LLM Gateway & Operations Platform
A secure internal platform that centralizes access to local LLM services, applies routing and credential controls, and turns multi-source usage and GPU telemetry into operational reporting.
LLM Gateway & Ops · Telemetry Pipeline
- Access Layer01
LLM Gateway
API Keys, Routing & GPU Mon
- 02
Read-Only Sync
Advisory Lock & Watermarks
- 03
PostgreSQL Marts
Minute Facts & Daily Rollups
- 04
Token Metering
Persisted vs Estimated
- 05
Ops Dashboard
Drill-Downs & Async Export
- 06
Weekly Reports
Scheduled PDF & HTML
Decoupled read-only analytics keeps heavy reporting queries completely isolated from live inference gateways.
Before
Teams used fragmented model endpoints, credential flows, and log scripts, making model adoption, usage attribution, and GPU capacity difficult to understand without touching production systems.
After
Unified model access and routing with auditable usage metering and a decoupled operations analytics layer, giving teams and supervisors a reliable view of service health, cost, and capacity.
Context & Background
- As internal LLM applications expanded across different gateways and GPU nodes, teams needed a consistent access layer while leadership needed reliable oversight of adoption trends, cost allocation, and capacity bottlenecks.
- Existing systems operated independently with separate logging formats, requiring a dedicated analytical store to correlate user demand with model serving health.
- The solution had to respect strict corporate security policies, air-gapped constraints, and read-only boundaries.
Constraints
- Zero-write access to source databases: upstream gateways and model servers must never be modified or subjected to analytical query spikes.
- Measurement integrity: token metrics must transparently distinguish true recorded tokens from calculated estimates to prevent misleading cost attributions.
- Production releases required versioned Docker image packages, checksum verification, and environment variable isolation.
Problem Solved
- Resolved blind spots in tracking company-wide Generative AI tool usage and model adoption across departments.
- Prevented reporting queries from causing latency spikes on active LLM serving instances.
- Standardized LLM usage accounting and performance attribution across diverse gateway systems.
My Role
- Designed the internal gateway and decoupled analytical architecture, PostgreSQL schema, and containerized deployment configuration.
- Implemented model access controls, routing, API key management, usage metering, and the operational reporting surface.
- Implemented read-only connector boundaries for SSH/SCP snapshot retrieval, database sync, and Prometheus/HTTP GPU metrics.
- Built the idempotent synchronization worker, background export processor, and automated weekly report generation service.
System Design
- Telemetry Collection: Scheduled workers fetch remote gateway database records, download SQLite snapshots via SCP, and poll GPU monitor endpoints.
- ETL & Normalization: Ingests into staging tables, validates SQLite schema integrity, deduplicates events via natural keys, and updates watermarks atomically.
- Analytical Storage: PostgreSQL stores minute-level facts for deep investigation and continuously maintains hourly/daily summary rollups.
- Supervisor Portal: FastAPI backend serves authenticated dashboard queries, dispatches asynchronous export jobs, and triggers weekly PDF compilations.
Implementation Highlights
- Designed atomic SQLite snapshot validation to prevent corrupted or partial file transfers from entering the analytics warehouse.
- Built an overlapping sync window mechanism with PostgreSQL advisory locks to guarantee idempotent ingestion across network retries.
- Established clear token provenance labeling (
persistedvsestimated) to preserve data honesty in user usage reports. - Automated recurring weekly HTML and PDF report delivery, including adoption graphs, top models, and GPU peak load analysis.
Outcome & Impact
- Provided management and platform engineers with unified, real-time visibility into internal LLM usage and hardware efficiency.
- Eliminated manual reporting labor while safeguarding production inference servers against analytical query overhead.
- Enabled proactive capacity planning by pinpointing cluster congestion and usage spikes across teams.