In complex software systems, our traditional definition of operational health has always been comfortably binary. For over a decade, site reliability engineering (SRE) teams have relied on the industry-standard ‘Four Golden Signals’ — latency, traffic, errors and saturation — as the ultimate truth of platform stability. If our API-response times are hovering at sub-100 ms, network throughput is steady, CPU cores aren’t pegged and the HTTP 500 error rate is flatlined at zero, we sleep soundly. We check our Grafana dashboards, see an entirely green pasture and assume that our platform is delivering flawless value to the business.
Then came production AI.
With organizations rapidly transitioning from deterministic, code-driven microservices to non-deterministic, LLM-powered applications, this foundational telemetry framework is facing a quiet crisis. In an AI-driven ecosystem, a system can be structurally flawless while failing functionally. An API gateway can return a crisp HTTP 200 OK in record time, yet the payload it carries could be a hallucinated financial projection, an injection exploit or a toxic output that violates compliance. The infrastructure is entirely healthy, but the system is broken. To build truly trustworthy AI at scale, platform and site reliability engineers must look beyond hardware and network states and evolve our telemetry for a non-deterministic world.
Decoding the AI Blindspot: The Green Dashboard Paradox
The core challenge of traditional telemetry in an AI world comes down to the concept of determinism. Conventional software architectures operate on absolute rules where, given Input A and Condition B, the application will always produce Output C. If it doesn’t, a distinct exception is thrown, a 5xx error code is emitted and an on-call engineer is paged.
Generative AI, retrieval-augmented generation (RAG) pipelines and autonomous agent frameworks break this paradigm completely. These systems are inherently probabilistic. Since models rely on high-dimensional semantic spaces and complex vector retrieval, the same input can yield entirely different outputs across sequential requests.
This introduces what I call the “Green Dashboard Paradox.” Consider a high-transaction financial enterprise platform processing automated customer queries. A traditional SRE dashboard monitoring the service shows a pristine state:
- Latency: Minimal
- Traffic: Well within standard thresholds
- Errors: Zero dropped packets or server exceptions
- Saturation: GPU and memory utilization perfectly optimized
However, beneath the surface, the system is failing contextually. A slight shift in the vector database’s embedding space has caused the retrieval mechanism to fetch outdated data, causing the model to hallucinate incorrect loan rates for thousands of users. Since the transport layer successfully delivered the data without crashing, traditional infrastructure monitoring remains completely blind to the failure. We are validating that the pipes aren’t leaking, but we have no idea if the water flowing through them has turned toxic.
Where the Classic SRE Model Loses Its Grip
To understand how to fix this, we have to look closely at where the classic Google SRE handbook breaks down when interacting with LLMs and inference clusters:
- The Misleading Nature of Latency: In classic web applications, high latency means a degraded user experience, which is usually fixed by scaling up instances or optimizing database queries. In LLM applications, total response latency is highly variable based on prompt size and token length. A long response isn’t necessarily an unhealthy one. Conversely, a blazing-fast response could indicate that a safety filter immediately aborted the query, meaning low latency could actually signal a high user rejection rate.
- The Saturation Illusion: Traditional saturation measures CPU, memory and disk I/O. In AI infrastructure, workloads live on GPUs. GPU memory (VRAM) behaves fundamentally differently because it is often aggressively pre-allocated by inference engines, such as vLLM or Hugging Face TGI, to optimize performance. A traditional monitoring agent looking at VRAM will report 95% saturation constantly, rendering it useless as a reactive alerting trigger.
- The Vanification of Error Codes: The classic error signal relies heavily on protocol-level telemetry such as HTTP status codes and gRPC status tracking. In an AI pipeline, an application failure often occurs at the semantic or alignment layer. If an LLM generates a response containing proprietary system instructions because of a prompt injection attack, it is a catastrophic security failure. However, to your load balancer, it is just a highly successful text transmission.
Operationalizing Trust: The New Evolutionary Telemetry
We don’t need to throw away the Four Golden Signals entirely. Rather, we must treat them as the baseline infrastructure layer and build a new, intelligent tier of telemetry on top of them. To engineer trustworthy, resilient AI systems, the telemetry pipeline must ingest semantic, structural and alignment signals natively.
When designing a modern observability stack for non-deterministic software, teams should look to instrument four alternative ‘Golden Signals of AI Architecture’:
| Classic SRE Metric | The AI Telemetry Alternative | What it Safely Measures in Production |
| Latency | Time to First Token (TTFT) | It measures the duration between request submission and the arrival of the initial streaming token. This isolates model inference lag from total delivery time. |
| Traffic | Token Velocity and Throughput | It tracks the volume of input (prompt) versus output (completion) tokens and is critical for forecasting provider cost, managing memory buffers and preventing rate-limiting. |
| Errors | Guardrail Intervention Rate | It tracks how frequently secondary safety layers (such as Llama Guard or NeMo) intercept, filter or rewrite inputs and outputs before they hit the user. |
| (Generic) | Semantic Drift and Faithfulness | It measures the statistical degradation of output vector embeddings over a rolling window compared to known-good baselines to catch silent model decay. |
By feeding these primitives into open standard frameworks such as OpenTelemetry (OTel), platform engineers can track a transaction seamlessly as it moves from user interaction, travels through API routes, runs queries across a vector store such as Pinecone or Milvus and executes inference on a GPU node.
Putting it to Work: Building AI-Focused SLOs
A meaningful observability strategy doesn’t stop at gathering data. It requires defining explicit service level objectives (SLOs) that align system performance with true business utility and trust.
When working with non-deterministic systems, your service level indicators (SLIs) must shift from pure technical uptime to semantic compliance.
For instance, instead of establishing an objective stating that 99.9% of API requests must return an HTTP 200 within 200 ms, a mature platform engineering team operating an AI service should define metrics focused on output safety and retrieval accuracy:
- Guardrail Health SLI: The percentage of transactions that successfully navigate the alignment pipeline without triggering a safety policy violation or malicious prompt injection filter.
- Target SLO: 99.5% of monthly traffic passes validation without structural or safety intervention.
- RAG Context Faithfulness SLI: The mathematical cosine similarity score between the retrieved grounding documents and the generated completion, evaluated via automated, asynchronous evaluation LLMs over a rolling 5-minute window.
- Target SLO: 98% of generated answers must maintain a faithfulness score above 0.85, alerting the SRE rotation if vector database drift occurs.
Conclusion: The Infrastructure of Integrity
The shift to AI does not make the discipline of SRE obsolete. Instead, it dramatically elevates it. The ultimate goal of the platform engineer is no longer to simply ensure that computing power is accessible and data packages move quickly from point A to point B. Our responsibility has expanded to guarding the functional integrity and safety of the system itself.
By implementing advanced telemetry pipelines that monitor past the boundaries of traditional transport protocols, engineering teams can safely embrace the immense power of non-deterministic software. When we proactively measure token velocity, time to first token and guardrail interventions, we bridge the gap between abstract AI safety and hard platform engineering. The result is a robust, modern observability culture where our dashboards aren’t just superficial indicators of hardware health, but are true reflections of systemic trust, accuracy and enterprise resilience.