Running your own large language model isn't like spinning up a standard web server. You aren't just managing CPU and RAM; you're dealing with volatile GPU memory, massive token throughput, and the inherent unpredictability of generative AI. When you move away from a managed API and go the self-hosted route, the burden of reliability shifts entirely to your shoulders. If the model starts hallucinating or the latency spikes to ten seconds per token, there's no cloud provider to blame-it's your SRE practices that will determine whether your service stays up or crashes under the weight of a few concurrent users.

The biggest myth in AI operations right now is that the models can manage themselves. You might think an LLM could act as an autonomous on-call engineer, but real-world data shows otherwise. Recent tests by ClickHouse involving GPT-5 and OpenTelemetry data proved that autonomous root cause analysis (RCA) just isn't there yet. Even the most advanced models struggle to consistently outperform a human engineer who knows the system. The goal isn't to replace the SRE, but to build an observability stack that makes the SRE's life easier.

The Core Metrics That Actually Matter

In a traditional app, you look at request rates and error codes. With self-hosted LLMs, those are secondary. You need to look at the GPU and the queue. If you're using vLLM is a high-throughput serving framework for LLMs that optimizes GPU memory through PagedAttention, you have a goldmine of Prometheus metrics available. Don't just track if the pod is "running"; track how the model is breathing.

Focus on these four critical signals to avoid production disasters:

• Request Queueing: Track vllm_num_requests_waiting. If this number climbs, your users are staring at a loading spinner while their requests sit in a line. It's the first sign you need to scale.
• Active Processing: Monitor vllm_num_requests_running to understand your actual concurrency limits.
• GPU Cache Pressure: Watch vllm_gpu_cache_usage_perc. LLMs use a KV cache to remember the context of a conversation. If this hits 100%, the system will either crash or slow down to a crawl.
• Token Throughput: Use vllm_avg_generation_throughput_toks_per_s. This is your real "speedometer." If throughput drops while GPU usage is high, you might have a bottleneck in your weights loading or a noisy neighbor on the node.

Integrating LLMs into the SRE Workflow

Since we know LLMs can't autonomously fix your cluster, how should you actually use them? The shift is from "Autonomous Agent" to "Intelligent Assistant." Instead of letting a bot restart pods, use it to parse the noise. SREs are often buried in logs from Kubernetes is an open-source container orchestration system for automating software deployment and scaling, and the sheer volume of container events can be overwhelming.

A practical workflow involves feeding the LLM specific slices of data. For example, when a pod enters a CrashLoopBackOff, don't ask the LLM "Why is the app broken?" Instead, pipe the last 50 lines of the pod log and the current resource limits into the model. Ask it to summarize the error pattern and suggest a specific change to the requests or limits in your YAML. This turns a 20-minute manual log dive into a 30-second review.

Beyond logs, you can use models to correlate signals. If you see a spike in 500 errors coinciding with a GPU memory leak, an LLM can quickly synthesize these two disparate alerts into a single narrative: "GPU OOM event causing API gateway timeouts." It's not doing the discovery; it's doing the synthesis.

Moving Toward AI-Native Automation

We are starting to see a transition from manual SRE work to what the CNCF calls AI-native automation. This isn't about a chatbot in the CLI; it's about integrating models directly into the control loop of your infrastructure. While some of these are still emerging, they represent the future of self-hosting.

Keep an eye on these three capabilities:

1. Smart Sizing: Moving beyond static resource requests. Imagine a system that uses ML to analyze your LLM's memory usage patterns and automatically adjusts vertical scaling to optimize cost without risking an Out-Of-Memory (OOM) kill.
2. Autopilot Scaling: Traditional horizontal pod autoscalers (HPAs) are often too slow or too blunt for LLMs. AI-driven scaling looks at workload patterns and predicts surges before they hit the queue, adjusting replicas in real-time.
3. Pod Recovery AI: This goes a step beyond simple restarts. It analyzes the failure event, recognizes a known pattern (like a CUDA driver mismatch), and triggers a specific recovery action rather than just rebooting the container into the same error state.

The LLMOps vs. MLOps Divide

One of the biggest mistakes experienced SREs make is treating LLMOps is the operationalization of large language models, focusing on the deployment, monitoring, and maintenance of generative AI systems as if it were just a subset of MLOps is a set of practices that aims to deploy and maintain machine learning models in production reliably and efficiently. In traditional MLOps, you're often worried about data drift and retraining pipelines. In LLMOps, you're fighting a different battle.

The challenges are fundamentally different. You're dealing with Retrieval-Augmented Generation (RAG) pipelines, where the observability needs to extend into the vector database. You're managing context windows and prompt versions. If a model's output quality drops, it's not necessarily "drift" in the statistical sense-it could be a change in how the system retrieves documents or a subtle shift in the prompt template.

To succeed here, SREs need to expand their toolkit. You can't just be a Kubernetes expert; you need to understand the basics of tokenization, how KV caches work, and why GPU memory fragmentation happens. It's a fusion of systems engineering and AI internals.

Common Pitfalls in LLM Observability

Avoid the temptation to monitor everything with a "black box" approach. Many teams just track the API response time and call it a day. This is a recipe for failure. If your response time is slow, you won't know if it's because the GPU is throttled, the prompt is too long, or the vector DB is lagging.

Another common mistake is ignoring the "cost of observability." Running a separate LLM to monitor your production LLM can quickly double your compute costs. Be strategic. Use lightweight models or specialized tools like Openlit for tracing and quality tracking, and reserve the heavy-duty models for complex root cause synthesis during actual incidents.

Next Steps for Your Infrastructure

If you're just starting with self-hosted LLMs, don't try to build a fully autonomous system. Start by setting up a robust Prometheus scrape config for your serving framework. Get your alerts right-alert on queue growth and GPU cache saturation before you alert on generic pod restarts.

Once your metrics are stable, introduce an LLM-assisted log analysis workflow. Create a simple script or use a tool that pipes error logs to a model for synthesis. As your scale grows, look into the CNCF's roadmap for AI-native automation to see when Pod Recovery AI and Autopilot scaling become viable for your specific workload.