Building a Local AI Tool for Kubernetes Cost Optimization

Kubernetes has become the default platform for modern application delivery, but its cost story remains messy. Most teams use Kubecost to instrument cluster spend — and Kubecost does that job well — yet the moment you open the dashboard, you are staring at hundreds of rows of namespace, workload, and resource allocations with very little guidance on what actually matters.

FinOps engineers end up doing the same exercise every month: reading the tables, eye-balling idle workloads, identifying over-provisioning, building a list of recommendations, and writing an executive summary. The work is repetitive, time-consuming, and the quality varies with the experience of the engineer doing it.

What if a local AI model could do the analysis for you, in the customer’s own environment, without sending a single line of cluster data to an external provider? That question led to Steward.

A completed Steward cost-analysis report against a test EKS cluster, showing the at-a-glance view, executive summary, and severity-graded findings.

The problem with manual cost reviews

A typical monthly review involves steps that don’t scale: reading 200-row allocation tables, identifying idle workloads and PVC waste by visual inspection, cross-referencing efficiency ratios against benchmarks, building a prioritised fix list by hand, and writing an executive summary. For a single cluster this takes half a day; across an MSP portfolio the effort compounds fast.

SaaS tools like CAST AI, Spot.io, and Vantage automate cost analysis, but they require pushing cluster data — namespace names, workload identifiers, cost figures, sometimes auth tokens — into the vendor’s environment. For regulated industries (healthcare, finance, government), that is a hard stop. External LLM providers have the same problem: even anonymised data leaving the customer environment can derail an evaluation.

What we built

Steward is an open-source, local-first FinOps reporting tool for Kubernetes. It connects to a customer’s Kubecost installation, runs analysis with a locally hosted LLM, and produces executive-grade cost reports through a modern web UI.

The key design constraint: no cluster cost data, namespace names, workload identifiers, or auth tokens ever leave the customer’s infrastructure. The LLM runs on a local Ollama daemon, the vector store on local ChromaDB, the database on local SQLite or Postgres. That’s the entire network surface.

System architecture

Six containerised services, all on the customer’s own Docker host:

  +---------------------+        +------------------+        +-----------------+
  |  Next.js frontend   | -----> |  FastAPI backend  | -----> |  Kubecost API   |
  |  (TS + shadcn/ui)   |        |  (async)          |        |  (per env)      |
  +---------------------+        +--------+----------+        +-----------------+
                                          |
                       +------------------+----------------+
                       v                  v                v
               +--------------+   +--------------+  +--------------+
               |   SQLite/PG  |   |    Ollama    |  |   ChromaDB   |
               |  (history)   |   |   (LLM)      |  |   (RAG)      |
               +--------------+   +--------------+  +--------------+

• Frontend — Next.js 15, TypeScript, Tailwind v4, shadcn/ui.
• Backend — FastAPI, Pydantic v2, SQLAlchemy 2.0 async.
• Worker — arq job runner executing the scan pipeline.
• Ollama — Local LLM daemon (default: qwen2.5:7b-instruct).
• ChromaDB — Vector store seeded with FinOps reference material.
• Redis — Job queue and cache.

How a scan works

When the user clicks Scan, the worker runs an eight-phase pipeline. Each phase commits to the database so the frontend’s polling sees live progress: mark RUNNING → concurrent Kubecost fetches (allocation, assets, savings) → build a structured digest with grounding fields → retrieve RAG context from ChromaDB → send system prompt + digest + RAG snippets to Ollama → post-LLM validation → enrich findings with dollar impacts → persist the report.

The clever bit: the grounded-LLM pattern

Anyone who has tried to use a 7B-parameter LLM for structured analysis has hit the same problem: the model hallucinates. It claims “no idle workloads” when the input has four. It calls a 15% efficiency score “healthy.” Steward solves this with the grounded-LLM pattern — a three-step contract between the preprocessor, the prompt, and a deterministic validator.

Step 1 — Grounding fields. The preprocessor computes a digest with bucketed thresholds the model must use verbatim:

{
  "cluster_scale": "trivial",          // trivial | small | production
  "efficiency_grade": "critical",      // healthy | mediocre | poor | critical
  "analysis_hints": {
    "idle_workload_count": 3,
    "over_provisioned_count": 0,
    "efficiency_grade": "critical",
    "cluster_scale": "trivial"
  },
  "cluster_efficiency": { "cpu": 0.037, "memory": 0.825, "overall": 0.126 }
}

“trivial” means <$50/mo run-rate. “critical” means 7×+ over-provisioning. These words become the vocabulary the LLM must use in its prose.

Step 2 — Grounded system prompt. The prompt enforces a strict contract: use exact grade and scale names from the digest, never describe a critical-grade cluster as “healthy,” and forbid AI filler phrases like “leverage” or “synergy.” Each finding must include a digest_reference pointer so the worker can resolve concrete dollar impacts.

Step 3 — Deterministic validator. After the LLM returns, a Python validator checks for negation contradictions, healthy-downplay phrases, boilerplate recommendations, severity violations on trivial clusters, and missing dollar impacts. Violations trigger a single repair round. If the model still fails, violations are logged but the report persists — a flawed report with a warning beats no report.

What the output looks like

Every report opens with cards summarising cluster scale, per-resource efficiency (CPU / memory / overall with grade-driven colouring), and signal counts. A stacked bar shows the namespace cost breakdown — the “read the room in five seconds” view.

Below that, the LLM-written executive summary uses the digest’s exact grade names. Findings are sorted by severity then dollar impact, each with a severity-coloured border, a specific workload identifier, a concrete recommendation, and the dollar impact attached.

The Cost Analysis report with efficiency grade, signal counts, and severity-graded findings naming specific deployments. Multi-environment dashboard with at-a-glance status and aggregated metrics. Reports page showing cost trend over time and sortable scan history.

Manual workflow vs. Steward

Getting started

Steward runs entirely from Docker Compose:

git clone https://github.com/Deepanshu846/Steward_Scanner.git
cd Steward
cp .env.example .env

# Generate a SECRET_KEY:
python3 -c "from cryptography.fernet import Fernet; print(Fernet.generate_key().decode())"
# Paste output into .env as SECRET_KEY=...

docker compose up -d
docker compose exec ollama ollama pull qwen2.5:7b-instruct

Then open http://localhost:3000, add a Kubecost environment, and click Scan. You’ll need Docker with Compose v2, 8 GB RAM allocated to Docker, ~10 GB free disk, and a reachable Kubecost installation on an AWS EKS cluster (any v2.x).

Conclusion

Kubernetes cost optimisation does not have to mean reading endless allocation tables, and AI-driven FinOps does not have to mean shipping cluster data to a third party. By combining a structured digest, a grounded system prompt, a deterministic validator, and a locally hosted LLM, Steward demonstrates that small open-source models can produce reliable, executive-grade FinOps reports — entirely on customer infrastructure.

For MSPs and platform teams, this approach standardises a typically artisanal process and opens conversations with regulated-industry clients who previously could not adopt SaaS FinOps tools. The full source is on GitHub.

Want to take Kubernetes cost optimisation to the next level?

Don’t just read about local AI for FinOps — experience it. Explore how TO THE NEW helps teams optimise and modernise their cloud at tothenew.com, or schedule a call with us today.

Want the technical details? The project is open-source on GitHub. Follow us for more engineering write-ups, and if you have questions or ideas, leave a comment — I’d love to hear how you handle Kubernetes FinOps on your team.