AI agents at scale – a fleet, not a product

Every product that calls a paid LLM emits Langfuse traces tagged with its product slug. That slug is also recorded on the owning row in the agent-fleet-dashboard, which reads Langfuse in near-real-time and rolls spend up per product and per customer. This is a house rule; a product cannot ship without it. The reason is compounding: without per-product attribution, a fleet of 13 products devolves into a single invoice, and every cost conversation restarts from "who spent what".

The dashboard itself is the proof that the pattern scales. Two-project Langfuse routing keeps client-billed and internal spend from ever blending. Silent-nil for uncatalogued products keeps a stray call from misattributing itself. Live spend reads in under 500ms with a 5-minute cache. When cost spikes happen, they surface within the business day rather than in the next invoice.

Rankings, scores, prices, and eligibility decisions are computed by deterministic code. LLMs sit alongside these as explanation, extraction, or drafting – never as the primary signal in a business decision.

• prospector's ICP scoring is pure functions with SHA-256 config-hash staleness detection. Missing inputs are first-class unscored state, never zero-defaulted. No LLM in the score.
• preservation's market-intel rank is a deterministic, multi-dimension weighted score, with the reasoning shown beside every score. Claude is used after the rank for opportunity analysis, never as a signal.
• fabrication-quoting uses Claude Sonnet 5 for supplier-PDF extraction; the quoting math downstream is pure.

The reason is explainability. A client asks "why is this HOT?" or "why did your engine miss this quote?" – the answer has to cite specifics that live in code, not an opinion that lives in a model.

Paid LLM and enrichment APIs (Firecrawl, Hunter, Perplexity, Scrapingdog, and Claude/OpenAI at volume) can spend a month's budget in a stray rerun. Every one is gated:

• preservation requires CLOUD_GATE_ACK=enabled as an environment variable before any paid endpoint is reached.
• agent-fleet-dashboard caps the Insights chat at 50 calls/day per user and $20/day per workspace, enforced via a Langfuse tag filter.
• Model choice is stratified: saga uses Opus 4.6 for user-facing chat, Sonnet 4.6 for batch drafts, Haiku 4.5 for cheap verification tasks. Every call is tagged with its model for post-hoc totals.

Confidentiality gates run in both directions. Client pipeline data does not enter the prompt in the first place, so the model cannot reveal it even if the leak-out check misses. See saga's two-sided gate: detectConfidentialLeak on the way out plus isConfidentialMetric + protectedClientNames() on the way in. The gate was designed in response to a real leak incident, and it is verified with the actual leaking-draft example in the test suite.

Individually, these rules read as engineering hygiene. Together they compound: the dashboard sees the spend, the deterministic scoring lets us defend a rank in a client review, the paid-API gate prevents disasters, and the safety gates keep client material out of drafts. A new product ships with all four attached from day one – because the standard is that a new product does not ship without them.

The fleet is not a portfolio of AI products; it is a portfolio operated by four rules that make an AI product safe to ship in the first place.