Our Production AI Architecture
This is the architecture we reach for when we start a new AI product. It is not the only way to build, but it is battle-tested across the products we run for users in India and the World, and it is deliberately boring where boring buys reliability.
The shape: thin handler, fat background
Every AI feature has the same skeleton. A Next.js App Router route handler does validation, authorization, and a small synchronous write, then returns. The expensive LLM work runs either streamed to the user or in `after()`/a job. The user never waits on a route handler that is itself waiting on a model unless we are streaming.
``` client -> route handler (validate, authorize, write pending) -> after()/queue -> model call -> persist result client <- subscribe/poll for result (or streamed tokens) ```
RAG grounding: compute first, retrieve second
Our most important architectural opinion is that the model should reason over data we computed, not data it hallucinated. In GrahAI, the AI Jyotish chat does not 'know' astrology from its weights. We compute the user's chart deterministically — planetary positions with the Lahiri ayanamsa, houses, dashas, yogas — using our own validated engine, with timezone auto-derived from the birth coordinates. That structured chart becomes the retrieval context for the chat.
```js const chart = computeChart(birth); // deterministic, validated const context = serializeRelevantFacts(chart, userQuestion); const answer = await model.chat({ system: groundingInstructions, context, // facts, not vibes question: userQuestion, }); ```
The model's job is interpretation and language, not arithmetic. This makes outputs consistent, auditable, and far cheaper than stuffing a giant prompt and praying. The same principle applies elsewhere: ground on the computed tax figure, the real option payoff, the actual proposal inputs.
Model routing: cheap first, escalate on need
We route between a fast inexpensive model and a stronger one. The router is a small policy function, not a framework:
```js function pickModel(task) { if (task.kind === 'classify' || task.kind === 'extract') return CHEAP; if (task.tokensIn > BIG || task.kind === 'longform') return STRONG; return CHEAP; // default down, escalate deliberately } ```
We default down and escalate on purpose, because the failure mode of routing everything to the strong model is a quiet 5x on your bill. For some flows we run the cheap model first and only escalate if a validation check on its output fails — a cascade, not a coin flip.
Retries with backoff and a hard ceiling
Model APIs rate-limit and occasionally 5xx. Every model call goes through a wrapper with exponential backoff, jitter, and a max-attempts ceiling. Crucially, retries are bounded by the function's remaining lifetime — there is no point retrying past the moment the serverless function dies. On terminal failure we mark the job failed with the error, so the self-healing read path or a human can act, rather than leaving it pending forever.
Caching: the cheapest token is the one you do not send
We cache at two layers. Prompt caching reuses large stable prefixes (system instructions, grounding boilerplate) so we are not re-billing for the same context every call. Output caching keys on a hash of the full input — for deterministic, repeatable generations (a report for the same inputs, an OG card) the second request is a cache hit, not a model call. For anything user-specific and stable, caching is the single biggest cost lever we have.
Evals: a harness, not vibes
We do not ship prompt changes on gut feel. We keep deterministic eval scripts that run a fixed set of inputs through the pipeline and assert on structure (every report has its required pages, no runaway repetition, no empty sections) plus a sampled empirical check on content quality. There is also a daily quality cron that audits live outputs and emails a digest. A prompt edit that regresses the eval does not ship.
Queueing without a queue
On serverless we have no always-on worker, so 'queue' is a pattern, not a service: write a pending record, process it via `after()` for tail work or a cron-drained collection for heavier batches, and let the read path self-heal anything dropped. For scheduled work (reminders, renewals, audits) a cron-triggered route is the worker. It is less elegant than a dedicated queue, but it has near-zero idle cost and no infrastructure to babysit.
Secrets, geo, and the edges
Secrets live in the platform's env, read at module scope, never shipped to the client. Geo-aware behavior (currency, sometimes content) is decided server-side from a geo signal plus a cookie, with a pre-paint cookie read so visitors never see the wrong currency flash. Runtime selection matters too: node runtime where we need fonts or Chromium for PDF rendering, edge only where the function is light and latency-sensitive — bundled fonts on edge tend to break, so OG and PDF stay on node.
The throughline
Compute what you can, ground the model on it, route to the cheapest model that clears the bar, cache aggressively, bound your retries, and gate every change behind an eval. That is the whole architecture. The intelligence is in the data you feed the model, not in the model alone.
If you want a reference architecture like this designed around your domain, GrahAI Systems can help — reach us at support@grahai.com.