An embedded AI assistant for Gigover

Problem

Gigover wanted an AI assistant inside the web app. The spec I was handed assumed Perplexity sonar-pro, proxied through the Java backend, with per-user token metering, a "gas gauge" in the UI and a resale markup on top.

Most of that didn't fit. The assistant answers questions about a static domain: Gigover's own features and PMBoK methodology. Live web search buys nothing there. And the backend developer was about to go on leave, so anything that needed Java changes would sit blocked for weeks.

Constraints

• No Java backend work. It had to be something I could own end to end in TypeScript.
• The API key never reaches the browser.
• The limits have to be enforced on the server. The client can show what's left, but it doesn't get a say.
• A large, mostly fixed system prompt. That makes prompt caching matter more than the per-token output price.
• Multi-tenant data. Once the assistant could read projects and bids, it could never see something the user couldn't.

Architecture

I picked Gemini 2.5 Flash with grounding off. Output was roughly ten times cheaper than sonar-pro, it supports context caching (which Perplexity didn't), and with grounding off, the terms that only apply to grounded results don't apply. PMBoK belongs in the prompt anyway, so web grounding added nothing. Later, after an eval run, the agent moved to a newer Flash tier.

It's a Firebase onCall callable, not onRequest. Callables verify the caller's ID token for free. The older functions in the project were open onRequest handlers with a wildcard CORS header, which is the wrong template for a paid endpoint. The key lives in Secret Manager through defineSecret. The system prompt lives in the repo, so I can iterate on it without anyone on the backend.

When the assistant needs the user's data, the function forwards the caller's Firebase ID token to the Java API. The backend already accepts that token and opens a session, so authorisation stays where it already lives. I didn't re-implement it at the edge, which is how multi-tenant AI features leak data.

What shipped

• Streaming chat. A floating panel with stop, retry, copy, smart auto-scroll, a full-screen layout on mobile, and markdown rendering.
• Eval harness. Real questions with rubrics, answered by the production prompt and graded by a stronger model through structured output. It started at 8 questions and now runs in CI as its own workflow, only on prompt or eval changes. The last run I have notes on scored 22/22 with an average of 4.95/5.
• More capabilities. Jurisdiction awareness from the location's country, a bid comparison that turns several offers into one table, and a site photo turned into a work-order draft. The comparison shows and never recommends. The photo draft describes and never grades severity, and the JSON schema enforces both, not just the prompt.
• Rollout gate. A kill switch plus an email allowlist, checked on the server.
• Spend hardening. Quota is reserved before the paid call and refunded if it fails. There's a global daily ceiling as a hard stop, and client access to the RTDB instance holding the counters is denied entirely.
• Question logging. It goes to Cloud Logging with 30-day retention, and the panel tells users their questions are recorded.

I also designed the next step on paper: thirteen decisions for letting the agent write. Routing lives in code, and the model can downgrade a write to a draft but never upgrade it. Anything irreversible, or anything that notifies someone else, stays draft-only. A tool only gets promoted to direct writes after a live write-then-read-back. Separately, I worked through exposing the API as an MCP server over OAuth 2.1 with dynamic client registration.

What I'd do differently

• Check the stores before designing on them. I planned question logging in RTDB before I read its rules. Any signed-in user had read and write at the root, and RTDB rules cascade, so a private child node was impossible. Logging every user's question there would have exposed it to every other user.
• Check where the API key comes from. A key minted in AI Studio landed on an auto-created project with no billing, which means free tier, where prompts can be used for training. That has to be fixed before rollout.
• A judge that only sees question, rubric and answer can't verify product claims. It failed a good answer for "inventing" a real feature. It couldn't have caught a real invented one either.
• Read the whole prompt at once. Two sections contradicted each other, one saying to decline legal advice and one saying to orient users on law. You only ever read one at a time.
• Count before spending can't cap spend. The first quota counted after success and was writable from devtools. Reserve-then-refund should have been the first design.