Skip to content
Engineering

How to add AI to an existing SaaS product: skip the chat widget, build this first

22 min read
GammaEdge Team

The first AI feature should be invisible

The chat widget is the wrong first AI feature for almost every SaaS product. Teams ship it because it's the fastest thing to demo, then watch customers ask it two questions and go back to the support form. Your first AI feature should be one your customers never notice: a reply draft already waiting in a ticket when the agent opens it, an invoice explained in plain language, a test suite that writes its own first drafts. It should sit on data you already have, do one job, and stay out of sight until you can prove it works.

This post is how we'd do that in a product that already has paying customers. It covers what we'd build, what we'd refuse to build, the architecture, the code for the parts teams usually get wrong, and the math on cost and latency.

What we refuse to build first

We turn these down as a first feature, even when a client asks for them.

  1. •A general "ask anything" chatbot. It has no defined job, so there's no test that tells you it works. It answers questions your docs don't cover, and it invents answers when retrieval comes back empty.
  2. •Anything that writes to billing. No refunds, credits, plan changes or cancellations by the model. A wrong sentence in a draft costs an agent a minute. A wrong refund costs money, and you can't take it back quietly.
  3. •Customer-facing auto-replies on day one. Auto-reply comes after weeks of drafts that agents sent unedited, per ticket type. Not before.
  4. •An agent with write access across your product. Multi-step agents that click through your app with admin rights multiply every failure mode. Start with one read-only tool.
  5. •A fine-tuned model before you have an eval set. You can't tell whether the fine-tune helped if you have nothing to score it against.
  6. •An "AI insights" dashboard nobody asked for. If you can't name the decision a person makes differently after reading it, don't build it.

How to pick the first feature

We score each candidate on four questions:

  1. •Does the data already exist? Ticket history, help docs, invoices, product events and test suites all count.
  2. •Is there a right answer you can check? Resolved tickets with the reply an agent actually sent give you ground truth.
  3. •What does a wrong answer cost? Drafts reviewed by a person are cheap to get wrong. Money movements are not.
  4. •Does a person already do this work? Then you have a baseline to compare against and a reviewer during rollout.

A feature that scores well on all four is usually boring. Build the boring one.

The candidates, in the order we'd build them

Tier-1 support drafts

This is our default first feature. When a ticket arrives, the system retrieves similar resolved tickets and the relevant help articles, reads the customer's plan and recent activity, and writes a draft into the ticket. The agent edits and sends it. Customers never see raw model output.

You get a reviewer on every output, and every edit is a labelled example of where the model is wrong. After a few weeks you can sort ticket types by how often agents send the draft unchanged. The top of that list is where auto-reply starts.

We haven't shipped a support bot, but we built a proposal evaluation tool for a client that runs formal vendor tenders, and it uses the same draft-then-review pattern. The tool checks each vendor proposal against the tender's eligibility criteria and drafts a score for each one. Evaluators can override any score or text, but they have to give a reason, and every change is kept in an audit trail. Those reasons work like agent edits in a support queue: each one is a labelled example of where the model was wrong. One early problem came from the tender documents themselves. When a scoring table listed criteria but no points, the model merged four criteria into one and gave it the whole technical weighting of 60 marks. We reproduced it, changed how extraction handles tables without points, and shipped the fix.

Billing answers, read-only

Billing tickets repeat: why was I charged this, when does my plan renew, where is my invoice. The answers sit in your billing tables, so the model can quote facts instead of guessing. The model reads and explains. A refund request becomes a handoff to a person, with the details already collected.

Onboarding help

Good candidates: mapping a customer's CSV columns to your fields during import, filling a setup form from a pasted config, suggesting the next setup step from what the account has already done. We rank it third because activation moves for many reasons. You need a comparison group and enough signups to read it, so it takes longer to prove.

AI test automation

This is internal, so a wrong output never reaches a customer. If slow releases hurt you more than support load, start here. The useful work: drafting test cases from user stories, writing first-pass end-to-end tests for flows with no coverage, and grouping failed runs by root cause.

The catch is that a generated test asserts what the app does today, bugs included. A person has to review each test against intended behavior. Track flaky failures too, because a suite that fails at random trains your team to ignore it.

The architecture

Here is the shape we'd build for support drafts and billing answers. The feature flag decides the path. Permissions resolve before retrieval, never after. Everything the model sees and says goes to a log that feeds the eval set.

Retrieval that can't cross tenants

The most dangerous bug in a multi-tenant AI feature is retrieval that fetches another tenant's chunk. Asking the model to ignore it doesn't work. The filter has to sit in the query, and a second layer in the database should catch you when someone forgets it.

Schema and the database-level guard:

CREATE EXTENSION IF NOT EXISTS vector;

CREATE TABLE doc_chunks (
    id            bigserial PRIMARY KEY,
    tenant_id     uuid        NOT NULL,
    source_id     text        NOT NULL,
    visibility    text        NOT NULL CHECK (visibility IN ('tenant', 'restricted')),
    acl_group_ids uuid[]      NOT NULL DEFAULT '{}',
    body          text        NOT NULL,
    embedding     vector(1536) NOT NULL,
    updated_at    timestamptz NOT NULL DEFAULT now()
);

CREATE INDEX ON doc_chunks (tenant_id);
CREATE INDEX ON doc_chunks USING hnsw (embedding vector_cosine_ops);

ALTER TABLE doc_chunks ENABLE ROW LEVEL SECURITY;
CREATE POLICY tenant_isolation ON doc_chunks
    USING (tenant_id = current_setting('app.tenant_id')::uuid);

Row-level security doesn't apply to the table owner unless you add FORCE ROW LEVEL SECURITY, so the app should connect as a separate role. The query, in Python with psycopg 3 and pgvector-python:

from dataclasses import dataclass
from uuid import UUID

import numpy as np
import psycopg
from pgvector.psycopg import register_vector


@dataclass(frozen=True)
class RequestContext:
    tenant_id: UUID
    user_id: UUID
    group_ids: tuple[UUID, ...]      # resolved server-side from the session
    permissions: frozenset[str]


SEARCH_SQL = """
SELECT id, source_id, body, embedding <=> %(qvec)s AS distance
FROM doc_chunks
WHERE tenant_id = %(tenant_id)s
  AND (visibility = 'tenant' OR acl_group_ids && %(group_ids)s::uuid[])
ORDER BY embedding <=> %(qvec)s
LIMIT %(k)s
"""


def search_chunks(conn: psycopg.Connection, ctx: RequestContext,
                  query_embedding: np.ndarray, k: int = 4) -> list[dict]:
    register_vector(conn)
    with conn.transaction():
        # Scope this transaction for the RLS policy. set_config(..., true) is SET LOCAL.
        conn.execute("SELECT set_config('app.tenant_id', %s, true)", (str(ctx.tenant_id),))
        # pgvector >= 0.8.0: keep scanning the HNSW index when filters drop rows.
        conn.execute("SET LOCAL hnsw.iterative_scan = relaxed_order")
        rows = conn.execute(SEARCH_SQL, {
            "qvec": query_embedding,
            "tenant_id": ctx.tenant_id,
            "group_ids": list(ctx.group_ids),
            "k": k,
        }).fetchall()
    return [{"id": r[0], "source_id": r[1], "body": r[2], "distance": float(r[3])} for r in rows]

Two details matter here. The group IDs come from your session, never from the request body or the model. And the iterative_scan line is there for a reason: with an approximate index, pgvector applies the WHERE clause after the index scan. The pgvector README gives the example that a filter matching 10% of rows, with the default hnsw.ef_search of 40, returns about 4 rows on average. Without iterative scans, a small tenant in a big shared index gets thin results and worse answers.

On the proposal evaluation tool, projects played the part tenants play here. Search and chat were scoped by project membership: every search query carried a project filter, and every chat endpoint checked that the user was a member of the project. A security review then found one route that didn't. The report export checked only that the user was logged in, so any signed-in user could download the evaluation report for any project. The fix shipped, and it reuses the same membership check the chat endpoints use. Everyone checks the search path. The routes around it, like exports, are easier to miss.

Billing lookups as a read-only tool call

For billing, the model gets one tool, and that tool can only read. There is no issue_refund tool, so no prompt can talk the model into calling one. The tool schema uses strict structured outputs so arguments arrive in a known shape, and the server checks them again anyway.

import json

from openai import OpenAI
from pydantic import BaseModel, Field, ValidationError

client = OpenAI()

GET_INVOICE_TOOL = {
    "type": "function",
    "function": {
        "name": "get_invoice",
        "description": "Read one invoice belonging to the current customer. Read-only.",
        "strict": True,
        "parameters": {
            "type": "object",
            "properties": {
                "invoice_number": {
                    "type": "string",
                    "description": "Invoice number as shown to the customer, e.g. INV-20418",
                }
            },
            "required": ["invoice_number"],
            "additionalProperties": False,
        },
    },
}


class GetInvoiceArgs(BaseModel):
    invoice_number: str = Field(pattern=r"^INV-\d{1,10}$")


def get_invoice(conn, ctx: RequestContext, args: GetInvoiceArgs) -> dict:
    if "billing:read" not in ctx.permissions:
        return {"error": "not_permitted"}
    row = conn.execute(
        """SELECT number, status, total_cents, currency, period_start, period_end
           FROM invoices WHERE tenant_id = %s AND number = %s""",
        (ctx.tenant_id, args.invoice_number),
    ).fetchone()
    if row is None:
        return {"error": "not_found"}
    number, status, total_cents, currency, start, end = row
    return {"number": number, "status": status, "total": f"{total_cents / 100:.2f} {currency}",
            "period": f"{start:%Y-%m-%d} to {end:%Y-%m-%d}"}


SYSTEM = ("You answer billing questions for one customer. Use get_invoice for facts. "
          "You cannot change plans or issue refunds. If asked, say a teammate will follow up.")


def answer_billing_question(conn, ctx: RequestContext, question: str, model: str) -> str | None:
    messages = [{"role": "system", "content": SYSTEM}, {"role": "user", "content": question}]
    for _ in range(3):  # hard cap on tool round trips
        msg = client.chat.completions.create(
            model=model, messages=messages, tools=[GET_INVOICE_TOOL]
        ).choices[0].message
        if not msg.tool_calls:
            return msg.content
        messages.append(msg.model_dump(exclude_none=True))
        for call in msg.tool_calls:
            try:
                if call.function.name != "get_invoice":
                    raise ValueError("unknown tool")
                result = get_invoice(conn, ctx, GetInvoiceArgs.model_validate_json(call.function.arguments))
            except (ValidationError, ValueError):
                result = {"error": "invalid_request"}
            messages.append({"role": "tool", "tool_call_id": call.id, "content": json.dumps(result)})
    return None  # no answer within the cap: hand off to a person

The tenant filter in get_invoice comes from ctx, not from the model's arguments. The model only names an invoice number. If it names one from another account, the query returns nothing.

RAG or fine-tuning

For a first feature, use retrieval. Our reasoning:

  • •Your facts change. Plans, prices, docs and invoices update weekly. With retrieval, you fix a help article and the next answer reflects it. With a fine-tune, you retrain.
  • •Permissions live in the query. A retrieval query filters by tenant and group on every request. Model weights have no idea who is asking.
  • •Fine-tuning a shared model on customer tickets can leak data between tenants. Train one model on tenant A's tickets and it can repeat tenant A's details to tenant B. We won't do it without per-tenant models or aggressive scrubbing, and usually neither is worth it.
  • •You can trace a retrieval answer back to the chunks it used. You can't trace a fine-tune's answer back to its training rows.

Fine-tuning is worth it later, on narrow jobs with labelled data: a ticket classifier, a router, a fixed output format. Your shadow-mode logs become that labelled data.

pgvector or a dedicated vector database

If your product already runs on Postgres, start with pgvector. The vectors sit next to the ACL tables, so the permission filter is a WHERE clause and a join, inside the same transaction, behind the same row-level security and the same backups. A dedicated vector database adds a second copy of your permission data and a sync job to keep it current. When that sync lags, a user who lost access to a document can still retrieve it.

We'd move to a dedicated store when measurements on your own data say so: recall falls on filtered queries even with iterative scans, the HNSW index no longer fits in memory on a database you can afford, or vector writes start competing with your transactional load. Before moving, try what pgvector's README suggests for multi-tenant data, list partitioning by tenant, so one big tenant's vectors stop hurting a small tenant's recall.

Per-request cost, worked through

The formula:

cost per request = (input_tokens × input_price + output_tokens × output_price) / 1,000,000
monthly cost     = requests per month × cost per request

Prices are per million tokens and change often, so take them from your provider's current pricing page. Here is a worked example with token counts we made up for illustration. Measure your own.

Part of the prompt (example)Tokens
System prompt and instructions600
4 retrieved chunks × 400 tokens1,600
Ticket text and account context400
Input total2,600
Draft reply (output)300

At an example volume of 20,000 tickets a month, all on the large model, that is 52 million input tokens and 6 million output tokens a month.

Two observations from the table. Retrieved chunks are the biggest line, so k is your biggest cost lever: dropping from 4 chunks to 2 cuts input by 800 tokens, about 31% of input. Check it against your eval set first, since fewer chunks can mean worse answers. Second, the system prompt is identical on every call, so if your provider offers prompt caching, put the fixed text first.

Now add routing. A small model classifies every ticket first, with about 500 input and 5 output tokens. Suppose, as an example, 60% of tickets are routine intents the small model can draft well, and 40% go to the large model:

  • •Small model: 20,000 × 500 for routing, plus 12,000 × 2,600 for drafts, is 41.2 million input tokens. Output is 0.1 million for routing plus 12,000 × 300, about 3.7 million.
  • •Large model: 8,000 × 2,600 is 20.8 million input tokens and 8,000 × 300 is 2.4 million output tokens.

Large-model tokens fall by 60%. Whether total cost falls depends on the price ratio. As an assumption, say the small model's per-token price is a tenth of the large model's. Check the real figures. Then the small model's 41.2 million input tokens cost the same as 4.1 million large-model tokens. Total input spend then equals about 24.9 million large-model tokens instead of 52 million. Your eval set decides the 60% figure: route a ticket type to the small model only if it passes on that type.

The latency budget

Split features by whether someone is waiting.

Support drafts have no one waiting. Generate the draft from the ticket-created webhook in a background job, and it's ready before the agent opens the ticket. Latency hardly matters here. Cost and quality do.

An in-app billing answer has a person waiting. Write down a budget per step before you build, then measure each step in staging. A budget sheet looks like this:

StepWhat to measure
Resolve session, tenant and permissionsShould be a cached lookup, not a fresh query
Embed the questionOne embedding call
Filtered vector searchDatabase time with your real tenant sizes
Model call 1, decide on the toolTime to the tool call
Invoice lookupOne indexed query
Model call 2, write the answerTime to first token, then streaming

The expensive line is the second model call. A tool call costs a full extra round trip. If most billing questions are about the latest invoice, fetch it with plain code before the model call and put it in the prompt. You pay a few hundred input tokens and save a round trip on most requests. Stream the answer so the user sees words early, and set a timeout that falls back to a link to the invoice page.

Build it into the product, not a bolt-on widget

A third-party chat widget is quick to install. It reads your public docs and nothing else. It doesn't know the user's plan, invoices, usage or permissions, so it can't answer the questions that fill your queue. It runs on its own auth, and your customers' questions go to another vendor.

Building in the product is more work. You need the retrieval layer, the permission plumbing and the logging above. In return the AI sees what the user can see, its output lands in screens people already use, and your team owns the code and prompts. A widget is fine for a public docs page on your marketing site. Inside the product, build it yourself.

An eval harness over your old tickets

Before launch, build a test set from resolved tickets. Each case keeps the question, a snapshot of the account facts at that time, the facts a correct answer must mention, and anything it must not say. The harness runs your drafting function against a fake backend seeded from the snapshot, so evals never touch live data.

import json
import re
import sys
from collections import defaultdict
from dataclasses import dataclass, field
from typing import Callable

FORBIDDEN = [
    r"\brefund (has been|was|will be) (issued|processed)\b",
    r"\bI(?:'ve| have) (cancel+ed|upgraded|downgraded)\b",
    r"\bguarantee\b",
]


@dataclass
class Case:
    ticket_id: str
    category: str
    question: str
    account_facts: dict
    must_mention: list[str]
    must_not: list[str] = field(default_factory=list)


def grade(draft: str, case: Case) -> list[str]:
    failures = [f"missing: {fact}" for fact in case.must_mention if fact.lower() not in draft.lower()]
    failures += [f"forbidden: {p}" for p in FORBIDDEN + case.must_not if re.search(p, draft, re.IGNORECASE)]
    return failures


def run(path: str, draft_fn: Callable[[Case], str], min_pass_rate: float) -> None:
    totals: dict[str, list[int]] = defaultdict(lambda: [0, 0])
    failed = []
    with open(path) as f:
        for line in f:
            case = Case(**json.loads(line))
            errors = grade(draft_fn(case), case)
            totals[case.category][1] += 1
            if errors:
                failed.append((case.ticket_id, errors))
            else:
                totals[case.category][0] += 1
    for category, (ok, n) in sorted(totals.items()):
        print(f"{category:<24} {ok}/{n}")
    for ticket_id, errors in failed[:20]:
        print(ticket_id, "; ".join(errors))
    passed = sum(ok for ok, _ in totals.values())
    total = sum(n for _, n in totals.values())
    if total == 0 or passed / total < min_pass_rate:
        sys.exit(1)

Run it in CI on every prompt change and every model version change, and set min_pass_rate to the last accepted baseline. These string checks catch the expensive failures: wrong amounts, missing dates, promises your team wouldn't make. Add a model-graded check for tone later if you want one, but keep reading a sample of failures by hand every week. The per-category output matters more than the total, because it tells you which ticket types are ready for the small model or for auto-reply.

Shadow mode behind a feature flag

Shadow mode runs the feature on real traffic, logs what it would have said, and shows nothing. When the ticket resolves, you compare the shadow draft with the reply the agent actually sent. That comparison is your best evidence before any customer sees an output.

import logging
from enum import Enum

log = logging.getLogger("ai_drafts")


class FlagState(str, Enum):
    OFF = "off"
    SHADOW = "shadow"
    ON = "on"


def flag_state(conn, tenant_id, key: str) -> FlagState:
    row = conn.execute(
        "SELECT state FROM feature_flags WHERE tenant_id = %s AND key = %s", (tenant_id, key)
    ).fetchone()
    return FlagState(row[0]) if row else FlagState.OFF


def on_ticket_created(conn, ticket, ctx: RequestContext) -> None:
    state = flag_state(conn, ctx.tenant_id, "ai_ticket_draft")
    if state is FlagState.OFF:
        return
    try:
        draft = draft_reply(conn, ticket, ctx)  # retrieval + model call, with its own timeout
    except Exception:
        log.exception("draft failed", extra={"ticket_id": ticket.id})
        return  # the ticket flows exactly as it did before AI existed
    conn.execute(
        """INSERT INTO ai_drafts (ticket_id, tenant_id, body, source_ids, input_tokens,
                                  output_tokens, latency_ms, shown)
           VALUES (%s, %s, %s, %s, %s, %s, %s, %s)""",
        (ticket.id, ctx.tenant_id, draft.body, draft.source_ids, draft.input_tokens,
         draft.output_tokens, draft.latency_ms, state is FlagState.ON),
    )
    if state is FlagState.ON:
        attach_draft_to_ticket(ticket.id, draft.body)

Any flag service works if it can target by tenant. The rollout order we use: shadow on a few tenants, on for your own internal tenant, on for a few customers who agreed to try it, then a wider share. Setting the flag to off is the kill switch, and the old path never stops working.

This one wasn't shadow mode. It was a test run on real-size data, and it makes the same point: run the feature on real inputs and read what it produced before anyone relies on it. On the proposal evaluation tool, a test run with one vendor's full submission caught a problem early. The proposal was 13 documents, about 681,000 characters. Before the eligibility check, the code kept only the first 15,000 characters, so 97.8% of the text never reached the model. The model then reported "not found" for 11 of 17 criteria whose evidence sat in the part it never saw. Nothing errored, and the results looked believable. We stopped sending the proposal as one block of text. Each criterion now retrieves its 5 most relevant chunks, and the fallback limit went from 15,000 to 100,000 characters. We caught it before evaluators relied on the results. A demo on a short sample would have passed.

Measure one number

Pick the headline metric before writing code.

  • •Support drafts: the share of drafts sent unedited or lightly edited, and the reopen rate on those tickets.
  • •Billing answers: billing questions closed without a handoff, and whether they reopen.
  • •Onboarding: time from signup to the first meaningful action, against a comparison group.
  • •Test automation: bugs that reach production, and test failures unrelated to code changes.

Also track cost per resolved ticket from the ai_drafts table, and read the thumbs-down feedback every week. If the number doesn't move after a fair trial, turn the flag off and build the next candidate. The retrieval layer, eval harness and flag setup carry over, so the second feature costs less than the first.

If you want us to build it

GammaEdge is an AI services firm with offices in Indore and Bangalore. We build AI features inside existing SaaS products, in your stack, through our AI integration services and AI agent development services. You own all the code and prompts at handover.

A first feature starts with a conversation about your product and your data. Then we write a scoped plan: the first feature, the architecture, the risks and a quote. You get that plan in writing before any build starts.

Team size and the engagement model are shaped around what you need, whether that's a fixed-scope project or a dedicated team.

See what we build for SaaS teams and get in touch on our AI for SaaS page. Pricing is on request.

FAQ

How do SaaS companies automate onboarding, billing and tier-1 support with AI?

One at a time, starting with tier-1 support as drafts an agent reviews, grounded in past tickets and help docs. Billing comes next as read-only answers from the billing tables, with refunds and plan changes left to people. Onboarding help, such as import mapping or setup suggestions, comes once the retrieval, permission and rollout plumbing exists.

What is the best first AI feature to add to an existing SaaS product?

One that uses data you already have, has a checkable right answer, costs little when wrong, and replaces work a person already does. For most products that's AI-drafted replies on tier-1 support tickets. If slow releases are the bigger problem, start with AI test automation.

Should we fine-tune a model or use RAG for our SaaS product?

Use RAG first. Your facts change often, retrieval can filter by tenant and permission on every request, and you can trace each answer to its sources. A shared model fine-tuned on customer tickets can leak one tenant's data to another. Fine-tune later for narrow jobs like classification or routing.

Is pgvector good enough, or do we need a vector database?

If you run Postgres, start with pgvector. Permissions, row-level security and backups stay in one place. Move to a dedicated store when filtered recall, index memory or write load on your own data says you have to.

Can AI write automated tests for our SaaS product?

Yes. It can draft test cases from user stories, write first-pass end-to-end tests and group failed runs by cause. A person still reviews each test, because a generated test asserts current behavior, bugs included.

How do you roll out an AI feature without breaking the product?

Behind a per-tenant feature flag with three states: off, shadow and on. Shadow mode runs on real traffic and logs output without showing it. Then turn it on for your own team, then a few customers, and keep the old path working as the kill switch.

saasai featuresragpgvectorevals

Authored by:

GammaEdge Team

Six years shipping production AI. We write about the problems nobody talks about.

Read more

Subscribe to Newsletter

Stay up-to-date on what's happening at GammaEdge.io