Most web applications built in the last decade share three properties their authors never had to think about. Response times are broadly consistent. Compute cost per request is effectively zero. The same input produces the same output.
Adding a model breaks all three at once, and the interesting engineering is not the API call. It is what the rest of the application has to do differently now that one step is slow, priced per use, and not guaranteed to behave identically twice.
Latency stops being uniform
A database query returns in milliseconds. A model call can take several seconds, and the variance is wide rather than a stable average.
That breaks assumptions throughout an application. Synchronous request handling that was fine at 80 milliseconds becomes a held connection and an exhausted worker pool. Timeouts set for a fast backend start firing. Users stare at a spinner with no indication whether anything is happening.
The architectural answer is to stop treating it as a request. Accept the input, return immediately with a reference, do the work in the background, and deliver the result when ready through polling or a push. That is more work than an inline call and it is the difference between a feature that degrades gracefully under load and one that takes the application down with it.
Where the interaction genuinely needs to feel immediate, stream. Showing output as it arrives makes several seconds feel responsive, because the user sees progress rather than absence.
This surprises the teams we work with more than cost does. A request that used to complete in a predictable fraction of a second now has a long tail, and an interface built on the old assumption feels broken rather than slow.
Cost per request is now real
Traditional application requests are effectively free at the margin. Model calls are not, and that changes decisions that were previously automatic.
Retry logic is the clearest example. Retrying a failed database query three times costs nothing. Retrying a model call three times costs three times as much, and a retry storm during an outage produces a bill nobody expected.
The same applies to anything that runs speculatively. Prefetching, background enrichment, generating something the user might want. All were free habits and all now have a price attached.
Practical containment: cache aggressively where inputs repeat, which is more often than teams assume. Set per user and per tenant limits before launch rather than after an incident. Log spend per feature so you can tell which one is expensive, because a single monthly figure tells you nothing actionable.
Read More - AI Sends a Fraction of the Traffic and Several Times the Customers
The same input may not produce the same output
This is the property that causes the most trouble, because it invalidates habits built over years.
Tests that assert exact output become flaky. Caching by input hash may serve a result that differs from what a fresh call would produce. Any downstream code parsing the response must handle a shape it has not seen, because eventually it will get one.
Two things help more than anything else. Constrain the output format and validate it before it enters your system, treating the model's response as untrusted input in the same way you would treat a form submission. And test behaviour rather than exact text: that the result contains a valid date, that the classification is one of five permitted values, that the summary is under the length limit.
Where determinism genuinely matters, do not use a model for that step. Extracting a total from an invoice is better done with parsing rules and a model as a fallback than the other way round.
Design for the model being wrong
It will be, at some rate, and the feature design determines whether that is tolerable or damaging.
The safest patterns put a person between the output and the consequence. Draft that someone sends. Suggestion someone accepts. Classification someone can override, with the override captured so you learn where it fails.
The dangerous patterns act directly on something irreversible. Sending, deleting, charging, or publishing without review. Those require a much higher confidence bar and usually a confirmation step, and the speed cost is generally smaller than teams fear.
Make the uncertainty visible in the interface rather than hiding it. An output presented as a suggestion invites scrutiny. The same output presented as a fact does not, and users who trust it uncritically are how a model error becomes a customer problem. We covered the accuracy dimension separately in stopping confident wrong answers.
We treat the failure path as the first thing to build rather than the last, because it decides whether anyone trusts the feature. A confident wrong answer with no way to correct it does more damage than no feature at all.
Read more - Your CRM Vendor Added an Agent Nobody Asked For
Keep the model swappable
One structural decision matters more than the rest, and it is cheap at the start and expensive later.
Put the model behind an interface your application owns. Keep prompts in one place, versioned. Normalise responses into your own types rather than passing provider objects through your codebase. Then switching provider is a contained change rather than a rewrite.
This is not theoretical caution. Model capability and pricing have both moved substantially within single quarters, and applications that hard coded one provider are paying whatever that provider charges because moving is too expensive to contemplate.
Worth planning for the other direction too, because agents increasingly arrive at your application as visitors rather than as features, which we covered in when the agent is the user.
If you are adding AI to an application that already has users, our web app development team treats the surrounding architecture as the actual work, because the API call is the easy part.
Frequently Asked Questions
Should the model call be synchronous or background?
Background for anything that might take more than a second or two, with streaming where the interaction needs to feel immediate. Synchronous calls hold connections and consume workers, and an application that was comfortable at a hundred concurrent users can fall over at twenty once each request occupies a worker for four seconds.
How do we control cost before it becomes a problem?
Per user and per tenant limits from day one, cache where inputs repeat, and log spend per feature rather than in aggregate. Most cost surprises come from retries and speculative calls rather than from normal use, so look there first.
How do we test something non deterministic?
Assert properties rather than exact strings. Valid format, permitted value, within length, contains required fields. Keep a set of representative inputs with acceptable and unacceptable outputs, and run it when prompts or models change. Exact match assertions will fail constantly and teach the team to ignore the suite.
Can we cache model responses?
Yes, and it is usually the largest single cost saving available. Cache on a normalised version of the input so trivial differences still hit. Be careful where responses should reflect changing data, and set expiry accordingly rather than caching indefinitely.
What happens when the provider has an outage?
Decide in advance whether the feature degrades or the application fails. Usually it should degrade: hide the AI assisted step, fall back to the manual path, and tell the user plainly. An outage at a third party should not take down functionality that has nothing to do with it.
Should we fine tune or use prompting?
Start with prompting and good context retrieval. Fine tuning adds a training and maintenance cycle, and it locks you to a model at a moment when models are improving quickly. Most business tasks are solved adequately without it, and you will know if yours is not.
How do we know whether the feature is actually working?
Instrument the human response. Acceptance rate of suggestions, edit rate on drafts, override rate on classifications. Those tell you what users think of the output, which is what matters, and they cost almost nothing to capture at the point of interaction.
Talk to our team if you are adding AI to an application that already has users depending on it.