· ship
Cost control on mobile AI
For a mobile product, “cost per request” is not the same as cost per useful outcome. One tap can trigger a request, a retry after a network interruption, a tool call, and a second generation after the user edits the prompt. A cheaper response can also create more review work. Budget the complete path from the user’s action to an accepted result.
Map the cost drivers
For one feature, count expected calls per session, input and output size, any retrieval or image processing, retries, moderation, and any human review. Use the provider’s current billing rules for the model and account you actually plan to use; rates and available models change. Then measure what share of generated results users keep, edit or abandon. Avoid putting raw prompt content into analytics unless the product has a clear need and a suitable data policy.
Hypothetical example: a writing app offers “make this note clearer.” The team tracks one initial request, any user-triggered regeneration, the amount of supplied context, completion or timeout, and whether the user copies or meaningfully edits the draft. It evaluates the quality of a sample separately from the cost ledger. If most users ask for a second generation because the result is too formal, the first fix may be product controls for tone—not routing every request to a larger model. This is an illustrative measurement plan, not real usage data.
Put limits at the points users can understand
- Limit input length and retrieved context to what this feature needs.
- Set a maximum request time and a clear retry rule; do not retry every failure automatically.
- Offer a lighter route for tasks that pass the same quality threshold, and verify the threshold with evaluations.
- Cache only when the data, freshness and privacy rules make reuse safe.
- Choose an explicit per-user or per-account budget and give product owners a way to disable the feature.
Do not move computation onto the device only to call it free. On-device inference can reduce network dependence and keep some inputs local, but it adds model distribution, device compatibility, storage, battery and support work. A server route centralizes control and updates but needs a trustworthy data path and a way to handle outages. Test both against the product’s actual constraints.
Keep a user-visible fallback, such as editing the original text or saving a draft, so a budget limit does not strand someone mid-task. For routing, see multi-model routing in apps; for interaction design, see latency budgets users feel. Review app implementation support or share the feature and its usage assumptions.