· ship
Shipping LLM features without melting CI
An AI feature ships as part of an app, so a successful model response is only one part of a successful release. The app still has to handle slow or missing responses, bad network connections, empty input, changed model behavior and a user who needs to recover without losing their work. Treat the model call as one dependency in a user-facing flow.
Separate checks by what they can catch
Keep deterministic checks in the normal pull-request path: request and response schemas, input limits, error mapping, cancellation, and whether the screen renders loading, partial, empty and failure states. Add a small, saved set of representative prompts and expected properties for the model-backed behavior. For example, a summarization feature might be checked for required source facts, a maximum length and a refusal to invent a missing date. Avoid exact-string tests for outputs that are expected to vary.
Run broader or slower evaluations on a schedule or before changing a model, prompt or retrieval source. Have people review samples from high-impact flows; an automated score is a signal, not a substitute for product judgment. Store the prompt/model version alongside the test result so a regression can be traced to the change that introduced it.
Design the user path before optimizing the prompt
- Show what the feature is doing and let the user cancel a long request.
- Preserve the user’s original text while a draft is generated or edited.
- Give a useful next step when the service is unavailable, such as retrying or saving a local draft.
- For generated actions, preview the exact change and ask for approval before it is sent.
- Keep provider credentials and policy decisions behind a service boundary you control.
Hypothetical example: a mobile expense app adds a receipt explanation draft. A pull request checks that the response maps to the app’s typed fields and that the screen handles no receipt, a timeout and an invalid response. A separate fixture set checks that the draft refers only to supplied receipt fields. The first release exposes the action to a small internal group behind a feature flag; support can turn it off while the conventional expense form remains available. This is an example workflow, not a claim about an existing product.
Before rollout, assign an owner for user reports and a way to compare failures by app version, prompt version and model route without collecting unnecessary personal text. For interface patterns, see streaming UX and empty states for model features. If the product needs an end-to-end implementation plan, review Custom AI Apps services or discuss the user flow.