Inference Public

AI engineering on AWS

AI that your service
can stand behind.

Generative AI features designed, evaluated and operated with the same discipline as the rest of the service.

Public administration staff using document scanning and digital casework tools.

Start with the task, not the model

Generative AI can help staff find, summarise and draft, but only if the answers are good enough for the task and someone owns them. A demonstration proves something is possible; operating it needs evidence about quality, failure and cost.

We design, evaluate and operate AI features with the same discipline as the rest of the service: identity, network boundaries, pipelines, monitoring and a named owner for each AI-assisted decision. Sometimes the right answer is search, rules or a clearer form, and we will say so.

We use AI assistants in our own engineering work too. Their output goes through the same review, testing and pipeline evidence as any other change.

When to bring us in

  • A promising prototype needs to become something a team can run and defend.
  • Staff spend hours finding or summarising information held in approved documents.
  • You need to decide whether an AI use case is worth pursuing, and how to tell.

What we take on, and what stays with your team

We take on: the use-case assessment, the engineering of the AI feature, its evaluation suite and guardrails, and the operating procedures.

Your team keeps: the judgement of answer quality, the decision on acceptable error, information governance, and accountability for decisions the AI assists.

Engineering scope

Retrieval over approved sources

Amazon Bedrock for model access and Bedrock Knowledge Bases or a purpose-built retrieval layer. Retrieval respects access controls, so users only see answers drawn from content they may see.

Evaluation baseline

An evaluation set of representative, awkward and adversarial questions with expected answers agreed by the service owner, run on every change and compared with the baseline, including a non-AI alternative where relevant.

Abstention and escalation

The feature says when it does not know, and hands over to a person, rather than guessing.

Guardrails and prompt injection

Bedrock Guardrails for content and sensitive information, limits on what actions the model can trigger, and defences against prompt injection through retrieved documents.

Model change and withdrawal

Model, prompt and source changes go through review and rerun the evaluation. There is a plan for model versions being retired, and for withdrawing the feature without taking the service down.

An illustrative project

Illustrative example, not a client engagement

Starting state. Staff answer recurring questions by searching several hundred policy documents. A prototype chatbot gives fluent but sometimes wrong answers.

Agreed work. Agree the task and acceptable error with the service owner, build an evaluation set from real questions, add retrieval with citations and access control, introduce abstention and guardrails, and compare against the current search.

Deliverables. A use-case decision record, the AI feature with guardrails, the evaluation suite and baseline results, monitoring for quality, cost and misuse, and runbooks for model changes and withdrawal.

Acceptance. Results on representative and adverse cases are compared with the baseline, unresolved limitations are documented, and a named owner accepts operating responsibility.

What we need from you

  • A service owner who can judge answer quality, and representative questions with agreed answers.
  • Access to the approved content and its access rules.
  • Contacts in information governance and security, and a view on whether an Algorithmic Transparency Recording Standard record is needed.

Costs, risks and trade-offs

Model calls, retrieval, evaluation runs and human review all cost money. Measure cost per useful task, not just cost per request, and compare it with doing the task without AI. More capable models cost more and are not always more accurate on a narrow task.

FinOps and cloud cost engineering covers how running costs are made attributable and managed.

Related

Read Putting a generative AI feature into service and Verifying AI-generated code and tests.

See anonymised experience. Compare ways to work together.

A sensible first step

Bring one candidate use case. We will discuss the task, the data involved, the consequences of a wrong answer and how success could be demonstrated.

A first conversation does not commit either side to anything, and we will not ask for access to your systems until a scope is agreed.

Discuss one AI use case