· · Matthew Ford · 5 min read
AI agent development services: what a production-ready build includes
The easiest part of an AI agent project is making it look clever. Give it a tidy example and it will draft a reply, summarise a document or call an API. The hard part starts when you let it near live data and real customers.
That is what to probe when comparing AI agent development services. A model can produce a convincing demo. The workflow around it decides whether the agent is safe to use.
Start with one job
"Build us an AI agent" is not a scope. "Read an incoming enquiry, draft a reply using our current product information and leave it for a person to approve" is a scope. It tells you the input, the output and who owns the result.
Other examples:
- check a document against a policy and flag missing evidence;
- update a CRM after a call when the required fields are present;
- watch an operations report and alert someone when a number changes unexpectedly.
Notice that none of these requires a grand "autonomous employee". They are bounded jobs with a reviewer or an escalation path. That is usually the right place to start.
Decide what the agent can touch
An agent is different from a chat interface because it can act. It might read a ticket, update a record, send a message or trigger a payment. Each permission changes the risk.
Ask the supplier to write down:
- which systems the agent can read;
- which systems it can change;
- which actions need approval;
- what it should do when information is missing or contradictory;
- what evidence is kept for an investigation;
- how the whole thing is switched off.
If those answers are vague, the proposal is not ready for production.
This is also where privacy and security work belongs. The ICO has guidance on UK GDPR, explaining AI-assisted decisions and assessing risks to people's rights.[2] The right controls depend on the data and the decision. They should be designed into the workflow, not bolted on after a successful demo.
A fast prototype proves less than you think
Visual builders are useful for getting a first flow working. They can also hide the awkward parts: permissions, evaluation, audit trails and what happens when a vendor changes course.
OpenAI provides a current warning about that last point. Its AgentKit page says Agent Builder and Evals are being wound down, with availability ending on 30 November 2026. OpenAI recommends the Agents SDK for workflows that should continue as code.[3]
That is not an argument for writing everything from scratch. It is a reason to ask what you would keep if the tool disappeared: workflow definitions, prompts, evaluation sets, integration code and logs. If none of that can be exported or rebuilt, you may be renting more than the demo suggests.
Evaluation should include the bad days
A good demo uses clean inputs. A useful evaluation asks what happens when the document is incomplete, the policy is ambiguous, the API times out or the right answer is "ask a person".
The measures should fit the job. You might look at routing accuracy, factual errors, correction effort, appropriate escalations or cost per completed item. The important thing is to agree them before the supplier shows you the best outputs.
NIST's AI Risk Management Framework is a voluntary reference for building trustworthiness into AI design, use and evaluation.[1] You do not need to implement a whole framework for one agent. You do need to know what "good enough" means and who gets to say so.
Human review has to fit the work
"Human in the loop" can become a comforting phrase with very little behind it. A reviewer needs enough context to check the result, time to check it and authority to reject it.
Measure that effort. If reviewing an answer takes longer than writing it, faster generation has not saved anything. If a reviewer is expected to approve items they cannot investigate, the control is mostly theatre.
Someone has to own the agent after launch
Models change. APIs change. The documents and rules the agent relies on change too. A production agent needs an owner who can review output quality, investigate failures and approve changes.
Our internal Watchkeeper project is deliberately limited. It reviews server-security evidence and flags items for people to investigate. People still approve changes and baselines. That boundary is the useful part of the example, not the fact that an agent is involved.
Questions worth asking before you sign
- What exactly will the agent do, and what is explicitly out of scope?
- Which systems can it read or write to?
- Which actions require a person's approval?
- What examples will you use to test difficult and unsuccessful cases?
- What evidence will be logged, and who can access it?
- How will the agent behave when information is missing or contradictory?
- What happens if the model, framework or vendor changes?
- Who owns evaluation, support and new permissions after launch?
For a selected use case, our AI consulting and automation service covers those assessment and implementation questions. If the approach is still uncertain, start with our AI readiness assessment guide and work through the first decision before choosing a tool.
Sources
Discuss your software, technical leadership or AI plans
Tell us what you need to build, improve or decide.
We can help with software development and ongoing support, fractional CTO leadership, and practical AI adoption. Speak to our senior team about your priorities, our approach and availability.
Choose 15 or 30 minutes. Booking requests require confirmation by email.
