A prototype changes when people depend on it
An AI demonstration can work well with a small set of carefully chosen examples. A production service must also handle incomplete requests, changing information, unexpected inputs and unavailable dependencies. The question shifts from whether the model can produce a useful result to whether the surrounding system can support a defined task consistently enough for its intended use.
Consider an internal document assistant. A convincing answer is only one requirement. People also need the right access to source material, a way to understand the answer’s basis and a clear route when the system is uncertain. Defining the user, the task and the consequences of an incorrect output helps a team decide what to build—and what should still require a person’s judgment.
Data engineering
Dependable inputs
AI engineering
Evaluate & integrate
Platform engineering
Operate & monitor
Data and retrieval need continuing ownership
Data engineering remains central when an application uses an existing model. Someone must maintain ingestion, transformation, quality checks and access controls. Documents change; sources can become unavailable; permissions can differ between users. A pipeline that worked during a demonstration needs monitoring and a clear owner once it supplies a service people use.
Retrieval-augmented generation can give a model relevant source material before it answers. It does not guarantee a correct answer. Retrieval can miss useful information, return outdated content or surface material a user should not see. Data and AI engineers need to work together on indexing, permissions, freshness and the way retrieved evidence reaches the application. A larger context window does not remove those responsibilities.
Evaluation is an engineering practice
Useful evaluation begins with examples that represent the task, including difficult cases and situations where the system should decline to answer. Domain specialists help define what a good result means. AI engineers can turn those examples into repeatable checks and compare changes to models, prompts, retrieval or application logic against the same baseline.
A single quality score can hide different failures. A document assistant might retrieve the wrong source, misread the right one or express unwarranted confidence. Separating these behaviours makes investigation more useful. Human review remains important where the cost of a mistake is substantial. Evaluation should continue as the data and application change; passing a test set once is not a lasting guarantee. Versioning the evaluation set and recording why a change was made helps later reviewers understand what improved and what remains unresolved.
Production is
a shared system.
Data engineering
Dependable inputs and access.
AI engineering
Evaluate and integrate.
Platform engineering
Operate and monitor.
Agree how responsibilities are shared.
The team behind production AI
Data engineering maintains dependable inputs and access. AI engineering evaluates and integrates the application. Platform engineering operates and monitors the service. Agree how the team shares these responsibilities.
Software and platform skills make it operable
The application still needs dependable software: authentication, tested interfaces, sensible timeouts and a useful experience when a dependency fails. Model outputs should be treated as untrusted inputs, especially if they can trigger another action. Application engineers need to understand where validation, permission checks and human confirmation belong rather than letting fluent text stand in for a trustworthy decision.
Platform and reliability specialists help operate the service. Their work can include deployment, observability, capacity and cost controls, credential management and recovery. A team needs to see both technical health and application quality: a successful API response can still contain a poor answer. Monitoring should respect privacy and avoid collecting sensitive prompts or documents simply because logging is easy.
Build the team around the gaps
An organisation does not necessarily need a separate hire for every responsibility. It does need to know which capabilities already exist and where ownership is missing. Data engineers, AI engineers, software engineers, platform specialists and domain experts may share the work in different ways. Training a model and integrating an existing one also call for different experience.
For recruitment, look for evidence that matches the gap. A data specialist should be able to discuss quality and access problems they handled; an AI engineer should explain evaluation and integration decisions; a platform specialist should describe operating constraints and recovery. Ask about the tradeoffs, not just the tools. The goal is a team that can maintain the application responsibly after the demonstration ends.
Further reading
- NIST: AI Risk Management Framework (opens in a new tab)
- Google: Rules of Machine Learning (opens in a new tab)
General guidance, rather than a description of a specific vacancy or a substitute for advice about your circumstances.