From AI Experiment to Production. What's Actually Missing?

An LLM Gateway Isn't an Agent Platform.

A model gateway can provide approved model access, routing, limits and consumption visibility for requests passing through it. An agent may also retrieve data, call tools, retain context and change systems through connections that never pass through the gateway.

Those actions need permissions and operational visibility of their own. Seeing a successful model request tells us little about whether a subsequent action completed correctly. Likewise, removing model access may leave other credentials or already running work active.

An agent platform can bring together more of the runtime, tool integration and lifecycle capabilities. It still needs to be configured and operated for the workloads depending on it. Availability, capacity, isolation, support hours and recovery arrangements determine what a team can actually rely on. A product feature needs to become a tested capability within that service.

Atul Kumar's article on agents becoming infrastructure describes an agent control plane spanning identity, policy, tools, observability, memory, lifecycle and cost. It also addresses ownership and intervention.

They may be distributed across existing identity systems, runtimes, gateways and enterprise applications. The design needs to follow the whole task through those systems. A platform originally supporting interactive use may need different capacity and recovery arrangements when unattended workflows start depending on it.

The Infrastructure Agents Actually Need.

I would start by establishing what is running and what authority it carries. An inventory needs to connect each agent to its purpose, owner, environment and dependencies, with a way to find deployments that have escaped the normal onboarding process. Its identity should make actions attributable, including whose authority it is using. Shared credentials make that harder to establish.

Access needs to be enforced where the action happens. An agent reading cost data should have access to the agreed data and nothing more. If an action requires human approval, the check needs to apply to the specific action and target. Instructions in a prompt cannot substitute for permissions enforced by the receiving service.

Execution also needs boundaries. Limit duration, concurrency, retries, model requests and tool calls. Decide what happens when a limit is reached, including whether the task can resume safely. A retry mechanism that starts a fresh task whenever a limit stops the previous one defeats the purpose.

Responders need to follow a task across its model requests, approvals, tool calls and resulting changes. Useful records connect those events without requiring every system to log every piece of sensitive content. Access, redaction and retention need attention because the diagnostic records can themselves become a sensitive data store.

Deployment and change handling belong here too. Model, prompt, tool and permission changes can alter behaviour. Keep versions identifiable, evaluate the workflow after relevant changes and understand what recovery requires. Rolling back code does not undo an external action or necessarily restore retained context.

Cost needs the same view of the whole workflow. Include model requests, retries, tool APIs, compute, storage, telemetry, human review and support. Compare the cost of an accepted result with the process being improved. A cheap model call can still contribute to an expensive outcome if it causes repeated work or substantial correction.

For each capability, establish whether the platform supplies it, the application implements it or a dependency provides it. That makes the remaining engineering work visible and gives responders somewhere definite to turn when it fails.

Bring the relevant people into the conversation while their answers can still influence the design. In a small organisation, one person may cover several roles. What matters is that the responsibilities are covered and the people providing the services agree what they can support.

Let's Break an Agent.

Suppose a team builds an agent to investigate unusual cloud spending. It reads approved cost data, identifies a possible cause and proposes an investigation ticket. A person approves the ticket before it is created. The agent cannot change infrastructure.

Begin by checking usefulness. Does the ticket direct a reviewer to relevant evidence and the responsible team? Does it reduce investigation effort without generating unnecessary work? Compare those results and the operating cost with the existing process.

Then test the boundaries in an isolated environment. Try a read outside the permitted data scope. Change the proposed ticket after approval and check that the altered request is rejected. Exhaust the execution limit and establish whether the task stops cleanly. These tests reveal whether the intended restrictions hold where the work happens.

Now interrupt a run after the ticket service accepts a request but before the agent receives confirmation. The ticket exists, but the agent may believe the request failed. On restart, an immediate retry could create a duplicate.

The workflow needs a way to establish the result before repeating the action. Where the target service supports it, a stable idempotency key can let retries refer to the same operation. Otherwise, reconciliation needs to identify whether the intended action occurred. If the result remains uncertain, the workflow may need to pause for investigation. Repeatedly asking the model what to do cannot establish the state of the ticket service.

Use the incident to test the response as well. Can the operator identify the affected task, trace the request and determine what changed? Suspend scheduling and execution, then revoke downstream access. Check active processes, existing sessions and queued tasks. A request already accepted by another service may still complete, so containment includes finding and reconciling those effects.

Recovery needs an explicit decision about what can resume. Confirm completed actions before restarting incomplete work, and check that the backlog will not create a sudden burst of requests. If the agent remains unavailable, someone needs to own the outstanding investigations and continue them manually where necessary.

Keep the test results and unresolved limits concise enough that the next operator can use them. Repeat the relevant tests when authority expands. Giving this agent permission to resize infrastructure would introduce different consequences and recovery requirements.

Build Once, Reuse Everywhere.

The first workload should reveal which capabilities will be useful to others. Supported identity patterns, approved tool integrations, execution limits, tracing and deployment templates can save subsequent teams considerable effort. Reuse should come with clear operating limits and someone maintaining the shared components.

Learn from real workflows before generalising. Some need long running tasks; others need low latency or stronger isolation. Shared capabilities should accommodate those differences.

The application team still needs to evaluate its agent, handle workflow errors and support the outcomes it produces. A shared runtime cannot determine whether a cost investigation is useful. It can make the underlying execution easier to operate and investigate.

This is an application of the paved road: the supported route needs to be practical enough that teams choose it. Clear examples, useful defaults and access to help matter alongside technical controls. Feedback from teams using the service should influence what gets built next.

To begin, take one agent and follow a task from its starting identity to its final external action. Identify the services and permissions involved. Run the interrupted-action and containment tests, then work through recovery with the people who would actually respond. Use the gaps to decide what the workload needs before production and which improvements belong in the shared platform. That leaves the next team with more working infrastructure to use and fewer problems to rediscover.