The Agent Loop Thinks. The Control Plane Decides.

A production architecture guide for policy, approvals, observability, retries, and rollback in agentic workflows

The boundary that matters in production

Agents are easy to demo and hard to operate, because the failure modes show up after the model has already acted.

If you are building agentic workflows that can mutate state, call tools, or trigger downstream actions, the hard problem is usually not whether the model can propose something plausible. The hard problem is who is allowed to let that proposal become a real action, what policy applies, how the action is recorded, and what happens when the system needs to recover.

This post is for AI engineers, platform engineers, DevOps/SRE practitioners, and engineering leaders who are trying to decide whether an agentic system is ready for production.

The thesis is simple:

The agent loop thinks; the control plane decides, constrains, records, and recovers.

That is not a prompt-engineering claim. It is an operational boundary claim.

I am not trying to argue that agents should be fully autonomous, or that human review should disappear, or that every system needs the same governance overhead. I am also not doing a benchmark comparison, a vendor shootout, or a security/compliance checklist with legal guarantees.

What follows is a production architecture model: where responsibilities belong, why the boundary matters, and how to design policy, approvals, retries, audit, and rollback so an agentic system can act safely and recover predictably.

Why agentic workflows need a control plane now

The reason this topic matters now is that agentic systems are moving beyond suggestions and into systems that can actually change state or trigger downstream effects. Once that happens, runtime governance stops being optional.

A prototype loop usually looks like this:

1. observe some state,
2. ask a model what to do,
3. call a tool,
4. repeat.

That is enough to show a demo. It is not enough to safely operate production systems where actions have durable side effects.

The moment an agent can create tickets, modify records, send messages, provision infrastructure, or kick off other workflows, the system needs deterministic answers to questions the model should not own by itself:

• Is this action allowed?
• Does this action require approval?
• What exactly changed?
• Can we prove who approved it?
• Can we reverse it?
• If we cannot reverse it, how do we contain it?

Those are control-plane questions.

Diagram note: This layered view is meant to separate model reasoning from deterministic governance. The key point is not the exact component names; it is that policy, approval, execution, audit, and rollback should sit in infrastructure that can be reasoned about without reading prompt transcripts.

Why prototype-grade agent loops break in production

The failure modes are familiar if you have operated any system that can take actions automatically, but they become more visible once the decision maker is a model rather than a fixed rule set.

Prototype assumptions vs. production requirements

Prototype assumption	Production requirement
The model can decide and act directly	Policy should be checked before side effects.
A retry just repeats the call	Retries should be bounded and policy-aware; they should not bypass approval or duplicate unsafe actions.
The transcript explains what happened	The control plane should store action lifecycle and governance events as the source of truth.
Human approval means every action needs review	Approval should be risk-based and tied to specific action classes.
Rollback can be improvised after incident	Rollback or compensation semantics should be declared ahead of time.

Common breakpoints include:

• unsafe actions that were never meant to execute,
• unclear responsibility boundaries between the model, the application, and the operator,
• missing audit trails,
• brittle retries that repeat the wrong action,
• weak rollback paths,
• and no obvious enforcement point for policy.

A lot of teams discover the same thing the hard way: the model can be good enough to recommend an action, but the system around it is not yet good enough to let that recommendation become reality.

This is where a common objection shows up: “But we already have workflow engines, queues, and orchestration tools.”

Yes, and those tools still matter. They are necessary. They are not sufficient.

A workflow engine can move work through steps. A queue can buffer execution. An orchestrator can schedule jobs. None of those, by themselves, solve the full problem of model-driven uncertainty, policy enforcement before side effects, approval routing, auditability, or recovery semantics.

Another objection: “Won’t human approval make the system useless?”

Not if you use it as a risk-based gate instead of a blanket rule.

The point is not to insert a person into every action. The point is to prevent unbounded blast radius where the system is uncertain, high-risk, or operating outside policy. Human approval is a throughput cost in some paths, but it can also be the mechanism that makes the rest of the system trustworthy enough to use. That is a tradeoff, not a moral statement.

Failure mode table description: A useful way to review this is to compare prototype assumptions against production requirements. For example, “the model can decide and act directly” becomes “the system should check policy before side effects.” “A retry just repeats the call” becomes “a retry should be bounded and policy-aware.” “The transcript explains what happened” becomes “the control plane should store the action lifecycle as the source of truth.”

Split responsibilities between the agent loop and the control plane

The most useful mental model here is to split the system into two parts:

• the agent loop, which reasons about the problem and proposes actions,
• the control plane, which governs whether those actions can happen, under what conditions, and how they are recorded and recovered.

Inside the agent loop, keep the work that is inherently model-driven:

• reasoning,
• planning,
• tool selection candidates,
• bounded local retries that do not create durable side effects on their own.

Inside the control plane, keep the work that must be deterministic and auditable:

• permissions,
• policy enforcement,
• state transitions,
• approval gates,
• scheduling,
• execution coordination,
• audit logging,
• rollback boundaries,
• observability.

The important rule is this:

If a decision needs consistency, traceability, or rollback semantics, it belongs outside the prompt.

That does not mean the model is irrelevant. It means the model should operate inside a bounded loop where its job is to reason, not to act as the system of record.

The control plane should own action state, not the model transcript. The transcript can be useful context, but it is not a durable governance boundary.

The minimum responsibilities that should not live in the model

If you are designing the platform layer for an agentic system, these are the responsibilities that should be explicitly owned outside the agent loop.

1. Policy enforcement before execution

A policy check should happen before side effects, not after.

That sounds obvious until you look at how many systems implicitly treat the model as both planner and gatekeeper. If the model decides and executes in one step, you have no deterministic point where policy can stop the action before it matters.

Policy should answer questions like:

• Is this action allowed at all?
• Is it allowed for this principal?
• Is it allowed in this context?
• Does it exceed the action class or risk level for automatic execution?

That policy engine is not a prompt. It is a decision boundary.

2. Approval gating for high-risk or ambiguous actions

Approval should attach to specific actions or action classes, not to the vague idea of “agent behavior.”

That distinction matters. A blanket approval flow becomes unmanageable. A risk-based gate is much easier to reason about:

• low-risk actions may proceed automatically if policy allows,
• ambiguous actions can be routed for review,
• high-risk actions should not proceed without a deterministic approval step.

Human approval is best treated as one policy outcome among several, not as a default tax on every agent interaction.

3. Audit logging as a first-class responsibility

The system should record:

• the requested action,
• the approved action,
• the executed action,
• timestamps,
• identities,
• and outcomes.

Do not rely on prompts or ad hoc application logs to reconstruct this later. If you need to answer what happened after the fact, the control plane needs to keep the trace at the action level.

4. Explicit state machine ownership

Represent the action lifecycle as a state machine, not as a loose collection of events or best-effort logs.

The model may generate text, but the control plane should own the action’s lifecycle states:

• requested,
• policy-checked,
• approved or rejected,
• executing,
• verified,
• closed,
• rolled back,
• or escalated.

That explicit state makes it possible to define valid transitions, terminal states, and recovery behavior.

5. Failure containment and rollback boundaries

If one action fails, the failure should not automatically cascade through the rest of the system.

That means the control plane needs a declared rollback boundary:

• what can be undone,
• what needs a compensating action,
• and what must stop immediately and be escalated.

Not every action is reversible. Pretending otherwise is how you end up with brittle recovery logic and false confidence.

What changes if you treat the control plane as the source of truth

A lot of the operational confusion in agentic systems comes from putting too much trust in the model transcript.

If the transcript is treated as the main artifact, then you end up inferring state from text. That is fragile. It is also hard to audit.

If the control plane is the source of truth, then every meaningful action has a durable record and an explicit lifecycle. That gives you a few things the model cannot provide on its own:

• deterministic state transitions,
• reproducible approval outcomes,
• consistent enforcement of policy,
• traceable execution history,
• recovery semantics that do not depend on prompt reconstruction.

This is the part skeptics often miss. The control plane is not there because the model is bad. It is there because production systems need guarantees the model is not designed to provide.

That is also why this is not just generic platform engineering repackaged for AI. The specific problem here is that model output influences execution. Once output can become action, you need a place to enforce policy and preserve meaning across retries, failures, and rollbacks.

How to enforce policy, approvals, retries, and rollback without making the system brittle

A useful implementation pattern is:

1. the agent proposes an action,
2. the control plane evaluates policy,
3. the control plane decides whether approval is needed,
4. an approved action is executed,
5. the execution outcome is verified,
6. the result is logged,
7. failures are contained or compensated.

That seems straightforward, but the details matter.

Use explicit action identifiers

Every request, approval, execution, retry, and rollback should be correlated to the same action identifier.

Without a stable identifier, you cannot tell whether:

• a retry is a continuation or a duplicate,
• a human approval applies to the current execution attempt,
• a rollback corresponds to the right action,
• or a post-incident review is reconstructing the right sequence.

Model retries as policy-aware, not just transport-aware

Retries are one of the easiest places to get this wrong.

A transient transport failure is not the same thing as a policy rejection. An execution failure is not the same thing as permission to try again. If the control plane does not distinguish these cases, retries can become a way to repeat unsafe behavior instead of recovering from infrastructure noise.

A practical rule:

• retry transport and execution failures only when the action is still policy-valid,
• never retry a rejected action as though it were a transient failure,
• and never let a retry bypass approval boundaries.

Declare rollback semantics before deployment

Rollback is not something to improvise after an incident.

Before the system goes live, the control plane should know:

• whether the action is reversible,
• whether compensation is required instead of reversal,
• and whether the safest response is to halt and escalate.

This matters especially for actions that touch external systems. Some state changes can be undone cleanly. Others can only be counteracted. Some should never be automatically reversed because reversal itself is risky.

Bind approval to action classes, not generic agent intent

If the approval gate is “approve the agent,” it will drift into a vague human rubber stamp.

If the approval gate is “approve this action class under these conditions,” it becomes a governance mechanism.

That distinction is what lets you scale human oversight without turning the system into a manual queue for everything.

Observability and auditability are the same problem from two angles

A control plane is also an observability layer.

When something goes wrong, the questions you need to answer are action-level questions, not just prompt-level questions:

• What was requested?
• Why was it allowed?
• Who approved it?
• What executed?
• What changed?
• Did verification succeed?
• Was rollback attempted?
• Did the system recover?

If you cannot answer those questions after the fact, then your system is not really governed. It is only narrated.

That is why the audit log should record policy decisions, approval decisions, execution results, and rollback outcomes as first-class events.

The model transcript can be attached as supporting context, but it should not be the only way to reconstruct the action path.

A useful validation question for architecture review is simple:

Can we explain every meaningful action without trusting the model to remember what it did?

If the answer is no, the observability boundary is in the wrong place.

Architecture review questions

Use these in design review or readiness review:

1. What is the source of truth for action state? If it is the transcript, that is a warning sign.
2. Where is policy enforced? It should happen before side effects.
3. Which actions require approval, and why? Approval should map to action class or risk.
4. Can retries duplicate or amplify side effects? They should not.
5. Is rollback defined before deployment, including compensation for irreversible actions?
6. Can you reconstruct the full action path after the fact?
7. What happens when recovery is not possible—does the system escalate deterministically?

If the answers are vague, the system is not ready for production use where actions matter.

What not to do

There are a few tempting designs that look simpler than they are.

Let the agent decide and execute directly

This is the fastest path from prototype to incident.

You may get a system that feels impressive in a demo, but it will be hard to audit, hard to contain, and hard to reverse once it touches real state.

Use a workflow engine and assume governance is solved

Workflow engines are valuable, but they are not automatically a policy layer.

They can move steps forward. They can enforce order. They do not inherently answer whether a model-derived action is permitted, whether it needs approval, or whether it is safe to retry.

Put governance in prompts

Prompts are not a durable boundary.

They are useful for shaping behavior, but they are not a system of record for permissions, approvals, or rollback. If governance only exists in text, you do not have deterministic control.

Require human approval for everything

This can be a reasonable temporary choice in very sensitive systems, but it is not a good default design.

If every action requires a human, the system stops being an agentic workflow and becomes a ticket queue with model-generated drafts. That may still be useful, but it is a different product shape.

How to tell whether your control plane is working

Since there is no benchmark or vendor comparison here, the practical question is readiness.

A control plane is doing its job if you can answer these questions confidently:

• Can blocked actions be blocked for policy reasons before side effects occur?
• Can approved actions be reproduced and attributed?
• Can retries occur without bypassing policy or duplicating side effects?
• Can rollback paths be exercised before a real incident forces them?
• Can audit logs reconstruct the decision chain end to end?

For a design review, that becomes a simple checklist:

• Policy is enforced before any external side effect.
• Approvals are tied to specific action classes or risk levels.
• Every action has a stable identifier across request, approval, execution, retry, and rollback.
• The action lifecycle is modeled explicitly, not inferred from free-form text.
• Retries are bounded and cannot bypass policy or approval.
• Rollback or compensation behavior is defined before deployment.
• Audit logs can reconstruct who requested, approved, executed, and closed the action.
• The system has a deterministic escalation path when recovery is impossible.

If you later have real system data, this is where you would measure things like approval turnaround, retry outcomes, rollback success, and audit coverage. I am not inventing numbers here because none were provided.

Tradeoffs: what you gain, what you give up

A control plane for agentic systems gives you a set of concrete operational benefits:

• bounded actions,
• clearer responsibility,
• better auditability,
• safer recovery,
• more predictable production behavior.

It also costs something:

• more plumbing,
• more policy logic,
• more operational surface area,
• potentially slower paths for sensitive actions.

That tradeoff is real.

A system with more control is not automatically better in every dimension. It may be less fast in the narrow sense while being much more usable in the broader sense because teams can trust it enough to deploy it. That is a production engineering tradeoff, not a philosophical one.

The right amount of control depends on risk.

Not every action needs the same gate. Not every workflow needs the same rollback semantics. The control boundary should be shaped by the consequences of failure, not by a desire to add infrastructure for its own sake.

Copy this / don’t copy this

Copy this	Don’t copy this
“The agent loop thinks; the control plane decides.”	“Just add guardrails in the prompt.”
“Policy should happen before side effects.”	“We’ll review it after the model acts.”
“Approval should attach to action classes or risk levels.”	“Approve the agent in general.”
“The control plane is the source of truth for action state.”	“We can reconstruct it from logs and transcripts later.”
“Rollback semantics should be declared before deployment.”	“We’ll figure out rollback in the incident.”

Conclusion: autonomy becomes production-grade when control is deterministic

The core model is worth repeating:

The agent loop thinks; the control plane decides, constrains, records, and recovers.

That is the practical boundary between an impressive demo and a production system you can actually operate.

Production readiness does not come from removing oversight. It comes from bounded autonomy with deterministic governance around the places where the model should not be making the final call.

If your agent can change state, the control plane should own the rules that make that change safe, explainable, and recoverable.

If you are designing this layer, the question is not whether to add a control plane. The question is how much responsibility it should own before side effects happen.