Highly Deterministic Agents: A Case Study

At Aguru, we started building agents the way most people do: an LLM, a system prompt, and a set of tools. The model receives a goal and decides what to do next.

This works very well in a chat interface. The task is usually short, and a human is immediately available to correct the agent when it goes in the wrong direction.

Our workflows are different. They run unattended for days, weeks, or even months. If the model ignores an instruction or takes the wrong action, there may be no one around to notice and correct it.

My first response was to add more instructions to the system prompt. But each new rule made the prompt longer and the behavior harder to predict, and none of it addressed why the model ignored instructions in the first place.

Eventually, I realized that the problem was not the wording of the prompt. We had given a probabilistic model ownership of the workflow's control flow.

So we flipped the relationship. Deterministic code now owns the workflow and calls small, specialized LLMs only when it encounters a language problem. That inversion is the foundation of what I call a highly deterministic agent.

The LLM became a library

The best analogy I found is the difference between a framework and a library. A framework owns the control flow and calls your code. With a library, your code owns the control flow and calls the library when it needs it.

A regular tool-calling agent works like a framework. The LLM owns the loop. It decides which tool to call, reads the result, and then decides what to do next. Your code is just a collection of tools waiting for the model to call them.

In a highly deterministic agent, the relationship is reversed. The workflow owns the loop and decides what happens next. When it needs to understand a message or write a response, it calls an LLM, gets the result, and continues. The LLM is one library used by the system; it is not the system itself.

The word highly is important. The LLM is still probabilistic, so two runs may produce slightly different wording. I am not trying to make natural language deterministic. I am making sure that decisions such as when to send a message, when to wait, and when the workflow is complete do not depend on the model remembering to follow an instruction.

One stakeholder, many workflows

One of the first places we used this design was stakeholder communication. At Aguru, several workflows can be running for the same stakeholder at the same time. Each workflow may need something different, and the stakeholder may reply through email, SMS, WhatsApp, or phone.

Letting every workflow send its own messages looks simple at first, but it quickly falls apart. The stakeholder receives several disconnected conversations, sometimes asking for the same thing twice. When a reply arrives, we know who sent it, but we don't necessarily know which workflow should handle it. The same reply may even answer more than one workflow.

That is why we built the Stakeholder Communication Module (SCM). Other workflows do not communicate with the stakeholder directly. They register what they need with SCM, and SCM owns one conversation with that stakeholder.

From the calling workflow's point of view, SCM hides a free-text conversation behind a typed interface. The workflow asks for a date, a number, a file, a choice, or a boolean, then waits for the result.

For example, imagine a property compliance workflow that needs to know whether the landlord will arrange an inspection or wants the agency to arrange it. Its contract with SCM could look like this:

const result = await scm({
  stakeholder: landlord,
  goal: "Decide who will arrange the inspection",
  slots: [
    {
      name: "arrangedBy",
      kind: "enum",
      options: ["landlord", "agency"],
      description:
        "Would you prefer to arrange the inspection yourself, or have us arrange it for you?",
    },
  ],
  timeout: "15d",
});
 
if (result.outcome === "fulfilled") {
  result.slots.arrangedBy.value; // "landlord" | "agency"
}

The landlord does not need to answer with the word agency. They might write, “Please arrange it for me” and SCM returns the typed value. The calling workflow does not need to read the conversation or work out which message contains the answer.

SCM owns the rest of the communication as well. It sends the first request, asks for clarification when an answer is unclear, and follows up when the stakeholder goes quiet. Once the need is satisfied, it returns the result to the workflow that requested it.

This sounds like a natural fit for a single conversational agent. That was our first implementation.

We started with a regular agent

Our first implementation was one conversational agent. It received the conversation, the outstanding goals, and a set of tools. From there, the model was expected to understand the reply, remember what was still missing, and decide what to do next.

At first, it worked well. We asked for an appointment date, the stakeholder replied with a date, and the agent returned it to the workflow.

The trouble started as we added more conversation scenarios and rules. We would tell the agent to wait in a particular situation, and it would continue. We would give it a specific follow-up policy, and it would follow up at the wrong time.

Our response to each failure was another prompt instruction. Sometimes that fixed the failing conversation, but changed the behavior of another one. After enough changes, I no longer felt confident that a prompt update was local.

The hardest part was that one model was doing everything. When something went wrong, it was difficult to tell whether the agent misunderstood the message, forgot the state, ignored a policy, or simply wrote the wrong response.

That was when I stopped trying to make one agent own the whole conversation.

I stopped using the transcript as state

In the first implementation, the conversation history was effectively our state. On every turn, we gave the transcript back to the model and expected it to reconstruct what had happened, what was still missing, and what it should do next.

That was too much responsibility to hide inside a prompt.

In the current SCM, every workflow registers a typed need with a goal and the values it is waiting for. SCM stores the progress of that need explicitly. It knows which values are still missing, what it has already asked, whether a reply is waiting to be processed, and whether the conversation is paused for a human.

The transcript still matters, but it is evidence—not the source of truth for the workflow. If a piece of information can change what the system does next, I want it to have a name, a type, and a place in state.

For reference, this is an abridged view of the state SCM carries:

type ScmState = {
  stakeholder: Stakeholder;
  needs: Need[];
  inbox: InboundMessage[];
  pendingNeedOperations: NeedOperation[];
  pendingReply: ReplyItem[];
  conversation: {
    slots: Record<string, SlotState>;
    attachments: Attachment[];
    transcript: { prior: Message[]; current: Message[] };
    followUp: { attempts: number; suppressedUntil?: string };
    asked: SlotKey[];
    briefed: BriefKey[];
    operatorGuidance: string[];
  };
  caughtUp: boolean;
  agents: AgentHolds;
  pendingQuickActions: QuickAction[];
  contacts: ContactSet;
  nextCallerProbeAt?: number;
};

It is a lot more state than an array of chat messages, and that is deliberate. The model no longer has to remember the workflow on our behalf.

Code decides what happens next

Once the state was explicit, we stopped asking the model what the system should do next. SCM now runs as a state machine, and its main loop is deliberately boring:

const phase = determinePhase(state, Date.now());
await runPhase(state, phase);

determinePhase is a pure function with an ordered set of rules. If a conversation is waiting for human review, it stays there and cannot send another message. If an inbound reply fills the final slot, the need is settled before SCM considers writing another response.

One detail I particularly like is that we do not store the current phase. We derive it again from state after every operation. That removes another piece of state that could become stale or disagree with the rest of the workflow.

The model still helps us understand messages and write replies, but it never decides which phase runs next. Code owns that decision.

One agent became several small LLM calls

After moving state and control flow into code, I looked at what was left. Most of it was language work.

We still needed an LLM to read an existing conversation and recover useful values from it. We needed another call to understand each new message. We also needed writers for the response, the email reminder, and the shorter SMS reminder. Those became separate model calls instead of responsibilities hidden inside one large agent.

Each call has a small job and very little authority. The extractor cannot send an email. The response writer cannot decide that a reminder is due. The reminder writer cannot decide that a need is complete.

This made the system much easier to change. If extraction is weak, I can work on extraction without touching the follow-up logic. If the reminder sounds wrong, I can change its writer without changing how the conversation state is updated.

I give the model a form to fill in

Making the LLM calls smaller helped, but the extractor could still jump to an answer without paying attention to an important detail. Telling it to “think carefully” was not enough.

We use Attentive Reasoning Queries (ARQs) for this. The technique comes from the paper Attentive Reasoning Queries: A Systematic Method for Optimizing Instruction-Following in Large Language Models. I think of ARQ as a form the model must fill in before it can give an answer. Instead of hiding every instruction in the prompt, we turn the important questions into required fields in the structured output.

Here is a simplified version of the real form SCM generates for the arrangedBy slot:

{
  "type": "object",
  "additionalProperties": false,
  "required": [
    "whatTheMessageSaysAboutIt",
    "isAClearValuePresent",
    "isThisAValueOrAPromiseToProvideLater",
    "wouldIBeGuessingIfISetIt",
    "conflictsWithCurrentValue",
    "proposedValue"
  ],
  "properties": {
    "whatTheMessageSaysAboutIt": { "type": "string" },
    "isAClearValuePresent": { "type": "boolean" },
    "isThisAValueOrAPromiseToProvideLater": {
      "type": "string",
      "enum": ["value", "promise", "neither"]
    },
    "wouldIBeGuessingIfISetIt": { "type": "boolean" },
    "conflictsWithCurrentValue": { "type": "boolean" },
    "proposedValue": { "type": ["string", "null"] }
  }
}

Before proposing a value, the model has to write down what the message says, whether the value is clear, and whether it would be guessing. Because these fields are required by the schema, it cannot silently skip one. We can also add self-checks for a particular slot when it needs extra care.

This does not make the model correct. It makes its proposal easier to inspect and much harder to produce without considering the questions we care about. We still validate the result in code before allowing it to change state.

I treat model output as untrusted input

Even with ARQ, the model never writes to SCM state directly. It returns a structured proposal, and code decides whether that proposal is allowed.

I treat this in the same way I would treat input coming into an API. Before anything changes, we check it against the current state and the rules of the slot. If it does not pass, SCM asks for clarification or pauses the conversation for a human.

Consider a stakeholder replying:

I contacted the tenant, and he will not be able to attend the inspection. Can I pass by the office to grab the keys?

The question is relevant to the inspection, but SCM has not been given the authority to let someone collect the keys. The model may understand the request perfectly, but it still cannot answer it. The conversation pauses for a human.

We use the same boundary for outgoing messages. Code decides that a message is needed and what it must communicate. The LLM only writes it.

I can test most of the system without an LLM

The LLM still makes mistakes. I just stopped letting every model mistake become a workflow mistake.

Most of SCM can now be tested like normal code. I don't need an LLM evaluation to check whether a deadline has passed, a reminder is due, or a need is complete. The LLM evaluations are limited to the parts where we actually use a model: understanding a message and writing a response.

Debugging also became much easier. With the old agent, we often had to read the whole conversation and guess what the model thought was happening. Now I can look at the state and see what the workflow is waiting for and why.

Of course, we paid for this with more code—a lot more code. We had to model the domain, name the states, and decide what every transition should do. The old agent loop was much smaller.

I would not use this architecture for every agent. For a short-lived task with a human watching, a regular tool-calling agent is often the better choice. But for a workflow that runs unattended and communicates with real people for weeks, the extra code is worth it.

The smallest possible job

In SCM, the LLM reads messy conversations and writes natural responses. It does not own the clock, keep the workflow state, decide when to follow up, or decide when a need is complete. Code does all of that.

This has become a useful rule for me when building agents. Before giving something to an LLM, I ask whether I can express it reliably in code. If I can, I keep it in code. If I cannot, I give the model the smallest possible job and validate what comes back.

That is what “highly deterministic” means to me. The AI is still important, but it is no longer pretending to be the whole system.