When Do AI Agents Make Sense?

When Do AI Agents Make Sense?

Understanding when deterministic workflows are sufficient, when dynamic action selection becomes useful, and the engineering trade-offs involved.

In the previous article, we examined what happens when an LLM is given access to external tools. A model can generate a request to retrieve a document or inspect a repository, but the surrounding application must validate, authorize, and execute that operation.

Consider our investigation into the order service and the payment service. We have two technical specifications describing different communication mechanisms, and we need more evidence to determine which one reflects the current architecture.

We could build an application that retrieves a predefined set of documents, sends their contents to an LLM, and produces a report. We could also allow the model to suggest which document to examine next, based on what the investigation reveals.

Both approaches can use the same language model and the same tools. What changes is how the application determines its next operation.

That distinction raises an important architectural question: when is a predefined workflow sufficient, and when does letting a model select actions provide enough value to justify the added complexity?

To answer it, we need to examine what deterministic workflows already accomplish, where they become restrictive, and what responsibilities emerge when execution decisions depend on newly discovered information.

Why Deterministic Workflows Are Often Enough

Let us simplify our document analysis task.

Imagine the engineering team maintains a standardized documentation structure. Every service has an architecture overview, an API specification, and an integration contract. These documents follow known formats and are stored in predictable locations.

The team wants an automated process that compares these documents and identifies potential inconsistencies.

Because the required inputs and processing steps are known, we can design the application around a predetermined workflow.

The application retrieves the three documents, extracts the relevant sections, and verifies that the required information is present. It then sends the extracted content to a language model and instructs it to identify inconsistencies and link each finding to supporting passages.

After receiving the model's response, the application validates the expected output structure and produces a report. The language model interprets technical descriptions and identifies relationships between them. The application remains responsible for determining which operations occur and in what order.

A simplified representation of this workflow is:

D —load→ C —analyze→ F —format→ R

Here, D represents the input documents, C the extracted content, F the potential findings, and the resulting report.

The diagram describes a sequence of transformations. The application determines the execution path, while the language model may implement the analysis stage.

This design is not inherently simplistic.

The document-loading stage could retrieve files from several repositories. The analysis stage could involve multiple model calls, each focused on a particular architectural concern. The reporting stage could apply validation rules, check for evidence references, and reject malformed results.

We could also introduce conditional branches. If a required document is missing, the application might stop the process and report the missing dependency. If a document exceeds the model's context limits, the application could divide it into sections and analyze them separately.

These mechanisms can make the workflow considerably more sophisticated without requiring the model to determine its overall execution structure.

Predictability is an architectural advantage

One benefit of this approach is that the execution path is relatively easy to understand.

If the application is designed to retrieve three documents and perform a defined set of comparisons, engineers can estimate the operation and set reasonable expectations for execution time and resource consumption.

They can also test individual stages independently.

For example, they can test the document-retrieval component using known files and expected error conditions. You can evaluate the extraction logic against documents with different structures. You can test the reporting component to ensure it rejects outputs that don't meet the required schema.

The model's analytical behavior still introduces uncertainty. It may overlook an inconsistency or produce an unsupported conclusion. A deterministic workflow does not eliminate language models' limitations.

What it does provide is a more controlled execution structure around those limitations.

When a report is incorrect, the team can investigate whether the relevant documents were retrieved, whether the expected content reached the model, and whether the generated findings passed the application's validation rules.

The known sequence of operations makes this investigation more manageable.

Another advantage is that business rules can be implemented directly in software.

If the application must never access documents outside a designated repository, the retrieval component can enforce that restriction. If the report must include evidence references, the output contract can require them, although checking that those references genuinely support the findings still requires additional validation.

These controls do not depend on the model following a rule expressed in natural language.

Complexity does not automatically justify an agent

It is worth distinguishing a complicated workflow from a workflow that requires dynamic action selection.

Imagine a document-validation pipeline with twenty processing stages. Some stages run in parallel, others depend on previous results, and several invoke language models for different analytical tasks.

The pipeline may be difficult to implement and operate, but software can still define its control flow.

Now imagine a much smaller task: investigating why two technical specifications disagree.

The investigation might require only three or four operations, but the appropriate sequence depends on what each operation reveals.

The second task may benefit more from model-driven decisions than the first, even though it involves fewer steps.

This distinction matters because the number of operations does not determine whether an agent is appropriate.

The relevant question is whether the application can reliably specify the necessary decisions in advance.

If it can, a deterministic workflow may provide better predictability, simpler testing, and lower operational complexity.

If it cannot, allowing a model to help select subsequent actions becomes an option worth considering.

When the Next Action Depends on What We Discover

Return to the order-processing system introduced at the beginning of this article.

We have two documents describing different communication mechanisms between the order service and the payment service.

The architecture diagram shows synchronous HTTP communication. The integration specification describes an OrderCreated event published to a message broker.

A straightforward comparison can identify this discrepancy, but it cannot establish why the documents disagree. To understand the current architecture, we need additional evidence.

Suppose the integration specification contains a reference to an architectural decision record:

DOC-0001 - Migration to Event-Driven Order Processing.

The reference changes the investigation.

Before discovering it, we knew that the documents described different integration patterns. We did not know whether the difference represented an error, a migration, or two valid mechanisms serving different purposes.

The reference suggests that the system may have undergone an architectural change. Examining the decision record could help determine whether the asynchronous integration replaced the synchronous one.

A deterministic workflow could automatically follow document references. It might extract every referenced document identifier, retrieve the corresponding files, and include their contents in the analysis.

That approach can work, particularly when references follow a predictable structure and the number of related documents is manageable.

But consider what happens when a specification references dozens of other documents.

Some references may describe unrelated services. Others may concern historical decisions that no longer affect the current architecture. Following every reference could consume substantial resources without improving the analysis.

The application needs a way to determine which information is worth investigating.

One possibility is to encode that decision in software. We could define rules based on document types, reference patterns, or known architectural relationships.

Another option is to let the language model help select the next operation.

After analyzing the initial specifications, the model might identify DOC-0001 as a relevant source and ask the application to retrieve it.

{
    "tool": "read_document",
    "arguments": {
    "document_id": "DOC-0001"
    }
}

Assuming the request is valid and authorized, the application executes the operation and returns the result.

Now imagine that the decision record states that the migration to event-driven communication was approved, but only for newly created orders. Refund processing continues to use synchronous HTTP requests.

This information changes how we interpret the original discrepancy.

The two communication mechanisms may coexist because they support different operations. The architecture diagram might be incomplete rather than entirely outdated, and the integration specification might describe only one part of the service's behavior.

The model can use the new evidence to refine its analysis.

It may decide that the available information is sufficient to produce a report, or it may identify another unresolved question. For example, the documents might not establish whether the migration has actually been deployed to production.

At that point, inspecting deployment configuration or implementation details could be useful, provided the application has the appropriate tools and permissions.

The investigation develops through a sequence of decisions and observations.

An operation produces information. That information changes the system's understanding of the task. The updated understanding influences which operation, if any, should be performed next.

The Relationship Between actions and observations

The interaction is closely related to the ideas explored in ReAct: Synergizing Reasoning and Acting in Language Models, by Yao and colleagues.

ReAct investigates how language models can combine reasoning-oriented outputs with actions that interact with external environments. The observations those environments return can then inform the model's subsequent behavior.

For our document-analysis example, you can understand the pattern without adopting any particular framework.

The model receives a task and the information currently available. It proposes an operation that may help resolve an uncertainty. The application executes the operation and returns an observation. The model then uses that observation to determine how to continue.

The important feature is not the presence of a specific tool or the number of mode calls.

It is the fact that information obtained during execution can influence subsequent action selection.

This introduces flexibility that a fixed execution sequence may not provide.

The model can respond to unexpected information, investigate newly discovered references, and adapt its actions to the evidence available at each stage.

However, this does not mean that every agent must use the exact method described in ReAct.

Some systems select actions incrementally without maintaining an explicit plan. Others generate a plan before execution and revise it when new information appears. Some combine model-driven decisions with deterministic subworkflows.

These approaches trade off flexibility, predictability, cost, and operational complexity. The distinction, then, is not whether an application uses a language model, invokes tools, or performs multiple operations. A deterministic workflow can do all of those things while keeping its execution path under conventional software control.

An agent becomes useful when the next action cannot be reliably determined in advance, and the model can help select an appropriate operation based on what the system has discovered.

That flexibility is valuable, but it changes the responsibilities of the surrounding application.

If the model can decide which operation to request next, what happens when it selects an inappropriate tool, repeats an unsuccessful action, or encounters instructions embedded in an untrusted document?

The question is no longer only whether dynamic action selection can help complete a task. It is also how the application can maintain control over an execution path that is not entirely predetermined.

We will examine these engineering problems in the next article, Dynamic Decisions Create New Engineering Responsibilities.

References & Further Reading

Anthropic - Building Effective Agents

Yao et al. - ReAct: Synergizing Reasoning and Acting in Language Models

Schick et al. - Toolformer: Language Models Can Teach Themselves to Use Tools

OpenAI - A Practical Guide to Building Agents