A Task That Requires More Than a Response.

A Task That Requires More Than a Response

You have just joined a team responsible for an order-processing system. One of your first tasks is to understand what happens after a customer places an order, particularly how the order service communicates with the payment service.

According to the architecture diagram, the order service sends an HTTP request to the payment service and waits for a response before continuing. That appears straightforward enough, until you notice something different in the integration specification: payments are initiated through an OrderCreated event published to a message broker.

Now you have two descriptions of what appears to be the same operation, but they imply different system actions.

If the services communicate synchronously, the order service may have to wait for the payment service to respond. If communication happens through a message broker, payment processing can continue independently, and the system needs some way to handle the eventual result.

These are not minor documentation differences. They affect how you reason about failures, response times, and the consistency of an order's state.

So which description should you trust?

The diagram might be outdated. The event-driven integration might belong to a migration that was never completed. It is also possible that both mechanisms exist but serve different purposes.

At this point, reading two documents again will not necessarily resolve the uncertainty. You need more information about how the system was designed and how it actually behaves.

An architectural decision record might explain when the communication pattern changed. The service's source code could reveal which integration is currently implemented. A deployment configuration might show whether the message broker is part of the production environment. Each source answers a different question, and none should automatically be treated as the complete picture.

An engineer investigating this difference would probably start by deciding which evidence is most useful. If the documentation references a migration, they might examine the corresponding architectural decision. If that record is missing or inconclusive, they might inspect the code or ask someone familiar with the system.

The investigation develops as new information becomes available. What the engineer discovers determines what they need to examine next.

Now imagine asking an AI-powered application to perform the same investigation.

You provide access to the technical documents and request a report identifying architectural inconsistencies, supported by references to the relevant documents. An LLM could compare the available descriptions and explain why synchronous HTTP communication differs from event-driven messaging. It might even suggest plausible reasons for the discrepancy.

But identifying possible explanations is not the same as determining which explanation the evidence supports.

To investigate the discrepancy, the application may need to locate additional documents, inspect their contents, compare the information they contain, and decide whether another operation is necessary before producing the report.

This changes the nature of the problem. Instead of using a language model only to interpret information prepared in advance, we are considering a system in which the model can help decide how an investigation should proceed.

That is one motivation behind modern LLM-based agents.

Generating an Answer Is Not the Same as Completing a Task

Suppose we copy the contents of the architecture diagram and integration specification into a prompt and ask an LLM to find inconsistencies. ​ The model might correctly observe that one document describes synchronous HTTP communication while the other describes asynchronous messaging. It could explain how these approaches differ and identify questions that an engineer should investigate.

For that particular request, a single model interaction might be sufficient. The application has already provided the relevant information. The model's responsibility is to examine it and respond.

Now consider a slightly different request:

Investigate the connection between the order service and the payment service, determine whether the technical specifications correspond to the current architecture, and produce an evidence-based report.

This request requires more than comparing two descriptions.

The system needs to determine what constitutes sufficient evidence. The documentation might be outdated, the implementation might have changed, or the two integration mechanisms might coexist. A report that ignores these possibilities could present a documentation discrepancy as an implementation defect.

To complete the investigation, the application may need to interact with external resources. It might search a repository for architectural decision records, retrieve additional specifications, or inspect relevant configuration files.

These operations add responsibilities that don't belong to language generation itself.

A model can generate a statement claiming that it inspected an architectural decision record. That statement does not establish that the record was actually retrieved. Similarly, a model can produce a convincing explanation of how a system works without having examined the implementation.

For an engineering report, this distinction matters because conclusions should be traceable to evidence.

If the system claims that a migration replaced synchronous communication with asynchronous messaging, we should be able to identify the information supporting that conclusion. If a referenced document could not be accessed, the report needs to acknowledge that limitation rather than silently assume its contents.

The application therefore needs a way to distinguish between information that was actually retrieved, interpretations generated from that information, and hypotheses that remain unverified.

It also needs to coordinate the operations required to gather evidence.

A language model need not select these operations. A conventional application can retrieve documents, inspect predefined resources, and organize the information before sending it to an LLM.

For many tasks, that approach is preferable. The architectural question becomes more interesting when the application cannot determine every necessary operation before the investigation begins.

If the next useful step depends on what the system discovers, we need to consider how it should decide.

The challenge, then, is not simply to give a language model access to more information or more tools. It is to design a system that can determine what evidence is needed, obtain it, and use what it discovers to decide how to proceed. Understanding when decisions must follow predefined rules and when they might benefit from a language model is where the distinction between conventional LLM applications, deterministic workflows, and agents becomes important.

References & Further Reading

Anthropic - Building Effective Agents Yao et al. - ReAct: Synergizing Reasoning and Acting in Language Models Schick et al. - Toolformer: Language Models Can Teach Themselves to Use Tools OpenAI - A Practical Guide to Building Agents