Integrating an LLM into an Agent Without Frameworks.

Integrating an LLM into an Agent Without Frameworks.

What we will build. We will extend the deterministic API-latency investigation from Part 01. The model will propose the next action; the application will decide whether that proposal is valid, execute a simulated observation, and update the state. No agent framework is required.

From fixed Steps to Model-Driven Decisions

In Part 01, our latency-investigation agent followed a predetermined sequence: inspect latency, inspect deployment information, then summarize the evidence. The loop was useful because it made execution state, step limits, and termination explicit. But it could not adapt its next step to an unexpected observation.

Suppose the first check reports a P95 latency of 850 ms. Should the agent inspect a recent deployment, collect another metric, or stop? A fixed sequence answers before seeing the evidence. In this version, we ask an LLM to propose the next action based on the current task and recorded observations. The important change is decision selection, not unrestricted execution.

Figure 1. Part 01 selects predefined steps. Part 02 introduces a model decision, but retains the controller.

Defining the LLM's Responsibility

We separate three responsibilities. The model proposes an action and a reason. The validator checks the proposal against a closed set of allowed actions and the current state. The controller executes the authorized action, records its observation, and enforces the termination limit. A plausible model response is not automatically a permitted operation.

For this experiment, the available actions are observe_latency, inspect_deployment, and finish. They return simulated observations. The model cannot create new action names or invoke arbitrary Python functions. Even a syntactically valid finish proposal does not prove the incident was resolved; it only requests termination of this investigation.

Representing State as Model Context

An LLM request does not automatically know what happened in previous loop iterations. We must serialize the relevant state into the request: the task, available actions, prior observations, and any execution constraints. The application owns this state; the prompt is a representation of it, not the authoritative record.

    task = "Investigate an increase in API latency"
    observations = ["P95 latency is 850 ms"]
    allowed_actions = ["observe_latency", "inspect_deployment", "finish"]

    context = (
        f"Task: {task}\n"
        f"Allowed actions: {', '.join(allowed_actions)}\n"
        f"Observations: {observations}"
    )

A useful context contains enough evidence for the next decision without copying unrelated logs or secrets. Later iterations rebuild it from updated state. In production systems, context selection also affects token cost, latency, and exposure of sensitive data.

Making Our First LLM Request

We will use the official OpenAI Python SDK and the responses API. Structured Outputs can parse a response into a Pydantic model. The constraints the response shape, but it does not establish that the selected action is appropriate. Configure the OPENAI_API_KEY and OPENAI_MODEL in the environment; choose a model that supports structured outputs.

import os
from typing import Literal
from openai import OpenAI
from pydantic import BaseModel

class AgentDecision(BaseModel):
    action: Literal["observe_latency", "inspect_deployment", "finish"]
    reason: str

    client = OpenAI()
    response = client.responses.parse(

        model=os.environ["OPENAI_MODEL"],
        instructions=("Choose one allowed action for an API latency "
                      "investigation. Do not execute actions."),
        input=context,
        text_format=AgentDecision,
    )

    if response.status != "completed" or response.output_parsed is None:
        raise RuntimeError("No completed structured decision")

    decison = response.output_parsed

This is a proposed integration, not a measured result. The request has not been executed for this article version. When running it, record the selected action, latency, token usage, and any refusal or API error instead of assuming a particular model answer.

Turning Model Output into a Decision

A schema answers a structural question: Is the action one of the declared strings, and is the reasoning a string? Application validation answers a different question: is this action permitted now? For example, finishing before collecting any observation may violate our investigation policy, even though finish is a valid schema value.

def validate_decision(decision: AgentDecision, observations: list[str]):
    
    if decision.action == "finish" and not observations:
        raise ValueError("Cannot finish before collecting evidence")
    
    if decision.action == "observe_latency" and any(
        "P95 latency" in item for item in observations
    ):
        raise ValueError("Latency was already observed")

    return decision

These are deliberately simple teaching policies, not universal rules for incident response. The validator is a trust boundary: model output remains untrusted until the application accepts it.

Figure 2. The OpenAI adapter returns a proposal. Only the controller can turn an accepted proposal into an observation or terminal state.

Integrating Decisions into the Execution Loop.

We can now replace the fixed step selector with a decision provider. The provider can be backed by the OpenAI API or a fake implementation for testing. The Controller remains responsible for the loop. Its maximum step count is a hard bound, regardless of what the model requests.

def run_agent(provider, execute, task, max_steps=4):
    observations = []

    for _ in range(max_steps):
        decision = provider.decide(task, observations)

        validate_decision(decision, observations)

        if decision.action == "finish":
            return {
                "status": "completed",
                "observations": observations,
            }

        observation = execute(decision.action)  # Simulated action
        observations.append(observation)

    return {
        "status": "step_limit",
        "observations": observations,
    }

Handling Invalid Responses and Failures

There are at least three distinct failure classes. A transport failure prevents the API call from completing. A parsing or refusal outcome prevents us from obtaining a usable structured decision. A policy failure occurs when a well-formed decision is invalid for the current state. These should not be collapsed into a generic retry.

For the first implementation, fail closed: do not execute an action if parsing or validation fails. Record the failure and terminate with an explicit error state. You can add retries, backoff, and fallback decisions later, but only with bounded attempts and a documented policy. Logging should exclude credentials and unnecessary sensitive context.

Testing the LLM-Driven Loop

Live API calls test integration, but their outputs can vary. Controller correctness needs deterministic tests. A fake provider can return a predefined sequence of decisions, including invalid ones, without network access or token cost. Test the acceptance of a valid action, rejection, or premature finish, and the enforcement of the step limit.

class FakeProvider:
    def __init__(self, decisions):
        self.decisions = iter(decisions)

    def decide(self, task, observations):
        return next(self.decisions)


fake = FakeProvider([
    AgentDecision(
        action="observe_latency",
        reason="Collect baseline",
    ),
    AgentDecision(
        action="finish",
        reason="Evidence collected",
    ),
])

result = run_agent(
    fake,
    lambda action: "P95 latency is 850 ms",
    "Investigate API latency",
)

assert result["status"] == "completed"
assert len(result["observations"]) == 1

The test checks control flow, not whether the model made the best investigative decision. In a separate live experiment, capture the observed action, elapsed time, and usage data across repeated identical inputs. Do not infer model reliability from a single successful call.

What Built and What Comes Next

We started with a deterministic loop and introduced one new capability: selecting the next action from model-generated output. The resulting architecture keeps task state in the application, sends relevant context to the LLM, validates its structured proposal, and executes only authorized actions within explicit limits.

The trade-off is clear. Model-driven selection can adapt to observations, but it adds latency, cost, nondeterminism, and new failure modes. Structured output improves interface reliability; it does not replace engineering judgment or application policy. In Part 03, we can evolve the simulated executor into explicit tool calling while preserving these boundaries.

References & Further Reading

OpenAI. Responses API reference
OpenAI. Structured Outputs guide
OpenAI. Official Python SDK
Pydantic. Models documentation
VictorHSN. Building an AI Agent Without Framework