CloudSignal AI

Blog

Reliable JSON from an LLM: structured outputs and function calling

By Ashwin Rajendraprasad Tutorials

The fastest way to make an LLM feature production-grade is to stop treating its output as prose and start constraining it to a schema. You’ll build a small pipeline that returns validated, typed JSON every time, then extend it to function calling so the model can pick a tool and hand you typed arguments.

1. The problem: prose out, exceptions in

Free-text output is unparseable at scale. “Just ask for JSON” gets you JSON most of the time. Then a stray sentence of preamble, a trailing comma, or a hallucinated field takes your parser down at 2 a.m. Two guarantees fix this: the output is shaped the way your code expects, and you catch it when it isn’t. That’s structured outputs plus validation, and the pattern is the same across every provider, so we’ll keep the code vendor-neutral.

2. Structured outputs: hand the model a schema

Major LLM APIs support structured outputs: you supply a JSON Schema and the API constrains the response to conform to it. Start by writing the shape you want back (types, required fields, enums, nesting):

# structured.py (Python 3.11+)
# pip install "requests==2.34.2" "jsonschema==4.23.0"
import os, json, requests
from jsonschema import validate, ValidationError

BASE_URL = os.environ["LLM_BASE_URL"]   # any OpenAI-compatible endpoint
API_KEY  = os.environ["LLM_API_KEY"]
MODEL    = os.environ["LLM_MODEL"]      # a model your endpoint serves that supports JSON-schema output

# The exact shape we demand back.
INVOICE_SCHEMA = {
    "type": "object",
    "additionalProperties": False,
    "required": ["vendor", "total", "currency", "line_items"],
    "properties": {
        "vendor":   {"type": "string"},
        "total":    {"type": "number"},
        "currency": {"type": "string", "enum": ["USD", "EUR", "GBP"]},
        "line_items": {
            "type": "array",
            "items": {
                "type": "object",
                "additionalProperties": False,
                "required": ["description", "amount"],
                "properties": {
                    "description": {"type": "string"},
                    "amount":      {"type": "number"},
                },
            },
        },
    },
}

Then request it in structured-output mode. Using an OpenAI-compatible Chat Completions endpoint (the shape most providers and many local runtimes now speak), that’s the response_format field, as documented in OpenAI’s Structured Outputs guide:

def ask_json(prompt):
    r = requests.post(
        f"{BASE_URL}/chat/completions",
        headers={"Authorization": f"Bearer {API_KEY}"},
        json={
            "model": MODEL,
            "messages": [{"role": "user", "content": prompt}],
            "response_format": {                       # structured-output mode
                "type": "json_schema",
                "json_schema": {"name": "invoice",
                                "schema": INVOICE_SCHEMA,
                                "strict": True},
            },
        },
        timeout=120,
    )
    r.raise_for_status()
    return r.json()["choices"][0]["message"]["content"]

Anthropic expresses the same idea through tool use with an input schema. Different field names, identical concept. Whichever provider you use, note the promise precisely: structured outputs constrain the shape of the reply, not the truth of the values inside it.

3. Validate, then trust

Because “schema mode” support and strictness vary by provider and model, never trust the shape blindly: validate it yourself and retry on failure. It’s belt-and-suspenders, and it’s the line between a demo and a service. Here’s a provider-neutral loop using the jsonschema library:

def extract(prompt, schema, max_tries=3):
    last_error = None
    for _ in range(max_tries):
        ask = prompt if last_error is None else (
            f"{prompt}\n\nYour previous reply was invalid ({last_error}). "
            "Return only JSON matching the schema."
        )
        raw = ask_json(ask)
        if raw is None:                                # refusal: content comes back null
            raise RuntimeError("Model declined the request; re-asking will not help.")
        try:
            data = json.loads(raw)
            validate(instance=data, schema=schema)     # jsonschema
            return data
        except (json.JSONDecodeError, ValidationError) as e:
            last_error = str(e).splitlines()[0]
    raise RuntimeError(f"No schema-valid JSON after {max_tries} tries: {last_error}")

if __name__ == "__main__":
    text = "Acme Tools: 2 hammers at $18 each, 1 saw at $34. Total $70, USD."
    print(extract(f"Extract this invoice as JSON:\n{text}", INVOICE_SCHEMA))

Three tries, each re-asking with the specific validation error, then a hard failure you can log and alert on. Those logged failures are the best signal you’ll get for improving the schema or the prompt.

The raw is None check earns its line. When a model declines, an OpenAI-compatible endpoint returns null content with the reason in a separate refusal field, so json.loads(raw) raises TypeError and sails straight past an except clause that only expects malformed JSON. Refusals get their own exit because retrying one just spends money to be told no again (see section 6).

4. Function calling: let the model pick a tool

Structured outputs get you typed data. Function (tool) calling gets you typed actions: you describe one or more tools (each a name plus a JSON Schema for its arguments), the model chooses one and emits arguments, your code executes it, and you feed the result back. The request carries the tool definitions; the reply carries a structured call instead of prose:

# --- function calling: typed actions, not just typed data ---
TOOLS = [{
    "type": "function",
    "function": {
        "name": "get_exchange_rate",
        "description": "Convert an amount between two ISO currency codes.",
        "parameters": {
            "type": "object",
            "additionalProperties": False,
            "required": ["amount", "from_currency", "to_currency"],
            "properties": {
                "amount":        {"type": "number"},
                "from_currency": {"type": "string"},
                "to_currency":   {"type": "string"},
            },
        },
    },
}]

def get_exchange_rate(amount, from_currency, to_currency):
    rate = live_rate(from_currency, to_currency)   # your real implementation
    return {"converted": amount * rate, "rate": rate}

def call_with_tools(prompt):
    r = requests.post(
        f"{BASE_URL}/chat/completions",
        headers={"Authorization": f"Bearer {API_KEY}"},
        json={"model": MODEL,
              "messages": [{"role": "user", "content": prompt}],
              "tools": TOOLS},
        timeout=120,
    )
    r.raise_for_status()
    msg = r.json()["choices"][0]["message"]
    for call in msg.get("tool_calls", []):
        args = json.loads(call["function"]["arguments"])   # typed, per our schema
        return get_exchange_rate(**args)                   # you execute it,
        # ...then feed the result back as a tool message for the model's final reply
    return msg.get("content")

This is the app-level mechanism behind “agents.” The argument schema does the same job as the response schema in section 2: it keeps the boundary between the model and your code typed and checkable. (Field names differ across providers; Anthropic’s tool-use API carries the same idea with its own shapes, so check the docs linked below.)

5. Where this meets MCP

Tool calling is the pattern; the Model Context Protocol, the subject of our first MCP server tutorial, is a standard way to transport it. Write a tool once behind MCP and many applications can call it, instead of re-wiring the same function into every app. Three altitudes of one idea: structured outputs give you typed data, tool calling gives you typed actions, and MCP gives you a standard socket those actions plug into. Point it at your retrieval pipeline and “answer from my documents” becomes a typed, callable tool.

6. The failure modes

  • Schema-valid ≠ semantically correct. The model can return a perfectly-shaped, wrong answer (a valid date that’s the wrong date, a total that doesn’t add up). Validate the values, not just the types: check ranges, enums, and cross-field arithmetic against your own rules.
  • Cap and watch your tokens. A response truncated by a token limit is invalid JSON. Set a generous max_tokens, and treat truncation as a validation failure that triggers a retry.
  • Handle refusals. A model may decline rather than answer; detect the refusal path explicitly instead of parsing it as data.
  • Bound your retries. Retry a small, capped number of times and back off. An infinite re-ask loop against a genuinely impossible request just burns money.

Get those right and an LLM stops being a text generator you hope to parse and becomes a typed component you can build on. Wiring LLMs to real systems — where a malformed call means a failed transaction, not a bad paragraph — is precisely where a CloudSignal architecture review earns its keep.

Sources / further reading

Written by Ashwin Rajendraprasad for CloudSignal AI. The code above is free to reuse.