Can LLM use tool calling when structured output is enabled?

chatlab

An interactive model-communication inspector. One conversation at a time, with the wire visible.

┌─ CHAT ──────────────────────────┬─ WIRE ────────────────────────────┐
│ what a human reads:             │ what actually went over the wire: │
│ your messages, the answer,      │ the exact request JSON, every     │
│ one line per tool call/result   │ token as it arrives, usage, 400s  │
└─────────────────────────────────┴───────────────────────────────────┘

Both panes are timestamped Y-m-d H:M:S, and each is written to its own file. Switch models mid-conversation, toggle tools, switch how structured output is requested, watch cost accumulate.

It exists because the interesting questions about an LLM are rarely "is it good" — they are did it even see my tool, what did that schema turn into on the wire, why is this model silent. Those are invisible in a normal chat UI, and they are one glance away here.

---

Quick start

python -m pip install -r requirements.txt   # or: python -m pip install -e .

cp .env.example .env        # add OPENAI_API_KEY and/or NEBIUS_API_KEY
python -m chatlab           # http://127.0.0.1:8137

The virtualenv lives outside this repo. Use whatever environment and interpreter you like — nothing in here names one, and nothing writes inside the chatlab/ package.

python -m chatlab                   # web UI
python -m chatlab --tui             # terminal renderer; /help for commands
python -m chatlab --refresh-models  # re-fetch provider catalogues, then run
python -m chatlab --replay 'logs/wire-*.jsonl'   # re-print a past session, spends nothing

Flags are only per-launch things: --config --port --model --db --env-file --refresh-models --tui --replay -v. Everything else is a control you can click, which is why it is not a flag.

Where files go

work_dir is the directory holding the chatlab.yaml that was loaded.

logs/, sandbox/, providers.yaml and data/ all resolve against it, so the app behaves the same from any working directory and never writes inside the installed package. chatlab.yaml is found by --config, then $CHATLAB_CONFIG, then ./chatlab.yaml, then the one beside the package — so from a checkout, plain python -m chatlab just works.

Credentials come from the environment only. Search order: --env-file, $CHATLAB_ENV, ./.env. A real environment variable always wins, so OPENAI_API_KEY=sk-… python -m chatlab needs no file. Never pass a key as an argument — argv is visible in the process table.

The model catalogue

28 models ship with the app: 5 OpenAI + 23 Nebius, with price, context window, region and claimed capabilities. No network and no API key needed to start.

Every provider is an OpenAI-compatible /v1/chat/completions endpoint, so a provider is data, not code — adding one is a YAML entry and nothing else:

# providers.yaml
providers:
  - name: local-ollama
    base_url: http://127.0.0.1:11434/v1/
    api_key_env: OLLAMA_API_KEY
    discover: false          # this endpoint publishes no pricing; don't ask it
    models:
      - model: llama3.2
        context_length: 131072
        features: [tools]

file	what it is
providers.yaml	who you can talk to. Each entry is a base URL, a key env var, and either a hand-written model list (with pricing, because most vendors publish none) or discover: true to read /v1/models.
data/catalog-<provider>.json	the last discovery result for a discovering provider, bundled so the app starts offline. Refresh with --refresh-models [PROVIDER].

For a discovering provider, rows under models: are overlays: only the keys you actually write are applied, so adding a note to a model never blanks the price discovery found for it.

Prices are USD per 1M tokens. An unknown price stays null and the cost column reads n/a — a made-up number is worse than an absent one. pricing_unit is stated per provider rather than sniffed, because per-token and per-million differ by a factor of a million and guessing wrong silently is worse than showing nothing.

--refresh-models rewrites the snapshot in place, so a price change is a one-command update and shows up in a diff. Region matters if you handle personal data: EU / UK / US are reported separately and an unrecognised country code is reported as itself rather than guessed.

supported_features is a claim, not a measurement. Several models that do not list structured_outputs handle schemas fine, and several that do list it still cannot combine a response format with tools. See below.

Structured output vs tool calling — read this before debugging a "broken" model

Measured across the catalogue, same prompt, same tool:

model	claims structured_outputs	off	tool	provider	provider_final
gpt-4o-mini (OpenAI)	yes	1 call	2 calls	1 call	3 calls
zai-org/GLM-5.3-Flash	no	1 call	2 calls	0 calls	—
Qwen/Qwen3.5-397B-A17B	no	1 call	2 calls	0 calls	—
deepseek-ai/DeepSeek-V4-Flash	yes	1 call	2 calls	0 calls	3 calls
openai/gpt-oss-120b	yes	1 call	1 call	error	—

provider mode sends response_format: {type: json_schema} alongside your tools. OpenAI can still emit a tool call under that constraint. No Nebius model can — the schema constrains generation, so the model narrates what it would query ("Let me check the database schema for you.") and never calls anything. It fails silently: no error, no warning, just a model that seems to have forgotten its tools.

Why — measured at the wire, not inferred

19 raw requests to /v1/chat/completions, stdlib urllib, no LangChain in the way, same system prompt and same run_sql schema chatlab itself sends. Against deepseek-ai/DeepSeek-V4-Flash-0731:

request	tool calls
tools, no response_format	1 ✅
tools + json_schema	0
tools + json_schema, strict: true	0
tools + json_object	0
tools + json_schema, strict removed from the tool	0
tools + json_schema + tool_choice: "required"	0 — silently ignored
tools + json_schema + tool_choice: {name: run_sql}	0 — silently ignored
tools + json_schema, mid-loop (a tool result already in history)	0
tools + json_schema, permissive schema (additionalProperties: true)	0
no tools + json_schema	n/a — schema honoured, so the JSON path is healthy

So it is not the prompt, not LangChain's injected tool strict: true, not schema strictness, and not json_schema specifically — any constrained decoding suppresses tool calls. Two rows say it outright:

• json_object + tools makes the model emit {"sql_query": "SELECT name FROM sqlite_master ..."} as content. The tool-call intent survives; the grammar redirects it into the text channel.

• zai-org/GLM-5.3-Flash refuses the combination in so many words:

  400 - tool_choice 'required' or a named tool cannot be combined with response_format,
        regex, or ebnf: the tool-call constraint and the output constraint cannot both
        be honored.
  

  DeepSeek ignores the same request silently and Qwen honours the tool_choice instead of the schema (2 tool calls) — three backends, three behaviours, one underlying conflict.

There is no provider-side escape hatch. response_format: {type: "structural_tag"} is rejected (422 - Input should be 'text', 'json_object' or 'json_schema'), and vLLM's own guided_json is accepted and silently ignored — as is not_a_real_parameter_at_all, which is how we know.

The fix: provider_final

Don't fight the grammar — stop sending it on the calls where a tool call is what you want. provider_final runs every step of the loop unconstrained with the tools bound, and only once the model stops asking for tools does it re-issue that same request under response_format with tool_choice: "none". One extra model call per turn, on the last step.

provider        1 request   tools + response_format          -> narration, 0 tool calls
provider_final  3 requests  tools                            -> run_sql(...)
                            tools                            -> draft, no tool call
                            tools + response_format + none   -> {"response": "Categories, ..."}
                                    ^ carrying the draft + "return that as JSON"

That last call reformats; it does not re-answer. Without the draft in front of it the model writes a fresh answer under the grammar, and a fresh answer is a much shorter one: google/gemma-3-27b-it turned a four-point reply into "Hello! How can I help you debug today?" — 12% of the original. The two trailing messages are not cosmetic either; measured n=4 on two providers:

final-call shape	gemma-3-27b (Nebius)	gpt-4o-mini (OpenAI)
no draft, re-answer	shortened to 12%	preserved
draft as the last message	preserved	echoes the SCHEMA back, 4/4
draft + "return that as JSON"	preserved	preserved
draft inside one user turn	HTTP 400 (wants role alternation)	preserved

The last request is the payload every model in the table accepts. tools: [] would be tidier and is not portable — Qwen/Qwen3.5-397B-A17B answers 400 - "tools" must not be an empty array. Either provide at least one tool or omit the field entirely, and OpenAI rejects it too; pinning tool_choice keeps the body byte-identical to the request already measured to work.

It is stock LangChain — an AgentMiddleware (deferred.py) on create_agent(middleware=...), no new dependency. If you want tools to work on a Nebius model, use provider_final (or tool, or off).

One more trap, and it is not about tools: a grammar will not fill an optional field. Asked for a chart with charts: Optional[list] = None in the schema, the model drew ASCII art inside response instead — 0/5 times did it use the field. Make payload fields required with [] legal, and say so in the prompt.

The orchestrator — the other way out

provider_final defers the schema. The orchestrator never creates the conflict: it splits the two mechanisms across two roles that each only ever do one thing.

orchestrator   no tools, one structured output:        sub-agent   tools,
               either "call these agents" or                       no response_format,
               "here is the final answer"                          answers in prose

Nothing asks a model to do the impossible thing, so nothing needs a middleware and nothing depends on LangChain internals that might move. One turn, list db tables, four requests:

POST · 2 msgs · response_format: json_schema          -> delegate to sql
POST · 2 msgs · tools: run_sql                        -> run_sql(...)
POST · 4 msgs · tools: run_sql                        -> prose report
POST · 3 msgs · response_format: json_schema          -> {"response": "Categories, ..."}

Toggle it with the orchestrated checkbox, or /orch in the TUI. It is orthogonal to the structured mode, which still decides the shape of the final answer.

The union must be tagged. The action schema is a discriminated union on a kind field, not two optional fields — measured, n=5 on two backends: tagged, 29/30 correct branch choices; two optional fields, the model fills neither 9 times in 10. Same lesson as charts: Optional[...]: a grammar takes the shortest valid path, and "both null" is the shortest one.

Sub-agents come from the enabled tool groups: sqlite → sql, fs → files. max_rounds bounds the delegate→report loop.

What you can change mid-session

Model · structured mode · either tool on/off · system prompt · temperature · max_tokens · reasoning_effort · recursion_limit · whether the optional schema fields are required.

Every control routes through Session.apply(), so the web UI and the TUI share one surface, and every change is an event — it lands in both panes and both logs, in order. Your wire log shows the moment you flipped a setting, sitting between the request that failed and the one that worked.

Re-run replays the last message against the currently selected model, after truncating history back to where that turn started — so the second model sees exactly what the first one saw.

Structured modes

One schema — response plus optional confidence, tags, sources — and seven ways of asking:

mode	what goes on the wire
off	plain text
auto	a bare pydantic type; create_agent picks. It keys off the model name, so anything behind a base_url always lands on ToolStrategy regardless of what it supports
provider	response_format: {type: json_schema} — and it injects strict: true into your tool schemas too
provider_final	the same, but applied only to the call that ends the turn, so the tool loop runs unconstrained. That last call hands the draft back to be reformatted, not re-answered. One extra model call; the only structured mode in which tools work on Nebius
tool	the schema as a second tool, with tool_choice: "required" on every step — the model can never stop without calling something
json_mode	response_format: {type: json_object} + the schema in the prompt
prompted	format instructions in the prompt, parsed at the end

"require confidence/tags/sources" makes the three optional fields mandatory — a quick way to see a model produce a perfectly good answer and have it rejected for missing bookkeeping.

Tools

• sqlite — read-only, over the bundled data/northwind.db (--db or chatlab.yaml to point elsewhere). Read-only is the connection URI (file:…?mode=ro), not a SELECT prefix check: SQLite allows WITH … INSERT, so the prefix check alone is bypassable and exists only to return a friendly message. Verified — a CTE-smuggled write comes back attempt to write a readonly database. The description names the dialect, because omitting it makes models emit INTERVAL '90 days' and DATEADD, which SQLite rejects.
• fs — list_files / read_file / write_file inside sandbox/ (gitignored). Writes run immediately, no approval step. Escapes are refused by resolving both sides, which covers ../ and symlinks alike.

Logs

logs/chat-<stamp>.log mirrors the chat pane verbatim. logs/wire-<stamp>.jsonl is one event per line — the pane's whole value is byte-exact JSON, and pretty text would either truncate it or become a 600-line wall per turn that nothing can query. --replay renders it back through the same function that drew the pane. Both files open with a header carrying the full settings, so a log is self-describing a month later. Reset closes both and opens a new pair.

Per-token lines are coalesced into one response event per model call carrying {ttft_s, tok_per_s, chunks, usage_reported}. Set log_tokens: true only to debug streaming itself.

Two honesty rules worth knowing when you read them:

• usage_reported: false means the provider sent no usage block and the token count is a chunk-count proxy, not a measurement.
• tok_per_s: null means the step arrived as a single chunk, so there was nothing to measure between. Better than a confident 49000000 tok/s.

A wire log contains every prompt you sent. logs/ is gitignored; keep it that way.

Configuration

chatlab.yaml — default model, system prompt, sampling params, recursion_limit, structured mode, tool paths, server host/port, log settings. Everything in it is also a control in the UI; the file just sets where a session starts. Its directory is work_dir, so every path it contains is relative to itself.

params.reasoning_effort: null means do not send it, which lets a model's own extra_args win. That matters: gpt-5.6-luna carries reasoning_effort: "none" because without it every tool-bound call returns

400 - Function tools with reasoning_effort are not supported for gpt-5.6-luna in
      /v1/chat/completions. To use function tools, use /v1/responses or set
      reasoning_effort to 'none'.

Precedence is providers.yaml extra_args < chatlab.yaml params < CLI < UI, and overriding a model's own default is announced in the status line rather than done quietly.

How it hangs together

chatlab.yaml        session defaults -- and its directory is work_dir
providers.yaml      who you can talk to, and what a call costs
data/               catalogue snapshots + the demo database
logs/  sandbox/     everything the app writes, never inside the package
chatlab/            code, plus the one asset it serves

events.py      the Event dataclass + chat_line()/wire_line() -- the whole UI boundary
session.py     state, the two logs, the fan-out; apply/run/rerun/reset
turn.py        create_agent + astream -> timestamped events
wire.py        the httpx request hook that captures the exact outbound body
catalog.py     Provider/Candidate/ModelSpec, the bundled data, discovery, usage + cost
client.py      init_chat_model with the capture client
structured.py  the schema and the seven modes
deferred.py    ProviderStrategy held back until the call that ends the turn
orchestrator.py  a model with no tools, sub-agents with no schema
tools.py       sqlite + fs
web.py         FastAPI: one GET /events SSE stream, small POSTs in
tui.py         the second renderer, stdlib only
static/index.html   the entire UI: no npm, no build step, no framework

Session.emit() is sync and the renderers are pure Event -> str. The log files are a renderer — chat.log is literally chat_line() appended to a handle. The TUI exists mainly to keep that boundary honest: with one front-end, web-shaped assumptions drift into the core within a week, and a consumer that imports only asyncio, session and events makes that a syntax error.

The web UI is one long-lived GET /events SSE stream plus small POSTs — not SSE-as-the-response- to-the-POST. That is what lets a control click land in the stream as an ordered, timestamped event while a turn is running, and it means EventSource does the reconnecting for us.

Traps worth knowing (all verified, several the hard way)

1. model.bind(stream_usage=True) does not survive create_agent: bind_tools resolves through RunnableBinding.__getattr__ to the unwrapped model and drops it. It must be a constructor kwarg.
2. Passing http_async_client disables langchain-openai's automatic stream_usage (it self-enables only when every client field is None and there is no base_url). The wire capture and the token counts are coupled — set stream_usage=True explicitly or every cost reads zero.
3. stream_mode must be a list containing "messages". Drop it and the request silently becomes "stream": false — a wire-format switch, not a display preference.
4. Any response_format routes the SDK through beta.chat.completions.stream(), which refuses non-strict tools. json_mode + tools will raise; chatlab says so before you hit it.
5. Never raise_for_status() in an httpx response hook — it destroys APIStatusError.body. response.elapsed raises there too, so timing uses our own monotonic clock.
6. recursion_limit belongs in the RunnableConfig, not in create_agent. LangGraph's default is 25 super-steps, and one step can contain many parallel tool calls.
7. YAML 1.1 reads a bare off as boolean False — hence structured_mode: "off" is quoted.

Ideas

• A side-by-side mode: same prompt, two models, two chat panes, one wire pane each.
• Record the tool/provider matrix above automatically, as a command, so it stays current as the catalogue changes.
• Token-level timing histogram — TTFT is recorded per call already; the shape of the gaps is where streaming stalls show up.