Structured Outputs and JSON Mode for Reliable Pipelines

The number one cause of LLM pipeline failures? Parsing errors. The model returns something almost-JSON, or wraps it in markdown code fences, or adds a helpful preamble before the actual data. Structured output mode fixes this.

The Problem

Without explicit constraints, LLMs love to be helpful:

Sure! Here's the classification:

```json
{"category": "billing", "confidence": 0.92}

Let me know if you need anything else!

Your `Jason.decode!/1` call just exploded. The actual JSON is buried in markdown. This happens 5-10% of the time — enough to break production.

## Solution 1: API-Level JSON Mode

Most LLM providers now support structured output modes:

```elixir
defmodule LLM.Structured do
  def complete(prompt, schema) do
    {:ok, response} = LLM.API.complete(prompt,
      response_format: %{
        type: "json_object",
        schema: schema
      }
    )

    Jason.decode!(response)
  end
end

When available, this is the best approach. The model is constrained at the token generation level to only produce valid JSON matching your schema.

Solution 2: Prompt-Level Enforcement

When API-level support isn’t available, enforce it in the prompt:

@prompt """
<instructions>
Classify the support ticket into a category.
</instructions>

<output_format>
Respond with ONLY a JSON object. No markdown, no explanation, no preamble.
Schema: {"category": "billing|technical|account|other", "confidence": 0.0-1.0}
</output_format>

<input>
<%= ticket_text %>
</input>
"""

XML tags help the model distinguish between instructions and output format. The explicit “No markdown, no explanation, no preamble” cuts down on wrapper text.

Solution 3: Robust Parsing

Belt and suspenders — even with JSON mode, parse defensively:

defmodule LLM.Parser do
  def parse_json(response) do
    response
    |> String.trim()
    |> strip_markdown_fences()
    |> strip_preamble()
    |> Jason.decode()
  end

  defp strip_markdown_fences(text) do
    text
    |> String.replace(~r/^```json\s*/m, "")
    |> String.replace(~r/^```\s*/m, "")
    |> String.trim()
  end

  defp strip_preamble(text) do
    case Regex.run(~r/\{[\s\S]*\}/, text) do
      [json] -> json
      nil -> text
    end
  end
end

Schema Validation with Ecto

Use Ecto embedded schemas to validate LLM output with proper types:

defmodule LLM.Schemas.Classification do
  use Ecto.Schema
  import Ecto.Changeset

  @primary_key false
  embedded_schema do
    field :category, Ecto.Enum, values: [:billing, :technical, :account, :other]
    field :confidence, :float
  end

  def validate(attrs) do
    %__MODULE__{}
    |> cast(attrs, [:category, :confidence])
    |> validate_required([:category, :confidence])
    |> validate_number(:confidence, greater_than_or_equal_to: 0, less_than_or_equal_to: 1)
    |> apply_action(:validate)
  end
end

The Reliability Stack

For production use, layer all three:

1. Use API-level JSON mode when available
2. Structure your prompt with explicit output format instructions
3. Parse defensively with fallback extraction
4. Validate against a typed schema

With all four layers, our JSON parsing success rate went from ~92% to 99.97%. The remaining 0.03% are genuine model errors that get caught and retried.