LLM-Powered Documentation Generation

Documentation is the first thing to go stale and the last thing developers want to update. LLMs can change that — not by replacing human-written docs, but by generating first drafts and flagging when existing docs drift from the code.

The Documentation Pipeline

Our approach: extract code context, generate docs, diff against existing docs, and create PRs for review.

defmodule Docs.Generator do
  def generate_for_module(module_path) do
    source = File.read!(module_path)

    prompt = """
    Generate documentation for this Elixir module.
    Include:
    - Module overview (2-3 sentences)
    - Each public function: purpose, params, return value, example usage
    - Any important notes about side effects or error handling

    Format as Markdown. Be concise and accurate.

    ```elixir
    #{source}
    ```
    """

    {:ok, docs} = LLM.complete(prompt, model: "claude-sonnet-4-6")
    {:ok, docs}
  end
end

Drift Detection

The real value is catching when docs don’t match code:

defmodule Docs.DriftDetector do
  def check(module_path, doc_path) do
    source = File.read!(module_path)
    existing_docs = File.read!(doc_path)

    prompt = """
    Compare this code against its documentation.
    Identify any discrepancies:
    - Functions documented but not in code
    - Functions in code but not documented
    - Parameter mismatches
    - Incorrect descriptions

    Code:
    ```elixir
    #{source}
    ```

    Documentation:
    ```markdown
    #{existing_docs}
    ```

    Respond with JSON:
    {
      "is_current": true/false,
      "issues": [{"type": "missing|outdated|incorrect", "description": "..."}]
    }
    """

    {:ok, response} = LLM.complete(prompt)
    Jason.decode!(response)
  end
end

Integrating into CI

Run drift detection on every PR that touches source code:

defmodule Docs.CI do
  def check_pr(changed_files) do
    changed_files
    |> Enum.filter(&source_file?/1)
    |> Enum.map(fn file ->
      doc_path = source_to_doc_path(file)

      if File.exists?(doc_path) do
        Docs.DriftDetector.check(file, doc_path)
      else
        %{is_current: false, issues: [%{type: "missing", description: "No docs for #{file}"}]}
      end
    end)
    |> Enum.filter(&(not &1.is_current))
  end
end

What Works Well

LLMs are excellent at generating API reference documentation, function-level docstrings, and changelog entries. They’re less reliable for architectural overviews and tutorials, which require understanding intent and context that goes beyond the code.

The sweet spot: let the LLM generate the first draft, then have a human review and refine. This cuts documentation time by roughly 60% while maintaining quality.

Changelog Generation

One particularly effective workflow — auto-generating changelogs from commit history:

defmodule Docs.Changelog do
  def generate(from_tag, to_tag) do
    {:ok, commits} = Git.log(from_tag, to_tag)

    prompt = """
    Generate a user-facing changelog from these git commits.
    Group by: Added, Changed, Fixed, Removed.
    Write for end users, not developers — focus on what changed, not how.
    Skip internal refactoring and dependency updates.

    Commits:
    #{format_commits(commits)}
    """

    {:ok, changelog} = LLM.complete(prompt)
    {:ok, changelog}
  end
end

This runs as part of our release process. The LLM draft goes into a PR, gets a quick human review, and ships with the release. No more “we’ll write the changelog later” that never happens.