Artifact first: a documentation sample that has never been executed is not a sample. It is a hypothesis. Conclusion: if your docs build does not compile and run the code it publishes, your documentation is a claim your CI has never tested.
Consider a quickstart that fails in front of a customer because the sample calls a method that was renamed two releases earlier. The docs build is green. The API build is green. Nothing connects them. That is the gap in miniature: two pipelines, one contract, zero enforcement.
What “docs as code” actually commits you to
The Write the Docs guide on Docs as Code lists five practices: issue trackers, version control, plain-text markup, code review, and automated tests. Four of those are about where the text lives. The fifth is about whether it is true. Most teams adopt the first four and quietly skip the fifth, because the first four are cheap and the fifth requires deciding what “true” means for a snippet.
That decision is the whole job. A snippet is not a unit test. It has no assertion, no fixture, no teardown, and often no imports. To put it in CI you have to give it all four, or accept that you are only checking syntax.
Three levels of snippet verification, and what each one catches
Level 1: parse. Run the snippet through the language’s parser or a linter. This catches typos, unbalanced brackets, and syntax that changed between language versions. It does not catch a renamed method, a changed argument order, or a removed endpoint. It is cheap and it is better than nothing, which is the faintest praise available.
Level 2: compile or type-check against the real client library. This catches signature drift. If your snippet calls client.createUser(name, email) and the library now takes a request object, the compiler tells you. This is the level most teams should target first, because signature drift is the most common way a published sample goes stale.
Level 3: execute against a live or mocked service and assert on the response. This catches everything above plus behavioral drift: the endpoint that now returns 202 instead of 200, the pagination cursor that changed shape, the idempotency key that is now required. It is also the level that costs the most to maintain, because it needs a service to talk to and a way to keep that service’s state clean between runs.
My recommendation, stated as a recommendation and not a finding: start at level 2 for every snippet, and promote to level 3 only the snippets that appear in a quickstart or a migration guide. Those are the ones a reader will copy verbatim into a fresh project, and those are the ones whose failure is most expensive.
Extraction versus inline: the trade-off nobody documents
There are two ways to get a snippet into CI. You can extract it from the prose at build time, or you can keep it inline in a file that the docs build includes and the test suite imports.
Extraction keeps the prose as the source of truth. The reader sees exactly what the test ran, because the test ran the reader’s copy. The cost is that extraction is fragile: a fence marker, an indentation change, or a stray blank line can silently change what gets extracted, and a snippet that fails to extract may be skipped rather than failed.
Inline inclusion keeps the code as the source of truth. The snippet lives in a real file with real imports, and the docs build pulls it in. The cost is that the reader may see a fragment that does not stand alone, and the prose has to explain the surrounding context that the file provides.
Both work. The failure mode to avoid is the hybrid where the snippet is extracted, the extraction fails, and the pipeline treats a missing snippet as a passing test. If your extractor cannot find a snippet, that is a failure, not a skip.
Where the CI actually runs
GitHub Actions workflows are YAML files stored in .github/workflows, triggered by events such as push or pull_request, with optional path filters so a docs-only change does not rebuild the world. GitLab CI/CD uses .gitlab-ci.yml, with include for pulling in shared configuration and rules for conditional jobs. Azure Pipelines combines continuous integration, continuous testing, and continuous delivery, and runs test tasks across the major language ecosystems.
The mechanics differ; the shape does not. You need a job that runs on the same trigger as the code it documents, checks out the same revision, installs the same dependency versions, and fails the build when a snippet does not compile or run. If the docs job runs on a different schedule than the code job, you have built a staleness generator with extra steps.
One detail worth getting right: pin the dependency versions in the docs job to the versions the snippet claims to support. A snippet that compiles against latest and fails against the version in your support matrix is not verified. It is verified against a moving target.
What to measure, and what not to
Postman’s API testing material frames the distinction between testing, which happens during development to catch issues before production, and monitoring, which happens after deployment and includes telemetry for historical analysis. Snippet verification is testing. It belongs in the pipeline, not in a dashboard someone checks on Fridays.
Useful signals, in order of how much they tell you:
- Snippet pass rate per release: the fraction of published snippets that compiled and ran in the release pipeline. A drop here is a leading indicator of support tickets.
- Time since last successful execution per snippet: staleness. A snippet that has not run in six months is a liability regardless of whether it currently passes.
- Extraction failures: snippets the build could not find. These should fail the build, and the count should be zero.
Signals that look useful and are not: total snippet count, docs build duration, and the number of snippets with a test file. None of those tell you whether a reader can copy the code and get a working result.
The decision record you should write before you build this
The reason snippet CI stalls is rarely technical. It is that nobody wrote down which snippets are covered, which level of verification applies to each, and who owns the fix when a snippet breaks because the API changed. That is an architecture decision record, and it should be short.
It needs to state: the scope (which docs directories are in), the verification level per category, the trigger (same as the code it documents), the failure policy (build fails, no exceptions), and the owner (the team that owns the API, not the docs team). The last one matters most. If the docs team owns snippet failures, the API team has no incentive to keep the contract stable, and the gap reopens.
A postmortem template helps here too. When a snippet ships broken, the postmortem should answer one question: why did the pipeline not catch this? The answer is usually that the snippet was not in scope, or that the extraction silently skipped it. Both are fixable, and both are invisible until you ask.
What I would do differently
I would not start with a general-purpose extractor. I would start with a list of the twenty snippets that appear in quickstarts and migration guides, put them in real files, and wire those files into the existing test suite. That is a week of work, not a quarter, and it covers the snippets whose failure costs the most.
I would also fail the build on extraction errors from day one. The temptation to warn instead of fail is strong, and it is how you end up with a green pipeline and a broken quickstart. A warning that nobody reads is a skip with better branding.
FAQ
Do I need to run every snippet against a live service?
No. Compile or type-check against the real client library catches the most common failure, which is signature drift. Reserve live execution for snippets a reader will copy into a fresh project.
What if my snippets are in a language without a compiler?
Use the language’s parser or linter for syntax, and add a smoke test that imports the module and calls the function with representative arguments. That gets you most of the way to level 2.
How do I keep the docs job from slowing down every pull request?
Use path filters so the job runs only when the docs or the client library change. GitHub Actions supports paths filters on push and pull_request; GitLab supports rules:changes.
Who should own snippet failures?
The team that owns the API. If the docs team owns them, the API team has no feedback loop, and the snippets will drift again.
Is this worth it for a small API?
If you publish a quickstart, yes. The cost of one broken quickstart in a support ticket is usually higher than the cost of wiring twenty snippets into CI.
Sources
- Write the Docs, Docs as Code — the five practices, including automated tests.
- GitHub, Workflow syntax for GitHub Actions — workflow file location, triggers, and path filters.
- GitLab, CI/CD YAML syntax reference —
include, rules, and job configuration.
- Microsoft, What is Azure Pipelines? — continuous integration, continuous testing, and test task support.
- Postman, API testing — the testing-versus-monitoring distinction and common API test types.
Heuristic: a snippet that has never run in CI is a hypothesis, not a sample. Treat extraction failures as build failures, and put the twenty quickstart snippets in real files before you build a general-purpose extractor.