Greetings & Salutations!
sheldonhull.com
Back to posts

Asciidoc Distributed Docs as Code

devops docs-as-code tech

The Problem

  • I want to keep my code and my documentation in the same place.
  • I want to separate the presentation layer from my documentation content.
  • I want to be flexible to publish documentation to a variety of endpoints and formats as processes evolve, without my content being impacted.
  • I want to write a solid runbook for things that can’t be fully automated, but still include scripts and other code in their native format.

Documentation is such an important part of a developer’s life. I think we often take it for granted, and it’s an afterthought in many projects. However, as I consider my work, I know that I’m not reinventing the wheel very often 😀. Most of what I do is built on the back of others’ work. When I use tooling, I’m reading the documentation and using it as my basis to get work done. When I use my notes and blog posts as a reference, I’m using my informal version of knowledge gathering.

INVEST in documenting your work as you go, for the person behind you. You don’t find time to do it, you make time to do it while you work, as a first class citizen of your work, not an after-thought. Think of all the times you’ve had to dig for answers and save someone else that experience.

You code and document not as much for yourself, but for the person that comes behind you.

Asciidoctor

I’ve found a happy solution in the Asciidoctor documentation format over markdown. You can go google this for more expanded understanding, but I’ve decided that other than for basic notes and blog posts which are very simplistic, I now choose Asciidoctor.

Why use Asciidoc format over markdown comes down to the needs of technical documentation.

Here are some key reasons why I’ve found Asciidoc format to be worth learning:

  • I can reference code files with a simple include::file[] statement, while markdown would require me to embed my code directly as a code block.
  • I can generate a table from a csv file, further helping me automate a refresh of the underlying data that is rendered to a table display
  • I can create tables much more cleanly and with control than in markdown, even allowing nested tables for complicated process documentation.
  • Automatic admonition callouts without extensions using simple statements like IMPORTANT: foo

Presentation

Since the common documentation system used where I am at is Confluence, I decided to leverage the incredible confluence-publisher project that made this entire process a breeze. Check the repo and the linked documentation out here: Confluence Publisher

In the future, if I didn’t use confluence, I’d explore rendering as a static website through Hugo (that’s what this site is generated from) or revisit Antora and maybe merge my content into the format required by Atora programmatically.

Use Docker

Since Asciidoc is written in Ruby, use docker and you won’t have to deal with dependency nightmares, especially on Windows.

$RepoDirectoryName = 'taco-ops-docs'
echo "🌮🌮🌮🌮🌮🌮🌮🌮🌮🌮🌮🌮🌮"
echo "Running confluence publisher 🌮"
echo "📃 Publishing $RepoDirectoryName repo contents"

docker run --rm -v $BUILD_SOURCESDIRECTORY/$RepoDirectoryName/docs:/var/asciidoc-root-folder -e ROOT_CONFLUENCE_URL=$ROOT_CONFLUENCE_URL \
-e SKIP_SSL_VERIFICATION=false \
-e USERNAME=$USERNAME \
-e PASSWORD=$PASSWORD \
-e SPACE_KEY=$SPACE_KEY \
-e ANCESTOR_ID=$ANCESTOR_ID \
-e PUBLISHING_STRATEGY=$PUBLISHING_STRATEGY \
confluencepublisher/confluence-publisher:0.0.0-SNAPSHOT
echo "📃 Publishing $RepoDirectoryName repo contents finished"

Yes… I know. I get bored reading log messages when debugging so my new year premise was to add some emoji for variety. Don’t judge. 😁

Distributed Docs Structure

So the above approach is fantastic for a single repo. I wanted to take it to a different level by solving this problem for distributed documentation. By distributed I meant that instead of containing all the documentation in a single “wiki” style repo, I wanted to grab documentation from the repositories I choose and render it. This would allow the documentation related to being contained in the repository it is related to.

For instance, what if I wanted to render the documentation in the following structure:


** General Documentation**
taco-ops-runbook
---> building-tacos
--------> topic.adoc
---> eating-tacos
--------> topic.adoc
---> taco-policies
--------> topic.adoc
---> taco-as-code
--------> topic.adoc

** Repo Oriented Documentation**
github-repos
---> taco-migration
--------> category-1
------------> topic.adoc
------------> topic.adoc
--------> category-2
------------> topic.adoc
------------> topic.adoc
---> taco-monitoring
--------> category-1
------------> topic.adoc
------------> topic.adoc
--------> category-2
------------> topic.adoc
------------> topic.adoc

The only current solution found was Antora. Antora is very promising and great for more disciplined documentation approaches by software development teams. The limitation I faced was complexity and rigidity in structure. For Antora to generate a beautiful documentation site, you have to ensure the documentation is structured in a much more complex format. For example, the docs might be under docs/modules/ROOT/pages/doc.adoc and have a nav.adoc file as well. While this promises a solid solution, retrofitting or expecting adoption might be tricky if your team has never even done markdown.

Azure DevOps Pipeline

I ended using an Azure DevOps pipeline (YAML of course 🤘) that provides a nice easy way to get this done.

First, for proper linking, you should follow the directions Azure DevOps gives on the creation of a Github Service Connection which uses OAUTH. This will ensure your setup isn’t brittle and using your access token.

Code 
name: $(BuildDefinitionName).$(DayOfYear)$(Rev:.r).$(Build.RequestedFor)

trigger:
 branches:
  include:
    - master

pr: none # don't trigger on pr, only when merging to master

pool:
  vmImage: 'ubuntu-latest'

variables:
  - group: confluence-publisher # Go to Library in Azure taco-ops and create your variables below, so you can easily change if needed in future
    #  ROOT_CONFLUENCE_URL
    #  USERNAME
    #  PASSWORD
  - name: SPACE_KEY
    value: taco-ops
  - name: ANCESTOR_ID
    value: 123456 # 🚩 the parent page you want to put everything under. Anything under this will be 🚩🚩🚩🚩 purged 🚩🚩🚩🚩

    # this is the repo to send distributed docs from github repos, so you can seperate the repos from your main run book
  - name: GITHUB_ANCESTOR_ID
    value: 1234567 # 🚩 the parent page you want to put everything under. Anything under this will be 🚩🚩🚩🚩 purged 🚩🚩🚩🚩

  - name: PUBLISHING_STRATEGY
    value: APPEND_TO_ANCESTOR

  - name: VERSION_MESSAGE # I'm not using, but could be possible to use gitversion tool or other to tag this stuff with semver automatically. I choose not to do that so it doesn't create more possible noise right now in updates
    value: ''

  - name: CONSOLIDATED_FOLDER_NAME # This is a folder to copy all the github matched repo doc folders to so you can generate subpages automatically on all of them
    value: 'consolidated-documentation'

resources:
  repositories:
  - repository: taco-ops-runbook
    type: github
    endpoint: github-taco-ops-connection
    name: taco/taco-ops-runbook

  - repository: terraform-taco
    type: github
    endpoint: github-taco-ops-connection
    name: taco/terraform-taco

  - repository: taco-ops-packer
    type: github
    endpoint: github-taco-ops-connection
    name: taco/taco-ops-packer
    # ref: develop

  - repository: taco-ops-reports
    type: github
    endpoint: github-taco-ops-connection
    name: taco/taco-ops-reports

stages:
- stage: PublishDocumentation
  jobs:
    - deployment: taco-ops-runbook
      displayName: Publish taco-ops-runbook
      timeoutInMinutes: 5
      environment: taco-ops
      strategy:
        runOnce:
          deploy:
            steps:
              - checkout: taco-ops-runbook
                clean: false
                persistCredentials: true

              - checkout: terraform-taco
                clean: false
                persistCredentials: true

              - checkout: taco-ops-packer
                clean: false
                persistCredentials: true

              - checkout: taco-ops-reports
                clean: false
                persistCredentials: true

              # didn't get working due to multiple repos, so disabled, but kept here for reference
              # - task: GitVersion@5
              #   inputs:
              #     runtime: 'core'
              #     configFilePath: '$(Build.SourcesDirectory)/taco-ops-runbook/GitVersion.yml'
              # fetchDepth: 1 #GITVERSION needs history to calculate property versioning
              # https://confluence-publisher.atlassian.net/wiki/spaces/CPD/overview?mode=global

              - pwsh: |
                  Write-Host "Searching for docs folders under `$ENV:PIPELINE_WORKSPACE: $ENV:PIPELINE_WORKSPACE"
                  $Directories = (Get-ChildItem -Path $ENV:PIPELINE_WORKSPACE -Directory -Filter 'docs' -Recurse | Where-Object FullName -notmatch 'taco-ops-runbook').FullName
                  Write-Host "--- Directories to Copy ---`n$($Directories | Format-Table -Autosize -Wrap | Out-String)"
                  Write-Host "👌 Creating folder called $ENV:CONSOLIDATED_FOLDER_NAME in current path to simplify multi-repo doc publishing"
                  $consolidatedfolder = (New-Item -Path $ENV:CONSOLIDATED_FOLDER_NAME -ItemType Directory -Force).FullName

                  $Directories | ForEach-Object {
                    $dir = $_
                    Write-Host "👌 Copying contents of $dir into $consolidatedfolder"
                    Copy-Item $dir -Recurse -Destination $consolidatedfolder -Verbose -Force
                  }
                  Write-Host "🎉 done copying contents, here is a report of the files in destination for your ease of debugging"
                  Write-Debug "$(Get-ChildItem $consolidatedfolder -Recurse | Format-Table -Autosize -Wrap | Out-String)"

                displayName: Copy GitHub docs to central folder to simplify adding repos for publishing
                env:
                  CONSOLIDATED_FOLDER_NAME: $(CONSOLIDATED_FOLDER_NAME)

              - bash: |
                  echo "🌮🌮🌮🌮🌮🌮🌮🌮🌮🌮🌮🌮🌮"
                  echo "Running confluence publisher 🌮"
                  echo "📃 Publishing taco-ops-runbook repo contents"
                  docker run --rm -v $BUILD_SOURCESDIRECTORY/taco-ops-runbook/docs:/var/asciidoc-root-folder -e ROOT_CONFLUENCE_URL=$ROOT_CONFLUENCE_URL \
                    -e SKIP_SSL_VERIFICATION=false \
                    -e USERNAME=$USERNAME \
                    -e PASSWORD=$PASSWORD \
                    -e SPACE_KEY=$SPACE_KEY \
                    -e ANCESTOR_ID=$ANCESTOR_ID \
                    -e PUBLISHING_STRATEGY=$PUBLISHING_STRATEGY \
                    confluencepublisher/confluence-publisher:0.0.0-SNAPSHOT

                  echo "📃 Publishing $CONSOLIDATED_FOLDER_NAME repo contents"

                  docker run --rm -v $BUILD_SOURCESDIRECTORY/$CONSOLIDATED_FOLDER_NAME/docs:/var/asciidoc-root-folder -e ROOT_CONFLUENCE_URL=$ROOT_CONFLUENCE_URL \
                    -e SKIP_SSL_VERIFICATION=false \
                    -e USERNAME=$USERNAME \
                    -e PASSWORD=$PASSWORD \
                    -e SPACE_KEY=$SPACE_KEY \
                    -e ANCESTOR_ID=$GITHUB_ANCESTOR_ID \
                    -e PUBLISHING_STRATEGY=$PUBLISHING_STRATEGY \
                    confluencepublisher/confluence-publisher:0.0.0-SNAPSHOT

                  echo "documentation should be published"
                  echo "🌮🌮🌮🌮🌮🌮🌮🌮🌮🌮🌮🌮🌮"
                workingDirectory: $(Build.SourcesDirectory)
                displayName: 'run confluence publisher'
                name: confluencepublisher
                env:
                  ROOT_CONFLUENCE_URL: $(ROOT_CONFLUENCE_URL)
                  USERNAME: $(USERNAME)
                  PASSWORD: $(PASSWORD)
                  SPACE_KEY: $(SPACE_KEY)
                  ANCESTOR_ID: $(ANCESTOR_ID)
                  PUBLISHING_STRATEGY: $(PUBLISHING_STRATEGY)
                  VERSION_MESSAGE: $(VERSION_MESSAGE)
                  CONSOLIDATED_FOLDER_NAME: $(CONSOLIDATED_FOLDER_NAME)
                timeoutInMinutes: 5

              # - task: printAllVariables@1
              #   condition: always()

Things to Know

  • Ensure you use the format shown here for documentation to render in confluence correctly. You need to have the names match in the doc/folder for it to know to render the child pages.
** Repo Oriented Documentation**
taco-ops-repo
README.adoc  -- optional, but I always include for landing page, and point to the docs folder using link:./docs/myrepo.adoc
---> [docs]
------> [resources]  -- optional, but keeps the scripts organized and consistent, or any images
------> process.adoc
------> another-process.adoc
---> taco-ops-repo.adoc
  • Include your scripts by using include::./resources/_myscript.ps1[]. You may have to test that relative path issue if doing multiple repos.
  • Ensure your non-asciidoc contents are prefaced with an underscore in the title name. I don’t like this, but it’s a requirement from confluence-publisher. This ensures it won’t try to render as a page.
  • Anything in the target directory (ancestor) gets purged in the process. I recommend a dedicated confluence space you create just for this to minimize risk and disable manual edits.