Sphinx integreren in uw CMake build systeem

By Eli van Es

Sphinx is a documentation tool for software projects that can create easily searchable HTML pages and excellent PDFs among other output types. It is well suited for documenting scientific code bases due to its support for LaTeX-style equations and numerous extensions for plots, diagrams and much more. Compared to LaTeX, Sphinx provides more output types, like HTML pages, has a lighter syntax and is more suitable for documenting APIs. Compared to Doxygen, the output and layout of Sphinx can be more easily configured and Sphinx’s reStructuredText format provides more features for writing technical documentation.

It’s not uncommon for scientific software to be written in C, Fortran or C++, for which the de-facto build-system is CMake. For ease of use and to pass information from CMake to Sphinx, it would be nice if we could integrate the two for our CMake-based projects. This blog describes an approach to do just that.

The Goal

After configuring our CMake build-system using something along the lines of:

cmake ~/Projects/myproject

we would like to be able to build all our documentation with a simple:

make doc

Lets say we have a few documentation source files in Sphinx’s reStructuredText format index.rst, introduction.rst, model.rst and api.rst, and a Sphinx configuration file conf.py with some fields to be filled in by CMake:

# ...

copyright = '@SPHINX_TARGET_YEAR@, VORtech'

# ...

version = '@SPHINX_TARGET_VERSION_MAJOR@.@SPHINX_TARGET_VERSION_MINOR@'
release = '@SPHINX_TARGET_VERSION@'

# ...

latex_documents = [
    (master_doc, '@SPHINX_TARGET_NAME@.tex', 'My User Manual',
     author, 'manual'),
]

# ...

We would like to be able to specify our documentation target in a fashion that feels familiar to CMake, almost like how a library target is specified with a add_library(...) call:

find_package(Sphinx)

if(Sphinx_FOUND)
    add_sphinx_document(
        my-user-manual
        CONF_FILE "${CMAKE_CURRENT_LIST_DIR}/doc/user/conf.py"
        "${CMAKE_CURRENT_LIST_DIR}/doc/user/index.rst"
        "${CMAKE_CURRENT_LIST_DIR}/doc/user/introduction.rst"
        "${CMAKE_CURRENT_LIST_DIR}/doc/user/models.rst"
        "${CMAKE_CURRENT_LIST_DIR}/doc/user/api.rst")
else()
    message(WARNING "No Sphinx found. Documentation target not available.")
endif()

Finding Sphinx From CMake

To make sure the CMake call to find_package(Sphinx) works, we need to add a file called FindSphinx.cmake. This file will contain some CMake machinery for figuring out where Sphinx is installed on your system:

include(FindPackageHandleStandardArgs)

# We are likely to find Sphinx near the Python interpreter
find_package(PythonInterp)
if(PYTHONINTERP_FOUND)
    get_filename_component(_PYTHON_DIR "${PYTHON_EXECUTABLE}" DIRECTORY)
    set(
        _PYTHON_PATHS
        "${_PYTHON_DIR}"
        "${_PYTHON_DIR}/bin"
        "${_PYTHON_DIR}/Scripts")
endif()

find_program(
    SPHINX_EXECUTABLE
    NAMES sphinx-build sphinx-build.exe
    HINTS ${_PYTHON_PATHS})
mark_as_advanced(SPHINX_EXECUTABLE)

find_package_handle_standard_args(Sphinx DEFAULT_MSG SPHINX_EXECUTABLE)

The call to find_program(...) will attempt to find the sphinx-build executable on your system. The call to find_package_handle_standard_args(...) is there to notify CMake about the result of the find-attempt, setting Sphinx_FOUND, dealing with the find_package REQUIRE argument, etc.

Function for Creating the Target

Next up is implementing the add_sphinx_document(...) function for our CMakeLists.txt-file. We can add it at the end of our FindSphinx.cmake:

# If finding Sphinx fails, there is no use in defining
# add_sphinx_document, so return early
if(NOT Sphinx_FOUND)
    return()
endif()

# add_sphinx_document(
#   <name>
#   CONF_FILE <conf-py-filename>
#   [C_API <c-api-header-file>]
#   [SKIP_HTML] [SKIP_PDF]
#   <rst-src-file>...)
#
# Function for creating Sphinx documentation targets.
function(add_sphinx_document TARGET_NAME)
    # ...
endfunction()

First part of the function is parsing the arguments:

cmake_parse_arguments(
    ${TARGET_NAME}
    "SKIP_HTML;SKIP_PDF"
    "CONF_FILE"
    ""
    ${ARGN})

We will go into the purpose of these arguments in the following steps.

For use further below, we will set up variables for the Sphinx source, intermediate and output directories:

get_filename_component(SRCDIR "${${TARGET_NAME}_CONF_FILE}" DIRECTORY)
set(INTDIR "${CMAKE_CURRENT_BINARY_DIR}/${TARGET_NAME}/source")
set(OUTDIR "${CMAKE_CURRENT_BINARY_DIR}/${TARGET_NAME}/build")

Next up, we configure the Sphinx conf.py file to include information from CMake, such as ${PROJECT_VERSION}:

string(TIMESTAMP SPHINX_TARGET_YEAR "%Y" UTC)

add_custom_command(
    OUTPUT "${INTDIR}/conf.py"
    COMMAND "${CMAKE_COMMAND}" -E make_directory "${INTDIR}"
    COMMAND
        "${CMAKE_COMMAND}"
        "-DCONFIGURE_FILE_IN=${${TARGET_NAME}_CONF_FILE}"
        "-DCONFIGURE_FILE_OUT=${INTDIR}/conf.py"
        "-DSPHINX_TARGET_NAME=${TARGET_NAME}"
        "-DSPHINX_TARGET_VERSION=${PROJECT_VERSION}"
        "-DSPHINX_TARGET_VERSION_MAJOR=${PROJECT_VERSION_MAJOR}"
        "-DSPHINX_TARGET_VERSION_MINOR=${PROJECT_VERSION_MINOR}"
        "-DSPHINX_TARGET_YEAR=${SPHINX_TARGET_YEAR}"
        -P "${_SPHINX_SCRIPT_DIR}/BuildTimeConfigureFile.cmake"
    DEPENDS "${${TARGET_NAME}_CONF_FILE}")

set(SPHINX_DEPENDS "${INTDIR}/conf.py")

This references a new CMake file BuildTimeConfigureFile.cmake. Since we need to run this command at build-time rather than at CMake configure-time, we cannot use CMake’s configure_file(...) function directly. Instead we have defined this small wrapper called BuildTimeConfigureFile.cmake to be called at build-time by the custom-command:

configure_file("${CONFIGURE_FILE_IN}" "${CONFIGURE_FILE_OUT}" @ONLY)

Back to our add_sphinx_document(...) function, we are also referencing a variable ${_SPHINX_SCRIPT_DIR} for finding BuildTimeConfigureFile.cmake. This can be defined at the top of FindSphinx.cmake as:

set(_SPHINX_SCRIPT_DIR ${CMAKE_CURRENT_LIST_DIR})

Note that we cannot use ${CMAKE_CURRENT_LIST_DIR} directly inside the add_sphinx_document(...) function, since then it will not contain the path to FindSphinx.cmake, but the path to CMakeLists.txt-file that is calling add_sphinx_document(...).

Another important thing to note about the call the add_custom_command are the DEPENDS and OUTPUT options. These make sure that the build-system will re-run the command whenever the output gets outdated compared to the source conf.py-file. This is very useful since it will make our documentation source very much like our regular program source, rebuilding along with the rest of the project as changes are made.

Continuing our implementation of add_sphinx_document(...), we add a loop to copy the Sphinx source files to the intermediate directory:

foreach(DOCFILE ${${TARGET_NAME}_UNPARSED_ARGUMENTS})
    get_filename_component(DOCFILE_INTDIR "${DOCFILE}" DIRECTORY)
    string(
        REPLACE
        "${SRCDIR}" "${INTDIR}"
        DOCFILE_INTDIR "${DOCFILE_INTDIR}")

    get_filename_component(DOCFILE_DEST "${DOCFILE}" NAME)
    set(DOCFILE_DEST "${DOCFILE_INTDIR}/${DOCFILE_DEST}")

    add_custom_command(
        OUTPUT "${DOCFILE_DEST}"
        COMMAND
            "${CMAKE_COMMAND}" -E make_directory "${DOCFILE_INTDIR}"
        COMMAND
            "${CMAKE_COMMAND}" -E copy_if_different
            "${DOCFILE}" "${DOCFILE_DEST}"
        DEPENDS "${DOCFILE}")

    list(APPEND SPHINX_DEPENDS "${DOCFILE_DEST}")
endforeach()

Now that we have the commands set up for getting our Sphinx source files into the intermediate directory, we can add the commands for calling Sphinx to build the actual documentation. The following command will build the Sphinx HTML output:

set(TARGET_DEPENDS)

if(NOT ${TARGET_NAME}_SKIP_HTML)
    add_custom_command(
        OUTPUT "${OUTDIR}/html.stamp"
        # Create the _static directory required by Sphinx in case it
        # wasn't added as one of the source files
        COMMAND "${CMAKE_COMMAND}" -E make_directory "${INTDIR}/_static"
        COMMAND "${SPHINX_EXECUTABLE}" -M html "${INTDIR}" "${OUTDIR}"
        COMMAND "${CMAKE_COMMAND}" -E touch "${OUTDIR}/html.stamp"
        DEPENDS ${SPHINX_DEPENDS})

    list(APPEND TARGET_DEPENDS "${OUTDIR}/html.stamp")
endif()

Note the ${SPHINX_DEPENDS} list that was built in the previous steps to serve as DEPENDS for this command.

Since the Sphinx HTML build will output many different files, we use a html.stamp-file to serve as the build-system’s source of determining if the output is up-to-date.

The next command is for calling Sphinx to build the PDF output:

if(NOT ${TARGET_NAME}_SKIP_PDF)
    find_package(LATEX COMPONENTS PDFLATEX)

    if(LATEX_PDFLATEX_FOUND)
        add_custom_command(
            OUTPUT "${OUTDIR}/latex/${TARGET_NAME}.tex"
            COMMAND "${SPHINX_EXECUTABLE}" -M latex "${INTDIR}" "${OUTDIR}"
            DEPENDS ${SPHINX_DEPENDS})

        add_custom_command(
            OUTPUT "${OUTDIR}/latex/${TARGET_NAME}.pdf"
            # Three times' the charm for PdfLaTeX to get all xrefs right
            COMMAND "${PDFLATEX_COMPILER}" "${TARGET_NAME}.tex"
            COMMAND "${PDFLATEX_COMPILER}" "${TARGET_NAME}.tex"
            COMMAND "${PDFLATEX_COMPILER}" "${TARGET_NAME}.tex"
            WORKING_DIRECTORY "${OUTDIR}/latex"
            DEPENDS "${OUTDIR}/latex/${TARGET_NAME}.tex")

        list(APPEND TARGET_DEPENDS "${OUTDIR}/latex/${TARGET_NAME}.pdf")
    else()
        message(WARNING "No PdfLaTeX found. PDF output not available.")
    endif()
endif()

We have wrapped the previous two steps into ifs checking for the SKIP_HTML and SKIP_PDF arguments of the add_sphinx_document(...) function such that its users can opt-out of one of the two forms of documentation. Obviously this could also be implemented the other way around if you prefer an opt-in mechanism.

The PDF command above can be simplified using Sphinx’s latexpdf builder instead of latex, however, this requires the latexmk-script and Perl. We would rather not add these as dependencies, but if that is no issue for your use case, then the block above can become:

if(NOT ${TARGET_NAME}_SKIP_PDF)
    add_custom_command(
        OUTPUT "${OUTDIR}/latex/${TARGET_NAME}.pdf"
        COMMAND "${SPHINX_EXECUTABLE}" -M latexpdf "${INTDIR}" "${OUTDIR}"
        DEPENDS ${SPHINX_DEPENDS})

    list(APPEND TARGET_DEPENDS "${OUTDIR}/latex/${TARGET_NAME}.pdf")
endif()

Now that all our commands are in place, we can create the CMake targets for the documentation:

add_custom_target(
    ${TARGET_NAME}
    DEPENDS ${TARGET_DEPENDS})

if(NOT TARGET doc)
    add_custom_target(doc)
endif()
add_dependencies(doc ${TARGET_NAME})

You could also use add_custom_target(doc ALL) instead of add_custom_target(doc) if you would like the doc target to be build whenever the default target is build.

Closing Remarks

We have introduced a reusable FindSphinx.cmake-script that allows us to easily add documentation targets to our CMakeLists.txt-files in a way that makes the documentation sources very similar to the program sources. Being able to pass information from CMake to Sphinx allows us to specify things such as version numbers in a single place.

It took quite a few steps to set up and while it does not alleviate the need to write good documentation, it does provide a build-system to make this easier.

Scientific Software Engineering