NQBP Gen2 is a Python based build system that I have used in some form or another over many years building embedded projects. NQBP stands for Not Quite Benv–Python. The primary features of NQBP that I have come to rely on are:

• Adding a new file, renaming, or deleting a file in an existing directory requires zero effort.
• Adding a new source directory to a build is done simply by adding the new directory’s name and path as a single-line entry to a single file.
• Supporting both Windows and Linux host platforms.
• Not having to deal with makefiles.

A more detailed list of NQBP Gen2 features:

• Multi-host build engine for C, C++, and assembler builds.
• Targeted specifically for embedded development.
• Speed. Uses Ninja as the underlying build system.
• Full dependency checking and incremental builds.
• Command line based.
• Supports many compiler toolchains.
• Source code reusability. That is, NQBP assumes that code will be shared across many projects.
• Reusability of compiler toolchains. That is, after a particular compiler toolchain has been created or defined, it can be reused across an unlimited number of projects.

What is Benv?

Benv is the original build system that NQBP gets its how-to-select-source-files to build and no-makefiles paradigm from. I have included a brief history NQBP at the end of article for the curious reader.

What is Gen2?

NQBP Gen2 is a major improvement over NQBP classic. The significant change is that Gen2 use the ninja build system for dependency checking and the invoking the compiler/linker toolchain. Beside faster builds, by using ninja I was able to include dependency checking and incremental builds (without a lot of effort) to NQBP. Effectively removing the Not Quite qualifier with respect to the original Benv system.

Why another build engine?

Besides the obvious answer of: because I can ;-). The practical answer is that it is very simple to create the ‘build scripts’ for project, i.e. just specify a toolchain and list of directories to build and your done. After the second time I created a new project using this paradigm I was hooked. I get frustrated with the complexity of other build system when compared to NQBP. A developer shouldn’t have to buy a book or take classes (or both) in order to specify what files to build for a project (yes, I am not fan of CMake).

Of course there is some pain with NQBP, for example creating a new toolchain for a new compiler or a new executable output. But this scenario happens maybe handful of times during a project. Where as adding, moving, renaming, and deleting files happens all of the time. YMMV.

Selecting What to Build

The principal mechanism for selecting which files to build is the libdirs.b file in the build directory and the optional sources.b files in source directories. The sources.b file simply contains file names which are listed singly on separate lines of the file. The libdirs.b file contains the directory names which are listed singly on separate lines of the file and specify which directories to compile and link. However, there is additional syntax and semantics for the libdirs.b file.

• Directories are referenced relative to the NQBP_PKG_ROOT directory.
• Entries in the file can reference another libdirs.b files.
• Blank lines or lines starting with # are ignored.
• Operating system environment variables can be referenced using leading and trailing $ characters to identify directories or partial directory paths.
• An entry that starts with enclosed within square brackets ( [ ] ) will only be compiled when matches the build variant specified when NQBP is invoked.
• An optional pipe symbol ( | ) can be used to include multiple variants inside the square brackets. For example, you could specify [arm7|arm9].
• Entries with no variant prefix specified are compiled for all variants.
• Entries can specify an optional trailing list of source files (in the specified directory) to either be omitted from the build or to only be included in the build. The less than character ( < ) is used to specify a build-only list of files. The greater than character ( > ) is used to specify an excluded list of files.

Here are some examples of lines that can be included in a libdirs.b file:

# Build the src/foo directory in my package
src/foo

# Build the src/foo directory but do NOT build the hello.c and world.cpp files
src/foo > hello.c world.cpp

# Build the src/foo directory but ONLY build the hello.c and world.cpp files
src/foobar < hello.c world.cpp

# Include a common libdirs.b file (use relative-to-the-package root syntax)
/top/my_common_libdirs.b

# build the third-party module Uncle under the xsrc/ directory
xsrc/Uncle/src

# Build an using an absolute path that the base path is specified by an environment variable
# where ARDUINO_TOOLS=c:\Progra~2\Ardunio
$ARDUINO_TOOLS$/hardware/arduino/cores/arduino

# Directory specific to the 'cpp1' variant
[cpp11] src/Cpl/System/_cpp11

# Build all directories specified in the following file (relative to my project directory)
../../../libdirs.b

# The set of directories are built for both 'win32' and 'win64' variants
[win32|win64] /top/libdirs/platform_win32_default_for_test_libdirs.b

Build Variants

NQBP supports the concepts of build variants. A build variant is where the same basic set of code is compiled and linked against different targets. For example, in the PIM repository the automated unit test for the Cpl::Dm namespace using the MinGW compiler has three build variants: win32, win64, and cpp11. Here is a description of these variants:

• win32 is a 32-bit application build using the native Win32 API for threading.
• win64 is a 64-bit application build using the native Win32 API for threading.
• cpp11 is a 64-bit application build using the C++11 threading interfaces.

Each build variant can be built independently from the others. That is, if you build variant A, it does not delete the final output files of variant B. NQBP does not consider a debug build a build variant. This means building with debug or without debug enabled will overwrite the previous build derived files.

Object Files and Libraries

When NQBP compiles directories, it places all the object files into a library file for each directory built. The exception is NQBP does not create a library file for the objects in the build directory (i.e., the directory where you run the nqbp.py script). During the link phase, it links your executable image against the object files in the build directory and the individual libraries it created during the compile phase. This has the positive effect of only including the code your application uses from a specific directory instead of including all of the object files for an entire directory into your application. Once again, the exception to this rule is all object files in the build directory are always linked into the application.

For example (from the PIM repo), if your application uses the Cpl::Container::Dictionary class, but does not use any of the other classes from the Cpl::Container namespace, at compile time the NQBP build scripts will compile the entire Cpl/Container directory. However, at link time, your application will only link in the Cpl::Container::Dictionary object code from the library. This is the C/C++ language–defined behavior for linking against libraries.

There is one downside to this approach: if there are no references in your application code to a variable or function in an object file that is placed in a library, then it will not be linked. Oddly enough, there can be required variables and functions that need to be linked that are not explicitly referenced. Here are some example cases:

• C/C++ runtime code—This includes things like the code that executes when the reset interrupt occurs, that is, the microcontroller’s vector table. Your application does not have an explicit function call to any of the entries in the vector table. The vector table is typically placed into RAM at a very specific location by the linker script.
• Self-registered (with a container) C++ modules—For this scenario, there is no calling module that references the self-registered instances directly by their names, only indirectly using a reference from the container. The Catch2 unit tests are an example of this. Each Catch2 test case self-registers with the test runner. At runtime, the test runner walks through its list of tests to execute the individual tests.

NQBP provides a mechanism—.firstobjs and .lastobjs parameters—to explicitly force linking against an arbitrary set of object files in addition to linking against directory libraries. When linking directly against object files, the object files are unconditionally included in the final image. In the PIM repository, see the tests/Storm/Component/_0test/linux/gcc/mytoolchain.py file for an example of this.

Top Level Directory Structure

NQBP separates the build directories from the source code directories. It further separates the build directories into two buckets: unit tests (the tests/ directory) and applications (the projects/ directory). The NQBP build scripts will only work if they are executed under one of these two directories. Whether you have both or only one of these directories is strictly your choice. The following diagram illustrates the directory structure

Build Scripts

Some of the Python scripts that NQBP uses are common across projects, and others are unique to individual projects. The table below describes the primary components that make up a complete build script.

File	Description
<compilerToolchain>.py	This script contains the compiler and linker script commands, options, configurations, etc. that are needed to use a specific compiler to build a specific set of outputs. After a compiler toolchain has been created, it can be reused on an unlimited number of projects. These scripts are located under the nqbp/nqbplib/toolchains directory. If you are using a compiler that NQBP does not currently support, you will need to create a compiler toolchain script. See the nqbp/top/start_here.html file for details on how to do this.
nqbp.py	This script is used to perform the builds. A copy of this script must be placed in each build directory. The content of this script is minimal; it basically calls scripts inside the nqbp/nqbplib directory to perform the actual builds.
mytoolchain.py	This script is used to specify which compiler toolchain to use and to provide project-specific customization of the referenced compiler toolchain. Each build directory is required to have a mytoolchain.py file.
libdirs.b	This file is used to specify which directories to build. Each build directory is required to have a libdirs.b file.
sources.b	This file is optional. When used, this file specifies which .c|.cpp|. asm|.s files in a given directory to build. By default, NQBP builds all .c|.cpp|.asm|.s files found in directories specified by the libdirs.b file or in the build directory itself.

Installing NQBP

1. Install Ninja and update the command path to include the executable’s directory.
2. Install Python 3.x.
3. Get the source code from the nqbp2 repo. It is your choice on where to store retrieved code. A single installation of the NQBP files can be shared across multi-users and/or multiple projects/workspaces on a single box.
4. For a given repository, NQBP requires the following environment variables to be set:
   • NQBP_BIN set to the full path to the root directory where the NQBP package is located.
   • NQBP_PKG_ROOT set to the full path to the package that is actively being worked on. Typically this is root directory of your local repository.
   • NQBP_WORK_ROOT set to the full path of the directory containing one or more packages being developed. In practice this is set to the parent directory of NQBP_PKG_ROOT.
   • NQB_XPKGS_ROOT set to the full path of the root directory containing external or third-party source. Typically this is set to NQBP_PKG_ROOT/xsrc directory. This environment variable can be omitted if there is no third-party source code.

Brief History of NQBP

First we need to start with Benv. Benv is build engine that was developed to ‘free’ developers from authoring makefiles and provide a simple and resusable method for building C/C++ source code. Benv was originally develop for *NIX platforms. A fork of the project ported Benv to run on Windows. A principle issue with running Benv on Windows is that because of Benv’s *NIX roots, the Windows port relies on the Cygwin.dll. Cygwin is great alternative for running in a Linux like environment on a Windows box – but it does incur a performance hit.

Some time later I was working on a personal project and needed a build environment to compile example programs under Windows. Benv was my first choice, but the overhead of installing it and slow performance on Windows made it not feasible for the project. However, creating a very light weight, “Benv Like” Windows only solution was something that fit the time frame/scope of the project. Thus the idea of Not-Quite-Benv was born, i.e. a build environment that provides the same core concepts and co-exist with Benv – but without the performance penalty. The end was product was a Windows/DOS batch based build engine that consumed the same core input files (lists of which source files to build, etc.) as Benv that had extremely fast build times. The faster build times where accomplished by using Windows native batch files and by omitting any/all dependency checking, i.e. NQB built all files (almost) all the time. For my project, building all the files all the time was a non-issue since the programs were small. In addition, I found that for several embedded open source projects I worked on, NQB could build all files, faster than running Benv under Windows with full dependency checking and only building the set of changed files.

The NQB concept worked well – especially for embedded development or smallish projects – but it was a Windows only solution. So I decided that some day I would develop a multi-host solution for the Not-Quite-Benv concept. Thus was born NQBP.

Over time microcontrollers only got bigger and better and contained more flash and RAM. This in turn has led to larger code bases for embedded projects. More code, meant the NQBP build-all model was less and less practical. Over the years, numerous command line options where added to NQBP to allow a developer to do manual incremental builds as a stop gap. Updating NQBP to support dependency checking and incremental build had been on the TODO list for quite some time – and now the long wait is over ;-).

For those who are overly curious the ‘Not-Quite’ paradigm was inspired by Not-Quite-C. I’ll leave the details/origins of NQC to the readers.