Automated Build System

Have you ever thought that too much time is spent on setting up projects for building programs?

Why do we have to manually set up include paths? Why do we have to manually set up libraries for linking and define which libraries use which other libraries? What is the minimal amount of information required to build a program?

The OovAide program must be easy to set up to analyze programs that may be built using many different build systems. For this reason it is important that it is easy to set up.

In order to allow the programmer to set up us little as possible, modern systems should find this information automatically, and update the information if any files have changed. There are some problems doing this for all programs, but for many programs, the constraints and tasks are not that difficult.

We do not need yet another build system, but maybe some of these features will someday be incorporated into more intelligent build systems.

C++ Automated Build

Include Paths

Some of the difficulties with automatically finding include paths are described below.

C++ code is difficult to parse. This problem is pretty much solved by using a modern parser such as the CLang parser, and building some code around them to search for include paths.
C++ macros can alter the files that are included. Two different include files can include a single include files with a different #define setting so that the included file has different include dependency results. Many projects do not have this problem at least for the header files that are defined within the project. The future C++ module feature may be required to solve the macro problem for difficult projects.
In C++, an include path can be something like "<gtk/gtk.h>", which means that the include directory flag given to the compiler has to specify a directory one level higher in the directory tree.
Many include files within a project could have the same name.
Most projects rely on external project include paths, such as C++ standard libraries, or some other framework. At the moment these cannot be found automatically, at least not on all operating systems.

Setting up Build Tools

It would also be nice if the build system automatically found the build tools. For example, it may search for g++ or for clang++. On Windows or Linux, it may search the "Path" environment variable, or on may search some standard directories. In addition, some compiler options may be setup by default for debug and release versions.

Specifying Source Code Grouping

For simple projects, it may be that all source code under a directory is used to build a single executable. For other projects, there can be many components of different types, such as static library, dynamic library or executable. Each dynamic library or executable can use any static library. At the moment, there is no way to automatically determine which source code in different directories are used to build a component. This must require some additional information from the programmer, but usually there should be no need to list all source or header files, or to specify the dependencies.

Libraries

In some operating systems, the linker may require the libraries to be listed in order. This is not a big problem and is easily solved with available tools.

External Packages

In addition to external project include files, there are typically also external libraries required by projects. These projects may also require additional compile or link time switches. Under Linux, there can be quite a bit of information available, but under Windows, the task is a bit more difficult.

OovAide Build System

The original purpose of OovAide is to be able to investigate projects where there is minimal knowledge of the project. There may be build files present, but there are so many build systems that it is impossible to handle them all. OovAide at some point may be able to read a few, at least for the purpose of analysis, but it does not currently.

An attempt was made to build an easy to use build system in the OovAide program. This solution may not work for very large or complicated projects, but does work on projects that have multiple include paths, external packages, and builds multiple binary components.

The following sections are defined roughly in the order that a programmer would go through to set up a project for building using the OovAide program. Performing only analysis using OovAide only requires a subset of the steps defined here.

OovAide allows specifying custom compile switches for the project. OovAide actually runs a command line build program called oovBuilder, which allows performing builds from the command line. OovAide also runs many of the build tasks in parallel by multitasking the processes.

Specify Project Code Location

The programmer must select a single source root directory for all of the source code that will be in the project. It is possible to specify exclusion directories within the source root directory to specify directories that should not be analyzed or built.

Setting Up Build Tools

The OovAide program searches for the g++ and clang++ tools. It does this differently on Windows and Linux. It also requires the ar and nm tools for finding library dependencies.

On Linux, the system include paths must be found. OovAide runs the compiler to query the built-in include paths. On Windows, the include files are supplied with OovAide, and do not need to be found using the compiler.

Setting up Build Configurations

OovAide defines the standard Debug and Release configurations and sets up some default compiler switches for each configuration. The goal here is that external packages only need to be set once, and every configuration will use the common arguments. OovAide also supports adding more configurations for things such as cross compiling, but at the moment, a separate project may be required if the configurations are too different.

Typically the external packages must be defined differently on Linux or Windows because the package systems are not the same on all systems.

There is much room for improvement here.

External Packages

The advantage of having predefined external packages is that the programmer can select a single name such as MySQL or wxWidgets, and the appropriate compiler switches (with include paths), link switches, and link libraries can be used by a project. The OovAide program reads the package details and sends the appropriate information to the compiler or linker.

On Linux, the pkgconfig program provides definitions for many popular packages. The pkgconfig program is on many Linux systems, and defines the compile and link information needed to use the package. The OovAide GUI provides a simple way to select external packages to add to the program.

On Windows, a similar interface is provided for selecting libraries, but there is no common database that provides definitions of external libraries. A few library definitions are provided by OovAide, but others can be added by the programmer. This is not a good long term solution, but there is not a common package manager on all systems yet.

Directory Scan

The OovAide program scans the directories under the source root directory to find all of the source and header files.

The OovAide project assumes that an external editor may have been used to edit the files, and performs a directory scan on the whole source root directory tree in order to find modified files. This could be optimized in the future by performing directory monitoring, since scanning all directories may limit the use for very large projects.

Specifying Source Code Grouping

The following directory tree will be used to discuss some common scenarios.

An important point to remember is that the OovAide grouping mechanism is used for grouping source files into components. OovAide will set include paths for header files no matter where they are as long as the file names are unique.

OovAide uses a simple method for defining components. Any directory in the source root directory can be designated as a component.

Some of the grouping rules:

A component at any level of the tree will be built using all source files in all subdirectories that are not defined as a component. In this diagram, the top nodes ("Dir Root", "Dir A", and "Dir B-2") are defined as components.
The groupings do not determine where header files can be found. They only determine which source files are compiled into the component. Header files can be anywhere and will be automatically found during analysis.
Component "Dir A" could be a library, which will be linked into other components automatically.

Example 1

In the first example, "Dir Root" is defined as an executable component. Include files could be present in any directory, and these directories will be supplied to the compiler in order to build the object files from the source files. All object files will be linked into the executable.

Example 2

This example starts from the first example, and defines "Dir A" as a static library component. This means that the programmer only has to select one item in the directory tree as a component, and OovAide will build a static library and link it into the executable without specifying and source paths, etc. OovAide provides the StaticLib example that is similar to this. Examples

Example 3

This example starts from the first example, and defines "Dir A" as a run-time library component. This is not as nice as example 2, since the source code must be modified to define an interface and use the interface for the run-time bindings.

Example 4

A good example of a slightly more complex system is the OovAide project. It contains multiple executables (OovAide, OovBuilder, ClangView, OovCMaker, OovCovInstr, OovCppParser), multiple static libraries (OovCommon, and OovGuiCommon), a Java program (OovJavaParser), and a run-time library (OovDbWriter). There are include files in all of these component directories, and they are all properly built.

Analysis

The CLang parser is run on all C++ files to generate include dependencies. The include dependencies are used to build object files, and to determine when they must be rebuilt when a source file changes. The include dependencies are also used to determine which external packages are used by each component.

A CRC is used on the command line switches to determine when the analysis and include dependencies must be updated. A separate directory is used for each CRC so that switching between settings is faster.

While parsing the C++ files, include flags are set for all directories in the project that contain header files so that the CLang compiler can function correctly. In addition, OOvAide specifies the parent and grandparent directories so that relative paths such as <gtk/gtk.h> will compile. The number of include directories may be a problem on some versions of Windows for larger projects since the command line length may be limited to 32767 characters.

It would be nice if CLang could request include flags dynamically during parsing to reduce possible problems that too many include directories could create.

During the actual compile, only the include files that are required are specified as directories for faster compilation.

At the moment, if the project is changed in a major manner, the conservative approach is taken, and all files are rebuilt.

Build

The build performs the directory scan to see what which files were modified. The file extensions are used to determine which tool to run on each file. The nm tool is used to find the names in internal and external project libraries to determine library dependencies and link order. Only files that are out of date are built.

The include dependency directories are used to determine which external package libraries are needed to build a component that is linked. OovAide does have constraints that the header file have to be unique in some cases in order to prevent linking the wrong library or finding the wrong include during a compile. This is typically not a problem since most external packages have few files that are included directly in a project.

At the moment, all project libraries are included in the link. This does not seem to cost much time since the unused libraries will be last during the link. This rule may be changed in the future.

More build details can be seen at OovAide Build Help or OovAide Build Design

Some Future Tasks

For a broader vision, see the Completely Redesigned Build below. Optimize the use of the CLang compiler. The CLang compiler is invoked multiple times.

The editor uses the compiler in order to perform syntax highlighting and to find symbols for code completion. This is fairly transparent to the user since it is performed on a background thread.
The compiler is used to generate include dependencies and analysis information.
The source code is compiled into object files.

Running the compiler multiple times could be a bit of a speed problem, but it is alleviated since OovAide performs multitasking for the build processes. It doesn't appear that CLang supports analyzing and building at the same time using the C-Index interface to the compiler. In the future, the analysis information could be generated in the editor after it is detected that the editor has has compiled some source code cleanly. Performing the compilation automatically could also be performed, but makes more sense when a source file is being compiled rather than a header file. There is definitely room for improvement here.

The other major problem that could be partially solved is include files that have the same name. The source code grouping could be used to first find include files within the current component. This still may not be enough for some projects though.

Java Automated Build

OovAide can also build very simple Java programs (generally a single jar) that are in the same directories along with C++ files.

Directory Scan

While the scan for C++ files is run under the source root directory, the java files are also found.

Specifying Source Code Grouping

The same directory setup is used as the C++ grouping. The component types that are used for java are "Library Jar" or "Executable Jar". The difference is that a Manifest.txt file is required in any directory that is defined as an "Executable Jar". The manifest must have the Main-Class attribute set. See OovJavaParser directory as an example.

Setting up Build Configurations

The Java setup allows setting up the class path to allow references to external jars, and other arguments.

Analysis and Build

At the moment, the java parser is used to find the include dependencies. This may mean that the files must compile successfully in order for full analysis to be performed. The jar files are not built in any order at this time, so if one is dependent on another, sorry. A workaround is to define multiple OovAide projects, and to define exclude directories if needed.

The correct way to fix this would be to have a pass that finds include dependencies before performing analysis or builds. I am not sure if the Java parser supports this, but it appears that it might. In addition, the jar files have to be built in order which presents some interesting problems to solve in order to support multitasking.

Completely Redesigned Build

The following are ideas for a completely redesigned build after presenting this article on Reddit. Many people think that the current build systems (make or cmake) are fine. My opinion is that the current build systems have the following problems.

There is too much manual setup
They could be much faster
Conditional logic should be reduced or removed for simplicity

A couple of unique features to OovAide is that the build should be extremely easy to set up so that analysis can be performed on unknown projects or projects with strange build systems. Another point is that analysis can be performed even when a successful build is not possible.

There are multiple phases to a more automated build. These phases are not listed in exact order since some phases are iterative. For example, some types of directory grouping may be done at any time.

Build Setup
- Setting up tool information
- Download externally required packages
Editor creates or modifies source files (Requires include paths)
Component identification/directory grouping
Build
- Either the build scans for modified files, or performs directory monitoring, or is told by the editor which files have changed.
- Find include dependencies (using something like -M flags)
- Build files/components by running final commands (executables require running nm on libraries for determining link order)
Scanning for new files and finding dependencies does not need to be done as often as the final build phase.

Implementation Goals.

Try to default in standard ways first, then allow overriding by user. For example, default to including all include paths in project, then if duplicate include files, allow user to specify order. Default to building all files in recursive directories. This can reduce file size and complexity.
Reduce the required number of build files. This reduces the complexity for projects (especially for small projects), and increases build speed.
Reduce directory scanning (like -M) so that they do not need to be done for every build.
Allow building debug/release, cross compile, platform, etc.
Multiple targets? Allow building things like doxygen, C assembly, etc?
At the moment, this build does not support building arbitrary files using rules like ninja.

Some implementation details.

Implementation solution for overriding defaults. For include directories and build tool arguments, this is done using indexed filtered variables
When do build files need to be loaded and how many need to be loaded? (build files per component) Location of build file has no meaning. Build files are all combined into a single build mass. Separated build files may be useful if variables can be inherited/extended from parent build files. This may not be useful until variables can be used from other variables? There should not need to be a build file in each component, but there can be to provide sub-projects that can be included into various parent projects. Parent build files define settings for child build files. Add and subtract from previous settings. Conditional for linux/windows Parent build "import" child builds.
Can the build files stay loaded and be reexecuted for each source file update? Saves scanning for new files.
Build files required to determine what to build should be small, then other files can be loaded to build certain parts?
Dependency generation must be performed before build phase for Java?
Analysis, docs, etc. output should be generated after build output? Include dependency analysis should be done by editor. Include dependency info should be saved by directory? Derived from C++ modules? C++ modules created automatically if specified? The editor can indicate to the build system whether the include dependencies need to be updated, and implicitly when a file has been created or changed.
Allow specifying include paths that must be included first. Allow user to choose from multiple duplicates? How to specify order?
Library dependency ordering - when/how does it happen, what are rules?
All stored paths that are within a project should be relative.
Should allow external projects to be on different drives. (Windows)
Still two module proposals: Microsoft open-std.org, CLang module.modulemap, std.vector. Neither module proposal specifies location of external dependencies? Improved import may handle dup names?
Does CMake get include directories from library specifications?
Must support debug/release output.
The new C++ module system should be used. This is an extra file in the source directories that can specify include dependencies, etc. This looks like it can be generated by compilers, but it is too late in the build phases.
Components / grouping should allow specifying wildcard source lists
To solve the duplicate header file problem, rely on C++ modules? Worst case manual override?
Should skip builds if input requirements (packages) do not exist
Ninja ideas:
- Ninja is a low level build program (the last build phase). It does not find packages like CMake. It does support building packages?
- Ninja is good for complete builds. It seems like it could be faster if it was told which parts of a program to build. I am pretty sure it must load all ninja files to do a build. This http://www.aosabook.org/en/posa/ninja.html says that there are 10 MB of ninja files to do a chrome build. This https://ninja-build.org/manual.html says it takes less than a second to start a build. It could use default mechanisms to reduce file size. Such as defaulting to *.cpp (more implicit rather than explicit) for a build rule. It seems like it could be faster also if it did not use the -M flags and could have two stages for scan/build.
- Fixed variables can be defined. These are basically literal strings that can define command names, and command arguments. Variables inherit from parent file. There are no conditionals for platform, debug/release, etc.
- Targets allow specifying which build outputs to build.
- "rules" provide common information about building a file type. Rules support variable substitution and extracting parts of the output filename and input filenames. $in has an extension but $out does not?
- "build" applies the rules to a specific output file.
- I view ninja type files as intermediate files. So they should not be stored in source directories. Dependency info like C++ modules can be stored in source directories.
Bug fix: Support multiple drives?

Evolve OovAide Build

The main reason that the project files must be updated is to support different configurations for different components. The main feature is to support different include paths for any component.

Variables

A base variable can be defined as having a value, then it can either be reassigned or appended to for certain build states.

Variable as stored in a project file:

Entity	Description	Example
Var name	An identifier	CppArgs
Filter	[filtername : filtervalue & filtername : filtervalue]	[comp:oovaide & plat:Windows]
Function	Set(=), insert(^), or append(+) to a variable	+
Value separator	\|	\|
Value	any string or compound values	-lnk-Wl,--subsystem,windows;

The following are filters that are prebuilt into the program.

Filter Name	Possible Values
mode	Analyze, Build
plat	Linux, Windows
cfg	Analysis, Debug, Release, [custom]
comp	Any component name within the project

The comp filter is not yet supported. Another that may be added is filetype (cpp,c,asm,obj,lib/java). Also coverage does not have filters yet.

The following is an example configuration file.

	CppArgs[]=|-c;-x;c++;-std=c++11;
	CppArgs[cfg:Debug]+|-O0;-g3;

The previous configuration would result in the following. The arguments are shown as separated with semicolons. This shows that release mode is using the base variable arguments, and the Debug mode has extra arguments appended.

Mode	Flags
Debug	-c;-x;c++;-std=c++11;-O0;-g3;
Release	-c;-x;c++;-std=c++11;

Future

Some of the following may be already implmented in the latest versions of OovAide.

Dependencies - Add dependency dialog per component/dir. Clear and find, if manually set up, should builder ever scan? In future, could scan from within selected directories. Add external and internal project dependency scanning. First only add internal dependencies? Add Comp-intDeps[] to oovaide-comptypes.txt for internal dependencies? If empty, use scan. Scan or CMake results go to Comp-scannedIntDeps or pulled out of incdeps? Allow different file types in same directory? Dependencies different? OovBuilder can scan or use CMake files to set up manual dependencies. GUI reads incdeps to assist updating manual dependencies? NEW: ComponentFinder::getComponentIncludeDirs()
Clean - Should be per component/dir. Add dependency/scan cleaning?
Build CRC - Fix to handle changed external projects per component? CRC per component? Other option is that external packages must be set, and components can select from external packages.
Projects - Add import for child components. Import only defines a way to optionally separate definitions so that multiple parents can import a child file. Info from oovaide-comptypes.txt file gets moved into oovaide.txt. oovaide-tmp-compsources.txt changes
Change build configurations to allow Windows/Linux settings - causes change in External Packages for Windows/Linux Analysis
Types of tools: convert=compile,analyze,coverage combine=lib(.obj),link(.lib) javac - For each component *.java converts to *.class link - combines lib and obj files java dups flag is handled strangely, analysis requires ordering by component jar dependencies
Rules - JavaAnalysis $javaToolPath $javaArgs -cp $tools.jar $javaAnalyzerTool $in $srcRootDir $analDir $args
Option - OovAide needs option for placement of dependency information in source or build dirs?
Option - Need option for when to rescan (when editor changes file). Allow manual rescan or clean. Clean by component.
Add option to specify relative include path levels? #include
Read CMake files to create build files. add_subdirectory
IncDeps - oovaide-incdeps.txt is specific for each component. There are two types of incdeps - compiler generated (files), and user specified (paths, previously oovaide-extdirs.txt)
Incdeps - Change incdeps file and other build files to use internal directory variables to reduce filepath duplication to reduce file sizes
oovaide-extdirs.txt (initial include paths) is specific for each component or defined as variable that builds inherit
Java - external packages should be specified by component, sets classpath