OpenVDB AX

AX C++ Documentation

The following documentation focuses on the C++ design and API of OpenVDB AX, not the front end AX language. For the latter, see the AX Language documentation. For more specific function API information, search for the relevant function doxygen.

Note

These documents are actively being worked on and are subject to change. See the API doxygen comments or contact us for more information.

Contents

• Versioning, API and ABI
• OpenVDB AX Repository
  • The Grammar
  • The AST
    • Scanners
  • The Compiler Pipeline
    • Logging
    • Executables
    • Custom Data
    • Codegen
    • The Compiler
  • The Command Line Binary
• OpenVDB / OpenVDB AX Types
• Extending OpenVDB AX

---

Versioning, API and ABI

AX is a self contained library which currently has a one way dependency on OpenVDB; any downstream artifact that uses AX will need to build and link against both OpenVDB and OpenVDB AX. There are a few important factors to be aware of with this integration:

• OpenVDB AX is tied to the OpenVDB library version. There is no explicit OpenVDB AX version. This includes the OpenVDB namespaced API, the shared library version suffix on relevant platforms and the OpenVDB ABI. It may be possible to build OpenVDB AX against a different version of OpenVDB however this is not explicitly supported.
• There are no ABI compatibility guarantees for OpenVDB AX components i.e. AX components do not support the transfer of themselves as binary representations across dynamic library boundaries to applications where AX has been compiled at a different location in the OpenVDB repository. Support for this may change in the future.

In general, anything in the public namespace (defined as anything not labelled "internal") comes with standard API guarantees; that is, classes and methods will not change visible behaviour or signatures and deprecations will occur with appropriate replacements (where necessary) prior to removal. However, due to the nature of the project and its infancy, it's encouraged to continue reading below into the relevant AX components. These provide a better description of their intended public consumption and completeness.

---

OpenVDB AX Repository

The AX repository has been laid out using a modular design. Each subdirectory encompasses a set of similar tools with a fairly consistent (one-way) dependency diagram to other tools in the repository. The following provides a short description of each directory, listed in this ad-hoc dependency order (from highest to lowest):

• grammar: Provides the flex/bison lexer/grammar rules for generating the OpenVDB AX language, as well as the pre-generated .c files.
• ast: The definition of an AX Abstract Syntax Tree, its node types and methods to work explicitly with them.
• compiler: The frontend C++ user interface for building and using AX.
• codegen: The backend LLVM IR code generation for AX.

Additional tools include (in no particular order):

• cmd: Command line binary tools for AX.
• math: Various mathematical methods, typically used by the AX code generators when building native functions.
• test: Unit testing framework for AX.

The Grammar

The subdirectory grammar contains the rules used to generate an AX abstract syntax tree, as defined by the AX Language specification. These files do not provide an API per-se and are primarily used internally by the ast module. The grammar syntax rules are specified in files with .y suffix and the tokenziers are specified by the .l suffix. These files are designed to work with two UNIX tools, GNU Bison and Flex. Together these tools provide a workflow for developers to create develop a wide range of language parsers. It's encouraged to visit the corresponding manuals for more information on their use.

The AX grammar is defined with a LR parser. In simplest terms, this means that the syntactical rules define a grammar which is parsed bottom-up, left-to-right, deterministically and context free (for the most part). These are fairly typical traits of parsers used to define computer languages. The parser is not designed to be interactive, nor reentrant (is not thread safe, each string is parsed sequentially) however such limitations are not imposed by AX and may be removed in the future. Note that the internal function symbols are correctly prefixed (with ax), so AX should be compatible with other software that also uses Bison/Flex.

The .y and .l files are typically not part of the normal build process. Bison and Flex use them to generate compatible C/C++ code, however this code is always pre-generated and provided with the AX repository within the grammar/generated subdirectory. This C/C++ code is compiled into the AX library. Thus, the grammar is provided for two reasons; to demonstrate how the syntactical rules are implemented and to provide expert users the ability to regenerate the C/C++ bindings (see the build instructions). Importantly, AX does not install any of the files in the grammar folder; the parser is accessed through the ast module.

The AST

The ast module provides:

• The complete implemenation of an AX abstract syntax tree
• The openvdb::ax::ast::parse() method
• The AST Visitor
• AST scanners and tools

All methods in this module are designed to work around an AX AST, which represents the syntactical constructs of the AX language as a hierarchy of C++ objects (nodes). The complete AST hierarchy is visualized by the doxygen class graph for an openvdb::ax::ast::Node, which represents the top most node in the AST. The AST API exists in a nested namespace (openvdb::ax::ast) and can be used by clients for more granular control and modifications to an AST. Importantly, the AST API contains the openvdb::ax::ast::parse() method, which invokes the C/C++ functions built by the grammar to iteratively construct an AST from a provided character string.

A number of other tools are provided by the module to work with an AST such as tools to re-interpret the AST back to AX compatible code, printing of the AST's node hierarchy/layout and scanners which are able to query details from AST branches. All these methods use the AX visitor framework, a visitor pattern traversal class strongly influenced by Clang's RecursiveVisitor. Clients can use the AX visitor to implement their own AST analysis or modifications (see Extending OpenVDB AX).

Scanners

todo

The Compiler Pipeline

The compiler subdirectory contains the bulk of the C++ API intended to be used directly by clients of OpenVDB AX. It links together the grammar, AST and code generation to produce AX executable objects, forming a pipeline from an initial string of AX code to final execution across some geometry. The following sections detail the core components to the Compiler pipeline.

Logging

Anyone familiar with programming languages will undoubtedly be aware of the huge variety and quantity of feedback that's presented to clients during use. This information, when properly presented, can be invaluable and help to advise users during their usage of the language. AX provides a Logger class which can be used throughout the Compiler pipeline to store and report this information. By default, this class reports all errors to std::cerr and swallows warnings, but can be customised depending on your needs e.g:

#include <openvdb_ax/compiler/Logger.h>

// Create a logger which sends errors and warnings to std::cerr
openvdb::ax::Logger
    logs([](const std::string& msg) { std::cerr << msg << std::endl; },
         [](const std::string& msg) { std::cerr << msg << std::endl; });

logs.setMaxErrors(5); // stop reporting after 5 errors
logs.setWarningsAsErrors(false); // don't count warnings as an error
logs.setPrintLines(true); // print code lines
logs.setNumberedOutput(true); // number each error/warning reported

A Logger should be passed to the relevant pipeline methods as detailed in the following sections.

Executables

The "executable" terminology comes from the binary representation of the compiled function which is invoked. AX executables essentially wrap the generated function calls from the AX codegen with an efficient multi-threaded invoke for the type of geometry being processed. The executable classes are designed to be lightweight to store, modify and copy with a consolidated interface for running and customising execution.

There are two main types of executables; the VolumeExecutable, designed to work with all OpenVDB Volumes and the PointExecutable which works with OpenVDB PointDataGrids. They are not expected to be created directly; instead these objects are returned by the AX Compiler depending on the selected Compiler::compile() function.

Note

There is no inherent limitation to the design of the VolumeExecutable which stops it working on PointDataGrids but it's not particular useful in it's current form and can be confusing so it is explicitly disallowed.

The executables are distinctly detached from the Compiler once they have been built and can be safely used no matter previous or subsequent compilations of AX code. There are a variety of settings on the executables to control runtime behaviour - below demonstrates an example of using this API:

using namespace openvdb;

ax::Compiler compiler;  // compiler

FloatGrid a;
a.setName("a");
a.tree().setValueOn({0,0,0}, 1.0f); // set single coordinate to active 1.0f

const ax::VolumeExecutable::Ptr exe1 =
    compiler.compile<ax::VolumeExecutable>("f@a += 1.0f;");
exe1->execute(a); // run over active leaf voxels in parallel

ax::VolumeExecutable::Ptr exe2 =
    compiler.compile<ax::VolumeExecutable>("f@a += 2.0f;");

// set to process all voxels
exe2->setValueIterator(ax::VolumeExecutable::IterType::ALL);
exe2->execute(a); // run over active and inactive leaf voxels in parallel

std::cout << a.tree().getValue({0,0,0}) << std::endl; // prints 4.0f
std::cout << a.tree().getValue({1,0,0}) << std::endl; // prints 2.0f

Executable Thread Safety

The executables are responsible for launching the compiled AX kernels and may internally use multiple threads. Multiple executables of any type can co-exist and be executed concurrently with unique arguments. For example:

FloatGrid a1, a2; // assume both are named "a"
auto exe1 = compiler.compile<ax::VolumeExecutable>("f@a += 1.0f;");
auto exe2 = compiler.compile<ax::VolumeExecutable>("f@a += 2.0f;");

std::thread t1([&]() { exe1->execute(a1); } );
std::thread t2([&]() {
    exe1->execute(a2); // safe even if exe1::execute is being used by another thread
    exe2->execute(a2); // safe, can be running alongside another executable
});

This is, however, only safe depending on the access pattern of the AX code, the iteration patterns of the executables and the grid data being fed to the execute methods. When we talk about the thread safety of these classes we are referring to the invocation of their execution method with the same argument data by multiple threads, not how many threads the executables themselves use (which can be configured by the grainSize() setting). Importantly, for a given VDB grid or VDB Points attribute "foo":

• For The VolumeExecutable it is safe to call VolumeExecutable::execute from multiple threads only if:
  • All relevant AX kernels do not write to grid foo.
• For The PointExecutable it is safe to call PointExecutable::execute from multiple threads only if:
  • All relevant AX kernels do not write to attribute foo.
  • All relevant AX kernels do not create any attributes or groups.
  • All relevant AX kernels do not modify the position "P" attribute.

For example:

auto exe = compiler.compile<ax::VolumeExecutable>("@a = @b");

// Grid a1 and a2 called "a", b called "b". Grid b is given to both threads.
// This is safe as we guarantee b is only read from
std::thread t1([&]() { GridPtrVec v{a1,b}; exe1->execute(v); } );
std::thread t2([&]() { GridPtrVec v{a2,b}; exe1->execute(v); } );

You can use the tools provided in Scanners.h to query data accesses on AX ASTs to verify access patterns of your data.

Custom Data

The AX language provides two main tokens to access memory which has not been allocated by AX itself. The first token, @ (the "at" symbol), is designed to provide read/write access to geometry attributes. The second token is the dollar character $. This allows read only access to custom data that can be modified in C++ i.e. outside the scope of an AX program. In the same way as the @attribute syntax, the value of this data can change without the need for AX to re-compile the program. It is, however, solely on the C++ clients of AX to make sure that this data is setup correctly when integrating AX into their applications.

Consider this trivial example:

f@density = f$my_value;
// f$my_value = f@density; NOTE: writing to $ values is disallowed

When compiled, this program will attempt to read a single float type value with the storage name "my_value". If the value cannot be found, a zero intialized block of memory (with size of the desired type) is returned. So, with no further setup, this will assign each value of density to 0.0f.

The CustomData class provides the intermediary storage for this data between the code generation and the compiler. Clients should use this API to create and modify custom data which will become available to AX programs. Although the data is stored as abstract openvdb::Metadata objects, the values are embedded into AX programs as pointers to their location in memory. In practice this means that accessing data stored in this way is almost as fast as accessing an AX local variable, though certain optimizations, such as constant folding, will not apply. As previously mentioned, this also means that the AX program does not need to be recompiled should this value need to be changed.

The CustomData starts off detached from an executable and thus must be provided when invoking Compiler::compile(). It is then tied to the AX executable for its duration. The compiler will validate the types of all accessed data of an AX program and, if it does not exist, the data will be created. Note that multiple executables can access the same CustomData instance.

using namespace openvdb;

FloatGrid density;
density.setName("density");

ax::Compiler compiler;  // compiler
ax::CustomData::Ptr data(new ax::CustomData);  // empty custom data

// data provided to compile is now registered with this exe. "m_value" is created
ax::VolumeExecutable::Ptr exe =
    compiler.compile<ax::VolumeExecutable>("f@density = f$my_value;", data);
exe->execute(density); // my_value doesn't exist, density set to 0.0f

// Compiler::compile will have created this
assert(data->hasValue<TypedMetadata<float>>("my_value"));

// get the data in its typed storage wrapper
TypedMetadata<float>* meta = data->getValue<TypedMetadata<float>>("my_value");
meta->set(1.0f);  // set the value of "my_value" to 1.0f

// don't need to re-compile
exe->execute(density); // density set to 1.0f

Warning

Care should be taken to ensure that modifications to custom data values are peformed outside of any calls to execute(). Any attempt to modify the data whilst executables attempt to access it will result in undefined behaviour.

Codegen

todo

The Compiler

The Compiler brings together the above components. It is responsible setting up LLVM contexts and modules for a given AX program, invoking the compute generators, binding custom data, performing any necessary AX and LLVM optimisations and finally JIT compiling the program to a callable function. The Compiler::compile() member functions execute the aforementioned steps and, if successful, return a pointer to an executable.

There are two main ways to invoke compilation; one with an AX Logger and one without. The prior will either return a valid pointer to a generated executable on success or a nullptr on failure. It is guaranteed to never throw a code generation runtime error and expects the caller to either query or setup the state of the logger to handle warnings and errors. The latter internally creates a default logger which swallows warnings and prints all possible errors to std::cerr. Note that the latter will throw a runtime error on failure and is thus guaranteed to never return a nullptr.

using namespace openvdb;

ax::Compiler compiler;
std::vector<std::string> warn, err;
// custom logger to collect warnings and errors
ax::Logger logger(
    [&warn]() { warn.emplace_back(msg); },
    [&err]()  { err.emplace_back(msg);  }
);

// logger provided, exe could be null if input is invalid
auto exe = compiler.compile<ax::VolumeExecutable>("f@a = 1.0f;", logger);
// print warnings
for (cosnt auto& msg : warn) std::cout << msg << std::endl;
if (!exe) {
    for (cosnt auto& msg : err) std::cout << msg << std::endl;
}

// without logger, divison by zero warning won't be reported
exe = compiler.compile<ax::VolumeExecutable>("f@a = 1.0f / 0.0f;");

// without logger, invalid AX will cause a runtime error but
// also print errors to std::cerr
try {
    exe = compiler.compile<ax::VolumeExecutable>("foo");
}
catch (...) {
    std::cerr << "See above error logs..." << std::endl;
}

Warning

A single instance of the AX Compiler uses a uniquely constructed LLVMContext. This context is shared between copies of the Compiler, however new Compilers create new LLVMContexts. It is therefor unsafe to invoke concurrent calls to Compiler::compile() on Compilers which share the same underlying LLVMContext.

ax::Compiler C1, C2;

// @warning Not safe!
// std::thread t1([&]() { C1.compile<ax::VolumeExecutable>("@a;") } );
// std::thread t2([&]() { C1.compile<ax::VolumeExecutable>("@b;") } );

// the following is safe as C1 and C2 are unique instances
std::thread t1([&]() { C1.compile<ax::VolumeExecutable>("@a;") } );
std::thread t2([&]() { C2.compile<ax::VolumeExecutable>("@a;") } );

The Command Line Binary

todo

---

OpenVDB / OpenVDB AX Types

Whilst OpenVDB AX is primarily designed for OpenVDB, it has its own notion of types. This is so that the actual AX language is able to support arithmetic that OpenVDB geometry would perhaps otherwise not support. For example, AX supports 3x3 and 4x4 matrix types. These types are akin to the openvdb::math::Mat types that exist in OpenVDB library - however, historically, these types are not "registered Grid types" by default by OpenVDB. This means that, although you could theoretically create an openvdb::Mat3dGrid OpenVDB may not support the reading or writing of these grid types out the box. Whilst this may change in the future, there may exist other examples of types in AX which suffer from the same limitation of existing geometry types in OpenVDB.

Native OpenVDB types are types which are a) registered for serialization and b) compilable by the default C++ installation of OpenVDB. This is a bit convoluted, so what does this actually mean in practice? Generally, accessing an attribute in AX requires that data to exist on the underlying geometry. As such, reading or writing to a given attribute (for example bool@myattr) requires both the AX compiler and the underlying geometry to support the given type. As mentioned above, there may be times where this isn't the case. Consider the following:

vec4i a = {1,2,3,4};
vec4i@attr = a;

Now lets assume that a notion of four int32 values is not supported for serialization of OpenVDB points or volumes (we assume the type can at least be instantiated). Whilst this code may execute, serializing this data to disk may fail due to unregistered OpenVDB types. AX does not register any additional OpenVDB types. This choice is left to downstream software. In this example, to allow for serialization of vec4 types, some variant of the following can be implemented in your application.

Warning

The choice to register Grid types in OpenVDB is left to the OpenVDB maintainers. There are important consequences to registering custom types - other default installations of OpenVDB will not be able to read your files unless compiled with similar instantiations.

openvdb::initialize(); // standard init of vdb
openvdb::ax::initialize(); // standard init of ax
// Define a Vec4I grid type from the openvdb::math::Vec4<int32_t> class.
// Use a ValueConverter to ensure the grid has the same configuration as
// a standard OpenVDB grid type.
using Vec4IGrid = openvdb::BoolGrid::ValueConverter<openvdb::Vec4I>::Type;
// register this type
Vec4IGrid::registerGrid();

---

Extending OpenVDB AX