SWIG Monologues : Part 1 Getting Started With CMake

As no one besides me is ever going to end-up here, I’ll use this space to distill my hard earned knowledge for my own use in later projects.

Jumping straight to SWIG, with a focus on writing Python bindings.

I’m a Visual Studio developer, so I need a solution file to achieve something. Instead of wasting time writing the SWIG rules, I use CMake. I have no love for CMake, but when it works it really takes care of the dirty work.

If you don’t care about understanding how CMake creates the solution, you can find a working CMakeList.txt file here and stop reading.

To make your own CMakeList.txt, start by putting these lines:

find_package(SWIG REQUIRED)
include(${SWIG_USE_FILE})

They will define CMake functions we need later to create SWIG module targets.

As we will write Python bindings, we need a way to find the Python libraries. Since I’m a virtual environment user, I don’t want CMake to go use the C:\Python27 or C:\Python35 paths that the default installer use. When working in a team, you’ll want to use virtualenv. Therefore, I launch cmake from a virtualenv command line. Then the following CMake script will find all the required path to link against the right Python libraries.

find_package(PythonInterp REQUIRED)
message (STATUS "Using this Python interpreter: " ${PYTHON_EXECUTABLE})

find_package(PythonLibs REQUIRED)
include_directories(${PYTHON_INCLUDE_PATH})

Most of the time, I will need to interface with numpy arrays. So the bindings must link with the numpy array library. These line will launch the current python interpreter to get its library folder (i.e. site-packages).

# must replace the backslash by forward slash otherwise there's an error in swig_link_libraries
 execute_process ( COMMAND python -c "from distutils.sysconfig import get_python_lib; print(get_python_lib().replace('\\\\', '/'))" OUTPUT_VARIABLE PYTHON_SITE_PACKAGES OUTPUT_STRIP_TRAILING_WHITESPACE)

And this my own variables to find numpy includes and library

set(NUMPY_INCLUDE_DIR ${PYTHON_SITE_PACKAGES}/numpy/core/include)
set(NUMPY_LIB ${PYTHON_SITE_PACKAGES}/numpy/core/lib/npymath.lib)
message (STATUS "Numpy include path: " ${NUMPY_INCLUDE_DIR} )
message (STATUS "Numpy lib: " ${NUMPY_LIB} )
include_directories(${NUMPY_INCLUDE_DIR})

Note that you will most probably want to use numpy’s typemap files found here. You need the numpy.i and pyfragments.swg files. You should put these file in your own source control. Building numpy is tedious, depending on how you install numpy, conda, Christoph Gohlke or pip, you might not find this file in your site-packages. Besides, you probably want the latest one from github to follow the recent SWIG patches. Here, I assumed that you have specified the path to a folder with these two files in the NUMPY_SWIG_DIR variable by adding this option to the cmake command:

-DNUMPY_SWIG_DIR=C:\dev\numpy\tools\swig

Note that this my local path, obviously use your own.

Next you can do this in your CMake file:

include_directories(${NUMPY_SWIG_DIR})

Now, when building Python bindings with Visual Studio, we always have to deal with the Debug version. If like me, you don’t want to debug the Python interpreter and all the linked C code of the modules like numpy, you’ll want to avoid the default Python C API behavior. There’s a pragma in this API that will autolink the debug version of the python library when your own project is in debug. That’s really not what you want.

The solution is to use a precompiler define, to avoid the autolink pragma:

# This preprocessor variable disables the automatic linking of pythonXY_d.lib to use the release version of python.lib
add_definitions(-DSWIG_PYTHON_INTERPRETER_NO_DEBUG)
# With Visual Studio we need to turn off iterator debugging because we are linking against the release version of pythonXY.lib
if(MSVC)
 add_definitions(-D_ITERATOR_DEBUG_LEVEL=0)
endif()

Note that _ITERATOR_DEBUG_LEVEL=0 is mandatory for successful linking. Visual C++ linker checks for this explicitly. That also means that all the other debug library you will want to link will have to be compiled with this define value. This can be a major pain but it’s still less painful than having to get all the debug version of python binaries. Furthermore, iterator debugging really kills the performance of any code using the STL in debug. We all understand that the debug version is slow, but not THAT slow. This variable should be defined by the STL maintainers and left at zero for all others. Sadly this is not the default settings.

Here’s how we define the SWIG targets:

set_source_files_properties(SOURCE swig_tests.i PROPERTIES CPLUSPLUS ON)

# On windows the _WIN64 is autodefined by the compiler cl.exe when using the Win64 toolchain
if (${CMAKE_SIZEOF_VOID_P} EQUAL 8)
   set_source_files_properties(SOURCE swig_tests.i PROPERTIES SWIG_FLAGS "-D_WIN64")
endif()

swig_add_module(swig_tests python swig_tests.i)
swig_link_libraries(swig_tests ${PYTHON_LIBRARIES} ${NUMPY_LIB})

if(MSVC)
   set_target_properties(${SWIG_MODULE_swig_tests_REAL_NAME} PROPERTIES COMPILE_FLAGS "/bigobj")
endif()

In the first line, we inform SWIG that we are wrapping C++, this is translated into the -c++ option.

We have to define _WIN64 manually for the swig parser for it to see the same thing as the Microsoft Compiler cl.exe which automatically defines this variable when building for x64. If you don’t do this you end-up with mismatched types at compile time.

The last invocation is to tell the Microsoft Compiler that SWIG generated files are huge. You won’t need that at first, but you will probably end up needing it, mostly in Debug.

That covers all the hard earned bits.

Here’s an example out of folder cmake command adapt it with your own paths:

mkdir build
cd build
cmake .. -G "Visual Studio 14 2015 Win64" -DSWIG_DIR=C:\DEV\swigwin-3.0.8 -DSWIG_EXECUTABLE=C:\dev\swigwin-3.0.8\swig.exe -DNUMPY_SWIG_DIR=C:\dev\numpy\tools\swig -DCMAKE_INSTALL_PREFIX=..

Always prefer out of folder CMake builds. They are much more cleaner. You can just delete the build folder to restart from a perfectly clean slate to test your CMake scripts. There is a lot of caching in CMake, something that worked after a few trial and error might not work when starting from scratch.

Laisser un commentaire

Votre adresse de messagerie ne sera pas publiée. Les champs obligatoires sont indiqués avec *