Boost.Python: C++ and Python Integration

Boost.Python
C++ & Python Integration

Motivation
Leveraging of strong sides in both languages:
1. C++:
a. performance,
b. effective resource management
2. Python:
a. high coding performance,
b. variety of available solutions
c. easy to start

Why Boost.Python?
Python C-API: requires too much manual work
SWIG (aka generators): usually requires
manual adjusting - take time to understand
“someones” code.
Boost.Python (middleware lib): most flexible -
expose what you need in the way you like.

Getting Boost ready
Getting Boost ready:
get your copy of boost and build it:
./bootstrap.sh --prefix=../../ --with-python-version=3.4
--with-python-version=X.Y
--with-python=/path/to/bin/python
--with-python-root=/path/to
./b2 -j4 address-model=64 --with-python stage
builds python 3.4 lib (only) for 64 bit os, all binaries are in
stage folder

… pick what to wrap
Simple C API to wrap into Python
int LZ4_compress(const char* source,
char* dest, int sourceSize);
int LZ4_decompress_safe (
const char* source, char* dest,
int compressedSize, int maxDecompressedSize);
basic API of lz4 compression library

Step #1: declare Python module
#include <boost/python.hpp>
#include <lz4.h>
using namespace boost::python;
BOOST_PYTHON_MODULE(lz4py)
// make sure that your output binary will have same name
// lz4py.so (not liblz4py.so ) or lz4py.dll
{
// place for our wrappers
}

basic wrappers...
def("dump", // name of function on Python side
LZ4_compress
/*, return_value_policy<>() - optional */ );
def("load", LZ4_decompress_safe);

… but
def() will wrap functions as they are …
and call on Python side will not be so comfortable:
lz4py.dump(
input, # data to compress
output, # ALLOCATED OUTPUT BUFFER (!?)
size) # size of input data (!?)
and function will return size of used bytes in output buffer
it is too much C style - this is not Python...

Step #2: make it more Python
add tiny wrapper in C++ code:
const std::string compress(const std::string& d);
const std::string decompress(const std::string& d);
and wrap it for Python:
def("dump", compress );
def("load", decompress);
and … new trouble

… string and bytes
In Python 3 all string are UNICODE:
1. can’t return binary data as string - will fail
a. could work for python 2 - but this is hidden
problem in future
2. need a way to return binary data
a. convert output to bytes()

Step 4: type conversion
Declare own type:
class buffer {
public:
typedef boost::shared_ptr<char> data_ptr_t;
public:
buffer(data_ptr_t& data, int size): data_(data), size_(size){}
int len() const { return size_; }
char* const get_data() const { return data_.get(); }
private:
data_ptr_t data_;
int size_;
};

Converter to Python
Converter interface declaration:
template< typename T >
struct type_into_python {
static PyObject* convert( T const &);
};
Converter implementation:
template<> PyObject* type_into_python<buffer>::convert(
buffer const& data){
return PyBytes_FromStringAndSize(data.get_data(), data.len());
}

Converter from Python: declaration
template< typename T >
struct type_from_python {
type_from_python(){
converter::registry::push_back( convertible,
construct, type_id<T>() );
}
static void* convertible( PyObject* );
static void construct(
PyObject*,
converter::rvalue_from_python_stage1_data* );
};

Convert from Python: implement
template<> void* type_from_python<buffer>::convertible( PyObject* obj){
return PyBytes_Check(obj) ? obj : nullptr;
}
template<> void type_from_python<buffer>::construct( PyObject* obj,
converter::rvalue_from_python_stage1_data* data){
auto storage = reinterpret_cast< converter::rvalue_from_python_storage<buffer>* >(
data )->storage.bytes;
int size = PyBytes_Size(obj);
buffer::data_ptr_t buffer_data = buffer::data_ptr_t(new char[size]);
memcpy(buffer_data.get(), PyBytes_AsString(obj), size);
new(storage) buffer(buffer_data, size);
data->convertible = storage;
}

New API
New C++ functions:
const buffer compress(buffer const& src);
const buffer compress(std::string const& src);
const buffer decompress(buffer const& src);
Python exports:
def("dump", static_cast<const buffer (*)(buffer const&)>( compress ));
def("dump", static_cast<const buffer (*)(std::string const&)>( compress ));
def("load", decompress);

Step #5: class
Wrapp interface:
class lz4_c {
class bytes_ref{
PyObject* bytes_;
public:
int len() const { return
PyBytes_Size(bytes_);}
char const* data() const { return
PyBytes_AsString(bytes_); }
};
class buffer{
private: lz4_c& owner_;
};
public:
// initialize & use pool of preallocated blocks
lz4_c(int block_max_size, int count);
const buffer compress(bytes_ref const& src);
const buffer compress(std::string const& src);
const buffer decompress(bytes_ref const&
src);
void release(data_block_t&);
};

Boost.Python class wrapper
Declare Python class for lz4_c:
class_<lz4_c>("lz4",
init<int, int>( args("block_max_size", "count")))
//.def(init<int, int>( args("block_max_size", "count")))
● “lz4” - class name on Python side;
● second argument - constructor, can be skipped if class have default one;
● other constructors could be added using .def() like functions;
● args - you can give names to your arguments and use named args in
python;

Class methods
extend our class by methods:
.def("dump", static_cast<const lz4_c::buffer (lz4_c::*)(
lz4_c::bytes_ref const&)>(&lz4_c::compress))
.def("dump", static_cast<const lz4_c::buffer (lz4_c::*)(std::string const&)>(
&lz4_c::compress))
.def("load", &lz4_c::decompress)
similar to regular functions.

Class properties
can be exposed to python code:
// read only
.add_property("ratio", &lz4_c::ratio)
// read / write
.add_property("digit", make_getter(&lz4_c::digit),
make_setter(&lz4_c::digit))

Exceptions
class my_ex: public std::exception {};
...
void translate_error( error const& my_ex);
...
//boost::python::register_exception_translator<T,F>(F)
boost::python::register_exception_translator<my_ex>(
translate_error)

And more
STL type conversion:
● std::map <-> dict();
● std::vector <-> list(), tuple()
Define Python operators
● __str__, __repr__, __ call__
● __add__, __lt__:
o .def(self + self)
o .def(self < self)
Inheritance:
● make available in Python
● declare C++ relation
Default args:
● wrapper to allow default
args:
Python function pointers:
● typedef boost::function<void
(string s) > funcptr;

Return Value Policy
● with_custodian_and_ward: Ties lifetimes of the arguments
● with_custodian_and_ward_postcall: Ties lifetimes of the arguments and
results
● return_internal_reference: Ties lifetime of one argument to that of result
● return_value_policy<T> with T one of:
o reference_existing_object: naive (dangerous) approach
o copy_const_reference: Boost.Python v1 approach
o copy_non_const_reference:
o manage_new_object: Adopt a pointer and hold the instance

Thank
You !
Andriy.Ohorodnyk@globallogic.com

Boost.Python: C++ and Python Integration

More Related Content

What's hot (20)

Viewers also liked (20)

Similar to Boost.Python: C++ and Python Integration (20)

More from GlobalLogic Ukraine (20)

Boost.Python: C++ and Python Integration