Development_C_Extension_with_Pybind11.pdf
0 likes2,630 views
This document discusses using Pybind11 to create C extensions for Python. Pybind11 helps extend Python with C/C++ and provides functionality similar to CFFI. Examples are given of integrating NumPy and the Eigen linear algebra library. Testing C code with Google Test (GTEST) is recommended. GTEST example code is provided. Memory leaks should be checked with Valgrind. A case study of using Azure DevOps for continuous integration is also presented. In summary, Pybind11 allows mixing C/C++ with Python for efficiency reasons while leveraging Python libraries, and testing C code is important.
Development_C_Extension_with_Pybind11.pdf
- 1. Development C-extension with using PyBind11 PyConJP 2022 Takayuki Suzuki Shiseido, co, ltd,.
- 2. Table of contents 1. Introduction 2. What is Pybind11? 3. Pybind11 usage and integration with peripheral libraries 4. Test & Memory Check 5. DevOps Case Studies 6. Summary 1
- 3. Intro
- 4. Self Introduction • Python Experience：about 10 years • Working at Shiseido Co. • Research and development of evaluation methods for skin images 2
- 5. Awareness of the problem Organizations have cultures, and programming languages accompany cultures. Especially when efficiency is required, there are historical assets, or for reasons of member skills. There are cases where development in C/C++ is necessary. On the other hand, we want to use Python because of its properties as a glue language and the abundance of libraries. What to do? In this hour, I will introduce the libraries I have used to link Python and C and give examples of how to test them! • Source code is available on Github • https://github.com/SzTk/pyconjp2022_dev_with_pybind11 3
- 6. How Python and the C language work together How to call C from Python? The official document says... • How to include and build Python.h on the C language side (C extension) • How to import ctypes on the Python side and call the library • CFFI Currently, CFFI seems to be the best choice(?). PyBind11 basically provides functionality similar to CFFI → shared this time 4
- 8. What is Pybind11? Pybind11 is a library that helps Python extend C Apparently derived from Boost.Python. The machine learning library dlib also uses it. 5
- 9. Quick Start Examples of usage C language side https://pybind11.readthedocs.io/en/stable/basics.html setup.py 6
- 10. Quick Start Examples of usage C language, so different data types will cause an error. 7
- 11. Pybind11 usage and integration with peripheral libraries
- 12. Numby data type Modules can also be created that use Numpy as input/output. C language side • Function that takes two np.arrays and returns an element-wise summed np.array • The returned numpy type data cannot be created in standard C (of course), so it is created by calling Python’s API. PyBind11 provides a wrapper in pybind11/numpy.h. • There are multiple ways to access numpy data. Here, we use the method of accessing without range checks, etc., giving priority to efficiency. 8
- 13. Numby data type Examples of usage Naturally, if a type other than the specified type is received, an error (or unexpected behavior) will occur. 9
- 14. Eigen Eigen is a Header Only linear algebra library available in C++ Fast enough to be comparable to GOTO Blas 0 2000 4000 6000 8000 10000 12000 14000 16000 18000 20000 10 100 1000 MFLOPS matrix size matrix matrix product ’eigen3’ ’GOTO2’ ’INTEL_MKL’ ’eigen2’ ’ATLAS’ ’gmm’ ’ublas’ http://download.tuxfamily.org/eigen/btl-results-110323/axpy.pdf It seems that the matrix-matrix multiplication is no different from GOTO2/INTEL MKL 10
- 15. Eigen is also available C language side setup.py • Function that performs multiplication of two matrices using Eigen and returns the result • In setup.py, specify the folder where you downloaded eigen for include dirs • The returned data is allocated memory space on the C language side (unlike the numpy example). • The method of memory management (whether to manage in C or to pass management to Python) can be specified by specifying return value policy. 11
- 17. Eigen Notes • Efficient execution of matrix operations (not only Eigen) requires attention to the order in which data is stored in memory. • Numpy arrays are C CONTIGUOUSS (=row-major) by default • Eigen’s default is column-major • We need to either explicitly specify that Eigen also receives RowMajor (which is the method employed in the example above) or specify F CONTIGUOUSS in numpy 13
- 18. Eigen Notes In the above example, an error will occur if the array is passed after setting F CONTIGUOUSS (because Raw Major = C CONTIGUOUSS in the argument type) 14
- 19. Linking to libraries If the C language side depends on an external library (.dll for Windows, .so for Linux, etc.), the dependent library should be listed in setup.py. This example is linked with opencv in /usr/local/lib If you distribute packages, you need to figure out how to distribute libraries as well 15
- 20. Test & Memory Check
- 21. Test About the Test When developing extensions in C or C++, Python’s GC does not manage memory areas allocated in C modules. It is necessary to check for memory leaks in the C language part. I will share my method of checking. 16
- 22. GTEST GTEST can be used for testing at the C language level. https://github.com/google/googletest GTEST is a library for testing the C++ language developed by Google Note that testing at the Python level is done using nosetests. 17
- 23. GTEST Usage Example My practice of GTEST • Separate IF and C language parts with Python • Test the C language part with GTEST • Since it is C language, it is compiled (using CMake, etc.) to generate an executable binary for testing. • Python part tested with nosetests 18
- 24. GTEST Usage Example Consider testing the matmul function used in eigen Cut out only the matmul function to a separate file for testing This has an intentional memory leak in it. 19
- 25. GTEST Usage Example Write test code for GTEST 20
- 26. GTEST Usage Example 例 CMakeFile.txt 21
- 27. GTEST Usage Example Build and Run Execution Result 22
- 28. GTEST Usage Exampl The Python IF part can be done as follows • Write matmul is defined externally • Add the file in which matmul is defined to setup.py 23
- 29. memory check In addition to testing, memory leaks must also be checked. • Valgrind is an OSS development tool for memory debugging, memory leak detection, profiling, etc. • This time, the test is done by passing the GTEST run command to valgrind. 24
- 30. memory check • For memory leaks, just look at LEAK SUMMARY. • The leak summary points out that 40 bytes of memory have been forgotten to be deleted. • should be reduced to zero, including POSSIBLY 25
- 32. DevOps Pipeline In our case, the skin analysis program is available for internal use. To automate the process up to this point In my case, I am using Azure DevOps to create a pipeline After access management using Azure Active Directory It is useful to be able to publish packages for internal use 26
- 33. Summary
- 34. Summary Summary • Depending on organizational culture and efficiency requirements, a mixture of C and Python may be used for development • Introduced how to use Pybind11 to link Python and C languages • Practical methods of testing the C language portion of the program were introduced. • Introduced a case study of DevOps use in our company. Even today, there are still cultural areas where the C language is still alive. I believe that working with other languages is also a cross-cultural exchange. Why don’t you all try to touch the C extension of Python and have a cross-cultural exchange. 27
- 35. Questions? 27