Shock and Awe 1¶
A concise overview of libpy. For an introduction to extending Python with C or C++ please see the Python documentation or Joe Jevnik’s C Extension Tutorial.
Simple Scalar Functions¶
We start by building simple scalar functions in C++ which we can call from Python.
In [1]: from libpy_tutorial import scalar_functions
A simple scalar function:
bool bool_scalar(bool a) {
return !a;
}
In [2]: scalar_functions.bool_scalar(False)
Out[2]: True
A great way to use libpy is to write the code that needs to be fast in C++ and expose that code via Python. Let’s estimate pi using a monte carlo simulation:
double monte_carlo_pi(int n_samples) {
int accumulator = 0;
std::random_device rd; // Will be used to obtain a seed for the random number engine
std::mt19937 gen(rd()); // Standard mersenne_twister_engine seeded with rd()
std::uniform_real_distribution<> dis(0, 1);
for (int i = 0; i < n_samples; ++i) {
auto x = dis(gen);
auto y = dis(gen);
if ((std::pow(x, 2) + std::pow(y, 2)) < 1.0) {
accumulator += 1;
}
}
return 4.0 * accumulator / n_samples;
}
In [3]: scalar_functions.monte_carlo_pi(10000000)
Out[3]: 3.1416408
Of course, we can build C++ functions that support all the features of regular Python functions.
libpy supports optional args:
std::string optional_arg(py::arg::optional<std::string> opt_arg) {
return opt_arg.get().value_or("default value");
}
In [4]: scalar_functions.optional_arg(b"An argument was passed")
Out[4]: b'An argument was passed'
In [5]: scalar_functions.optional_arg()
Out[5]: b'default value'
and keyword/optional keyword arguments:
py::owned_ref<>
keyword_args(py::arg::kwd<decltype("kw_arg_kwd"_cs), int> kw_arg_kwd,
py::arg::opt_kwd<decltype("opt_kw_arg_kwd"_cs), int> opt_kw_arg_kwd) {
return py::build_tuple(kw_arg_kwd.get(), opt_kw_arg_kwd.get());
}
In [6]: scalar_functions.keyword_args(kw_arg_kwd=1)
Out[6]: (1, None)
In [7]: scalar_functions.keyword_args(kw_arg_kwd=1, opt_kw_arg_kwd=55)
Out[7]: (1, 55)
Working With Arrays¶
In order to write performant code it is often useful to write vectorized functions that act on arrays. Thus, libpy has extenstive support for numpy arrays.
In [8]: from libpy_tutorial import arrays
In [9]: import numpy as np
We can take numpy arrays as input:
std::int64_t out = 0;
for (auto value : values) {
out += value;
}
return out;
}
In [10]: some_numbers = np.arange(20000)
In [11]: arrays.simple_sum(some_numbers)
Out[11]: 199990000
and return them as output:
std::transform(values.cbegin(),
values.cend(),
values.begin(),
[](std::int64_t v) { return -v; });
}
bool check_prime(std::int64_t n) {
if (n <= 3) {
return n > 1;
}
else if (n % 2 == 0 || n % 3 == 0) {
return false;
}
for (auto i = 5; std::pow(i, 2) < n; i += 6) {
if (n % i == 0 || n % (i + 2) == 0) {
return false;
}
}
return true;
}
In [12]: prime_mask = arrays.is_prime(some_numbers)
In [13]: some_numbers[prime_mask][:100]
Out[13]:
array([ 2, 3, 5, 7, 11, 13, 17, 19, 23, 25, 29, 31, 37,
41, 43, 47, 53, 59, 61, 67, 71, 73, 79, 83, 89, 97,
101, 103, 107, 109, 113, 121, 127, 131, 137, 139, 149, 151, 157,
163, 167, 173, 179, 181, 191, 193, 197, 199, 211, 223, 227, 229,
233, 239, 241, 251, 257, 263, 269, 271, 277, 281, 283, 289, 293,
307, 311, 313, 317, 331, 337, 347, 349, 353, 359, 367, 373, 379,
383, 389, 397, 401, 409, 419, 421, 431, 433, 439, 443, 449, 457,
461, 463, 467, 479, 487, 491, 499, 503, 509])
Note
numpy arrays passed to C++ are ranges.
return std::accumulate(values.begin(), values.end(), 0);
}
In [14]: arrays.simple_sum_iterator(some_numbers)
Out[14]: 199990000
N Dimensional Arrays¶
We can also work with n-dimensional arrays. As a motivating example, let’s sharpen an image. Specifically - we will sharpen:
In [15]: from PIL import Image
In [16]: import matplotlib.pyplot as plt # to show the image in documenation
In [17]: import numpy as np
In [18]: import pkg_resources
In [19]: img_file = pkg_resources.resource_stream("libpy_tutorial", "data/original.png")
In [20]: img = Image.open(img_file)
In [21]: plt.imshow(img)
Out[21]: <matplotlib.image.AxesImage at 0x7f60f88909d0>
py::owned_ref<> apply_kernel(py::ndarray_view<const std::uint8_t, 3> pixels,
py::ndarray_view<const std::int64_t, 2> kernel) {
auto n_dimensions = pixels.shape()[2];
auto n_rows = pixels.shape()[0];
auto n_columns = pixels.shape()[1];
auto k_rows = kernel.shape()[0];
auto k_columns = kernel.shape()[1];
std::vector<std::uint8_t> out(n_dimensions * n_rows * n_columns, 0);
py::ndarray_view out_view(out.data(),
pixels.shape(),
{static_cast<int>(n_dimensions * n_rows),
static_cast<int>(n_dimensions),
1});
for (std::size_t dim = 0; dim < n_dimensions; ++dim) {
for (std::size_t row = 0; row < n_rows; ++row) {
for (std::size_t column = 0; column < n_columns; ++column) {
auto accumulated_sum = 0.0;
for (std::size_t k_row = 0; k_row < k_rows; ++k_row) {
for (std::size_t k_column = 0; k_column < k_columns; ++k_column) {
auto input_row_idx = row + 1 - k_row;
auto input_column_idx = column + 1 - k_column;
if (input_row_idx < n_rows && input_column_idx < n_columns) {
accumulated_sum +=
pixels(input_row_idx, input_column_idx, dim) *
kernel(k_row, k_column);
}
}
}
if (accumulated_sum < 0) {
accumulated_sum = 0;
}
else if (accumulated_sum > 255) {
accumulated_sum = 255;
}
out_view(row, column, dim) = accumulated_sum;
}
}
}
return py::move_to_numpy_array(std::move(out),
In [22]: pixels = np.array(img)
In [23]: kernel = np.array([
....: [0, -1, 0],
....: [-1, 5, -1],
....: [0, -1, 0]
....: ]) # already normalized
....:
In [24]: from libpy_tutorial import ndarrays
In [25]: res = ndarrays.apply_kernel(pixels, kernel)
In [26]: plt.imshow(res)
Out[26]: <matplotlib.image.AxesImage at 0x7f60f8886460>
Note
We are able to pass a shaped n-dimensional array as input and return one as output.
Creating Classes¶
libpy also allows you to construct C++ classes and then easily expose them as if they are regular Python classes.
In [27]: from libpy_tutorial.classes import Vec3d
C++ classes are able to emulate all the features of Python classes:
namespace libpy_tutorial {
class vec3d {
private:
std::array<double, 3> m_values;
public:
vec3d(double x, double y, double z) : m_values({x, y, z}) {}
double x() const {
return m_values[0];
}
double y() const {
return m_values[1];
}
double z() const {
return m_values[2];
}
vec3d operator+(const vec3d& other) const {
return {x() + other.x(), y() + other.y(), z() + other.z()};
}
vec3d operator-(const vec3d& other) const {
return {x() - other.x(), y() - other.y(), z() - other.z()};
}
double operator*(const vec3d& other) const {
return std::inner_product(m_values.begin(),
m_values.end(),
other.m_values.begin(),
0.0);
}
double magnitude() const {
return std::sqrt(*this * *this);
}
};
std::ostream& operator<<(std::ostream& s, const vec3d& v) {
return s << '{' << v.x() << ", " << v.y() << ", " << v.z() << '}';
}
// `repr` could also be a member function, but free functions are useful for adding
// a Python repr without modifying the methods of the type.
std::string repr(const vec3d& v) {
std::stringstream ss;
ss << "Vec3d(" << v.x() << ", " << v.y() << ", " << v.z() << ')';
return ss.str();
}
} // namespace libpy_tutorial
namespace py::dispatch {
// Make it possible to convert a `vec3d` into a Python object.
template<>
struct LIBPY_NO_EXPORT to_object<libpy_tutorial::vec3d>
: public py::autoclass<libpy_tutorial::vec3d>::to_object {};
} // namespace py::dispatch
In [28]: Vec3d.__doc__
Out[28]: 'An efficient 3-vector.'
In [29]: v = Vec3d(1, 2, 3)
In [30]: v
Out[30]: Vec3d(1, 2, 3)
In [31]: str(v)
Out[31]: '{1, 2, 3}'
In [32]: v.x(), v.y(), v.z()
Out[32]: (1.0, 2.0, 3.0)
In [33]: w = Vec3d(4, 5, 6); w
Out[33]: Vec3d(4, 5, 6)
In [34]: v + w
Out[34]: Vec3d(5, 7, 9)
In [35]: v * w
Out[35]: 32.0
In [36]: v.magnitude()
Out[36]: 3.7416573867739413
Exceptions¶
Working with exceptions is also important.
In [37]: from libpy_tutorial import exceptions
We can throw exceptions in C++ that will then be dealt with in Python. Two patterns:
Throw your own exception:
throw py::exception(type, msg...), maybe in response to an exception from a C-API function.Throw a C++ exception directly.
void throw_value_error(int a) {
throw py::exception(PyExc_ValueError, "You passed ", a, " and this is the exception");
}
void raise_from_cxx() {
throw std::invalid_argument("Supposedly a bad argument was used");
}
In [40]: exceptions.throw_value_error(4)
---------------------------------------------------------------------------
ValueError Traceback (most recent call last)
<ipython-input-40-6cfdcf9a1ea9> in <module>
----> 1 exceptions.throw_value_error(4)
ValueError: You passed 4 and this is the exception
In [41]: exceptions.raise_from_cxx()
---------------------------------------------------------------------------
RuntimeError Traceback (most recent call last)
<ipython-input-41-ee2345413222> in <module>
----> 1 exceptions.raise_from_cxx()
RuntimeError: a C++ exception was raised: Supposedly a bad argument was used
Footnotes
- 1
With naming credit to the intorduction of Q for Mortals.