OpenDreamKit

A project funded by the Horizon 2020

European Research Infrastructures Work Program

Workshop on Low level interfaces with libraries: Project and progress reports

During the workshop on Low level interfaces with libraries, we maintained on this page a list of projects and progress reports on them.

Desired format: for each project that was worked on during the workshop, please include 1-3 paragraphs about what the project was about and what was achieved. You may want to mention the impact and, if relevant whether workshop provided some unique setting that enabled those achievements. Please include links to all relevant tickets / issues that has been worked on. If possible, please tag those tickets / issues with the label Cernay2018.

Survey paper

The workshop was the occasion to build a shared vision among the participants on the current landscape of tools and strategies to integrate low-level libraries in (math) systems. To capitalize on this vision, we started writing collaboratively a survey paper

GAP as a library

Jeroen, Nicolas, Samuel, Sebastian, Thomas

TODO: copy this paragraph into the paper

Context

GAP is an open source system for computational discrete algebra. It’s composed of a kernel written in C, and a library (and many packages) written in its own bespoke language. Originally it was developed as a standalone system. Since a decade however, other systems like SageMath have started using it as a library. The main approaches for interfacing have been text based (either interacting directly with the GAP interpreter through pexpect, or using the OpenMath/SCSCP protocol over a socket). Such interfaces are not suitable for low granularity computations (latency) or when large amounts of data are involved.

To circumvent this, a low interface for GAP – called libgap – is in development, enabling programs to interact with GAP through a C-level API.

Existing or planned applications include:

A couple years ago a prototype was implemented as a fork of GAP for use in SageMath under the lead of Volker Braun. Being a fork involves non trivial maintenance overhead. Over the last year, in particular at the occasion of a previous workshop in Cernay, libgap has been reimplemented within GAP itself under the lead of Markus Pfeiffer, with ultimate goal to make the GAP interpreter itself into a thin layer over libgap.

The work consists in (1) disentangling the GAP kernel from the interpreter, (2) updating the build system, and (3) handling the ABI to reduce risks of symbols conflicts at link time. In the prototype, (3) is handled by mangling all symbols (this feature is dubbed prefixed libgap); the caveat is that it breaks the ABI-level compatibility with a standard-built GAP. In practice Sage’s libgap currently can’t load GAP packages that involve dynamic modules (e.g. Semigroups). In the last two years, a lot of work was put into refactoring GAP’s build system, making (2) much easier. But also making GAP 4.9 a dependency for GAP’s official libgap.

Current status

GAP 4.9 has been released recently; however more work is needed on libgap, and its inclusion was postponed for a later version, hopefully GAP 4.10. See https://github.com/gap-system/gap/pull/1205#issuecomment-385670578

Achievements

The workshop was the occasion to coordinate among the parties, and move forward.

First, some work was done to prepare the upgrade to GAP 4.9 in SageMath (see tickets below).

Sebastian also did some updates to Markus’ libgap, which now compiles and works in Julia: https://github.com/sebasguts/libgap .

Finally we explored potential solutions for (3) and discussed the desirable API and ABI. One difficulty is that Python/Sage cannot deal with a plain unprefixed libgap due to conflicts, due to conflicts with the T_NUM equivalent defines (T_BOOL, etc…) in https://github.com/python/cpython/blob/master/Include/structmember.h A proposed strategy is to define a small clean C-API, and hide all other symbols from the C-kernel in the libgap.so file. (see http://blog.fesnel.com/blog/2009/08/19/hiding-whats-exposed-in-a-shared-library/ ). This definition will be nurtured by reviewing the available GAP kernel symbols and which one have been used in practice by e.g. SageMath or Julia.

We also discussed how to smoothen the transition. We concluded that maintaining ABI-level backward compatibility is not important: GAP users either use precompiled GAP packages, or recompile their GAP packages whenever they upgrade GAP. As for API-level backward compatibility, this should be easy through the use of macros. Fixing Sage to use unprefixed symbols should be just a matter of query-replacing.

References

Volker says:

I don’t anticipate any particular issues with the libgap update. The proof is in the pudding, of course. Not all files are in the tarball, you should work on the libgap repo (which does contain src/libgap.map)

Upgrade to Arb 2.13.0 in SageMath

https://trac.sagemath.org/ticket/24927 https://trac.sagemath.org/ticket/24928 https://github.com/fredrik-johansson/arb/issues/152

Vincent, Vincent

LinBox-Sage interface

–> should we try to find how to pass matrix data with cppyy?

Vincent, Jeroen

Use GAP’s semigroup package in sage-semigroups

Nicolas, Florent, James

(requires GAP 4.9 in SageMath)

cppyy tutorial

Nicolas, James, Florent, Sylvain, Robert, Stefan, …

github repo with the notebook used for the tutorial: https://github.com/bluehood/cppyy_tutorial

Python bindings for libsemigroups

Experimenting with Cython, Pybind11 , cppyy James, Florent, Nicolas, TODO: James: report :-)

We managed to do some simple semigroup computations from Python using cppyy

Use libsemigroups in sage-semigroups

Nicolas, Florent, James

Redesign libsemigroups, for easier interfacing and performance

Two things need to be done:

TODO: Discussion with Jeroen, Enrico, …

Reproduce and report Cython bugs

Vincent Delecroix

Cython: discuss PEP 575

See https://www.python.org/dev/peps/pep-0575/ and https://github.com/jdemeyer/cpython/tree/pep575

Goal: unify functions and methods implemented in Python and C, in order to have a consistent API (will help uniform introspection, e.g. for Sphinx documentation) without losing performance.

It would be good if Cython could somehow “officially” support this PEP by writing to python-dev. (Done)

BIG PROBLEM: people have concerns about backwards compatibility. But it’s difficult to measure this problem.

-> one way is to run the testsuites of a lot of Python modules. It may also be possible to get most of the benefits without backwards compatibility issues, and defer cleanup ‘till later.

Jeroen, Erik(?), Antoine(?), Stefan? Robert? Florent, Serge

Cython: improve installation of Cython projects

Push Cython-specific distutils hooks from Sage to Cython upstream. See https://github.com/sagemath/sage/blob/8.1/src/setup.py#L195

Better integration between Cython and distutils Make it easy to ship the Cython-generated .c files in the source tarball Best practices for shipping Cython-depending code Lessons learned from SageMath, Astropy …

2008-05-03: Plan formulated.

Cython: discuss “late includes” feature

Cython allows to embed arbitrary C code in a Cython module, for example with cdef extern from “foo.c”. However, it’s difficult to access Cython objects from that C code. An experimental feature was added to Cython 0.28 to make this possible but we should now find a proper solution.

Needed for cysignals. Jeroen promised to finish by tomorrow.

Jeroen, Robert, Stefan

Cython: tracebacks are broken for Cython code in Python 3

See https://trac.sagemath.org/ticket/24681 See​ https://bugs.python.org/issue32797 Upstream Python fix at https://github.com/python/cpython/pull/6653

Jeroen, Robert, Stefan, Erik

Cython: discuss tp_new / tp_dealloc customization

It would be great if Cython could allow further customization of allocating and deallocating objects. This could be used for example to improve the “Integer pool” (https://trac.sagemath.org/ticket/17670)

Jeroen, Florent

Triage a bunch of Cython bugs

a bunch! (>= 10) TODO: add links if easy

Triaged: https://github.com/cython/cython/issues/2232 https://github.com/cython/cython/issues/2231 https://github.com/cython/cython/issues/2229 https://github.com/cython/cython/issues/2222 https://github.com/cython/cython/issues/2214 https://github.com/cython/cython/issues/2091 https://github.com/cython/cython/issues/1818 https://github.com/cython/cython/pull/1842 https://github.com/cython/cython/pull/2228 https://github.com/cython/cython/pull/2207

Fixed: https://github.com/cython/cython/issues/2216 https://github.com/cython/cython/issues/1397 https://github.com/cython/cython/issues/1399 https://github.com/cython/cython/issues/1393 https://github.com/cython/cython/issues/1771 https://github.com/cython/cython/pull/2239 https://github.com/cython/cython/pull/2219 https://github.com/cython/cython/pull/2234 https://github.com/cython/cython/pull/391

Plus some other optimizations with dicts and slots (both done and ongoing).

Cython: discuss cpdef functions with slots = []

Bug or feature? https://github.com/cython/cython/issues/1771

Fast callable and arbitrary precision (path) integration in Sage

There is now code for performing certified integration with arb (https://trac.sagemath.org/ticket/24686). However, the input needs to be a “arb” function (with some knowledge about analytic features). Ideally, we would have the possibility of integrating symbolic expressions directly… and possibly pass via a fast callable that would compile the expression into more efficient C code.

Vincent Delecroix

Use Cython’s multiple inheritance

People have complained for years that Cython doesn’t allow multiple inheritance. Now they do allow it (under certain conditions), but what are the possible uses? (Some discussion.)

usecase: category framework in Sage

Nicolas, Robert, Jeroen, Stefan, Florent ???

Explore the possibility of using import hooks for .sage files

Luca: Erik, I need your help

Work on a couple of overdue OpenDreamKit deliverables

Luca

Cross-binding-library fast function calls (and general C-level exposure)

Cython, cppyy, PyBind11, NumPY, etc. should all be able to call each other without passing through the Python layer.

See https://annuel2.framapad.org/p/pywrapper-interop

tutorial on cppyy

https://mybinder.org/v2/gh/bluehood/cppyy_tutorial/master?filepath=cppyy_tutorial.ipynb

Stefan, Robert, Jim, … ?

Leverage cppyy for Cython .pxd generation

Simple prototype Upstream PR: https://bitbucket.org/wlav/cppyy-backend/pull-requests/13/export-some-more-information-in-cppyy/diff

Robert

Integration xtensor <-> Pythran

Serge, Sylvain, …

Fight ongoing :-) Numpy can be faster than xtensor!

cysignals on Windows (Cygwin + native)

Good progress is being made

Erik, Jeroen, Vincent Klein

Better support for small matrix algebra in xtensor

Hadrien, Sylvain, Wolf, Johan

During a Google Summer of Code project, the LAL computing group would like to explore adding support for efficient small matrix algebra to the xtensor stack. As a first step, Hadrien ported existing small matrix algebra work of his group from Boost.SIMD to xsimd, the SIMD abstraction layer used by xtensor.

This work went very smoothly, as xsimd was initially inspired by Boost.SIMD and had very similar API logic. However, the work also revealed some feature gaps of xsimd with respect to Boost.SIMD: - No way to emulate vector sizes which do not exist in hardware (PR: https://github.com/QuantStack/xsimd/pull/100 ) - No way to efficiently leverage compile-time knowledge of the mask for a blend/select operation (ticket: https://github.com/QuantStack/xsimd/issues/97 )

Both issues wore worked on during the workshop by Hadrien and the QuandStack team, and are now close to being resolved. This was also an excellent opportunity for the two teams to get to know each other and start investigating future collaboration.

Get ROOT, PyROOT, JupyROOT installed in CoCalc

Samuel, Jim, Enrico

Samuel and Enrico spent some time on getting a recent (i.e. v6.12 or later) ROOT version installed in CoCalc. ROOT is compile-able and installable in CoCalc as a local package, Sam got in touch with CoCalc maintainers to actually get a recent version pre-installed in the container environment. PyROOT (ROOT’s python module) also works form the commandline interpreter. JupyROOT (jupyter with the ROOT C++ kernel) was not working properly (kernel loop-crashinig), possibly because of missing python packages (e.g. metakernel). Enrico is available for help in case Sam requires further assistance.

Python callable to C++

Jim, Enrico, Serge, Robert, Stefan

Jim put together a proof of concept that shows it is possible to jit a python function with numba and then invoke the jitted C function from C++ code jitted with cppyy. This effectively makes it possible to make numba-friendly python callables invokable from C++. Then Enrico spent some time trying to come up with a non-expert-user-friendly interface that would do most of this work under the hood, given that the C types of the arguments that will be passed to the python function are known. Some progress was made and this will hopefully make it to pyROOT. More relevant for the world at large is the discussion with Stefan/Robert on Friday (see topic further up)

Using VecCore with the whole Cython/Jupyter thing (yellow lines)

day 1:

–> Discovered that compilation errors go to stdout on Jupyter server, rather than in notebook. –> Literal C++ string option (verbatim includes) is a lifesaver but needs to be documented!

day 2:

–> switched to xsimd (part of xtensor), much easier interface –> xsimd lacks AVX-512 support; Wolf started implementing them –> easier to work in C++ directly, non-Jupyter, because working out indexes in the algorithm isn’t the hard part— finding the right intrinsics is –> Hadrien had a much better idea for the algorithm (conversation on the walk); now using Hillis & Steele –> implemented it and vectorized algorithm is 10x faster than -O2 sequential, 3x faster than -O3 sequential –> porting back to Cython + Numpy to check against numpy.cumsum

Jim, Robert (answering some questions), Wolf Vollprecht, Hadrien Grasland

Conda brainstorm

Themes:

Sylvain, Julian, Nicolas, Jim, Stefan, Samuel

Support “external” datatypes in Sage functions (is_prime, factor, etc)

Mainly focused on gmpy2 for now https://trac.sagemath.org/ticket/24677

Vincent Klein

Cython users support group

Friday at 11:00am

Document installing Xeus-Cling in a CoCalc project

Samuel, with help from Sylvain, Julian, Hadrien

This might help with the system-wide install on CoCalc, tracked as

CoCalc is in the process of upgrading to Ubuntu 18.04, see announcement on Twitter – https://twitter.com/cocalc_com/status/992021191161499650 – saying “We’re already at the stage of testing a major upgrade to @ubuntu #1804 internally.” CoCalc should provide xeus-cling once the move to Ubuntu 18.04 is done. The CoCalc team offered to let one of Samuel’s CoCalc projects use the Ubuntu-18.04-based CoCalc image, which is still in beta.

Document installing ROOT in a CoCalc project

Samuel, Enrico

Still needs some love.

This should help with the system-wide install on CoCalc, tracked as

Full GAP in CoCalc

Samuel, Sebastian?

This is tracked at:

There was progress in end-of-Dec-2017 / beg-of-Jan-2018, but it has been stalled since, although it’s almost complete.

Polymake on CoCalc

Samuel, Andreas

Better etherpad service

Samuel

The new etherpad-lite 1.6.6 was released just after our Cernay workshop, and has a fix for the issue which caused us a lot of frustration:

https://github.com/ether/etherpad-lite/releases/tag/1.6.6
FIX: text entered between connection loss and reconnection was not saved

In early May 2018, Framapad runs etherpad-lite 1.2.13. Samuel’s request to upgrade to etherpad-lite 1.6.6:

<