Summer update

A bit more than 3 months after the last update, I think it is really time to share some updates about MPTCP! I have been quiet recently, not because I was on holiday – even if I should probably take some :) – but because there were always higher priority tasks than writing a new blog post which is something I’m slow at! Bug-fixes, improvements, apps support, conferences, what’s next: read on to find out more about what happened recently!

More bug fixes

In my previous post, I mentioned I was working on fixing issues around the path-manager. A few more fixes have been added since then, and a lot of backports have been done: hundreds of commits have been added to the different stable kernels, up to v5.10 when needed. If you see odd behaviour with the MPTCP path-manager, please check with an up-to-date kernel version first, maybe your issue has already been fixed!

One important issue has also been fixed recently: some (weird) middleboxes were causing MPTCP connections to be reset, because MPTCP options were dropped later on during the connection, but not at the beginning. When possible, such connections are now forced to fall back to TCP. Middleboxes are still able to surprise us! :)

New improvements

Some improvements have also been added during the last few months, e.g.

If MPTCP is explicitly blocked by a firewall, after 3 unsuccessful connection requests, the kernel will fall back to TCP, and future connections will directly use TCP for a configurable period of time – 1 hour by default, then it is exponential.
SO_KEEPALIVE socket option – to send keep-alive packets in case of connection inactivity – has been supported by MPTCP for a few years now. But it was not possible to fine tune this behaviour using TCP_KEEP* socket options. This has been fixed, and exceptionally backported to stable versions, because it was not normal to support the main option, but not these other ones, this was blocking some deployments, and the code to support these socket options is simple and not impacting the rest.
More MIB counters – visible with nstat – have been added, some of them have also been backported: this will help kernel developers and users to understand some behaviour like why an expected additional subflow has not been established, without having to instrument the kernel.
The documentation about MPTCP endpoints configuration has been improved to avoid confusion. These modifications are visible in the IPRoute manual, and on our website.
The CI now generates code coverage stats (also published on Coveralls). Code coverage is a valuable piece of information to know how much we can trust a test suite, and easily find out what needs to be improved. A majority of the code is covered (~90%), which is great! There are some areas that can be improved – e.g. diag and socket options – but not critical ones. We can also notice many error branches are not covered by the MPTCP test suite: that’s because that would be a lot of work to cover most of them while most of the time the action that has to be taken in this case is very simple. That’s why we rely on Syzkaller, a fuzzing tool, to cover most of these error branches.
Still about the CI, the BPF MPTCP selftests are now also executed with a debug kernel config. More analytic tools will cover this area that is growing.

More apps natively supporting MPTCP

Thanks to the help from Anthony Doeraene, even more apps are now natively supporting MPTCP:

NGinx and FreeNGinx might support it soon too!

An important change as well is on the Go Language: the next version will have MPTCP enabled by default on the server side. (src)

Additionally, Anthony added more languages in the MPTCP Hello repository: Elixir, Erlang, Swift (macOS), and Objective-C (macOS). Plus a dedicated page about how to use MPTCP on macOS.

Other apps and tools also gained MPTCP support recently:

OpenWrt v24 will also have MPTCP support available by default! (src)

Conferences

Netconf 2024

Netconf is a Linux community conference, by-invitation-only. The agenda has a clear focus on kernel level networking. Attendees are the main maintainers and developers of the Linux networking subsystem.

This year, I was grateful to have been invited to the 2024 edition in Vienna, and I learned a lot. Even if MPTCP has been invoked, I was mostly there to discuss CI aspects. For a while now, there is an MPTCP CI validating patches that are shared on the mailing list. It took quite sometime to put that in place, and it regularly needs tweaking. There is still some work to do, like having a proper performance lab in place which is what I’m currently looking at that. On Netdev side, that’s only for less than one year the CI is validating functional tests. A good time to discuss it, how useful it is, what can be improved, and what is important to know about it.

We had very useful discussions there. Amongst the topics that have been discussed around the CI, we have:

How important it is to know the code coverage.
A container image would be useful to easily reproduce issues locally.
A debug kernel config causes a very slow environment, and results in tests to be more unstable, but that can maybe be ignored: what is important here is to look at new kernel warnings found by the debug tools.
Other subsystems can re-use code from the MPTCP CI if they want a similar CI setup on their side. Still, it would be good to have an “official” service applying patches in a Git repository to ease the bootstrap of new CIs.
Stable kernel are often being validated using the last version of the kernel selftests. In theory, that makes sense, and looks good to have the biggest coverage and check for regressions, but most network related selftests expect to be executed on the kernel version it is linked to, and then fail if a feature is missing. That’s an issue because older stable kernels have a lot of failed tests due to that, and probably nobody is monitoring the different results. Maybe there should be an exception for the network tests, because many tests are looking at the internals like how packets are created, not only at what is exposed to the userspace via the different simple APIs, like the socket one that hides all the complexity to the userspace.
Managing the CI takes time, and resources, plus it might be good to have some hardware tests done in the same conditions, from a neutral lab. It might be good to create a dedicated entity to share costs.

LPC 2024

The Linux Plumbers Conference (or LPC) is the premier event for developers working at all levels of the plumbing layer and beyond.

The 2024 edition was also in Vienna, just after Netconf.

There as well, I talked about the CI. The Netdev CI is particular: it runs many different tests, while there are many patches regularly posted on the list. That’s why patches are validated by batches. Due to the complexity – but also to avoid increasing the traffic on the Netdev mailing with people sending patches just to fix unworthy details reported by the CI, instead of checking that upfront – what is being done by the CI is supposed to be checked by experienced reviewers and maintainers. It is then important to agree on what should be advertised, and what to expect from kernel developers, reviewers and maintainers.

(Note that my presentation has been recorded, but there were some issues with the microphone at the beginning, apparently.)

What’s next?

Good news, my work will continue to be supported by the NGI0 Core fund – managed by the NLnet Foundation (a big thank you :)) – for another round. NGI0 Core is made possible with financial support from the European Commission’s Next Generation Internet programme, under the aegis of DG Communications Networks, Content and Technology. This project has received funding from the European Union’s Horizon Europe research and innovation programme under grant agreement No 101092990.

This support will help to cover the maintenance, and some new features I will hopefully talk about soon!

Team work

As always, it is important to note that what I presented here so far is mostly what I was working on. But I’m not alone in this project. For example, Geliang is looking at BPF iter for the subflows, and having a path-manage controlled with BPF ; Paolo helped fix some issues found by Syzbot, but also by users around the path-manager ; Davide switched to a better RST code in case of middleboxes interference ; Paolo and Mat helped with the code reviews ; Christoph continued the SyzKaller infrastructure maintenance.

A great community!