Running Applications on the NetBSD Rump Kernel
Slides at http://eurobsdcon.myriabit.eu/
What is a rump kernel?
It is the (NetBSD) kernel without support for
- executing binaries
- scheduling threads
- managing hardware privilege levels
- most memory management eg virtual memory
- Drivers are the bit of the kernel that turns raw hardware into nice abstractions like files and sockets
- Get something else (host) to do the other parts (host environment)
- Or do without it, just compile in single purpose application.
What use is that?
- You get the missing capabilities from a hypercall layer.
- Like a very simple virtual machine
- man 3 rumpuser
- Provides memory allocation, threads, mutexes, console output, random numbers, clock
- This can run in userspace for example, but can be implemented in any environment fairy easily
What use is that?
- Original use cases around testing drivers.
- You can test a kernel driver without booting the OS.
- You can test much of the OS without booting into it.
- Debugging is easy as it is just a userpsace program
- Also for running kernel drivers in userspace eg for file systems
How was it used?
- Typically either written for rump like the NetBSD tests
- or using
LD_PRELOAD to change some calls using
- The interface was essentially the syscall interface (minus parts not supported by rump)
- In August last year we got a Xen port of the rump kernel working. In addition to porting the hypercall layer, we also got NetBSD libc to compile against the rump kernel.
- We replaced the syscalls in libc with calls to the rump kernel syscall implementations.
- This enabled running real applications, in particular we had LuaJIT running, and a simple web server.
- Not just syscalls, more of a POSIX API.
The POSIX implementation of the rump kernel now runs on
- FreeBSD, OpenBSD, DragonflyBSD
- Linux, including Android
- on architectures including arm, x86, amd64, powerpc, sparc64 and mips
There is continuous integration using TravisCI and buildbot
- Tests all commits to the buildrump.sh script on multiple operating systems and architectures
- Tests NetBSD -current hourly on multiple operating systems and architectures
- Good test of NetBSD portability issues
- Uses ljsyscall to test functionality - good syscall coverage, fast.
- Late last year I started work on rumprun, which allowed running unmodified NetBSD binaries with the rump kernel.
- The main issue with this is that you have two libcs involved, so there are symbol conflicts. However with judicious symbol renaming it turned out to be possible to fix this.
- In particular you can compile userspace binaries for the core NetBSD commands like ifconfig, etc.
- Can test userspace without booting into OS
How does it work?
- Compile NetBSD object code and libraries on host machine
- Link code and libraries to new object with
read symbols to
- Fix up
main and other things which will be called from a stub
- Localise symbols to avoid conflicts
- Link with host libc and rump libraries
- See script for more details, it is messy...
# ./rumpdyn/bin/rump_allserver unix://sock
# export RUMP_SERVER=unix://sock
lo0: flags=8049 mtu 33648
inet6 ::1 prefixlen 128
inet6 fe80::1%lo0 prefixlen 64 scopeid 0x1
inet 127.0.0.1 netmask 0xff000000
# ./bin/ping -o 127.0.0.1
PING 127.0.0.1 (127.0.0.1): 64 data bytes
64 bytes from 127.0.0.1: icmp_seq=0 ttl=255 time=0.000000 ms
----127.0.0.1 PING Statistics----
1 packets transmitted, 1 packets received, 0.0% packet loss
round-trip min/avg/max/stddev = 0.000000/0.000000/0.000000/0.000000 ms
arp cp dumpfs fsck_msdos ls mount_ext2fs ndp pax reboot sysctl wpa_supplicant
cat dd ed halt mkdir mount_ffs newfs pcictl rm umount
cgdconfig df fsck ifconfig mknod mount_msdos newfs_ext2fs ping rmdir vnconfig
chmod disklabel fsck_ext2fs ktrace modstat mount_tmpfs newfs_msdos ping6 rndctl wlanctl
chown dump fsck_ffs ln mount mv npfctl raidctl route wpa_passphrase
- Almost anything in the NetBSD core can probably be compiled
- There is now a cross compiler for rumprun, that should work with most code
- There is experimental pthreads support, not yet well tested
- Programs that fork or use other things not supported by rump may be a problem
- Some of the missing rump features are emulated or partially emulated, eg anon mmap
rumprun - TODOs
- Clean up and upstream libc build modifications
- Fix other architectures, currently only working on x86, amd64
- Build more of userspace tools and libraries
- Use for NetBSD tests - ability to run tests on a kernel without booting it very useful when developing.
- Improve build process
- Continuous integration and testing on NetBSD current.
rump on green threads
- A few months back I added a "green threads" userspace implementation to the userspace rump implementation.
- The default implementation uses pthreads.
- Uniprocessor, single thread
- Useful for embedded implementations
- Also useful if you want to make a deterministic implementation
- Some PCI support has been added
- Linux userspace support.
- Linux has uio and vfio frameworks for userspace pci drivers.
- uio supported now, only works for some cards
- vfio may be later; requires an iommu.
- Can be used for developing NetBSD drivers even so - used for wireless development
- Really want to add BSD support
rump on bare metal
- New project to run a rump kernel on (currently x86) bare metal
- Includes PCI support, with virtio drivers for KVM, bhyve
- Only a few hundred lines of code to deal with interrupts etc, written in a week
- Planning ARM port to mbed
rump on microkernels
- Genode, a microkernel OS framework, has started using the rump kernel to support NetBSD file system drivers on microkernels.
- Less than 3,000 lines of (untrusted) glue code.
- See genode.org for more
- Minix3 is another potential user, it already uses a lot of NetBSD code.
Four different environments
- hosted, e.g. userspace
- paravirtualized, e.g. Xen
- "bare metal", e.g. hardware or hypervisor with virtio
- microkernels (as servers)
- Driver development
- Drivers for other environments
- Applications with userspace drivers, eg networking
- Running code securely eg file system code
- Very lightweight "containers" with their own OS library
What needs doing?
- Upstream and improve rumprun
- Improve documentation
- Portability: native Windows, OSX
- Userspace IP stacks: need good performance on 10Gb