In Linux, virtual memory is divided into user space and kernel space. The kernel is the core part of the operating system that serves as the interface between physical hardware devices and running processes. Kernel space protects that interface from faults (if the operating system itself fails, how does it recover?) and from malicious programs (trying to circumvent operating system security or modify another program).
A diagram of Linux kernel space and user space from Wikipedia (source)
Although most programs are written for user space, there might be reasons why you need to work at kernel space. For example, you could be writing a device driver (e.g., for a graphics card), observability or monitoring at a system level, or implementing custom networking logic. In addition, programs in kernel space are often much faster than user space programs since they don't need to cross the user space/kernel space boundary. So how do you write a program that runs in kernel space?
Extending the kernel or debugging programs in kernel space isn't easy. Linux is a monolithic kernel but supports loading and unloading kernel modules. You wouldn't want the Linux kernel to have the code for supporting every type of graphics card, keyboard, and joystick. A broken kernel module will crash the system. Kernel modules also have to be updated with every kernel version.
Berkley Packet Filter (BPF) has been around since 1992. It was developed to analyze network traffic. It lets you filter packets at the data link layers in the kernel (among other things). For example, you could write a user space program that filters packets (which is much faster than forwarding every packet). BPF accomplishes the user-defined filtering by implementing a small 32-bit virtual machine inside the kernel.
eBPF (extended BPF) extends this virtual machine to ten 64-bit registers. It extends BPF's interface beyond just networking – you can attach it to arbitrary system calls, userspace applications, or arbitrary trace points in the kernel. This means you can achieve a high level of system observability and deep application observability without making any modifications to the kernel. Before, you'd have to load a particular kernel module or run a modified kernel.
What's eBPF used for today?
- High-performance load balancing. Facebook developed Katran, a BPF program that performs layer 4 load balancing.
- System tracing. bpftrace is one of many utilities to collect observability and application metrics on a deeper level.
- Container observability. Falco by Sysdig.
- Compiler toolchains. bcc makes it easier to write eBPF programs, either in Python or Lua.
- Kubernetes networking and security. Cilium does load balancing (layer 3/4) and firewalling (layer 7).
- Solana uses a standalone eBPF as the execution engine for its smart contracts.