Throughput per CPU cycle: the number that survives contact with a router

On two physical routers, moving real traffic over a real gigabit LAN, our tunnel core pushed 122 Mbit/s where the router's own kernel tunnel implementation pushed 66.8 Mbit/s — on the same sender, at less CPU per Mbit. Neither figure is the whole story on its own: the two arms had to use different traffic generators, because no single generator runs on both sides of that comparison. So the ratio to argue with is each arm against its own no-tunnel ceiling, which is in the table below and favours us by more, not less. A four-VM lab run on a laptop says something faster in absolute terms and, on one axis, something less flattering. We trust the router number more, because a router is the only one of the two that a customer's traffic actually crosses.

That is the finding worth leading with, and also the reason this post has two result sections instead of one: the lab and the hardware disagree about which axis matters, and papering over that disagreement would cost more credibility than either number gains.

What was measured, and on what

Two independent environments, because a benchmark is only honest within the environment it ran in — an iperf3 number and a bandwidth-test number are not one number wearing two names, and a VM sharing a laptop's cores is not a router.

Hardware — two MikroTik routers, one gigabit LAN, 2026-09-01. RB3011UiAS (ARMv7, 2×1400 MHz) as sender, cAP ax (ARM64, 4×864 MHz) as receiver, RouterOS 7.24.1 on both. Our core ran as fv, in a RouterOS container, in userspace, against the router's own in-kernel WireGuard-protocol implementation — the only apples-to-apples comparator available on that hardware, since it is the one alternative that ships in RouterOS itself. Two generators, because none crosses implementations cleanly: dd | nc for our container run, RouterOS's own bandwidth-test for the kernel run — each carries its own no-tunnel ceiling as a control, so a reader can sanity-check the ceiling before trusting the ratio.

Lab — four aarch64 Debian guests under QEMU/HVF on one Apple-silicon host, 2026-08-31. Same keypairs, addresses, ports, MTU and interface bring-up across every binary under test: our core, boringtun 0.7 (the userspace core ours was forked from), wireguard-go (the WireGuard-protocol reference implementation, userspace), and the Linux kernel's own WireGuard-protocol module. 15-second iperf3 runs, three samples each, both Rust cores pinned to one data-plane thread for the reason in the callout above.

task test:impls    # lab: ours vs boringtun vs wireguard-go vs the kernel
task test:all      # lab: single-hop, multi-hop, and the comparison

The router run isn't a task target yet — it's fv started by hand in a RouterOS container on each box, against RouterOS's built-in WireGuard-protocol server. Full topology and every raw row sits in our own docs/benchmarks/results.csv.

The number that matters: on the router

Two independent framings agree, which is what makes the result believable rather than cherry-picked: absolute throughput favours our core 1.8×, and throughput retained against each generator's own uncontested ceiling favours it 2.2×. What the table does not say: the two generators are not the same tool, which is why both no-tunnel ceilings are printed rather than one headline ratio, and both figures would move if the sender were receiving real network traffic instead of generating it locally. Unlike the lab table below, these are single runs: the percentages in brackets are the range the CPU wandered over during that run, not a spread across repeated trials.

The lab: three userspace cores, one kernel, equalised threads

CPU cost is where the lab earns its place in this post — it is the one axis the router table can't show, because a router's in-kernel comparator gives you no CPU breakdown to compare against.

Where we did not win

Two places, both explained rather than buried in the CSV:

User-mode CPU, in the lab. 0.347 s/Gbit against the cheaper Rust comparator's 0.290 — the one column in that table where it's ahead of us. Our reading is that it's bought, not wasted: our core does TUN checksum offload in both directions, which moves work out of the kernel and into userspace CPU accounted as user time — and the system-CPU column, where that trade is repaid, is comfortably ahead (0.549 vs 0.787). Total CPU still wins. The user/sys split above is shown specifically so this trade is visible rather than asserted.

Round-trip time, on the router. 1.22 ms against the kernel tunnel's 0.92 ms — the one hardware axis where in-kernel wins outright, for the ordinary reason that our path costs a container's veth hop and a userspace scheduling turn that an in-kernel path doesn't pay. Interestingly the lab shows the opposite ordering on p99 latency (0.94 ms vs the kernel's 0.96 ms) — we don't read anything into that beyond "different environments, different bottlenecks," and say so rather than picking whichever number flatters more.

What's fresh and what's inherited

Every figure above carries a date, and the dates matter: the router table is 2026-09-01, the lab table 2026-08-31. Since then, a coordinated major-version bump moved the underlying crypto crates our data plane calls on every packet — chacha20poly1305, blake2, curve25519-dalek — and we have not re-run the full lab or router comparison against that tree. The numbers above are real and were measured carefully, but they are six days old as of publishing, on a tree that predates that bump. We're publishing them with that stated rather than silently, and we'll date a re-run when we have one. The router figure also isn't pinned to a specific commit in our own records — read "2026-09-01" as "early September," not as tied to one optimization change.

What we could re-run today, on this machine, without a router or a VM lab, is the part of the path the crypto bump touched most directly: encapsulate and decapsulate cost, isolated from every socket, TUN and threading concern above.

Re-run 2026-09-05, Apple M3 Max, cargo bench -p forestvpn-quantum --bench datapath_benches -- --measurement-time 3 --warm-up-time 1, criterion 0.8.2, 100 samples per point. No socket, no TUN, no threads — one completed handshake, then encapsulate/decapsulate in a loop. Results are mixed by direction and packet size, all within a few percent either way, which is exactly what we'd expect: our core forked this path from its upstream, and both now sit on the same generation of RustCrypto primitives. This is not a throughput claim — it says the forked crypto path itself costs about what upstream's does, within noise, which is what makes the lab and router wins above worth trusting: they come from elsewhere in the stack (syscall batching, offload, threading), not from a faster cipher.

Reproducing

task test:impls          # lab: ours vs boringtun vs wireguard-go vs the kernel
task test:all            # lab: single-hop, multi-hop, and the multihop cost table
cargo bench -p forestvpn-quantum --bench datapath_benches -- --measurement-time 3 --warm-up-time 1

The router run is manual today: fv in a RouterOS container on each box, against RouterOS's own WireGuard-protocol server.

Being straight about what that buys you: the harness and the raw results.csv behind every table here live in our tree, which is not public today. Read the commands above as an exact statement of what we ran, not as something you can clone and repeat. The claim least dependent on us is also the one most worth checking — boringtun and wireguard-go are public, iperf3 is public, and the thread-count trap at the top of this post (one flag moving a comparator between roughly 789 and 1427 Mbit/s on a single host) is reproducible by anyone with those three and none of our code in the loop. If it holds up for you, the equalised rows above are the ones to argue with. If it doesn't, we want to hear about it.