Throughput per CPU cycle on real routers
A lab number and a hardware number rarely agree. We measured both, kept the intervals, disclosed the trap that gave a wrong result, and said where we lost.
On this page (7)
On two physical routers, moving real traffic over a real gigabit LAN, our tunnel core pushed 122 Mbit/s where the router's own kernel tunnel implementation pushed 66.8 Mbit/s — on the same sender, at less CPU per Mbit. Neither figure is the whole story on its own: the two arms had to use different traffic generators, because no single generator runs on both sides of that comparison. So the ratio to argue with is each arm against its own no-tunnel ceiling, which is in the table below and favours us by more, not less. A four-VM lab run on a laptop (2026-08-31, now history; see the update below) says something faster in absolute terms and, on one axis, something less flattering. We trust the router number more, because a router is the only one of the two that a customer's traffic actually crosses.
That is the finding worth leading with, and also the reason this post has two result sections instead of one: the lab and the hardware disagree about which axis matters, and papering over that disagreement would cost more credibility than either number gains.
Read this before the numbers. Our data-plane defaults to one thread. The comparator we ran alongside it in the lab defaults to four. Left unequalised, that flag alone swung its throughput from ~789 to 1427 Mbit/s in the laptop lab of 2026-08-31 — a gap in defaults, not in datapaths — and it has already produced one wrong published number here. On dedicated x86 on 2026-10-17, boringtun at its default four threads carried 1317 Mbit/s (see the update below). Every lab figure below pins both Rust cores to one thread. The Go and the kernel implementations in that table ran at their own defaults, which use more than one — if anything that flatters them against our single thread. The as-shipped-default gap is shown separately, once, so it isn't hidden by omission either.
What was measured, and on what
Two independent environments, because a benchmark is only honest within the environment it ran in — an iperf3 number and a bandwidth-test number are not one number wearing two names, and a VM sharing a laptop's cores is not a router.
Hardware — two MikroTik routers, one gigabit LAN, 2026-09-01. RB3011UiAS (ARMv7, 2×1400 MHz) as sender, cAP ax (ARM64, 4×864 MHz) as receiver, RouterOS 7.24.1 on both. Our core ran as fv, in a RouterOS container, in userspace, against the router's own in-kernel WireGuard-protocol implementation — the only apples-to-apples comparator available on that hardware, since it is the one alternative that ships in RouterOS itself. Two generators, because none crosses implementations cleanly: dd | nc for our container run, RouterOS's own bandwidth-test for the kernel run — each carries its own no-tunnel ceiling as a control, so a reader can sanity-check the ceiling before trusting the ratio.
Lab — four aarch64 Debian guests under QEMU/HVF on one Apple-silicon host, 2026-08-31. Same keypairs, addresses, ports, MTU and interface bring-up across every binary under test: our core, boringtun 0.7 (Cloudflare's userspace WireGuard implementation in Rust), wireguard-go (the WireGuard-protocol reference implementation, userspace), and the Linux kernel's own WireGuard-protocol module. 15-second iperf3 runs, three samples each, both Rust cores pinned to one data-plane thread for the reason in the callout above.
task test:impls # lab: ours vs boringtun vs wireguard-go vs the kernel
task test:all # lab: single-hop, multi-hop, and the comparisonThe router run isn't a task target yet — it's fv started by hand in a RouterOS container on each box, against RouterOS's built-in WireGuard-protocol server. Full topology and every raw row sits in our own docs/benchmarks/results.csv.
The number that matters: on the router
| Measure | our core (userspace) | router's own kernel tunnel |
|---|---|---|
| throughput | 122 Mbit/s | 66.8 Mbit/s |
| CPU, sender | 65% (63–67) | 74% (70–79) |
| CPU, receiver | 18% (10–22) | 20% (18–21) |
| CPU per Mbit | 0.53 %/Mbit | 1.10 %/Mbit |
| RTT | 1.22 ms | 0.92 ms |
| resident memory (sender / receiver) | 3.0 / 3.1 MB | in-kernel, not separable |
| generator | dd | nc | bandwidth-test |
| that generator's no-tunnel ceiling | 187 Mbit/s | 220.5 Mbit/s |
| retained against its own ceiling | 65% | 30% |
Bar length ∝ Mbit/s, axis 0–140. Same RB3011 sender in both runs; it is the CPU ceiling throughout, at 65–79% while the receiving cAP ax idles near 20%.
Two independent framings agree, which is what makes the result believable rather than cherry-picked: absolute throughput favours our core 1.8×, and throughput retained against each generator's own uncontested ceiling favours it 2.2×. What the table does not say: the two generators are not the same tool, which is why both no-tunnel ceilings are printed rather than one headline ratio, and both figures would move if the sender were receiving real network traffic instead of generating it locally. Unlike the lab table below, these are single runs: the percentages in brackets are the range the CPU wandered over during that run, not a spread across repeated trials.
The lab: three userspace cores, one kernel, equalised threads
2026-08-31, four VMs on one laptop; history, see the update.
| core | kind | TCP Mbit/s (range, n=3) | peak memory | p99 latency |
|---|---|---|---|---|
| ours, 1 thread | userspace, Rust | 1535 (1506–1554) | 3.5 MB | 0.94 ms |
| boringtun, 1 thread | userspace, Rust | 1427 (1346–1479) | 5.3 MB | 1.15 ms |
| wireguard-go, default | userspace, Go | 1228 (1221–1239) | 34.8 MB | 1.39 ms |
| kernel WireGuard, default | kernel, C | 1376 (1366–1386) | — (kernel, not separable) | 0.96 ms |
As shipped, unequalised, in the 2026-08-31 laptop lab: our core defaults to 1 thread and ran 1535 Mbit/s; boringtun defaults to 4 threads and ran ~789 Mbit/s at that setting. Reporting only the default-vs-default number would repeat the same mistake in the flattering direction. Both numbers are real; only one of them isolates the datapath from the thread count.
CPU cost is where the lab earns its place in this post — it is the one axis the router table can't show, because a router's in-kernel comparator gives you no CPU breakdown to compare against.
Lower is better. Axis 0–2.5 s/Gbit, linear. Both segments stack left to right; the split is where the next section's loss shows up.
Log scale, 1–50 MB, stated because it isn't the honest default: on a linear axis wireguard-go's 34.8 MB would flatten the other two bars to hairlines.
Where we did not win
Three places, all explained rather than buried in the CSV:
Throughput on dedicated Linux servers, 2026-10-16 and 2026-10-17. Kernel WireGuard and wireguard-go carried about 1.9 to 2.4 times Quantum's single-stream throughput at its best thread count (2026-10-17), and about three times at one thread (2026-10-16). See the update below.
User-mode CPU, in the 2026-08-31 lab. 0.347 s/Gbit against the cheaper Rust comparator's 0.290 — the one column in that table where it's ahead of us. Our reading is that it's bought, not wasted: our core does TUN checksum offload in both directions, which moves work out of the kernel and into userspace CPU accounted as user time — and the system-CPU column, where that trade is repaid, is comfortably ahead (0.549 vs 0.787). Total CPU still wins. The user/sys split above is shown specifically so this trade is visible rather than asserted.
Round-trip time, on the router. 1.22 ms against the kernel tunnel's 0.92 ms — the one hardware axis where in-kernel wins outright, for the ordinary reason that our path costs a container's veth hop and a userspace scheduling turn that an in-kernel path doesn't pay. Interestingly the 2026-08-31 laptop lab shows the opposite ordering on p99 latency (0.94 ms vs the kernel's 0.96 ms) — we don't read anything into that beyond "different environments, different bottlenecks," and say so rather than picking whichever number flatters more.
Update, 2026-10-16 and 2026-10-17: Linux servers
The laptop lab (2026-08-31) and the router run (2026-09-01) above are dated history. On 2026-10-16 we measured four Linux server platforms at one thread; on the dedicated types kernel WireGuard carried about three times Quantum's throughput. On 2026-10-17 we swept thread counts on the two dedicated types: the Linux daemon run as fv run --threads N (its default is 1; these are not the apps), extra key exchange off, boringtun at its default four threads, one TCP stream, median of 5 runs. At Quantum's best thread count (4 threads on x86, 8 on arm64), kernel WireGuard and wireguard-go still carried about 1.9 to 2.4 times its throughput (2026-10-17), so the lab ranking above does not hold on servers. Against boringtun, both at four threads on x86, Quantum carried 1.6 times as much (2026-10-17), and against wireguard-go it used about half the CPU cycles per byte while wireguard-go carried about twice the traffic. Method, spreads and every loss: the thread sweep.
Which figures are fresh, and which are dated
Every figure above carries a date: the router table is 2026-09-01, the lab table 2026-08-31. Both are history, measured on a tree from before a coordinated major-version bump of the crypto crates our data plane calls on every packet — chacha20poly1305, blake2, curve25519-dalek. The router figure also isn't pinned to a specific commit — read “2026-09-01” as “early September”. The current Linux-server measurement is the update above.
What we could re-run today, on this machine, without a router or a VM lab, is the part of the path the crypto bump touched most directly: encapsulate and decapsulate cost, isolated from every socket, TUN and threading concern above.
| operation, packet size | our core | boringtun 0.7.1 |
|---|---|---|
| encapsulate, 64 B | 219.2 ns | 218.7 ns |
| encapsulate, 512 B | 436.4 ns | 442.5 ns |
| encapsulate, 1420 B | 938.1 ns | 968.3 ns |
| decapsulate, 64 B | 238.8 ns | 226.7 ns |
| decapsulate, 512 B | 562.1 ns | 539.2 ns |
| decapsulate, 1420 B | 1099.5 ns | 1069.4 ns |
Re-run 2026-09-05, Apple M3 Max, cargo bench -p forestvpn-quantum --bench datapath_benches -- --measurement-time 3 --warm-up-time 1, criterion 0.8.2, 100 samples per point. No socket, no TUN, no threads — one completed handshake, then encapsulate/decapsulate in a loop. Results are mixed by direction and packet size, all within a few percent either way, which is exactly what we'd expect: both cores seal and open every data packet with the same ChaCha20-Poly1305 code, from the ring crate. This is not a throughput claim — it says our crypto path costs about what boringtun's does, within noise, which is what makes the router wins above worth trusting: they come from elsewhere in the stack (syscall batching, offload, threading), not from a faster cipher.
Reproducing
task test:impls # lab: ours vs boringtun vs wireguard-go vs the kernel
task test:all # lab: single-hop, multi-hop, and the multihop cost table
cargo bench -p forestvpn-quantum --bench datapath_benches -- --measurement-time 3 --warm-up-time 1The router run is manual today: fv in a RouterOS container on each box, against RouterOS's own WireGuard-protocol server.
Being straight about what that buys you: the harness and the raw results.csv behind every table here live in our tree, which is not public today. Read the commands above as an exact statement of what we ran, not as something you can clone and repeat. The claim least dependent on us is also the one most worth checking — boringtun and wireguard-go are public, iperf3 is public, and the thread-count trap at the top of this post (in the 2026-08-31 laptop lab, one flag moving a comparator between roughly 789 and 1427 Mbit/s on a single host) is reproducible by anyone with those three and none of our code in the loop. If it holds up for you, the equalised rows above are the ones to argue with. If it doesn't, we want to hear about it.