On 8 September 2026 one flag on a virtual machine’s network card took a path from 3 Mbit/s to 509. Before we found it, we had named four causes. Each one was plausible, and every one was wrong.
Three days, four wrong causes, and a written list of what it was not.
What looked healthy
Traffic between virtual machines on different physical hosts, carrying a network tag, crawled. Nothing else said anything was wrong. Every interface on both hosts showed zero dropped packets and zero errors. There were zero checksum errors. Ping on the same path was close to perfect, 837 Mbit/s at 0.23% loss, because ping packets are never split into segments.
The only evidence was on the receiving end, and only if you knew to ask: 1,947 of 26,812 segments arriving out of order, and retransmissions. That is why it took three days.
Four causes, each measured and discarded
-
“Traffic between hosts is slow when it is routed”
The first machine placed on its own network segment, on a different host from the router, ran at under 2 Mbit/s. We moved it beside the router and wrote the move down as a workaround, because the symptom vanished without anything being fixed. The cause on file said routed. Two days later the same pair of machines measured 529 Mbit/s untagged and 3 Mbit/s tagged, and the slow path never touched a router.
Stood 2 days -
“It is the switch”
The operator cleared the switch’s rate limiting, and the full tagging configuration was read off the device. It was uniform and correct. The defect stayed.
Measured · wrong -
“It is how the bridges are built”
This estate had deliberately differed from the estate it was copied from in how tagged traffic reaches a machine, and now that difference was the thing measured to be slow. It fitted every fact. We built both designs and measured them at 4 Mbit/s and 4 Mbit/s. It was the operator’s hypothesis and the agent’s.
Built both · 4 and 4 -
“It is checksum offload”
If the network card were corrupting checksums, the receiver would count errors. It counted 0.
0 errors · wrong
The cause
Then one variable at a time, on the same two machines, 20 MB each. All offloads on: 3 Mbit/s. Two of the three offloads off, one at a time: 3 and 3. TCP segmentation offload off, alone: 509. Back on: 3. Off again: 656. That is a cause in both directions, which is the only kind worth writing down.
The mechanism, in short: the virtual machine hands the host very large packets and trusts the hardware to cut them into segments. Once a network tag is added on the way out, that cutting goes wrong and packets vanish without a counter noticing.
Two findings made it believable. Turning the same offload off on both physical hosts changed nothing; it was still 3 Mbit/s, so only the virtual machine’s setting matters. And the one fast path between machines all along, 221 Mbit/s, ran through a USB network adapter whose driver refuses offload. The one host that worked was the one that could not do the broken thing. We had read it as an unexplained outlier for hours.
What it cost
A wrong cause is cheap to produce. It becomes expensive when someone acts on it, and on the way to this one we acted on several.
- The whole local network went down. Testing the bridge-design cause, the agent built a second bridge on a network card the first bridge still owned, on two hosts at once. It had named that exact risk in its previous message. Recovery was unplugging two switch ports by hand.
- Three measurements were worthless, and every one read as a result. A transfer over a port the firewall blocks measured the firewall, not the wire. A transfer that failed had its 50 MB divided by the time it spent waiting, and printed a plausible 2 Mbit/s. It was nearly reported as a throughput.
- A comparison was called controlled when it was not. The 529-against-3 test was written up as one pair, one transport, one payload, differing only in the tag. The fast path also used a different network card and a different switch. Another confident conclusion, that every slow number involved one particular host, fell only because the operator insisted on fixing a switch port first.
And one thing went right, because it was named. Moving the monitoring machine beside the router was written down as a workaround, not a placement. So the defect kept an owner, and the moment the cause was found, the workaround stopped being forced.
The rule it bought
One measured cause, or say “unconfirmed” out loud. And write down what was eliminated, with the measurement that eliminated it.
This is a working-with-the-agent lesson, not a networking one, because of how an agent fails. It produces plausible causes without limit and at speed, and each arrives sounding as sure as the last. That is a hazard: a confident wrong diagnosis costs more than none, because it gets acted on. An agent also starts every session with no memory of the last. Without the list of what was measured and discarded, the next session will reason its way to the same four causes, each as convincing as it was the first time. So the record of this defect lists the four wrong causes, with the instruction not to re-derive them.
The same discipline catches the measurements themselves. A number is not a measurement until the command behind it succeeded, and a comparison is not controlled until you have named everything that differs.
Four good-sounding causes were not four steps toward the answer. The answer came from changing one thing at a time.
None of this is new
In his 1974 commencement address at Caltech, Richard Feynman described how science began: a method for separating ideas, “try one to see if it worked, and if it didn’t work, to eliminate it”. Then the part that matters here: “The first principle is that you must not fool yourself—and you are the easiest person to fool.”
He told of a student who wanted to change one condition in someone else’s experiment without first repeating the original, and explained that repeating it first was the only way she “would know that the real difference was the thing she thought she had under control.” Our 529-against-3 comparison failed exactly that test.
What changed is not the principle. It is the rate. A person fools themselves at the pace they can think up a theory. An agent drafts theories faster than anyone can measure them, so the principle cannot rest on temperament. It has to be a rule, and the rule has to be written where the next session will read it.