2024 — present
7 min read
Finding a fault that was not where everyone was looking
In a park this size, “the internet is slow” almost never means what it says. Fibre barely adds latency; the fault is a dirty connector, a failing SFP, a switching loop, a bent cable. A method for diagnosing by layer instead of by the loudest guess.
IT Support Officer · Bright Industrial Park, Ghana
- delay fibre adds per km
- 5 µs
- where the fault usually hides
- Layer 1
- faults diagnosed by guessing
- 0
Why the fibre is almost never the fibre
Optical fibre adds about five microseconds of delay per kilometre. Across a park, that is nothing — the fibre is almost never the cause of a latency complaint, whatever the complaint says. What reaches me is a symptom: “the internet is slow,” “the ERP keeps disconnecting,” “the CCTV freezes every few seconds.” Where the fault actually sits is somewhere else entirely.
That somewhere is usually the physical layer — a dirty connector, a failing transceiver, a fibre bent too tight or cut by an excavator, a splice that drifts as it heats. Sometimes it is a layer higher: a duplex mismatch, a saturated uplink, a switching loop, a port on the wrong VLAN. The fault is rarely where the complaint points, and the work is to find it without being led there by the loudest guess.
Working up from the bottom
The discipline is to diagnose by layer, from the physical layer up, and to measure at each step rather than suspect. The first question is scope — one PC, one building, or the whole park — because that alone rules out half the possibilities before a single tool comes out of the bag.
After that it is measurement, not opinion. A continuous ping to the gateway separates a local fault from a WAN one and shows whether the loss is steady or spikes under load. The switch's own interface counters — CRC errors, receive drops, interface flaps — point straight at Layer 1 the moment they start climbing. Optical power tells the rest: a healthy receive level sits around -8 to -15 dBm, and a reading of -20 dBm is a light budget quietly falling apart. An iperf run across the LAN separates a bandwidth ceiling from a latency problem. Each measurement rules a layer in or out, so by the time I actually touch the fibre I already know it is the fibre.
The fault a thirty-second clean fixes
One building reports the ERP dropping and the CCTV stuttering. A ping to the gateway runs at one or two milliseconds, then spikes to 350 and times out, then returns to normal — the signature of a link shedding frames, not an application failing. The switch confirms it: CRC errors and receive drops climbing on that one port, which puts the fault at Layer 1 and takes routing and VLANs off the table.
Optical power reads -20 dBm on receive, well under the healthy window. The cause turns out to be almost boringly common — a dirty LC connector, a speck of dust on a fibre end face a fraction of a hair wide, enough to scatter the light. A one-click cleaner, an inspection under the scope, reconnect, and receive power comes back to -10 dBm with the latency gone. It looked like a software bug for a week and it was a smudge.
When you have to find the break
Not every fault is that gentle. A whole factory goes dark at once — link down, no light on the transceiver at all. That is not degradation, it is a break, and the only question is where. A visual fault locator, a red laser sent down the fibre, shows it glowing halfway along the conduit, where an excavator cutting a road trench had gone through the cable. A fusion splice, a retest, and the factory is back.
The subtler version is the splice that holds in the morning and fails in the afternoon heat, or a fibre tied hard around a metal pole so its receive power sags and recovers as it flexes. Those never show as a clean break; they show as loss, and an OTDR is what finds them — a 2.4 dB spike at 320 metres where the trace should read a rounding error, pointing at the exact splice to redo.
The loop that takes down everything
The faults that do not stay in one building are the worst. The whole park goes unusable at once, every switch's CPU pinned near the ceiling, the network drowning in broadcast traffic. That pattern has one usual cause — a loop. Someone has patched two switch ports into each other, often while innocently tidying a cabinet, and the network is now forwarding the same frames around a ring with no end.
Spanning tree gives it away as constant topology changes. Pulling the loop cable ends it, but the real fix is the guard that should have kept one patch lead from becoming a park-wide outage: RSTP with BPDU guard and loop protection, so the switch shuts the offending port instead of letting it take down eight thousand people's network.
What I would do differently
Almost every fault here is visible before a user feels it. A connector does not fail in an instant — its receive power drifts down over days. A splice that gives out in the heat has been marginal for weeks. The single biggest improvement is to monitor the physical layer: optical power and interface error counters, with an alert when a link crosses a threshold, so a port sliding toward -18 dBm raises a ticket before it drops a frame.
That turns the whole thing from reactive to proactive — instead of diagnosing an outage after the CCTV freezes, you replace a connector on a Tuesday because a graph told you to. The method in this piece is how you find a fault once it has happened; the point of the monitoring is to need it less often.