MMO ISP route latency comparison mtr tracert

You ping the MMO login gateway right before a raid, latency reads 38 ms. The fight starts, your screen freezes for 1.8 seconds, you blink-die in front of the boss. You alt-tab out and run ping again. 38 ms. Nothing on the game's own status page is yellow. You have no data, only a recent corpse and a vague sense that something between your modem and the data centre dropped the ball for two seconds and then forgot it ever happened.

Single-shot tools (ping, one-pass tracert) tell you what the path looked like the moment you ran them. They cannot tell you whether one specific hop spends 2 percent of every hour above 200 ms. The signal you need is per-hop latency over a long enough window that the slow moments stop hiding inside noise. That is what mtr already does on a single invocation, and what you can build on top of by running mtr on a 30-minute schedule against the same gateway IP for a few days.

This article walks through a small Python CLI you can clone and run today. It wraps mtr --json, writes each snapshot as one JSONL line, aggregates per-hop median and p95 across many snapshots, and produces a side-by-side comparison between two route labels (your ISP versus a VPN tunnel, or your old ISP versus your new one). The closing piece flags hops that look like bufferbloat candidates, the kind of intermediate router whose queue depth spikes under load while its baseline ping looks fine. Public source is at https://github.com/vytharion/mmo-isp-route-latency-mtr-snapshots and every step below cites the commit you should check out to follow along.

Step 1: Wrap `mtr --json` (commit d69baf1)

The first decision is which tool to use. The three usual suspects each answer a different question:

Tool	What it measures	Time shape	Hop visibility
`ping`	Round-trip time to one destination	One sample per packet	None, destination only
`tracert` (Windows) / `traceroute` (Unix)	First-packet RTT per hop	One packet per hop, no repeats	Yes, but one-shot
`mtr`	RTT plus loss plus jitter per hop, repeated	N packets per hop, default 10	Yes, with statistics

mtr was written in 1997 specifically to fix the problem where a single traceroute shows one hop with 250 ms and you cannot tell whether that hop is genuinely slow or just rate-limiting one ICMP packet. Upstream lives at https://github.com/traviscross/mtr and most package managers ship a recent build. The flag we care about is --json, which emits a single document with a report.hubs array, one row per hop.

A typical document for six hops to 1.1.1.1 looks like this:

{
  "report": {
    "hubs": [
      {"count": 1, "host": "192.168.1.1", "Loss%": 0.0, "Snt": 100, "Avg": 1.4, "StDev": 0.4},
      {"count": 2, "host": "10.20.0.1", "Loss%": 0.0, "Snt": 100, "Avg": 8.5, "StDev": 0.7},
      {"count": 3, "host": "isp-edge1.example.net", "Loss%": 0.0, "Snt": 100, "Avg": 13.0, "StDev": 1.8},
      {"count": 4, "host": "isp-core1.example.net", "Loss%": 2.0, "Snt": 100, "Avg": 41.2, "StDev": 32.1},
      {"count": 5, "host": "ix-peer.example.net", "Loss%": 0.0, "Snt": 100, "Avg": 31.5, "StDev": 1.2},
      {"count": 6, "host": "1.1.1.1", "Loss%": 0.0, "Snt": 100, "Avg": 33.4, "StDev": 0.6}
    ]
  }
}

Hop 4 already shows the symptom we care about: 2 percent loss, 41 ms average, and 32 ms of standard deviation on a route where every other hop has sub-2 ms stdev. That is what an oversubscribed peering point looks like on paper, well before anyone calls support.

The Python wrapper takes the raw JSON, parses each hub row into a frozen dataclass, and stamps a UTC timestamp on the whole thing. Pure parsing lives in parse_mtr_json; the subprocess call lives in run_mtr. Splitting them means tests replay captured JSON fixtures and the network code is exercised only by integration runs.

def parse_mtr_json(raw: str, target: str, label: str, now: datetime | None = None) -> Snapshot:
    doc = json.loads(raw)
    report = doc.get("report", {})
    raw_hops = report.get("hubs", [])
    hops = [
        Hop(
            hop_no=int(h["count"]),
            host=str(h.get("host", "???")),
            loss_pct=float(h.get("Loss%", 0.0)),
            sent=int(h.get("Snt", 0)),
            last_ms=float(h.get("Last", 0.0)),
            avg_ms=float(h.get("Avg", 0.0)),
            best_ms=float(h.get("Best", 0.0)),
            worst_ms=float(h.get("Wrst", 0.0)),
            stdev_ms=float(h.get("StDev", 0.0)),
        )
        for h in raw_hops
    ]
    ts = (now or datetime.now(timezone.utc)).isoformat()
    return Snapshot(timestamp=ts, target=target, label=label, hops=hops)

Run one probe with uv run mtr-snap probe --target <gateway-ip> --label baseline --count 100. On Linux you need sudo because raw ICMP sockets require capabilities, on macOS the Homebrew build sets the setuid bit during install. The captured JSON lands at snapshots/baseline.jsonl as one line.

Step 2: Snapshot scheduler (commit 779bb0f)

Running one probe every 30 minutes for a few days is the cheapest way to catch a hop that only misbehaves under load. 30 minutes is the sweet spot for three reasons. One minute floods your ISP's ICMP rate limit and produces a noisier dataset. One hour is too sparse to catch sub-hour congestion windows; peak evening hours on a residential link tend to be 30 to 60 minutes long. 30 minutes also matches the default sampling cadence of public bufferbloat datasets at https://www.bufferbloat.net/projects/, so your numbers stay comparable to community traces.

The persistence format is JSONL, one snapshot per line. JSONL beats SQLite for this workload because the file is append-only, survives partial writes, and replays trivially with cat snapshots/baseline.jsonl | jq. No schema migration, no DB lock, no operator intervention when the loop stops and starts.

async def run_schedule(
    target: str,
    label: str,
    interval_s: float,
    out_dir: Path,
    stop_event: asyncio.Event,
    probe: Callable[[str, str], Snapshot] = lambda t, l: run_mtr(t, l),
    count_limit: int | None = None,
) -> int:
    jsonl_path = out_dir / f"{label}.jsonl"
    written = 0
    while not stop_event.is_set():
        snapshot = await asyncio.to_thread(probe, target, label)
        append_snapshot(snapshot, jsonl_path)
        written += 1
        if count_limit is not None and written >= count_limit:
            return written
        try:
            await asyncio.wait_for(stop_event.wait(), timeout=interval_s)
        except asyncio.TimeoutError:
            continue
    return written

Two patterns matter here. First, asyncio.to_thread(probe, ...) runs the blocking mtr subprocess on a worker thread so the asyncio event loop is free to wait on stop_event concurrently. If we called subprocess.run directly the loop would block for 100 seconds per probe and any future caller that wants to schedule two labels in parallel would queue them serially. Second, the cancellation pattern is asyncio.wait_for(stop_event.wait(), timeout=interval_s) instead of asyncio.sleep(interval_s). The two look similar but only the first stops mid-interval when the operator sends Ctrl-C. A sleep swallows the cancellation until the full interval elapses, which during a 30-minute cycle would feel like the program hung.

count_limit is a safety cap used only in tests. Real runs leave it None and let the operator stop the schedule with a signal. Verify with uv run mtr-snap schedule --target <gateway-ip> --label baseline --interval 1800. The loop writes one JSON line every 30 minutes. Leave it running for 48 hours minimum, ideally a full week, before you trust the aggregate.

Step 3: Per-hop aggregation (commit 44326da)

Sixteen snapshots over eight hours produce a table the aggregator turns into one row per hop. The grouping key is hop_no, the integer index, not the host name. This matters because the host at, say, hop 5 occasionally flaps between two upstream peers when one of them is saturating. If you grouped by host, the same physical position in the route would show up as two separate aggregate rows and you would lose the per-position view that lets you reason about "the 5th hop in my route".

The summary fields per hop are median latency, p95 latency, median stdev (jitter proxy), packet loss percentage, and the ordered list of hosts seen at that hop number. Median resists single-snapshot outliers in a way that mean does not. A 30-minute schedule sometimes catches an unrelated burst (DHCP renewal, ARP storm on the local segment, switch port flap) and the mean for that hop spikes by 80 ms while the median moves 2 ms. p95 is the complement, the value the worst 5 percent of snapshots exceed, which is where the gameplay-relevant tail lives.

def aggregate_snapshots(snapshots: list[dict]) -> list[HopAggregate]:
    by_hop: dict[int, list[dict]] = {}
    for snap in snapshots:
        for hop in snap.get("hops", []):
            by_hop.setdefault(int(hop["hop_no"]), []).append(hop)
    result: list[HopAggregate] = []
    for hop_no in sorted(by_hop):
        rows = by_hop[hop_no]
        hosts_seen: list[str] = []
        for r in rows:
            host = str(r.get("host", "???"))
            if host not in hosts_seen:
                hosts_seen.append(host)
        if all(r.get("host", "???") == "???" for r in rows):
            continue
        avg_values = [float(r.get("avg_ms", 0.0)) for r in rows]
        stdev_values = [float(r.get("stdev_ms", 0.0)) for r in rows]
        loss_values = [float(r.get("loss_pct", 0.0)) for r in rows]
        result.append(
            HopAggregate(
                hop_no=hop_no,
                hosts_seen=tuple(hosts_seen),
                sample_count=len(rows),
                median_avg_ms=statistics.median(avg_values),
                p95_avg_ms=percentile(avg_values, 95.0),
                median_stdev_ms=statistics.median(stdev_values),
                loss_pct=statistics.mean(loss_values),
            )
        )
    return result

The percentile uses the nearest-rank definition rather than linear interpolation. With small N (16 to 30 samples is typical for a multi-day schedule) the nearest-rank result is also a real observed value, which makes the table easier to defend in conversation with ISP support. The fully-silent hop skip is necessary because some intermediate routers drop all ICMP TTL-exceeded responses; aggregating them produces a row of zeros that pollutes the comparison table later.

Step 4: Two-label comparison report (commit 9ff9579)

The comparison is the part operators actually look at. You run two schedulers in parallel, one against the gateway via your ISP's default route, one via a VPN tunnel, both writing to different JSONL files. After a day or two you ask the report whether the VPN actually helped.

The output is a small terminal table:

label          hops  end-to-end ms    worst hop
---------------------------------------------------
baseline          6           33.0  #4 41.2ms
vpn-tunnel        8           29.5  #5 18.4ms

verdict: alt_faster (delta -3.5 ms)

Three things are worth pointing out. The hop count almost always differs between routes; a VPN tunnel adds the VPN provider's ingress and egress hops, so an 8-hop VPN path is normal when the baseline is 6 hops. The end-to-end median is the destination hop's median, which is what a player on the game's network stack measures end-to-end. The verdict applies a 5 ms wash threshold; anything smaller than that on a wired connection is well within snapshot-to-snapshot noise, and reporting "alt_faster" when the delta is 2 ms would mislead.

def compare_routes(
    baseline_hops: list[HopAggregate],
    alt_hops: list[HopAggregate],
    baseline_label: str = "baseline",
    alt_label: str = "alt",
    wash_threshold_ms: float = 5.0,
) -> ComparisonReport:
    a = summarise_route(baseline_label, baseline_hops)
    b = summarise_route(alt_label, alt_hops)
    delta = b.end_to_end_median_ms - a.end_to_end_median_ms
    if abs(delta) < wash_threshold_ms:
        verdict = "wash"
    elif delta < 0:
        verdict = "alt_faster"
    else:
        verdict = "alt_slower"
    return ComparisonReport(routes=[a, b], end_to_end_delta_ms=delta, verdict=verdict)

The interesting failure mode is alt_slower when the worst hop's host name is the same on both routes. That means the VPN is sending your traffic right back through the same congested ISP peering point, just with two extra hops of tunnel overhead. Three out of every four "gaming VPN" trials run against an MMO gateway in a residential setup have shown that shape: the marketing copy describes a parallel backbone but the routing tables run through the same internet exchange as the baseline.

Step 5: Bufferbloat candidate detection (commit 94bbd80)

The last piece flags intermediate hops that look like an oversubscribed queue. Bufferbloat is the phenomenon where a router with a fat output queue under load produces low average latency (the queue absorbs short bursts) yet a fat tail (sustained load fills the queue and adds 100 to 400 ms of delay per packet). The background measurements that named the phenomenon live at https://www.bufferbloat.net/projects/ along with the original lab traces.

Two signals must hold for a hop to qualify. First, the hop's median stdev is at least 3x the median stdev across all hops in the route. A noisy hop relative to its neighbours suggests queue depth is being added there. Second, the hop's p95 latency is at least 2x its median latency. The hop's tail is much fatter than its centre, the classic bufferbloat shape.

def detect_bufferbloat_candidates(
    hops: list[HopAggregate],
    jitter_ratio_threshold: float = 3.0,
    spike_ratio_threshold: float = 2.0,
    min_median_stdev_ms: float = 2.0,
) -> list[BufferbloatCandidate]:
    if not hops:
        return []
    route_median_stdev = statistics.median(h.median_stdev_ms for h in hops)
    if route_median_stdev <= 0:
        route_median_stdev = 1e-6
    candidates: list[BufferbloatCandidate] = []
    for h in hops:
        if h.median_stdev_ms < min_median_stdev_ms:
            continue
        jitter_ratio = h.median_stdev_ms / route_median_stdev
        spike_ratio = h.p95_avg_ms / h.median_avg_ms if h.median_avg_ms > 0 else 0.0
        if jitter_ratio < jitter_ratio_threshold:
            continue
        if spike_ratio < spike_ratio_threshold:
            continue
        why = (
            f"jitter {h.median_stdev_ms:.1f}ms is {jitter_ratio:.1f}x route median; "
            f"p95 {h.p95_avg_ms:.1f}ms is {spike_ratio:.1f}x median"
        )
        candidates.append(
            BufferbloatCandidate(
                hop_no=h.hop_no,
                host=h.hosts_seen[0] if h.hosts_seen else "???",
                median_ms=h.median_avg_ms,
                p95_ms=h.p95_avg_ms,
                jitter_ratio=jitter_ratio,
                spike_ratio=spike_ratio,
                why=why,
            )
        )
    return candidates

The min_median_stdev_ms floor at 2 ms keeps the local home gateway out of the list. A consumer-grade router will have 0.5 ms stdev across 100 packets on a quiet network, so the jitter-ratio calculation can produce a 5x ratio against another hop with 0.1 ms stdev, yet the absolute number is too small to matter. Filtering by absolute stdev first avoids the trap.

The destination hop stays in the candidate pool because if the bufferbloat lives at the last mile to the game gateway, that IS the latency a player feels. A common scenario in practice: a regional ISP peers with the game publisher through a single saturated link, and the destination hop's stdev jumps from 0.6 ms during off-peak hours to 28 ms during the 21:00 to 23:00 raid window.

Sample output once detection is wired into the CLI:

bufferbloat candidates:
  hop 4 (isp-core1.example.net): jitter 32.1ms is 17.8x route median; p95 180.5ms is 4.4x median

That row is what you screenshot and forward to ISP support. With the timestamp range from the schedule and the per-hop median, it is data they can take to their NOC, not a vague complaint about "the internet feels laggy when I play".

Repository

Full source at https://github.com/vytharion/mmo-isp-route-latency-mtr-snapshots.

Step 0 (scaffold) , dd8955a , pyproject, gitignore, README, smoke tests
Step 1 (mtr wrapper) , d69baf1 , parse mtr --json into typed dataclasses
Step 2 (scheduler) , 779bb0f , asyncio 30-minute loop with JSONL append
Step 3 (aggregation) , 44326da , median, p95, route-flap detection
Step 4 (compare) , 9ff9579 , two-label side-by-side report with wash threshold
Step 5 (bufferbloat) , 94bbd80 , jitter and spike ratio thresholds, CLI wiring

The tree passes uv run pytest -q at every commit. To replay a step: git checkout <sha> && uv run pytest -q. 22 tests green throughout.

Closing

Clone the repo, point one scheduler at the game gateway IP your client logs the first time you connect (the login server, not your raid party), and let it run for at least 48 hours. If you suspect a VPN will help, run a second label in parallel for the same window. The numbers either justify the subscription or save you the cash.

The scope is intentionally narrow. This tool talks to public-facing gateway IPs over ICMP, the same protocol an ISP support engineer would use to diagnose your line. It never touches the game process, the game client's network stack, or any in-game packet. If the publisher in question has a stricter stance on ICMP probes than the rest of the industry, respect it and stop. Network diagnostics is a publisher-permitted activity precisely because it stays outside the game; keep it that way and the data you collect will be useful in the conversations where you actually need it.

Step 1: Wrap mtr --json (commit d69baf1)