`vat(1)`

showed that the bulk of the
packet loss recurred periodically, with a period around 70-75 seconds.
The cyclic loss varies in intensity, but persists from day to day, week to week.
Where was this loss coming from, and why was it so neatly periodic?
The main tool for examining the state of the multicast Internet is
`mtrace(1)`

.

It was used here in the form

and left to run for a while.mtrace -lNASA-MBone-sourcesome-remote-receivermulticast-address

`mtrace -l`

traces out the path from
the source to the receiver (assumed to be a member of the given
multicast group, i.e. to be currently listening in on that MBone broadcast)
and gathers path and packet-loss statistics at 10-second intervals.
These statistics are plotted below.
For each graph, there's a table of MBone routers in order from source to receiver (hop 0 being the source, etc.). On the graph itself, the various MBone routers are ordered by the range of measured variation in the packet-loss rate, with the widest-swinging MBone router being first.

- NASA -> DelMarva
- NASA -> Denmark
- NASA -> UMinn
- NASA -> ISI
- These first four all show periodic loss,
- all point to FIXW-MBONE as the loss point, and
- all route from FIXW-MBONE to dec3800-X-fddi-Y-SanFrancisco.mci.net.

- NASA -> LBL
- NASA -> SDSC
- These two both show very low loss.
- For both, though FIXW-MBONE is in the path, MCI isn't!

- NASA -> Worldcom
- Wierd path flapping -- note that there are 6 graphs.
- some paths low-loss (nasa-worldcom-2?) (via BBNPLANET)
- some periodic-loss (nasa-worldcom, nasa-worldcom-6) (both point to FIXW-MBONE, both via MCI)
- some lossy but nonperiodic (nasa-worldcom-5)

Please send questions or comments to Stuart Levy, slevy@geom.umn.edu, Geometry Center, University of Minnesota.

mdecode -s -h mtrace-output >> stuff.html

for each mtrace file in turn. The table-of-contents header was added by hand.

Nodes, with loss variation (percent over mtrace -l's 10-second intervals):

Hop 0 loss 0.. 0% 224.0.1.32 Hop 1 loss -5.. 5% arcbone.arc.nasa.gov Hop 2 loss -1.. 2% ARC-MBONE.NSN.NASA.GOV Hop 3 loss 0..20% FIXW-MBONE.NSN.NASA.GOV Hop 4 loss 0.. 4% dec3800-2-fddi-1.SanFrancisco.mci.net Hop 5 loss 0.. 3% dec3800-1-fddi-1.Dallas.mci.net Hop 6 loss 0.. 2% dec3800-2-fddi-1.Atlanta.mci.net Hop 7 loss 0.. 2% dec3800-1-fddi-1.NorthRoyalton.mci.net Hop 8 loss -1.. 2% dec3800-1-fddi-1.WestOrange.mci.net Hop 9 loss 0.. 0% multicast.delmarva.com Hop 10 loss 0.. 0% 128.101.25.56

Nodes, with loss variation (percent over mtrace -l's 10-second intervals):

Hop 0 loss 0.. 0% 224.0.1.32 Hop 1 loss 0.. 1% arcbone.arc.nasa.gov Hop 2 loss 0.. 0% ARC-MBONE.NSN.NASA.GOV Hop 3 loss -4..30% FIXW-MBONE.NSN.NASA.GOV Hop 4 loss 0.. 2% dec3800-2-fddi-1.SanFrancisco.mci.net Hop 5 loss 0.. 1% dec3800-1-fddi-1.Dallas.mci.net Hop 6 loss 0.. 3% dec3800-2-fddi-1.Atlanta.mci.net Hop 7 loss 0.. 1% dec3800-1-fddi-1.NorthRoyalton.mci.net Hop 8 loss 0.. 1% dec3800-1-fddi-0.WestOrange.mci.net Hop 9 loss -3.. 4% schnell.ebone.net Hop 10 loss -2.. 3% stockholm.mbone.ebone.net Hop 11 loss 0.. 0% patella.uni-c.dk Hop 12 loss 0.. 0% rhodium.daimi.aau.dk Hop 13 loss 0.. 0% 128.101.25.56

Nodes, with loss variation (percent over mtrace -l's 10-second intervals):

Hop 0 loss 0.. 0% 224.0.1.32 Hop 1 loss -5.. 7% arcbone.arc.nasa.gov Hop 2 loss 0.. 0% ARC-MBONE.NSN.NASA.GOV Hop 3 loss 0..24% FIXW-MBONE.NSN.NASA.GOV Hop 4 loss -1.. 6% dec3800-2-fddi-1.SanFrancisco.mci.net Hop 5 loss -1.. 6% dec3800-2-fddi-1.Denver.mci.net Hop 6 loss 0.. 7% dec3800-1-fddi-1.WillowSprings.mci.net Hop 7 loss -1.. 4% bridge.mr.net Hop 8 loss 0.. 0% hausdorff.geom.umn.edu Hop 9 loss 0.. 0% 128.101.25.56

Nodes, with loss variation (percent over mtrace -l's 10-second intervals):

Hop 0 loss 0.. 0% 224.0.1.32 Hop 1 loss -4.. 6% arcbone.arc.nasa.gov Hop 2 loss 0.. 0% ARC-MBONE.NSN.NASA.GOV Hop 3 loss -1..25% FIXW-MBONE.NSN.NASA.GOV Hop 4 loss 0.. 2% dec3800-2-fddi-0.SanFrancisco.mci.net Hop 5 loss 0.. 1% dec3800-1-fddi-0.LosAngeles.mci.net Hop 6 loss 0.. 5% mbone.ln.net Hop 7 loss 0.. 0% mb80.isi.edu Hop 8 loss 0.. 0% 128.101.25.56

Nodes, with loss variation (percent over mtrace -l's 10-second intervals):

Hop 0 loss 0.. 0% 224.0.1.32 Hop 1 loss -9..10% arcbone.arc.nasa.gov Hop 2 loss 0.. 1% ARC-MBONE.NSN.NASA.GOV Hop 3 loss -3.. 4% FIXW-MBONE.NSN.NASA.GOV Hop 4 loss -3.. 1% mbone.nsi.nasa.gov Hop 5 loss 0.. 0% llnl-mr2.es.net Hop 6 loss 0.. 0% lbl-mr1-f.es.net Hop 7 loss 0.. 0% er1gw.lbl.gov Hop 8 loss 0.. 0% ir40gw.lbl.gov Hop 9 loss 0.. 0% 128.101.25.56

Nodes, with loss variation (percent over mtrace -l's 10-second intervals):

Hop 0 loss 0.. 0% 224.0.1.32 Hop 1 loss -15..14% arcbone.arc.nasa.gov Hop 2 loss 0.. 0% ARC-MBONE.NSN.NASA.GOV Hop 3 loss 0.. 1% FIXW-MBONE.NSN.NASA.GOV Hop 4 loss 0.. 0% nccosc-mbone.dren.net Hop 5 loss 0.. 0% 128.101.25.56

Nodes, with loss variation (percent over mtrace -l's 10-second intervals):

Hop 0 loss 0.. 0% 224.0.1.32 Hop 1 loss 0.. 0% arcbone.arc.nasa.gov Hop 2 loss 0.. 0% ARC-MBONE.NSN.NASA.GOV Hop 3 loss 0..25% FIXW-MBONE.NSN.NASA.GOV Hop 4 loss 0.. 3% dec3800-2-fddi-1.SanFrancisco.mci.net Hop 5 loss 0.. 1% dec3800-1-fddi-1.Dallas.mci.net Hop 6 loss 0.. 3% dec3800-2-fddi-1.Atlanta.mci.net Hop 7 loss -1.. 2% atlanta3-mbone1.bbnplanet.net Hop 8 loss 0.. 6% collegepk-mbone1.bbnplanet.net Hop 9 loss 0.. 2% cambridge1-mbone1.bbnplanet.net Hop 10 loss -1.. 3% MBONE3.UU.NET Hop 11 loss 0.. 0% st6.wcomcast.com Hop 12 loss 0.. 0% 128.101.25.56

Nodes, with loss variation (percent over mtrace -l's 10-second intervals):

Hop 0 loss 0.. 0% 224.0.1.32 Hop 1 loss 0.. 1% arcbone.arc.nasa.gov Hop 2 loss 0.. 0% ARC-MBONE.NSN.NASA.GOV Hop 3 loss 0.. 3% maew-mbone.nsn.nasa.gov Hop 4 loss 16..94% sanjose1-mbone1.bbnplanet.net Hop 5 loss -24.. 0% paloalto-mbone1.bbnplanet.net Hop 6 loss -7..10% collegepk-mbone1.bbnplanet.net Hop 7 loss 0.. 2% cambridge1-mbone1.bbnplanet.net Hop 8 loss -2.. 2% MBONE3.UU.NET Hop 9 loss 0.. 0% st6.wcomcast.com Hop 10 loss 0.. 0% 128.101.25.56

Nodes, with loss variation (percent over mtrace -l's 10-second intervals):

Hop 0 loss 0.. 0% cambridge1-mbone1.bbnplanet.net Hop 1 loss 0.. 0% MBONE3.UU.NET Hop 2 loss 0.. 0% st6.wcomcast.com Hop 3 loss 0.. 0% 128.101.25.56

Nodes, with loss variation (percent over mtrace -l's 10-second intervals):

Hop 0 loss 0.. 0% 224.0.1.32 Hop 1 loss -4.. 5% arcbone.arc.nasa.gov Hop 2 loss -2.. 3% ARC-MBONE.NSN.NASA.GOV Hop 3 loss 0.. 0% maew-mbone.nsn.nasa.gov Hop 4 loss 0.. 0% mbone-w.ans.net Hop 5 loss 0.. 0% mbone-e.ans.net Hop 6 loss 0.. 0% collegepk-mbone1.bbnplanet.net Hop 7 loss 0.. 0% cambridge1-mbone1.bbnplanet.net Hop 8 loss 0.. 0% MBONE3.UU.NET Hop 9 loss 0.. 0% st6.wcomcast.com Hop 10 loss 0.. 0% 128.101.25.56

Nodes, with loss variation (percent over mtrace -l's 10-second intervals):

Hop 0 loss 0.. 0% 224.0.1.32 Hop 1 loss 0.. 0% arcbone.arc.nasa.gov Hop 2 loss 0.. 0% ARC-MBONE.NSN.NASA.GOV Hop 3 loss 0.. 0% FIXW-MBONE.NSN.NASA.GOV Hop 4 loss 0.. 1% mbone.nsi.nasa.gov Hop 5 loss 0.. 3% stk-mbone-1.sprintlink.net Hop 6 loss 0.. 7% fw-mbone-1.sprintlink.net Hop 7 loss -99.. 0% dc-mbone-1.sprintlink.net Hop 8 loss 21..28% pen-mbone-1.sprintlink.net Hop 9 loss 0.. 1% mbone1.nap.net Hop 10 loss -1.. 2% MBONE3.UU.NET Hop 11 loss 0.. 0% st6.wcomcast.com Hop 12 loss 0.. 0% 128.101.25.56

Nodes, with loss variation (percent over mtrace -l's 10-second intervals):

Hop 0 loss 0.. 0% 224.0.1.32 Hop 1 loss 0.. 1% arcbone.arc.nasa.gov Hop 2 loss 0.. 0% ARC-MBONE.NSN.NASA.GOV Hop 3 loss 0..21% FIXW-MBONE.NSN.NASA.GOV Hop 4 loss -1.. 3% dec3800-2-fddi-1.SanFrancisco.mci.net Hop 5 loss -1.. 2% dec3800-1-fddi-1.Dallas.mci.net Hop 6 loss 0.. 3% dec3800-2-fddi-1.Atlanta.mci.net Hop 7 loss -2.. 2% atlanta3-mbone1.bbnplanet.net Hop 8 loss -1.. 4% collegepk-mbone1.bbnplanet.net Hop 9 loss -1.. 3% cambridge1-mbone1.bbnplanet.net Hop 10 loss -2.. 5% MBONE3.UU.NET Hop 11 loss 0.. 0% st6.wcomcast.com Hop 12 loss 0.. 0% 128.101.25.56