Packet Loss on talktalk backhaul, host-78-144-5-89.as13285.net

[talkwhatever]

I've been frustrated by periodic glitching on Teams calls, so went exploring. Using mtr (MyTraceRoute) to give an overview via repeated traceroutes, I can see that one host on the talktalk network (host-78-144-5-89.as13285.net) has consistently high packet loss (see first snapshot below).

You can see that all hops, to a typical example destination, show no loss apart from what for me is the 5th and last hop in your network. Maybe your peering with Cloudflare is congested? You can see these glitches happen now and again (see second snapshot below) and I notice that they correspond to breakup in my Teams conversations.

How do I get you guys to get your network folk to investigate this congestion please?

                                                           Packets               Pings
 Host                                                    Loss%   Snt   Last   Avg  Best  Wrst StDev
 1. 10.0.1.1                                              0.0%  7476    1.7   3.0   1.3 2329.  53.0
 2. host-80-47-128-1.as13285.net                          0.0%  7475   33.0  12.1   5.9 2183.  52.7
 3. ae50-ner002.msp.as13285.net                           0.0%  7475    9.4  14.8   7.4 2074.  49.9
 4. ae50-scr002-msp.as13285.net                           0.0%  7475   32.1  12.8   7.3 1898.  44.3
 5. host-78-144-5-89.as13285.net                          7.9%  7475   10.2  14.8   8.0 2220.  57.4
 6. 162.158.32.11                                         0.0%  7475   11.1  13.8   7.9 2768.  69.0
 7. one.one.one.one                                       0.0%  7475    8.4  10.7   7.4 2477.  55.8

                             Last  70 pings
 1. 10.0.1.1                 ......................................................................
 2. host-80-47-128-1.as13285 ......................................................................
 3. ae50-ner002.msp.as13285. ......................................................................
 4. ae50-scr002-msp.as13285. ..................................................................... 
 5. host-78-144-5-89.as13285 .........?...........?..?.......?...?.......?.?..??.............?....
 6. 162.158.32.11            ..................................................................... 
 7. one.one.one.one          ..................................................................... 

Thanks

[Chris-TalkTalk]

Hi talkwhatever,

A colleague in our Network Team says that he's tried to contact you a couple of times but has been unsuccessful, what is the best time to contact you?

Chris

[Chris-TalkTalk]

Hi talkwhatever,

Thanks for the information. I've passed this over to our Network Team and asked them for an update. I'll let you know when I have any more information 

Chris

[talkwhatever]

No significant change:

                                                               Packets               Pings
 Host                                                        Loss%   Snt   Last   Avg  Best  Wrst StDev
 1. ???                10.0.1.1                               0.0%  2011    1.8   2.5   1.2 162.2   4.9
 2. 80.46.0.0/15       host-80-47-128-1.as13285.net           0.0%  2011    7.9  10.6   5.9 149.3   8.1
 3. 78.144.0.0/15      ae50-ner002.msp.as13285.net            0.0%  2011    9.1  12.1   7.2 320.1  10.3
 4. 78.144.0.0/15      ae50-scr002-msp.as13285.net            0.0%  2011   11.2  11.4   7.3 313.2  11.1
 5. 78.144.0.0/15      host-78-144-5-89.as13285.net          17.6%  2011   17.3  13.1   8.0 318.1  11.0
 6. 162.158.32.0/22    162.158.32.11                          0.0%  2011   11.2  11.6   7.8 356.4  12.8
 7. 1.1.1.0/24         one.one.one.one                        0.0%  2010   10.0   9.4   7.2 312.3   9.5

EDIT: Average loss on that hop (now the kids are home) is up to 21.2% after another 4500 seconds of monitoring.

                                                               Packets               Pings
 Host                                                        Loss%   Snt   Last   Avg  Best  Wrst StDev
 1. ???                10.0.1.1                               0.0%  6572    2.3   1.9   1.1 312.7   9.7
 2. 80.46.0.0/15       host-80-47-128-1.as13285.net           0.0%  6571    7.6   9.5   5.9 591.6  11.6
 3. 78.144.0.0/15      ae50-ner002.msp.as13285.net            0.0%  6571    9.4  11.5   7.2 437.7  11.6
 4. 78.144.0.0/15      ae50-scr002-msp.as13285.net            0.0%  6571    9.2  10.2   7.3 355.6  10.7
 5. 78.144.0.0/15      host-78-144-5-89.as13285.net          21.2%  6571   14.4  12.2   7.9 318.1  10.0
 6. 162.158.32.0/22    162.158.32.11                          0.0%  6571   10.5   9.8   7.7 356.4   9.8
 7. 1.1.1.0/24         one.one.one.one                        0.0%  6571    8.6   8.6   7.2 462.8  10.9

                                                Last  55 pings
 1. ???                10.0.1.1                 .......................................................
 2. 80.46.0.0/15       host-80-47-128-1.as13285 .......................................................
 3. 78.144.0.0/15      ae50-ner002.msp.as13285. ...................................................... 
 4. 78.144.0.0/15      ae50-scr002-msp.as13285. ...................................................... 
 5. 78.144.0.0/15      host-78-144-5-89.as13285 ..?..?..?..?.?........?..?.....?........??.???..??....
 6. 162.158.32.0/22    162.158.32.11            ...................................................... 
 7. 1.1.1.0/24         one.one.one.one          ...................................................... 

[Chris-TalkTalk]

Sorry, there are no updates yet, we'll let you know when there are

Chris

[talkwhatever]

Any feedback from networks? Symptoms unchanged today:

                                                               Packets               Pings
 Host                                                        Loss%   Snt   Last   Avg  Best  Wrst StDev
 1. ???                10.0.1.1                               0.0%  1947    1.9   6.2   1.1 2430.  71.8
 2. 80.46.0.0/15       host-80-47-128-1.as13285.net           0.0%  1947    7.8  15.0   6.0 2292.  81.4
 3. 78.144.0.0/15      ae50-ner002.msp.as13285.net            0.0%  1947    8.3  18.9   7.5 2148.  75.5
 4. 78.144.0.0/15      ae50-scr002-msp.as13285.net            0.0%  1947   10.4  15.4   7.6 2005.  68.0
 5. 78.144.0.0/15      host-78-144-5-89.as13285.net          16.0%  1946   11.6  16.5   8.0 2868.  84.0
 6. 162.158.32.0/22    162.158.32.11                          0.0%  1946   10.9  15.6   7.9 2725.  83.5
 7. 1.1.1.0/24         one.one.one.one                        0.1%  1946    8.2  13.2   7.3 2580.  77.3

[Karl-TalkTalk]

Hi

ok, we can see what networks advise when they get back to us.

Karl

[talkwhatever]

Hi Karl,

Nope. Just the usual minor packet loss, but nothing problematic.

But looking now, I see the problem is as it was last week before the long holiday weekend. It seems like it's probably related to traffic volumes... 

                                                              Packets               Pings
 Host                                                        Loss%   Snt   Last   Avg  Best  Wrst StDev
 1. 10.0.1.1                                                  0.0%    99    2.1   2.7   1.3  11.7   1.4
 2. host-80-47-128-1.as13285.net                              0.0%    99    6.9  10.0   6.0  40.0   4.7
 3. ae50-ner002.msp.as13285.net                               0.0%    98    8.8  12.6   7.9  49.4   8.6
 4. ae50-scr002-msp.as13285.net                               0.0%    98    8.8  12.5   7.5  41.3   6.1
 5. host-78-144-5-89.as13285.net                             17.3%    98    8.9  13.0   8.2  31.3   5.5
 6. 162.158.32.11                                             0.0%    98   10.0  11.9   8.3  36.5   5.1
 7. one.one.one.one                                           0.0%    98    9.1   9.2   7.4  16.0   1.1

EDIT: A longer set of data showing the issue is consistent:

                                                               Packets               Pings
 Host                                                        Loss%   Snt   Last   Avg  Best  Wrst StDev
 1. ???                10.0.1.1                               0.0%   369    5.1   2.8   1.3  13.2   1.7
 2. 80.46.0.0/15       host-80-47-128-1.as13285.net           0.0%   369    9.2  10.6   6.0  66.9   6.3
 3. 78.144.0.0/15      ae50-ner002.msp.as13285.net            0.0%   368   11.3  12.9   7.6  49.4   8.1
 4. 78.144.0.0/15      ae50-scr002-msp.as13285.net            0.0%   368   15.7  11.7   7.5  41.3   4.6
 5. 78.144.0.0/15      host-78-144-5-89.as13285.net          17.4%   368   12.2  14.2   8.1 286.3  16.8
 6. 162.158.32.0/22    162.158.32.11                          0.0%   368   13.4  12.0   8.1 144.8   8.6
 7. 1.1.1.0/24         one.one.one.one                        0.0%   368   17.6   9.4   7.3  23.5   2.0

[Karl-TalkTalk]

Hi

have you seen any packet loss over the weekend?

Thanks  

Karl. 

[talkwhatever]

Oddly, I've just had an email from Talk Talk (not the community) saying:

---

We're sorry you've had to report a problem. Your complaint has now been logged and we'll do all we can to put things right as quickly as possible.

---

I assume it's this thread that triggered my complaint, but I wonder why? 

Meanwhile, perhaps due to today being a bank holiday (hence less traffic), I've seen no significant packet loss today. 🤷

[Chris-TalkTalk]

Thanks. I'll ask our network team to take a look. We'll let you know when we receive an update 

Thanks

Chris

[talkwhatever]

As requested by Karl, that last traceroute didn't involve a router, it was via a direct connection to the Openreach modem. Moreover, it is highly unlikely that source routers effect the behaviour of routers on subsequent hops.

The issue with Talk Talk host-78-144-5-89.as13285.net remains at the moment. 

                                                               Packets               Pings
 Host                                                        Loss%   Snt   Last   Avg  Best  Wrst StDev
 1. ???    10.0.1.1                                           0.0%   112    4.1  71.7   1.5 1719. 237.9
 2. 9105   host-80-47-128-1.as13285.net                       0.0%   112   66.4  79.6   6.7 1576. 214.4
 3. 13285  ae50-ner002.msp.as13285.net                        0.0%   112   29.6  79.2   7.9 1433. 190.6
 4. 13285  ae50-scr002-msp.as13285.net                        0.0%   112   25.4  74.8   8.2 1301. 173.0
 5. 13285  host-78-144-5-89.as13285.net                      20.5%   112    9.9 111.7   8.6 2173. 334.8
 6. 13335  162.158.32.11                                      0.0%   112  161.6  93.0   8.6 2030. 287.7
 7. 13335  one.one.one.one                                    0.0%   112   27.0  85.9   7.8 1872. 262.8

I feel we've done enough preamble to warrant an escalation to the team who can investigate congestion on your peering with AS13335 – Cloudflare, Inc.

If you prefer a simple traceroute (note dropped packet on hop 5):

% traceroute 1.1.1.1
traceroute to 1.1.1.1 (1.1.1.1), 64 hops max, 52 byte packets
 1  10.0.1.1 (10.0.1.1)  2.821 ms  2.597 ms  1.571 ms
 2  host-80-47-128-1.as13285.net (80.47.128.1)  8.315 ms  7.697 ms  7.579 ms
 3  ae50-ner002.msp.as13285.net (78.144.1.35)  9.408 ms  11.337 ms  8.760 ms
 4  ae50-scr002-msp.as13285.net (78.144.1.34)  9.266 ms  7.653 ms  15.594 ms
 5  host-78-144-5-89.as13285.net (78.144.5.89)  24.843 ms *  12.403 ms
 6  162.158.32.11 (162.158.32.11)  8.869 ms  8.307 ms  9.214 ms
 7  one.one.one.one (1.1.1.1)  7.978 ms  8.757 ms  9.821 ms

[Chris-TalkTalk]

Hi talkwhatever,

Could you try switching the router off for 30 minutes, then switch back on and run another test. Please let us know how you get on

Chris

[talkwhatever]

% traceroute 1.1.1.1
traceroute to 1.1.1.1 (1.1.1.1), 64 hops max, 52 byte packets
 1  host-80-47-128-1 (80.47.128.1)  5.098 ms  11.629 ms  11.587 ms
 2  ae50-ner002.msp.as13285.net (78.144.1.35)  6.283 ms  6.245 ms  6.467 ms
 3  ae50-scr002-msp (78.144.1.34)  6.557 ms  6.025 ms  6.686 ms
 4  * host-78-144-5-89 (78.144.5.89)  8.069 ms  6.698 ms
 5  162.158.72.3 (162.158.72.3)  6.362 ms  6.277 ms
    162.158.32.9 (162.158.32.9)  6.806 ms
 6  one.one.one.one (1.1.1.1)  6.222 ms  6.317 ms  6.155 ms
% traceroute 1.1.1.1
traceroute to 1.1.1.1 (1.1.1.1), 64 hops max, 52 byte packets
 1  host-80-47-128-1 (80.47.128.1)  5.098 ms  4.860 ms  4.567 ms
 2  ae50-ner002.msp.as13285.net (78.144.1.35)  6.603 ms  6.160 ms  20.997 ms
 3  ae50-scr002-msp (78.144.1.34)  6.069 ms  16.110 ms  6.135 ms
 4  host-78-144-5-89 (78.144.5.89)  6.824 ms  6.671 ms  6.755 ms
 5  162.158.32.9 (162.158.32.9)  6.628 ms
    162.158.72.3 (162.158.72.3)  14.290 ms  6.972 ms
 6  one.one.one.one (1.1.1.1)  6.162 ms  6.242 ms  6.295 ms
% traceroute 1.1.1.1
traceroute to 1.1.1.1 (1.1.1.1), 64 hops max, 52 byte packets
 1  host-80-47-128-1 (80.47.128.1)  5.092 ms  5.391 ms  4.641 ms
 2  ae50-ner002.msp.as13285.net (78.144.1.35)  27.653 ms  6.210 ms  6.505 ms
 3  ae50-scr002-msp (78.144.1.34)  6.321 ms  6.396 ms  6.011 ms
 4  host-78-144-5-89 (78.144.5.89)  7.223 ms  7.177 ms *
 5  162.158.32.11 (162.158.32.11)  6.909 ms  6.722 ms
    162.158.32.9 (162.158.32.9)  7.080 ms
 6  one.one.one.one (1.1.1.1)  6.389 ms  6.112 ms  6.326 ms

[Karl-TalkTalk]

Hi,  

If possible, these are the things that our network teams will ask if a pass some traces to them, so I'm just trying to get everything down to basics for them, so they don't bounce the issue back to us.

Karl.

[talkwhatever]

I know you want to eliminate router, Wi-Fi and LAN as an issue, but do you really want to connect my machine directly to the OpenReach modem? Is seems fairly clear that the issue is further into the internet than my house.

[talkwhatever]

Yup, I'm running a class A internal LAN. 192.168 subnets take so long to type! 😉 

[Karl-TalkTalk]

Hi

Quick question regarding your setup, your traces all start with a 10. Ip address, this is a class A internal range, usually private, the trace would normally start from the router with a 192 address, are you running a trace from a machine over a private network etc, rather that direct to the router ?

Thanks  

Karl. 

[talkwhatever]

Here's one that happened to capture one of the regular glitches:

% traceroute 1.1.1.1
traceroute to 1.1.1.1 (1.1.1.1), 64 hops max, 52 byte packets
 1  10.0.1.1 (10.0.1.1)  2.412 ms  1.568 ms  2.632 ms
 2  host-80-47-128-1.as13285.net (80.47.128.1)  8.845 ms  8.663 ms  7.002 ms
 3  ae50-ner002.msp.as13285.net (78.144.1.35)  8.547 ms  8.994 ms  10.200 ms
 4  ae50-scr002-msp.as13285.net (78.144.1.34)  8.549 ms  9.932 ms  9.122 ms
 5  host-78-144-5-89.as13285.net (78.144.5.89)  9.215 ms *  11.301 ms
 6  162.158.72.3 (162.158.72.3)  8.819 ms  8.522 ms
    162.158.32.11 (162.158.32.11)  10.290 ms
 7  one.one.one.one (1.1.1.1)  8.578 ms  8.884 ms  8.959 ms

Note the timeout (packet loss on the second ping of hop 5)

[talkwhatever]

% traceroute 1.1.1.1
traceroute to 1.1.1.1 (1.1.1.1), 64 hops max, 52 byte packets
 1  10.0.1.1 (10.0.1.1)  2.330 ms  2.105 ms  3.495 ms
 2  host-80-47-128-1.as13285.net (80.47.128.1)  9.481 ms  11.003 ms  12.654 ms
 3  ae50-ner002.msp.as13285.net (78.144.1.35)  8.287 ms  16.507 ms  14.705 ms
 4  ae50-scr002-msp.as13285.net (78.144.1.34)  15.058 ms  11.431 ms  8.897 ms
 5  host-78-144-5-89.as13285.net (78.144.5.89)  21.885 ms  9.537 ms  9.103 ms
 6  162.158.32.11 (162.158.32.11)  9.070 ms
    162.158.32.9 (162.158.32.9)  10.186 ms
    162.158.72.3 (162.158.72.3)  60.006 ms
 7  one.one.one.one (1.1.1.1)  9.842 ms  9.587 ms  9.327 ms
% traceroute 1.1.1.1
traceroute to 1.1.1.1 (1.1.1.1), 64 hops max, 52 byte packets
 1  10.0.1.1 (10.0.1.1)  2.867 ms  2.854 ms  1.462 ms
 2  host-80-47-128-1.as13285.net (80.47.128.1)  9.484 ms  6.815 ms  6.546 ms
 3  ae50-ner002.msp.as13285.net (78.144.1.35)  11.013 ms  8.956 ms  9.963 ms
 4  ae50-scr002-msp.as13285.net (78.144.1.34)  8.864 ms  10.999 ms  18.432 ms
 5  host-78-144-5-89.as13285.net (78.144.5.89)  10.753 ms  8.817 ms  9.188 ms
 6  162.158.32.9 (162.158.32.9)  10.472 ms
    162.158.32.11 (162.158.32.11)  9.572 ms  10.933 ms
 7  one.one.one.one (1.1.1.1)  10.432 ms  9.861 ms  9.751 ms

[Karl-TalkTalk]

Hi

can you run a couple of traceroutes directly from your machine wired to the router and let me see the results.

Thanks  

Karl.