Packet Loss on talktalk backhaul, host-78-144-5-89.as13285.net

[talkwhatever]

I've been frustrated by periodic glitching on Teams calls, so went exploring. Using mtr (MyTraceRoute) to give an overview via repeated traceroutes, I can see that one host on the talktalk network (host-78-144-5-89.as13285.net) has consistently high packet loss (see first snapshot below).

You can see that all hops, to a typical example destination, show no loss apart from what for me is the 5th and last hop in your network. Maybe your peering with Cloudflare is congested? You can see these glitches happen now and again (see second snapshot below) and I notice that they correspond to breakup in my Teams conversations.

How do I get you guys to get your network folk to investigate this congestion please?

                                                           Packets               Pings
 Host                                                    Loss%   Snt   Last   Avg  Best  Wrst StDev
 1. 10.0.1.1                                              0.0%  7476    1.7   3.0   1.3 2329.  53.0
 2. host-80-47-128-1.as13285.net                          0.0%  7475   33.0  12.1   5.9 2183.  52.7
 3. ae50-ner002.msp.as13285.net                           0.0%  7475    9.4  14.8   7.4 2074.  49.9
 4. ae50-scr002-msp.as13285.net                           0.0%  7475   32.1  12.8   7.3 1898.  44.3
 5. host-78-144-5-89.as13285.net                          7.9%  7475   10.2  14.8   8.0 2220.  57.4
 6. 162.158.32.11                                         0.0%  7475   11.1  13.8   7.9 2768.  69.0
 7. one.one.one.one                                       0.0%  7475    8.4  10.7   7.4 2477.  55.8

                             Last  70 pings
 1. 10.0.1.1                 ......................................................................
 2. host-80-47-128-1.as13285 ......................................................................
 3. ae50-ner002.msp.as13285. ......................................................................
 4. ae50-scr002-msp.as13285. ..................................................................... 
 5. host-78-144-5-89.as13285 .........?...........?..?.......?...?.......?.?..??.............?....
 6. 162.158.32.11            ..................................................................... 
 7. one.one.one.one          ..................................................................... 

Thanks

[Michelle-TalkTalk]

Morning,

Ok thanks for the update and please let us know if you need any further assistance.

Thanks

[talkwhatever]

So, I had one of my infrequent Teams calls today and it went off fuss free. Yeah! My (regular) Zoom calls have also been ok too!

Interestingly (ok that depends on your geeky perspective), the Talk Talk peering with Cloudflare has switched to another host;

162.158.72.3 rather than 162.158.32.9 previously. Yet the packet loss seems similar on your egress server host-78-144-5-89.as13285.net:

                                                             Packets               Pings
 Host                                                      Loss%   Snt   Last   Avg  Best  Wrst StDev
 1. AS???    10.0.1.1                                       0.0%   581    2.7   2.8   1.2 235.8   9.7
 2. AS9105   host-80-47-128-1.as13285.net                   0.0%   581   11.6  11.7   5.9 305.1  17.2
 3. AS13285  ae50-ner002.msp.as13285.net                    0.0%   581    8.8  11.2   7.2 155.7   7.7
 4. AS13285  ae50-scr002-msp.as13285.net                    0.0%   581   11.6  12.3   7.3  48.5   5.5
 5. AS13285  host-78-144-5-89.as13285.net                  15.3%   581   17.0  13.0   8.2 150.6   8.2
 6. AS13335  162.158.72.3                                   0.0%   581   13.0  14.3   7.6 163.2  16.8
 7. AS13335  one.one.one.one                                0.0%   581    9.8   9.3   7.1  25.6   1.2

Perhaps the issue was some up-link noise issue that didn't show on these tests and the congestion (dropped packets) on host-78-144-5-89.as13285.net is not causing significant issues (just packet retransmissions)?

At fear of jinxing it, it seems ok for now! Time to box up your TT wifi/router/modem and stick it back in the cupboard and stay with my simple LAN friendly Openreach modem...

[Chris-TalkTalk]

Glad to hear there's been some improvement, thanks for letting me know

Chris

[talkwhatever]

Hi Chris,

I just had a call without any problems, so fingers crossed. I'll see how it goes. 🤞 

(I reverted to my Openreach modem to solve the LAN issue. I'm getting identical speed tests with that compared to what I saw with the Sagemcom modem/router. So maybe the wiring refresh helped?)

[Chris-TalkTalk]

Hi talkwhatever,

Thanks for the update. Just to confirm, has there been any improvement in Face Time, Zoom and Teams calls?

Chris

[talkwhatever]

I spoke to the network folk and they suggested they could not escalate this further. They wanted someone to come out and sort my upstream issue. So I now have a shiny new master socket. The Cube boy (TT sub-contractor) installed the Sagem Wifi Hub that had been sat in its box for years. I now have a slightly faster sync speed (lower noise margin 7dB rather than 15dB previously with my OpenReach modem).

However, the issue of packet loss in the Talk Talk network remains. Plus the router adds its own packet loss and offers no way of operating in bridge mode (modem only). So I have double NAT (two daisy chained LANs) and bonus packet loss on my LAN!

                                                             Packets               Pings
 Host                                                      Loss%   Snt   Last   Avg  Best  Wrst StDev
 1. AS???    10.0.1.1                                       0.0%   889    2.5   3.1   1.1 131.8   9.1
 2. AS???    192.168.1.1                                   12.7%   889    5.2   4.1   2.1 131.3   6.5
 3. AS9105   host-80-47-128-1.as13285.net                   0.0%   889   12.8  11.8   6.2 128.1  10.3
 4. AS13285  ae50-ner002.msp.as13285.net                    0.0%   888   10.9  12.8   7.6 130.7  10.0
 5. AS13285  ae50-scr002-msp.as13285.net                    0.0%   888   12.1  12.3   7.7 128.5   8.9
 6. AS13285  host-78-144-5-89.as13285.net                  10.0%   888   10.9  14.3   8.5 129.8  11.8
 7. AS13335  162.158.32.9                                   0.0%   888   11.9  12.7   8.1 334.3  18.6
 8. AS13335  one.one.one.one                                0.1%   888   10.7  11.4   7.6 259.4  16.0

There must be someone who can take ownership of the issue with talk talk host-78-144-5-89.as13285.net and the poor service effecting my video calls?

[Chris-TalkTalk]

OK thanks, I'll ask them to give you a call

Chris

[talkwhatever]

Hi Chris,

OK, I've found an old phone and plugged it in. Let's see how that approach goes please?

[Chris-TalkTalk]

Hi talkwhatever,

I understand your frustration but I'm not hiding behind the Network team. As you can probably appreciate, this isn't something that I can investigate directly myself, I have to escalate it. The network team aren't elusive, as I mentioned in my early post, they told me that they tried to contact you directly and I also offered to arrange a callback from them at a time convenient to yourself to discuss the issue further.

Chris

[talkwhatever]

I appreciate your familiar response Chris, but we need to try a different approach. Look at what I hear from you guys!

on ‎05-04-2023 02:49 PM

I reported the issue and you efficiently took me through some first line stages. However, since then you seem to be hiding behind the elusive networks team:

on ‎06-04-2023 12:27 PM Message 12 of 31
Thanks. I'll ask our network team to take a look. We'll let you know when we receive an update 

on ‎11-04-2023 11:04 AM Message 16 of 31
ok, we can see what networks advise when they get back to us.

on ‎17-04-2023 07:46 AM Message 18 of 31
Sorry, there are no updates yet, we'll let you know when there are

on ‎25-04-2023 12:51 PM Message 20 of 31

Thanks for the information. I've passed this over to our Network Team and asked them for an update. I'll let you know when I have any more information 

on ‎28-04-2023 08:55 AM Message 27 of 31
Thanks for the information, I'll pass it on to our network team

on ‎02-05-2023 10:28 AM Message 29 of 31
OK thanks. I'll let you know when I have any further information

on ‎03-05-2023 11:29 AM Message 31 of 31
I've asked for an update and passed on the latest information that you've posted. I'll let you know when I have any further information

What do we need to do to persuade me there might be some work towards an actual resolution? You must have some other options you can use...

[Chris-TalkTalk]

Hi,

I've asked for an update and passed on the latest information that you've posted. I'll let you know when I have any further information

Chris

[talkwhatever]

Just had to abandon a Zoom call as I was sounding like a robot. We switched to Google Meet and all was well. Looking at the traceroute I see that at hop 4, my traffic takes a different route avoiding the problematic router host-78-144-5-89.as13285.net shown above.

Please can you find out what is the delay in responding to this issue? It's been a month now and no meaningful response from your networks team! Do I need to switch providers, or are you going to investigate and help me?

Good network performance when peering with google:

                                                             Packets               Pings
 Host                                                      Loss%   Snt   Last   Avg  Best  Wrst StDev
 1. AS???    10.0.1.1                                       0.0%  2191    2.1   3.8   1.1 2129.  52.0
 2. AS9105   host-80-47-128-1.as13285.net                   0.0%  2191   12.2  15.1   5.8 3039.  82.0
 3. AS13285  ae50-ner002.msp.as13285.net                    0.0%  2191    8.9  13.8   7.1 2925.  77.3
 4. AS13285  ae50-scr002-msp.as13285.net                    0.0%  2190   10.6  13.7   7.3 2809.  73.6
 5. AS13285  ae63-scr102.loh.as13285.net                    0.0%  2190   15.1  21.1  13.9 2710.  70.3
 6. AS13285  host-78-144-4-35.as13285.net                   0.0%  2190   17.6  19.4  14.4 2582.  66.0
 7. AS15169  209.85.248.229                                 0.0%  2190   15.9  18.6  14.4 2467.  62.1
 8. AS15169  142.251.54.25                                  0.0%  2190   16.0  18.7  14.5 2355.  59.4
 9. AS15169  lhr25s31-in-f14.1e100.net                      0.0%  2190   15.6  17.5  13.7 2243.  55.5

[Chris-TalkTalk]

OK thanks. I'll let you know when I have any further information

Chris

[talkwhatever]

Just to update, this issue also affects FaceTime and Zoom as well as Teams calls.

[Chris-TalkTalk]

Thanks for the information, I'll pass it on to our network team

Chris

[talkwhatever]

Hi Chris,

As discussed earlier in this thread, I can not alter how your egress routers behave.

For illustration I've run two separate MyTraceRoutes which use two different TalkTalk egress routers, both on your 78.144.0.0/15 subnet. The first trace (to one.one.one.one dns) is the subject of this ticket and shows the packet loss issue with your host-78-144-5-89.as13285.net. Dropped packets here coincide with interruptions in Teams video calls.

                                                             Packets               Pings
 Host                                                      Loss%   Snt   Last   Avg  Best  Wrst StDev
 1. ???                10.0.1.1                             0.1%  6157    2.7  12.8   1.2 2217. 100.1
 2. 80.46.0.0/15       host-80-47-128-1.as13285.net         0.1%  6157    9.0  21.2   5.9 2263. 101.6
 3. 78.144.0.0/15      ae50-ner002.msp.as13285.net          0.1%  6157    8.3  23.9   7.2 2123.  98.4
 4. 78.144.0.0/15      ae50-scr002-msp.as13285.net          0.0%  6157   11.8  22.3   7.2 2214. 100.5
 5. 78.144.0.0/15      host-78-144-5-89.as13285.net        11.9%  6157   11.3  21.4   7.7 2070.  90.3
 6. 162.158.32.0/22    162.158.32.11                        0.0%  6157    9.9  21.4   7.8 2507. 101.3
 7. 1.1.1.0/24         one.one.one.one                      0.1%  6157    8.2  18.9   7.2 2360.  94.5

The second trace (to dns.google) happens to egress via a different one of your boundary servers, host-78-144-4-35.as13285.net and shows no packet loss.

                                                             Packets               Pings
 Host                                                      Loss%   Snt   Last   Avg  Best  Wrst StDev
 1. ???                10.0.1.1                             0.0%  6148    1.9   7.4   1.0 1983.  71.1
 2. 80.46.0.0/15       host-80-47-128-1.as13285.net         0.1%  6148    7.4  16.4   5.9 2538.  78.2
 3. 78.144.0.0/15      ae50-ner002.msp.as13285.net          0.1%  6148   10.7  16.4   7.2 2425.  76.2
 4. 78.144.0.0/15      ae50-scr002-msp.as13285.net          0.0%  6148   10.9  17.4   7.4 2313.  82.7
 5. 78.144.0.0/15      ae63-scr102.loh.as13285.net          0.0%  6148   16.6  23.8  13.7 2211.  77.2
 6. 78.144.0.0/15      host-78-144-4-35.as13285.net         0.0%  6147   15.5  21.3  13.7 2089.  71.4
 7. 209.85.128.0/17    209.85.249.149                       0.1%  6147   17.0  23.0  14.5 2326.  84.1
 8. 142.251.54.0/24    142.251.54.49                        0.0%  6147   15.4  21.4  13.7 2241.  78.1
 9. 8.8.8.0/24         dns.google                           0.0%  6147   15.7  21.4  13.9 2126.  77.8

Please ask your network folk to look at the peering from host-78-144-5-89.as13285.net and check for congestion. Once (if) confirmed, perhaps they can adjust the routing, or install additional capacity?

Many thanks 

[Chris-TalkTalk]

Hi,

I don't know to be honest, they would just run through some diagnostics with you.

Chris

[talkwhatever]

I don't have a phone plugged into my landline, so there'll never be an answer there. Is there a particular question they want to ask (and I can answer here)?

[Chris-TalkTalk]

They usually try your landline and mobile (if you have a mobile number registered on the system). If you'd like to add your mobile number to your community profile I can ask them to contact you on that

Chris

[talkwhatever]

Hi Chris, I haven't seen any messages. How is your colleague trying to reach me?