Long Delay receiving emails forwarded from 123.reg

[agpvm]

(See also thread here: Long-Delay-receiving-emails - 4090443 )

I too am getting long delays, and even the occasional total failure with emails being forwarded from 123-reg to Virgin Media.

It appears that the reason is that Virgin Media have set a Limit of just 20 for the number of simultaneous connections that 123 can have to Virgin Media servers when sending/forwarding emails to them.  123 are very frequently exceeding this limit, causing their connections to be rejected.  Our messages then get delayed and  tried again later, and each time it fails the delay gets even longer.  So often messages aren't getting through for many hours.

Unfortunately Virgin Media are blaming 123 and 123 are blaming Virgin Media, with neither company passing the information on to the people who could actually resolve this.

Virgin Media's Limit of 20 is the default value - but they could increase it.  123 are exceeding the default value but could set their limit to 20 when sending to Virgin Media, or could just not back off for so long.

Here's part of an example of a rejection message:

A message that you sent has not yet been delivered to one or more of its
recipients after more than 6 hours on the queue on smtp12.mailcore.me.
...
The address to which the message has not yet been delivered is:
...
host mx.tb.ukmail.iss.as9143.net [212.54.56.11]
Delay reason: SMTP error from remote mail server after initial connection:
421 mx1.tb.ukmail.iss.as9143.net mx1.tb.ukmail.iss.as9143.net
logid=SMTPRC 421 MXIN106 Parallel SessionLimit of 20 for your 94.136.40.149 exceeded. Retry later.
;id=PdHgiN4hD3aVY;sid=PdHgiN4hD3aVY;mta=mx1.tb;d=20191030;t=030028[CET];ipsrc=94.136.40.149;

Alan

[agpvm]

I too am getting long delays, and even the occasional total failure with emails being forwarded from 123-reg to Virgin Media.

It appears that the reason is that Virgin Media have set a Limit of just 20 for the number of simultaneous connections that 123 can have to Virgin Media servers when sending/forwarding emails to them.  123 are very frequently exceeding this limit, causing their connections to be rejected.  Our messages then get delayed and  tried again later, and each time it fails the delay gets even longer.  So often messages aren't getting through for many hours.

Unfortunately Virgin Media are blaming 123 and 123 are blaming Virgin Media, with neither company passing the information on to the people who could actually resolve this.

Virgin Media's Limit of 20 is the default value - but they could increase it.  123 are exceeding the default value but could set their limit to 20 when sending to Virgin Media, or could just not back off for so long.

Here's part of an example of a rejection message:

A message that you sent has not yet been delivered to one or more of its
recipients after more than 6 hours on the queue on smtp12.mailcore.me.
...
The address to which the message has not yet been delivered is:
...
host mx.tb.ukmail.iss.as9143.net [212.54.56.11]
Delay reason: SMTP error from remote mail server after initial connection:
421 mx1.tb.ukmail.iss.as9143.net mx1.tb.ukmail.iss.as9143.net
logid=SMTPRC 421 MXIN106 Parallel SessionLimit of 20 for your 94.136.40.149 exceeded. Retry later.
;id=PdHgiN4hD3aVY;sid=PdHgiN4hD3aVY;mta=mx1.tb;d=20191030;t=030028[CET];ipsrc=94.136.40.149;

Alan

[Kev_B]

Hey @ravenstar68 

Thanks for the nudge, and for your continued help with this.

I've had some headers from both @9fingers and @3_g - thanks for those.

@9fingers

It looks like the delay in the header you sent is introduced before it arrives with our mail servers. Once it reaches our end it seems to pass through fine and we're not seeing the errors that were showing in headers before any tweaks were made. I'd like to have a look at some other headers where you have a delay just to be sure, if possible?

@3_g 

Thanks for your headers, of the 3 I could only see a delay within our mail exchangers on the email that was delayed by 5 minutes. The 14-minute delay seems to happen on this step:

Received: from mailex.mailcore.me ([94.136.40.61])
by mx3.tb.ukmail.iss.as9143.net with ESMTP

There's 14 minutes between the mailcore.me server receiving and then the next step where we receive it. Tim - does this point to a delay at the receiving exchanger or the sending one? I'm happy to pass these headers along, but since we're no longer showing the parallel sessions exceeded errors then I'd like to be sure that the delay is happening at mailcore.me or when it arrives at our exchanger.

[Grindathotte]

I'm still using the technique suggested in this thread of forwarding from 123-reg to Google Mail and then to VM and still getting no delays (maximum 2 minutes; usually instantaneously).  This does suggest that delays are occurring on the path from 123-reg to VM.

[agpvm]

@Kev_B

The issue is that 123 cannot pass the emails to VM, as VM frequently refuses the connection.  When the connection is refused 123 waits a while then tries again, and if that fails then waits again, and so on....

As VM frequently fails to let 123 connect, emails can take many hours to arrive.  Or even worse 123 eventually gives up trying.

The 14 minute delay you see is exactly because of this.  VM will have failed 123's connection attempts a number of times during those 14 minutes.  The only header you see is when it finally allowed the connection.

Some of us have set up 123 so that any mail we receive gets forwarded to both VM and googlemail so that we can monitor this issue (and make sure we get emails promptly).  123 NEVER has any delays forwarding the email to googlemail, but frequently struggles to be allowed to pass the email to VM. 

@Kev_B - I will PM you the headers for an email received today that took 1 second for 123 to pass the email to googlemail, but 18 minutes 27 seconds for 123 to pass the same email to VM.

@ravenstar68 - on Saturday you said that you had an example where 123 took 5 hours to accept an email.  I have never seen such an example -  but if that situation occurred to an email sent to me it would delay the copies sent to both googlemail and VM equally - so wouldn't be part of the problem I am seeing.  However would it be possible for you to let me see the example, or for you to ask the original person to let me see it, as I would like to take a look.

@Kev_B - With regards the email that was delayed internally by VM for 5 minutes - I've seen that example and it is NOT the issue that this thread is about - though may indicate some other problem - but please ignore it for this discussion.

When you say that you are not seeing parallel session exceeded errors - do you mean that the community haven't recently provided such examples, or do you mean that VM engineers are actively monitoring VM servers for that particular error and that they used to see such errors logged but no longer see any now?

Maybe VM engineers should try monitoring for when VM servers reject a connection attempt from 123.  This is the problem that needs to be addressed regardless of whether a parallel session limit exceeded log message is being generated.

It is disappointed that after 10 weeks of having this problem, that we seem to have got nowhere. 

Alan

[ravenstar68]

@Kev_B 

There's 14 minutes between the mailcore.me server receiving and then the next step where we receive it. Tim - does this point to a delay at the receiving exchanger or the sending one? I'm happy to pass these headers along, but since we're no longer showing the parallel sessions exceeded errors then I'd like to be sure that the delay is happening at mailcore.me or when it arrives at our exchanger.

As a general rule, as soon as an MTA receives the mail, it tries sending it on immediately, so if we see a delay between one step and the next, the delay is often caused by the receiving server.  The sending mail administrator would also be able to see the 4xx errors from the receiving server in the logs.  The receiving mail admin can also check to see the 4xx errors in their logs as well.

@agpvm - I'd had anonymised headers shared with me, but I would need permission before I share them.

However I would say that 9fingers has shared Some headers since that shows a 2 hour delay passing from 123 to Virgin Media's mail exchangers.  So I do agree that the problem is definitely there.  But it does show that not every delay is down to the same thing.  @Kev_B I'll PM you the header in question.

Tim

[Kev_B]

@agpvm 

Thanks for taking the time to explain what you're seeing Alan, really appreciate that.

When I said we couldn't see the parallel session limit errors I was just referring to the example headers I'd received recently. When this issue started every header showed that error, but I've not seen it since the limits were amended (barring anything that's in my inbox at the moment).

One of our email team did monitor the traffic for a period and made a couple of adjustments independently of my request. I'll go back with the recent findings and headers and see if we can pinpoint the issue further.

[agpvm]

@Kev_B

Hi Kev,

It's actually quite rare for us to see a parallel sessions limit error even though I assume they are happening a lot of the time.  This is because:  i) 123 rarely passes the errors back to the sender - usually 123 just keeps quietly re-trying, and usually eventually succeeds;  ii) the errors are reported to the sender, not to us  the receivers of the emails - most senders won't bother to tell us when they received an error message.

Mostly what we see is just emails taking a long time to be delivered from 123 to VM, without us or the senders seeing any error message.  The headers in those messages just show that 123 struggles to pass the message to VM.

But I believe that the parallel sessions limit errors are the "smoking gun" - giving us a clue to why the delays are occurring - especially as they showed the limit was 20 which seemed very low.  Presumable VM engineers have tried increasing that limit.  They surely can monitor to see if they are still rejecting connections from 123, either due to the parallel sessions limit, or possibly some other reason.

Alan

[3_g]

What's the latest on this issue? @KevB

Delays are still happening, and if anything are getting worse.

Here is an email delayed yesterday.  From my interpretation of the headers there is a 5.5 hour delay between Virgin Media accepting the email from 123-Reg (mx4.tb.ukmail.iss.as9143.net receiving from mailex.mailcore.me) and a further 5 minute delay within Virgin Media (md9.tb.ukmail.iss.local receiving from smtpclienthelo)

I don't analyse all the emails and headers, and this is not an isolated case. 

Spoiler

Delivered-To: $$$$$$$@blueyonder.co.uk

Received: from md9.tb.ukmail.iss.local ([212.54.57.71])

by mc44.tb.ukmail.iss.local with LMTP id mEdmAP5WHl6+TQAA7mw0Ow

for <$$$$$$$@blueyonder.co.uk>; Wed, 15 Jan 2020 01:04:14 +0100

Received: from smtpclienthelo ([212.54.57.71])

by md9.tb.ukmail.iss.local with LMTP id ID5mJ/1WHl5qBQAABc+m+w

; Wed, 15 Jan 2020 01:04:14 +0100

Authentication-Results: ukmail.iss.as9143.net;

 spf=none (94.136.40.142; $$$$$$$.co.uk);

 dkim=pass header.d=email.strava.com;

 dmarc=none header.from=strava.com (dis=no_record);

X-Env-Mailfrom: srs0=pzaqqd5rew=3d=2+z9410ab5bfp7nh@ $$$$$$$.co.uk

X-Env-Rcptto: $$$$$$$@blueyonder.co.uk

X-SourceIP: 94.136.40.142

X-CNFS-Analysis: v=2.3 cv=f+s2+96M c=1 sm=1 tr=0 cx=a_idp_d

 a=s1hRAmXuQnGNrIj+3lWWVA==:117 a=NJNoIObYlB/4mjJSX2j46A==:17

 a=IkcTkHD0fZMA:10 a=Jdjhy38mL1oA:10 a=OsfJwRc1M5QA:10 a=MWnK1AarAAAA:8

 a=1XWaLZrsAAAA:8 a=Ug6ySvPiAAAA:8 a=t-IPkPogAAAA:8 a=OaXAtWZrAAAA:8

 a=3j4BkbkPAAAA:8 a=JqEG_dyiAAAA:8 a=BYtrmIwOAAAA:8 a=BkpHzsgbAAAA:8

 a=VVeT47NKAAAA:8 a=aue7zkMoYA5UKjHyZ-IA:9 a=QEXdDO2ut3YA:10 a=-FEs8UIgK8oA:10

 a=NWVoK91CQyQA:10 a=0zlweaWEAAAA:8 a=wibOD-iIAAAA:8 a=JB6VtYVu5ZfSMxtNAB0A:9

 a=vw6iwVVO5kbN4bvWlfsksWyUBTs=:19 a=N_K_M_A7JfmVSLrH:21 a=frz4AuCg-hUA:10

 a=_W_S_7VecoQA:10 a=Gkm4cp9a2kcA:10 a=F35pNLLn1F7ZugcVQCRp:22

 a=qMWJMbTbSoGuja_d3_bY:22 a=BoIfj7G3OZOUBUCdTYW0:22 a=gG9K73sxWPwKcekTYlAt:22

 a=ag8x4kDO9xKyHf3OLBXg:22 a=Y_njGK6ubMN9Bv3oJazI:22

Received: from mailex.mailcore.me ([94.136.40.142])

by mx4.tb.ukmail.iss.as9143.net with ESMTP

id rW5QiUF2riCTOrW5QixBSD; Wed, 15 Jan 2020 00:59:04 +0100

Received: from o2.email.strava.com ([167.89.2.22])

by smtp13.mailcore.me with esmtp (Exim 4.92.3)

(envelope-from <bounces+33699-7497-strava= $$$$$$$.co.uk@email.strava.com>)

id 1irPsI-0000IZ-Nq

for $$$$$$$@ $$$$$$$.co.uk; Tue, 14 Jan 2020 17:21:09 +0000

[emailwoes]

Agreed about getting worse, I was having 5hr delays yesterday.

[Kev_B · Accepted solution]

Hi all,

Apologies for the delayed response on this one.

I've heard back from our email team. They've doubled the rate limits on some of the IPs coming into our mail exchange, they've advised this should've taken care of a lot of the delays that have been happening.

How have things been over the past few days for everybody? If there are still delays, could you send me some new headers please so I can evidence these if I need to pass anything back to the email team.

[9fingers]

Kev, I've been monitoring sporadically and it seem much better. Difficult to say "yes it is fixed" but hats off to the email team for the support so far.
I will keep monitoring and posting here.

Bob