Ravenstar's Email FAQ's

[ravenstar68]

The aim of this thread is to collate some of the stickies I have made in the past into one thread.

In here you will find the following FAQ's

• Understanding Email Block Lists
• SPF, DKIM, and DMARC explained
• Generic/Dynamic host names prohibited - an explanation
• Why EVERYONE should use SSL for their email
• SMTP Error - 530 Authentication Required (VM401) Explained
• Non Delivery of Emails to Virgin Customers

[ravenstar68]

               How Email Block lists Work

One of the common complaints encountered across the web is that people check their own IP address and find that it is on one or more mail block lists. This can lead to some consternation as people take this to mean that their IP has been delivering spam. However delving into the world of email reveals a few truths about block lists and how they work.

Understanding Email Delivery

When an email is sent by a user of a mail service (either an ISP's or a dedicated email provider) It follows a defined process.

 The sender sends a mail through their providers Mail Submission Agent (MSA) – Ideally this should be done using SMTP authentication although some ISP's still allow their own IP addresses to send mail without authenticating.

The Mail is then transferred using the Providers Mail Transfer Agent (MTA), if it is for another of the mail providers users then it is delivered to their inbox, if not then the Mail is sent to the Mail Exchanger (MX) of the recipients email domain and finally delivered on to the recipient.

Because of the way email works the transfer to the MX has to be done without authentication. This would seem to be a weak point for spammers to attack, and this is where block lists come into play.

Block Lists block direct connections only.

Block lists are used by Mail Exchangers to determine if they should allow a direct connection from an individual IP.

As previously stated Mail Exchangers are a potential weak link for the entry of spam into a network. With spammers being able to set up on any address and send directly to an MX server, there has to be a way of identifying bogus mail sources. Block lists work in one of two ways.

Identifying addresses that should not be connecting directly – Policy Block Lists (PBL)

Identifying addresses that are known to send spam - Spam Block Lists (SBL)

Policy Block Lists

These are IP addresses that should not be connecting directly to mail exchangers.

Most residential customers of ISP's will find themselves on at least one of these. Two of the most well known are Spamhaus PBL and SORBS DUHL.

It's important to note that being on a PBL DOES NOT mean that your IP has been sending spam.

These zones in theory should list all public addresses assigned to customers via DHCP, (Spamhaus has also suggested it would also like to see all static IP's which do not host mail servers in this list as well). Therefore if you are on a PBL you should note that this is nothing to be worried about, it won't affect you sending mail.

Note also that if you are listed on on either PBL then your address may also appear in the master zones Spamhaus zen and dnsbl.sorbs.net, again provided you are sending through a legitimate mail service, this won't affect your mails as it's the Mail Transfer Agent's address that is being checked against the block list and not your own address.

Spam Block Lists

These are lists of IP's that are known to be sending spam. Often these are generated by cataloguing mail that is delivered to spamtraps as well as using aggregate spam reports from mail recipients. While finding your own IP on one of these is a little more concerning, especially if the report is current, again it shouldn't stop you sending mail through a legitimate email service.

If you have any questions about block lists please post them in a new thread and I will try to answer them for you.

Ravenstar68

[ravenstar68]

CBL and XBL Blacklist Entries

I recently had problems sending mail - in my case to Hotmail account.

What I found was that my Home IP had become blacklisted on the CBL (Composite Block List) see the site below for more info and a lookup can be done here.

http://cbl.abuseat.org/ or here

http://www.spamhaus.org/lookup/

Spamhaus XBL includes CBL entries so I would recommend using their check.

While I don't normally advocate checking if your home IP is on a blacklist.  I do recommend checking this one.  As if you are on the CBL or XBL it would suggest a device on your home network is hosting malware.

Here's an example of a normal Spamhaus Listing

Blocklist Lookup Results

77.100.xxx.xxx is not listed in the SBL

77.100.xxx.xxx is listed in the PBL, in the following records:

• PBL182609

77.100.xxx.xxx is not listed in the XBL

The middle entry is normal  and so can be safely ignored.

However if you see

77.100.xxx.xxx is listed in the XBL, in the following records:

Then you need to click the link underneath.

This will take you to a page detailing why the IP has been listed and giving advice on how to deal with the problem.

In the case of the CBL there is also a link to click to de-list your IP, however it is important you check all PC's in your home for infection first, as if you don't rectify the problem your IP could be re-listed.

Note the CBL uses various methods to identify compromised IP's, and does de-list them when activity from that IP stops for a long enough period of time.

With that in mind it's important that an XBL entry is not just dismissed, Virgin IP's are very sticky (I've have mine for a few months now).  So if you are on the XBL or CBL you should check EVERY computer on your home network.

In my case getting my IP removed from the CBL did solve my email problem, and I was able to get mails through to Hotmail again.

Ravenstar68

[ravenstar68]

This is a plea for everyone to upgrade their email settings to use the SSL ports when collecting and sending mail.

Old standards

When the mail protocols were created in the 1980's the internet was a much smaller and much less hostile place.

As such little thought was given in the original standards to security of logins and passwords.

This meant that when you sent an email username and password to log in to your mail server they are actually visible to anyone who can eavesdrop on the web traffic.

Consider the exchange here between a pop3.blueyonder.co.uk and myself. (Server resonses are in red)

telnet pop3.blueyonder.co.uk 110

+OK Virgin Media POP3 server ready [ e4c558782BY ].
user myaddress@blueyonder.co.uk
+OK send PASS
pass mypassword
+OK Welcome.
quit
+OK Farewell.

Obviously I'm not posting my real address and password on here. 😉

The point here is, that although I am using telnet here to connect to the server - your mail client does exactly the same thing.  And the username and password is sent as you see them on the screen.

Collecting your email from home - the risks might not seem that great, but when you start to use public wifi with phones/tablets and Laptops logging in to grab mail when they connect the risks go up.

While many email servers support MD5 passwords which would offer some protection.  Most email providers still use clear text authentication.

Addressing the Weakness

The email standards were updated to take account of this vulnerability as long ago as 1999 with TLS (Previously known as SSL) being added.  This required new ports 995 for POP3, 993 for IMAP and 465 for SMTP - the last while still in use has actually been deprecated in favour of port 587 using StartTLS.

The outcome of this change is that email providers can use clear text authentication safely as the connection is now encrypted so usernames and passwords are protected from prying eyes.

Virginmedia.com and SSL

On it's own named servers Virginmedia has been using SSL more or less ever since it took over the mantle.  By those I mean:-

pop3.virginmedia.com  - Port 995 SSL

imap.virginmedia.com  - Port 993 SSL

smtp.virginmedia.com   - Port 465 SSL

These are the ONLY settings that these servers can use. So with these there is no issue regarding security.

Users of the legacy servers however have a choice.  Virgin chose not to enforce the changes upon users of those servers, in order to prevent Tech support headaches,  and after witnessing the furore caused by the recent issue with smtp authentication - who can blame them.

Just because you can continue to use the old settings though, it doesn't mean you should do so.

People might argue that so long as it works there's no need to change.  However would you leave your front door open.  most people wouldn't

To a hacker people surfing the net using insecure settings is as bad as leaving the door open.

In most cases changing ports is fairly painless.  The server names themselves don't need to change.

With that in mind I would urge everyone using blueyonder, ntlworld, and virgin.net to check ALL their email clients and update them.  Virgin have posted the updated settings as long ago as 2011.  Make 2015 the year you close one door to would be hackers.

Ravenstar68

[ravenstar68]

SMTP ERROR 530 What it means

SMTP error 530 is defined as meaning Access Denied.  When this is sent by a server the server sends additional information as to why it's been denied.  Two common reasons are blacklisting and authentication issues.

Blacklisting should not be a problem for home users, I've only included that here for completeness.

The message itself explains the reason  in this case.

Authentication Denied (VM401)

Unfortunately mail client error messages can somtimes confuse the issue,  Take a look at this one from Outlook

The message could not be sent because the server rejected the sender's e-mail address. The sender's e-mail address was someaddress@ntlworld.com'.

Subject 'hhh'
Server Error: 530
Server Response: 530 Authentication Required (VM401)
Server: 'smtp.ntlworld.com'
Windows Live Mail Error ID: 0x800CCC78
Protocol: SMTP
Port: 25
Secure(SSL): No

At first glance it looks as if the problem was with the senders email address.  However looking further down we see Authentication Required.  In other words - it's not the email address that's causing the issue.  It's the fact that the mail client has not authenticated beforehand.

What do I mean?  Lets talk to an email server in this case smtp.virginmedia.com

220 know-smtprelay-10-imp bizsmtp ESMTP server ready
ehlo desktop
250-know-smtprelay-10-imp hello [77.100.xxx.xxx], pleased to meet you
250-HELP
250-AUTH LOGIN PLAIN
250-SIZE 52000000
250-8BITMIME
250-STARTTLS
250 OK
mail from: <myaddress@blueyonder.co.uk>
530 Authentication Required (VM401)

 Lines in blue are messages from the server back to me.  The mail from: line is me sending out the address that the mail is coming from.  The server refuses it because I failed to authenticate first.  So lets try again.

220 know-smtprelay-10-imp bizsmtp ESMTP server ready
ehlo desktop
250-know-smtprelay-10-imp hello [77.100.xxx.xxx], pleased to meet you
250-HELP
250-AUTH LOGIN PLAIN
250-SIZE 52000000
250-8BITMIME
250-STARTTLS
250 OK
auth login
334 VXNlcm5hbWU6
bXlhZGRyZXNzQGJsdWV5b25kZXIuY28udWs=
334 UGFzc3dvcmQ6
bXlwYXNzd29yZA==
235 ... authentication succeeded
mail from: myaddress@blueyonder.co.uk
250 <myaddress@blueyonder.co.uk> sender ok

 This time I send an auth login command and the server asks for my username and password the request and response are sent in base64 encoding - note that encoding is not the same as encryption, the text can be read by going to a base64 decoder.

Note to mods - My username and password have been removed.  The encoded text simply reads myaddress@blueyonder.co.uk and mypassword.

Now when I send the mail from: command the address is accepted and I can go on and complete the exchange and send my mail should I wish to.

I've included the above as a background.  The long and the short is that by turning on email authentication in your client you should be able to send your mails.

But I could send mails before - what's changed?

Technically nothings changed smtp.blueyonder.co.uk, smtp.ntlworld.co.uk and smtp.virgin.net have had authentication switched on since at least 2011.

However in the interests of making things easier for the customer the old settings were also left in place.  This means that if you connect from a Virgin address and you were using the above servers then authentication is not needed.

However if you try to send from outside the Virgin network you still need to authenticate.

Whit the changeover of ADSL users from Virgin to TalkTalk Virgin appear to have removed some IP blocks from the lists of addresses allowed to send without authenticating.  Unfortunately some of the IP ranges were in use by blueyonder and ntlworld customers as well.  I say some, because despite the threads complaining of this.  I was able to connect and send mail unauthenticated to smtp.blueyonder.co.uk - although I should add this was just for testing.  I normally use SMTP authentication in ALL my clients.

Virgin have since restored the functionality so ntlworld and blueyonder users should now be able to connect.

So I can leave my settings as they are?

If you are using outdated settings it's really advisable to update them to the recommended settings for two reasons:

1. Should the problem that's caused this issue recur, then you shouldn't be affected as you are already authenticating (the smtp server won't complain if you authenticate when you don't need to).
2. Outdated settings pose a security risk.

As already stated the settings have been on Virgin's website since at least 2011 so a quick Google search will bring them up.  However here the recommended outgoing settings:

Server

• smtp.blueyonder.co.uk, smtp.ntlworld.com, smtp.virgin.net - to match the end of your email address - OR
• smtp.virginmedia.com - can currently be used for all four VM domains.

Port: 465

Security: SSL (Outlook) or SSL/TLS (Thunderbird) Other clients should show one format or another.

Authentication: Enabled This is found in more settings in Outlook.  On Macs and Ipads/Iphones this should be set to password.

Username and Password: same as incoming

Note that if you use POP3 with recent: before you username to collect mail, then you need to configure the smtp settings to login using your email without the recent: as the username.

Ravenstar68

[ravenstar68]

Finding the correct Email settings

Finding the settings for your email domain can be found by going here.

The page also contains advice on how to set up some of the most common email clients.

[ravenstar68]

Non Delivery of emails to Virgin Customers

I'm going to start this post with a disclaimer. While I am a Virgin Superuser - I do not work for Virgin Media. All opinions posted here are my own, however I will clearly delineate between my opinions and the facts where appropriate.

While users have been deluged with Spam it has been clear from day one that some mail has not been getting through to the end users. That senders are now having to post on these boards detailing their problems is proof of that.

This post aims to educate Virgin Email users as to one of the major causes of non-delivery, and help get genuine mail flowing into their email folders again.

Fact: Virgin Default Email Settings
Virgin made a choice when setting up the new email system to set the default rules for mails that are deemed to be spam to be rejected by their email system.

While I sincerely believe that this was done with the best of intentions in mind, it was a flawed choice for the following reasons.

• It put 100% faith in the spam filters to be working correctly from day one.
• It does not allow the training of spam filters to understand false positives.
• Most importantly it means that customers DO NOT have sight of any misclassified mails.

This choice has led to frustration from customers who are not receiving their mails, and for legitimate senders who are getting their mail bounced back with the message "5.5.4 Spam Content Found."

The Immediate Fix

While Virgin may or may not be able to force a blanket change from their end, and I am trying to get an answer on this. Users SHOULD be able to do something to get mail flowing back to them.

1. Log into webmail - preferably from a desktop PC
2. Click the cog in the top right of the screen then click settings
   
3. Click Spam settings and then change the radio button to Move message to Spam folder
   
4. BEFORE closing settings change the category on the left to any other category. If you fail to do this the SETTING WILL NOT BE CHANGED. I spent half an hour yesterday trying to figure out how to get the change to stick on a new dummy account I created.

Why Move to Spam Folder?

One of the things we need to be able to do is to train the Spam filters. This is the ONLY setting that will allow us as users to be able to do this. Mail in the Spam Folder can be marked Not Spam by clicking the thumbs up icon. While it sometimes takes multiple markings to actually get the spam filters to pick up the change. The majority of my genuine blueyonder mail does come to my inbox (there are still a few notable holdouts such as some of the Twitter info mails ).

I've changed the setting but my mail is still being bounced!

If after changing the settings, you find that the mail is still not getting to you. Try and get the sender to send you a copy of the bounce message to verify the reason the mail is being bounced and start a new thread. This solution will ONLY deal with bounces with the "5.5.4 Spam content found message."

If the sender is getting this message and you've changed the settings then Virgin will need to look at your account, as some users report that although the settings were changed, and have provided screenshots to demonstrate this, the new settings haven't always taken effect.

Closing comments.

Virgin do bounce incoming mails for a number of other reasons. Two notable ones are

• Senders IP is in the RBL - that is blacklisted, in that instance it's up to the sending mail provider to find out why their mail server is in the RBL
• SPF Check failed - This is a contentious one. SPF is used to verify that the sender sent mail through a server that is authorized to send for that domain. Unfortunately, it is broken for example if you have your own domain and then forward the mails on to another email provider.

In those instances, the fix detailed above will not stop the mail being bounced - which is why if the mail is still not getting through after doing the above, Virgin need a copy of the bounce message from the sender.

This post will be locked - any queries about this should be made in a new thread.

Ravenstar68

[ravenstar68]

SPF, DKIM, DMARC Explained

The purpose of this post is to explain 3 technologies that are increasingly being used in email.  It aims to look at the pro's and cons of the technologies as well.

I'll cover SPF and DKIM and then move on to DMARC as the latter actually requires the former two technologies in order to work.

General Introduction

SPF, DKIM and DMARC are all designed to protect domains emails from being spoofed by third parties.  They are are all implemented at some level by adding DNS records by the domain administrators.  Although DKIM does also use cryptographic methods to sign an email to verify the emails integrity.

Broadly speaking.

SPF announces the mail servers that should be delivering mail for a domain DIRECTLY to the destination mail servers.

DKIM embeds a signature in the mail to prove that the mail is genuine.

DMARC provides instruction to the receiving server, as to what to do should a mail fail SPF and/or DKIM.

Message Envelopes

Before going any further I also need to discuss message envelopes as these are important in SPF.
Just like real mail.  When an email is sent  a message envelope is constructed.  This is similar to when you address an envelope in the real world, except that your email program does this automatically.

The message envelope is constructed by sending two commands to the mail server.

mail from: <senders-address>
rcpt to: <recipients-address>

While in most basic e-mails the senders-address will be the same as the From: address in the email and the recipients-address will be the same as the To: address in the email - there are times when this is not the case.  This is something that spammers know and can take advantage of.

The important thing to note is that mail is delivered to the <recipients-address> and not to the To: address in an email.  Likewise bounced mail is returned to the <senders-address> and not the From: address.

SPF was designed to use the domain in the <senders-address> when evaluating if mail is coming from an appropriate source.

DKIM and DMARC on the other hand both use the domain in the From: address.

SPF - SENDER POLICY FRAMWORK

As discussed above SPF is a way for domain administrators to let the world know which servers deliver their outbound mail.  The administrator publishes a text record under the bare domain.

For example the domain example.com might publish the following TXT record

"v=spf1 a -all"

In this case the record states that the mail server is found on the same address as indicated by the A record lookup for example.com.  The -all at the end indicates that no other addresses should be used.

So if someone sends me a mail.  My inbound server looks at the <senders-address> discussed above.  It takes the domain and then does a DNS lookup to see if an SPF record exists.

If so it then does another DNS lookup for the A record to get the IP address.

Finally it does a comparison of the IP address and the IP address of the server that it's talking to.  If the two match then the SPF passes.  If the two differ then  the server continues to evaluate the next entry on the SPF string.  In this case the next entry is -all which tells it that all other addresses should be failed.

The SPF record Syntax is discussed in more detail here

http://www.openspf.org/SPF_Record_Syntax

Possible Final Outcomes

The last parameter provides for a number of possible outcomes should the mail server not match any addresses provided by the SPF record.

+all - pass the mail.  Effectively saying "I don't see any value in SPF."
?all - neutral result (this is the default)  It allows the mail through without saying anything about it's authenticity.
~all - soft fail.  Mark the mail as failed but don't act on the failure.  Useful when testing SPF records.
-all - Hard fail mark the mail as failed and act appropriately.

SPF and Email Forwarders.

Because SPF tells a server which mail servers should be directly delivering a domains mail to it.  SPF fails when mail is passed to it's final destination via a forwarding service.  One possible suggestion to overcome this was that Forwarders rewrite the <senders-address> when forwarding the mail.  While some forwarders do this.  Many do not meaning SPF on it's own only has real value when mail is being delivered directly to the recipient's email address.

Server response to SPF fails.

Up til the creation of DMARC, there was no mechanism in place for telling recipient servers what to do if an SPF check failed.  It was left to the administrators of the receiving domain's email servers to decide the best policy.  Most providers record the result of the SPF check in the headers, although Virgin does not at this time.  Other possible solutions were to quarantine failed mail, deliver it to the spam folder or reject it altogether.

SPF means attitudes of Senders needs to change.

One common thing a lot of people do is use the outgoing servers of their current ISP to send mail for a number of different email addresses.  In the past this was not an issue.  In fact it's the way the mail standard was built.  Unfortunately the reality of the need to protect domains from spammers has changed that.

When sending mail from any email address - senders need to use the smtp servers associated with that email address - ESPECIALLY when the address has an SPF record deployed.  Failure to do so will mean the mail fails SPF checks and is more likely to be marked as spam in the first instance.  If the domain has a DMARC reject policy deployed, the mail is likely to be rejected on those grounds as well.

DKIM - DOMAINKEYS IDENTIFIED MAIL

DKIM was an extension of a similar scheme created by Yahoo called DomainKeys.  DKIM proves the authenticity of an email by signing it with a DKIM Header.  This Header contains the following.

• A key to identify which record to use.
• Information as to any headers that are to be used to authenticate the mail.  As well as whether to use relaxed or canonical checking.
• A hash of the text of the bodies and any selected headers.

DKIM works in the following way.  The server administrator produces a pair of Cryptographic keys.  The Private key is used by the email server and the public key is published in a TXT record via DNS

When a mail is sent through the outbound server the text of the mail is parsed and a DKIM header added to the mail using the private key and the information in the body and chosen headers, as well as an index which tells it which particular DKIM record it should be using.

When the mail is received by an exchanger it looks at the DKIM record and then uses the domain in the From: address to look up the DKIM entry index._domainkey.<domain_name>  It will then parse the body and selected headers to determine if the signature in the DKIM header is valid.

Provided no changes have been made to the relevant parts of the mail, these checks will pass.  It is even possible to pass the mail through a forwarding service and the mail will still pass PROVIDED the forwarding service does not add anything to the body of the mail.

I say this because I've seen at least one example on these Forums where a mail has been rejected because it was being sent to an institution that forwarded the mail, but before doing do added their standard disclaimer to the bottom of the mail.  This caused the mail to fail DKIM, and as the sender had a DMARC reject policy set up, the mail was bounced by Virgin.

Like SPF, DKIM alone does not tell the recipient domain what to do when a mail fails checks.  It simply offers a pass/fail/neutral result.

DMARC - Domain-based Message Authentication, Reporting & Conformance

DMARC builds on the above two protocols.  It allows the reporting of SPF and DKIM failures as well as providing a means to announce a policy that advises recipient servers what they should do with failed mail.

Again this relies on DNS entries to work effectively.  And again uses the address in the From: address of an email in order to look up the DMARC record for a domain.

Lets's have a look at a real world DMARC entry - in this case yahoo.com

The DMARC record is published as _dmarc.<domain>

> set type=txt
> _dmarc.yahoo.com
Server:  cache1.service.virginmedia.net
Address:  194.168.4.100

Non-authoritative answer:
_dmarc.yahoo.com        text =

        "v=DMARC1; p=reject; pct=100; rua=mailto:dmarc_y_rua@yahoo.com;"

In this the record shows.

1. Version - v=DMARC1
2. Policy - p = reject in this case mails that fail DKIM or SPF should be bounced.
3. Percentage of mail to check - pct=100 in this case all of it.
4. Where to send aggregate failure reports.

Note that DMARC can also be used to send reports on individual mail failures as well.

DMARC is useful as it standardises the outcome of SPF and DKIM and puts the choice as to what to do with the failed mail back in the hands of the sending domain.  It should be noted that

DMARC on it's own is pointless.
DMARC with SPF alone will encounter problems where mail ends up being sent via forwarders.

For that reason IMHO DMARC is best deployed when the sending domain utilises BOTH DKIM and SPF.

Ravenstar68

[ravenstar68]

Generic/Dynamic host names prohibited

I'd like to assist businesses who are trying to send emails and are getting rejected with the above message by explaining it in more detail and what you can do to resolve this particular error.

Many ISP's might give a Generic host name to a customer's IP address.  This means that when you do a reverse lookup using dig or nslookup, you might get something such as:

Default Server: cache1.service.virginmedia.net
Address: 194.168.4.100

> set type=ptr
> 94.173.xxx.xxx
Server: cache1.service.virginmedia.net
Address: 194.168.4.100

Non-authoritative answer:
xxx.xxx.173.94.in-addr.arpa name = cpc35-sutt4-2-0-custx.xx-x.cable.virginm.net
>

(Note this is my home IP so I have masked some of it out somewhat.)

While the above is for a residential connection I have seen similar for static business connections on both bt and comcast.

If you are running a mail server however the PTR should not point back to a record of this form.  If it does Virgin reject the mail.  

While some other servers may accept that mail it should be noted that mail is more likely to end up in a customer's spam folder.

It is important when running a mail server that the ptr record for the IP address points back to the fully qualified domain name (FQDN) of the mail server.  The server should also quote it's FQDN in it's HELO/EHLO exchange.

For example if we take a real world example from a mail I recently received.

Received: from mail-qt0-f199.google.com ([209.85.216.199])

The mail server has greeted Virgin's server with it's FQDN and Virgin's server has also noted the IP address of the server.

Plugging the IP address in to nslookup.

> 209.85.216.199
Server:  cache1.service.virginmedia.net
Address:  194.168.4.100

Non-authoritative answer:
199.216.85.209.in-addr.arpa     name = mail-qt0-f199.google.com
>

When you are assigned a static IP address then you should by rights have been given control of the reverse DNS zone for the IP addresses - however if not you may need to ask your ISP to set the reverse DNS to match your servers FQDN and wait for it to propagate across the web.

Ravenstar68