Virgin Media DNS servers completely dropped from our website DNS zones

[PMilne87]

Hi there, 

I work at Marys Meals International and as of late Thursday evening, 1st Feb, Virgin Media and another US ISP appear to have been completely dropped from our website's DNS Zones. What should happen is if it doesn't have a zone, then it should query the root DNS servers globally to pull down a copy of the zone and cache it. But that's not happening, so I suspect their DNS zones have got corrupted somehow at Virgin (or in their upstream provider). That is what happened with Sky last year.

This has probably been a maintenance/update on your side which has gone awry.

Our DNS Provider has website called DNS Hospital that you can use to query what otherDNS servers return. I'll past the link below. It doesn't have a valid SSL certificate (never has had) so you will get a warning that anything entered in the site isn't secure 

If I input your Virgin DNS servers:

No results at all

https://www.dnshospital.com/cgi-bin/dig.cgi?d=marysmealsusa.org&n=194.168.4.100&q=ANY

Google DNS shows full listings: https://www.dnshospital.com/cgi-bin/dig.cgi?d=marysmealsusa.org&n=8.8.8.8&q=ANY 

Thousand Eyes Outages map shown that "Level 3 - A Major USA DNS Provider Has Issues". Are these incidents related and can you please re-run your DNS Servers to be added back onto our Zones?

Affected sites:

• www.marysmeals.au 
• www.marysmeals.ba 
• www.marysmeals.ca 
• www.marysmeals.ch 
• www.marysmeals.es 
• www.marysmeals.fr 
• www.marysmeals.ie
• www.marysmeals.it 
• www.marysmeals.nl
• www.marysmeals.pl
• www.marysmealsusa.org

[Client62]

Public DNS are (still) resolving the host names
however the authoritative DNS servers are not available.

https://www.nslookup.io/domains/marysmeals.fr/dns-records/#authoritative

[PMilne87]

Please see above CMD Prompt and below email from Blackbaud Online

[MarysMeals]

The SOA and NS records for the domain are showing across the 3 major providers listed at nslookup.io and dnschecker.org. Why would the authorative tab be empty on nslookup.io?
https://dnschecker.org/#SOA/marysmeals.fr
https://dnschecker.org/#NS/marysmeals.fr

[Client62]

The nslookup.io Authoritative tab may be empty because the specified Name Servers are not responding.

If I try these commands they both time out :-

nslookup marysmeals.fr ns2.bb-online.org

nslookup marysmeals.fr ns1.bb-online.net

[MarysMeals]

That's very odd. I get responses from both these servers when querying them directly using a local nslookup tool.
I wonder if there's any DDOS issues

[Client62]

What is also unusual is looking at nslookup.io for marysmeals.fr the public DNS are showing Revalidate in 5mins,  just as if they have all expired and are re-trying a refresh at short intervals.

Below are the timeout I see :

C:\Users\Philip>nslookup marysmeals.fr ns2.bb-online.org
DNS request timed out.
timeout was 2 seconds.
Server: UnKnown
Address: 185.206.123.20

DNS request timed out.
timeout was 2 seconds.
DNS request timed out.
timeout was 2 seconds.
DNS request timed out.
timeout was 2 seconds.
DNS request timed out.
timeout was 2 seconds.
*** Request to UnKnown timed-out

C:\Users\Philip>
C:\Users\Philip>
C:\Users\Philip>nslookup marysmeals.fr ns1.bb-online.net
DNS request timed out.
timeout was 2 seconds.
Server: UnKnown
Address: 185.206.123.1

DNS request timed out.
timeout was 2 seconds.
DNS request timed out.
timeout was 2 seconds.
DNS request timed out.
timeout was 2 seconds.
DNS request timed out.
timeout was 2 seconds.
*** Request to UnKnown timed-out

[MarysMeals]

The TTL on that domain has been lowered pending a website migration which could account for that revalidation you're seeing, but I'm still unsure why those BB-Online name servers are not showing as unreachable (same output below from 2 different ISPs outwith Virgin Media):

===========(Trooli DNS)

C:\Users\Stephen>nslookup marysmeals.fr ns1.bb-online.org
*** Can't find server address for 'ns1.bb-online.org':
Server: UnKnown
Address: 2a04:204:291f:c700::1

Non-authoritative answer:
Name: marysmeals.fr
Address: 52.18.15.4

C:\Users\Stephen>nslookup marysmeals.fr ns2.bb-online.org
Server: ns2.bb-online.org
Address: 185.206.123.20

Name: marysmeals.fr
Address: 52.18.15.4
===============

===============(Cisco DNS)

C:\Users\Stephen>nslookup marysmeals.fr ns1.bb-online.org
*** Can't find server address for 'ns1.bb-online.org':
Server: dns.sse.cisco.com
Address: 208.67.222.222

Non-authoritative answer:
Name: marysmeals.fr
Address: 52.18.15.4

C:\Users\Stephen>nslookup marysmeals.fr ns2.bb-online.org
Server: ns2.bb-online.org
Address: 185.206.123.20

Name: marysmeals.fr
Address: 52.18.15.4

===================

Are you querying these nameservers from within the Virgin Media network?

[legacy1]

if I run nslookup on o2 I get

C:\Users\_>nslookup marysmeals.fr ns2.bb-online.org
Server: ns2.bb-online.org
Address: 185.206.123.20

Name: marysmeals.fr
Address: 52.18.15.4

C:\temp>tracetcp ns2.bb-online.org:53

Tracing route to 185.206.123.20 [ns2.bb-online.org] on port 53
Over a maximum of 30 hops.
1 2 ms 2 ms 1 ms 192.168.255.247
2 19 ms 20 ms 18 ms 192.168.8.1
3 * * * Request timed out.
4 * * * Request timed out.
5 * * * Request timed out.
6 * * * Request timed out.
7 Destination Reached in 46 ms. Connection established to 185.206.123.20
Trace Complete.

on VM

C:\Users\_>nslookup marysmeals.fr ns2.bb-online.org
DNS request timed out.
timeout was 2 seconds.
Server: UnKnown
Address: 185.206.123.20

DNS request timed out.
timeout was 2 seconds.
DNS request timed out.
timeout was 2 seconds.
DNS request timed out.
timeout was 2 seconds.
DNS request timed out.
timeout was 2 seconds.
*** Request to UnKnown timed-out

C:\tracetcp_v1.0.3>tracetcp ns2.bb-online.org:53

Tracing route to 185.206.123.20 [ns2.bb-online.org] on port 53
Over a maximum of 30 hops.
1 10 ms 7 ms 8 ms 10.112.32.133
2 10 ms 10 ms 10 ms 80.1.81.185 [basl-core-2b-ae63-650.network.virginmedia.net]
3 * * * Request timed out.
4 * * * Request timed out.
5 * * * Request timed out.
6 19 ms 19 ms 19 ms 62.254.85.86 [86.85-254-62.static.virginmediabusiness.co.uk]
7 * * * Request timed out.
8 31 ms 21 ms 45 ms 130.117.51.73 [be2348.ccr41.lon13.atlas.cogentco.com]
9 30 ms 27 ms 34 ms 154.54.56.94 [be12194.ccr41.ams03.atlas.cogentco.com]
10 35 ms 33 ms 89 ms 130.117.0.122 [be2813.ccr41.fra03.atlas.cogentco.com]
11 43 ms 40 ms 41 ms 154.54.36.54 [be2959.ccr21.muc03.atlas.cogentco.com]
12 45 ms 45 ms 42 ms 130.117.0.17 [be3072.ccr51.zrh02.atlas.cogentco.com]
13 44 ms 44 ms 46 ms 154.54.39.158 [te0-0-1-3.agr11.zrh02.atlas.cogentco.com]
14 46 ms 44 ms 44 ms 154.25.7.66 [te0-0-2-2.nr11.b046110-0.zrh02.atlas.cogentco.com]
15 49 ms 46 ms 48 ms 146.228.52.155
16 46 ms 43 ms 52 ms 94.177.122.250 [vbgp01.4b42.net]
17 * * * Request timed out.
18 * * * Request timed out.
19 * * * Request timed out.
20 * * * Request timed out.
21 * Destination Reached in 45 ms. Port closed on 185.206.123.20
Trace Complete.

[asim18]

The auth servers ns1.bb-online.net & ns2.bb-online.org are responding to ICMP ping, meaining it's definitely reachable and routing OK from my VM connection. However DNS querying the servers using port 53 is timing out.

Very weird indeed.

Maybe this has something to do with it: https://docs.aws.amazon.com/Route53/latest/DeveloperGuide/routing-policy-geo.html

"Geolocation works by mapping IP addresses to locations. However, some IP addresses aren't mapped to geographic locations, so even if you create geolocation records that cover all seven continents, Amazon Route 53 will receive some DNS queries from locations that it can't identify. You can create a default record that handles both queries from IP addresses that aren't mapped to any location and queries that come from locations that you haven't created geolocation records for. If you don't create a default record, Route 53 returns a "no answer" response for queries from those locations."