Hub 5 crashes from RTSP stream

[Riyo]

Since the hub 4 to hub 5 swap I've began experiencing a fault which takes the entire internet down.

At first it was difficult to identify the cause. From my personal usage, it occurred when playing the game VRChat and I assumed that it was related to the game. However I've been working on narrowing down the cause and it seems worse than that.

First, I called an engineer out to check power levels (it's always "check power levels!") To be safe the virgin engineer checked and reduced power levels them so they're now certainly within range. However, the problem persisted.

Errors in the network logs are always the following:

16-04-2023 00:22:53	critical	Cable Modem Reboot due to RG crash
16-04-2023 00:18:22	critical	SYNC Timing Synchronization failure - Failed to acquire QAM/QPSK symbol timing;;CM-MAC=**:**:**:**:**:**;CMTS-MAC=**:**:**:**:**:**;CM-QOS=1.1;CM-VER=3.1;
16-04-2023 00:18:11	critical	Cable Modem Reboot due to RG crash
16-04-2023 00:13:48	critical	SYNC Timing Synchronization failure - Failed to acquire QAM/QPSK symbol timing;;CM-MAC=**:**:**:**:**:**;CMTS-MAC=**:**:**:**:**:**;CM-QOS=1.1;CM-VER=3.1;

This does not appear to be an isolated issue with the hub 5. The following similar issues were found:
https://community.virginmedia.com/t5/Gaming-Support/Connection-issues-with-VRChat-RG-crash-Packet-lo...
https://community.virginmedia.com/t5/Gaming-Support/SYNC-Timing-Synchronization-failure-while-in-vr/...

The pattern that I noticed was that this crash occurred when I was in VRChat worlds playing an RTSP music stream. Music streams are popular in this game. After the internet goes down, If I left the game client open, even as I had disconnected from the instance (and all other players in it) the hub 5 would still keep crashing because the stream was still trying to load in my own (private) world.

In the latest incident the music stream was an RTSP stream on TCP 554. The URL host being: https://stream.vrcdn.live

After noticing this pattern I tested a stream using this URL in VLC media player completely outside of VR. It crashed the hub 5 with the exact same symptoms.

I tested an HTTPS .ts of the exact same stream and it loaded instantly and ran fine. This seems to be entirely due to the RTSP stream.

Please advise. If this is a flaw with the Hub 5 is requires a critical fix. A protocol-phobic router is, unsurprisingly, a BIG problem.

[eth0]

I've been having exactly the same issue!

It's an issue with the Hub 5 firmware LG-RDK_4.6.24-2206.7

Specifically the Application Level Gateway (ALG) process is detecting RTSP traffic in an effort to start opening ports to make the RTSP stream more efficient. It's most common for SIP traffic and has caused security issues in the past

The ALG is failing to properly understand the RTSP stream and is crashing catastrophically, probably causing a kernel panic on the router

For the Virgin engineers, it's nice and easy to replicate with this command from Linux. Running this on any machine on a network behind a Hub 5 will trigger the crash, reboot the router and disconnect the internet for 5-10 minutes.

echo "SETUP A" | nc portquiz.net 554

On some routers, you can disable the ALG in the admin interface. But not on the Hub 5, or any of the Virgin routers.

There are two possible immediate fixes until Virgin deploy a new firmware:

Block 445 Outbound:

In your router admin page: Advanced Settings > Security > IP and Port filtering > Create a new rule
The rule should then be set to:

Source address: All, Destination address: All, Protocol: TCP, Source port: All, Destination port: 554, Enabled (checked)

This will simply stop any RTSP streams from being established. In VRChat specifically you will lose world music, but everything else should keep working.

VPN:

Another option is to just use a VPN so the router can't see the RTSP traffic at all, so the ALG is never triggered. For normal streams any VPN type would be fine, but for VRChat you'll probably need something with minimal overhead like the Wireguard protocol.

Hopefully this helps :)!

[eth0]

I've been having exactly the same issue!

Looks like an issue with the Application Level Gateway (ALG) in Hub 5 firmware LG-RDK_4.6.24-2206.7

You're 100% right it's caused by RTSP streams in the background for music/video in VRChat, the router's ALG would normally try and open ports/speed up real time connections like SIP/RTSP, but in this case the ALG is extremely fragile and crashes even with well formed RTSP traffic.

Normally a crashing ALG isn't a huge issue, but the Hub 5 compounds the issue with these 3 points:

• ALG is easy to crash
• ALG is running with enough privileges to crash the entire router OS
• ALG can't be disabled on Hub 5

The crash can also be replicated with the following Linux command, without even needing to fully set up an RTSP stream. It simply sends a string starting with "SETUP" to port 554, immediately triggering the Hub 5 to crash.

echo "SETUP A" | nc portquiz.net 554

Needless to say, being able to crash a router with 6 letters sent over a common port is both a stability and security issue that Virgin needs to fix ASAP

There will be no proper fix until Virgin's engineers patch and deploy a new firmware version. But there are 2 short-term options:

Block 445 Outbound:

In your router admin page: Advanced Settings > Security > IP and Port filtering > Create a new rule
The rule should then be set to:

Source address: All, Destination address: All, Protocol: TCP, Source port: All, Destination port: 554, Enabled (checked)

This will simply stop any RTSP streams from being established and thankfully it also stops the ALG from trying to "help"

In VRChat specifically you will lose world music/video in some worlds, but everything else like voice, movement, avatar and world loading should keep working.

VPN:

Another option is to just use a VPN so the router can't see the RTSP traffic at all, so the ALG is never triggered.

For normal streams any VPN type would be fine, but for VRChat you'll probably need something with minimal overhead like the Wireguard protocol.

---

Note: This post has been edited to remove images, links and technical references, since my last reply was removed.

[IPFreely]

eth0: Legend. 

[Lisa_CC]

Hi and thanks for letting us know about this. We've passed it onto our back office team to investigate and will update this thread as soon as we can.

Regards,

Lisa

[Lisa_CC]

Hi @Riyo @eth0 

Thank you for your patience whilst we look into this. Our back office Team are unable to replicate the connection drop. Please provide the actual RTSP stream URL that causes the drop so we can continue with our investigation.

Thanks,

Lisa

[24hourmusic]

hi @Lisa_CC,

I am one of the directors for the company listed in the opening post, this issue has been plaguing us for a little while as our whole service revolves around streaming, we have a demo URL that is live 24/7 and follows all the same routes as a users stream, hopefully this will aide in the investigation (I did try finding a contact URL to email it in but couldn't find one)!

This issue can be replicated with RTSP under the following URL:

rtsp://stream.vrcdn.live/live/vrcdn

(this is just a demo stream that just plays a slideshow of images with some audio)

We've already managed to raise this as an issue to other DOCSIS-based ISPs such as Ziggo and Vodafone which have confirmed this to likely be in firmware.

Kind Regards,

Aaron Francis (24HourMusic)

[Lisa_CC]

Hi Aaron and thank you for the information.

Sorry it's been affecting those using your services and I'll add the information you've provided to them to help with the fix. We don't have an ETA but now it's on our radar, we'll be working to resolve this as soon as we can.

Warm Regards,

Lisa

..................

Edit - We've managed to replicate the issue and have raised a bug for it

[ThomasMcDonald]

After hours of searching online, I came across a post that resonates with my own experience as a VRChat player and event host on the platform. It's quite disheartening as I initially suspected the issue might be with my modem, leading to a visit from Virgin technicians who even replaced it. In our group, we rely on the use of "https://stream.vrcdn.live" to stream content into the VR world for events. While everything worked fine with the Hub 4 router, ever since upgrading to the 1GB Hub 5, I can no longer fulfil my role as a host and staff member for these events in VR. Some people have suggested downgrading to the Hub 4, but it feels like a step back considering the Hub 5 is supposedly an improvement.

[ThomasMcDonald]

Hello Lisa,

I have been following this topic as it affects the work that I do on the VRC platform, I have an event that I am staff for on VRChat and it is planned for the 24th of this month. We ran a test night to check performer and stage equipment for the streams and services and this issue reported here still affects me.

Have there been any updates from the team since you replicated the problem and raised a bug for it? as I am worried I will not be able to work for the event team I am a part of on the 24th.