Forum Discussion

Riyo's avatar
Riyo
Joining in
2 years ago

Hub 5 crashes from RTSP stream

Since the hub 4 to hub 5 swap I've began experiencing a fault which takes the entire internet down.

At first it was difficult to identify the cause. From my personal usage, it occurred when playing the game VRChat and I assumed that it was related to the game. However I've been working on narrowing down the cause and it seems worse than that.

First, I called an engineer out to check power levels (it's always "check power levels!") To be safe the virgin engineer checked and reduced power levels them so they're now certainly within range. However, the problem persisted.

Errors in the network logs are always the following:

16-04-2023 00:22:53criticalCable Modem Reboot due to RG crash
16-04-2023 00:18:22criticalSYNC Timing Synchronization failure - Failed to acquire QAM/QPSK symbol timing;;CM-MAC=**:**:**:**:**:**;CMTS-MAC=**:**:**:**:**:**;CM-QOS=1.1;CM-VER=3.1;
16-04-2023 00:18:11criticalCable Modem Reboot due to RG crash
16-04-2023 00:13:48criticalSYNC Timing Synchronization failure - Failed to acquire QAM/QPSK symbol timing;;CM-MAC=**:**:**:**:**:**;CMTS-MAC=**:**:**:**:**:**;CM-QOS=1.1;CM-VER=3.1;


This does not appear to be an isolated issue with the hub 5. The following similar issues were found:
https://community.virginmedia.com/t5/Gaming-Support/Connection-issues-with-VRChat-RG-crash-Packet-loss/td-p/5303479
https://community.virginmedia.com/t5/Gaming-Support/SYNC-Timing-Synchronization-failure-while-in-vr/td-p/5291040

The pattern that I noticed was that this crash occurred when I was in VRChat worlds playing an RTSP music stream. Music streams are popular in this game. After the internet goes down, If I left the game client open, even as I had disconnected from the instance (and all other players in it) the hub 5 would still keep crashing because the stream was still trying to load in my own (private) world.

In the latest incident the music stream was an RTSP stream on TCP 554. The URL host being: https://stream.vrcdn.live

After noticing this pattern I tested a stream using this URL in VLC media player completely outside of VR. It crashed the hub 5 with the exact same symptoms.

I tested an HTTPS .ts of the exact same stream and it loaded instantly and ran fine. This seems to be entirely due to the RTSP stream.

Please advise. If this is a flaw with the Hub 5 is requires a critical fix. A protocol-phobic router is, unsurprisingly, a BIG problem.

  • I've been having exactly the same issue!

    It's an issue with the Hub 5 firmware LG-RDK_4.6.24-2206.7

    Specifically the Application Level Gateway (ALG) process is detecting RTSP traffic in an effort to start opening ports to make the RTSP stream more efficient. It's most common for SIP traffic and has caused security issues in the past

    The ALG is failing to properly understand the RTSP stream and is crashing catastrophically, probably causing a kernel panic on the router

    For the Virgin engineers, it's nice and easy to replicate with this command from Linux. Running this on any machine on a network behind a Hub 5 will trigger the crash, reboot the router and disconnect the internet for 5-10 minutes.

    echo "SETUP A" | nc portquiz.net 554

    On some routers, you can disable the ALG in the admin interface. But not on the Hub 5, or any of the Virgin routers.

    There are two possible immediate fixes until Virgin deploy a new firmware:

    Block 445 Outbound:

    In your router admin page: Advanced Settings > Security > IP and Port filtering > Create a new rule
    The rule should then be set to:

    Source address: All, Destination address: All, Protocol: TCP, Source port: All, Destination port: 554, Enabled (checked)

    This will simply stop any RTSP streams from being established. In VRChat specifically you will lose world music, but everything else should keep working.

    VPN:

    Another option is to just use a VPN so the router can't see the RTSP traffic at all, so the ALG is never triggered. For normal streams any VPN type would be fine, but for VRChat you'll probably need something with minimal overhead like the Wireguard protocol.

    Hopefully this helps :)!

  • I've been having exactly the same issue!

    Looks like an issue with the Application Level Gateway (ALG) in Hub 5 firmware LG-RDK_4.6.24-2206.7

    You're 100% right it's caused by RTSP streams in the background for music/video in VRChat, the router's ALG would normally try and open ports/speed up real time connections like SIP/RTSP, but in this case the ALG is extremely fragile and crashes even with well formed RTSP traffic.

    Normally a crashing ALG isn't a huge issue, but the Hub 5 compounds the issue with these 3 points:

    • ALG is easy to crash
    • ALG is running with enough privileges to crash the entire router OS
    • ALG can't be disabled on Hub 5

    The crash can also be replicated with the following Linux command, without even needing to fully set up an RTSP stream. It simply sends a string starting with "SETUP" to port 554, immediately triggering the Hub 5 to crash.

     

    echo "SETUP A" | nc portquiz.net 554

     

    Needless to say, being able to crash a router with 6 letters sent over a common port is both a stability and security issue that Virgin needs to fix ASAP

    There will be no proper fix until Virgin's engineers patch and deploy a new firmware version. But there are 2 short-term options:

    Block 445 Outbound:

    In your router admin page: Advanced Settings > Security > IP and Port filtering > Create a new rule
    The rule should then be set to:

    Source address: All, Destination address: All, Protocol: TCP, Source port: All, Destination port: 554, Enabled (checked)

    This will simply stop any RTSP streams from being established and thankfully it also stops the ALG from trying to "help"

    In VRChat specifically you will lose world music/video in some worlds, but everything else like voice, movement, avatar and world loading should keep working.

    VPN:

    Another option is to just use a VPN so the router can't see the RTSP traffic at all, so the ALG is never triggered.

    For normal streams any VPN type would be fine, but for VRChat you'll probably need something with minimal overhead like the Wireguard protocol.

    ---

    Note: This post has been edited to remove images, links and technical references, since my last reply was removed.

      • Lisa_CC's avatar
        Lisa_CC
        Moderator
        Hi and thanks for letting us know about this. We've passed it onto our back office team to investigate and will update this thread as soon as we can.

        Regards,

        Lisa
  • Hi Aaron and thank you for the information.

    Sorry it's been affecting those using your services and I'll add the information you've provided to them to help with the fix. We don't have an ETA but now it's on our radar, we'll be working to resolve this as soon as we can.

     

    Warm Regards,

     

    Lisa

    ..................

    Edit - We've managed to replicate the issue and have raised a bug for it

     

     

    • ThomasMcDonald's avatar
      ThomasMcDonald
      Joining in

      Hello Lisa,

      I have been following this topic as it affects the work that I do on the VRC platform, I have an event that I am staff for on VRChat and it is planned for the 24th of this month. We ran a test night to check performer and stage equipment for the streams and services and this issue reported here still affects me.

      Have there been any updates from the team since you replicated the problem and raised a bug for it? as I am worried I will not be able to work for the event team I am a part of on the 24th.

  • After hours of searching online, I came across a post that resonates with my own experience as a VRChat player and event host on the platform. It's quite disheartening as I initially suspected the issue might be with my modem, leading to a visit from Virgin technicians who even replaced it. In our group, we rely on the use of "https://stream.vrcdn.live" to stream content into the VR world for events. While everything worked fine with the Hub 4 router, ever since upgrading to the 1GB Hub 5, I can no longer fulfil my role as a host and staff member for these events in VR. Some people have suggested downgrading to the Hub 4, but it feels like a step back considering the Hub 5 is supposedly an improvement.

  • Quick update to this issue, it has been been raised as a security issue and has been assigned the following ID by Mitre for security vulnerabilities: CVE-2023-31709

  • legacy1's avatar
    legacy1
    Alessandro Volta

    All we need now is to make the hub crash by incoming traffic for VM to do some thing like tell everyone to use modem mode.

  • 3 weeks and no sign of a fix? there are 3 massive events going on this weekend on the platform for VRCHat this issue is stopping me from getting to them and doing coverage for them. how is RTSP steam issue still going on with this new hub5?

    • Sabrina_B's avatar
      Sabrina_B
      Forum Team

      Hi ThomasMcDonald.

      Thanks for reaching out to us. Apologies you are having issues in relation to the Hub. In order to look into this further for you we would need to bring you into a private message. Please be on the lookout for the envelope on the top right of web browser or if you are using a mobile device it will be located under your profile icon. 

      Sabrina

      • icoleman_uk's avatar
        icoleman_uk
        Tuning in

        I am having this exact issue, and it's 100% repeatable.  It's obviously a firmware bug.  Any ideas on when this will be fixed?

  • There hasn't been any sort of update to this issue since has there? I recently upgraded from a hub 3 to a hub 6 and have been experiencing the same issues (From my research the hub 6 seems to be a refresh of the hub 5 hense the same issues). I had tried setting up the port filtering but I couldn't seem to get it to work (I probably just set it up wrong). I'd rather not have to play minesweeper when joining different VRChat instances to figure out which use VRCDN or have to leave whenever my friends want to play sets so any update on this would be nice to know. 

    • Kath_P's avatar
      Kath_P
      Forum Team

      Hi Thrwawy03, 

      Thanks for taking the time to post in the Community. We don't currently offer a Hub 6. Can you please check the hub you have and pop back with an update?

      Thanks, 

  • Thrwawy03 - Just to update on my end (as the original poster) my eventual workaround for this which would still allow me to hear music in VRChat was to purchase a 3rd party router and then simply switch the Hub 5 to modem mode. Sadly this does not resolve the issue for the hub itself but it does circumvent it simply by avoiding the Hub 5 router functionality entirely. I was able to play RTSP streams from then on without the same issues.

    This is if they have not fixed the issue already with the Hub 5 yet, of course.