Each hub, router, or mesh includes a chipset that manages the Dynamic Host Control Protocol (DHCP) - doesn't that sound impressive? But what it means is that the device has a brain that actively manages all connected devices, so that your single internet connection can be used simultaneously by a houseful of gadgets. Each connection needs a DHCP processor, but it must not have two, otherwise they often squabble over how devices connect. Outward symptoms are often not obvious, but it can cause all manner of weird, erratic and intermittent problems, and these probably won't be immediately apparent.
Putting your hub in modem mode turns off the hub's DHCP processor and wireless, stops the weird problems and allows the Nest wifi to work as its makers intend.
When you put your hub into modem mode, all devices connected to the hub will lose internet connection. When the hub has come back online (allow ten minutes) but nothing appears to be working, just turn the hub off, and the Nest unit off. Then make sure the first Nest unit is the only thing connected to the hub by ethernet cable, and turn the Hub on (another 7-10 minute wait, I'm afraid) and then the Nest. At this point all should be working.
If this turns into some domestic disaster, then your get out of jail card is the pinhole reset button on the back of the hub, then come back here and we can run through getting it back together.
