This has nothing to do with IPv6 itself. I pull in 4K YouTube videos over IPv6 just fine. My IPv6 routes actually have lower latency than my IPv4 routes, funnily enough.

Sounds like your ISP has broken their IPv6 routes, or your modem is outdated and can't do IPv6 hardware acceleration. Disabling IPv6 to downgrade your connection will work as a workaround, at least until your ISP switches over to something using IPv6 as the connection backbone (like DS-Lite, which would allow your ISP to significantly reduce their IPv4 space and make a quick profit selling off their allocations, which is unfortunately becoming more and more common).

Your ISP or modem manufacturer needs to fix the actual problem here.