Transfer 500GB Instead Of 70GB To Save Time
A month ago I needed to download a quite big package from the Internet Archive. Unfortunately their HTTP servers throttled the connections to ~300KB/s in case of the files I was interested in. Luckily, they also offer option to use BitTorrent protocol to transfer files.
Downloading 70GB would take roughly 3 days at 300KB/s. I knew I could do better than that. But I didn’t want to circumvent any mechanisms they put in place to protect their infrastructure from people like me.
I was sceptic at first but I started my qBittorrent and loaded the torrent from the website. To my surprise, there were two HTTP mirrors instead of one. That’s twice the original speed already! There were no other peers at the time.
My theory was that everything would speed up substantially if more people were to join because the HTTP mirrors are artificially slowed down. So I invited some friends to the experiment. After an hour there were 7 of us downloading different parts at “full” speed from the Archive mirrors and sharing with the rest.
Some time later one of the upstream hosts went down but I managed to download the whole thing in under 8 hours. One of the friends keeps seeding it on his server and reached the ratio of 9.7 over last month. So there are other people interested in this package after all.
Shout-outs to aquaz, dexterw, Halamix2, Kaadzik, pitpo and PokerFacowaty for help.