Llogiq on stuff

Wasted

We are letting a lot of network traffic going to waste.

How so, I hear some of you asking? I tweeted about it a while ago:

Why doesn't rustup delta-compress binaries? At least something like BSDiff would seriously reduce network load on both server&client!


Actually, that doesn’t only apply to rustup, but also to cargo. In fact, it’s even easier for cargo (as cargo packages are predominantly code), but as the rust updates are (usually) bigger, the effect per download may be more pronounced on rustup.

Also there is a package from Google called courgette that can compress binary files even better than BSDiff. It uses a primitive dis-/assembler to undo the packer-defeating effects of relative jumps on current CPU architectures.

At least cargo appears to use max gzip compression for the crates. Though there are more capable algorithms available today, e.g. dropbox’ rust-brotli.

But why? I hear you asking. Bandwith is cheap, after all, right? Not for everyone, I remind those who indeed held this question, also cheap doesn’t necessary mean free – someone has to pay for the network bandwidth. In some countries (or some areas) this may be more expensive than in others. Think McMurdo station, the ISS or rural German areas.

Don’t we owe it to the people there to do what we can to conserve bandwidth?

So here’s my suggestion:

  1. Add differential updates to both rustup and cargo. I suspect this is going to have the biggest impact in reducing network traffic for rust-related updates. The simple solution would be to use a BSDiff implementation, otherwise Courgette would probably save even more bandwidth. Alas, it’s written in C++, and probably not too easy to port.

  2. Switch from gzip to brotli. Dropbox` implementation is battle-tested and blazingly fast, so this should be a no-brainer. There should be a grace period where both formats are supported while gradually phasing out gzip.

What’s more, I’m going to put my programming acumen where my mouth is; I’m porting BSDiff to Rust to make it easier to use for Cargo and rustup. Stay tuned.

Update: redditor /u/jdubjdub tells us he’s been quietly working on a zsync port (which is obviously superior to BSDiff).