Skip to content

Instantly share code, notes, and snippets.

@elenakrittik

elenakrittik/CRATES_MIRROR.md

Last active November 19, 2023 13:43
Show Gist options
  • Save elenakrittik/47387ee33edbb16c289d34027cbd1ef6 to your computer and use it in GitHub Desktop.
Save elenakrittik/47387ee33edbb16c289d34027cbd1ef6 to your computer and use it in GitHub Desktop.
Download ZIP
crates.io mirror using ROMT as of 12 April, 2022
Raw
CRATES_MIRROR.md

This gist is intended to loosely describe what to expect when you want to mirror crates.io (in particulair, using romt)

This is written on 12 of April, 2022

  • crates.io has 80,843 crates
  • Download size is ~70GB
  • Download lasted for about 6 hours

The command i have used for downloading crates.io:

romt crate -v --keep-going update -j=64
  • -v: verbose
  • --keep-going: ignore errors
  • -j=64: Number of jobs

Note: if you see messages like ResourseClosedError, downgrade the number of jobs or somehow optimize your pc.

Also, if you see errors like ConnectTimeout and/or ReadTimeout try downgrading number of jobs or look at the issue; these errors mean that you hit limit of your network (ConnectTimeout) or disk (ReadTimeout) power.

At the end, here are characteristics of the PC (actually, laptop) i've used:

  • OS: Windows 10
  • RAM: 8GB
  • ROM: ~360GB free before mirroring, HDD
  • Processor: Intel Core i7-3537U

P.S. panamax has a bug on Windows that locks you to download crates.io only to C:, therefore i didn't use it. For details, see this issue

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment