Skip to content

Instantly share code, notes, and snippets.

@snickell

snickell/migrate-code-dot-org-to-lfs.sh

Last active February 3, 2024 02:32
Show Gist options
  • Save snickell/ca717b3266568b4ee05b0af20f5a25d8 to your computer and use it in GitHub Desktop.
Save snickell/ca717b3266568b4ee05b0af20f5a25d8 to your computer and use it in GitHub Desktop.
Download ZIP
Raw
migrate-code-dot-org-to-lfs.sh
#!/bin/bash
set -x
REPO=git@github.com:code-dot-org/code-dot-org.git
# We use a --mirror clone because non-mirror clones were not having
# all remote refspecs updated by `git lfs migrate`, which meant
# we weren't succesfully removing the giant old tree with blobs.
git clone $REPO --mirror
cd code-dot-org.git
git lfs migrate import \
--everything \
--include='apps/i18n/**,dashboard/config/locales/**,i18n/locales/**,apps/static/**,dashboard/app/assets/**,dashboard/public/**,shared/images/**,pegasus/sites.v3/**,pegasus/sites/**' \
--exclude='apps/i18n/**/en_us.json,dashboard/config/locales/**/*en.yml,i18n/locales/source/**,pegasus/sites.v3/hourofcode.com/i18n/en.yml,pegasus/sites.v3/**/*.collate,pegasus/sites.v3/**/*.css,pegasus/sites.v3/**/*.erb,pegasus/sites.v3/**/*.fetch,pegasus/sites.v3/**/*.haml,pegasus/sites.v3/**/*.html,pegasus/sites.v3/**/*.js,pegasus/sites.v3/**/*.md,pegasus/sites.v3/**/*.partial,pegasus/sites.v3/**/*.moved,pegasus/sites.v3/**/*.redirect,pegasus/sites.v3/**/*.scss,pegasus/sites/**/*.collate,pegasus/sites/**/*.css,pegasus/sites/**/*.erb,pegasus/sites/**/*.fetch,pegasus/sites/**/*.haml,pegasus/sites/**/*.html,pegasus/sites/**/*.js.,pegasus/sites/**/*.md,pegasus/sites/**/*.partial,pegasus/sites/**/*.moved,pegasus/sites/**/*.redirect,pegasus/sites/**/*.scss'
git gc --prune=now
# GitHub doesn't accept pushes with more than 2GB of objects
# the canonical solution is to break the push into chunks
# but the implementation uses refspecs, which don't work on
# a --mirror clone. Our solution is to clone our mirror clone
# locally, and push our chunked staging branch from there.
cd ..
git clone code-dot-org.git code-dot-org-clone
cd code-dot-org-clone
git remote set-url origin $REPO
# Push to remote staging in 10,000 commit chunks
step_commits=$(git log --oneline --reverse refs/heads/staging | awk 'NR % 10000 == 0')
n_steps=$(echo "$step_commits" | wc -l)
echo "Push is broken into $n_steps steps"
echo "$step_commits" | while read commit message; do echo $commit; git push --force origin $commit:refs/heads/staging; echo; echo; done
git push origin staging
echo; echo; echo
# Finally, return to the mirror clone to push ALL the remaining refspecs (branches, notes, etc etc)
cd ../code-dot-org.git
echo; echo;
git push --force --progress -v
echo; echo; echo; echo;
echo "Done!"
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment