Skip to content

Instantly share code, notes, and snippets.

@mbostock
Last active July 30, 2024 03:51
Show Gist options
  • Save mbostock/3883098 to your computer and use it in GitHub Desktop.
Save mbostock/3883098 to your computer and use it in GitHub Desktop.
The Gist to Clone All Gists
license: gpl-3.0

Run like so:

chmod u+x gist-clone-all
./gist-clone-all username OAUTH_TOKEN

You'll want to replace “username” with your own username and “OAUTH_TOKEN” with an OAuth token.

This script clones using the push URL, so you should probably be the owner of the gists. You could also use this to clone someone else's gists, but in that case you may wish to edit the code to use gist_pull_url instead.

#!/usr/bin/env node
var fs = require("fs"),
https = require("https"),
exec = require("child_process").exec;
// TODO --pull or --push
var user = process.argv[2],
token = process.argv[3];
fetchAndClone(1, function callback(error, nextPage) {
if (error) throw error;
if (nextPage > 0) fetchAndClone(nextPage, callback);
});
function fetchAndClone(page, callback) {
fetch(page, function(error, gists) {
if (error) return callback(error);
if (gists.length) ++page; else page = -1;
cloneNext(gists.pop());
function cloneNext(gist) {
if (!gist) return callback(null, page);
if (directoryExists(gist.id)) return cloneNext(gists.pop());
console.log("cloning " + gist.id);
exec("git clone git@gist.github.com:" + gist.id + ".git", function(error, stdout, stderr) {
if (error) return callback(error);
cloneNext(gists.pop());
});
}
});
}
function fetch(page, callback) {
var request = https.request({
hostname: "api.github.com",
port: 443,
path: "/users/" + encodeURIComponent(user) + "/gists?page=" + page,
method: "GET",
headers: {
"Accept": "application/vnd.github.v3+json",
"Authorization": "token " + token,
"User-Agent": "mbostock/gist-clone-all"
}
}, function(response) {
var chunks = [];
response.setEncoding("utf8");
response.on("data", function(chunk) { chunks.push(chunk); });
response.on("end", function() { callback(null, JSON.parse(chunks.join(""))); });
});
request.on("error", callback);
request.end();
}
function directoryExists(path) {
try {
return fs.lstatSync(path).isDirectory();
} catch (ignored) {
return false;
}
}
#!/bin/bash
for i in *; do
if [ -d $i ]; then
pushd $i
git commit -am '' --allow-empty-message && git push
popd
fi
done
@batandwa
Copy link

Superb gist. Thank you.

@weakish
Copy link

weakish commented Oct 21, 2014

https://gist.github.com/mbostock/3883098#file-gist-clone-all-js-L1

#!/usr/local/bin/node

Why not #!/usr/bin/env node?


@batandwa This only clones public gists. If you wnt to clone all your gists (including private ones), you can have a look at gister sync.

@nanda-malve
Copy link

I am unable to use the above script, please help me in this

root@dev-node:~/git# node gist-clone-all

/root/git/gist-clone-all:21
cloneNext(gists.pop());
^
TypeError: Object # has no method 'pop'
at /root/git/gist-clone-all:21:21
at IncomingMessage. (/root/git/gist-clone-all:50:37)
at IncomingMessage.emit (events.js:88:20)
at HTTPParser.onMessageComplete (http.js:137:23)
at CleartextStream.ondata (http.js:1150:24)
at CleartextStream._push (tls.js:375:27)
at SecurePair.cycle (tls.js:734:20)
at EncryptedStream.write (tls.js:130:13)
at Socket.ondata (stream.js:38:26)
at Socket.emit (events.js:67:17)

@maowug
Copy link

maowug commented Mar 26, 2016

@nanda-malve

I'v made one much simpler to use. No OAUTH_TOKEN, for both github.com and github enterprise.
https://github.com/maowug/backup-gists

@tianchaijz
Copy link

tianchaijz commented May 10, 2016

@vetras
Copy link

vetras commented May 18, 2016

The error:

root@dev-node:~/git# node gist-clone-all

/root/git/gist-clone-all:21
cloneNext(gists.pop());
^
TypeError: Object # has no method 'pop'
(...)

Is due to a invalid token.
If you add a log to the GIST object:

(...)
console.log(gists);
cloneNext(gists.pop());
(...)

It comes with the GitHub message:

{ message: 'Bad credentials',
documentation_url: 'https://developer.github.com/v3'
}

My solution was to generate another token with more permissions.

Great stuff, many thanks!

@hyang0
Copy link

hyang0 commented Sep 2, 2016

I tried. It works. The environment is ubuntu 16.04 64bit.

First of all, you should install nodejs.

➜ curl -sL https://deb.nodesource.com/setup_4.x | sudo -E bash -

Installing the NodeSource Node.js v4.x LTS Argon repo...

➜ sudo apt-get install nodejs
Reading package lists... Done

Then you should generate a ssh public key and add it to you github account.

➜ ssh-keygen
Generating public/private rsa key pair.
Enter file in which to save the key (/home/hyang0/.ssh/id_rsa):

Finnally, retry the script to sync your gists.

➜ ./gist-clone-all.sh GIST-ACCOUNT Personal-access-token
cloning 7d5e4348f307f7cb2fbe
cloning 6986436
cloning 8572463
cloning 8572515
cloning 8572569
cloning a59175560823bd3cf383
cloning 5234efc9170aa3302dd9
cloning 75c39d5a6186e5bae304
cloning 159e719feb8b230a63fa
cloning 1eeb1affd91e91ba73c7
cloning 2498e1494bf7af5503b9

Cool job. Thanks!

@lmj0011
Copy link

lmj0011 commented Dec 30, 2016

I only had to set the gist permission for the OAuth token, but you may not even need that set.

@mbostock
Would be nice to have, which permission should be set for the OAuth token, in the README.md .

@lmj0011
Copy link

lmj0011 commented Dec 30, 2016

I only had to set the gist permission for the OAuth token, but you may not even need that set.

correction: Only the "gist" scope is needed for the OAuth token.

@tianyiii
Copy link

what is gist_pull_url?

@nlevick
Copy link

nlevick commented Mar 30, 2017

@maowug
Great work! Thank you!

@endolith
Copy link

I've been using https://github.com/aprescott/gist-backup but I was wondering if there was a way to clone all other people's forks of my gists, too, so I can easily see what changes have been made. I've been adding them manually with username as remote name.

@erbanku
Copy link

erbanku commented Mar 30, 2022

I have failed many times with this. but after searching the internet I found an easy one to do the same job, ghcloneall 1.10.1, enjoy!

@chenchun
Copy link

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment