Skip to content

Instantly share code, notes, and snippets.

@IAmStoxe

IAmStoxe/falcon180b_dl.sh

Last active September 7, 2023 16:05
Show Gist options
  • Save IAmStoxe/785830adf0b48eef4c7c5051fe4c9634 to your computer and use it in GitHub Desktop.
Save IAmStoxe/785830adf0b48eef4c7c5051fe4c9634 to your computer and use it in GitHub Desktop.
Download ZIP
This script helps you to automatically download and assemble the split files of the Falcon 180B Chat GGUF dataset from Hugging Face.
Raw
readme.md

Falcon 180B Chat GGUF File Download and Assembly

This script helps you to automatically download and assemble the split files of the Falcon 180B Chat GGUF dataset from Hugging Face.

Requirements

  • Bash shell (Linux/Unix/macOS)
  • aria2 - A lightweight multi-protocol & multi-source command-line download utility. It supports HTTP/HTTPS, FTP, SFTP, BitTorrent and Metalink.

Usage

  1. Ensure you have aria2 installed on your system. You can install it using the following command on Ubuntu/Debian systems:

    sudo apt-get install aria2

  2. Save the script in a file, for instance download_and_assemble.sh.

  3. Give execute permissions to the script:

    chmod +x download_and_assemble.sh

  4. Run the script:

    ./download_and_assemble.sh

  5. The assembled file will be available in your home directory with the name falcon-180b-chat.Q4_K_M.gguf.

Raw
falcon180b_dl.sh
#!/bin/bash
# Create a file with the list of URLs to download
echo -e "#\nhttps://huggingface.co/TheBloke/Falcon-180B-Chat-GGUF/resolve/main/falcon-180b-chat.Q4_K_M.gguf-split-a\n out=falcon-180b-chat.Q4_K_M.gguf-split-a\nhttps://huggingface.co/TheBloke/Falcon-180B-Chat-GGUF/resolve/main/falcon-180b-chat.Q4_K_M.gguf-split-b\n out=falcon-180b-chat.Q4_K_M.gguf-split-b\nhttps://huggingface.co/TheBloke/Falcon-180B-Chat-GGUF/resolve/main/falcon-180b-chat.Q4_K_M.gguf-split-c\n out=falcon-180b-chat.Q4_K_M.gguf-split-c" > ~/urls.txt
# Use aria2 to download files in parallel
aria2c -x 6 -i ~/urls.txt -d ~
# Concatenate the downloaded files
cat ~/falcon-180b-chat.Q4_K_M.gguf-split-a ~/falcon-180b-chat.Q4_K_M.gguf-split-b ~/falcon-180b-chat.Q4_K_M.gguf-split-c > ~/falcon-180b-chat.Q4_K_M.gguf
# Remove the temporary files
rm ~/urls.txt ~/falcon-180b-chat.Q4_K_M.gguf-split-*
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment