Skip to content

Instantly share code, notes, and snippets.

@egidijus
Forked from lxe/README.md
Created March 14, 2023 18:36
Show Gist options
  • Save egidijus/268af77829d5cd895d73267127f99e73 to your computer and use it in GitHub Desktop.
Save egidijus/268af77829d5cd895d73267127f99e73 to your computer and use it in GitHub Desktop.
How to get oobabooga/text-generation-webui running on Windows or Linux with LLaMa-30b 4bit mode via GPTQ-for-LLaMa on an RTX 3090 start to finish.

How to get oobabooga/text-generation-webui running on Windows or Linux with LLaMa-30b 4bit mode via GPTQ-for-LLaMa on an RTX 3090 start to finish.

This guide actually works well for linux too. Just don't bother with the powershell envs

  1. Download prerequisites

  2. (Windows Only) Open the Conda Powershell.

    • Alternatively, open the regular PowerShell and activate the Conda environment:
      pwsh -ExecutionPolicy ByPass -NoExit -Command "& ~\miniconda3\shell\condabin\conda-hook.ps1 ; conda activate ~\miniconda3"'
    • Sometimes for some reason the GPTQ compilation fails if 'cl' is not in the path. You can try using the x64 Native Tools Command Prompt for VS 2019 shell instead or, load both conda and VS build tools shell like this:
      cmd /k '"C:\Program Files (x86)\Microsoft Visual Studio\2019\BuildTools\VC\Auxiliary\Build\vcvars64.bat" && pwsh -ExecutionPolicy ByPass -NoExit -Command "& ~\miniconda3\shell\condabin\conda-hook.ps1 ; conda activate ~\miniconda3"'
  3. You'll need the CUDA compiler and torch that matches the version in order to build the GPTQ extesions which allows for 4 bit prequantized models. Create a conda env and install python, cuda, and torch that matches the cuda version, as well as ninja for fast compilation

    conda create -n tgwui
    conda activate tgwui
    conda install python=3.10

    Installing pytorch and cuda is the hardest part of machine learning I've come up with this install line from the following sources:

    conda install cuda pytorch torchvision torchaudio pytorch-cuda=11.7 -c pytorch -c nvidia/label/cuda-11.7.0
    python -c 'import torch; print(torch.cuda.is_available())'
  4. Download text-generation-webui and GPTQ-for-LLaMa

    git clone https://github.com/oobabooga/text-generation-webui.git
    cd text-generation-webui
    mkdir repositories
    cd repositories
    git clone https://github.com/qwopqwop200/GPTQ-for-LLaMa.git
    cd GPTQ-for-LLaMa
  5. Build and install gptq package and CUDA kernel (you should be in the GPTQ-for-LLaMa directory)

    pip install ninja
    python setup_cuda.py install
    
  6. Install the text-generation-webui dependencies

    cd ../..
    pip install -r requirements.txt
    
  7. Download the 13b model from huggingface

    python download-model.py decapoda-research/llama-13b-hf
    

    This will take some time. After it's done, rename the folder to llama-13b

    The llama-13b prequantized is available here. Download the llama-13b-4bit.pt file and place it in models directory, alongside the llama-13b folder.

  8. Run the text-generation-webui with llama-13b to test it out

    python server.py --cai-chat --load-in-4bit --model llama-13b --no-stream
    
  9. Download the hf version 30b model from huggingface

    python download-model.py decapoda-research/llama-30b-hf
    

    You can download the pre-quantized 4 bit versions of the model here.

    Alternatively, you'll need to quantize it yourself using GPTQ-for-LLaMa (this will take a while):

    cd ../repositories/GPTQ-for-LLaMa
    pip install datasets
    HUGGING_FACE_HUB_TOKEN={your huggingface token} CUDA_VISIBLE_DEVICES=0 python llama.py ../../models/llama-30b-hf c4 --wbits 4 --save llama-30b-4bit.pt
    

    Place the llama30b-4bit.pt in models in models directory, alongside the llama-30b folder.

  10. Run the text-generation-webui with llama-30b

    python server.py --cai-chat --load-in-4bit --model llama-30b --no-stream
    
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment