Skip to content

Instantly share code, notes, and snippets.

@jondurbin
jondurbin / check_copyright.py
Last active July 12, 2024 00:41
Copyright check
@jondurbin
jondurbin / create_tokenizer.py
Last active December 28, 2023 17:31
AR examples
import re
import gc
import os
import glob
import json
from copy import deepcopy
from datasets import concatenate_datasets, Dataset
from transformers import AutoTokenizer
from huggingface_hub import snapshot_download
@jondurbin
jondurbin / airoboros-m-7b-3.1.2.md
Last active October 21, 2023 12:52
airoboros-m-7b-3.1.2.md

Trained on 10x a6000 GPUs on runpod.io.

I actually ran many fine-tunes, including multiple full-finetunes, fp16 loras, and qloras, and the below qlora actually did best in my testing.

dataset: https://hf.co/datasets/jondurbin/airoboros-3.1 (plus a few unpublished de-censoring instructions)

training script: https://github.com/jondurbin/qlora specifically commit 8cd269bf9bd7753c92164934269019e12f23314f

export BASE_DIR=/workspace
@jondurbin
jondurbin / airoboros-m-7b-3.0-tuning.md
Created October 2, 2023 15:07
airoboros-m-7b-3.0-tuning.md

My fork of qlora (but using the --full-finetune option), commit b5771f3caa9a5ea3ec397526a09720e957dc03d0 (main branch as of 2023-10-02):

6x 80GB a100s on runpod.io

Dataset: https://hf.co/datasets/jondurbin/airoboros-3.0 (private currently, until fine-tunes are finished)

Script:

@jondurbin
jondurbin / airoboros-l2-7b-3.0-training.md
Created September 30, 2023 12:52
airoboros-l2-7b-3.0-training.md
@jondurbin
jondurbin / airoboros-l2-70b-2.2-tuning.md
Created September 12, 2023 14:32
airoboros-l2-70b-2.2-tuning
@jondurbin
jondurbin / spicyboros-70b-2.2-tuning.md
Last active September 18, 2024 18:42
spicyboros-70b-2.2-tuning
@jondurbin
jondurbin / airoboros-l2-7b-2.2-tuning.md
Last active September 10, 2023 11:41
airoboros-l2-7b-2.2-tuning
@jondurbin
jondurbin / airoboros-l2-13b-2.2-tuning.md
Created September 10, 2023 11:32
airoboros-l2-13b-2.2-tuning
@jondurbin
jondurbin / spicyboros-13b-2.2-tuning.md
Last active September 10, 2023 11:33
spicyboros-13b-2.2-tuning