Docker Desktop requires an expensive license for commercial use: https://www.docker.com/pricing/faq/
# Install minikube
brew install minikube
# Install Docker CLI
brew install docker
"""Evaluate BM25 on MTEB tasks | |
Usage: | |
python bm25.py -t <task name> --output_folder=./data/results | |
Notes: | |
- https://github.com/xhluca/bm25s (promissing implememntation) | |
- https://github.com/beir-cellar/beir/blob/main/examples/retrieval/evaluation/lexical/evaluate_bm25.py | |
- https://colab.research.google.com/drive/1HfutiEhHMJLXiWGT8pcipxT5L2TpYEdt?usp=sharing#scrollTo=nqotyXuIBPt6 |
Docker Desktop requires an expensive license for commercial use: https://www.docker.com/pricing/faq/
# Install minikube
brew install minikube
# Install Docker CLI
brew install docker
#!/bin/bash | |
#SBATCH --job-name=oxw-bloom-1b7-twc-german | |
#SBATCH --ntasks-per-node=1 # crucial - only 1 task per dist per node! | |
#SBATCH --nodes=4 | |
#SBATCH --gres=gpu:4 # ---> does not matter on JUWELS | |
#SBATCH --cpus-per-task=48 # number of cores per tasks | |
#SBATCH --hint=nomultithread # we get physical cores not logical | |
#SBATCH --time=0-12:00:00 | |
#SBATCH --output=%j.%x.out | |
#SBATCH --partition=booster |
# Copyright 2022 EleutherAI and The HuggingFace Inc. team. All rights reserved. | |
# | |
# Licensed under the Apache License, Version 2.0 (the "License"); | |
# you may not use this file except in compliance with the License. | |
# You may obtain a copy of the License at | |
# | |
# http://www.apache.org/licenses/LICENSE-2.0 | |
# | |
# Unless required by applicable law or agreed to in writing, software | |
# distributed under the License is distributed on an "AS IS" BASIS, |
Being able to SSH directly into a compute job has the advantage of using all remote development tools such as using your IDE's debugger also for GPU jobs (VSCode, PyCharm, ...).
General problem:
import argparse | |
import os | |
import torch | |
from transformers.models.auto import AutoModelForCausalLM | |
LAYER_FILE_PREFIX = 'layer_' | |
MODEL_FILE_PREFIX = 'model_' | |
EMBEDDING_LAYER_INDEX = 1 |
# Obtain Lets-Encrypt SSL Certificate via Docker DNS challenge | |
# adjust: | |
# - domains (-d foo.me) | |
mkdir letsencrypt_etc letsencrypt_var | |
docker run -it --rm --name certbot \ | |
-v "./letsencrypt_etc:/etc/letsencrypt" \ | |
-v "./letsencrypt_var:/var/lib/letsencrypt" \ | |
certbot/certbot certonly -d foo.me -d *.foo.me --manual --preferred-challenges dns |
[AutoDJ] | |
[Master] | |
[VinylControl] | |
[PreviewDeck1] | |
[Channel1] | |
play y |