in the 2nd step, just for the sake of example and easy testing, I put "echo" in front of each command.
To run for real, delete the echo preceding every line from ln 27-30.
This pypyr pipeline pretty much does the same thing as https://github.com/huggingface/transformers/blob/master/valohai.yaml
I'm not exactly sure where the /valohai directory comes from - I don't see it in the huggingface repo root or in the pytorch container, so maybe there is a script/generator somewhere that creates this directory. As it is the command will fail because that directory isn't available.
The parameters
input in the first step (ln. 9) is pretty much a straight copy-paste from valohai.yaml - this just being an example, I didn't bother tweaking it too much beyond just showing you the idea. Strictly speaking you don't need the name
key, the pipeline only uses the pass-as
and default
keys to map the argument name to a value. I just picked 3 arbitrary parameters - you can of course include the entire list under parameters
from valohai.yaml
. (just note the pass-as is only --argname
, the ={v}
part is unnecessary for pypyr.)
pypyr will
- set some default values for you like docker_image and task_name - you can override these from the cmdline
- parse the parameters list collection into a flat string list of args
- run the docker container, mount the current host directory into the container workspace, and pass the command and flattened list of args to it
To run this, put the attached docker-run-example.yaml
pipeline into your repo working directory and from that directory:
# run with defaults
$ pypyr docker-run-example
# override default task_name
$ pypyr docker-run-example task_name=arb-task-name
# override default task_name and model_name_or_path
$ pypyr docker-run-example task_name=arb-task-name model_name=arb-name