Created
June 1, 2022 15:23
-
-
Save yaythomas/c1d171d9b3477f77dc06efc795ddc6e7 to your computer and use it in GitHub Desktop.
pypyr pipeline with dependencies & idempotency
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
# run this like follows: | |
# Run only the common/shared dependency: | |
# $ pypyr deps-example | |
# | |
# Run either my_task or another_task: | |
# $ pypyr deps-example my_task | |
# $ pypyr deps-example another_task | |
# | |
# And to run both tasks: | |
# $ pypyr deps-example my_task another_task | |
context_parser: pypyr.parser.list | |
# steps is the standard entry point - this runs 1st | |
steps: | |
- name: pypyr.steps.call | |
description: -> common_setup - think of this as dependencies that have to run every time | |
in: | |
call: common_setup | |
- name: pypyr.steps.call | |
comment: optionally do extras like my_task or another_task after common | |
run: '{argList}' | |
in: | |
call: '{argList}' | |
- name: pypyr.steps.echo | |
in: | |
echoMe: done! | |
common_setup: | |
- name: pypyr.steps.pyimport | |
comment: you can import anything here | |
we'll be using Path to check if file exists on disk to decide | |
whether to run a step again or not for idempotency | |
in: | |
pyImport: from pathlib import Path | |
- name: pypyr.steps.echo | |
in: | |
echoMe: common 1 | |
- name: pypyr.steps.echo | |
in: | |
echoMe: common 2 | |
dep1: | |
# dep1 somewhat arbitrarily creates out/arb.txt | |
# dep1 will only create out/arb.txt ONCE, if file already exists will stop | |
# and exit this step-group immediately without doing anything | |
- name: pypyr.steps.stopstepgroup | |
comment: exit this group is out/arb.txt already exists | |
run: !py Path('out/arb.txt').exists() | |
- name: pypyr.steps.cmd | |
description: -> doing dep1 | |
in: | |
cmd: mkdir -p out | |
- name: pypyr.steps.cmd | |
comment: each dependency can have multiple steps in it | |
in: | |
cmd: touch out/arb.txt | |
dep2: | |
# dep2 somewhat arbitrarily creates out/arb2.txt | |
# dep2 shows a different way of achieving idempotency: | |
# 1) when another_task calls dep2 it uses the `skip` decorator to skip group | |
# if out/arb.txt already exists | |
# 2) unlike dep1, dep2 does not itself check whether it should run or not. | |
# therefore it will ALWAYS run when my_task calls it - it's up to the caller | |
# to decide idempotent behavior | |
- name: pypyr.steps.cmd | |
description: -> doing dep2 | |
in: | |
cmd: mkdir -p out | |
- name: pypyr.steps.cmd | |
comment: each dependency can have multiple steps in it | |
in: | |
cmd: touch out/arb2.txt | |
my_task: | |
- name: pypyr.steps.call | |
description: -> my_task calls dependencies dep1 & dep2 1st | |
in: | |
call: [dep1, dep2] | |
- name: pypyr.steps.cmd | |
description: -> now doing my_task stuff | |
in: | |
cmd: echo my_task stuff | |
another_task: | |
- name: pypyr.steps.call | |
comment: skip the dependency if out/arb2.txt exists already | |
skip: !py Path('out/arb2.txt').exists() | |
in: | |
call: dep2 | |
- name: pypyr.steps.cmd | |
in: | |
cmd: echo another_task stuff |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment