Skip to content

Instantly share code, notes, and snippets.

@Enteee
Last active December 26, 2022 22:05
Show Gist options
  • Save Enteee/c8c11d46a95568be4d331ba58a702b62 to your computer and use it in GitHub Desktop.
Save Enteee/c8c11d46a95568be4d331ba58a702b62 to your computer and use it in GitHub Desktop.
Bash: Parallel execute and retain exit code
#!/usr/bin/env bash
# Wait for pids to terminate. If one pid exits with
# a non zero exit code, send the TERM signal to all
# processes and retain that exit code
#
# usage:
# :wait 123 32
function :wait(){
local pids=("$@")
[ ${#pids} -eq 0 ] && return $?
trap 'kill -INT "${pids[@]}" &>/dev/null || true; trap - INT' INT
trap 'kill -TERM "${pids[@]}" &>/dev/null || true; trap - RETURN TERM' RETURN TERM
for pid in "${pids[@]}"; do
wait "${pid}" || return $?
done
trap - INT RETURN TERM
}
# Run a function in parallel for each argument.
# Stop all instances if one exits with a non zero
# exit code
#
# usage:
# :for func 1 2 3
#
# env:
# FOR_PARALLEL: Max functions running in parallel
function :for(){
local f="${1}" && shift
local i=0
local pids=()
for arg in "$@"; do
( ${f} "${arg}" ) &
pids+=("$!")
if [ ! -z ${FOR_PARALLEL+x} ]; then
(( i=(i+1)%${FOR_PARALLEL} ))
if (( i==0 )) ;then
:wait "${pids[@]}" || return $?
pids=()
fi
fi
done && [ ${#pids} -eq 0 ] || :wait "${pids[@]}" || return $?
}
#!/usr/bin/env bash
set -ex
# import :for
source <(curl -Ls https://goo.gl/Wr4QZZ)
msg="You should see this three times"
:(){
i="${1}" && shift
echo "${msg}"
sleep 1
if [ "$i" == "1" ]; then sleep 1
elif [ "$i" == "2" ]; then false
elif [ "$i" == "3" ]; then
sleep 3
echo "You should never see this"
fi
} && :for : 1 2 3 || exit $?
echo "You should never see this"
$ ./usage
+ source /dev/fd/63
++ curl -Ls https://goo.gl/Wr4QZZ
+ msg='You should see this three times'
+ :for : 1 2 3
+ local f=:
+ shift
+ local i=0
+ pids=()
+ local pids
+ for arg in "$@"
+ pids+=("$!")
+ : 1
+ '[' '!' -z ']'
+ for arg in "$@"
+ i=1
+ shift
+ echo 'You should see this three times'
You should see this three times
+ pids+=("$!")
+ sleep 1
+ '[' '!' -z ']'
+ for arg in "$@"
+ pids+=("$!")
+ '[' '!' -z ']'
+ '[' 5 -eq 0 ']'
+ :wait 24933 24934 24935
+ pids=("$@")
+ local pids
+ '[' 5 -eq 0 ']'
+ trap 'kill -INT "${pids[@]}" &>/dev/null || true; trap - INT' INT
+ trap 'kill -TERM "${pids[@]}" &>/dev/null || true; trap - RETURN TERM' RETURN TERM
+ for pid in "${pids[@]}"
+ wait 24933
+ : 3
+ i=3
+ shift
+ echo 'You should see this three times'
You should see this three times
+ sleep 1
+ : 2
+ i=2
+ shift
+ echo 'You should see this three times'
You should see this three times
+ sleep 1
+ '[' 3 == 1 ']'
+ '[' 3 == 2 ']'
+ '[' 3 == 3 ']'
+ sleep 3
+ '[' 1 == 1 ']'
+ sleep 1
+ '[' 2 == 1 ']'
+ '[' 2 == 2 ']'
+ false
+ for pid in "${pids[@]}"
+ wait 24934
+ return 1
++ kill -TERM 24933 24934 24935
++ trap - RETURN TERM
+ return 1
+ exit 1
@Enteee
Copy link
Author

Enteee commented Mar 9, 2018

A similar implementation posted on reddit:

#!/usr/bin/env bash
set -ex

msg="You should see this three times"

:(){
  i="${1}" && shift

  echo "${msg}"

  sleep 1
  if   [ "$i" == "1" ]; then sleep 1
  elif [ "$i" == "2" ]; then false
  elif [ "$i" == "3" ]; then
    sleep 3
    echo "You should never see this"
  fi
} && export -f : && xargs -n 1 -P 0 bash -c ': "$@"' -- <<<'1 2 3' || exit $?


echo "You should never see this"

Problems with the xargs -P alternative:

  • Spawns a new shell, which means:
    • does not preserve shell options: set -ex
    • does not forward variables (see echo "${msg}" in new usage)
  • The third process won't be stopped from printing You should never see this
  • Does not retain exit code (returns 123 instead of 1)
$ ./usage.sh 
+ msg='You should see this three times'
+ export -f :
+ xargs -n 1 -P 0 bash -c ': "$@"' --



You should never see this
+ exit 123

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment