Skip to content

Instantly share code, notes, and snippets.

@lcatlett

lcatlett/README.md

Last active August 28, 2024 14:04
Show Gist options
  • Save lcatlett/27367cadfff22a99a017be37f95c1bc7 to your computer and use it in GitHub Desktop.
Save lcatlett/27367cadfff22a99a017be37f95c1bc7 to your computer and use it in GitHub Desktop.
Download ZIP
Enterprise Site Portfolio Deployments Scripting Library
Raw
README.md

Enterprise Many Site Deployment and Scripting Recommendations

This repository contains an example set of shell scripts that provide recommendations for Voya to execute CI / CD tasks at scale againt a large portfolio of Pantheon sites. Each script provided has inline docmentation for required functionality and additional examples of how to address common enterprise deployment challenges / requirements.

Features

  • Parallel execution of code and deployment tasks on multiple sites
  • Race condition handling for workflows, terminus commands, and git commands
  • Configuration of environment variables commonly required in enterprise deployment scripts
  • Automatic retries of failed site deployments
  • Setup of SSH and git configuration to avoid password and other interactive prompts

Usage

To use these scripts, follow these steps:

  1. Clone this repository to your local machine.
  2. Make sure you have the necessary dependencies installed (e.g., parallel, php, etc.).
  3. Set the appropriate environment variables in the script.
  4. Run the "wrapper" script for your desired task, eg:
./deploy-from-local-to-dev-wrapper.sh
./deploy-from-dev-to-test.sh
./deploy-to-live-wrapper.sh

Configuration

The script provides several configuration options that you can modify to suit your needs. These options include:

  • ENV: The target environment (e.g., dev, test, etc.).
  • SITES: An array of sites to execute the deployment tasks on.
  • --jobs: The number of parallel jobs to run for code push and deployment tasks.

Retry Mechanism

The script includes a retry mechanism that allows failed site deployments to be retried automatically. Failed sites are added to a list, and the script will continue retrying until all sites are deployed successfully or a timeout is reached.

Logs

The script generates log files for each deployment attempt. You can view the log file by running cat <log_filename>.

Raw
bg.sh
#!/usr/bin/env bash
# Runs a command as a background job gets exit code of child processes.
# Allows for more granular error handling and logging of Pantheon / terminus commands.
set -m # allow for job control
EXIT_CODE=0 # exit code of overall script
# may need to set -o posix if we run into issues with the for loop returning a 127 or bad_trap error
INPUT_CMD=$1
function wait_and_get_exit_codes() {
children=("$@")
EXIT_CODE=0
for job in "${children[@]}"; do
echo "PID => ${job}"
CODE=0
wait ${job} || CODE=$?
if [[ "${CODE}" != "0" ]]; then
echo "At least one test failed with exit code => ${CODE}"
EXIT_CODE=1
fi
done
}
DIRN=$(dirname "$0")
commands=(
"{ $INPUT_CMD; }"
)
clen=$(expr "${#commands[@]}" - 1) # get length of commands - 1
children_pids=()
for i in $(seq 0 "$clen"); do
(echo "${commands[$i]}" | bash) & # run the command via bash in subshell
children_pids+=("$!")
echo "$i ith command has been issued as a background job"
done
wait_and_get_exit_codes "${children_pids[@]}"
echo "EXIT_CODE => $EXIT_CODE"
exit "$EXIT_CODE"
Raw
deploy-dev
#!/bin/bash
# Runs tasks to deploy code from bitbucket/github/local vcs to a Pantheon dev environment including:
# - Setup of SSH and git configuration to avoid password prompts
# - Race condition handling for workflows, terminus commands, and git commands
# - Logging of failed code deployment task for automatic retries
# - Configuration of environment variables commonly required in enterprise deployment scripts
# !! IMPORTANT !!
# Set SSH_PRIVATE_KEY_PATH to the path of the private key you want to use for deployment to Pantheon. This key should not be password protected.
# Usage:
# ./deploy-dev <site-name or uuid> <environment>
#
# Additional examples of common enterprise deployment tasks are included in this script, but are not enabled by default.
#
# ###
SITE=$1
ENV=$2
current_dir=$3
START=$SECONDS
echo -e "Starting ${SITE} code deployment to ${ENV} environment" | sed "s/^/[$SITE] /" >>"${current_dir}/logs/deploy-$ENV.log"
# Configure script variable for use in other functions
function setup_variables {
# !! IMPORTANT !! Change this value to the path of the private key you want to use for deployment to Pantheon. This key should not be password protected.
# Example: SSH_PRIVATE_KEY_PATH="~/.ssh/id_rsa"
SSH_PRIVATE_KEY_PATH=
# if the SSH_PRIVATE_KEY_PATH is not set, ask the user which key to use. Exit if the user doesn't provide a key.
if [[ -z "$SSH_PRIVATE_KEY_PATH" ]]; then
echo "SSH_PRIVATE_KEY_PATH is not set. Please enter the path to the private key you want to use for deployment to Pantheon."
read -p "Enter the path to the private key: " SSH_PRIVATE_KEY_PATH
if [[ -z "$SSH_PRIVATE_KEY_PATH" ]]; then
echo "SSH_PRIVATE_KEY_PATH is not set. Exiting."
exit 1
fi
fi
# Set the Pantheon remote URL for the site
PANTHEON_REMOTE=$(terminus connection:info "$SITE.dev" --field=git_url)
echo "Adding Pantheon remote to git repo. Value of PANTHEON_REMOTE: $PANTHEON_REMOTE"
}
# Setup ssh config for drush.in hosts.
# This is to help avoid password prompts when using git and ssh.
function ssh_config_setup() {
echo -e "Host *.drush.in\n StrictHostKeyChecking no\n PasswordAuthentication no\n HostkeyAlgorithms +ssh-rsa\n PubkeyAcceptedAlgorithms +ssh-rsa" >> ~/.ssh/config
}
# Set system environment variables for ssh / git ASKPASS in case a GUI is currently being used
function ssh_askpass_setup() {
unset SSH_ASKPASS
export GIT_ASKPASS=
}
# Ensure that the SSH agent is running and has the private key loaded for this site.
# This is critical for avoiding password prompts when using git and ssh.
function ssh_agent_setup() {
# set SSH_PRIVATE_KEY to the contents of the private key
SSH_PRIVATE_KEY=$(cat $SSH_PRIVATE_KEY_PATH)
echo "$SSH_PRIVATE_KEY" > private.key
chmod 600 private.key
eval `ssh-agent -s`
ssh-add private.key
ssh-add -L | ssh-keygen -l -E md5 -f - | awk '{print substr($2,5)}'
}
# Scan for and add the Pantheon SSH host keys to the known_hosts file for this site's code and app servers.
# This is to avoid prompts to confirm the authenticity of the host when connecting via SSH.
function ssh_host_key_scan {
SITE_ID=$(terminus site:lookup "$SITE")
ssh-keyscan -t rsa -p 2222 "appserver.dev.${SITE_ID}.drush.in" >>~/.ssh/known_hosts
ssh-keyscan -t rsa -p 2222 "codeserver.dev.${SITE_ID}.drush.in" >>~/.ssh/known_hosts
}
# Basic code push to Pantheon environment, assumes master branch is checked out
function code_push() {
# Push code to Pantheon environment
echo "Pushing code to $ENV environment"
git remote add pantheon $PANTHEON_REMOTE
git push pantheon HEAD:refs/heads/master
# if the push fails with a non-zero exit code, add the site to the failed-sites-<env>.txt file
if [[ "$?" -ne 0 ]]; then
echo "Code push failed for $SITE.$ENV"
echo "[code] $SITE" >>"${current_dir}/logs/failed-sites-$ENV.txt"
fi
}
# Wait for running workflows to complete - this is a workaround for workflow:wait unreliable behavior
# Configure TIMEOUT_SEC to set the maximum time to wait for workflows to complete per site
# Configure retries to set the maximum number of times to check for running workflows
function check_workflows {
local id=$1
START_TIME="$(date -u +%s)"
TIMEOUT_SEC=1400
local retries=25
while (($(date +%s) < $START_TIME + $TIMEOUT_SEC)); do
set +e
CURRENT_TIME="$(date -u +%s)"
elapsed_seconds=$(($CURRENT_TIME - $START_TIME))
if [[ -z "$id" ]]; then
WORKFLOW_RUNNING=$(terminus workflow:list "$SITE" --fields=id,status,env,workflow | grep $ENV)
else
WORKFLOW_STATUS=$(terminus workflow:info:status $SITE --id=$id --field=status)
fi
set +e
if [ $elapsed_seconds -gt $TIMEOUT_SEC ]; then
echo "timeout of $TIMEOUT_SEC sec"
# The workflow is still running after the timeout, so add the site to the failed-sites-$ENV.txt file
echo "[code] $SITE" >>"${current_dir}/logs/failed-sites-$ENV.txt"
exit 124
fi
echo "Total workflow time elapsed is $elapsed_seconds sec"
if [[ "$WORKFLOW_RUNNING" =~ "running" ]]; then
echo "Workflows running, waiting 15 seconds and checking again."
sleep 15
elif [[ "$WORKFLOW_RUNNING" =~ "failed" || "$WORKFLOW_STATUS" =~ "failed" ]]; then
echo "Workflows failed, exiting."
# The workflow failed, so add the site to the failed-sites-<env>.txt file
echo "[code] $SITE" >>"${current_dir}/logs/failed-sites-$ENV.txt"
return 1
else
echo "Workflows completed successfully."
break
fi
done
}
# Example: run a terminus command as a background task - eg to create a multidev environment if it doesn't exist
function background_task() {
set +e
# Create multidev in background if it doesn't exist - this is to speed up builds
echo "Creating multidev as a background task if it doesn't exist."
# Push to multidev env, create branch on Pantheon if it doesnt exist
# Check if multidev exists for current site and env
ENV_EXISTS=$(terminus multidev:list $SITE --format=list | grep -w $ENV)
# Doesn't exist; create it
# If ENV_exists is not empty, the environment exists. Check that MD_STATUS is not 0, if it is, then the multidev exists and we can skip creation.
if [[ -n "$ENV_EXISTS" ]]; then
echo "Site $SITE has a $ENV multidev. Skipping creation."
MD_STATUS=0
else
[[ -z "$ENV_EXISTS" ]]
echo "Site $SITE does not have a $ENV multidev. Creating."
./bg.sh "terminus -n multidev:create $SITE.dev $ENV --yes"
id=$(terminus workflow:list "$SITE" --fields=id,status,env,workflow | grep -w 'Create a Multidev environment' | grep -v "succeeded" | awk '{print $1}' | head -n 1)
check_workflows $id
MD_STATUS=0
fi
return $MD_STATUS
}
# Example: Remove untracked files from the git repository
function remove_untracked() {
set +e
UNTRACKED_FILES=$(git status -u)
if [[ -n "$UNTRACKED_FILES" ]]; then
echo "Untracked files found, attempting to remove."
git clean -f -d
else
echo "No untracked files found."
fi
}
# Example: Advanced automatic merge conflict resolution
function resolve_conflicts {
# SET CONFLICTS_LIST to the output of git conflicts
CONFLICTS_LIST=$(git conflicts)
if [[ -n "$CONFLICTS_LIST" ]]; then
echo "Merge conflicts found, attempting to resolve."
git theirs composer.lock
git commit --no-edit
fi
}
# Example: automatic cleanup of uncommitted files
function git_cleanup() {
MODIFIED_OR_ADDED_LIST=$(git status --porcelain)
if [[ -n "$MODIFIED_OR_ADDED_LIST" ]]; then
echo "Uncommitted files found:"
echo $MODIFIED_OR_ADDED_LIST
echo "Attempting to remove uncommitted files."
git clean -fdX
fi
}
# Optional: Push code to Pantheon environment, retrying up to 3 times if it failed.
# This is useful for managing sites with large codebases/diffs that may fail to push due to timeouts or other issues.
function code_push_retry() {
local retries=3
while [ $retries -gt 0 ]; do
set +e
echo "Pushing code to $ENV environment"
git remote add pantheon $PANTHEON_REMOTE
git push pantheon HEAD:refs/heads/master
PUSH_STATUS="$?"
if [[ "$PUSH_STATUS" == 0 ]]; then
echo "Pushed to Pantheon $SITE.$ENV successfully."
break
else
echo "Code push not successful, waiting 15 seconds and trying sync again."
sleep 15
retries=$((retries - 1))
fi
done
}
setup_variables
ssh_config_setup
ssh_askpass_setup
ssh_agent_setup
ssh_host_key_scan
# Uncomment the function call below to retry the code push to Pantheon if it fails
# code_push_retry
code_push 2>&1 | sed "s/^/[$SITE] /" >>"${current_dir}/logs/deploy-$ENV.log"
check_workflows 2>&1 | sed "s/^/[$SITE] /" >>"${current_dir}/logs/deploy-$ENV.log"
# if any of the functions above fail, the code deployment has failed. Add the site to the failed-sites-<env>.txt file.
if [[ $? -eq 0 ]]; then
echo "Code push tasks successful for $SITE.$ENV" | sed "s/^/[$SITE] /" >>"${current_dir}/logs/deploy-$ENV.log"
# if the site was
if [[ -f "${current_dir}/logs/failed-sites-$ENV.txt" ]]; then
sed -i "/$SITE/d" "${current_dir}/logs/failed-sites-$ENV.txt"
fi
else
echo "Code push tasks failed for $SITE.$ENV" | sed "s/^/[$SITE] /" >>"${current_dir}/logs/deploy-$ENV.log"
echo "[code] $SITE" >>"${current_dir}/logs/failed-sites-$ENV.txt"
fi
# Report time to results.
DURATION=$((SECONDS - START))
TIME_DIFF=$(bc <<<"scale=2; $DURATION / 60")
MIN=$(printf "%.2f" $TIME_DIFF)
echo -e "Finished dev code push tasks ${SITE} in ${MIN} minutes" | sed "s/^/[$SITE] /" >>"${current_dir}/logs/deploy-$ENV.log"
Raw
deploy-from-dev-to-test-wrapper.sh
#!/bin/bash
# Wrapper script to deploy code from Pantheon dev to test.
# This script enhances standard deployment workflows by adding:
# - Parallel execution of code and deployment tasks on many sites
# - Race condition handling for workflows, terminus commands, and git commands
# - Automatic retries of failed site deployments
# To use this script, run:
# ./deploy-from-dev-to-test-wrapper.sh
current_dir=$(dirname "$(realpath "${BASH_SOURCE[0]}")")
ENV=test
# Array of sites the script will run againt. Commonly the output of a terminus command. Eg: SITES=$(terminus org:sites [your-pantheon-org] --field name --tag deploy --format=list)
SITES=$(terminus org:sites [your-pantheon-org] --field name --tag deploy --format=list)
# export vars to be used in other scripts
export current_dir=$current_dir
export ENV=$ENV
export SITES=("${SITES[@]}")
# Create file with a list of sites to retry in failed-sites-<env>.txt.
# A site will be added to this list if code push or deployment tasks fail.
touch "${current_dir}/logs/failed-sites-$ENV.txt"
# Run code deployment tasks from Pantheon dev -> test environment in parallel.
# Adjust the number of jobs by modifying the --jobs flag. eg --jobs 30
function deploy_code_to_test() {
local SITES=("$@")
echo "Running code push tasks to Pantheon $ENV environment"
parallel --jobs 25 ./deploy-test "${SITES[@]}" "$ENV" "$current_dir"
}
# Run standard site deployment tasks such as drush commands in the Pantheon test environment in parallel.
# Adjust the number of jobs by modifying the --jobs flag. eg --jobs 30
function run_deployment_tasks() {
local SITES=("$@")
echo "Running deployment tasks on site $ENV environment"
parallel --jobs 25 ./deploy-tasks "${SITES[@]}" $ENV "$current_dir"
}
# Retry failed sites for up to 2 hours until all sites are deployed successfully.
# A site can be in the failed-sites-<env>.txt file if it fails to deploy code or run deployment tasks.
# deploy_code_to_test is re-run for each failed site listed in failed-sites-<env>.txt with the [code] tag.
# run_deployment tasks is re-run for each failed site listed in failed-sites-<env>.txt with the [deploy] tag.
# Update timeout to change the maximum time to retry failed sites.
function retry_failed_deployments() {
echo "Retrying failed sites in the Pantheon $ENV environment" >>"${current_dir}/logs/deploy-$ENV.log"
local start_time=$(date +%s)
local timeout=7200 # 2 hours
while [ -s "${current_dir}/logs/failed-sites-$ENV.txt" ]; do
local current_time=$(date +%s)
local elapsed_time=$((current_time - start_time))
if [ $elapsed_time -ge $timeout ]; then
echo "Timeout reached without resolving all site failures." >>"${current_dir}/logs/deploy-$ENV.log"
break
fi
if grep -q '\[code\]' "${current_dir}/logs/failed-sites-$ENV.txt"; then
mapfile -t SITES < <(grep '\[code\]' "${current_dir}/logs/failed-sites-$ENV.txt")
deploy_code_to_test "${SITES[@]}"
else
mapfile -t SITES < <(grep '\[deploy\]' "${current_dir}/logs/failed-sites-$ENV.txt")
run_deployment_tasks "${SITES[@]}"
fi
done
echo "Finished retrying failed sites" >>"${current_dir}/logs/deploy-$ENV.log"
}
deploy_code_to_test "${SITES[@]}"
run_deployment_tasks "${SITES[@]}"
retry_failed_deployments
TIMESTAMP=$(date "+%Y-%m-%d#%H:%M:%S")
LOG_FILENAME="~/release/logs/pantheon-dev-to-$ENV.$TIMESTAMP.log"
mv "${current_dir}/logs/deploy-$ENV.log" $LOG_FILENAME
echo "View log by running: cat $LOG_FILENAME"
Raw
deploy-from-local-to-dev-wrapper.sh
#!/bin/bash
# Wrapper script to deploy code from local to dev.
# This script enhances standard "git push" deployment processes by adding:
# - Parallel execution of code and deployment tasks on many sites
# - Setup of SSH and git configuration to avoid password prompts
# - Race condition handling for workflows, terminus commands, and git commands
# - Configuration of environment variables commonly required in enterprise deployment scripts
# - Automatic retries of failed site deployments
# To use this script, run:
# ./deploy-from-local-to-dev-wrapper.sh
current_dir=$(dirname "$(realpath "${BASH_SOURCE[0]}")")
ENV=dev
# Array of sites the script will run againt. Commonly the output of a terminus command. Eg: SITES=$(terminus org:sites [your-pantheon-org] --field name --tag deploy --format=list)
SITES=$(terminus org:sites [your-pantheon-org] --field name --tag deploy --format=list)
# export vars to be used in other scripts
export current_dir=$current_dir
export ENV=$ENV
export SITES=("${SITES[@]}")
# Create logs directory if it does not exist.
mkdir -p "${current_dir}/logs"
# Create file with a list of sites to retry in failed-sites-<env>.txt.
# A site will be added to this list if code push or deployment tasks fail.
touch "${current_dir}/logs/failed-sites-$ENV.txt"
# Run code deployment tasks to Pantheon dev environment in parallel.
# Adjust the number of jobs by modifying the --jobs flag. eg --jobs 30
function deploy_code_to_dev() {
local SITES=("$@")
echo "Running code push tasks to Pantheon $ENV environment"
parallel --jobs 25 ./deploy-dev "${SITES[@]}" "$ENV" "$current_dir"
}
# Run standard site deployment tasks such as drush commands in the Pantheon dev environment in parallel.
# Adjust the number of jobs by modifying the --jobs flag. eg --jobs 30
function run_deployment_tasks() {
local SITES=("$@")
echo "Running deployment tasks on site $ENV environment"
parallel --jobs 25 ./deploy-tasks "${SITES[@]}" $ENV "$current_dir"
}
# Retry failed sites for up to 2 hours until all sites are deployed successfully.
# A site can be in the failed-sites-dev.txt file if it fails to deploy code or run deployment tasks.
# deploy_code_to_dev is re-run for each failed site listed in failed-sites-dev.txt with the [code] tag.
# run_deployment tasks is re-run for each failed site listed in failed-sites-dev.txt with the [deploy] tag.
# Update timeout to change the maximum time to retry failed sites.
function retry_failed_deployments() {
echo "Retrying failed sites in the Pantheon $ENV environment" >>"${current_dir}/logs/deploy-$ENV.log"
local start_time=$(date +%s)
local timeout=7200 # 2 hours
while [ -s "${current_dir}/logs/failed-sites-$ENV.txt" ]; do
local current_time=$(date +%s)
local elapsed_time=$((current_time - start_time))
if [ $elapsed_time -ge $timeout ]; then
echo "Timeout reached without resolving all site failures." >>"${current_dir}/logs/deploy-$ENV.log"
break
fi
if grep -q '\[code\]' "${current_dir}/logs/failed-sites-$ENV.txt"; then
mapfile -t SITES < <(grep '\[code\]' "${current_dir}/logs/failed-sites-$ENV.txt")
deploy_code_to_dev "${SITES[@]}"
else
mapfile -t SITES < <(grep '\[deploy\]' "${current_dir}/logs/failed-sites-$ENV.txt")
run_deployment_tasks "${SITES[@]}"
fi
done
echo "Finished retrying failed sites" >>"${current_dir}/logs/deploy-$ENV.log"
}
deploy_code_to_dev "${SITES[@]}"
run_deployment_tasks "${SITES[@]}"
retry_failed_deployments
TIMESTAMP=$(date "+%Y-%m-%d#%H:%M:%S")
LOG_FILENAME="~/release/logs/local-to-$ENV.$TIMESTAMP.log"
mv "${current_dir}/logs/deploy-$ENV.log" $LOG_FILENAME
echo "View log by running: cat $LOG_FILENAME"
Raw
deploy-live
#!/bin/bash
# Runs tasks to release code from Pantheon test -> live environment including:
# - Race condition handling for workflows, terminus commands, and git commands
# - Configuration of environment variables commonly required in enterprise deployment scripts
# - Logging of failed site deployments for automatic retries
# Usage:
# ./deploy-live <site-name or uuid> <environment>
#
# ###
SITE=$1
ENV=$2
current_dir=$3
DEV=$(echo "${SITE}.dev")
TEST=$(echo "${SITE}.test")
LIVE=$(echo "${SITE}.live")
START=$SECONDS
echo -e "Starting live deploy of Pantheon site ${SITE} from Test" | sed "s/^/[$SITE] /" >>"${current_dir}/logs/deploy-$ENV.log"
# Basic code deploy from Pantheon test -> live environment
function code_deploy() {
# Deploy code to Pantheon environment
echo "Deploying code from Pantheon test to $ENV environment"
terminus env:deploy $LIVE --cc -n -q
# if the push fails with a non-zero exit code, add the site to the failed-sites-<env>.txt file
if [[ "$?" -ne 0 ]]; then
echo "Code push failed for $SITE.$ENV"
echo "[code] $SITE" >>"${current_dir}/logs/failed-sites-$ENV.txt"
fi
}
code_deploy 2>&1 | sed "s/^/[$SITE] /" >>"${current_dir}/logs/deploy-$ENV.log"
# if any of the functions above fail, the code deployment has failed. Add the site to the failed-sites-<env>.txt file.
if [[ $? -eq 0 ]]; then
echo "Code deploy tasks successful for $SITE.$ENV" | sed "s/^/[$SITE] /" >>"${current_dir}/logs/deploy-$ENV.log"
# if the site was
if [[ -f "${current_dir}/logs/failed-sites-$ENV.txt" ]]; then
sed -i "/$SITE/d" "${current_dir}/logs/failed-sites-$ENV.txt"
fi
else
echo "Code deploy tasks failed for $SITE.$ENV" | sed "s/^/[$SITE] /" >>"${current_dir}/logs/deploy-$ENV.log"
echo "[code] $SITE" >>"${current_dir}/logs/failed-sites-$ENV.txt"
fi
# Report time to results.
DURATION=$((SECONDS - START))
TIME_DIFF=$(bc <<<"scale=2; $DURATION / 60")
MIN=$(printf "%.2f" $TIME_DIFF)
echo -e "Finished live code deploy for ${SITE} in ${MIN} minutes" | sed "s/^/[$SITE] /" >>"${current_dir}/logs/deploy-$ENV.log"
Raw
deploy-tasks
#!/bin/bash
# Recommended enterprise deployment tasks for a Pantheon site environment.
# Add retries and error handling to ensure that deployment tasks complete successfully.
# Resolves common enterprise deployment challenges related to race conditions, timeouts, and idle containers.
# Logs failed site deploy tasks for automatic retries
# Usage:
# ./deploy-tasks <site-name or uuid> <environment>
SITE=$1
ENV=$2
current_dir=$3
START=$SECONDS
# # Example: Run drush [command], retrying up to 3 times if it fails
# drush_command() {
# echo "Running drush [command]..."
# retries=3
# while [[ $retries -gt 0 ]]; do
# terminus -n drush ${SITE}.${ENV} -- [command] -y 2>&1
# if [ "$?" = "0" ]; then
# break
# else
# echo "::warning:: drush [command] command failed - retrying in 30 seconds."
# sleep 30
# retries=$((retries-1))
# fi
# done
# }
# drush_command
# Wake idle containers to ensure backend services are available to execute deployment tasks
function wake_env() {
echo "Ensuring that Pantheon environments are ready to execute deployment tasks."
echo "Waking $ENV environment."
terminus -n env:wake "$SITE.$ENV"
}
function drush_cr() {
# Clear Drupal cache, retrying up to 3 times if it fails
echo "Clearing Drupal cache..."
retries=3
while [[ $retries -gt 0 ]]; do
terminus -n drush ${SITE}.${ENV} -- cr 2>&1
if [ "$?" = "0" ]; then
break
else
echo "::warning:: drush cr command failed - retrying in 30 seconds."
sleep 30
retries=$((retries - 1))
fi
done
# if the deploy task fails with a non-zero exit code, add the site to the failed-sites-<env>.txt file
if [[ "$?" -ne 0 ]]; then
echo "Deploy tasks failed for $SITE.$ENV"
echo "[deploy] $SITE" >>"${current_dir}/logs/failed-sites-$ENV.txt"
fi
}
# Run drush updb, retrying up to 3 times if it fails
function drush_updb() {
echo "Running Drupal updates..."
retries=3
while [[ $retries -gt 0 ]]; do
terminus -n drush ${SITE}.${ENV} -- updb -y 2>&1
if [ "$?" = "0" ]; then
break
else
echo "::warning:: drush updb command failed - retrying in 30 seconds."
sleep 30
retries=$((retries - 1))
fi
done
# if the deploy task fails with a non-zero exit code, add the site to the failed-sites-<env>.txt file
if [[ "$?" -ne 0 ]]; then
echo "Deploy tasks failed for $SITE.$ENV"
echo "[deploy] $SITE" >>"${current_dir}/logs/failed-sites-$ENV.txt"
fi
}
# Check if Drupal is bootstrapped, retrying up to 3 times if it fails
function check_bootstrap() {
local retries=3
local DRUPAL_BOOTSTRAPPED=1
echo "Checking if drupal is bootstrapped"
while [ $retries -gt 0 ]; do
terminus -n drush ${SITE}.${ENV} -- status --field=bootstrap 2>&1
DRUPAL_BOOTSTRAPPED="$?"
if [[ "$DRUPAL_BOOTSTRAPPED" == 0 ]]; then
echo "Drupal bootstrapped successfully."
break
else
echo "Drupal not bootstrapped, waiting 15 seconds and checking again."
sleep 15
retries=$((retries - 1))
if [[ $retries -eq 0 ]]; then
echo "Drupal not bootstrapped after 3 retries, exiting."
DRUPAL_BOOTSTRAPPED=1
# deploy task failed with a non-zero exit code, so add the site to the failed-sites-<env>.txt file
echo "Deploy tasks failed for $SITE.$ENV"
echo "[deploy] $SITE" >>"${current_dir}/logs/failed-sites-$ENV.txt"
fi
fi
done
return $DRUPAL_BOOTSTRAPPED
}
# Clear the Pantheon platform cache, retrying up to 3 times if it fails
function platform_clear_cache() {
echo "Clearing all Pantheon caches..."
retries=3
while [[ $retries -gt 0 ]]; do
# Clear the environment cache
terminus -n env:clear-cache "$SITE"."$ENV"
if [ "$?" = "0" ]; then
break
else
echo "::warning:: clearing Pantheon caches failed - retrying in 30 seconds."
sleep 30
retries=$((retries - 1))
fi
done
# if the deploy task fails with a non-zero exit code, add the site to the failed-sites-<env>.txt file
if [[ "$?" -ne 0 ]]; then
echo "Deploy tasks failed for $SITE.$ENV"
echo "[deploy] $SITE" >>"${current_dir}/logs/failed-sites-$ENV.txt"
fi
}
echo "Starting site deploy tasks on on ${SITE}.${ENV}" | sed "s/^/[$SITE] /" >>"${current_dir}/logs/deploy-$ENV.log"
env_wake 2>&1 | sed "s/^/[$SITE] /" >>"${current_dir}/logs/deploy-$ENV.log"
check_bootstrap 2>&1 | sed "s/^/[$SITE] /" >>"${current_dir}/logs/deploy-$ENV.log"
drush_updb | sed "s/^/[$SITE] /" >>"${current_dir}/logs/deploy-$ENV.log"
drush_cc 2>&1 | sed "s/^/[$SITE] /" >>"${current_dir}/logs/deploy-$ENV.log"
platform_clear_cache 2>&1 | sed "s/^/[$SITE] /" >>"${current_dir}/logs/deploy-$ENV.log"
# Add any additional drush commands or deploy tasks which should run here - see the drush_command function in this file for a template.
# drush_command 2>&1 | sed "s/^/[$SITE] /" >> "${current_dir}/logs/deploy-$ENV.log"
# if any of the functions above fail, the code deployment has failed. Add the site to the failed-sites.txt file.
if [[ $? -eq 0 ]]; then
echo "Site deploy tasks successful for $SITE.$ENV" | sed "s/^/[$SITE] /" >>"${current_dir}/logs/deploy-$ENV.log"
# if the site was
if [[ -f "${current_dir}/logs/failed-sites-$ENV.txt" ]]; then
sed -i "/$SITE/d" "${current_dir}/logs/failed-sites-$ENV.txt"
fi
else
echo "Site deploy tasks failed for $SITE.$ENV" | sed "s/^/[$SITE] /" >>"${current_dir}/logs/deploy-$ENV.log"
echo "[deploy] $SITE" >>"${current_dir}/logs/failed-sites-$ENV.txt"
fi
# Report time to results.
DURATION=$((SECONDS - START))
TIME_DIFF=$(bc <<<"scale=2; $DURATION / 60")
MIN=$(printf "%.2f" $TIME_DIFF)
echo -e "Finished ${SITE}.${ENV} in ${MIN} minutes" | sed "s/^/[$SITE] /" >>"${current_dir}/logs/deploy-$ENV.log"
Raw
deploy-test
#!/bin/bash
# Runs tasks to deploy code from Pantheon dev -> test environment including:
# - Race condition handling for workflows, terminus commands, and git commands
# - Configuration of environment variables commonly required in enterprise deployment scripts
# - Logging of failed site deployments for automatic retries
# Usage:
# ./deploy-test <site-name or uuid> <environment>
#
# ###
SITE=$1
ENV=$2
current_dir=$3
DEV=$(echo "${SITE}.dev")
TEST=$(echo "${SITE}.test")
LIVE=$(echo "${SITE}.live")
START=$SECONDS
echo -e "Starting ${SITE} code deployment to Pantheon ${ENV} environment from Dev" | sed "s/^/[$SITE] /" >>"${current_dir}/logs/deploy-$ENV.log"
# Basic code deploy from Pantheon dev -> test environment
function code_deploy() {
# Deploy code to Pantheon environment
echo "Deploying code from Pantheon dev to $ENV environment"
terminus env:deploy $TEST --cc -n -q
# if the push fails with a non-zero exit code, add the site to the failed-sites-<env>.txt file
if [[ "$?" -ne 0 ]]; then
echo "Code push failed for $SITE.$ENV"
echo "[code] $SITE" >>"${current_dir}/logs/failed-sites-$ENV.txt"
fi
}
code_deploy 2>&1 | sed "s/^/[$SITE] /" >>"${current_dir}/logs/deploy-$ENV.log"
# if any of the functions above fail, the code deployment has failed. Add the site to the failed-sites-<env>.txt file.
if [[ $? -eq 0 ]]; then
echo "Code deploy tasks successful for $SITE.$ENV" | sed "s/^/[$SITE] /" >>"${current_dir}/logs/deploy-$ENV.log"
# if the site was
if [[ -f "${current_dir}/logs/failed-sites-$ENV.txt" ]]; then
sed -i "/$SITE/d" "${current_dir}/logs/failed-sites-$ENV.txt"
fi
else
echo "Code deploy tasks failed for $SITE.$ENV" | sed "s/^/[$SITE] /" >>"${current_dir}/logs/deploy-$ENV.log"
echo "[code] $SITE" >>"${current_dir}/logs/failed-sites-$ENV.txt"
fi
# Report time to results.
DURATION=$((SECONDS - START))
TIME_DIFF=$(bc <<<"scale=2; $DURATION / 60")
MIN=$(printf "%.2f" $TIME_DIFF)
echo -e "Finished dev -> test code deploy tasks ${SITE} in ${MIN} minutes" | sed "s/^/[$SITE] /" >>"${current_dir}/logs/deploy-$ENV.log"
Raw
deploy-to-live-wrapper.sh
#!/bin/bash
# Wrapper script to deploy code from Pantheon test to live.
# This script hardens standard deployment workfloiws by adding:
# - Parallel execution of code and deployment tasks on many sites
# - Race condition handling for workflows, terminus commands, and git commands
# - Automatic retries of failed site deployments
# To use this script, run:
# ./deploy-to-live-wrapper.sh
current_dir=$(dirname "$(realpath "${BASH_SOURCE[0]}")")
ENV=live
# Array of sites the script will run againt. Commonly the output of a terminus command. Eg: SITES=$(terminus org:sites [your-pantheon-org] --field name --tag deploy --format=list)
SITES=$(terminus org:sites [your-pantheon-org] --field name --tag deploy --format=list)
# export vars to be used in other scripts
export current_dir=$current_dir
export ENV=$ENV
export SITES=("${SITES[@]}")
# Create file with a list of sites to retry in failed-sites-<env>.txt.
# A site will be added to this list if code push or deployment tasks fail.
touch "${current_dir}/logs/failed-sites-$ENV.txt"
# Run code deployment tasks from Pantheon dev -> test environment in parallel.
# Adjust the number of jobs by modifying the --jobs flag. eg --jobs 30
function deploy_code_to_live() {
local SITES=("$@")
echo "Running code push tasks to Pantheon $ENV environment"
parallel --jobs 10 .//deploy-live "${SITES[@]}" "$ENV" "$current_dir"
}
# Run standard site deployment tasks such as drush commands in the Pantheon test environment in parallel.
# Adjust the number of jobs by modifying the --jobs flag. eg --jobs 30
function run_deployment_tasks() {
local SITES=("$@")
echo "Running deployment tasks on site $ENV environment"
parallel --jobs 10 ./deploy-tasks "${SITES[@]}" $ENV "$current_dir"
}
# Retry failed sites for up to 2 hours until all sites are deployed successfully.
# A site can be in the failed-sites-<env>.txt file if it fails to deploy code or run deployment tasks.
# deploy_code_to_live is re-run for each failed site listed in failed-sites-<env>.txt with the [code] tag.
# run_deployment tasks is re-run for each failed site listed in failed-sites-<env>.txt with the [deploy] tag.
# Update timeout to change the maximum time to retry failed sites.
function retry_failed_deployments() {
echo "Retrying failed sites in the Pantheon $ENV environment" >>"${current_dir}/logs/deploy-$ENV.log"
local start_time=$(date +%s)
local timeout=7200 # 2 hours
while [ -s "${current_dir}/logs/failed-sites-$ENV.txt" ]; do
local current_time=$(date +%s)
local elapsed_time=$((current_time - start_time))
if [ $elapsed_time -ge $timeout ]; then
echo "Timeout reached without resolving all site failures." >>"${current_dir}/logs/deploy-$ENV.log"
break
fi
if grep -q '\[code\]' "${current_dir}/logs/failed-sites-$ENV.txt"; then
mapfile -t SITES < <(grep '\[code\]' "${current_dir}/logs/failed-sites-$ENV.txt")
deploy_code_to_live "${SITES[@]}"
else
mapfile -t SITES < <(grep '\[deploy\]' "${current_dir}/logs/failed-sites-$ENV.txt")
run_deployment_tasks "${SITES[@]}"
fi
done
echo "Finished retrying failed sites" >>"${current_dir}/logs/deploy-$ENV.log"
}
deploy_code_to_live "${SITES[@]}"
run_deployment_tasks "${SITES[@]}"
retry_failed_deployments
TIMESTAMP=$(date "+%Y-%m-%d#%H:%M:%S")
LOG_FILENAME="~/release/logs/pantheon-dev-to-$ENV.$TIMESTAMP.log"
mv "${current_dir}/logs/deploy-$ENV.log" $LOG_FILENAME
echo "View log by running: cat $LOG_FILENAME"
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment