Skip to content

Instantly share code, notes, and snippets.

@hartfordfive

hartfordfive/README.md

Last active December 21, 2021 18:57
Show Gist options
  • Save hartfordfive/8a273872dfa50672b757cfb9f96d4b10 to your computer and use it in GitHub Desktop.
Save hartfordfive/8a273872dfa50672b757cfb9f96d4b10 to your computer and use it in GitHub Desktop.
Download ZIP
Raw
README.md

Finding Elasticsearch Loggers

Description

In cases of advanced troubleshooting of Elasticsearch issues, it may be necessary at times to increase the verbosity of a specific logger within Elasticsearch. Unlike with Logstash, the loggers in Elasticsearch aren't specifically documented as there are far too many (thousands+).

Note: Running this tool will result in the cloning of the Elasticsearch repo locally if it doesn't already exist within the specified repo_dir

Command Usage

Syntax: es-logger-finder -r <PATH_TO_ELASTICSEARCH_GIT_DIR> -v <ES_MAJOR_MINOR_VERSION> -p '<LOGGER_PATTERN>' -c <CACHE_FILE_DIR> [-f|-d]
options:
-r | --repo_dir           Specify the local directory of the Elasticsearch git repo
-v | --es_version         Specify the version of Elasticsearch to use (MAJOR.MINOR)
-f | --force_update       Pull latest changes from remote down to local elasticsearch repo
-c | --cache_dir          Specify the path to the directory the pre-generated files containing all loggers
-p | --logger_pattern     Pattern to use when searching for a specific logger
-d | --debug              Enable debug mode

Example

In a case you wanted to increase the logging verbosity for all packages relating to searchable snapshots in Elasticsearch 7.14, you would run the following command to identify those relevant logers:

./find-loggers -r /home/user1/git_projects/elasticsearch/ -v 7.14 -p 'snapshots' -c /home/user1/es-logger-finder/

2021-12-17T22:54:14-0500 [INFO] Serching for loggers matching 'snapshot' in v7.14 ...
Already on '7.14'
Your branch is up to date with 'origin/7.14'.
--------------------- RESULTS -----------------------
org.elasticsearch.action.admin.cluster.snapshots
org.elasticsearch.action.admin.cluster.snapshots.clone
org.elasticsearch.action.admin.cluster.snapshots.create
org.elasticsearch.action.admin.cluster.snapshots.delete
org.elasticsearch.action.admin.cluster.snapshots.features
org.elasticsearch.action.admin.cluster.snapshots.get
org.elasticsearch.action.admin.cluster.snapshots.restore
org.elasticsearch.action.admin.cluster.snapshots.status
org.elasticsearch.client.searchable_snapshots
org.elasticsearch.client.snapshots
org.elasticsearch.http.snapshots
org.elasticsearch.index.snapshots
org.elasticsearch.index.snapshots.blobstore
org.elasticsearch.snapshots
org.elasticsearch.snapshots.mockstore
org.elasticsearch.snapshots.sourceonly
org.elasticsearch.xpack.core.ml.job.snapshot.upgrade
org.elasticsearch.xpack.core.searchablesnapshots
org.elasticsearch.xpack.ml.job.snapshot.upgrader
org.elasticsearch.xpack.ml.modelsnapshots
org.elasticsearch.xpack.ml.rest.modelsnapshots
org.elasticsearch.xpack.searchablesnapshots
org.elasticsearch.xpack.searchablesnapshots.action
org.elasticsearch.xpack.searchablesnapshots.action.cache
org.elasticsearch.xpack.searchablesnapshots.allocation
org.elasticsearch.xpack.searchablesnapshots.allocation.decider
org.elasticsearch.xpack.searchablesnapshots.cache.blob
org.elasticsearch.xpack.searchablesnapshots.cache.common
org.elasticsearch.xpack.searchablesnapshots.cache.full
org.elasticsearch.xpack.searchablesnapshots.cache.shared
org.elasticsearch.xpack.searchablesnapshots.hdfs
org.elasticsearch.xpack.searchablesnapshots.minio
org.elasticsearch.xpack.searchablesnapshots.preallocate
org.elasticsearch.xpack.searchablesnapshots.recovery
org.elasticsearch.xpack.searchablesnapshots.rest
org.elasticsearch.xpack.searchablesnapshots.s3
org.elasticsearch.xpack.searchablesnapshots.store
org.elasticsearch.xpack.searchablesnapshots.store.input
org.elasticsearch.xpack.searchablesnapshots.upgrade
-----------------------------------------------------
Total matching loggers:       39
-----------------------------------------------------

In the list you can see the org.elasticsearch.xpack.searchablesnapshots logger has been indentified. You can then dynamically update the log level for this logger via the DevTools console:

PUT /_cluster/settings
{
  "persistent": {
    "logger.org.elasticsearch.xpack.searchablesnapshots: "DEBUG"
  }
}

or via a curl call:

curl -XPUT  https://my-elasticsearch-host:9200/_cluster/settings -H 'Content-Type: application/json' -d'{"persistent": {"logger.org.elasticsearch.xpack.searchablesnapshots":"DEBUG"}}'

Once you've completed your investigation, you can then unset the logger to return it to its default log level:

PUT /_cluster/settings
{
  "persistent": {
    "logger.org.elasticsearch.xpack.searchablesnapshots: null
  }
}
Raw
es-logger-finder
#!/bin/bash
GIT_URL=https://github.com/elastic/elasticsearch.git
CURR_DIR=`pwd`
function join_path() {
echo "${1:+$1/}$2" | sed 's#//#/#g'
}
show_help()
{
SCRIPT_NAME=$(basename $0)
echo "Generate a diagnostics dump of Elasticsearch cluster running on ECK."
echo
echo "Syntax: $SCRIPT_NAME -r <PATH_TO_ELASTICSEARCH_GIT_DIR> -v <ES_MAJOR_MINOR_VERSION> -p '<LOGGER_PATTERN>' -c <CACHE_FILE_DIR> [-f|-d]"
echo "options:"
echo "-r | --repo_dir Specify the local directory of the Elasticsearch git repo"
echo "-v | --es_version Specify the version of Elasticsearch to use"
echo "-f | --force_update Pull latest changes from remote down to local elasticsearch repo"
echo "-c | --cache_dir Specify the path to the directory the pre-generated files containing all loggers"
echo "-p | --logger_pattern Pattern to use when searching for a specific logger"
echo "-d | --debug Enable debug mode"
echo
}
LOGGER_HELPER_REPO_DIR=''
LOGGER_HELPER_CACHE_FILE=''
LOGGER_HELPER_LOGGER_PATTERN=''
LOGGER_HELPER_DEBUG=0
LOGGER_HELPER_ES_VERSION=''
LOGGER_HELPER_FORCE_UPDATE=0
LOGGER_HELPER_CACHE_DIR='~/es-logger-helper'
LOGGER_HELPER_USE_AG=0
while true; do
# echo "\$1:\"$1\" \$2:\"$2\""
case "$1" in
-h | --help ) show_help; exit; ;;
-r | --repo_dir ) LOGGER_HELPER_REPO_DIR="$2"; shift 2 ;;
-v | --repo_dir ) LOGGER_HELPER_ES_VERSION="$2"; shift 2 ;;
-c | --cache_dir ) LOGGER_HELPER_CACHE_DIR="$2"; shift 2 ;;
-f | --force_update ) LOGGER_HELPER_FORCE_UPDATE="$2"; shift 2 ;;
-p | --logger_pattern ) LOGGER_HELPER_LOGGER_PATTERN="$2"; shift 2 ;;
-d | --debug ) LOGGER_HELPER_DEBUG="1"; shift ;;
-- ) shift; break ;;
* ) break ;;
esac
done
if [ "$LOGGER_HELPER_DEBUG" == "0" ]; then
unset LOGGER_HELPER_DEBUG
fi
LOGGER_HELPER_CACHE_FILE=$(join_path ${LOGGER_HELPER_CACHE_DIR} "elasticsearch_loggers_${LOGGER_HELPER_ES_VERSION}.txt")
LOGGER_HELPER_REPO_DIR=${LOGGER_HELPER_REPO_DIR}
LOCALREPO_VC_DIR=$(join_path ${LOGGER_HELPER_REPO_DIR} .git)
function log_msg() {
SEVERITY=$(echo "$1" | tr '[:lower:]' '[:upper:]')
MSG=$2
if [ "$SEVERITY" == "DEBUG" ] && [ "${LOGGER_HELPER_DEBUG}" != "1" ]; then
return
fi
echo "$(date +"%Y-%m-%dT%H:%M:%S%z") [$SEVERITY] $MSG"
if [ "$SEVERITY" == "FATAL" ]; then
exit 1
fi
}
function check_dependencies() {
RES=$(which ag > /dev/null 2>&1)
if [ $? -ne "0" ]; then
RES=$(which grep > /dev/null 2>&1)
if [ "$RES" -ne "0" ]; then
log_msg FATAL "Missing comand 'grep'. Please install before using this command."
fi
export LOGGER_HELPER_USE_AG=0
log_msg WARN "Missing comand 'ag'. Using grep instead."
fi
RES=$(which git > /dev/null 2>&1)
if [ $? -ne "0" ]; then
log_msg FATAL "Missing comand 'git'. Please install before using this command."
fi
}
function create_cache_dir() {
if [ ! -d ${LOGGER_HELPER_CACHE_DIR} ]; then
log_msg INFO "Creating base cache directory: ${LOGGER_HELPER_CACHE_DIR}"
mkdir -p "${LOGGER_HELPER_CACHE_DIR}"
LOGGER_HELPER_CACHE_DIR=$(dirname $LOGGER_HELPER_CACHE_DIR)
log_msg INFO "Cache directory: ${LOGGER_HELPER_CACHE_DIR}"
fi
}
function checkout_version_branch() {
if [ ! -d ${LOCALREPO_VC_DIR} ]; then
log_msg INFO "Local repo not found. Cloning it."
git clone -b ${LOGGER_HELPER_ES_VERSION} ${GIT_URL} ${LOCAL_REPO}
else
cd ${LOGGER_HELPER_REPO_DIR}
#echo "Running: cd ${LOGGER_HELPER_REPO_DIR}; git checkout ${LOGGER_HELPER_ES_VERSION} ${GIT_URL}"
git checkout ${LOGGER_HELPER_ES_VERSION}
if [ "${LOGGER_HELPER_FORCE_UPDATE}" == "1" ]; then
[ "${LOGGER_HELPER_DEBUG}" == "1" ] && log_msg DEBUG "Updating codebase from remote..."
git fetch origin ${LOGGER_HELPER_ES_VERSION}
fi
fi
}
function extract_loggers_grep() {
[ "${LOGGER_HELPER_DEBUG}" == "1" ] && log_msg DEBUG "Extracting loggers with find+grep"
find ${LOGGER_HELPER_REPO_DIR} -name "*.java" | xargs grep '^package .*' | awk '{print $2}' | sed 's/;//g' | sort | uniq > ${LOGGER_HELPER_CACHE_FILE}
}
function extract_loggers_ag() {
[ "${LOGGER_HELPER_DEBUG}" == "1" ] && log_msg DEBUG "Extracting loggers with ag"
ag --java "^package .*" ${LOGGER_HELPER_REPO_DIR} | cut -d'\' -f2 | tr -d ';' | sort -u | awk '{print $2}' | sort -n | uniq > ${LOGGER_HELPER_CACHE_FILE}
}
function gen_cachefile() {
[ "${LOGGER_HELPER_DEBUG}" == "1" ] && log_msg DEBUG "Cache file name: ${LOGGER_HELPER_CACHE_FILE}"
[ "${LOGGER_HELPER_DEBUG}" == "1" ] && log_msg DEBUG "Repo dir: ${LOGGER_HELPER_REPO_DIR}"
if [ ! -f ${LOGGER_HELPER_CACHE_FILE} ]; then
[ "${LOGGER_HELPER_DEBUG}" == "1" ] && log_msg DEBUG "Populating logger cache file..."
log_msg DEBUG "Running search..."
if [ "${LOGGER_HELPER_USE_AG}" -eq "1" ]; then
extract_loggers_ag
else
extract_loggers_grep
fi
else
[ "${LOGGER_HELPER_DEBUG}" == "1" ] && log_msg DEBUG "Logger cache file ${LOGGER_HELPER_CACHE_FILE} exists."
fi
}
if [ "${LOGGER_HELPER_DEBUG}" == "1" ]; then
echo "'------------------- DEBUG ----------------------"
echo "LOGGER_HELPER_CACHE_FILE: ${LOGGER_HELPER_CACHE_FILE}"
echo "LOGGER_HELPER_REPO_DIR: ${LOGGER_HELPER_REPO_DIR}"
echo "LOCALREPO_VC_DIR: ${LOCALREPO_VC_DIR}"
echo "LOGGER_HELPER_CACHE_DIR: ${LOGGER_HELPER_CACHE_DIR}"
echo "LOGGER_HELPER_DEBUG: ${LOGGER_HELPER_DEBUG}"
echo "LOGGER_HELPER_USE_AG: ${LOGGER_HELPER_USE_AG}"
echo "-------------------------------------------------"
fi
log_msg INFO "Serching for loggers matching '${LOGGER_HELPER_LOGGER_PATTERN}' in Elasticsearch v${LOGGER_HELPER_ES_VERSION} ..."
check_dependencies
checkout_version_branch
create_cache_dir
gen_cachefile
RES=$(cat ${LOGGER_HELPER_CACHE_FILE} | grep "${LOGGER_HELPER_LOGGER_PATTERN}")
echo "--------------------- RESULTS -----------------------"
echo "$RES"
echo "-----------------------------------------------------"
echo "Total matching loggers: $(echo "$RES" | wc -l)"
echo "-----------------------------------------------------"
echo ""
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment