Created
December 17, 2021 06:16
-
-
Save skovmand/897aee6e8cb97116b9a6a699ddb29628 to your computer and use it in GitHub Desktop.
Entangled Parsers
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
// Entangled API Request Parser | |
// Ref: https://library.humio.com/stable/docs/parsers/creating-a-parser/ | |
// Example inputs | |
// Dec 16 17:43:34 xerecation node[698]: [info] [1.2.3.4] GET /tags [status: 200] [4.1 ms] | |
// Dec 16 17:46:15 xerecation node[698]: [info] [1.2.3.4] GET /tracks/activity/running?page=1&orderBy=popularity&direction=desc&limit=10 [status: 200] [12.6 ms] | |
// Nov 15 08:52:38 xerecation node[708]: [info] [1.2.3.4] POST /tracks/store [status: 201] [168.1 ms] | |
// Nov 15 20:39:35 xerecation node[708]: [info] [1.2.3.4] POST /tracks/suggest-tags [status: 200] [218.1 ms] | |
// Redact the origin of the request from rawstring to show 1.2.3.4 | |
replace(field=@rawstring, regex="\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3}", with="1.2.3.4") | |
// Parse the current format of the request log | |
| /(?<rawRequestTime>.{3} \d+ \d+:\d+:\d+) (?<nodeName>.+) node\[\d+\]: \[(?<logLevel>info|warn|error)\] \[(?<requestIp>\d+\.\d+\.\d+\.\d+)\] (?<requestMethod>GET|POST|PUT|PATCH|DELETE|OPTIONS) (?<requestPath>.+) \[status: (?<responseStatus>\d+)\] \[(?<responseTimeMs>[\d\.]+) ms\]$/ | |
| url := parseUrl(field=requestPath) | |
| logLevel := upper(logLevel) | |
| parseTimestamp("MMM dd HH:mm:ss", field="rawRequestTime", timezone="Europe/Copenhagen") | |
| drop([rawRequestTime, requestIp]) |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
// Entangled Metadata Worker | |
// Ref: https://library.humio.com/stable/docs/parsers/creating-a-parser/ | |
// Example inputs | |
// Nov 15 19:37:05 xerecation node[14799]: [info] Processed Meghan Trainor - NO (id: 5ae2e875d9e53a0eec979406) | |
// Dec 16 17:20:37 xerecation node[33698]: [info] Processed NOAH - På Vej Hjem (id: 5ae2e875d9e53a0eec979526) | |
/^(?<rawRequestTime>.{3} \d+ \d+:\d+:\d+) (?<nodeName>.+) node\[\d+\]: \[(?<logLevel>info|warn|error)\] Processed (?<artistAndTitle>.+) \(id: (?<trackId>.+)\)$/ | |
| logLevel := upper(logLevel) | |
| parseTimestamp("MMM dd HH:mm:ss", field="rawRequestTime", timezone="Europe/Copenhagen") | |
| drop([rawRequestTime]) |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment