The following gist contains all files needed to take raw NGINX access logs and then conver them into a suitable format that is ready to ingest within OpenSearch. These components include the following:
- Mock NGINX logs
- Fluent Bit configuration file
- Fluent Bit parsers.conf file (This is the default and a user would not require to define)
- When converting to the proper format we use a lua script to perform this inline. To reduce complexity we do this in a single line, however a user may wish to abstract this and add as a seperate file
- TraceID and SpanID are set as the same values seen in the schema and are not dynamically generated
- timestamp is not seen in Standard Output and instead is only seen when sending the data to OpenSearch
- As we know this data is nginx we have hard coded the event, name, domain, kind, result, and type - This of course can be further customized
https://www.loom.com/share/2b331a4b98b04aabbda639e9d44020a3
- Copy data.log, parsers.conf, and fluent-bit.conf into the
/tmp
directory - Run the following docker command
sudo docker run -it -v /tmp/:/tmp/ fluent/fluent-bit /bin/fluent-bit -R /tmp/parsers.conf -c /tmp/fluent-bit.conf