# run filter script tests
$ bin/logstash -t -e 'filter{ruby{path => "path/to/compact_event.rb"}}'
[2018-06-07T14:20:22,777][INFO ][logstash.filters.ruby.script] Test run complete {:script_path=>"path/to/compact_event.rb", :results=>{:passed=>5, :failed=>0, :errored=>0}}
Configuration OK
[2018-06-07T14:20:22,790][INFO ][logstash.runner ] Using config.test_and_exit mode. Config Validation Result: OK. Exiting Logstash
-
-
Save colinsurprenant/dc8f55d2afdc03ec72ff6a3927eabca0 to your computer and use it in GitHub Desktop.
# this is an alternate compact function implementation which also removes keys with empty string values | |
# writing the tests for this is left as an excercise to the reader :D | |
def compact(h) | |
h.inject({}) do |result, (k, v)| | |
if v.is_a?(Hash) | |
result[k] = compact(v) | |
elsif v.is_a?(String) | |
result[k] = v unless v.empty? | |
elsif !v.nil? | |
result[k] = v | |
end | |
result | |
end | |
end |
# this is anoter alternate compact function implementation which also removes empty values and nil values in arrays. | |
def compact(h) | |
h.inject({}) do |result, (k, v)| | |
case v | |
when Hash | |
c = compact(v) | |
result[k] = c unless c.empty? | |
when String | |
result[k] = v unless v.empty? | |
when Array | |
c = v.delete_if{|e| e.nil? || (e.is_a?(String) && e.empty?)} | |
result[k] = c unless c.empty? | |
when NilClass | |
# nothing | |
else | |
result[k] = v | |
end | |
result | |
end | |
end |
def compact(h) | |
h.inject({}) do |result, (k, v)| | |
unless v.nil? | |
result[k] = v.is_a?(Hash) ? compact(v) : v | |
end | |
result | |
end | |
end | |
def filter(event) | |
return [LogStash::Event.new(compact(event.to_hash_with_metadata))] | |
end | |
test "remove keys with nil values" do | |
in_event { { "foo" => 1, "bar" => nil, "nested" => { "baz" => nil, "biz" => "yo" }} } | |
expect("return a single event") do |events| | |
events.size == 1 | |
end | |
expect("kept the foo key") do |events| | |
events.first.get("[foo]") == 1 | |
end | |
expect("kept the [nested][biz] key") do |events| | |
events.first.get("[nested][biz]") == "yo" | |
end | |
expect("remove the bar key") do |events| | |
!events.first.include?("[bar]") | |
end | |
expect("remove the baz key") do |events| | |
!events.first.include?("[nested][baz]") | |
end | |
end |
filter { | |
ruby { | |
path => "path/to/compact_event.rb" | |
} | |
} |
@joshjacques sorry for the late reply - seems like the gist mention never got to me - need to verify my email filters.
Your proposed modification seems ok. Note that one of the advantages of using the file-based Ruby script is that you can easily embed tests as you can see in lines 14-37. I would suggest you add tests cases for the empty string and the hyphen, that way you'll know that it works the way you intend it.
Thanks a lot for this scripts . You saved me.
For me only the alternate version worked and I had to add the filter() method.
Again: Thanks a lot. :-)
The alternate compact function saved me too, thanks!! It worked perfectly with large json docs that have nested keys with a lot of null values I wanted to cleanup. I really appreciate it!
Looks like it fails to remove nil values from json array like
{
"title": [
{
"abc": 123,
"timecode": nil
"cust_id": {
"other_cust_id": nil,
"cust_ref": nil,
"external_key": nil
},
]
}
A solution that I found is
def compact(h) h.inject({}) do |result, (k, v)| if v.is_a?(Array) result[k]=compact(Hash[*v.flatten]) elsif v.is_a?(Hash) result[k] = compact(v) elsif v.is_a?(String) result[k] = v unless v.empty? elsif !v.nil? result[k] = v end result end end
@alxsss right - I just added a new alternate_compact_event2.rb
which also removes nil
values and empty values. Let me know if that works for you.
@colinsurprenant,
I've got a question for you.
It's regarding line 9 in your "alternate_compact_events.rb" code. If I also wanted to remove any fields which only contained a hyphen "-" for a value, could I amend that line to include the following.
Would there be a "better" way to do that?
Thanks