Skip to content

Commit

Permalink
idaholab#318, third party logs are not parsed correctly from fluentbi…
Browse files Browse the repository at this point in the history
…t -> fluentd aggregator -> Malcolm
  • Loading branch information
mmguero committed Jan 8, 2024
1 parent 8e714bb commit 41271fe
Showing 1 changed file with 14 additions and 15 deletions.
29 changes: 14 additions & 15 deletions logstash/pipelines/beats/11_beats_logs.conf
Original file line number Diff line number Diff line change
Expand Up @@ -8,6 +8,20 @@
filter {

# move a couple of things identifying the event source from under miscbeat up to the top level
if ([miscbeat][message][module]) and (![miscbeat][module]) {
# special case to handle fluent-bit -> fluentd.loomsystems -> filebeat TCP input
# move entire "message" contents up to root
# https://github.com/idaholab/Malcolm/issues/318
ruby {
id => "ruby_miscbeat_message_move_up"
code => "
event.get('[miscbeat][message]').each { |k, v|
event.set('[miscbeat][' + k + ']', v)
}
event.remove('[miscbeat][message]')
"
}
}
if ([miscbeat][host]) {
ruby {
id => "ruby_miscbeat_host_merge"
Expand Down Expand Up @@ -78,21 +92,6 @@ filter {
# https://docs.fluentbit.io/manual/
# https://www.elastic.co/guide/en/beats/filebeat/current/filebeat-input-tcp.html

if (![miscbeat][module]) and ([miscbeat][message][module]) {
# special case to handle fluent-bit -> fluentd.loomsystems -> filebeat TCP input
# move entire "message" contents up to root
# https://github.com/idaholab/Malcolm/issues/318
ruby {
id => "ruby_miscbeat_message_move_up"
code => "
event.get('[miscbeat][message]').each { |k, v|
event.set('[miscbeat][' + k + ']', v)
}
event.remove('[miscbeat][message]')
"
}
}

# set event.module to whatever we set "module" to when running fluentbit
if ([miscbeat][module]) {
mutate { id => "mutate_replace_event_module_miscbeat_module"
Expand Down

0 comments on commit 41271fe

Please sign in to comment.