Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

in_syslog: Provide appending source address parameter #7651

Merged
merged 12 commits into from
Jul 24, 2023

Conversation

cosmo0920
Copy link
Contributor

@cosmo0920 cosmo0920 commented Jul 5, 2023

Closes #7581

Currently, in_syslog plugin does not offer adding source address feature.
This PR provides this feature.


Enter [N/A] in the box, if an item is not applicable to your change.

Testing
Before we can approve your change; please submit the following in a comment:

  • Example configuration file for the change
[SERVICE]
  Parsers_File  /path/to/parsers.conf # For example, /etc/fluent-bit/parsers.conf

[INPUT]
  Name syslog
  Port 5140
  Listen   0.0.0.0
  Parser syslog-rfc5424
  Mode     tcp
  Source_Address_Key source_host
  # Raw_Message_Key raw # Also, it can be working with this parameter.

[OUTPUT]
  Name stdout
  • Debug log output from testing the change
Fluent Bit v2.1.7
* Copyright (C) 2015-2022 The Fluent Bit Authors
* Fluent Bit is a CNCF sub-project under the umbrella of Fluentd
* https://fluentbit.io

[2023/07/05 19:01:35] [ info] Configuration:
[2023/07/05 19:01:35] [ info]  flush time     | 1.000000 seconds
[2023/07/05 19:01:35] [ info]  grace          | 5 seconds
[2023/07/05 19:01:35] [ info]  daemon         | 0
[2023/07/05 19:01:35] [ info] ___________
[2023/07/05 19:01:35] [ info]  inputs:
[2023/07/05 19:01:35] [ info]      syslog
[2023/07/05 19:01:35] [ info] ___________
[2023/07/05 19:01:35] [ info]  filters:
[2023/07/05 19:01:35] [ info] ___________
[2023/07/05 19:01:35] [ info]  outputs:
[2023/07/05 19:01:35] [ info]      stdout.0
[2023/07/05 19:01:35] [ info] ___________
[2023/07/05 19:01:35] [ info]  collectors:
[2023/07/05 19:01:35] [ info] [fluent bit] version=2.1.7, commit=0714d10889, pid=110032
[2023/07/05 19:01:35] [debug] [engine] coroutine stack size: 24576 bytes (24.0K)
[2023/07/05 19:01:35] [ info] [storage] ver=1.1.6, type=memory, sync=normal, checksum=off, max_chunks_up=128
[2023/07/05 19:01:35] [ info] [cmetrics] version=0.6.3
[2023/07/05 19:01:35] [ info] [ctraces ] version=0.3.1
[2023/07/05 19:01:35] [ info] [input:syslog:syslog.0] initializing
[2023/07/05 19:01:35] [ info] [input:syslog:syslog.0] storage_strategy='memory' (memory only)
[2023/07/05 19:01:35] [debug] [syslog:syslog.0] created event channels: read=21 write=22
[2023/07/05 19:01:35] [debug] [downstream] listening on 0.0.0.0:5140
[2023/07/05 19:01:35] [ info] [in_syslog] TCP server binding 0.0.0.0:5140
[2023/07/05 19:01:35] [debug] [stdout:stdout.0] created event channels: read=24 write=25
[2023/07/05 19:01:35] [ info] [sp] stream processor started
[2023/07/05 19:01:35] [ info] [output:stdout:stdout.0] worker #0 started
[2023/07/05 19:01:39] [debug] [input:syslog:syslog.0] error appending source_address : 2
[2023/07/05 19:01:39] [debug] [input chunk] update output instances with new chunk size diff=161, records=1, input=syslog.0
[2023/07/05 19:01:40] [debug] [task] created task=0x7f066401fcd0 id=0 OK
[2023/07/05 19:01:40] [debug] [output:stdout:stdout.0] task_id=0 assigned to thread #0
[0] syslog.0: [[1688551296.281394000, {}], {"pri"=>"14", "time"=>"2023-07-05T10:01:36.281394Z", "host"=>"-", "ident"=>"-", "pid"=>"-", "msgid"=>"-", "extradata"=>"-", "message"=>"dummy", "source_host"=>"tcp://192.168.11.5:52195"}]
[2023/07/05 19:01:40] [debug] [out flush] cb_destroy coro_id=0
[2023/07/05 19:01:40] [debug] [task] destroy task=0x7f066401fcd0 (task_id=0)
[2023/07/05 19:01:40] [debug] [input:syslog:syslog.0] error appending source_address : 2
[2023/07/05 19:01:40] [debug] [input chunk] update output instances with new chunk size diff=161, records=1, input=syslog.0
[2023/07/05 19:01:41] [debug] [task] created task=0x7f06640242a0 id=0 OK
[2023/07/05 19:01:41] [debug] [output:stdout:stdout.0] task_id=0 assigned to thread #0
[0] syslog.0: [[1688551297.279265000, {}], {"pri"=>"14", "time"=>"2023-07-05T10:01:37.279265Z", "host"=>"-", "ident"=>"-", "pid"=>"-", "msgid"=>"-", "extradata"=>"-", "message"=>"dummy", "source_host"=>"tcp://192.168.11.5:52195"}]
[2023/07/05 19:01:41] [debug] [out flush] cb_destroy coro_id=1
[2023/07/05 19:01:41] [debug] [task] destroy task=0x7f06640242a0 (task_id=0)
^C[2023/07/05 19:01:46] [engine] caught signal (SIGINT)
[2023/07/05 19:01:46] [ warn] [engine] service will shutdown in max 5 seconds
[2023/07/05 19:01:46] [ info] [engine] service has stopped (0 pending tasks)
[2023/07/05 19:01:46] [ info] [output:stdout:stdout.0] thread worker #0 stopping...
[2023/07/05 19:01:46] [ info] [output:stdout:stdout.0] thread worker #0 stopped
  • Attached Valgrind output that shows no leaks or memory corruption was found
==110164== 
==110164== HEAP SUMMARY:
==110164==     in use at exit: 95,170 bytes in 702 blocks
==110164==   total heap usage: 5,646 allocs, 4,944 frees, 1,791,261 bytes allocated
==110164== 
==110164== LEAK SUMMARY:
==110164==    definitely lost: 0 bytes in 0 blocks
==110164==    indirectly lost: 0 bytes in 0 blocks
==110164==      possibly lost: 0 bytes in 0 blocks
==110164==    still reachable: 95,170 bytes in 702 blocks
==110164==         suppressed: 0 bytes in 0 blocks
==110164== Reachable blocks (those to which a pointer was found) are not shown.
==110164== To see them, rerun with: --leak-check=full --show-leak-kinds=all
==110164== 
==110164== For lists of detected and suppressed errors, rerun with: -s
==110164== ERROR SUMMARY: 0 errors from 0 contexts (suppressed: 0 from 0)

If this is a change to packaging of containers or native binaries then please confirm it works for all targets.

  • Run local packaging test showing all targets (including any new ones) build.
  • Set ok-package-test label to test for all targets (requires maintainer to do).

Documentation

  • Documentation required for this feature

fluent/fluent-bit-docs#1151

Backporting

  • Backport to latest stable release.

Fluent Bit is licensed under Apache 2.0, by submitting this pull request I understand that this code will be released under the terms of that license.

@cosmo0920 cosmo0920 temporarily deployed to pr July 5, 2023 10:03 — with GitHub Actions Inactive
@cosmo0920 cosmo0920 temporarily deployed to pr July 5, 2023 10:03 — with GitHub Actions Inactive
@cosmo0920 cosmo0920 temporarily deployed to pr July 5, 2023 10:03 — with GitHub Actions Inactive
@cosmo0920 cosmo0920 temporarily deployed to pr July 5, 2023 10:31 — with GitHub Actions Inactive
@cosmo0920 cosmo0920 temporarily deployed to pr July 5, 2023 11:03 — with GitHub Actions Inactive
@cosmo0920 cosmo0920 temporarily deployed to pr July 5, 2023 11:03 — with GitHub Actions Inactive
@cosmo0920 cosmo0920 temporarily deployed to pr July 5, 2023 11:03 — with GitHub Actions Inactive
@cosmo0920 cosmo0920 temporarily deployed to pr July 5, 2023 11:27 — with GitHub Actions Inactive
@leonardo-albertovich leonardo-albertovich marked this pull request as draft July 5, 2023 13:46
@cosmo0920 cosmo0920 temporarily deployed to pr July 6, 2023 03:32 — with GitHub Actions Inactive
@cosmo0920 cosmo0920 temporarily deployed to pr July 6, 2023 03:32 — with GitHub Actions Inactive
@cosmo0920 cosmo0920 temporarily deployed to pr July 6, 2023 03:32 — with GitHub Actions Inactive
@cosmo0920 cosmo0920 temporarily deployed to pr July 6, 2023 03:54 — with GitHub Actions Inactive
@cosmo0920 cosmo0920 marked this pull request as ready for review July 6, 2023 08:11
@cosmo0920 cosmo0920 temporarily deployed to pr July 6, 2023 08:58 — with GitHub Actions Inactive
@cosmo0920 cosmo0920 temporarily deployed to pr July 6, 2023 08:58 — with GitHub Actions Inactive
@cosmo0920 cosmo0920 temporarily deployed to pr July 6, 2023 08:58 — with GitHub Actions Inactive
@cosmo0920 cosmo0920 temporarily deployed to pr July 6, 2023 09:21 — with GitHub Actions Inactive
Copy link
Collaborator

@leonardo-albertovich leonardo-albertovich left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We need to make a few changes so this doesn't cause trouble in the future :

Unify the implementations of append_raw_message_to_record_data and append_source_address_to_record_data in a single generic function that receives the type, pointer and length of the new value. As a starting point just add support for MSGPACK_OBJECT_BIN and MSGPACK_OBJECT_STR and for any other types return an error.

Let's make the most of this opportunity to improve code in this newly created function by making these changes :

  • Because flb_msgpack_expand_map already allocates a new buffer we can just avoid repackaging the data and return that buffer directly.
  • Modify the code of the newly created function so it has 3 possible result values : SUCCESS, MAP NOT MODIFIED and MAP EXPANSION ERROR

Then you need to modify pack_line to properly process the result code and only print an error message if MAP_EXPANSION_ERROR is returned.

Signed-off-by: Hiroshi Hatake <[email protected]>
@cosmo0920 cosmo0920 temporarily deployed to pr July 19, 2023 15:30 — with GitHub Actions Inactive
@cosmo0920 cosmo0920 temporarily deployed to pr July 19, 2023 15:30 — with GitHub Actions Inactive
@cosmo0920 cosmo0920 temporarily deployed to pr July 19, 2023 15:30 — with GitHub Actions Inactive
@cosmo0920 cosmo0920 temporarily deployed to pr July 19, 2023 15:34 — with GitHub Actions Inactive
@cosmo0920 cosmo0920 temporarily deployed to pr July 19, 2023 15:34 — with GitHub Actions Inactive
@cosmo0920 cosmo0920 temporarily deployed to pr July 19, 2023 15:34 — with GitHub Actions Inactive
{
int i;
int result;
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Initialize result to FLB_MAP_NOT_MODIFIED, othwewise if message_key_name is NULL the check in line 87 is accessing garbage which considering that the value of FLB_MAP_EXPAND_SUCCESS is 0 could cause this function to return garbage.


*result_buffer = mp_sbuf.data;
*result_size = mp_sbuf.size;
if (result != FLB_MAP_EXPAND_SUCCESS) {
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

As per the comment in line 42 you need to invert the clause in this conditional block and remove the result override.

Then you need to add an else block to the clause in line 82 so for any other values returned by flb_msgpack_expand_map result is set to FLB_MAP_EXPANSION_ERROR.

message_entry.val.via.str.ptr = message_buffer;
}
else {
result = FLB_MAP_NOT_MODIFIED;
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Add a new constant named FLB_MAP_EXPANSION_INVALID_VALUE_TYPE and use it in this line, trying to use a type that's not acceptable is an error, we don't want to miss it.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Got it. Will do.

Copy link
Collaborator

@leonardo-albertovich leonardo-albertovich left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

All of the comments I made need to be addressed, I wasn't going to formally request changes but there are some that could cause issues so I'm forced to.


msgpack_pack_map(&mp_pck, unpacked_buffer.data.via.map.size + 1);
return result;
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Remove this return and wrap the code in lines 77-84 in a conditional that only executes if result equals FLB_MAP_NOT_MODIFIED

@cosmo0920 cosmo0920 temporarily deployed to pr July 19, 2023 16:04 — with GitHub Actions Inactive
Signed-off-by: Hiroshi Hatake <[email protected]>
@cosmo0920 cosmo0920 temporarily deployed to pr July 20, 2023 06:39 — with GitHub Actions Inactive
@cosmo0920 cosmo0920 temporarily deployed to pr July 20, 2023 06:39 — with GitHub Actions Inactive
@cosmo0920 cosmo0920 temporarily deployed to pr July 20, 2023 06:39 — with GitHub Actions Inactive
@cosmo0920 cosmo0920 temporarily deployed to pr July 20, 2023 07:09 — with GitHub Actions Inactive
@edsiper
Copy link
Member

edsiper commented Jul 24, 2023

@leonardo-albertovich pls review latest changes

@edsiper edsiper merged commit 659a4db into master Jul 24, 2023
39 of 42 checks passed
@edsiper edsiper deleted the cosmo0920-append-source-address-in_syslog branch July 24, 2023 23:01
Wiston999 pushed a commit to Wiston999/fluent-bit that referenced this pull request Aug 7, 2023
* in_syslog: Append source_address into records if needed
leonardo-albertovich pushed a commit that referenced this pull request Oct 5, 2023
* in_syslog: Append source_address into records if needed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Syslog plugin to add IP address of source
3 participants