Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

in_snmp: an input plugin for collect metrics by SNMP request #7671

Open
wants to merge 6 commits into
base: master
Choose a base branch
from
Open

in_snmp: an input plugin for collect metrics by SNMP request #7671

wants to merge 6 commits into from

Conversation

k402xxxcenxxx
Copy link

@k402xxxcenxxx k402xxxcenxxx commented Jul 9, 2023 •
edited
Loading

Enter [N/A] in the box, if an item is not applicable to your change.

Testing
Before we can approve your change; please submit the following in a comment:

  • Example configuration file for the change
[INPUT]
    Name snmp
    Tag snmp.test
    Target_Host 172.17.0.3
    Community public
    Oid_Type get
    Oid 1.3.6.1.2.1.1.3.0
    Timeout 1
    Retries 20
[OUTPUT]
    Name stdout
    Match *
  • Debug log output from testing the change
$ valgrind --leak-check=full --show-leak-kinds=all ./bin/fluent-bit -c in_snmp.conf  -v
==26== Memcheck, a memory error detector
==26== Copyright (C) 2002-2017, and GNU GPL'd, by Julian Seward et al.
==26== Using Valgrind-3.16.1 and LibVEX; rerun with -h for copyright info
==26== Command: ./bin/fluent-bit -c in_snmp.conf -v
==26== 
Fluent Bit v2.1.7
* Copyright (C) 2015-2022 The Fluent Bit Authors
* Fluent Bit is a CNCF sub-project under the umbrella of Fluentd
* https://fluentbit.io

[2023/07/11 06:00:49] [ info] Configuration:
[2023/07/11 06:00:49] [ info]  flush time     | 1.000000 seconds
[2023/07/11 06:00:49] [ info]  grace          | 5 seconds
[2023/07/11 06:00:49] [ info]  daemon         | 0
[2023/07/11 06:00:49] [ info] ___________
[2023/07/11 06:00:49] [ info]  inputs:
[2023/07/11 06:00:49] [ info]      snmp
[2023/07/11 06:00:49] [ info] ___________
[2023/07/11 06:00:49] [ info]  filters:
[2023/07/11 06:00:49] [ info] ___________
[2023/07/11 06:00:49] [ info]  outputs:
[2023/07/11 06:00:49] [ info]      stdout.0
[2023/07/11 06:00:49] [ info] ___________
[2023/07/11 06:00:49] [ info]  collectors:
[2023/07/11 06:00:49] [ info] [fluent bit] version=2.1.7, commit=7182681f78, pid=26
[2023/07/11 06:00:49] [debug] [engine] coroutine stack size: 24576 bytes (24.0K)
[2023/07/11 06:00:49] [ info] [storage] ver=1.4.0, type=memory, sync=normal, checksum=off, max_chunks_up=128
[2023/07/11 06:00:49] [ info] [cmetrics] version=0.6.3
[2023/07/11 06:00:49] [ info] [ctraces ] version=0.3.1
[2023/07/11 06:00:49] [ info] [input:snmp:snmp.0] initializing
[2023/07/11 06:00:49] [ info] [input:snmp:snmp.0] storage_strategy='memory' (memory only)
[2023/07/11 06:00:49] [debug] [snmp:snmp.0] created event channels: read=21 write=22
[2023/07/11 06:00:49] [debug] [stdout:stdout.0] created event channels: read=23 write=24
[2023/07/11 06:00:49] [ info] [sp] stream processor started
[2023/07/11 06:00:49] [ info] [output:stdout:stdout.0] worker #0 started
[2023/07/11 06:00:50] [debug] [input chunk] update output instances with new chunk size diff=53, records=1, input=snmp.0
[2023/07/11 06:00:51] [debug] [task] created task=0x54cc990 id=0 OK
[2023/07/11 06:00:51] [debug] [output:stdout:stdout.0] task_id=0 assigned to thread #0
[0] snmp.test: [[1689055250.198336300, {}], {"iso.3.6.1.2.1.1.3.0"=>"57"}]
[2023/07/11 06:00:51] [debug] [input chunk] update output instances with new chunk size diff=53, records=1, input=snmp.0
[2023/07/11 06:00:51] [debug] [out flush] cb_destroy coro_id=0
[2023/07/11 06:00:51] [debug] [task] destroy task=0x54cc990 (task_id=0)
[2023/07/11 06:00:52] [debug] [task] created task=0x5532f10 id=0 OK
[2023/07/11 06:00:52] [debug] [output:stdout:stdout.0] task_id=0 assigned to thread #0
[0] snmp.test: [[1689055251.182227200, {}], {"iso.3.6.1.2.1.1.3.0"=>"58"}]
[2023/07/11 06:00:52] [debug] [out flush] cb_destroy coro_id=1
[2023/07/11 06:00:52] [debug] [input chunk] update output instances with new chunk size diff=53, records=1, input=snmp.0
[2023/07/11 06:00:52] [debug] [task] destroy task=0x5532f10 (task_id=0)
[2023/07/11 06:00:53] [debug] [task] created task=0x5598fd0 id=0 OK
[2023/07/11 06:00:53] [debug] [output:stdout:stdout.0] task_id=0 assigned to thread #0
[0] snmp.test: [[1689055252.137364100, {}], {"iso.3.6.1.2.1.1.3.0"=>"59"}]
[2023/07/11 06:00:53] [debug] [out flush] cb_destroy coro_id=2
[2023/07/11 06:00:53] [debug] [input chunk] update output instances with new chunk size diff=53, records=1, input=snmp.0
[2023/07/11 06:00:53] [debug] [task] destroy task=0x5598fd0 (task_id=0)
[2023/07/11 06:00:54] [debug] [task] created task=0x56010d0 id=0 OK
[0] snmp.test: [[1689055253.136002200, {}], {"iso.3.6.1.2.1.1.3.0"=>"60"}]
[2023/07/11 06:00:54] [debug] [output:stdout:stdout.0] task_id=0 assigned to thread #0
[2023/07/11 06:00:54] [debug] [out flush] cb_destroy coro_id=3
[2023/07/11 06:00:54] [debug] [input chunk] update output instances with new chunk size diff=53, records=1, input=snmp.0
[2023/07/11 06:00:54] [debug] [task] destroy task=0x56010d0 (task_id=0)
^C[2023/07/11 06:00:54] [engine] caught signal (SIGINT)
[2023/07/11 06:00:54] [debug] [task] created task=0x56671e0 id=0 OK
[2023/07/11 06:00:54] [debug] [output:stdout:stdout.0] task_id=0 assigned to thread #0
[0] snmp.test: [[1689055254.149898200, {}], {"iso.3.6.1.2.1.1.3.0"=>"61"}]
[2023/07/11 06:00:54] [ warn] [engine] service will shutdown in max 5 seconds
[2023/07/11 06:00:54] [ info] [input] pausing snmp.0
[2023/07/11 06:00:54] [debug] [out flush] cb_destroy coro_id=4
[2023/07/11 06:00:54] [debug] [task] destroy task=0x56671e0 (task_id=0)
[2023/07/11 06:00:55] [ info] [engine] service has stopped (0 pending tasks)
read_config_store open failure on /var/lib/snmp/snmp.conf
[2023/07/11 06:00:55] [ info] [input] pausing snmp.0
read_config_store open failure on /var/lib/snmp/snmp.conf
read_config_store open failure on /var/lib/snmp/snmp.conf
[2023/07/11 06:00:55] [ info] [output:stdout:stdout.0] thread worker #0 stopping...
[2023/07/11 06:00:55] [ info] [output:stdout:stdout.0] thread worker #0 stopped
==26== 
==26== HEAP SUMMARY:
==26==     in use at exit: 4,800 bytes in 1 blocks
==26==   total heap usage: 3,723 allocs, 3,722 frees, 2,543,883 bytes allocated
==26== 
==26== 4,800 bytes in 1 blocks are still reachable in loss record 1 of 1
==26==    at 0x483AB65: calloc (in /usr/lib/x86_64-linux-gnu/valgrind/vgpreload_memcheck-amd64-linux.so)
==26==    by 0x4AE51F7: netsnmp_init_mib_internals (in /usr/lib/x86_64-linux-gnu/libnetsnmp.so.40.0.0)
==26==    by 0x4AE9C8B: ??? (in /usr/lib/x86_64-linux-gnu/libnetsnmp.so.40.0.0)
==26==    by 0x4AFB5F4: init_snmp (in /usr/lib/x86_64-linux-gnu/libnetsnmp.so.40.0.0)
==26==    by 0x42BBD6: in_snmp_init (in_snmp.c:372)
==26==    by 0x1FC78D: flb_input_instance_init (flb_input.c:1189)
==26==    by 0x1FC925: flb_input_init_all (flb_input.c:1248)
==26==    by 0x23D4B0: flb_engine_start (flb_engine.c:730)
==26==    by 0x1DED52: flb_lib_worker (flb_lib.c:638)
==26==    by 0x4855EA6: start_thread (pthread_create.c:477)
==26==    by 0x5069A2E: clone (clone.S:95)
==26== 
==26== LEAK SUMMARY:
==26==    definitely lost: 0 bytes in 0 blocks
==26==    indirectly lost: 0 bytes in 0 blocks
==26==      possibly lost: 0 bytes in 0 blocks
==26==    still reachable: 4,800 bytes in 1 blocks
==26==         suppressed: 0 bytes in 0 blocks
==26== 
==26== For lists of detected and suppressed errors, rerun with: -s
==26== ERROR SUMMARY: 0 errors from 0 contexts (suppressed: 0 from 0
  • Attached Valgrind output that shows no leaks or memory corruption was found
==26== 
==26== HEAP SUMMARY:
==26==     in use at exit: 4,800 bytes in 1 blocks
==26==   total heap usage: 3,723 allocs, 3,722 frees, 2,543,883 bytes allocated
==26== 
==26== 4,800 bytes in 1 blocks are still reachable in loss record 1 of 1
==26==    at 0x483AB65: calloc (in /usr/lib/x86_64-linux-gnu/valgrind/vgpreload_memcheck-amd64-linux.so)
==26==    by 0x4AE51F7: netsnmp_init_mib_internals (in /usr/lib/x86_64-linux-gnu/libnetsnmp.so.40.0.0)
==26==    by 0x4AE9C8B: ??? (in /usr/lib/x86_64-linux-gnu/libnetsnmp.so.40.0.0)
==26==    by 0x4AFB5F4: init_snmp (in /usr/lib/x86_64-linux-gnu/libnetsnmp.so.40.0.0)
==26==    by 0x42BBD6: in_snmp_init (in_snmp.c:372)
==26==    by 0x1FC78D: flb_input_instance_init (flb_input.c:1189)
==26==    by 0x1FC925: flb_input_init_all (flb_input.c:1248)
==26==    by 0x23D4B0: flb_engine_start (flb_engine.c:730)
==26==    by 0x1DED52: flb_lib_worker (flb_lib.c:638)
==26==    by 0x4855EA6: start_thread (pthread_create.c:477)
==26==    by 0x5069A2E: clone (clone.S:95)
==26== 
==26== LEAK SUMMARY:
==26==    definitely lost: 0 bytes in 0 blocks
==26==    indirectly lost: 0 bytes in 0 blocks
==26==      possibly lost: 0 bytes in 0 blocks
==26==    still reachable: 4,800 bytes in 1 blocks
==26==         suppressed: 0 bytes in 0 blocks
==26== 
==26== For lists of detected and suppressed errors, rerun with: -s
==26== ERROR SUMMARY: 0 errors from 0 contexts (suppressed: 0 from 0

There is one leak and can be fixed after this patch is release: net-snmp/net-snmp@4bd0d9a
If this is a change to packaging of containers or native binaries then please confirm it works for all targets.

  • Run local packaging test showing all targets (including any new ones) build.
  • Set ok-package-test label to test for all targets (requires maintainer to do).

Documentation

  • Documentation required for this feature

fluent/fluent-bit-docs#1162
Backporting

  • Backport to latest stable release.

Fluent Bit is licensed under Apache 2.0, by submitting this pull request I understand that this code will be released under the terms of that license.

@k402xxxcenxxx k402xxxcenxxx changed the title Feature/in snmp in_snmp: an input plugin for collect metrics by SNMP request Jul 15, 2023
@k402xxxcenxxx k402xxxcenxxx mentioned this pull request Jul 15, 2023
Open
@k402xxxcenxxx k402xxxcenxxx marked this pull request as ready for review July 15, 2023 07:17
@k402xxxcenxxx
Copy link
Author

k402xxxcenxxx commented Jul 15, 2023 •
edited
Loading

I have found a leak in the code, and it can be resolved by applying this patch: net-snmp/net-snmp@4bd0d9a.

However, I am unsure about the process of incorporating this fix into the fluent-bit project. Any suggestions or guidance on this matter would be greatly appreciated!

@patrick-stephens patrick-stephens added the ok-package-test Run PR packaging tests label Jul 17, 2023
patrick-stephens
patrick-stephens reviewed Jul 17, 2023
View reviewed changes
Copy link
Contributor

@patrick-stephens patrick-stephens left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I've triggered a full build of all targets to confirm it builds for them, we probably need to modify dependencies to build.

Can you confirm it works for non-Linux targets as well? It may build but that doesn't mean it works...

CMakeLists.txt
@@ -191,6 +192,7 @@ option(FLB_IN_HEAD "Enable Head input plugin"
option(FLB_IN_PROC "Enable Process input plugin" Yes)
option(FLB_IN_SYSTEMD "Enable Systemd input plugin" Yes)
option(FLB_IN_DUMMY "Enable Dummy input plugin" Yes)
option(FLB_IN_SNMP "Enable SNMP input plugin" Yes)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

To confirm - does this work on Windows and macOS as well? It will need disabling if not in their specific override cmake/ files.

dockerfiles/Dockerfile
@@ -53,6 +53,7 @@ RUN echo "deb http://deb.debian.org/debian bullseye-backports main" >> /etc/apt/
flex \
bison \
libyaml-dev \
libsnmp-dev \
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We presumably will also need this in the various package builds under packaging/distros too.

@k402xxxcenxxx
Copy link
Author

@patrick-stephens thanks, I will try to build and run on Windows device

@k402xxxcenxxx
snmp: borrow dummy plugin as a start point
9a28d22 
Add: ChatGPT in_snmp

Add: snmp get with fix request

Add: snmp with log recorder

Add: print value by sprint_realloc_by_type

Fix: refine code.

Add: walk get values

TMP

Signed-off-by: k402xxxcenxxx <[email protected]>
@k402xxxcenxxx
Add: unit test
7295681 
Add: unit test for snmp_walk
Signed-off-by: k402xxxcenxxx <[email protected]>
@k402xxxcenxxx
Fix: memory leak due to openssl lib used by net-snmp
82c8f8d 
Signed-off-by: k402xxxcenxxx <[email protected]>
@k402xxxcenxxx
Add: timeout and retris setting for net snmp lib
acef8b0 
Signed-off-by: k402xxxcenxxx <[email protected]>
@k402xxxcenxxx
Add: snmp in Dockerfile
0a0e1bb 
Signed-off-by: k402xxxcenxxx <[email protected]>
@k402xxxcenxxx
Copy link
Author

I still fail to build it on Windows, so I choose to disable it in https://github.com/fluent/fluent-bit/blob/master/cmake/windows-setup.cmake.
And I'm trying to build to various distribution

@k402xxxcenxxx k402xxxcenxxx temporarily deployed to pr August 21, 2023 10:22 — with GitHub Actions Inactive
@k402xxxcenxxx k402xxxcenxxx temporarily deployed to pr August 21, 2023 10:22 — with GitHub Actions Inactive
@k402xxxcenxxx k402xxxcenxxx temporarily deployed to pr August 21, 2023 10:22 — with GitHub Actions Inactive
@k402xxxcenxxx k402xxxcenxxx temporarily deployed to pr August 21, 2023 10:53 — with GitHub Actions Inactive
@github-actions GitHub Actions
Copy link
Contributor

This PR is stale because it has been open 45 days with no activity. Remove stale label or comment or this will be closed in 10 days.

@github-actions github-actions bot added the Stale label Dec 11, 2023
@github-actions github-actions bot removed the Stale label Aug 16, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@patrick-stephens patrick-stephens patrick-stephens left review comments

@niedbalski niedbalski Awaiting requested review from niedbalski niedbalski is a code owner

@fujimotos fujimotos Awaiting requested review from fujimotos fujimotos is a code owner

@edsiper edsiper Awaiting requested review from edsiper edsiper is a code owner

@leonardo-albertovich leonardo-albertovich Awaiting requested review from leonardo-albertovich leonardo-albertovich is a code owner

@koleini koleini Awaiting requested review from koleini koleini is a code owner

@celalettin1286 celalettin1286 Awaiting requested review from celalettin1286 celalettin1286 is a code owner

At least 1 approving review is required to merge this pull request.

Assignees
No one assigned
Labels
docs-required ok-package-test Run PR packaging tests
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@k402xxxcenxxx @patrick-stephens