log event decoder/encoder errors appear when using syslog input

[sergei-polikarpov-sympower]

Describe the bug
syslog input and stdout output are configured in Fluent Bit.
When sending JSON formatted logs to the Fluent Bit syslog input, there is a ~70% chance that Fluent Bit won't accept these logs and start returning Log event decoder error errors on every attempt.
Additionally, if there is a Lua filter configured, Fluent Bit may start showing Log event encoder error errors as well.

To Reproduce
Configure syslog TCP input with JSON parser and stdout output:

[SERVICE]
    Flush         1
    Log_Level     warning
    Parsers_File  /etc/fluent-bit/parsers.conf
    HTTP_Server   On

[INPUT]
    Name    syslog
    Tag     syslog.*
    Parser  json
    Mode    tcp
    Listen  127.0.0.1
    Port    24330

[OUTPUT]
    Name    stdout
    Match   syslog.*

Send JSON formatted message to the syslog input:

# echo '{"key1": "value1", "key2": "value2"}' | nc -w 1 127.0.0.1 24330

Check Fluent Bit stdout output:

Oct 17 10:04:00 test-host fluent-bit[328251]: [2024/10/17 10:04:00] [error] [output:stdout:stdout.0] Log event decoder error : -11

Expected behavior
Fluent Bit processes syslog messages without raising the Log event decoder error or Log event encoder error errors.

Screenshots

Your Environment

• Version used:
  3.1.0
• Configuration:

[SERVICE]
    Flush         1
    Log_Level     warning
    Parsers_File  /etc/fluent-bit/parsers.conf
    HTTP_Server   On

[INPUT]
    Name    syslog
    Tag     syslog.*
    Parser  json
    Mode    tcp
    Listen  127.0.0.1
    Port    24330

[OUTPUT]
    Name    stdout
    Match   syslog.*

• Environment:
  Ubuntu 22.04.5 LTS

Additional context
The problem started to appear on version 3.1.0 and can be reproduced on the latest (3.1.9 at the moment) as well.
I'm not able to reproduce the issue on version 3.0.7.

As mentioned in the description, there is a chance that after restart Fluent Bit won't raise any errors while receiving the same message:

# systemctl restart fluent-bit
# echo '{"key1": "value1", "key2": "value2"}' | nc -w 1 127.0.0.1 24330

Oct 17 10:20:04 test-host fluent-bit[330262]: [0] syslog.*: [[1250179456.1250591744, {}], {"key1"=>"value1", "key2"=>"value2"}]

As also mentioned, adding a random Lua script may result in two errors:

# echo '{"key1": "value1", "key2": "value2"}' | nc -w 1 127.0.0.1 24330

Oct 17 10:31:51 test-host fluent-bit[331875]: [2024/10/17 10:31:51] [error] [filter:lua:lua.0] Log event encoder error : -11
Oct 17 10:31:52 test-host fluent-bit[331875]: [2024/10/17 10:31:52] [error] [output:stdout:stdout.0] Log event decoder error : -11