Fix response header comparison in cloudwatch logs plugin

[b-markowski]

As RFC for HTTP/1.1 describes:

Each header field consists
of a name followed by a colon (":") and the field value. Field names
are case-insensitive.

Using strstr function can lead to errors with CloudWatch Logs plugin when the request goes via proxy which for example lower-case header names, despite AWS call being successful.

---

Enter [N/A] in the box, if an item is not applicable to your change.

Testing
Before we can approve your change; please submit the following in a comment:

• [N/A] Example configuration file for the change
• Debug log output from testing the change

fluent-bit-2.1.2 ./bin/fluent-bit -i forward -f 1 -o cloudwatch_logs -p log_group_name=fluentbit-test -p log_stream_name=docker-container-test-stream -p region=eu-central-1 -p endpoint=https://private-proxy.com -m '*'
Fluent Bit v2.1.2
* Copyright (C) 2015-2022 The Fluent Bit Authors
* Fluent Bit is a CNCF sub-project under the umbrella of Fluentd
* https://fluentbit.io

[2023/05/02 15:50:32] [ info] Configuration:
[2023/05/02 15:50:32] [ info]  flush time     | 1.000000 seconds
[2023/05/02 15:50:32] [ info]  grace          | 5 seconds
[2023/05/02 15:50:32] [ info]  daemon         | 0
[2023/05/02 15:50:32] [ info] ___________
[2023/05/02 15:50:32] [ info]  inputs:
[2023/05/02 15:50:32] [ info]      forward
[2023/05/02 15:50:32] [ info] ___________
[2023/05/02 15:50:32] [ info]  filters:
[2023/05/02 15:50:32] [ info] ___________
[2023/05/02 15:50:32] [ info]  outputs:
[2023/05/02 15:50:32] [ info]      cloudwatch_logs.0
[2023/05/02 15:50:32] [ info] ___________
[2023/05/02 15:50:32] [ info]  collectors:
[2023/05/02 15:50:32] [ info] [fluent bit] version=2.1.2, commit=, pid=166872
[2023/05/02 15:50:32] [debug] [engine] coroutine stack size: 24576 bytes (24.0K)
[2023/05/02 15:50:32] [ info] [storage] ver=1.4.0, type=memory, sync=normal, checksum=off, max_chunks_up=128
[2023/05/02 15:50:32] [ info] [cmetrics] version=0.6.1
[2023/05/02 15:50:32] [ info] [ctraces ] version=0.3.0
[2023/05/02 15:50:32] [ info] [input:forward:forward.0] initializing
[2023/05/02 15:50:32] [ info] [input:forward:forward.0] storage_strategy='memory' (memory only)
[2023/05/02 15:50:32] [debug] [forward:forward.0] created event channels: read=21 write=22
[2023/05/02 15:50:32] [debug] [in_fw] Listen='0.0.0.0' TCP_Port=24224
[2023/05/02 15:50:32] [debug] [downstream] listening on 0.0.0.0:24224
[2023/05/02 15:50:32] [ info] [input:forward:forward.0] listening on 0.0.0.0:24224
[2023/05/02 15:50:32] [debug] [cloudwatch_logs:cloudwatch_logs.0] created event channels: read=24 write=25
[2023/05/02 15:50:32] [debug] [aws_credentials] Initialized Env Provider in standard chain
[2023/05/02 15:50:32] [debug] [aws_credentials] creating profile (null) provider
[2023/05/02 15:50:32] [debug] [aws_credentials] Initialized AWS Profile Provider in standard chain
[2023/05/02 15:50:32] [debug] [aws_credentials] Not initializing EKS provider because AWS_ROLE_ARN was not set
[2023/05/02 15:50:32] [debug] [aws_credentials] Not initializing ECS Provider because AWS_CONTAINER_CREDENTIALS_RELATIVE_URI is not set
[2023/05/02 15:50:32] [debug] [aws_credentials] Initialized EC2 Provider in standard chain
[2023/05/02 15:50:32] [debug] [aws_credentials] Sync called on the EC2 provider
[2023/05/02 15:50:32] [debug] [aws_credentials] Init called on the env provider
[2023/05/02 15:50:32] [debug] [aws_credentials] Init called on the profile provider
[2023/05/02 15:50:32] [debug] [aws_credentials] Reading shared config file.
[2023/05/02 15:50:32] [debug] [aws_credentials] Reading shared credentials file.
[2023/05/02 15:50:32] [debug] [aws_credentials] upstream_set called on the EC2 provider
[2023/05/02 15:50:32] [ info] [sp] stream processor started
[2023/05/02 15:50:32] [ info] [output:cloudwatch_logs:cloudwatch_logs.0] worker #0 started
[2023/05/02 15:50:38] [debug] [input chunk] update output instances with new chunk size diff=226
[2023/05/02 15:50:38] [debug] [input chunk] update output instances with new chunk size diff=220
[2023/05/02 15:50:38] [debug] [task] created task=0x7f166c0d81c0 id=0 OK
[2023/05/02 15:50:38] [debug] [output:cloudwatch_logs:cloudwatch_logs.0] task_id=0 assigned to thread #0
[2023/05/02 15:50:38] [debug] [output:cloudwatch_logs:cloudwatch_logs.0] Using stream=docker-container-test-stream, group=fluentbit-test
[2023/05/02 15:50:38] [ info] [output:cloudwatch_logs:cloudwatch_logs.0] Creating log stream docker-container-test-stream in log group fluentbit-test
[2023/05/02 15:50:38] [debug] [upstream] KA connection #41 to private-proxy.com:443 is connected
[2023/05/02 15:50:38] [debug] [http_client] not using http_proxy for header
[2023/05/02 15:50:38] [debug] [aws_client] private-proxy.com: http_do=0, HTTP Status: 400
[2023/05/02 15:50:38] [debug] [upstream] KA connection #41 to private-proxy.com:443 is now available
[2023/05/02 15:50:38] [debug] [output:cloudwatch_logs:cloudwatch_logs.0] CreateLogStream http status=400
[2023/05/02 15:50:38] [ info] [output:cloudwatch_logs:cloudwatch_logs.0] Log Stream docker-container-test-stream already exists
[2023/05/02 15:50:38] [debug] [output:cloudwatch_logs:cloudwatch_logs.0] Using stream=docker-container-test-stream, group=fluentbit-test
[2023/05/02 15:50:38] [debug] [output:cloudwatch_logs:cloudwatch_logs.0] cloudwatch:PutLogEvents: events=2, payload=660 bytes
[2023/05/02 15:50:38] [debug] [output:cloudwatch_logs:cloudwatch_logs.0] Sending log events to log stream docker-container-test-stream
[2023/05/02 15:50:38] [debug] [upstream] KA connection #41 to private-proxy.com:443 has been assigned (recycled)
[2023/05/02 15:50:38] [debug] [http_client] not using http_proxy for header
[2023/05/02 15:50:39] [debug] [upstream] KA connection #41 to private-proxy.com:443 is now available
[2023/05/02 15:50:39] [debug] [output:cloudwatch_logs:cloudwatch_logs.0] PutLogEvents http status=200
[2023/05/02 15:50:39] [debug] [output:cloudwatch_logs:cloudwatch_logs.0] Sent events to docker-container-test-stream
[2023/05/02 15:50:39] [debug] [output:cloudwatch_logs:cloudwatch_logs.0] Sent 2 events to CloudWatch
[2023/05/02 15:50:39] [debug] [out flush] cb_destroy coro_id=0

• Attached Valgrind output that shows no leaks or memory corruption was found

==167870== 
==167870== HEAP SUMMARY:
==167870==     in use at exit: 761,444 bytes in 5,815 blocks
==167870==   total heap usage: 18,335 allocs, 12,520 frees, 6,678,915 bytes allocated
==167870== 
==167870== LEAK SUMMARY:
==167870==    definitely lost: 64 bytes in 1 blocks
==167870==    indirectly lost: 76 bytes in 2 blocks
==167870==      possibly lost: 0 bytes in 0 blocks
==167870==    still reachable: 761,304 bytes in 5,812 blocks
==167870==         suppressed: 0 bytes in 0 blocks
==167870== Rerun with --leak-check=full to see details of leaked memory
==167870== 
==167870== For lists of detected and suppressed errors, rerun with: -s
==167870== ERROR SUMMARY: 0 errors from 0 contexts (suppressed: 0 from 0)

If this is a change to packaging of containers or native binaries then please confirm it works for all targets.

• [ N/A ] Run local packaging test showing all targets (including any new ones) build.
• [ N/A ] Set ok-package-test label to test for all targets (requires maintainer to do).

Documentation

• [ N/A] Documentation required for this feature

Backporting

• [ N/A ] Backport to latest stable release.

---

Fluent Bit is licensed under Apache 2.0, by submitting this pull request I understand that this code will be released under the terms of that license.

[github-actions]

This PR is stale because it has been open 45 days with no activity. Remove stale label or comment or this will be closed in 10 days.

[PettitWesley]

@b-markowski We can merge this if you fix the commit message. it should be something like:

out_cloudwatch_logs: http headers are case insensitive 

https://github.com/fluent/fluent-bit/blob/master/CONTRIBUTING.md

[github-actions]

This PR is stale because it has been open 45 days with no activity. Remove stale label or comment or this will be closed in 10 days.