Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

core: endianness detection fix #9256

Merged
merged 2 commits into from
Aug 22, 2024

Conversation

leonardo-albertovich
Copy link
Collaborator

This PR addresses an issue with the byte order detection macro in flb_endian.h switching the buggy implementation with the proven cmake based detection mechanism used by msgpack-c.

@patrick-stephens
Copy link
Contributor

Should we have some tests to verify it is actually working as intended?

@rightblank
Copy link

LGTM

@leonardo-albertovich
Copy link
Collaborator Author

Should we have some tests to verify it is actually working as intended?

I tested this on my M3 mac and my s390 virtual machine and it worked as intended in both cases.

I'm quite confident because it's not an original snippet, it's the exact same thing msgpack-c does and it leverages cmake for it.

@rightblank
Copy link

Should we have some tests to verify it is actually working as intended?

I tested this on my M3 mac and my s390 virtual machine and it worked as intended in both cases.

I'm quite confident because it's not an original snippet, it's the exact same thing msgpack-c does and it leverages cmake for it.

@edsiper, would you please help to review and merge this PR? It should be merged before #9196

@leonardo-albertovich @mirko-lazarevic

flb-it-log_event_decoder and flb-it-log_event_encoder test output on x86
root@b5176f9f2a91:/home/yifu/fluent-bit/bin#  uname -a
Linux b5176f9f2a91 5.15.0-118-generic #128-Ubuntu SMP Fri Jul 5 09:28:59 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux

root@b5176f9f2a91:/home/yifu/fluent-bit/bin#  ./flb-it-log_event_decoder -vvvv
Test create_destroy:
  log_event_decoder.c:52: Check dec != NULL... ok
  SUCCESS: All conditions have passed.

Test init_destroy:
  log_event_decoder.c:67: Check ret == FLB_EVENT_DECODER_SUCCESS... ok
  SUCCESS: All conditions have passed.

Test decode_timestamp:
  log_event_decoder.c:94: Check ret == FLB_EVENT_DECODER_SUCCESS... ok
  log_event_decoder.c:99: Check tm.tm.tv_sec == 123456 && tm.tm.tv_nsec == 0... ok
  log_event_decoder.c:116: Check ret == FLB_EVENT_DECODER_SUCCESS... ok
  log_event_decoder.c:121: Check tm.tm.tv_sec == 123456 && tm.tm.tv_nsec == 123456... ok
  SUCCESS: All conditions have passed.

Test decode_object:
  log_event_decoder.c:166: Check ret == MSGPACK_UNPACK_SUCCESS... ok
  log_event_decoder.c:172: Check ret == FLB_EVENT_DECODER_SUCCESS... ok
  log_event_decoder.c:178: Check flb_time_equal(&tm, &event.timestamp)... ok
  log_event_decoder.c:184: Check json != NULL... ok
  log_event_decoder.c:188: Check strstr(json, "\"key1\":\"val1\"") != NULL... ok
  log_event_decoder.c:192: Check strstr(json, "\"key2\":\"val2\"") != NULL... ok
  SUCCESS: All conditions have passed.

Test decoder_next:
  log_event_decoder.c:236: Check ret == FLB_EVENT_DECODER_SUCCESS... ok
  log_event_decoder.c:243: Check ret == FLB_EVENT_DECODER_SUCCESS... ok
  log_event_decoder.c:248: Check flb_time_equal(&tm, &event.timestamp)... ok
  log_event_decoder.c:254: Check json != NULL... ok
  log_event_decoder.c:258: Check strstr(json, "\"key1\":\"val1\"") != NULL... ok
  log_event_decoder.c:262: Check strstr(json, "\"key2\":\"val2\"") != NULL... ok
  SUCCESS: All conditions have passed.

Summary:
  Count of all unit tests:        5
  Count of run unit tests:        5
  Count of failed unit tests:     0
  Count of skipped unit tests:    0
SUCCESS: All unit tests have passed.

root@b5176f9f2a91:/home/yifu/fluent-bit/bin#  ./flb-it-log_event_encoder -vvvv
Test basic_format_fluent_bit_v2:
  log_event_encoder.c:378: Check ret == FLB_EVENT_ENCODER_SUCCESS... ok
  log_event_encoder.c:384: Check ret == FLB_EVENT_ENCODER_SUCCESS... ok
  log_event_encoder.c:392: Check ret == FLB_EVENT_ENCODER_SUCCESS... ok
  log_event_encoder.c:407: Check ret == FLB_EVENT_ENCODER_SUCCESS... ok
  log_event_encoder.c:415: Check ret == FLB_EVENT_ENCODER_SUCCESS... ok
  log_event_encoder.c:144: Check msgpack_data != NULL... ok
  log_event_encoder.c:148: Check msgpack_size > 0... ok
  log_event_encoder.c:162: Check root.type == MSGPACK_OBJECT_ARRAY... ok
  log_event_encoder.c:167: Check root.via.array.size == 2... ok
  log_event_encoder.c:174: Check root.type == MSGPACK_OBJECT_ARRAY... ok
  log_event_encoder.c:179: Check root.via.array.size == 2... ok
  log_event_encoder.c:186: Check obj.type == MSGPACK_OBJECT_EXT || obj.type == MSGPACK_OBJECT_POSITIVE_INTEGER... ok
  log_event_encoder.c:193: Check obj.type == MSGPACK_OBJECT_MAP... ok
  log_event_encoder.c:206: Check obj.via.map.size == 0... ok
  log_event_encoder.c:104: Check map->type == MSGPACK_OBJECT_MAP... ok
  log_event_encoder.c:111: Check map->via.map.ptr[i_map].key.type == MSGPACK_OBJECT_STR... ok
  log_event_encoder.c:111: Check map->via.map.ptr[i_map].key.type == MSGPACK_OBJECT_STR... ok
  log_event_encoder.c:124: Check num == l->size/2... ok
  log_event_encoder.c:214: Check ret == 0... ok
  log_event_encoder.c:424: Check ret == 0... ok
  SUCCESS: All conditions have passed.

Test basic_format_fluent_bit_v1:
  log_event_encoder.c:442: Check ret == FLB_EVENT_ENCODER_SUCCESS... ok
  log_event_encoder.c:448: Check ret == FLB_EVENT_ENCODER_SUCCESS... ok
  log_event_encoder.c:456: Check ret == FLB_EVENT_ENCODER_SUCCESS... ok
  log_event_encoder.c:471: Check ret == FLB_EVENT_ENCODER_SUCCESS... ok
  log_event_encoder.c:479: Check ret == FLB_EVENT_ENCODER_SUCCESS... ok
  log_event_encoder.c:235: Check msgpack_data != NULL... ok
  log_event_encoder.c:239: Check msgpack_size > 0... ok
  log_event_encoder.c:253: Check root.type == MSGPACK_OBJECT_ARRAY... ok
  log_event_encoder.c:258: Check root.via.array.size == 2... ok
  log_event_encoder.c:265: Check obj.type == MSGPACK_OBJECT_EXT || obj.type == MSGPACK_OBJECT_POSITIVE_INTEGER... ok
  log_event_encoder.c:104: Check map->type == MSGPACK_OBJECT_MAP... ok
  log_event_encoder.c:111: Check map->via.map.ptr[i_map].key.type == MSGPACK_OBJECT_STR... ok
  log_event_encoder.c:111: Check map->via.map.ptr[i_map].key.type == MSGPACK_OBJECT_STR... ok
  log_event_encoder.c:124: Check num == l->size/2... ok
  log_event_encoder.c:274: Check ret == 0... ok
  log_event_encoder.c:488: Check ret == 0... ok
  SUCCESS: All conditions have passed.

Test basic_metadata_format_fluent_bit_v2:
  log_event_encoder.c:512: Check ret == FLB_EVENT_ENCODER_SUCCESS... ok
  log_event_encoder.c:518: Check ret == FLB_EVENT_ENCODER_SUCCESS... ok
  log_event_encoder.c:526: Check ret == FLB_EVENT_ENCODER_SUCCESS... ok
  log_event_encoder.c:541: Check ret == FLB_EVENT_ENCODER_SUCCESS... ok
  log_event_encoder.c:555: Check ret == FLB_EVENT_ENCODER_SUCCESS... ok
  log_event_encoder.c:564: Check ret == FLB_EVENT_ENCODER_SUCCESS... ok
  log_event_encoder.c:144: Check msgpack_data != NULL... ok
  log_event_encoder.c:148: Check msgpack_size > 0... ok
  log_event_encoder.c:162: Check root.type == MSGPACK_OBJECT_ARRAY... ok
  log_event_encoder.c:167: Check root.via.array.size == 2... ok
  log_event_encoder.c:174: Check root.type == MSGPACK_OBJECT_ARRAY... ok
  log_event_encoder.c:179: Check root.via.array.size == 2... ok
  log_event_encoder.c:186: Check obj.type == MSGPACK_OBJECT_EXT || obj.type == MSGPACK_OBJECT_POSITIVE_INTEGER... ok
  log_event_encoder.c:193: Check obj.type == MSGPACK_OBJECT_MAP... ok
  log_event_encoder.c:104: Check map->type == MSGPACK_OBJECT_MAP... ok
  log_event_encoder.c:111: Check map->via.map.ptr[i_map].key.type == MSGPACK_OBJECT_STR... ok
  log_event_encoder.c:111: Check map->via.map.ptr[i_map].key.type == MSGPACK_OBJECT_STR... ok
  log_event_encoder.c:124: Check num == l->size/2... ok
  log_event_encoder.c:200: Check ret == 0... ok
  log_event_encoder.c:104: Check map->type == MSGPACK_OBJECT_MAP... ok
  log_event_encoder.c:111: Check map->via.map.ptr[i_map].key.type == MSGPACK_OBJECT_STR... ok
  log_event_encoder.c:111: Check map->via.map.ptr[i_map].key.type == MSGPACK_OBJECT_STR... ok
  log_event_encoder.c:124: Check num == l->size/2... ok
  log_event_encoder.c:214: Check ret == 0... ok
  log_event_encoder.c:573: Check ret == 0... ok
  SUCCESS: All conditions have passed.

Test create_destroy:
  log_event_encoder.c:300: Check encoder != NULL... ok
  log_event_encoder.c:300: Check encoder != NULL... ok
  log_event_encoder.c:300: Check encoder != NULL... ok
  log_event_encoder.c:300: Check encoder != NULL... ok
  SUCCESS: All conditions have passed.

Test create_unsupported_format:
  log_event_encoder.c:319: Check encoder == NULL... ok
  log_event_encoder.c:319: Check encoder == NULL... ok
  SUCCESS: All conditions have passed.

Test init_destroy:
  log_event_encoder.c:341: Check ret == FLB_EVENT_ENCODER_SUCCESS... ok
  log_event_encoder.c:341: Check ret == FLB_EVENT_ENCODER_SUCCESS... ok
  log_event_encoder.c:341: Check ret == FLB_EVENT_ENCODER_SUCCESS... ok
  log_event_encoder.c:341: Check ret == FLB_EVENT_ENCODER_SUCCESS... ok
  SUCCESS: All conditions have passed.

Test init_unsupported_format:
  log_event_encoder.c:361: Check ret != FLB_EVENT_ENCODER_SUCCESS... ok
  log_event_encoder.c:361: Check ret != FLB_EVENT_ENCODER_SUCCESS... ok
  SUCCESS: All conditions have passed.

Test emit_raw_record:
  log_event_encoder.c:590: Check ret == FLB_EVENT_ENCODER_SUCCESS... ok
  log_event_encoder.c:595: Check encoder.output_length == 0... ok
  log_event_encoder.c:602: Check ret == FLB_EVENT_ENCODER_SUCCESS... ok
  log_event_encoder.c:609: Check ret == FLB_EVENT_ENCODER_SUCCESS... ok
  log_event_encoder.c:617: Check ret == 0... ok
  log_event_encoder.c:622: Check buf_size > 0... ok
  log_event_encoder.c:629: Check ret == FLB_EVENT_ENCODER_SUCCESS... ok
  log_event_encoder.c:638: Check ret == FLB_EVENT_ENCODER_SUCCESS... ok
  log_event_encoder.c:646: Check encoder.output_length > 0... ok
  SUCCESS: All conditions have passed.

Summary:
  Count of all unit tests:        8
  Count of run unit tests:        8
  Count of failed unit tests:     0
  Count of skipped unit tests:    0
SUCCESS: All unit tests have passed.

flb-it-log_event_decoder and flb-it-log_event_encoder test output on s390x
linux1@fluentbitvm:~/code/fluent-bit/build/bin$  uname -a
Linux fluentbitvm 5.15.0-117-generic #127-Ubuntu SMP Fri Jul 5 20:13:04 UTC 2024 s390x s390x s390x GNU/Linux
linux1@fluentbitvm:~/code/fluent-bit/build/bin$  ./flb-it-log_event_decoder -vvvv
Test create_destroy:
  log_event_decoder.c:52: Check dec != NULL... ok
  SUCCESS: All conditions have passed.

Test init_destroy:
  log_event_decoder.c:67: Check ret == FLB_EVENT_DECODER_SUCCESS... ok
  SUCCESS: All conditions have passed.

Test decode_timestamp:
  log_event_decoder.c:94: Check ret == FLB_EVENT_DECODER_SUCCESS... ok
  log_event_decoder.c:99: Check tm.tm.tv_sec == 123456 && tm.tm.tv_nsec == 0... ok
  log_event_decoder.c:116: Check ret == FLB_EVENT_DECODER_SUCCESS... ok
  log_event_decoder.c:121: Check tm.tm.tv_sec == 123456 && tm.tm.tv_nsec == 123456... ok
  SUCCESS: All conditions have passed.

Test decode_object:
  log_event_decoder.c:166: Check ret == MSGPACK_UNPACK_SUCCESS... ok
  log_event_decoder.c:172: Check ret == FLB_EVENT_DECODER_SUCCESS... ok
  log_event_decoder.c:178: Check flb_time_equal(&tm, &event.timestamp)... ok
  log_event_decoder.c:184: Check json != NULL... ok
  log_event_decoder.c:188: Check strstr(json, "\"key1\":\"val1\"") != NULL... ok
  log_event_decoder.c:192: Check strstr(json, "\"key2\":\"val2\"") != NULL... ok
  SUCCESS: All conditions have passed.

Test decoder_next:
  log_event_decoder.c:236: Check ret == FLB_EVENT_DECODER_SUCCESS... ok
  log_event_decoder.c:243: Check ret == FLB_EVENT_DECODER_SUCCESS... ok
  log_event_decoder.c:248: Check flb_time_equal(&tm, &event.timestamp)... ok
  log_event_decoder.c:254: Check json != NULL... ok
  log_event_decoder.c:258: Check strstr(json, "\"key1\":\"val1\"") != NULL... ok
  log_event_decoder.c:262: Check strstr(json, "\"key2\":\"val2\"") != NULL... ok
  SUCCESS: All conditions have passed.

Summary:
  Count of all unit tests:        5
  Count of run unit tests:        5
  Count of failed unit tests:     0
  Count of skipped unit tests:    0
SUCCESS: All unit tests have passed.

linux1@fluentbitvm:~/code/fluent-bit/build/bin$  ./flb-it-log_event_encoder -vvvv
Test basic_format_fluent_bit_v2:
  log_event_encoder.c:378: Check ret == FLB_EVENT_ENCODER_SUCCESS... ok
  log_event_encoder.c:384: Check ret == FLB_EVENT_ENCODER_SUCCESS... ok
  log_event_encoder.c:392: Check ret == FLB_EVENT_ENCODER_SUCCESS... ok
  log_event_encoder.c:407: Check ret == FLB_EVENT_ENCODER_SUCCESS... ok
  log_event_encoder.c:415: Check ret == FLB_EVENT_ENCODER_SUCCESS... ok
  log_event_encoder.c:144: Check msgpack_data != NULL... ok
  log_event_encoder.c:148: Check msgpack_size > 0... ok
  log_event_encoder.c:162: Check root.type == MSGPACK_OBJECT_ARRAY... ok
  log_event_encoder.c:167: Check root.via.array.size == 2... ok
  log_event_encoder.c:174: Check root.type == MSGPACK_OBJECT_ARRAY... ok
  log_event_encoder.c:179: Check root.via.array.size == 2... ok
  log_event_encoder.c:186: Check obj.type == MSGPACK_OBJECT_EXT || obj.type == MSGPACK_OBJECT_POSITIVE_INTEGER... ok
  log_event_encoder.c:193: Check obj.type == MSGPACK_OBJECT_MAP... ok
  log_event_encoder.c:206: Check obj.via.map.size == 0... ok
  log_event_encoder.c:104: Check map->type == MSGPACK_OBJECT_MAP... ok
  log_event_encoder.c:111: Check map->via.map.ptr[i_map].key.type == MSGPACK_OBJECT_STR... ok
  log_event_encoder.c:111: Check map->via.map.ptr[i_map].key.type == MSGPACK_OBJECT_STR... ok
  log_event_encoder.c:124: Check num == l->size/2... ok
  log_event_encoder.c:214: Check ret == 0... ok
  log_event_encoder.c:424: Check ret == 0... ok
  SUCCESS: All conditions have passed.

Test basic_format_fluent_bit_v1:
  log_event_encoder.c:442: Check ret == FLB_EVENT_ENCODER_SUCCESS... ok
  log_event_encoder.c:448: Check ret == FLB_EVENT_ENCODER_SUCCESS... ok
  log_event_encoder.c:456: Check ret == FLB_EVENT_ENCODER_SUCCESS... ok
  log_event_encoder.c:471: Check ret == FLB_EVENT_ENCODER_SUCCESS... ok
  log_event_encoder.c:479: Check ret == FLB_EVENT_ENCODER_SUCCESS... ok
  log_event_encoder.c:235: Check msgpack_data != NULL... ok
  log_event_encoder.c:239: Check msgpack_size > 0... ok
  log_event_encoder.c:253: Check root.type == MSGPACK_OBJECT_ARRAY... ok
  log_event_encoder.c:258: Check root.via.array.size == 2... ok
  log_event_encoder.c:265: Check obj.type == MSGPACK_OBJECT_EXT || obj.type == MSGPACK_OBJECT_POSITIVE_INTEGER... ok
  log_event_encoder.c:104: Check map->type == MSGPACK_OBJECT_MAP... ok
  log_event_encoder.c:111: Check map->via.map.ptr[i_map].key.type == MSGPACK_OBJECT_STR... ok
  log_event_encoder.c:111: Check map->via.map.ptr[i_map].key.type == MSGPACK_OBJECT_STR... ok
  log_event_encoder.c:124: Check num == l->size/2... ok
  log_event_encoder.c:274: Check ret == 0... ok
  log_event_encoder.c:488: Check ret == 0... ok
  SUCCESS: All conditions have passed.

Test basic_metadata_format_fluent_bit_v2:
  log_event_encoder.c:512: Check ret == FLB_EVENT_ENCODER_SUCCESS... ok
  log_event_encoder.c:518: Check ret == FLB_EVENT_ENCODER_SUCCESS... ok
  log_event_encoder.c:526: Check ret == FLB_EVENT_ENCODER_SUCCESS... ok
  log_event_encoder.c:541: Check ret == FLB_EVENT_ENCODER_SUCCESS... ok
  log_event_encoder.c:555: Check ret == FLB_EVENT_ENCODER_SUCCESS... ok
  log_event_encoder.c:564: Check ret == FLB_EVENT_ENCODER_SUCCESS... ok
  log_event_encoder.c:144: Check msgpack_data != NULL... ok
  log_event_encoder.c:148: Check msgpack_size > 0... ok
  log_event_encoder.c:162: Check root.type == MSGPACK_OBJECT_ARRAY... ok
  log_event_encoder.c:167: Check root.via.array.size == 2... ok
  log_event_encoder.c:174: Check root.type == MSGPACK_OBJECT_ARRAY... ok
  log_event_encoder.c:179: Check root.via.array.size == 2... ok
  log_event_encoder.c:186: Check obj.type == MSGPACK_OBJECT_EXT || obj.type == MSGPACK_OBJECT_POSITIVE_INTEGER... ok
  log_event_encoder.c:193: Check obj.type == MSGPACK_OBJECT_MAP... ok
  log_event_encoder.c:104: Check map->type == MSGPACK_OBJECT_MAP... ok
  log_event_encoder.c:111: Check map->via.map.ptr[i_map].key.type == MSGPACK_OBJECT_STR... ok
  log_event_encoder.c:111: Check map->via.map.ptr[i_map].key.type == MSGPACK_OBJECT_STR... ok
  log_event_encoder.c:124: Check num == l->size/2... ok
  log_event_encoder.c:200: Check ret == 0... ok
  log_event_encoder.c:104: Check map->type == MSGPACK_OBJECT_MAP... ok
  log_event_encoder.c:111: Check map->via.map.ptr[i_map].key.type == MSGPACK_OBJECT_STR... ok
  log_event_encoder.c:111: Check map->via.map.ptr[i_map].key.type == MSGPACK_OBJECT_STR... ok
  log_event_encoder.c:124: Check num == l->size/2... ok
  log_event_encoder.c:214: Check ret == 0... ok
  log_event_encoder.c:573: Check ret == 0... ok
  SUCCESS: All conditions have passed.

Test create_destroy:
  log_event_encoder.c:300: Check encoder != NULL... ok
  log_event_encoder.c:300: Check encoder != NULL... ok
  log_event_encoder.c:300: Check encoder != NULL... ok
  log_event_encoder.c:300: Check encoder != NULL... ok
  SUCCESS: All conditions have passed.

Test create_unsupported_format:
  log_event_encoder.c:319: Check encoder == NULL... ok
  log_event_encoder.c:319: Check encoder == NULL... ok
  SUCCESS: All conditions have passed.

Test init_destroy:
  log_event_encoder.c:341: Check ret == FLB_EVENT_ENCODER_SUCCESS... ok
  log_event_encoder.c:341: Check ret == FLB_EVENT_ENCODER_SUCCESS... ok
  log_event_encoder.c:341: Check ret == FLB_EVENT_ENCODER_SUCCESS... ok
  log_event_encoder.c:341: Check ret == FLB_EVENT_ENCODER_SUCCESS... ok
  SUCCESS: All conditions have passed.

Test init_unsupported_format:
  log_event_encoder.c:361: Check ret != FLB_EVENT_ENCODER_SUCCESS... ok
  log_event_encoder.c:361: Check ret != FLB_EVENT_ENCODER_SUCCESS... ok
  SUCCESS: All conditions have passed.

Test emit_raw_record:
  log_event_encoder.c:590: Check ret == FLB_EVENT_ENCODER_SUCCESS... ok
  log_event_encoder.c:595: Check encoder.output_length == 0... ok
  log_event_encoder.c:602: Check ret == FLB_EVENT_ENCODER_SUCCESS... ok
  log_event_encoder.c:609: Check ret == FLB_EVENT_ENCODER_SUCCESS... ok
  log_event_encoder.c:617: Check ret == 0... ok
  log_event_encoder.c:622: Check buf_size > 0... ok
  log_event_encoder.c:629: Check ret == FLB_EVENT_ENCODER_SUCCESS... ok
  log_event_encoder.c:638: Check ret == FLB_EVENT_ENCODER_SUCCESS... ok
  log_event_encoder.c:646: Check encoder.output_length > 0... ok
  SUCCESS: All conditions have passed.

Summary:
  Count of all unit tests:        8
  Count of run unit tests:        8
  Count of failed unit tests:     0
  Count of skipped unit tests:    0
SUCCESS: All unit tests have passed.

@edsiper edsiper merged commit ee797ab into master Aug 22, 2024
88 of 90 checks passed
@edsiper edsiper deleted the leonardo-master-endianness_detection_fix branch August 22, 2024 16:43
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
docs-required ok-package-test Run PR packaging tests
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants