[BUG] test_from_json_ints integration test failed due to mismatch #11511
Labels
bug
Something isn't working
Comments
nartal1
added
? - Needs Triage
|
Do you have anything more as far as what data is different? Also #11500 recently ran into some issues on dataproc serverless. In those cases all of the errors appeared to be because the tests were running against something that appeared to be the wrong version of the plugin. Are we 100% sure that the correct version of the plugin is being used? |
I pasted the the details of the failure under "Details of test failure:" in the description. Other than the that I don't see much in the logs. Checking if other tests are also failing in the run |
|
I see some other json tests failing in json_test.py. I will close this issue and we can track why the tests are failing on dataproc serverless in #11500 |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Describe the bug
Integration test is failing in dataproc-serverless due to mismatch in CPU and GPU results
Details of test failure:
[2024-09-26T15:46:13.629Z] =================================== FAILURES =================================== [2024-09-26T15:46:13.629Z] ___________________ test_from_json_ints[int_formatted.json] ____________________ [2024-09-26T15:46:13.629Z] [2024-09-26T15:46:13.629Z] std_input_path = 'rapids-it-dataproc-serverless-33/integration_tests/src/test/resources' [2024-09-26T15:46:13.629Z] input_file = 'int_formatted.json' [2024-09-26T15:46:13.629Z] [2024-09-26T15:46:13.629Z] @pytest.mark.parametrize('input_file', COMMON_TEST_FILES) [2024-09-26T15:46:13.629Z] @allow_non_gpu(TEXT_INPUT_EXEC, *non_utc_allow) # https://github.com/NVIDIA/spark-rapids/issues/10453 [2024-09-26T15:46:13.629Z] def test_from_json_ints(std_input_path, input_file): [2024-09-26T15:46:13.629Z] schema = StructType([StructField("data", IntegerType())]) [2024-09-26T15:46:13.629Z] > assert_gpu_and_cpu_are_equal_collect( [2024-09-26T15:46:13.629Z] lambda spark : read_json_as_text(spark, std_input_path + '/' + input_file, "json").select(f.col('json'), f.from_json(f.col('json'), schema)), [2024-09-26T15:46:13.629Z] conf =_enable_json_to_structs_conf) [2024-09-26T15:46:13.629Z] [2024-09-26T15:46:13.629Z] rapids-it-dataproc-serverless-33/integration_tests/src/main/python/json_matrix_test.py:639: [2024-09-26T15:46:13.629Z] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [2024-09-26T15:46:13.629Z] rapids-it-dataproc-serverless-33/integration_tests/src/main/python/asserts.py:599: in assert_gpu_and_cpu_are_equal_collect [2024-09-26T15:46:13.629Z] _assert_gpu_and_cpu_are_equal(func, 'COLLECT', conf=conf, is_cpu_first=is_cpu_first, result_canonicalize_func_before_compare=result_canonicalize_func_before_compare) [2024-09-26T15:46:13.629Z] rapids-it-dataproc-serverless-33/integration_tests/src/main/python/asserts.py:521: in _assert_gpu_and_cpu_are_equal [2024-09-26T15:46:13.629Z] assert_equal(from_cpu, from_gpu) [2024-09-26T15:46:13.629Z] rapids-it-dataproc-serverless-33/integration_tests/src/main/python/asserts.py:111: in assert_equal [2024-09-26T15:46:13.629Z] _assert_equal(cpu, gpu, float_check=get_float_check(), path=[]) [2024-09-26T15:46:13.629Z] rapids-it-dataproc-serverless-33/integration_tests/src/main/python/asserts.py:43: in _assert_equal [2024-09-26T15:46:13.629Z] _assert_equal(cpu[index], gpu[index], float_check, path + [index]) [2024-09-26T15:46:13.629Z] rapids-it-dataproc-serverless-33/integration_tests/src/main/python/asserts.py:36: in _assert_equal [2024-09-26T15:46:13.629Z] _assert_equal(cpu[field], gpu[field], float_check, path + [field]) [2024-09-26T15:46:13.629Z] rapids-it-dataproc-serverless-33/integration_tests/src/main/python/asserts.py:36: in _assert_equal [2024-09-26T15:46:13.629Z] _assert_equal(cpu[field], gpu[field], float_check, path + [field]) [2024-09-26T15:46:13.629Z] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ [2024-09-26T15:46:13.629Z] [2024-09-26T15:46:13.629Z] cpu = 1, gpu = None [2024-09-26T15:46:13.629Z] float_check = . at 0x7f7436a5c310> [2024-09-26T15:46:13.629Z] path = [0, 'from_json(json)', 'data'] [2024-09-26T15:46:13.629Z] [2024-09-26T15:46:13.629Z] def _assert_equal(cpu, gpu, float_check, path): [2024-09-26T15:46:13.629Z] t = type(cpu) [2024-09-26T15:46:13.629Z] if (t is Row): [2024-09-26T15:46:13.629Z] assert len(cpu) == len(gpu), "CPU and GPU row have different lengths at {} CPU: {} GPU: {}".format(path, len(cpu), len(gpu)) [2024-09-26T15:46:13.629Z] if hasattr(cpu, "__fields__") and hasattr(gpu, "__fields__"): [2024-09-26T15:46:13.629Z] assert cpu.__fields__ == gpu.__fields__, "CPU and GPU row have different fields at {} CPU: {} GPU: {}".format(path, cpu.__fields__, gpu.__fields__) [2024-09-26T15:46:13.629Z] for field in cpu.__fields__: [2024-09-26T15:46:13.629Z] _assert_equal(cpu[field], gpu[field], float_check, path + [field]) [2024-09-26T15:46:13.629Z] else: [2024-09-26T15:46:13.629Z] for index in range(len(cpu)): [2024-09-26T15:46:13.629Z] _assert_equal(cpu[index], gpu[index], float_check, path + [index]) [2024-09-26T15:46:13.629Z] elif (t is list): [2024-09-26T15:46:13.629Z] assert len(cpu) == len(gpu), "CPU and GPU list have different lengths at {} CPU: {} GPU: {}".format(path, len(cpu), len(gpu)) [2024-09-26T15:46:13.629Z] for index in range(len(cpu)): [2024-09-26T15:46:13.629Z] _assert_equal(cpu[index], gpu[index], float_check, path + [index]) [2024-09-26T15:46:13.629Z] elif (t is tuple): [2024-09-26T15:46:13.629Z] assert len(cpu) == len(gpu), "CPU and GPU list have different lengths at {} CPU: {} GPU: {}".format(path, len(cpu), len(gpu)) [2024-09-26T15:46:13.629Z] for index in range(len(cpu)): [2024-09-26T15:46:13.629Z] _assert_equal(cpu[index], gpu[index], float_check, path + [index]) [2024-09-26T15:46:13.629Z] elif (t is pytypes.GeneratorType): [2024-09-26T15:46:13.629Z] index = 0 [2024-09-26T15:46:13.629Z] # generator has no zip :( so we have to do this the hard way [2024-09-26T15:46:13.629Z] done = False [2024-09-26T15:46:13.629Z] while not done: [2024-09-26T15:46:13.629Z] sub_cpu = None [2024-09-26T15:46:13.629Z] sub_gpu = None [2024-09-26T15:46:13.629Z] try: [2024-09-26T15:46:13.629Z] sub_cpu = next(cpu) [2024-09-26T15:46:13.629Z] except StopIteration: [2024-09-26T15:46:13.629Z] done = True [2024-09-26T15:46:13.629Z] [2024-09-26T15:46:13.629Z] try: [2024-09-26T15:46:13.629Z] sub_gpu = next(gpu) [2024-09-26T15:46:13.629Z] except StopIteration: [2024-09-26T15:46:13.629Z] done = True [2024-09-26T15:46:13.629Z] [2024-09-26T15:46:13.629Z] if done: [2024-09-26T15:46:13.629Z] assert sub_cpu == sub_gpu and sub_cpu == None, "CPU and GPU generators have different lengths at {}".format(path) [2024-09-26T15:46:13.629Z] else: [2024-09-26T15:46:13.629Z] _assert_equal(sub_cpu, sub_gpu, float_check, path + [index]) [2024-09-26T15:46:13.629Z] [2024-09-26T15:46:13.629Z] index = index + 1 [2024-09-26T15:46:13.629Z] elif (t is dict): [2024-09-26T15:46:13.629Z] # The order of key/values is not guaranteed in python dicts, nor are they guaranteed by Spark [2024-09-26T15:46:13.629Z] # so sort the items to do our best with ignoring the order of dicts [2024-09-26T15:46:13.629Z] cpu_items = list(cpu.items()).sort(key=_RowCmp) [2024-09-26T15:46:13.630Z] gpu_items = list(gpu.items()).sort(key=_RowCmp) [2024-09-26T15:46:13.630Z] _assert_equal(cpu_items, gpu_items, float_check, path + ["map"]) [2024-09-26T15:46:13.630Z] elif (t is int): [2024-09-26T15:46:13.630Z] > assert cpu == gpu, "GPU and CPU int values are different at {}".format(path) [2024-09-26T15:46:13.630Z] E AssertionError: GPU and CPU int values are different at [0, 'from_json(json)', 'data'] [2024-09-26T15:46:13.630Z] [2024-09-26T15:46:13.630Z] rapids-it-dataproc-serverless-33/integration_tests/src/main/python/asserts.py:78: AssertionError [2024-09-26T15:46:13.630Z] ----------------------------- Captured stdout call ----------------------------- [2024-09-26T15:46:13.630Z] ### CPU RUN ### [2024-09-26T15:46:13.630Z] ### GPU RUN ### [2024-09-26T15:46:13.630Z] ### COLLECT: GPU TOOK 0.17503833770751953 CPU TOOK 0.1917879581451416 ### [2024-09-26T15:46:13.630Z] --- CPU OUTPUT [2024-09-26T15:46:13.630Z] +++ GPU OUTPUT [2024-09-26T15:46:13.630Z] @@ -1,13 +1,13 @@ [2024-09-26T15:46:13.630Z] -Row(json='{"data": 1}', from_json(json)=Row(data=1)) [2024-09-26T15:46:13.630Z] -Row(json='{"data": -1}', from_json(json)=Row(data=-1)) [2024-09-26T15:46:13.630Z] -Row(json='{"data": -0}', from_json(json)=Row(data=0)) [2024-09-26T15:46:13.630Z] -Row(json='{"data": 0}', from_json(json)=Row(data=0)) [2024-09-26T15:46:13.630Z] -Row(json='{"data": 127}', from_json(json)=Row(data=127)) [2024-09-26T15:46:13.630Z] -Row(json='{"data": -128}', from_json(json)=Row(data=-128)) [2024-09-26T15:46:13.630Z] -Row(json='{"data": 32767}', from_json(json)=Row(data=32767)) [2024-09-26T15:46:13.630Z] -Row(json='{"data": -32768}', from_json(json)=Row(data=-32768)) [2024-09-26T15:46:13.630Z] -Row(json='{"data": 2147483647}', from_json(json)=Row(data=2147483647)) [2024-09-26T15:46:13.630Z] -Row(json='{"data": -2147483648}', from_json(json)=Row(data=-2147483648)) [2024-09-26T15:46:13.630Z] +Row(json='{"data": 1}', from_json(json)=Row(data=None)) [2024-09-26T15:46:13.630Z] +Row(json='{"data": -1}', from_json(json)=Row(data=None)) [2024-09-26T15:46:13.630Z] +Row(json='{"data": -0}', from_json(json)=Row(data=None)) [2024-09-26T15:46:13.630Z] +Row(json='{"data": 0}', from_json(json)=Row(data=None)) [2024-09-26T15:46:13.630Z] +Row(json='{"data": 127}', from_json(json)=Row(data=None)) [2024-09-26T15:46:13.630Z] +Row(json='{"data": -128}', from_json(json)=Row(data=None)) [2024-09-26T15:46:13.630Z] +Row(json='{"data": 32767}', from_json(json)=Row(data=None)) [2024-09-26T15:46:13.630Z] +Row(json='{"data": -32768}', from_json(json)=Row(data=None)) [2024-09-26T15:46:13.630Z] +Row(json='{"data": 2147483647}', from_json(json)=Row(data=None)) [2024-09-26T15:46:13.630Z] +Row(json='{"data": -2147483648}', from_json(json)=Row(data=None)) [2024-09-26T15:46:13.630Z] Row(json='{"data": 9223372036854775807}', from_json(json)=Row(data=None)) [2024-09-26T15:46:13.630Z] Row(json='{"data": -9223372036854775808}', from_json(json)=Row(data=None)) [2024-09-26T15:46:13.630Z] Row(json='{"data": 9223372036854775808}', from_json(json)=Row(data=None)) [2024-09-26T15:46:13.630Z] =============================== warnings summary =============================== [2024-09-26T15:46:13.630Z] rapids-it-dataproc-serverless-33/integration_tests/src/main/python/marks.py:19 [2024-09-26T15:46:13.630Z] /var/dataproc/tmp/srvls-batch-30d7d73a-9542-42ed-908e-f6dc4adf41f9/rapids-it-dataproc-serverless-33/integration_tests/src/main/python/marks.py:19: PytestUnknownMarkWarning: Unknown pytest.mark.disable_ansi_mode - is this a typo? You can register custom marks to avoid this warning - for details, see https://docs.pytest.org/en/stable/how-to/mark.html [2024-09-26T15:46:13.630Z] disable_ansi_mode = pytest.mark.disable_ansi_mode [2024-09-26T15:46:13.630Z] [2024-09-26T15:46:13.630Z] rapids-it-dataproc-serverless-33/integration_tests/src/main/python/cast_test.py:151 [2024-09-26T15:46:13.630Z] /var/dataproc/tmp/srvls-batch-30d7d73a-9542-42ed-908e-f6dc4adf41f9/rapids-it-dataproc-serverless-33/integration_tests/src/main/python/cast_test.py:151: DeprecationWarning: invalid escape sequence \. [2024-09-26T15:46:13.630Z] StringGen(date_start_1_1_1 + '[ |T][0-3][0-9]:[0-6][0-9]:[0-6][0-9]\.[0-9]{0,6}Z?') [2024-09-26T15:46:13.630Z] [2024-09-26T15:46:13.630Z] rapids-it-dataproc-serverless-33/integration_tests/src/main/python/string_test.py:797 [2024-09-26T15:46:13.630Z] /var/dataproc/tmp/srvls-batch-30d7d73a-9542-42ed-908e-f6dc4adf41f9/rapids-it-dataproc-serverless-33/integration_tests/src/main/python/string_test.py:797: DeprecationWarning: invalid escape sequence \/ [2024-09-26T15:46:13.630Z] gen = StringGen('\/lit\/') [2024-09-26T15:46:13.630Z] [2024-09-26T15:46:13.630Z] rapids-it-dataproc-serverless-33/integration_tests/src/main/python/json_tuple_test.py:75 [2024-09-26T15:46:13.630Z] /var/dataproc/tmp/srvls-batch-30d7d73a-9542-42ed-908e-f6dc4adf41f9/rapids-it-dataproc-serverless-33/integration_tests/src/main/python/json_tuple_test.py:75: DeprecationWarning: invalid escape sequence \/ [2024-09-26T15:46:13.630Z] data = [['{"url":"https:\/\/www.nvidia.com\/1\/pic\/-1234.jpg","item":[],"info":{"id":12345}}'], [2024-09-26T15:46:13.630Z] [2024-09-26T15:46:13.630Z] src/main/python/cast_test.py: 76 warnings [2024-09-26T15:46:13.630Z] src/main/python/cmp_test.py: 16 warnings [2024-09-26T15:46:13.630Z] src/main/python/csv_test.py: 694 warnings [2024-09-26T15:46:13.630Z] src/main/python/datasourcev2_read_test.py: 1 warning [2024-09-26T15:46:13.630Z] src/main/python/datasourcev2_write_test.py: 2 warnings [2024-09-26T15:46:13.630Z] src/main/python/date_time_test.py: 131 warnings [2024-09-26T15:46:13.630Z] src/main/python/hash_aggregate_test.py: 375 warnings [2024-09-26T15:46:13.630Z] src/main/python/hive_delimited_text_test.py: 102 warnings [2024-09-26T15:46:13.630Z] src/main/python/json_matrix_test.py: 40 warnings [2024-09-26T15:46:13.630Z] src/main/python/orc_cast_test.py: 9 warnings [2024-09-26T15:46:13.630Z] src/main/python/orc_write_test.py: 62 warnings [2024-09-26T15:46:13.630Z] src/main/python/repart_test.py: 275 warnings [2024-09-26T15:46:13.630Z] src/main/python/schema_evolution_test.py: 2 warnings [2024-09-26T15:46:13.630Z] src/main/python/sort_test.py: 155 warnings [2024-09-26T15:46:13.630Z] /var/dataproc/tmp/srvls-batch-30d7d73a-9542-42ed-908e-f6dc4adf41f9/rapids-it-dataproc-serverless-33/integration_tests/src/main/python/conftest.py:246: UserWarning: allow_non_gpu marker without anything allowed [2024-09-26T15:46:13.630Z] warnings.warn('allow_non_gpu marker without anything allowed') [2024-09-26T15:46:13.630Z] [2024-09-26T15:46:13.630Z] src/main/python/hash_aggregate_test.py: 45 warnings [2024-09-26T15:46:13.630Z] /usr/lib/spark/python/lib/pyspark.zip/pyspark/sql/functions.py:988: FutureWarning: Deprecated in 3.2, use sum_distinct instead. [2024-09-26T15:46:13.630Z] [2024-09-26T15:46:13.630Z] src/main/python/hive_delimited_text_test.py: 56 warnings [2024-09-26T15:46:13.630Z] /usr/lib/spark/python/lib/pyspark.zip/pyspark/sql/catalog.py:781: FutureWarning: createExternalTable is deprecated since Spark 2.2, please use createTable instead. [2024-09-26T15:46:13.630Z] [2024-09-26T15:46:13.630Z] -- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.htmlThe text was updated successfully, but these errors were encountered: