Logical type : TIME #143

saraswatpuneet · 2024-09-09T23:16:15Z

Problem

Part of #99

Support logical types in parquetjs starting with TIME support

Solution

Implementation following the parquet spec

Change summary:

Added a logical type to support parquet types
Implemented type conversions for logical type time to primitive type according to spec
Implement field, schema and file tests to ensure everything works together

Steps to Verify:

npm run test

…schema

lib/codec/types.ts

saraswatpuneet · 2024-09-11T18:29:04Z

lib/types.ts

+  value: string | bigint | number;
+  unit: 'MILLIS' | 'MICROS' | 'NANOS';
+  isAdjustedToUTC: boolean;
+}


Type for handling incoming time type and handle accordingly

lib/types.ts

test/jsonSchema.test.ts

test/decodeSchema.js

lib/jsonSchema.ts

test/decodeSchema.js

test/jsonSchema.test.ts

test/fields.test.ts

lib/jsonSchema.ts

saraswatpuneet · 2024-09-12T15:22:07Z

lib/jsonSchema.ts

+        if (!fieldValue.items.properties.unit.enum) {
+          throw new UnsupportedJsonSchemaError('Unit enum is not defined');
+        }
+        const unit = fieldValue.items.properties.unit.default || fieldValue.items.properties.unit.enum[0];


A question here, how will units be defined in schema, an enum? or plain string. if enum do we initialize all unit types as separate fields?

saraswatpuneet · 2024-09-12T16:39:35Z

test/test-files/time.schema.json

+        "unit": {
+          "type": "string",
+          "enum": ["MILLIS", "MICROS", "NANOS"],
+          "description": "The unit for the time value"


How we expect units be defined a string or enum?

I am fine with the enum or string, so what it is now is good by me.

shannonwells

Thanks for adding the tests.

saraswatpuneet added 21 commits September 5, 2024 12:54

minimal changes, passing test

fb4b59a

handle possible exceptions on conversion

1ca98b8

set the primitive type for TIME as INT64

8fcafe2

define field definition for time support and helper function

9949fd4

declare additional fields for TIME

679c96c

update jsonSchema to support TIME

d3a1382

add field tests for time

e22fbe4

annotate as int32 for MILLIS and int64 for rest

ddfcc0e

spec says int32 for milli, bigint for micro and nano

05e2414

redo based on generated types

38a1d57

add TIME related primitive converters

dddb09a

support time field for json schema

9401f67

finalize implementation side of TIME logical type

13007a5

set converted types for backward compatibility

ae58c96

set example field tests

42d72ad

set an example time schema

510f402

set schema test

bc9e11c

update schema files for testing

44f64aa

schema test result file

97b1475

adjust types based on ongoing file tests

6485753

logical types from parquet file cannot be reconstructed back without …

6927f31

…schema

saraswatpuneet marked this pull request as ready for review September 11, 2024 18:16

saraswatpuneet changed the title ~~Logical type : TIME~~ [WIP] Logical type : TIME Sep 11, 2024