schema_mode 'merge' | removing all other partitions

[dataMuley]

I need to change the schema and writing a new file using schema_mode = 'merge', the new schema is captured on the desired partition but the content in other partitions is lost. Reviewing the log file, all other partitions seem to be removed indicated by the 'remove' command. Has anyone else encountered the same or have a solution?

[sherlockbeard]

tried this its working fine.

import deltalake
import pyarrow as pa


deltalake.write_deltalake(
    './temp2',
    pa.Table.from_pydict(
        {
            "x": pa.array([x for x in range(5)]),
            "y": pa.array([x for x in range(5)]),
            "z": pa.array([x for x in range(5)]),
        }
    ),
    mode='append',
    engine="rust",
    partition_by=["x"]
) 

deltalake.write_deltalake(
    './temp2',
    pa.Table.from_pydict(
        {
            "x": pa.array([x for x in range(3, 8)]),
            "t": pa.array([x for x in range(3, 8)]),
            "k": pa.array([x for x in range(3, 8)]),
        }
    ),
    schema_mode='merge',
    mode='append',
    engine="rust",
    partition_by=["x"]
) 

Can you share a reproducible example ?

[dataMuley]

Thanks for the response -
In the above response, the 'second' write operation is an 'append' and a review of the inserted modified table will show both the first and second write operations. However, if you add a third write, and change to mode='overwrite', it seems that all partitions seems to be overwritten, not just that partitions then contained in the 'third' 'table.'

The 'schema' is indeed modified but prior partitions are overwritten. Below is the log detail from the 'third' operation. It would seem that x=2 would/still be retained but is removed. I don't recall this behavior when using engine='pyarrow'

Thanks again for your assistance

{"metaData":{"id":"122bcf4e-ff0f-49c3-9f87-edd5680937eb","name":null,"description":null,"format":{"provider":"parquet","options":{}},"schemaString":"{\"type\":\"struct\",\"fields\":[{\"name\":\"x\",\"type\":\"long\",\"nullable\":true,\"metadata\":{}},{\"name\":\"t\",\"type\":\"long\",\"nullable\":true,\"metadata\":{}},{\"name\":\"k\",\"type\":\"long\",\"nullable\":true,\"metadata\":{}}]}","partitionColumns":["x"],"createdTime":null,"configuration":{}}} {"add":{"path":"x=5/part-00001-ec98ed20-3ca6-46dd-bf46-50a467a67ae6-c000.snappy.parquet","partitionValues":{"x":"5"},"size":905,"modificationTime":1719851810639,"dataChange":true,"stats":"{\"numRecords\":1,\"minValues\":{\"k\":5,\"t\":5},\"maxValues\":{\"k\":5,\"t\":5},\"nullCount\":{\"k\":0,\"t\":0}}","tags":null,"deletionVector":null,"baseRowId":null,"defaultRowCommitVersion":null,"clusteringProvider":null}} {"add":{"path":"x=4/part-00001-e1700915-93ce-474a-84a2-d0a7f1aaeacd-c000.snappy.parquet","partitionValues":{"x":"4"},"size":905,"modificationTime":1719851810639,"dataChange":true,"stats":"{\"numRecords\":1,\"minValues\":{\"k\":4,\"t\":4},\"maxValues\":{\"t\":4,\"k\":4},\"nullCount\":{\"t\":0,\"k\":0}}","tags":null,"deletionVector":null,"baseRowId":null,"defaultRowCommitVersion":null,"clusteringProvider":null}} {"add":{"path":"x=7/part-00001-cedc5d38-4cd1-4098-9c88-ed18bfc2f199-c000.snappy.parquet","partitionValues":{"x":"7"},"size":905,"modificationTime":1719851810639,"dataChange":true,"stats":"{\"numRecords\":1,\"minValues\":{\"t\":7,\"k\":7},\"maxValues\":{\"t\":7,\"k\":7},\"nullCount\":{\"t\":0,\"k\":0}}","tags":null,"deletionVector":null,"baseRowId":null,"defaultRowCommitVersion":null,"clusteringProvider":null}} {"add":{"path":"x=3/part-00001-5c058cd8-a06e-4c31-9d42-bbef2a98a790-c000.snappy.parquet","partitionValues":{"x":"3"},"size":905,"modificationTime":1719851810639,"dataChange":true,"stats":"{\"numRecords\":1,\"minValues\":{\"t\":3,\"k\":3},\"maxValues\":{\"t\":3,\"k\":3},\"nullCount\":{\"t\":0,\"k\":0}}","tags":null,"deletionVector":null,"baseRowId":null,"defaultRowCommitVersion":null,"clusteringProvider":null}} {"add":{"path":"x=6/part-00001-a227072a-01bc-490d-8bf0-26c8d2aef792-c000.snappy.parquet","partitionValues":{"x":"6"},"size":905,"modificationTime":1719851810639,"dataChange":true,"stats":"{\"numRecords\":1,\"minValues\":{\"t\":6,\"k\":6},\"maxValues\":{\"k\":6,\"t\":6},\"nullCount\":{\"t\":0,\"k\":0}}","tags":null,"deletionVector":null,"baseRowId":null,"defaultRowCommitVersion":null,"clusteringProvider":null}} {"add":{"path":"x=8/part-00001-4096cdd4-e97c-46c2-914a-6e42997a74c5-c000.snappy.parquet","partitionValues":{"x":"8"},"size":905,"modificationTime":1719851810639,"dataChange":true,"stats":"{\"numRecords\":1,\"minValues\":{\"k\":8,\"t\":8},\"maxValues\":{\"t\":8,\"k\":8},\"nullCount\":{\"k\":0,\"t\":0}}","tags":null,"deletionVector":null,"baseRowId":null,"defaultRowCommitVersion":null,"clusteringProvider":null}} {"metaData":{"id":"39c05fe4-56c4-4a43-af1a-d41b95a59a4d","name":null,"description":null,"format":{"provider":"parquet","options":{}},"schemaString":"{\"type\":\"struct\",\"fields\":[{\"name\":\"x\",\"type\":\"long\",\"nullable\":true,\"metadata\":{}},{\"name\":\"t\",\"type\":\"long\",\"nullable\":true,\"metadata\":{}},{\"name\":\"k\",\"type\":\"long\",\"nullable\":true,\"metadata\":{}}]}","partitionColumns":["x"],"createdTime":null,"configuration":{}}} {"remove":{"path":"x=3/part-00001-7b6763ec-f83f-4454-b720-0a8a47e7d9d8-c000.snappy.parquet","dataChange":true,"deletionTimestamp":1719851810641,"extendedFileMetadata":true,"partitionValues":{"x":"3"},"size":1338}} {"remove":{"path":"x=4/part-00001-6dd6b296-f4a6-449b-98e0-ab9f0bc187c0-c000.snappy.parquet","dataChange":true,"deletionTimestamp":1719851810641,"extendedFileMetadata":true,"partitionValues":{"x":"4"},"size":1338}} {"remove":{"path":"x=5/part-00001-afd0f1fa-54bc-412b-a8b9-03e84619fb7a-c000.snappy.parquet","dataChange":true,"deletionTimestamp":1719851810641,"extendedFileMetadata":true,"partitionValues":{"x":"5"},"size":1338}} {"remove":{"path":"x=7/part-00001-8a3995d6-e825-4e74-94b0-89a85e66ed90-c000.snappy.parquet","dataChange":true,"deletionTimestamp":1719851810641,"extendedFileMetadata":true,"partitionValues":{"x":"7"},"size":1338}} {"remove":{"path":"x=8/part-00001-c52c0aa8-ffaf-4a51-bff5-a43c6e1aed8c-c000.snappy.parquet","dataChange":true,"deletionTimestamp":1719851810641,"extendedFileMetadata":true,"partitionValues":{"x":"8"},"size":1338}} {"remove":{"path":"x=6/part-00001-05590b65-bf15-4361-866b-10c4897d7905-c000.snappy.parquet","dataChange":true,"deletionTimestamp":1719851810641,"extendedFileMetadata":true,"partitionValues":{"x":"6"},"size":1338}} {"remove":{"path":"x=2/part-00001-fddfdcbb-e63b-4605-8b7b-536a09f8805d-c000.snappy.parquet","dataChange":true,"deletionTimestamp":1719851810641,"extendedFileMetadata":true,"partitionValues":{"x":"2"},"size":905}} {"remove":{"path":"x=4/part-00001-c64dee50-689c-4002-b14f-f1f38ab17a45-c000.snappy.parquet","dataChange":true,"deletionTimestamp":1719851810641,"extendedFileMetadata":true,"partitionValues":{"x":"4"},"size":905}} {"remove":{"path":"x=0/part-00001-f9fa008c-82a5-4ea5-accc-106fe4d48044-c000.snappy.parquet","dataChange":true,"deletionTimestamp":1719851810641,"extendedFileMetadata":true,"partitionValues":{"x":"0"},"size":905}} {"remove":{"path":"x=1/part-00001-8e871caf-f995-43ae-bd5d-1fac4c2ee5cf-c000.snappy.parquet","dataChange":true,"deletionTimestamp":1719851810641,"extendedFileMetadata":true,"partitionValues":{"x":"1"},"size":905}} {"remove":{"path":"x=3/part-00001-5c9259fb-198d-411c-a09e-3ea2e804a9a8-c000.snappy.parquet","dataChange":true,"deletionTimestamp":1719851810641,"extendedFileMetadata":true,"partitionValues":{"x":"3"},"size":905}} {"commitInfo":{"timestamp":1719851810641,"operation":"WRITE","operationParameters":{"partitionBy":"[\"x\"]","mode":"Overwrite"},"clientVersion":"delta-rs.0.18.0"}}

[sherlockbeard]

https://delta-io.github.io/delta-rs/usage/writing/#overwriting-part-of-the-table-data-using-a-predicate

import deltalake
import pyarrow as pa


deltalake.write_deltalake(
    './temp2',
    pa.Table.from_pydict(
        {
            "x": pa.array([x for x in range(5)]),
            "y": pa.array([x for x in range(5)]),
            "z": pa.array([x for x in range(5)]),
        }
    ),
    mode='append',
    engine="rust",
    partition_by=["x"]
) 

deltalake.write_deltalake(
    './temp2',
    pa.Table.from_pydict(
        {
            "x": pa.array([x for x in range(3, 8)]),
            "t": pa.array([x for x in range(3, 8)]),
            "k": pa.array([x for x in range(3, 8)]),
        }
    ),
    schema_mode='merge',
    mode='append',
    engine="rust",
    partition_by=["x"]
) 

deltalake.write_deltalake(
    './temp2',
    pa.Table.from_pydict(
        {
            "x": pa.array([x for x in range(7, 9)]),
            "t": pa.array([x for x in range(7, 9)]),
            "k": pa.array([x for x in range(7, 9)]),
            "y": pa.array([x for x in range(7, 9)]),
            "z": pa.array([x for x in range(7, 9)])
        }
    ),
    predicate="x > 6 and x < 9",
    mode='overwrite',
    engine="rust",
    partition_by=["x"]
) 

[dataMuley]

I completely overlooked the predicate option. thanks for the assistance