This is intended to demonstrate the Kryo v2 to v5 upgrade pull-request proposal. There are two projects:
-
flink-kryo-legacy: This is a Flink application built for the current stable version, Flink 1.17.0, which uses the Kryo v2 serialization framework. This uses custom Kryo v2 serializers and Kryo serialized keyed state to test edge cases for applications based on Kryo v2 serialization features in current versions of Flink. When this application is stopped with savepoints generated, the savepoints will have Kryo v2 serialized data that can't be correctly deserialized with Kryo v5.
-
flink-kryo-upgraded: This is an upgraded version of the same application for a Flink prototype, Flink 1.18-SNAPSHOT, which keeps Kryo v2 for backward compatibility with reading existing savepoints and checkpoints but otherwise upgrades Kryo to v5 and writes out new state with Kryo v5. This keeps the same custom Kryo v2 serializer and adds a custom Kryo v5 serializer. Note that these custom serializers are written so attempting to deserialize one version with data serialized by the other version will fail fast with a clear exception.
Note that this test application is a test + demo on top of the unit test suite that is part of the Apache Flink code base, which has a variety of backward compatibility tests included.
This repository also includes two savepoints directories: one savepoint is written with the most recent stable Flink and includes data serialized with Kryo v2. The second savepoint is written with the Flink prototype which includes data serialized with Kryo v5.
- Run flink-kryo-legacy with the current stable version of Flink, save the log output as a baseline for a correct execution.
- Run flink-kryo-legacy, and do a stop-with-savepoint part way through. This generates a savepoint with Kryo v2 serialized state.
- Run flink-kryo-legacy, and restore from the savepoint from the stable version of Flink with Kryo v2 serialized state. Verify that the application output matches the expectation.
- Run flink-kryo-upgraded with the prototype version of Flink, restore from the savepoint generated by the stable version of Flink with Kryo v2 serialized state. Let this run to completion. This demonstrates backwards compatibility with state from stable versions of Flink.
- Run flink-kryo-upgraded like above, but do a stop-with-savepoint part way through execution. This will generate a savepoint with the prototype version of Flink with Kryo v5 serialied state.
- Run flink-kryo-upgraded, restore from the new savepoint, and verify that the logs produce the desired output.