FLINK-36540.[Runtime / Task] Add Support for Hadoop Caller Context when using Flink to operate hdfs.

[liangyu-1]

… yarn

What is the purpose of the change

As described in FLINK-36540.
When we use Flink to delete or write or modify files on Hadoop filesystem, callerContext is a helpful feature if we want to trace who did the operation or count how many files an application can create on hadoop filesystem. So I add a new Option so that when we enable callerContext in Flink, we can set a ThreadLocal value for each task so that we can trace it in hadoop's audit.log.

What's more, with this new feature and history json files in history server, we can calculate how many read operations and write operations a Flink application did to hdfs, and find out if there is a pressure or bottleneck to operate on hdfs files.

Brief change log

• Add a new HadoopOption
• Add a new static method setCallerContext in HadoopUtils.java
• modify startTaskThread in Task.java and add a new method getIsCallerContextEnabled in Task.java

Verifying this change

Please make sure both new and modified tests in this PR follow the conventions for tests defined in our code quality guide.

This change added tests and can be verified as follows:

(example:)

• Added new Unit Test in HadoopUtilsTest.java and TaskTest.java
• Tested on our YARN CLUSTER

I rebuild this project, and test the new jar file in my cluster, it prints out the correct caller context as expected

Does this pull request potentially affect one of the following parts:

• Dependencies (does it add or upgrade a dependency): (yes / no)
• The public API, i.e., is any changed class annotated with @Public(Evolving): (yes / no)
• The serializers: (yes / no / don't know)
• The runtime per-record code paths (performance sensitive): (yes / no / don't know)
• Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn, ZooKeeper: (yes / no / don't know)
• The S3 file system connector: (yes / no / don't know)

Documentation

• Does this pull request introduce a new feature? (yes / no)
• If yes, how is the feature documented? (not applicable / docs / JavaDocs / not documented)

[flinkbot]

CI report:

• e3075d7 Azure: SUCCESS

Bot commands

The @flinkbot bot supports the following commands:

• @flinkbot run azure re-run the last Azure build

[liangyu-1]

@flinkbot run azure re-run the last Azure build

[liangyu-1]

@xintongsong
Hi, would you please help me check this issue?

[liangyu-1]

@dmvk @fapaul
Hi, would you please help me check this issue?