NIFI-13865: Enable the JVM to record a heap dump when an OOM error occurs

[mattyb149]

Summary

NIFI-13865. This PR adds the following arguments to bootstrap.conf for the NiFi JVM to enable taking a heap dump when an OutOfMemoryError occurs. I'm not sure about the feasibility of unit/integration testing

Tracking

Please complete the following tracking steps prior to pull request creation.

Issue Tracking

• Apache NiFi Jira issue created

Pull Request Tracking

• Pull Request title starts with Apache NiFi Jira issue number, such as NIFI-00000
• Pull Request commit message starts with Apache NiFi Jira issue number, as such NIFI-00000

Pull Request Formatting

• Pull Request based on current revision of the main branch
• Pull Request refers to a feature branch with one commit containing changes

Verification

Please indicate the verification steps performed prior to pull request creation.

Build

• Build completed using mvn clean install -P contrib-check
  • JDK 21

Licensing

• New dependencies are compatible with the Apache License 2.0 according to the License Policy
• New dependencies are documented in applicable LICENSE and NOTICE files

Documentation

• Documentation formatting appears as expected in rendered files

[joewitt]

This needs testing in real world deployments before it becomes an always on default. We've used this many times in the past but there was always some reason to avoid it being there as a default.

[exceptionfactory]

Thanks for proposing this change @mattyb149.

Although enabling heap dumps can be very useful for troubleshooting, it can also present security concerns due to the memory contents containing sensitive values. For this reason, it is not something that we should enable as part of the default configuration.

[mattyb149]

Makes sense, I'll add them commented out with some doc

[exceptionfactory]

Instead of adding this to the configuration file, it would be better to note this as an option in the Administrator's Guide. The security implications are worth highlighting, and although it requires going to the guide, this is also something that should not necessarily be used regularly.

[joewitt]

good idea @exceptionfactory. In general getting our out of the box files simpler/less options and then documenting how to do things would yield a better user experience.

[mattyb149]

I can add it to the Admin Guide too, but for folks supporting other NiFi users, it just seems easier to ask the user(s) to uncomment a couple of existing lines vs having them search in the Guide for something that they might not know they're looking for, mis-copying the text from the Guide, etc.

[exceptionfactory]

I can add it to the Admin Guide too, but for folks supporting other NiFi users, it just seems easier to ask the user(s) to uncomment a couple of existing lines vs having them search in the Guide for something that they might not know they're looking for, mis-copying the text from the Guide, etc.

I agree this is a fine line, but over the years, the default configuration files have grown to include lot of commented information and many defaults that rarely change. In this case, the security implications and potential disk utilization concerns surrounding heap dumps make me think this should not be included in the default bootstrap.conf, even as a commented option.

[exceptionfactory]

It is also worth noting that the while the default configuration files should be minimal, nothing prevents others from providing alternative defaults in customized packages. The goal of the default files, however, should be to provide sensible and secure defaults without overwhelming users.

[mattyb149]

I was using the debug and javaAgent entries as an example.

[exceptionfactory]

I agree the debug and javaAgent examples are notable. There are other common production-level options we could consider, such as maximum directory memory usage, and other memory settings, but these are more specific to given environments, and harder to generalize. That's where having a section of the Admin Guide, with copyable examples, seems like a better way to go more for more JVM options.

[mattyb149]

Ok I will change this PR/Jira to add this to the Admin Guide instead