Feat (Core): Migrate import contentlets action to job processor #30432
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Implemented RetryPolicyProcessor to manage retry strategies in job processing. Integrated ExponentialBackoffRetryPolicy annotation for configuring retry behaviors and updated related test cases and classes to accommodate this change.
Moved the logic for retrieving temporary files from LargeFileReader to the new JobUtil class for better separation of concerns. Introduced JobValidationException to encapsulate validation failures during job processing.
…mportContentletsAction-to-job-processor
Added a LongConsumer progress callback to provide real-time progress updates during CSV import. Finalized ImportContentletProcessor implementation to support content imports, progress tracking, cancellation, and proper error handling.
…mportContentletsAction-to-job-processor
…mportContentletsAction-to-job-processor
This change updates the class name to reflect its purpose more accurately. The queue and log references have also been updated to ensure consistency throughout the codebase.
Added @NoRetryPolicy to explicitly define no-retry behavior for job processors. Introduced @DefaultRetryStrategy for marking the default retry strategy implementation. Updated relevant classes to utilize these annotations for better code readability and maintainability.
Introduced a new abstract class `AbstractJobWatcher` and modified `RealTimeJobMonitor` to support job watcher filtering using predicates. Updated related test configurations to include the new classes for initialization. These changes enhance the monitoring functionality by allowing more precise control over job update notifications.
…mportContentletsAction-to-job-processor
Added a private constructor in Predicates to prevent instantiation and a default constructor in RetryPolicyProcessor for CDI proxy creation. This change improves the design by enforcing non-instantiability where necessary and ensuring the proper creation of CDI proxies.
…mportContentletsAction-to-job-processor
…mportContentletsAction-to-job-processor
Updated the `getFields` method to handle instances where `PARAMETER_FIELDS` might be of type `ArrayList`. This prevents potential `ClassCastException` at runtime by checking the type before casting.
Refactor and enhance job management functionality including type-safe parameters handling, retrieval of active, completed, canceled, and failed jobs using consolidated queries. Added new endpoint for job creation with parameters and updated related tests.
Renamed the FailJob class to FailSuccessJob and updated its process method to conditionally throw an exception based on job parameters. Refactored JobParams to simplify parameter parsing. Updated Postman tests to reflect these changes and added new tests for active jobs and job states.
…mportContentletsAction-to-job-processor
Added `ImportContentletsProcessorIntegrationTest` for end-to-end testing of content import functionality in both preview and publish modes. Refactored `generateMockRequest` method to `JobUtil` for reusability, removing the redundant `getRequest` method from `ImportContentletsProcessor`.
…mportContentletsAction-to-job-processor
…mportContentletsAction-to-job-processor # Conflicts: # dotCMS/src/main/java/com/dotcms/rest/api/v1/job/JobQueueResource.java # dotcms-integration/src/test/java/com/dotcms/TestBaseJunit5WeldInitiator.java # dotcms-integration/src/test/java/com/dotcms/util/IntegrationTestInitService.java
|
@jgambarios I was thinking we have a case in the job processing REST API, as there seems to be some disconnection in how we capture parameters and how the job is triggered. For example, let’s say a job requires three parameters, and when the REST captures those three parameters, one of them, say, is expected to be an integer, but it is received in the string. So, the job gets created, stored in the database, the processor starts working with it, and it fails because one of the expected parameters doesn’t have the correct format. This situation will keep happening because the processor will attempt to rerun the job created with incorrect parameters. this could be prevented if we had a validator in place to catch these errors before the job is created. That would be a significant improvement. |
Added `@EnableWeld` annotation and extended `ImportContentletsProcessorIntegrationTest` from `Junit5WeldBaseTest`. These changes integrate Weld with the JUnit5 testing framework, enabling dependency injection and enhancing the test's capabilities.
…mportContentletsAction-to-job-processor
fabrizzio-dotCMS
approved these changes
Oct 29, 2024
dotCMS/src/main/java/com/dotcms/jobs/business/processor/impl/ImportContentletsProcessor.java
Outdated
Show resolved
Hide resolved
dotCMS/src/main/java/com/dotcms/jobs/business/processor/impl/ImportContentletsProcessor.java
Outdated
Show resolved
Hide resolved
…mportContentletsAction-to-job-processor
|
nollymar
approved these changes
Oct 29, 2024
nollymar
deleted the
issue-29498-migrate-ImportContentletsAction-to-job-processor
branch
dsolistorres
pushed a commit
that referenced
this pull request
Nov 5, 2024
This pull request introduces a new processor `ImportContentletsProcessor`, which replicates the functionality we have in the `ImportContentletsAction` but using the new jobs infrastructureseveral, also enhancements and refactoring to the `JobQueueManagerAPI` and related classes. The key changes include adding new methods for retrieving job lists, integrating a retry policy processor, and improving the real-time job monitoring system. ### Enhancements to Job Retrieval: * Added new methods to `JobQueueManagerAPI` for retrieving lists of active, completed, canceled, and failed jobs with pagination support. (`dotCMS/src/main/java/com/dotcms/jobs/business/api/JobQueueManagerAPI.java`) [[1]](diffhunk://#diff-97639376a50922f533c812eb8848f70e1913df8f14f0fb5bb582243f5660e465R90-R101) [[2]](diffhunk://#diff-97639376a50922f533c812eb8848f70e1913df8f14f0fb5bb582243f5660e465L100-R148) * Implemented these methods in `JobQueueManagerAPIImpl` to interact with the job queue and handle exceptions. (`dotCMS/src/main/java/com/dotcms/jobs/business/api/JobQueueManagerAPIImpl.java`) [[1]](diffhunk://#diff-c092f8af2f800c0ca84f2c2f4aed0af60d0ed488e20fa32992967e154064f561R303-R313) [[2]](diffhunk://#diff-c092f8af2f800c0ca84f2c2f4aed0af60d0ed488e20fa32992967e154064f561L304-R364) ### Retry Policy Integration: * Introduced `RetryPolicyProcessor` to process retry policies for job processors and integrated it into `JobQueueManagerAPIImpl`. (`dotCMS/src/main/java/com/dotcms/jobs/business/api/JobQueueManagerAPIImpl.java`) [[1]](diffhunk://#diff-c092f8af2f800c0ca84f2c2f4aed0af60d0ed488e20fa32992967e154064f561R111) [[2]](diffhunk://#diff-c092f8af2f800c0ca84f2c2f4aed0af60d0ed488e20fa32992967e154064f561R251-R256) * Modified the constructor of `JobQueueManagerAPIImpl` to include the `RetryPolicyProcessor` parameter. (`dotCMS/src/main/java/com/dotcms/jobs/business/api/JobQueueManagerAPIImpl.java`) [[1]](diffhunk://#diff-c092f8af2f800c0ca84f2c2f4aed0af60d0ed488e20fa32992967e154064f561L144-R151) [[2]](diffhunk://#diff-c092f8af2f800c0ca84f2c2f4aed0af60d0ed488e20fa32992967e154064f561R160-R161) [[3]](diffhunk://#diff-c092f8af2f800c0ca84f2c2f4aed0af60d0ed488e20fa32992967e154064f561L166) ### Real-Time Job Monitoring: * Enhanced `RealTimeJobMonitor` to support filtered watching through predicates, allowing clients to receive specific job updates. Added detailed documentation and examples for usage. (`dotCMS/src/main/java/com/dotcms/jobs/business/api/events/RealTimeJobMonitor.java`) [[1]](diffhunk://#diff-ff9af41ece416b81fb0f12a7a8a9a8e5fe54ccbb278f4121bb0fce04b149cc36R4-R129) [[2]](diffhunk://#diff-ff9af41ece416b81fb0f12a7a8a9a8e5fe54ccbb278f4121bb0fce04b149cc36L59-R155) * Created `AbstractJobWatcher` to encapsulate a watcher and its filter predicate, supporting the new monitoring functionality. (`dotCMS/src/main/java/com/dotcms/jobs/business/api/events/AbstractJobWatcher.java`) ### Code Refactoring: * Simplified method signatures and improved exception handling in `JobQueueManagerAPIImpl`. (`dotCMS/src/main/java/com/dotcms/jobs/business/api/JobQueueManagerAPIImpl.java`) [[1]](diffhunk://#diff-c092f8af2f800c0ca84f2c2f4aed0af60d0ed488e20fa32992967e154064f561L685-R736) [[2]](diffhunk://#diff-c092f8af2f800c0ca84f2c2f4aed0af60d0ed488e20fa32992967e154064f561R860-R882) * Added missing Javadoc comments and improved existing ones for better clarity and documentation. (`dotCMS/src/main/java/com/dotcms/jobs/business/api/JobQueueManagerAPI.java`) [[1]](diffhunk://#diff-97639376a50922f533c812eb8848f70e1913df8f14f0fb5bb582243f5660e465R64) [[2]](diffhunk://#diff-97639376a50922f533c812eb8848f70e1913df8f14f0fb5bb582243f5660e465R178) These changes collectively improve the robustness, maintainability, and functionality of the job management and monitoring systems in the `dotCMS` project.
spbolton
pushed a commit
that referenced
this pull request
Nov 11, 2024
This pull request introduces a new processor `ImportContentletsProcessor`, which replicates the functionality we have in the `ImportContentletsAction` but using the new jobs infrastructureseveral, also enhancements and refactoring to the `JobQueueManagerAPI` and related classes. The key changes include adding new methods for retrieving job lists, integrating a retry policy processor, and improving the real-time job monitoring system. ### Enhancements to Job Retrieval: * Added new methods to `JobQueueManagerAPI` for retrieving lists of active, completed, canceled, and failed jobs with pagination support. (`dotCMS/src/main/java/com/dotcms/jobs/business/api/JobQueueManagerAPI.java`) [[1]](diffhunk://#diff-97639376a50922f533c812eb8848f70e1913df8f14f0fb5bb582243f5660e465R90-R101) [[2]](diffhunk://#diff-97639376a50922f533c812eb8848f70e1913df8f14f0fb5bb582243f5660e465L100-R148) * Implemented these methods in `JobQueueManagerAPIImpl` to interact with the job queue and handle exceptions. (`dotCMS/src/main/java/com/dotcms/jobs/business/api/JobQueueManagerAPIImpl.java`) [[1]](diffhunk://#diff-c092f8af2f800c0ca84f2c2f4aed0af60d0ed488e20fa32992967e154064f561R303-R313) [[2]](diffhunk://#diff-c092f8af2f800c0ca84f2c2f4aed0af60d0ed488e20fa32992967e154064f561L304-R364) ### Retry Policy Integration: * Introduced `RetryPolicyProcessor` to process retry policies for job processors and integrated it into `JobQueueManagerAPIImpl`. (`dotCMS/src/main/java/com/dotcms/jobs/business/api/JobQueueManagerAPIImpl.java`) [[1]](diffhunk://#diff-c092f8af2f800c0ca84f2c2f4aed0af60d0ed488e20fa32992967e154064f561R111) [[2]](diffhunk://#diff-c092f8af2f800c0ca84f2c2f4aed0af60d0ed488e20fa32992967e154064f561R251-R256) * Modified the constructor of `JobQueueManagerAPIImpl` to include the `RetryPolicyProcessor` parameter. (`dotCMS/src/main/java/com/dotcms/jobs/business/api/JobQueueManagerAPIImpl.java`) [[1]](diffhunk://#diff-c092f8af2f800c0ca84f2c2f4aed0af60d0ed488e20fa32992967e154064f561L144-R151) [[2]](diffhunk://#diff-c092f8af2f800c0ca84f2c2f4aed0af60d0ed488e20fa32992967e154064f561R160-R161) [[3]](diffhunk://#diff-c092f8af2f800c0ca84f2c2f4aed0af60d0ed488e20fa32992967e154064f561L166) ### Real-Time Job Monitoring: * Enhanced `RealTimeJobMonitor` to support filtered watching through predicates, allowing clients to receive specific job updates. Added detailed documentation and examples for usage. (`dotCMS/src/main/java/com/dotcms/jobs/business/api/events/RealTimeJobMonitor.java`) [[1]](diffhunk://#diff-ff9af41ece416b81fb0f12a7a8a9a8e5fe54ccbb278f4121bb0fce04b149cc36R4-R129) [[2]](diffhunk://#diff-ff9af41ece416b81fb0f12a7a8a9a8e5fe54ccbb278f4121bb0fce04b149cc36L59-R155) * Created `AbstractJobWatcher` to encapsulate a watcher and its filter predicate, supporting the new monitoring functionality. (`dotCMS/src/main/java/com/dotcms/jobs/business/api/events/AbstractJobWatcher.java`) ### Code Refactoring: * Simplified method signatures and improved exception handling in `JobQueueManagerAPIImpl`. (`dotCMS/src/main/java/com/dotcms/jobs/business/api/JobQueueManagerAPIImpl.java`) [[1]](diffhunk://#diff-c092f8af2f800c0ca84f2c2f4aed0af60d0ed488e20fa32992967e154064f561L685-R736) [[2]](diffhunk://#diff-c092f8af2f800c0ca84f2c2f4aed0af60d0ed488e20fa32992967e154064f561R860-R882) * Added missing Javadoc comments and improved existing ones for better clarity and documentation. (`dotCMS/src/main/java/com/dotcms/jobs/business/api/JobQueueManagerAPI.java`) [[1]](diffhunk://#diff-97639376a50922f533c812eb8848f70e1913df8f14f0fb5bb582243f5660e465R64) [[2]](diffhunk://#diff-97639376a50922f533c812eb8848f70e1913df8f14f0fb5bb582243f5660e465R178) These changes collectively improve the robustness, maintainability, and functionality of the job management and monitoring systems in the `dotCMS` project.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This pull request introduces a new processor
ImportContentletsProcessor, which replicates the functionality we have in theImportContentletsActionbut using the new jobs infrastructureseveral, also enhancements and refactoring to theJobQueueManagerAPIand related classes. The key changes include adding new methods for retrieving job lists, integrating a retry policy processor, and improving the real-time job monitoring system.Enhancements to Job Retrieval:
JobQueueManagerAPIfor retrieving lists of active, completed, canceled, and failed jobs with pagination support. (dotCMS/src/main/java/com/dotcms/jobs/business/api/JobQueueManagerAPI.java) [1] [2]JobQueueManagerAPIImplto interact with the job queue and handle exceptions. (dotCMS/src/main/java/com/dotcms/jobs/business/api/JobQueueManagerAPIImpl.java) [1] [2]Retry Policy Integration:
RetryPolicyProcessorto process retry policies for job processors and integrated it intoJobQueueManagerAPIImpl. (dotCMS/src/main/java/com/dotcms/jobs/business/api/JobQueueManagerAPIImpl.java) [1] [2]JobQueueManagerAPIImplto include theRetryPolicyProcessorparameter. (dotCMS/src/main/java/com/dotcms/jobs/business/api/JobQueueManagerAPIImpl.java) [1] [2] [3]Real-Time Job Monitoring:
RealTimeJobMonitorto support filtered watching through predicates, allowing clients to receive specific job updates. Added detailed documentation and examples for usage. (dotCMS/src/main/java/com/dotcms/jobs/business/api/events/RealTimeJobMonitor.java) [1] [2]AbstractJobWatcherto encapsulate a watcher and its filter predicate, supporting the new monitoring functionality. (dotCMS/src/main/java/com/dotcms/jobs/business/api/events/AbstractJobWatcher.java)Code Refactoring:
JobQueueManagerAPIImpl. (dotCMS/src/main/java/com/dotcms/jobs/business/api/JobQueueManagerAPIImpl.java) [1] [2]dotCMS/src/main/java/com/dotcms/jobs/business/api/JobQueueManagerAPI.java) [1] [2]These changes collectively improve the robustness, maintainability, and functionality of the job management and monitoring systems in the
dotCMSproject.This PR fixes: #29498