Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Create a Job with the relevant logic from ImportUtil and ImportContentletsAction to successfully import large files #29498

Closed
Tracked by #29482
fabrizzio-dotCMS opened this issue Aug 7, 2024 · 9 comments · Fixed by #30432 or #30617
Closed
Tracked by #29482

Create a Job with the relevant logic from ImportUtil and ImportContentletsAction to successfully import large files #29498

fabrizzio-dotCMS opened this issue Aug 7, 2024 · 9 comments · Fixed by #30432 or #30617
Labels
NW Removed QA : Passed Internal Team : Scout Type : Task

Comments

@fabrizzio-dotCMS
Copy link
Contributor

fabrizzio-dotCMS commented Aug 7, 2024 •
edited
Loading

Parent Issue

#29482

Task

Create a Job following the implementation of the Epic
This Job has to use the new BufferedCvsReader to extract the content data from a file to avoid memory issues
It is desirable to be able to start or restart the import from a given row number skipping all previous
The new Job can be instructed to perform a db commit after n rows are saved.
The new method should return an Immutable ImportSumary class instead of a HashMap reporting the results.
Consume a single class with all the required parameters instead of taking a large number of arguments. Right now it takes 15 parameters. When the maximum number of allowed params should be 7. Any private methods created here have to meet these requirements too.
Optionally we can Refactor the method ImportUtil.importFile to reduce its complexity and make it clearer to read and understand. Currently, it is 115 lines and the recommended is 15

Proposed Objective

Core Features

Proposed Priority

Priority 2 - Important

Acceptance Criteria

  1. The Job should be able to import successfully a large number of content items. e.g. 10K items.
  2. The Job should never get stuck or run out of memory.
  3. The same options passed to the Struts Action (preview, import, fields, and key selections should be accepted by this job) Since the functionality should remain as it is now.

External Links... Slack Conversations, Support Tickets, Figma Designs, etc.

No response

Assumptions & Initiation Needs

No response

Quality Assurance Notes & Workarounds

No response

Sub-Tasks & Estimates

No response
@fabrizzio-dotCMS fabrizzio-dotCMS changed the title Create a Job with the relevant logic from ImportUtil and ImportContentletsAction to successfully imports large files Create a Job with the relevant logic from ImportUtil and ImportContentletsAction to successfully import large files Aug 7, 2024
@fabrizzio-dotCMS fabrizzio-dotCMS mentioned this issue Aug 7, 2024
Open
@nollymar nollymar moved this from New to In Progress in dotCMS - Product Planning Oct 14, 2024
@nollymar nollymar removed the Triage label Oct 22, 2024
jgambarios added a commit that referenced this issue Oct 22, 2024
@jgambarios
#29498 Add progress callback and finalize ImportContentletProcessor
17b28a8 
Added a LongConsumer progress callback to provide real-time progress updates during CSV import. Finalized ImportContentletProcessor implementation to support content imports, progress tracking, cancellation, and proper error handling.
@github-actions GitHub Actions
Copy link

github-actions bot commented Oct 23, 2024 •
edited
Loading

PRs:

@jgambarios jgambarios linked a pull request Oct 23, 2024 that will close this issue
Feat (Core): Migrate import contentlets action to job processor #30432
Merged
jgambarios added a commit that referenced this issue Oct 23, 2024
@jgambarios
#29498 Rename ImportContentletProcessor to ImportContentletsProcessor
e999e20 
This change updates the class name to reflect its purpose more accurately. The queue and log references have also been updated to ensure consistency throughout the codebase.
jgambarios added a commit that referenced this issue Oct 23, 2024
@jgambarios
#29498 Create NoRetryPolicy and DefaultRetryStrategy annotations
ff5a516 
Added @NoRetryPolicy to explicitly define no-retry behavior for job processors. Introduced @DefaultRetryStrategy for marking the default retry strategy implementation. Updated relevant classes to utilize these annotations for better code readability and maintainability.
jgambarios added a commit that referenced this issue Oct 23, 2024
@jgambarios
#29498 Add job watcher filtering and predicates in RealTimeJobMonitor
dc929f5 
Introduced a new abstract class `AbstractJobWatcher` and modified `RealTimeJobMonitor` to support job watcher filtering using predicates. Updated related test configurations to include the new classes for initialization. These changes enhance the monitoring functionality by allowing more precise control over job update notifications.
jgambarios added a commit that referenced this issue Oct 23, 2024
@jgambarios
#29498 Sonarlint feedback.
5518ef3
jgambarios added a commit that referenced this issue Oct 23, 2024
@jgambarios
#29498 Add private constructors and default constructor for CDI
b9019cd 
Added a private constructor in Predicates to prevent instantiation and a default constructor in RetryPolicyProcessor for CDI proxy creation. This change improves the design by enforcing non-instantiability where necessary and ensuring the proper creation of CDI proxies.
jgambarios added a commit that referenced this issue Oct 23, 2024
@jgambarios
#29498 Fixing tests
a757bb0
jgambarios added a commit that referenced this issue Oct 24, 2024
@jgambarios
#29498 Fixing tests
7aa91f9
jgambarios added a commit that referenced this issue Oct 24, 2024
@jgambarios
#29498 Fix type casting issue in getFields method.
542414d 
Updated the `getFields` method to handle instances where `PARAMETER_FIELDS` might be of type `ArrayList`. This prevents potential `ClassCastException` at runtime by checking the type before casting.
jgambarios added a commit that referenced this issue Oct 25, 2024
@jgambarios
#29498 Refactor job management API and implement endpoint
834d256 
Refactor and enhance job management functionality including type-safe parameters handling, retrieval of active, completed, canceled, and failed jobs using consolidated queries. Added new endpoint for job creation with parameters and updated related tests.
jgambarios added a commit that referenced this issue Oct 26, 2024
@jgambarios
#29498 Rename FailJob to FailSuccessJob and improve job handling
9e43009 
Renamed the FailJob class to FailSuccessJob and updated its process method to conditionally throw an exception based on job parameters. Refactored JobParams to simplify parameter parsing. Updated Postman tests to reflect these changes and added new tests for active jobs and job states.
jgambarios added a commit that referenced this issue Oct 29, 2024
@jgambarios
#29498 Add integration test and refactor request generation
054f97d 
Added `ImportContentletsProcessorIntegrationTest` for end-to-end testing of content import functionality in both preview and publish modes. Refactored `generateMockRequest` method to `JobUtil` for reusability, removing the redundant `getRequest` method from `ImportContentletsProcessor`.
jgambarios added a commit that referenced this issue Oct 29, 2024
@jgambarios
#29498 Enable Weld JUnit5 in integration test
0d13629 
Added `@EnableWeld` annotation and extended `ImportContentletsProcessorIntegrationTest` from `Junit5WeldBaseTest`. These changes integrate Weld with the JUnit5 testing framework, enabling dependency injection and enhancing the test's capabilities.
jgambarios added a commit that referenced this issue Oct 29, 2024
@jgambarios
#29498 Applying code review feedback
d5e7f95
jgambarios added a commit that referenced this issue Oct 29, 2024
@jgambarios
#29498 Applying code review feedback
e601cea
@github-project-automation github-project-automation bot moved this from In Review to Internal QA in dotCMS - Product Planning Oct 30, 2024
@github-actions github-actions bot mentioned this issue Oct 30, 2024
Merged
jgambarios added a commit that referenced this issue Nov 6, 2024
@jgambarios
#29498 Add content type validation and exception handling
cc95b7b 
Refactor to introduce `findContentType` for content type retrieval and validation. This involves handling cases where content type is not found, adding detailed error messages, and ensuring security checks.
jgambarios added a commit that referenced this issue Nov 7, 2024
@jgambarios
#29498 Adding support for using language iso code in the language par…
d3ccbd3 
…ameter.
jgambarios added a commit that referenced this issue Nov 8, 2024
@jgambarios
#29498 Error handling improvements for the server sent events we use …
4e3ee0d 
…for the monitor of jobs
jgambarios added a commit that referenced this issue Nov 9, 2024
@jgambarios
#29498 Adjusting tests to the recent changes
718daad
jgambarios added a commit that referenced this issue Nov 11, 2024
@jgambarios
#29498 Reverting changes in LanguageFactoryImpl, leaving exactly as i…
4e0d89c 
…t was.
@jgambarios jgambarios linked a pull request Nov 11, 2024 that will close this issue
Feat (Core): Migrate import contentlets action to job processor - Feedback improvements #30617
Merged
jgambarios added a commit that referenced this issue Nov 11, 2024
@jgambarios
#29498 Applying PR feedback.
c99a4fa
jgambarios added a commit that referenced this issue Nov 11, 2024
@jgambarios
#29498 Improving postman tests
d04f7df
jgambarios added a commit that referenced this issue Nov 12, 2024
@jgambarios
#29498 Improving postman tests
be97d64
jgambarios added a commit that referenced this issue Nov 12, 2024
@jgambarios
#29498 Improving postman tests
c376421
jgambarios added a commit that referenced this issue Nov 12, 2024
@jgambarios
#29498 Fixing tests
8977a5f
@github-project-automation github-project-automation bot moved this from In Review to Internal QA in dotCMS - Product Planning Nov 13, 2024
@github-actions github-actions bot mentioned this issue Nov 13, 2024
Merged
@jgambarios jgambarios reopened this Nov 13, 2024
@github-project-automation github-project-automation bot moved this from Internal QA to Current Sprint Backlog in dotCMS - Product Planning Nov 13, 2024
@jgambarios jgambarios moved this from Current Sprint Backlog to Internal QA in dotCMS - Product Planning Nov 13, 2024
@jgambarios jgambarios removed their assignment Nov 13, 2024
@fabrizzio-dotCMS fabrizzio-dotCMS self-assigned this Nov 14, 2024
This was referenced Nov 15, 2024
Open
Open
Open
@fabrizzio-dotCMS
Copy link
Contributor Author

a few new issues were found and will be attended to in separate tickets
#30665
#30667
#30668

@nollymar nollymar moved this from Internal QA to QA - Backlog in dotCMS - Product Planning Nov 15, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
NW Removed QA : Passed Internal Team : Scout Type : Task
Projects
dotCMS - Product Planning
Status: QA - Backlog
Milestone
No milestone
4 participants
@fmontes @jgambarios @nollymar @fabrizzio-dotCMS