feat: list and get tasks

[JaeAeich]

Summary by Sourcery

Implement a Task Execution Service (TES) API with Kubernetes integration for managing and executing tasks. Enhance the project with new features, refactor the codebase, improve documentation, and set up CI/CD workflows for quality assurance and automated deployments.

New Features:

• Introduce a new Task Execution Service (TES) API for managing and executing tasks, including endpoints for creating, listing, and retrieving tasks.
• Add support for handling tasks with inputs and outputs using various protocols such as HTTP, FTP, and S3.
• Implement a Kubernetes-based task execution engine that converts TES tasks to Kubernetes jobs and manages their lifecycle.

Enhancements:

• Refactor the project structure to separate concerns, including API, Kubernetes interactions, and task management.
• Improve the documentation with detailed deployment instructions, examples, and a brief introduction to TES.
• Enhance the CI/CD workflows with new actions for code quality checks, vulnerability assessments, and automated builds and deployments.

Build:

• Add a Makefile for managing build tasks, including creating virtual environments, installing dependencies, and building Docker images.
• Introduce Dockerfiles for building and running the filer and taskmaster components as Docker containers.

CI:

• Set up GitHub Actions workflows for code quality checks, including linting, formatting, type checking, and spell checking.
• Implement workflows for building and publishing Docker images for the filer and taskmaster components.

Deployment:

• Provide Helm charts and deployment configurations for deploying TESK on Kubernetes, including support for different storage backends like FTP and S3.

Documentation:

• Update the README and add detailed documentation for deploying TESK, including integration with WES-ELIXIR and using shared file systems.
• Add examples of TES tasks and configurations for different input/output scenarios.

Tests:

• Introduce unit tests for various components, including the taskmaster, filer, and Kubernetes interactions, to ensure functionality and reliability.

[sourcery-ai]

Reviewer's Guide by Sourcery

This pull request implements significant changes to the TESK (Task Execution Service on Kubernetes) project, including a major restructuring of the codebase, addition of new features, and improvements to existing functionality. The changes encompass updates to the API, Kubernetes integration, documentation, testing, and project configuration.

Architecture diagram for TESK task execution

graph TD;
    A[TESK API Pod] -->|Translates TES requests| B[Taskmaster Pod];
    B -->|Creates| C[Filer Pod];
    C -->|Creates PVC| D[Persistent Volume Claim];
    D -->|Used by| E[Executor Pods];
    E -->|Execute tasks| F[Task Completion];
    F -->|Triggers| G[Output Processing by Filer];
    G -->|Uploads outputs| H[Remote Storage];
    B -->|Monitors| E;

Loading

Class diagram for TESK API models

classDiagram
    class TesTask {
        +Optional~str~ id
        +Optional~TesState~ state
        +Optional~str~ name
        +Optional~str~ description
        +Optional~List~ inputs
        +Optional~List~ outputs
        +Optional~TesResources~ resources
        +List~TesExecutor~ executors
        +Optional~List~ volumes
        +Optional~Dict~ tags
        +Optional~List~ logs
        +Optional~str~ creation_time
    }
    class TesExecutor {
        +str image
        +List~str~ command
        +Optional~str~ workdir
        +Optional~str~ stdin
        +Optional~str~ stdout
        +Optional~str~ stderr
        +Optional~Dict~ env
        +Optional~bool~ ignore_error
    }
    class TesResources {
        +Optional~int~ cpu_cores
        +Optional~bool~ preemptible
        +Optional~float~ ram_gb
        +Optional~float~ disk_gb
        +Optional~List~ zones
        +Optional~Dict~ backend_parameters
        +Optional~bool~ backend_parameters_strict
    }
    TesTask --> TesExecutor
    TesTask --> TesResources

Loading

File-Level Changes

Change	Details	Files
Restructured project layout and added new modules	Created new directories for API, Kubernetes, and service modules Added new files for constants, exceptions, and utility functions Implemented new classes for task handling and Kubernetes integration	tesk/api/ga4gh/tes/models.py tesk/k8s/converter/converter.py tesk/k8s/wrapper.py tesk/services/filer.py tesk/services/taskmaster.py tesk/app.py tesk/constants.py tesk/exceptions.py tesk/utils.py
Updated Kubernetes integration and job handling	Implemented new classes for Kubernetes job and task management Added functionality for converting TES tasks to Kubernetes jobs Updated job status checking and error handling	tesk/k8s/converter/data/job.py tesk/k8s/converter/data/task.py tesk/k8s/converter/data/task_builder.py tesk/services/job.py tesk/services/pvc.py
Enhanced API functionality and models	Added new API endpoints for task management Implemented service info endpoint Updated TES models and request handling	tesk/api/ga4gh/tes/controllers.py tesk/api/ga4gh/tes/service_info/service_info.py tesk/api/ga4gh/tes/task/create_task.py tesk/api/ga4gh/tes/task/get_task.py tesk/api/ga4gh/tes/task/list_tasks.py
Improved project configuration and build process	Added pyproject.toml for project configuration Updated Dockerfiles for filer and taskmaster Added new GitHub Actions workflows for testing and building	pyproject.toml deployment/containers/filer.Dockerfile deployment/containers/taskmaster.Dockerfile .github/workflows/code_quality.yaml .github/workflows/build_and_publish_charts.yaml .github/workflows/build_and_publish_filer.yaml .github/workflows/build_and_publish_taskmaster.yaml
Updated documentation and examples	Added new deployment documentation Updated README with new project structure and instructions Added new examples for task execution	deployment/documentation/deployment.md README.md docs/examples/inputFile.json docs/examples/inputHelloWorld.json docs/examples/inputHttp.json
Implemented new testing framework and added tests	Added unit tests for various modules Implemented new test utilities and fixtures Added configuration for test coverage and reporting	tests/test_unit/test_services/test_filer.py tests/test_unit/test_services/test_job.py tests/test_unit/test_services/test_taskmaster.py tests/test_unit/test_services/test_s3_filer.py tests/test_unit/test_services/test_filer_http_pytest.py tests/test_unit/test_services/test_filer_ftp_pytest.py

---

Tips and commands

Interacting with Sourcery

• Trigger a new review: Comment @sourcery-ai review on the pull request.
• Continue discussions: Reply directly to Sourcery's review comments.
• Generate a GitHub issue from a review comment: Ask Sourcery to create an
  issue from a review comment by replying to it.
• Generate a pull request title: Write @sourcery-ai anywhere in the pull
  request title to generate a title at any time.
• Generate a pull request summary: Write @sourcery-ai summary anywhere in
  the pull request body to generate a PR summary at any time. You can also use
  this command to specify where the summary should be inserted.

Customizing Your Experience

Access your dashboard to:

• Enable or disable review features such as the Sourcery-generated pull request
  summary, the reviewer's guide, and others.
• Change the review language.
• Add, remove or edit custom review instructions.
• Adjust other review settings.

Getting Help

• Contact our support team for questions or feedback.
• Visit our documentation for detailed guides and information.
• Keep in touch with the Sourcery team by following us on X/Twitter, LinkedIn or GitHub.

[JaeAeich]

Note the PR is still under progress and not tested, but feel free to go around and review.