Merge pull request #478 from Qovery/chore_multienv_autodeploy

chore: add special case for auto-deploy with multiple envs

website/docs/using-qovery/deployment.md

@@ -1,5 +1,5 @@
 ---
-last_modified_on: "2023-10-10"
+last_modified_on: "2024-10-15"
 title: Deployment
 description: "Everything you need to know about the deployment of your applications with Qovery"
 sidebar_label: hidden

website/docs/using-qovery/deployment/deployment-pipeline.md

@@ -1,5 +1,5 @@
 ---
-last_modified_on: "2023-04-27"
+last_modified_on: "2024-10-15"
 title: "Deployment Pipeline"
 description: "Learn how to the Environment Deployment Pipeline works"
 ---
@@ -36,26 +36,41 @@ Below you can find a visual example of how the pipeline looks like:
 
 When the deployment pipeline execute the deployment of a stage, the services within it will go through the `Build` and `Deployment` phases. 
 
-The Building process is managed by the Qovery CI which downloads your repository and generates the final image that will be run on your Kubernetes cluster. 
+### Application building
 
 <Alert type="info">
 
-By default, the nodes building your application have the following resources: 4CPU and 4 GB memory. If you need more resources, get in touch with our support.
+**Important note**: If you already have an image available on a container registry that has been previously created by your own CI/CD, it might be interesting for you to reuse it instead of re-building it again on Qovery side. Have a look at [this section][docs.using-qovery.integration.continuous-integration] on how to deploy it.
 
 </Alert>
 
+The Building process is managed by the Qovery CI which downloads your repository and generates the final image that will be run on your Kubernetes cluster. 
+
+The build operation of each service within a deployment stage is executed in parallel with a parallism of 7. 
+
+Once the operation is completed, the built image is pushed in the `mirroring registry` for deployment ([check here][docs.using-qovery.deployment.image-mirroring#application-built-via-the-qovery-deployment-pipeline] for more information).
+
+Check [this section](#best-practices-to-speed-up-build-time) to know more on how to reduce the build time of your application.
+
 <Alert type="info">
 
-**Important note**: If you already have an image available on a container registry that has been previously created by your own CI/CD, it might be interesting for you to reuse it instead of re-building it again on Qovery side. Have a look at [this section][docs.using-qovery.integration.continuous-integration] on how to deploy it.
+By default, the nodes building your application have the following resources: 4CPU and 4 GB memory. If you need more resources, get in touch with our support.
 
 </Alert>
 
-The build and deploy operation of each service within a deployment stage are executed in parallel with a parallism of 4. 
+
+### Application deployment
+
+The deployment process is managed by the Qovery engine via helm charts. The engine generates a custom chart to deploy your application/job, unless you have already provided your own helm chart.
+
+The deployed image is pulled from the `mirroring registry` ([check here][docs.using-qovery.deployment.image-mirroring] for more information)
+
+The deployment operation of each service within a deployment stage is executed in parallel with a parallism of 7. 
 
 **Example**
-If you have 6 applications to be deployed within a stage, Qovery will:
-* build 4 applications in parallel. Once the build of one application is terminated, Qovery will start immediately another one until all the applications are built.
-* deploy 4 applications in parallel on your Kubernetes cluster. Once the deployment of one application is terminated, Qovery will start immediately another one until all the applications are deployed.
+If you have 10 applications to be deployed within a stage, Qovery will:
+* build 7 applications in parallel. Once the build of one application is terminated, Qovery will start immediately another one until all the applications are built.
+* deploy 7 applications in parallel on your Kubernetes cluster. Once the deployment of one application is terminated, Qovery will start immediately another one until all the applications are deployed.
 
 <Alert type="info">
 
@@ -89,7 +104,55 @@ This default assignment is maintained as long as you do not delete or rename the
 
 You can access and modify the pipeline configuration from the environment settings. Have a look at [this section][docs.using-qovery.configuration.environment#deployment-pipeline] to know more.
 
+## Best practices to speed up build time
+
+Here you can find some best practices you can follow to ensure you benefit from all the caching system and reduce the build time.
+
+### Review your Dockerfile structure
+
+Review the content of your Dockerfile to find performance improvements. There's a list of suggestion on the [Docker website](https://docs.docker.com/build/building/best-practices), feel free to ask for help on [our forum](https://discuss.qovery.com/) 
+
+### Benefit from the build and deployment parallelism
+
+Try to put on the same deployment stage as many apps as you can, making sure there is no dependency between them. It will allow you to benefit from the parallel build and deployment.
+
+### Update your Qovery configuration based on your application repository structure
+
+#### Multi-stage deployment on the same environment
+
+If within the same environment you have multiple applications using the same git repository and build context, you can benefit from the image caching mechanism provided by the [mirriring registry][docs.using-qovery.deployment.image-mirroring#application-built-via-the-qovery-deployment-pipeline] by:
+
+1. having one application X on a stage A. This is the one that will be built each time
+2. having all the other application on other stages as long as they are after the stage A. For all these applications the build phase will be skipped since the image has already been built from application X
+
+
+#### Managing Preview environments
+
+No real configuration change is required here to speed up your deployment. Ensure to avoid as much as you can to use environment specific variables as ARGS within your Dockerfile (such as `QOVERY_ENVIRONMENT_ID`) since this will force a new build on each environment.
+
+
+#### Cross-environment deploy with auto-deploy enabled
+
+When the auto-deploy feature is enabled and you deploy the same application across multiple environments, the applications will be built separately on each environment and you can't benefit from the caching mechanism available for the already built images.
+
+A small diagram to explain it:
+
+<p align="center">
+ <img src="/img/deployment/autodeploy_build.png" alt="Auto-deploy schema" />
+</p>
+
+Example: two apps A and B are configured on Qovery pointing to a repo X.
+
+Flow:
+1. A commit is pushed on the repo X. Git(hub/lab) inform Qovery about the new commit
+2. Qovery starts the deployment of the two apps separately and checks the existence of the image. At this moment, the image does not exist and thus both deployments move forward with the build phase.
+3. Qovery starts building the image for each application and pushes it onto the repository.
+
+As you can see, every deployment is independent and the build choice is only based on the existence or not of the image on the container registry at the very beginning.
+
 
 [docs.using-qovery.configuration.environment#deployment-pipeline]: /docs/using-qovery/configuration/environment/#deployment-pipeline
 [docs.using-qovery.configuration.environment]: /docs/using-qovery/configuration/environment/
+[docs.using-qovery.deployment.image-mirroring#application-built-via-the-qovery-deployment-pipeline]: /docs/using-qovery/deployment/image-mirroring/#application-built-via-the-qovery-deployment-pipeline
+[docs.using-qovery.deployment.image-mirroring]: /docs/using-qovery/deployment/image-mirroring/
 [docs.using-qovery.integration.continuous-integration]: /docs/using-qovery/integration/continuous-integration/

website/docs/using-qovery/deployment/deployment-pipeline.md.erb

@@ -27,26 +27,41 @@ Below you can find a visual example of how the pipeline looks like:
 
 When the deployment pipeline execute the deployment of a stage, the services within it will go through the `Build` and `Deployment` phases. 
 
-The Building process is managed by the Qovery CI which downloads your repository and generates the final image that will be run on your Kubernetes cluster. 
+### Application building
 
 <Alert type="info">
 
-By default, the nodes building your application have the following resources: 4CPU and 4 GB memory. If you need more resources, get in touch with our support.
+**Important note**: If you already have an image available on a container registry that has been previously created by your own CI/CD, it might be interesting for you to reuse it instead of re-building it again on Qovery side. Have a look at [this section][docs.using-qovery.integration.continuous-integration] on how to deploy it.
 
 </Alert>
 
+The Building process is managed by the Qovery CI which downloads your repository and generates the final image that will be run on your Kubernetes cluster. 
+
+The build operation of each service within a deployment stage is executed in parallel with a parallism of 7. 
+
+Once the operation is completed, the built image is pushed in the `mirroring registry` for deployment ([check here][docs.using-qovery.deployment.image-mirroring#application-built-via-the-qovery-deployment-pipeline] for more information).
+
+Check [this section](#best-practices-to-speed-up-build-time) to know more on how to reduce the build time of your application.
+
 <Alert type="info">
 
-**Important note**: If you already have an image available on a container registry that has been previously created by your own CI/CD, it might be interesting for you to reuse it instead of re-building it again on Qovery side. Have a look at [this section][docs.using-qovery.integration.continuous-integration] on how to deploy it.
+By default, the nodes building your application have the following resources: 4CPU and 4 GB memory. If you need more resources, get in touch with our support.
 
 </Alert>
 
-The build and deploy operation of each service within a deployment stage are executed in parallel with a parallism of 4. 
+
+### Application deployment
+
+The deployment process is managed by the Qovery engine via helm charts. The engine generates a custom chart to deploy your application/job, unless you have already provided your own helm chart.
+
+The deployed image is pulled from the `mirroring registry` ([check here][docs.using-qovery.deployment.image-mirroring] for more information)
+
+The deployment operation of each service within a deployment stage is executed in parallel with a parallism of 7. 
 
 **Example**
-If you have 6 applications to be deployed within a stage, Qovery will:
-* build 4 applications in parallel. Once the build of one application is terminated, Qovery will start immediately another one until all the applications are built.
-* deploy 4 applications in parallel on your Kubernetes cluster. Once the deployment of one application is terminated, Qovery will start immediately another one until all the applications are deployed.
+If you have 10 applications to be deployed within a stage, Qovery will:
+* build 7 applications in parallel. Once the build of one application is terminated, Qovery will start immediately another one until all the applications are built.
+* deploy 7 applications in parallel on your Kubernetes cluster. Once the deployment of one application is terminated, Qovery will start immediately another one until all the applications are deployed.
 
 <Alert type="info">
 
@@ -80,3 +95,48 @@ This default assignment is maintained as long as you do not delete or rename the
 
 You can access and modify the pipeline configuration from the environment settings. Have a look at [this section][docs.using-qovery.configuration.environment#deployment-pipeline] to know more.
 
+## Best practices to speed up build time
+
+Here you can find some best practices you can follow to ensure you benefit from all the caching system and reduce the build time.
+
+### Review your Dockerfile structure
+
+Review the content of your Dockerfile to find performance improvements. There's a list of suggestion on the [Docker website](https://docs.docker.com/build/building/best-practices), feel free to ask for help on [our forum](https://discuss.qovery.com/) 
+
+### Benefit from the build and deployment parallelism
+
+Try to put on the same deployment stage as many apps as you can, making sure there is no dependency between them. It will allow you to benefit from the parallel build and deployment.
+
+### Update your Qovery configuration based on your application repository structure
+
+#### Multi-stage deployment on the same environment
+
+If within the same environment you have multiple applications using the same git repository and build context, you can benefit from the image caching mechanism provided by the [mirriring registry][docs.using-qovery.deployment.image-mirroring#application-built-via-the-qovery-deployment-pipeline] by:
+
+1. having one application X on a stage A. This is the one that will be built each time
+2. having all the other application on other stages as long as they are after the stage A. For all these applications the build phase will be skipped since the image has already been built from application X
+
+
+#### Managing Preview environments
+
+No real configuration change is required here to speed up your deployment. Ensure to avoid as much as you can to use environment specific variables as ARGS within your Dockerfile (such as `QOVERY_ENVIRONMENT_ID`) since this will force a new build on each environment.
+
+
+#### Cross-environment deploy with auto-deploy enabled
+
+When the auto-deploy feature is enabled and you deploy the same application across multiple environments, the applications will be built separately on each environment and you can't benefit from the caching mechanism available for the already built images.
+
+A small diagram to explain it:
+
+<p align="center">
+ <img src="/img/deployment/autodeploy_build.png" alt="Auto-deploy schema" />
+</p>
+
+Example: two apps A and B are configured on Qovery pointing to a repo X.
+
+Flow:
+1. A commit is pushed on the repo X. Git(hub/lab) inform Qovery about the new commit
+2. Qovery starts the deployment of the two apps separately and checks the existence of the image. At this moment, the image does not exist and thus both deployments move forward with the build phase.
+3. Qovery starts building the image for each application and pushes it onto the repository.
+
+As you can see, every deployment is independent and the build choice is only based on the existence or not of the image on the container registry at the very beginning.

website/docs/using-qovery/deployment/image-mirroring.md

@@ -1,5 +1,5 @@
 ---
-last_modified_on: "2024-10-09"
+last_modified_on: "2024-10-15"
 title: "Image Mirroring"
 description: "Learn how images are mirrored within your cloud account"
 ---
@@ -9,15 +9,15 @@ import Assumptions from '@site/src/components/Assumptions';
 
 When Qovery is running on your infrastructure, it requires an image registry to store the images built via the Qovery CI and to mirror the images deployed from a 3rd party container registry.
 
-This `mirroring registry` is available and configurable within the Qovery interface
+This `mirroring registry` is available and configurable within the Qovery interface within the `Image registry` section of your cluster.
 
 <p align="center">
  <img src="/img/deployment/mirror-registry.png" alt="Mirroring Repository" />
 </p>
 
 # How does it work
 
-Every time an application needs to be deployed on your cluster, the application image is mirrored on the mirroring registry.
+Every time an application needs to be deployed on your cluster, the deployed image is picked from the `mirroring registry`. How the image is pushed on the `mirroring registry` it depends if you build the application with the Qovery Deployment Pipeline or not.
 
 <!--
  THIS FILE IS AUTOGENERATED!
@@ -27,23 +27,28 @@ Every time an application needs to be deployed on your cluster, the application
  website/docs/using-qovery/deployment/image-mirroring.md.erb
 -->
 
-## Application built via the Qovery pipeline
+## Application built via the Qovery Deployment Pipeline
 
-Images in the mirroring registry are organized by Git repository. Each service, with identical build context parameters (commit ID, repository root path, Dockerfile path, Dockerfile content, and environment variables), is assigned its own repository, named z<short_cluster_id>-git_repo_name (or namespace, depending on the cloud provider). This ensures that each service's build and mirroring process is completely isolated from others.
+Images built by Qovery are organized in the mirroring registry by "Git repository", meaning that the image built on services having the same git repsitory will be pushed in the same repository, named z<short_cluster_id>-git_repo_name (or namespace, depending on the cloud provider).
+
+The tag assigned to the built image is based on the following elements (build context): commit ID, repository root path, Dockerfile path, Dockerfile content, and ARGS environment variables (present in the dockerfile). This ensures that each service's build and mirroring process is completely isolated from others.
 
 Before building the application A1, Qovery checks within mirroring registry at the repository of the application A1 if an image has already being built with the build context parameters (commit id, repository root path, dockerfile path, dockerfile content and environment variables) within the same cluster. 
 
-If the image already exists, the built is skipped and Qovery starts the deployment of that image on the Kubernetes cluster.
+If the image already exists, the build is skipped and Qovery starts the deployment of that image on the Kubernetes cluster.
 
-Otherwise, the image is built by the Qovery pipeline the resulting image is pushed on the mirroring registry at the repository of the application A1, deleting any previous image.
+Otherwise, the image is built by the Qovery pipeline the resulting image is pushed on the mirroring registry at the repository of the application A1. 
 
 <p align="center">
  <img src="/img/deployment/build-mirror.png" alt="Mirroring built image" />
 </p>
 
-In order to speed up the image build, we are using remote caches (available in AWS, GCP and Scaleway). It will avoid building the image from scratch, only the layers that changed will be built.
+Once an application is deleted, if no other application is using the same image name and tag, the image is deleted from the mirroring registry.
+
+In order to speed up the image build, we are using the mirroring registry as a remote cache (available in AWS, GCP and Scaleway). It will avoid building the image from scratch, only the layers that changed will be built.
+
+Check out the Best practices section below to know some best practices you can follow to ensure you benefit from all the caching system and reduce the build time.
 
-Given this isolation mechanism, if the same application is cloned (via the [clone][docs.using-qovery.configuration.environment#clone-environment] or [preview environment][docs.using-qovery.configuration.environment#preview-environment] feature), Qovery will re-build the application since the environment variables have changed (the ones at environment level).
 
 ## Application deployed from a container registry
 
@@ -127,5 +132,3 @@ You can push this image on the mirroring registry within the repository `nginx`,
 
 
 [docs.using-qovery.configuration.cluster-advanced-settings#image-registry]: /docs/using-qovery/configuration/cluster-advanced-settings/#image-registry
-[docs.using-qovery.configuration.environment#clone-environment]: /docs/using-qovery/configuration/environment/#clone-environment
-[docs.using-qovery.configuration.environment#preview-environment]: /docs/using-qovery/configuration/environment/#preview-environment