Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Restructured Kubeflow Pipelines docs #3737

Merged
merged 35 commits into from
Jun 20, 2024

Conversation

hbelmiro
Copy link
Contributor

Resolves #3716.

This PR:

cc @StefanoFioravanzo @diegolovison @milosjava

@StefanoFioravanzo
Copy link
Member

StefanoFioravanzo commented May 21, 2024

@hbelmiro Awesome work!

I have just a few comments:

  • Let's rename How-to/User guides to User guides
  • Thanks for moving content to Build a More Advanced ML Pipeline. I think it's a good temporary measure. For longer term, it seems like this is not exactly a "how to guide", but rather an "Advanced" quickstart. Each bullet in this guide could reference to its own how to guide, while this example serves as an advanced tutorial/quickstart. If you agree, we can have a separation issue describing the desired end state.
  • Now that we have a solid structure in place we should start building a "feature map" to understand what are the features that need documentation, the ones that are already documented but need improvement, and the ones that are green. I can see us doing this with a table like this
Feature Status Type
feat A Missing User guide
feat A Partial Reference
feat B Done User guide
feat B Missing Reference
... ... ...

WDYT?

@hbelmiro
Copy link
Contributor Author

/hold for review

Copy link

Check out this pull request on  ReviewNB

See visual diffs & provide feedback on Jupyter Notebooks.


Powered by ReviewNB

@hbelmiro
Copy link
Contributor Author

@StefanoFioravanzo

  • Let's rename How-to/User guides to User guides

Done.

  • Thanks for moving content to Build a More Advanced ML Pipeline. I think it's a good temporary measure. For longer term, it seems like this is not exactly a "how to guide", but rather an "Advanced" quickstart. Each bullet in this guide could reference to its own how to guide, while this example serves as an advanced tutorial/quickstart. If you agree, we can have a separation issue describing the desired end state.
  • Now that we have a solid structure in place we should start building a "feature map" to understand what are the features that need documentation, the ones that are already documented but need improvement, and the ones that are green. I can see us doing this with a table like this

Sounds good. I created two issues for that:

Should we add them (and new ones that may arise) to #3712 even if they may not be completed on time for 1.9?

@hbelmiro
Copy link
Contributor Author

/unhold
ready for review

Copy link
Member

@thesuperzapper thesuperzapper left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There are still redirect issues, we need to make sure that all the old pages still redirect correctly, namely:

Which both currently 404.

content/en/_redirects Outdated Show resolved Hide resolved
@hbelmiro hbelmiro marked this pull request as draft May 30, 2024 15:10
@hbelmiro hbelmiro force-pushed the restructure-kfp branch 2 times, most recently from bdfc735 to 9a7cf22 Compare May 30, 2024 17:08
@hbelmiro hbelmiro marked this pull request as ready for review May 30, 2024 17:53
@hbelmiro
Copy link
Contributor Author

@thesuperzapper @rimolive @HumairAK @milosjava
Can you guys take another look?

@google-oss-prow google-oss-prow bot removed the lgtm label Jun 13, 2024
@hbelmiro
Copy link
Contributor Author

@rimolive
Copy link
Member

@thesuperzapper @StefanoFioravanzo Can you please verify if there are other 404s remaining?

@hbelmiro
Copy link
Contributor Author

@thesuperzapper
Copy link
Member

@hbelmiro it was very hard to review because the ordering was random, so I sorted them alphabetically and went through each one to ensure its source and target were correct and that no pages were missing.

Here is the fixed copy, please update the PR to replace the redirects you added to end of the _redirects file as follows:
(NOTE: leave the redirects you added in the previous sections, as they are still needed).

# Restructured the pipeline docs (https://github.com/kubeflow/website/issues/3716)
/docs/components/pipelines/overview/quickstart/                             /docs/components/pipelines/overview/
/docs/components/pipelines/v1/                                              /docs/components/pipelines/legacy-v1/
/docs/components/pipelines/v1/concepts/                                     /docs/components/pipelines/concepts/
/docs/components/pipelines/v1/concepts/component/                           /docs/components/pipelines/concepts/component/
/docs/components/pipelines/v1/concepts/experiment/                          /docs/components/pipelines/concepts/experiment/
/docs/components/pipelines/v1/concepts/graph/                               /docs/components/pipelines/concepts/graph/
/docs/components/pipelines/v1/concepts/metadata/                            /docs/components/pipelines/concepts/metadata/
/docs/components/pipelines/v1/concepts/output-artifact/                     /docs/components/pipelines/concepts/output-artifact/
/docs/components/pipelines/v1/concepts/pipeline/                            /docs/components/pipelines/concepts/pipeline/
/docs/components/pipelines/v1/concepts/run-trigger/                         /docs/components/pipelines/concepts/run-trigger/
/docs/components/pipelines/v1/concepts/run/                                 /docs/components/pipelines/concepts/run/
/docs/components/pipelines/v1/concepts/step/                                /docs/components/pipelines/concepts/step/
/docs/components/pipelines/v1/installation/                                 /docs/components/pipelines/legacy-v1/installation/
/docs/components/pipelines/v1/installation/choose-executor/                 /docs/components/pipelines/legacy-v1/installation/choose-executor/
/docs/components/pipelines/v1/installation/compatibility-matrix/            /docs/components/pipelines/legacy-v1/installation/compatibility-matrix/
/docs/components/pipelines/v1/installation/localcluster-deployment/         /docs/components/pipelines/legacy-v1/installation/localcluster-deployment/
/docs/components/pipelines/v1/installation/overview/                        /docs/components/pipelines/legacy-v1/installation/overview/
/docs/components/pipelines/v1/installation/standalone-deployment/           /docs/components/pipelines/legacy-v1/installation/standalone-deployment/
/docs/components/pipelines/v1/installation/upgrade/                         /docs/components/pipelines/legacy-v1/installation/upgrade/
/docs/components/pipelines/v1/introduction/                                 /docs/components/pipelines/legacy-v1/introduction/
/docs/components/pipelines/v1/overview/                                     /docs/components/pipelines/legacy-v1/overview/
/docs/components/pipelines/v1/overview/caching/                             /docs/components/pipelines/legacy-v1/overview/caching/
/docs/components/pipelines/v1/overview/interfaces/                          /docs/components/pipelines/interfaces/
/docs/components/pipelines/v1/overview/multi-user/                          /docs/components/pipelines/legacy-v1/overview/multi-user/
/docs/components/pipelines/v1/overview/pipeline-root/                       /docs/components/pipelines/concepts/pipeline-root/
/docs/components/pipelines/v1/overview/quickstart/                          /docs/components/pipelines/legacy-v1/overview/quickstart/
/docs/components/pipelines/v1/reference/                                    /docs/components/pipelines/legacy-v1/reference/
/docs/components/pipelines/v1/reference/api/kubeflow-pipeline-api-spec/     /docs/components/pipelines/legacy-v1/reference/api/kubeflow-pipeline-api-spec/
/docs/components/pipelines/v1/reference/component-spec/                     /docs/components/pipelines/reference/component-spec/
/docs/components/pipelines/v1/reference/sdk/                                /docs/components/pipelines/legacy-v1/reference/sdk/
/docs/components/pipelines/v1/sdk/                                          /docs/components/pipelines/legacy-v1/sdk/
/docs/components/pipelines/v1/sdk/best-practices/                           /docs/components/pipelines/legacy-v1/sdk/best-practices/
/docs/components/pipelines/v1/sdk/build-pipeline/                           /docs/components/pipelines/legacy-v1/sdk/build-pipeline/
/docs/components/pipelines/v1/sdk/component-development/                    /docs/components/pipelines/legacy-v1/sdk/component-development/
/docs/components/pipelines/v1/sdk/connect-api/                              /docs/components/pipelines/user-guides/core-functions/connect-api/
/docs/components/pipelines/v1/sdk/dsl-recursion/                            /docs/components/pipelines/legacy-v1/sdk/dsl-recursion/
/docs/components/pipelines/v1/sdk/enviroment_variables/                     /docs/components/pipelines/legacy-v1/sdk/enviroment_variables/
/docs/components/pipelines/v1/sdk/gcp/                                      /docs/components/pipelines/legacy-v1/sdk/gcp/
/docs/components/pipelines/v1/sdk/install-sdk/                              /docs/components/pipelines/legacy-v1/sdk/install-sdk/
/docs/components/pipelines/v1/sdk/manipulate-resources/                     /docs/components/pipelines/legacy-v1/sdk/manipulate-resources/
/docs/components/pipelines/v1/sdk/output-viewer/                            /docs/components/pipelines/legacy-v1/sdk/output-viewer/
/docs/components/pipelines/v1/sdk/parameters/                               /docs/components/pipelines/legacy-v1/sdk/parameters/
/docs/components/pipelines/v1/sdk/pipelines-with-tekton/                    /docs/components/pipelines/legacy-v1/sdk/pipelines-with-tekton/
/docs/components/pipelines/v1/sdk/python-based-visualizations/              /docs/components/pipelines/legacy-v1/sdk/python-based-visualizations/
/docs/components/pipelines/v1/sdk/python-function-components/               /docs/components/pipelines/legacy-v1/sdk/python-function-components/
/docs/components/pipelines/v1/sdk/sdk-overview/                             /docs/components/pipelines/legacy-v1/sdk/sdk-overview/
/docs/components/pipelines/v1/sdk/static-type-checking/                     /docs/components/pipelines/legacy-v1/sdk/static-type-checking/
/docs/components/pipelines/v1/troubleshooting/                              /docs/components/pipelines/legacy-v1/troubleshooting/
/docs/components/pipelines/v1/tutorials/                                    /docs/components/pipelines/legacy-v1/tutorials/
/docs/components/pipelines/v1/tutorials/api-pipelines/                      /docs/components/pipelines/legacy-v1/tutorials/api-pipelines/
/docs/components/pipelines/v1/tutorials/benchmark-examples/                 /docs/components/pipelines/legacy-v1/tutorials/benchmark-examples/
/docs/components/pipelines/v1/tutorials/build-pipeline/                     /docs/components/pipelines/legacy-v1/tutorials/build-pipeline/
/docs/components/pipelines/v1/tutorials/cloud-tutorials/                    /docs/components/pipelines/legacy-v1/tutorials/cloud-tutorials/
/docs/components/pipelines/v1/tutorials/sdk-examples/                       /docs/components/pipelines/legacy-v1/tutorials/sdk-examples/
/docs/components/pipelines/v2/                                              /docs/components/pipelines/
/docs/components/pipelines/v2/administration/                               /docs/components/pipelines/operator-guides/
/docs/components/pipelines/v2/administration/server-config/                 /docs/components/pipelines/operator-guides/server-config/
/docs/components/pipelines/v2/caching/                                      /docs/components/pipelines/user-guides/core-functions/caching/
/docs/components/pipelines/v2/cli/                                          /docs/components/pipelines/user-guides/core-functions/cli/
/docs/components/pipelines/v2/community-and-support/                        /docs/components/pipelines/reference/community-and-support/
/docs/components/pipelines/v2/compile-a-pipeline/                           /docs/components/pipelines/user-guides/core-functions/compile-a-pipeline/
/docs/components/pipelines/v2/components/                                   /docs/components/pipelines/user-guides/components/
/docs/components/pipelines/v2/components/additional-functionality/          /docs/components/pipelines/user-guides/components/additional-functionality/
/docs/components/pipelines/v2/components/container-components/              /docs/components/pipelines/user-guides/components/container-components/
/docs/components/pipelines/v2/components/containerized-python-components/   /docs/components/pipelines/user-guides/components/containerized-python-components/
/docs/components/pipelines/v2/components/importer-component/                /docs/components/pipelines/user-guides/components/importer-component/
/docs/components/pipelines/v2/components/lightweight-python-components/     /docs/components/pipelines/user-guides/components/lightweight-python-components/
/docs/components/pipelines/v2/data-types/                                   /docs/components/pipelines/user-guides/data-handling/data-types/
/docs/components/pipelines/v2/data-types/artifacts/                         /docs/components/pipelines/user-guides/data-handling/artifacts/
/docs/components/pipelines/v2/data-types/parameters/                        /docs/components/pipelines/user-guides/data-handling/parameters/
/docs/components/pipelines/v2/hello-world/                                  /docs/components/pipelines/getting-started/
/docs/components/pipelines/v2/installation/                                 /docs/components/pipelines/operator-guides/installation/
/docs/components/pipelines/v2/installation/quickstart/                      /docs/components/pipelines/operator-guides/installation/
/docs/components/pipelines/v2/introduction/                                 /docs/components/pipelines/overview/
/docs/components/pipelines/v2/load-and-share-components/                    /docs/components/pipelines/user-guides/components/load-and-share-components/
/docs/components/pipelines/v2/local-execution/                              /docs/components/pipelines/user-guides/core-functions/execute-kfp-pipelines-locally/
/docs/components/pipelines/v2/migration/                                    /docs/components/pipelines/user-guides/migration/
/docs/components/pipelines/v2/pipelines/                                    /docs/components/pipelines/user-guides/core-functions/
/docs/components/pipelines/v2/pipelines/control-flow/                       /docs/components/pipelines/user-guides/core-functions/control-flow/
/docs/components/pipelines/v2/pipelines/pipeline-basics/                    /docs/components/pipelines/user-guides/components/compose-components-into-pipelines/
/docs/components/pipelines/v2/platform-specific-features/                   /docs/components/pipelines/user-guides/core-functions/platform-specific-features/
/docs/components/pipelines/v2/reference/                                    /docs/components/pipelines/reference/
/docs/components/pipelines/v2/reference/api/kubeflow-pipeline-api-spec/     /docs/components/pipelines/reference/api/kubeflow-pipeline-api-spec/
/docs/components/pipelines/v2/reference/sdk/                                /docs/components/pipelines/reference/sdk/
/docs/components/pipelines/v2/run-a-pipeline/                               /docs/components/pipelines/user-guides/core-functions/run-a-pipeline/
/docs/components/pipelines/v2/version-compatibility/                        /docs/components/pipelines/reference/version-compatibility/

After that I can LGTM.

Signed-off-by: hbelmiro <[email protected]>
@hbelmiro
Copy link
Contributor Author

@thesuperzapper done.

@thesuperzapper
Copy link
Member

@hbelmiro thanks for your work on this, I think its ok to merge now.

/lgtm

Note, we will still get 404 for all the pre-v1/v2 split pages, but since we didn't want to make a new "component" path (e.g. rename the root prefix from /docs/components/pipelines/, there is no avoiding this.

@thesuperzapper
Copy link
Member

@james-jwu or @zijianjoy can you please approve this significant refactor of the Kubeflow Pipelines docs if you agree?

We need a root approver because we are updating the redirects.

Copy link
Member

@andreyvelich andreyvelich left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you for this hard work @hbelmiro!
We need someone from the KFP team to approve it.
/assign @kubeflow/pipelines

@@ -1,353 +0,0 @@
+++
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think we should keep the v2/installation/quickstart content instead of redirecting people to v1. It contains practical guidance to actually getting started with the product.

Copy link
Contributor Author

@hbelmiro hbelmiro Jun 17, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@zijianjoy we plan to create a proper v2 installation page in a follow-up PR. This PR is intended just to move pages avoiding changing their contents, otherwise, the review would be too difficult.

@zijianjoy
Copy link
Contributor

It contains a huge change in website architecture, including @connor-mccarthy who originally introduced the current structure to take a look.

@hbelmiro
Copy link
Contributor Author

@chensun @connor-mccarthy @zijianjoy
This week's meeting has been canceled. We can't wait until the next one to have this merged. We have several other issues planned for the 1.9 KF release that depend on this PR.
Can we get this merged or do you have any other suggestions?

@connor-mccarthy
Copy link
Member

My 2c: with the removal of the quickstart page and the new structure, some very basic usage patterns (e.g., define and compile a component) are deeply nested on the documentation site. I would consider restoring a quickstart guide.

A typical and critical reason for reviewing the docs is to determine whether to use Pipelines or some other ML orchestration system. A key input to this choice is often the SDK authoring experience. With the current restructuring, "Run a pipeline" is quite deep in the tree.

image

I wont block on this, but want to raise it for consideration before merging.

@thesuperzapper
Copy link
Member

thesuperzapper commented Jun 18, 2024

@connor-mccarthy the "Getting Started" guide actually gives an example of running a pipeline:
https://deploy-preview-3737--competent-brattain-de2d6d.netlify.app/docs/components/pipelines/getting-started/

But it probably needs to be updated for "Kubeflow Platform" deployments, which need special steps to submit pipelines from outside the cluster, probably by linking to the "Connect the Pipelines SDK to Kubeflow Pipelines" guide:
https://deploy-preview-3737--competent-brattain-de2d6d.netlify.app/docs/components/pipelines/user-guides/core-functions/connect-api/

However, I agree that we should link users from the "Getting Started" page to the new "User Guides" section, and point them at the most important topics in a logical order that "reads like a story" for new users.

@hbelmiro
Copy link
Contributor Author

@connor-mccarthy @thesuperzapper
Thank you for the suggestions.
I created a new issue and added it to the follow-up tasks we need to address.

@StefanoFioravanzo
Copy link
Member

I agree on those points, especially providing links in the Getting Started section. As Helber mentioned, this is something we can work on right after merging this PR. @connor-mccarthy It's important we proceed quickly so that we can start working independently on several improvements, including what you suggested, in time for the Kubeflow 1.9 release.

If you don't have other concerns, could you approve? We will prfioritize addressing your concerns right after.

@connor-mccarthy
Copy link
Member

/lgtm
/approve

Thanks everyone for the effort and discussion on this!

@hbelmiro
Copy link
Contributor Author

@zijianjoy we still need your approval to get this merged.

@james-jwu
Copy link

James Liu is OOO. I will approve on his behalf.
/lgtm
/approve

Copy link

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: connor-mccarthy, james-jwu, milosjava

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@google-oss-prow google-oss-prow bot merged commit 8e56df7 into kubeflow:master Jun 20, 2024
6 checks passed
@hbelmiro hbelmiro deleted the restructure-kfp branch June 20, 2024 18:59
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Review the structure of the docs to reflect Kubeflow 1.9
10 participants