Skip to content

Commit

Permalink
Merge pull request #2079 from recommenders-team/staging
Browse files Browse the repository at this point in the history
Staging to main: Support python 3.10 and 3.11
  • Loading branch information
miguelgfierro authored Apr 5, 2024
2 parents 310370b + 433e912 commit c2ea583
Show file tree
Hide file tree
Showing 26 changed files with 416 additions and 324 deletions.
51 changes: 20 additions & 31 deletions .github/actions/azureml-test/action.yml
Original file line number Diff line number Diff line change
Expand Up @@ -69,56 +69,45 @@ runs:
using: "composite"
steps:
- name: Setup python
uses: actions/setup-python@v4
uses: actions/setup-python@v5
with:
python-version: "3.8"
- name: Install azureml-core and azure-cli on a GitHub hosted server
shell: bash
run: pip install --quiet "azureml-core>1,<2" "azure-cli>2,<3"
- name: Log in to Azure
uses: azure/login@v1
uses: azure/login@v2
with:
creds: ${{inputs.AZUREML_TEST_CREDENTIALS}}
- name: Install wheel package
shell: bash
run: pip install --quiet wheel
- name: Submit CPU tests to AzureML
- name: Submit tests to AzureML
shell: bash
if: contains(inputs.TEST_GROUP, 'cpu')
run: >-
python tests/ci/azureml_tests/submit_groupwise_azureml_pytest.py --clustername ${{inputs.CPU_CLUSTER_NAME}}
--subid ${{inputs.AZUREML_TEST_SUBID}} --reponame "recommenders" --branch ${{ github.ref }}
--rg ${{inputs.RG}} --wsname ${{inputs.WS}} --expname ${{inputs.EXP_NAME}}_${{inputs.TEST_GROUP}}
--testlogs ${{inputs.TEST_LOGS_PATH}} --testkind ${{inputs.TEST_KIND}}
--conda_pkg_python ${{inputs.PYTHON_VERSION}} --testgroup ${{inputs.TEST_GROUP}}
--disable-warnings --sha "${GITHUB_SHA}"
- name: Submit GPU tests to AzureML
shell: bash
if: contains(inputs.TEST_GROUP, 'gpu')
run: >-
python tests/ci/azureml_tests/submit_groupwise_azureml_pytest.py --clustername ${{inputs.GPU_CLUSTER_NAME}}
--subid ${{inputs.AZUREML_TEST_SUBID}} --reponame "recommenders" --branch ${{ github.ref }}
--rg ${{inputs.RG}} --wsname ${{inputs.WS}} --expname ${{inputs.EXP_NAME}}_${{inputs.TEST_GROUP}}
--testlogs ${{inputs.TEST_LOGS_PATH}} --add_gpu_dependencies --testkind ${{inputs.TEST_KIND}}
--conda_pkg_python ${{inputs.PYTHON_VERSION}} --testgroup ${{inputs.TEST_GROUP}}
--disable-warnings --sha "${GITHUB_SHA}"
- name: Submit PySpark tests to AzureML
shell: bash
if: contains(inputs.TEST_GROUP, 'spark')
run: >-
python tests/ci/azureml_tests/submit_groupwise_azureml_pytest.py --clustername ${{inputs.CPU_CLUSTER_NAME}}
--subid ${{inputs.AZUREML_TEST_SUBID}} --reponame "recommenders" --branch ${{ github.ref }}
--rg ${{inputs.RG}} --wsname ${{inputs.WS}} --expname ${{inputs.EXP_NAME}}_${{inputs.TEST_GROUP}}
--testlogs ${{inputs.TEST_LOGS_PATH}} --add_spark_dependencies --testkind ${{inputs.TEST_KIND}}
--conda_pkg_python ${{inputs.PYTHON_VERSION}} --testgroup ${{inputs.TEST_GROUP}}
--disable-warnings --sha "${GITHUB_SHA}"
python tests/ci/azureml_tests/submit_groupwise_azureml_pytest.py \
--subid ${{inputs.AZUREML_TEST_SUBID}} \
--reponame "recommenders" \
--branch ${{ github.ref }} \
--rg ${{inputs.RG}} \
--wsname ${{inputs.WS}} \
--expname ${{inputs.EXP_NAME}}_${{inputs.TEST_GROUP}} \
--testlogs ${{inputs.TEST_LOGS_PATH}} \
--testkind ${{inputs.TEST_KIND}} \
--conda_pkg_python ${{inputs.PYTHON_VERSION}} \
--testgroup ${{inputs.TEST_GROUP}} \
--disable-warnings \
--sha "${GITHUB_SHA}" \
--clustername $(if [[ ${{inputs.TEST_GROUP}} =~ "gpu" ]]; then echo "${{inputs.GPU_CLUSTER_NAME}}"; else echo "${{inputs.CPU_CLUSTER_NAME}}"; fi) \
$(if [[ ${{inputs.TEST_GROUP}} =~ "gpu" ]]; then echo "--add_gpu_dependencies"; fi) \
$(if [[ ${{inputs.TEST_GROUP}} =~ "spark" ]]; then echo "--add_spark_dependencies"; fi)
- name: Get exit status
shell: bash
id: exit_status
run: echo "code=$(cat ${{inputs.PYTEST_EXIT_CODE}})" >> $GITHUB_OUTPUT
- name: Check Success/Failure
if: ${{ steps.exit_status.outputs.code != 0 }}
uses: actions/github-script@v3
uses: actions/github-script@v7
with:
script: |
core.setFailed('All tests did not pass!')
6 changes: 3 additions & 3 deletions .github/workflows/azureml-cpu-nightly.yml
Original file line number Diff line number Diff line change
Expand Up @@ -47,7 +47,7 @@ jobs:
runs-on: ubuntu-latest
steps:
- name: Check out repository code
uses: actions/checkout@v3
uses: actions/checkout@v4
- name: Get test group names
id: get_test_groups
uses: ./.github/actions/get-test-groups
Expand All @@ -67,11 +67,11 @@ jobs:
strategy:
max-parallel: 50 # Usage limits: https://docs.github.com/en/actions/learn-github-actions/usage-limits-billing-and-administration
matrix:
python-version: ['"python=3.8"', '"python=3.9"']
python-version: ['"python=3.8"', '"python=3.9"', '"python=3.10"', '"python=3.11"']
test-group: ${{ fromJSON(needs.get-test-groups.outputs.test_groups) }}
steps:
- name: Check out repository code
uses: actions/checkout@v3
uses: actions/checkout@v4
- name: Execute tests
uses: ./.github/actions/azureml-test
id: execute_tests
Expand Down
6 changes: 3 additions & 3 deletions .github/workflows/azureml-gpu-nightly.yml
Original file line number Diff line number Diff line change
Expand Up @@ -47,7 +47,7 @@ jobs:
runs-on: ubuntu-latest
steps:
- name: Check out repository code
uses: actions/checkout@v3
uses: actions/checkout@v4
- name: Get test group names
id: get_test_groups
uses: ./.github/actions/get-test-groups
Expand All @@ -67,11 +67,11 @@ jobs:
strategy:
max-parallel: 50 # Usage limits: https://docs.github.com/en/actions/learn-github-actions/usage-limits-billing-and-administration
matrix:
python-version: ['"python=3.8"', '"python=3.9"']
python-version: ['"python=3.8"', '"python=3.9"', '"python=3.10"', '"python=3.11"']
test-group: ${{ fromJSON(needs.get-test-groups.outputs.test_groups) }}
steps:
- name: Check out repository code
uses: actions/checkout@v3
uses: actions/checkout@v4
- name: Execute tests
uses: ./.github/actions/azureml-test
id: execute_tests
Expand Down
4 changes: 2 additions & 2 deletions .github/workflows/azureml-release-pipeline.yml
Original file line number Diff line number Diff line change
Expand Up @@ -33,9 +33,9 @@ jobs:
needs: [unit-test-workflow, cpu-nightly-workflow, gpu-nightly-workflow, spark-nightly-workflow]
steps:
- name: Check out repository code
uses: actions/checkout@v3
uses: actions/checkout@v4
- name: Setup python
uses: actions/setup-python@v4
uses: actions/setup-python@v5
with:
python-version: "3.8"
- name: Install wheel package
Expand Down
6 changes: 3 additions & 3 deletions .github/workflows/azureml-spark-nightly.yml
Original file line number Diff line number Diff line change
Expand Up @@ -46,7 +46,7 @@ jobs:
runs-on: ubuntu-latest
steps:
- name: Check out repository code
uses: actions/checkout@v3
uses: actions/checkout@v4
- name: Get test group names
id: get_test_groups
uses: ./.github/actions/get-test-groups
Expand All @@ -66,11 +66,11 @@ jobs:
strategy:
max-parallel: 50 # Usage limits: https://docs.github.com/en/actions/learn-github-actions/usage-limits-billing-and-administration
matrix:
python-version: ['"python=3.8"', '"python=3.9"']
python-version: ['"python=3.8"', '"python=3.9"', '"python=3.10"', '"python=3.11"']
test-group: ${{ fromJSON(needs.get-test-groups.outputs.test_groups) }}
steps:
- name: Check out repository code
uses: actions/checkout@v3
uses: actions/checkout@v4
- name: Execute tests
uses: ./.github/actions/azureml-test
id: execute_tests
Expand Down
6 changes: 3 additions & 3 deletions .github/workflows/azureml-unit-tests.yml
Original file line number Diff line number Diff line change
Expand Up @@ -36,7 +36,7 @@ jobs:
runs-on: ubuntu-latest
steps:
- name: Check out repository code
uses: actions/checkout@v3
uses: actions/checkout@v4
- name: Get test group names
id: get_test_groups
uses: ./.github/actions/get-test-groups
Expand All @@ -56,11 +56,11 @@ jobs:
strategy:
max-parallel: 50 # Usage limits: https://docs.github.com/en/actions/learn-github-actions/usage-limits-billing-and-administration
matrix:
python-version: ['"python=3.8"', '"python=3.9"']
python-version: ['"python=3.8"', '"python=3.9"', '"python=3.10"', '"python=3.11"']
test-group: ${{ fromJSON(needs.get-test-groups.outputs.test_groups) }}
steps:
- name: Check out repository code
uses: actions/checkout@v3
uses: actions/checkout@v4
- name: Execute tests
uses: ./.github/actions/azureml-test
id: execute_tests
Expand Down
14 changes: 7 additions & 7 deletions .github/workflows/sarplus.yml
Original file line number Diff line number Diff line change
Expand Up @@ -39,12 +39,12 @@ jobs:
runs-on: ubuntu-22.04
strategy:
matrix:
python-version: ["3.8", "3.9"]
python-version: ["3.8", "3.9", "3.10", "3.11"]
steps:
- uses: actions/checkout@v3
- uses: actions/checkout@v4

- name: Set up Python ${{ matrix.python-version }}
uses: actions/setup-python@v4
uses: actions/setup-python@v5
with:
python-version: ${{ matrix.python-version }}

Expand Down Expand Up @@ -96,15 +96,15 @@ jobs:
- name: Upload Python wheel as GitHub artifact when merged into main
# Upload the whl file of the specific python version
if: github.ref == 'refs/heads/main'
uses: actions/upload-artifact@v2
uses: actions/upload-artifact@v4
with:
name: pysarplus-${{ env.sarplus_version }}-cp${{ matrix.python-version }}-wheel
path: ${{ env.PYTHON_ROOT }}/dist/*.whl

- name: Upload Python source as GitHub artifact when merged into main
# Only one pysarplus source tar file is needed
if: github.ref == 'refs/heads/main' && matrix.python-version == '3.10'
uses: actions/upload-artifact@v2
uses: actions/upload-artifact@v4
with:
name: pysarplus-${{ env.sarplus_version }}-source
path: ${{ env.PYTHON_ROOT }}/dist/*.tar.gz
Expand All @@ -131,7 +131,7 @@ jobs:
hadoop-version: "3.3.1"

steps:
- uses: actions/checkout@v3
- uses: actions/checkout@v4

- name: Test
run: |
Expand Down Expand Up @@ -180,7 +180,7 @@ jobs:
- name: Upload Scala bundle as GitHub artifact when merged into main
if: github.ref == 'refs/heads/main'
uses: actions/upload-artifact@v2
uses: actions/upload-artifact@v4
with:
name: sarplus-${{ env.sarplus_version }}-bundle_2.12-spark-${{ matrix.spark-version }}-jar
path: ${{ env.SCALA_ROOT }}/target/scala-2.12/*bundle*.jar
6 changes: 3 additions & 3 deletions .github/workflows/update_documentation.yml
Original file line number Diff line number Diff line change
Expand Up @@ -16,12 +16,12 @@ jobs:

steps:
- name: Checkout repository
uses: actions/checkout@v3
uses: actions/checkout@v4

- name: Set up Python
uses: actions/setup-python@v4
uses: actions/setup-python@v5
with:
python-version: 3.10
python-version: "3.10"

- name: Install dependencies
run: |
Expand Down
4 changes: 2 additions & 2 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -158,7 +158,7 @@ The nightly build tests are run daily on AzureML.

## References

- D. Li, J. Lian, L. Zhang, K. Ren, D. Lu, T. Wu, X. Xie, "Recommender Systems: Frontiers and Practices" (in Chinese), Publishing House of Electronics Industry, Beijing 2022.
- D. Li, J. Lian, L. Zhang, K. Ren, D. Lu, T. Wu, X. Xie, "Recommender Systems: Frontiers and Practices", Springer, Beijing, 2024. [Available on this link](https://www.amazon.com/Recommender-Systems-Frontiers-Practices-Dongsheng/dp/9819989639/).
- A. Argyriou, M. González-Fierro, and L. Zhang, "Microsoft Recommenders: Best Practices for Production-Ready Recommendation Systems", *WWW 2020: International World Wide Web Conference Taipei*, 2020. Available online: https://dl.acm.org/doi/abs/10.1145/3366424.3382692
- L. Zhang, T. Wu, X. Xie, A. Argyriou, M. González-Fierro and J. Lian, "Building Production-Ready Recommendation System at Scale", *ACM SIGKDD Conference on Knowledge Discovery and Data Mining 2019 (KDD 2019)*, 2019.
- S. Graham, J.K. Min, T. Wu, "Microsoft recommenders: tools to accelerate developing recommender systems", *RecSys '19: Proceedings of the 13th ACM Conference on Recommender Systems*, 2019. Available online: https://dl.acm.org/doi/10.1145/3298689.3346967
- L. Zhang, T. Wu, X. Xie, A. Argyriou, M. González-Fierro and J. Lian, "Building Production-Ready Recommendation System at Scale", *ACM SIGKDD Conference on Knowledge Discovery and Data Mining 2019 (KDD 2019)*, 2019.
3 changes: 2 additions & 1 deletion contrib/sarplus/python/setup.py
Original file line number Diff line number Diff line change
Expand Up @@ -42,14 +42,15 @@ def __str__(self):
"Programming Language :: Python :: 3.8",
"Programming Language :: Python :: 3.9",
"Programming Language :: Python :: 3.10",
"Programming Language :: Python :: 3.11",
"Intended Audience :: Developers",
"Intended Audience :: Science/Research",
"Topic :: Scientific/Engineering :: Mathematics",
],
setup_requires=["pytest-runner"],
install_requires=DEPENDENCIES,
tests_require=["pytest"],
python_requires=">=3.6,<3.11",
python_requires=">=3.6,<3.12",
packages=["pysarplus"],
package_data={"": ["VERSION"]},
ext_modules=[
Expand Down
Loading

0 comments on commit c2ea583

Please sign in to comment.