Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

ci-sage: Update #386

Merged
merged 2 commits into from
Jan 19, 2024
Merged

ci-sage: Update #386

Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
16800b4
.github/workflows/ci-sage.yml: Remove cygwin (no longer supported by …
mkoeppe Jan 2, 2024
f9e6938
.github/workflows/ci-sage.yml: Use reusable workflows
mkoeppe Jan 2, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
327 changes: 39 additions & 288 deletions .github/workflows/ci-sage.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,9 +1,9 @@
name: Run Sage CI for Linux/Cygwin/macOS
name: Run Sage CI for Linux/macOS

## This GitHub Actions workflow provides:
##
## - portability testing, by building and testing this project on many platforms
## (Linux variants and Cygwin), each with two configurations (installed packages),
## (Linux variants and macOS), each with two configurations (installed packages),
##
## - continuous integration, by building and testing other software
## that depends on this project.
Expand Down Expand Up @@ -46,22 +46,24 @@ on:
tags:
- 'SageCI'

concurrency:
# Cancel previous runs of this workflow for the same branch
group: ${{ github.workflow }}-${{ github.ref }}
cancel-in-progress: true

env:
# Ubuntu packages to install so that the projects' "make dist" can succeed
DIST_PREREQ: libtool libopenblas-dev libgmp-dev
# Name of this project in the Sage distribution
SPKG: fflas_ffpack
# Sage distribution packages to build
TARGETS_PRE: build/make/Makefile
TARGETS: SAGE_CHECK=no SAGE_CHECK_givaro=yes SAGE_CHECK_fflas_ffpack=yes SAGE_CHECK_linbox=yes fflas_ffpack sagemath_linbox
TARGETS: SAGE_CHECK=no SAGE_CHECK_givaro=yes SAGE_CHECK_fflas_ffpack=yes SAGE_CHECK_linbox=yes fflas_ffpack
TARGETS_OPTIONAL: build/make/Makefile
# Standard setting: Test the current beta release of Sage:
SAGE_REPO: sagemath/sage
SAGE_REF: develop
# Test with the branch from sage ticket 31444 merged (provides modularized package sagemath-linbox)
SAGE_TRAC_GIT: https://github.com/sagemath/sagetrac-mirror.git
SAGE_TICKET: 31444
REMOVE_PATCHES: "*"
SAGE_REPO: sagemath/sage
SAGE_REF: develop
REMOVE_PATCHES: "*"
EXTRA_CONFIGURE_ARGS: --without-system-givaro --without-system-fflas_ffpack --without-system-linbox

jobs:
Expand Down Expand Up @@ -105,290 +107,39 @@ jobs:
&& echo "sage-package create linbox --version git --tarball linbox-git.tar.gz --type=standard" >> upstream/update-pkgs.sh \
&& if [ -n "${{ env.REMOVE_PATCHES }}" ]; then echo "(cd ../build/pkgs/linbox/patches && rm -f ${{ env.REMOVE_PATCHES }}; :)" >> upstream/update-pkgs.sh; fi \
&& ls -l upstream/
- uses: actions/upload-artifact@v2
- uses: actions/upload-artifact@v3
with:
path: upstream
name: upstream

cygwin:
env:
STAGE: i-a
LOCAL_ARTIFACT_NAME: sage-local-commit-${{ github.sha }}-cygwin-${{ matrix.pkgs }}
LOGS_ARTIFACT_NAME: logs-commit-${{ github.sha }}-cygwin-${{ matrix.pkgs }}
MAKE: make -j8
SAGE_NUM_THREADS: 3
SAGE_CHECK: yes
SAGE_CHECK_PACKAGES: "!cython,!r,!python3,!python2,!nose,!pathpy,!gap,!cysignals,!linbox,!git,!ppl"
CYGWIN: winsymlinks:native
CONFIGURE_ARGS: --enable-experimental-packages --enable-download-from-upstream-url
SAGE_FAT_BINARY: yes
SAGE_LOCAL: /opt/sage-fflas-ffpack-${{ github.sha }}

runs-on: windows-latest

linux:
uses: sagemath/sage/.github/workflows/docker.yml@develop
with:
# Extra system packages to install. See available packages at
# https://github.com/sagemath/sage/tree/develop/build/pkgs
extra_sage_packages: "info"
# Sage distribution packages to build
targets: SAGE_CHECK=no SAGE_CHECK_givaro=yes SAGE_CHECK_fflas_ffpack=yes SAGE_CHECK_linbox=yes givaro fflas_ffpack linbox
# Standard setting: Test the current beta release of Sage:
sage_repo: sagemath/sage
sage_ref: develop
upstream_artifact: upstream
# Docker targets (stages) to tag
docker_targets: "with-targets"
# We prefix the image name with the SPKG name ("singular_") to avoid the error
# 'Package "sage-docker-..." is already associated with another repository.'
docker_push_repository: ghcr.io/${{ github.repository }}/fflas_ffpack_
needs: [dist]

strategy:
fail-fast: false
matrix:
pkgs: [minimal, standard]
steps:
- run: |
git config --global core.autocrlf false
git config --global core.symlinks true
- name: install cygwin with choco
shell: bash {0}
run: |
choco --version
choco install git python3 --source cygwin
- name: Check out SageMath
uses: actions/checkout@v2
with:
repository: ${{ env.SAGE_REPO }}
ref: ${{ env.SAGE_REF }}
fetch-depth: 2000
if: env.SAGE_REPO != ''
- name: Check out git-trac-command
uses: actions/checkout@v2
with:
repository: sagemath/git-trac-command
path: git-trac-command
if: env.SAGE_TRAC_GIT != ''
- name: Check out SageMath from trac.sagemath.org
shell: bash {0}
# Random sleep and retry to limit the load on trac.sagemath.org
run: |
git config --global user.email "[email protected]"
git config --global user.name "ci-sage workflow"
if [ ! -d .git ]; then git init; fi; git remote add trac ${{ env.SAGE_TRAC_GIT }} && x=1 && while [ $x -le 5 ]; do x=$(( $x + 1 )); sleep $(( $RANDOM % 60 + 1 )); if git-trac-command/git-trac fetch $SAGE_TICKET; then git merge FETCH_HEAD || echo "(ignored)"; exit 0; fi; sleep 40; done; exit 1
if: env.SAGE_TRAC_GIT != ''
- uses: actions/download-artifact@v2
with:
path: upstream
name: upstream
- name: install minimal prerequisites with choco
shell: bash {0}
run: |
choco --version
PACKAGES="python38 python38-pip"
choco install $PACKAGES --source cygwin
- name: Update Sage packages from upstream artifact
run: |
C:\\tools\\cygwin\\bin\\bash -l -c 'cd $(cygpath -u "$GITHUB_WORKSPACE") && ls -l upstream/ && export PATH="$(pwd)/build/bin:$PATH:/usr/local/bin:/usr/bin" && (cd upstream && bash -x update-pkgs.sh) && git diff'
- name: tox
run: |
C:\\tools\\cygwin\\bin\\bash -l -x -c 'python3.8 -m pip install tox'
C:\\tools\\cygwin\\bin\\bash -l -x -c 'cd $(cygpath -u "$GITHUB_WORKSPACE") && PREFIX="${{ env.SAGE_LOCAL }}" tox -e local-cygwin-choco-${{ matrix.pkgs }} -- $TARGETS'
- name: Prepare logs artifact
shell: bash
run: |
mkdir -p "artifacts/$LOGS_ARTIFACT_NAME"; for a in local/var/tmp/sage/build/*; do if [ -d $a ]; then tar -c --remove-files -f "artifacts/$LOGS_ARTIFACT_NAME/$(basename $a).tar" $a || tar -c --ignore-failed-read -f "artifacts/$LOGS_ARTIFACT_NAME/$(basename $a)-save.tar" $a ; fi; done; cp -r logs/* "artifacts/$LOGS_ARTIFACT_NAME"
if: always()
- uses: actions/upload-artifact@v2
with:
path: artifacts
name: ${{ env.LOGS_ARTIFACT_NAME }}
if: always()
- name: Print out logs for immediate inspection
# The markup in the output is a GitHub Actions logging command
# https://help.github.com/en/actions/automating-your-workflow-with-github-actions/development-tools-for-github-actions
shell: bash
run: |
find "artifacts/$LOGS_ARTIFACT_NAME" -type f -name "*.log" -exec sh -c 'if tail -20 "{}" 2>/dev/null | grep "^Error" >/dev/null; then echo :":"error file={}:":" ==== LOG FILE {} CONTAINS AN ERROR ====; cat {} ; fi' \;
if: always()
- name: Prepare sage-local artifact
# We specifically use the cygwin tar so that symlinks are saved/restored correctly on Windows.
# We remove the $SAGE_LOCAL/lib64 link, which will be recreated by the next stage.
run: |
C:\\tools\\cygwin\\bin\\bash -l -c 'cd $(cygpath -u "$GITHUB_WORKSPACE") && rm -f "${{ env.SAGE_LOCAL }}"/lib64; tar -cf /tmp/sage-local-${{ env.STAGE }}.tar --remove-files "${{ env.SAGE_LOCAL }}"'
if: always()
- uses: actions/upload-artifact@v2
# upload-artifact@v2 does not support whitespace in file names.
# so we tar up the directory ourselves
with:
path: C:\\tools\\cygwin\\tmp\\sage-local-${{ env.STAGE }}.tar
name: ${{ env.LOCAL_ARTIFACT_NAME }}
if: always()

docker:
runs-on: ubuntu-latest
needs: [dist]
strategy:
fail-fast: false
max-parallel: 32
matrix:
tox_system_factor: [ubuntu-xenial, ubuntu-bionic, ubuntu-focal, ubuntu-groovy, ubuntu-hirsute, debian-stretch, debian-buster, debian-bullseye, debian-sid, linuxmint-18, linuxmint-19, linuxmint-19.3, linuxmint-20.1, fedora-26, fedora-27, fedora-28, fedora-29, fedora-30, fedora-31, fedora-32, fedora-33, fedora-34, centos-8, gentoo, archlinux-latest, slackware-14.2, conda-forge, ubuntu-bionic-i386, debian-buster-i386, centos-7-i386, raspbian-buster-armhf]
tox_packages_factor: [minimal, standard]
env:
TOX_ENV: docker-${{ matrix.tox_system_factor }}-${{ matrix.tox_packages_factor }}
LOGS_ARTIFACT_NAME: logs-commit-${{ github.sha }}-tox-docker-${{ matrix.tox_system_factor }}-${{ matrix.tox_packages_factor }}
DOCKER_TARGETS: configured with-targets with-targets-optional
steps:
- name: Check out SageMath
uses: actions/checkout@v2
with:
repository: ${{ env.SAGE_REPO }}
ref: ${{ env.SAGE_REF }}
fetch-depth: 2000
if: env.SAGE_REPO != ''
- name: Check out git-trac-command
uses: actions/checkout@v2
with:
repository: sagemath/git-trac-command
path: git-trac-command
if: env.SAGE_TRAC_GIT != ''
- name: Check out SageMath from trac.sagemath.org
shell: bash {0}
run: |
git config --global user.email "[email protected]"
git config --global user.name "ci-sage workflow"
if [ ! -d .git ]; then git init; fi; git remote add trac ${{ env.SAGE_TRAC_GIT }} && x=1 && while [ $x -le 5 ]; do x=$(( $x + 1 )); sleep $(( $RANDOM % 60 + 1 )); if git-trac-command/git-trac fetch $SAGE_TICKET; then git merge FETCH_HEAD || echo "(ignored)"; exit 0; fi; sleep 40; done; exit 1
if: env.SAGE_TRAC_GIT != ''
- uses: actions/download-artifact@v2
with:
path: upstream
name: upstream
- name: Install test prerequisites
run: |
sudo DEBIAN_FRONTEND=noninteractive apt-get update
sudo DEBIAN_FRONTEND=noninteractive apt-get install tox python3-setuptools
- name: Update Sage packages from upstream artifact
run: |
(export PATH=$(pwd)/build/bin:$PATH; (cd upstream && bash -x update-pkgs.sh) && sed -i.bak '/upstream/d' .dockerignore && echo "/:toolchain:/i ADD upstream upstream" | sed -i.bak -f - build/bin/write-dockerfile.sh && git diff)
- name: Configure and build Sage distribution within a Docker container
run: |
set -o pipefail; EXTRA_DOCKER_BUILD_ARGS="--build-arg USE_MAKEFLAGS=\"-k V=0 SAGE_NUM_THREADS=3\"" tox -e $TOX_ENV -- $TARGETS 2>&1 | sed "/^configure: notice:/s|^|::warning file=artifacts/$LOGS_ARTIFACT_NAME/config.log::|;/^configure: warning:/s|^|::warning file=artifacts/$LOGS_ARTIFACT_NAME/config.log::|;/^configure: error:/s|^|::error file=artifacts/$LOGS_ARTIFACT_NAME/config.log::|;"
- name: Copy logs from the Docker image or build container
run: |
mkdir -p "artifacts/$LOGS_ARTIFACT_NAME"
cp -r .tox/$TOX_ENV/Dockerfile .tox/$TOX_ENV/log "artifacts/$LOGS_ARTIFACT_NAME"
if [ -f .tox/$TOX_ENV/Dockertags ]; then CONTAINERS=$(docker create $(tail -1 .tox/$TOX_ENV/Dockertags) /bin/bash || true); fi
if [ -n "$CONTAINERS" ]; then for CONTAINER in $CONTAINERS; do for ARTIFACT in /sage/logs; do docker cp $CONTAINER:$ARTIFACT artifacts/$LOGS_ARTIFACT_NAME && HAVE_LOG=1; done; if [ -n "$HAVE_LOG" ]; then break; fi; done; fi
if: always()
- uses: actions/upload-artifact@v2
with:
path: artifacts
name: ${{ env.LOGS_ARTIFACT_NAME }}
if: always()
- name: Print out logs for immediate inspection
# and markup the output with GitHub Actions logging commands
run: |
.github/workflows/scan-logs.sh "artifacts/$LOGS_ARTIFACT_NAME"
if: always()
- name: Push Docker images
run: |
if [ -f .tox/$TOX_ENV/Dockertags ]; then
TOKEN="${{ secrets.DOCKER_PKG_GITHUB_TOKEN }}"
if [ -z "$TOKEN" ]; then
TOKEN="${{ secrets.GITHUB_TOKEN }}"
fi
echo "$TOKEN" | docker login docker.pkg.github.com -u ${{ github.actor }} --password-stdin
for a in $(cat .tox/$TOX_ENV/Dockertags); do
FULL_TAG=docker.pkg.github.com/$(echo ${{ github.repository }}|tr 'A-Z' 'a-z')/$a
docker tag $a $FULL_TAG
echo Pushing $FULL_TAG
docker push $FULL_TAG
done || echo "(Ignoring errors)"
fi
if: always()

macos:

runs-on: macos-latest
strategy:
fail-fast: false
max-parallel: 4
matrix:
tox_system_factor: [homebrew-macos, conda-forge-macos]
tox_packages_factor: [minimal, standard]
# As of 2021-03, default xcode is 12.4
# https://github.com/actions/virtual-environments/blob/main/images/macos/macos-10.15-Readme.md#xcode
xcode_version_factor: [default]
os: [ macos-10.15, macos-11.0 ]
include:
# Test xcode 11.7 only on macos-10.15
- tox_system_factor: homebrew-macos
tox_packages_factor: minimal
xcode_version_factor: 11.7
os: macos-10.15
- tox_system_factor: homebrew-macos
tox_packages_factor: standard
xcode_version_factor: 11.7
os: macos-10.15
# python3_xcode is only accepted if enough packages are available from the system
# --> to test "minimal", we will need https://trac.sagemath.org/ticket/30949
- tox_system_factor: homebrew-macos-python3_xcode
tox_packages_factor: standard
xcode_version_factor: 11.7
os: macos-10.15
- tox_system_factor: homebrew-macos-python3_xcode
tox_packages_factor: standard
xcode_version_factor: default
os: macos-10.15
- tox_system_factor: homebrew-macos-python3_xcode
tox_packages_factor: standard
xcode_version_factor: default
os: macos-11.0

uses: sagemath/sage/.github/workflows/macos.yml@develop
with:
osversion_xcodeversion_toxenv_tuples: >-
[["latest", "", "homebrew-macos-usrlocal-minimal"],
["latest", "", "homebrew-macos-usrlocal-standard"],
["13", "xcode_15.0", "homebrew-macos-usrlocal-standard"]]
targets: SAGE_CHECK=no SAGE_CHECK_givaro=yes SAGE_CHECK_fflas_ffpack=yes SAGE_CHECK_linbox=yes givaro fflas_ffpack linbox
sage_repo: sagemath/sage
sage_ref: develop
upstream_artifact: upstream
needs: [dist]

env:
TOX_ENV: local-${{ matrix.tox_system_factor }}-${{ matrix.tox_packages_factor }}
LOGS_ARTIFACT_NAME: logs-commit-${{ github.sha }}-tox-local-${{ matrix.tox_system_factor }}-${{ matrix.tox_packages_factor }}-${{ matrix.os }}-xcode_${{ matrix.xcode_version_factor }}
DOCKER_TARGETS: configured with-targets with-targets-optional

steps:

- name: Select Xcode version
run: |
if [ ${{ matrix.xcode_version_factor }} != default ]; then sudo xcode-select -s /Applications/Xcode_${{ matrix.xcode_version_factor }}.app; fi
- name: Check out SageMath
uses: actions/checkout@v2
with:
repository: ${{ env.SAGE_REPO }}
ref: ${{ env.SAGE_REF }}
fetch-depth: 2000
if: env.SAGE_REPO != ''
- name: Check out git-trac-command
uses: actions/checkout@v2
with:
repository: sagemath/git-trac-command
path: git-trac-command
if: env.SAGE_TRAC_GIT != ''
- name: Check out SageMath from trac.sagemath.org
shell: bash {0}
run: |
git config --global user.email "[email protected]"
git config --global user.name "ci-sage workflow"
if [ ! -d .git ]; then git init; fi; git remote add trac ${{ env.SAGE_TRAC_GIT }} && x=1 && while [ $x -le 5 ]; do x=$(( $x + 1 )); sleep $(( $RANDOM % 60 + 1 )); if git-trac-command/git-trac fetch $SAGE_TICKET; then git merge FETCH_HEAD || echo "(ignored)"; exit 0; fi; sleep 40; done; exit 1
if: env.SAGE_TRAC_GIT != ''
- uses: actions/download-artifact@v2
with:
path: upstream
name: upstream
- name: Update Sage packages from upstream artifact
run: |
(export PATH=$(pwd)/build/bin:$PATH; (cd upstream && bash -x update-pkgs.sh) && git diff)

- name: Install test prerequisites
run: |
brew install tox
- name: Build and test with tox
# We use a high parallelization on purpose in order to catch possible parallelization bugs in the build scripts.
# For doctesting, we use a lower parallelization to avoid timeouts.
run: |
MAKE="make -j12" tox -e $TOX_ENV -- SAGE_NUM_THREADS=4 $TARGETS
- name: Prepare logs artifact
run: |
mkdir -p "artifacts/$LOGS_ARTIFACT_NAME"; cp -r .tox/*/log "artifacts/$LOGS_ARTIFACT_NAME"
if: always()
- uses: actions/upload-artifact@v1
with:
path: artifacts
name: ${{ env.LOGS_ARTIFACT_NAME }}
if: always()
- name: Print out logs for immediate inspection
# and markup the output with GitHub Actions logging commands
run: |
.github/workflows/scan-logs.sh "artifacts/$LOGS_ARTIFACT_NAME"
if: always()
Loading