Implement automated weekly smoke tests with GitHub Actions

[andylizf]

Fixes #4112

Implement an automated weekly smoke test run using GitHub Actions leveraging the existing tests/test_smoke.py script.

• Add a new GitHub Actions workflow file .github/workflows/weekly-smoke-tests.yml.
• Schedule the workflow to run weekly using cron syntax.
• Set up the environment and install dependencies.
• Run pytest tests/test_smoke.py --terminate-on-failure.
• Handle different test groups (AWS, GCP, Azure, Lambda, Kubernetes) as defined in the script.
• Add necessary cloud credentials as GitHub secrets.
• Upload test results as artifacts.

[Michaelvll]

Thanks for adding this @andylizf! There are several things to investigate for this, including, how we reduce the cost of the smoke test, how to reduce the flakiness of the smoke test, whether we should use a subset of the smoke tests instead of the entire tests for efficiency and cost, and whether we should use a third-party tool, buildkite, than using Github action, as seems many OSS projects are using that.

https://github.com/ray-project/ray/tree/master/.buildkite
https://github.com/vllm-project/vllm/tree/main/.buildkite
https://buildkite.com/

[Michaelvll]

Also, I just canceled all the smoke test actions, as that can take a long time and we don't have the credentials setup (should we?). I guess a better way to do this is to list the alternatives and routes we will go with in a design doc before submitting the PR. : )

[andylizf]

Yep, I’ve already flagged those tricky issues in the related GitHub issue, including cost-cutting and reducing flakiness in the smoke tests. I’m going to try running it on GitHub Actions using my own GCP credentials first to see how that works out

[Michaelvll]

Sounds good! Please be careful with the leaked resources in cloud and cost for running in CI. : )