Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Introduce shellcheck to lint shell scripts #15169

Open
wants to merge 6 commits into
base: master
Choose a base branch
from
+89 −55
Open

Introduce shellcheck to lint shell scripts #15169

Show file tree
Hide file tree
Changes from 3 commits
Commits
Show all changes
6 commits
Select commit Hold shift + click to select a range
076421e
shellcheck autofix
straight-shoota Oct 31, 2024
696b97f
shellcheck fixes
straight-shoota Oct 31, 2024
b5970e8
CI workflow
straight-shoota Oct 31, 2024
614183b
polish: Quote entire argument instead of just variable expansions
straight-shoota Nov 8, 2024
cb9047b
Fix `CI_NIX_SHELL_ARGS`
straight-shoota Nov 8, 2024
90a487f
Fix `changed_cr_files`
straight-shoota Nov 8, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
29 changes: 29 additions & 0 deletions .github/workflows/format.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,29 @@
on:
push:
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Might be worth defining paths here. Supposedly this check could then run very rarely, only when needed
https://docs.github.com/en/actions/writing-workflows/workflow-syntax-for-github-actions#onpushpull_requestpull_request_targetpathspaths-ignore

Copy link
Member Author

@straight-shoota straight-shoota Nov 8, 2024 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah, I was wondering about that as well. A bit of an obstacle is that there's no simple glob to catch all shell script. Some like bin/crystal have no extension. The GitHub action actually checks all files for a matching shebang.
Limiting paths would need to hard code all paths that don't identify as shell script by their extension. That means the check could miss out on new ones being added.
I suppose this is an acceptable limitation, though. I wouldn't expect too much movement in that regard.

On the other hand, this job finishes reallly quickly - especially compared to our other workflows - in just 7s so it's not too much overhead to run it on every commit.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah it's a fair point. Not a good tradeoff then

Copy link
Member Author

@straight-shoota straight-shoota Nov 8, 2024 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm actually not so sure. Either way seems quite fine. So maybe we can hear some other's comments on this.

It's also relevant to note that the discovery feature depends on the GitHub action and another comment is suggesting we might do without the action. Droping automatic path discovery would make that easier.

pull_request:

name: "Format checks"
permissions: {}

jobs:
shellcheck:
name: Shellcheck
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- name: Run ShellCheck
uses: ludeeus/action-shellcheck@master
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm not super happy with running an arbitrarily changing action with an arbitrarily changing version of shellcheck.
Using the shellcheck executable that's available out of the box in GHA or Ubuntu would be slightly better. Or ideally this would be a precise version of a GitHub release of shellcheck, with its version managed by https://docs.renovatebot.com/modules/datasource/github-releases/

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could be done in a follow-up, I suppose 🤔

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That's a good point. I just picked this up as it's an easy solution.
However, we can at least pin the version of the action.

I'm happy to use any other mechanism.

We could also simply use grep -l -E '^#!(/usr/bin/env bash|/bin/sh|/bin/bash)' $(git ls-files) to detect changed script files.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Integration with pre-commit would also be nice 🤷

with:
# shellcheck doesn't have good support for zsh, so we're ignoring this
ignore_paths: >-
*.zsh
severity: error

- name: Run ShellCheck Warnings
uses: ludeeus/action-shellcheck@master
with:
ignore_paths: >-
./etc/completion.zsh
./etc/completion.bash
./etc/win-ci/cygwin-build-iconv.sh
severity: warning
36 changes: 18 additions & 18 deletions bin/ci
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,14 +1,14 @@
#!/bin/sh

fail() {
echo "${@}" >&2
echo "${*}" >&2
exit 1
}

on_tag() {
if [ -n "$CURRENT_TAG" ]; then
echo "${@}"
eval "${@}"
echo "${*}"
eval "${*}"
return $?
else
return 0
Expand All @@ -20,7 +20,7 @@ fail_on_error() {

exit=$?
if [ "$exit" -ne "0" ]; then
fail "${@} exited with $exit"
fail "${*} exited with $exit"
fi

return 0
Expand Down Expand Up @@ -50,8 +50,8 @@ on_os() {
verify_environment

if [ "$TRAVIS_OS_NAME" = "$os" ]; then
echo "${@}"
eval "${@}"
echo "${*}"
eval "${*}"
return $?
else
return 0
Expand All @@ -62,30 +62,30 @@ on_os() {
}

on_linux() {
fail_on_error on_os "linux" "${@}"
fail_on_error on_os "linux" "${*}"
}

on_osx() {
fail_on_error on_os "osx" "${@}"
fail_on_error on_os "osx" "${*}"
}

on_nix_shell_eval() {
if [ -n "$CI_NIX_SHELL" ]; then
echo "${@}"
eval "${@}"
echo "${*}"
eval "${*}"
return $?
else
return 0
fi
}

on_nix_shell() {
fail_on_error on_nix_shell_eval "${@}"
fail_on_error on_nix_shell_eval "${*}"
}

on_github() {
if [ "$GITHUB_ACTIONS" = "true" ]; then
eval "${@}"
eval "${*}"
return $?
else
return 0
Expand Down Expand Up @@ -153,7 +153,7 @@ prepare_build() {

# Install a recent bash version for nix-shell.
# macos ships with an ancient one.
if [ `uname` = "Darwin" ]; then
if [ "$(uname)" = "Darwin" ]; then
on_nix_shell "brew install bash"
fi
# initialize nix environment
Expand All @@ -172,7 +172,7 @@ prepare_build() {

verify_version() {
# If building a tag, check it matches with file
FILE_VERSION=`cat ./src/VERSION`
FILE_VERSION=$(cat ./src/VERSION)

if [ "$FILE_VERSION" != "$CURRENT_TAG" ]
then
Expand Down Expand Up @@ -205,8 +205,8 @@ with_build_env() {

on_linux docker run \
--rm -t \
-u $(id -u) \
-v $PWD:/mnt \
-u "$(id -u)" \
-v "$PWD":/mnt \
-v /etc/passwd:/etc/passwd \
-v /etc/group:/etc/group \
-w /mnt \
Expand All @@ -222,7 +222,7 @@ with_build_env() {
CRYSTAL_CACHE_DIR="/tmp/crystal" \
/bin/sh -c "'$command'"

on_nix_shell nix-shell --pure $CI_NIX_SHELL_ARGS --run "'TZ=$TZ $command'"
on_nix_shell nix-shell --pure "$CI_NIX_SHELL_ARGS" --run "'TZ=$TZ $command'"
straight-shoota marked this conversation as resolved.
Show resolved Hide resolved

on_github echo "::endgroup::"
}
Expand Down Expand Up @@ -254,7 +254,7 @@ case $command in
prepare_build
;;
with_build_env)
target_command="${@}"
target_command="${*}"
with_build_env "$target_command"
;;
build)
Expand Down
2 changes: 1 addition & 1 deletion bin/crystal
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -175,7 +175,7 @@ fi
# CRYSTAL_PATH has all symlinks resolved. In order to avoid issues with duplicate file
# paths when the working directory is a symlink, we cd into the current directory
# with symlinks resolved as well (see https://github.com/crystal-lang/crystal/issues/12969).
cd "$(realpath "$(pwd)")"
cd "$(realpath "$(pwd)")" || exit
oprypin marked this conversation as resolved.
Show resolved Hide resolved

case "$(uname -s)" in
CYGWIN*|MSYS_NT*|MINGW32_NT*|MINGW64_NT*)
Expand Down
4 changes: 2 additions & 2 deletions scripts/git/pre-commit
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -22,7 +22,7 @@ changed_cr_files=$(git diff --cached --name-only --diff-filter=ACM | grep '\.cr$

if [ -x bin/crystal ]; then
# use bin/crystal wrapper when available to run local compiler build
exec bin/crystal tool format --check $changed_cr_files >&2
exec bin/crystal tool format --check "$changed_cr_files" >&2
straight-shoota marked this conversation as resolved.
Show resolved Hide resolved
else
exec crystal tool format --check $changed_cr_files >&2
exec crystal tool format --check "$changed_cr_files" >&2
fi
4 changes: 2 additions & 2 deletions scripts/release-update.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -42,13 +42,13 @@ sed -i -E "s|crystal: \"[0-9.]+\"|crystal: \"$CRYSTAL_VERSION\"|g" .github/workf

# Edit shell.nix latestCrystalBinary using nix-prefetch-url --unpack <url>
darwin_url="https://github.com/crystal-lang/crystal/releases/download/$CRYSTAL_VERSION/crystal-$CRYSTAL_VERSION-1-darwin-universal.tar.gz"
darwin_sha=$(nix-prefetch-url --unpack $darwin_url)
darwin_sha=$(nix-prefetch-url --unpack "$darwin_url")

sed -i -E "s|https://github.com/crystal-lang/crystal/releases/download/[0-9.]+/crystal-[0-9.]+-[0-9]-darwin-universal.tar.gz|$darwin_url|" shell.nix
sed -i -E "/darwin-universal\.tar\.gz/ {n;s|sha256:[^\"]+|sha256:$darwin_sha|}" shell.nix

linux_url="https://github.com/crystal-lang/crystal/releases/download/$CRYSTAL_VERSION/crystal-$CRYSTAL_VERSION-1-linux-x86_64.tar.gz"
linux_sha=$(nix-prefetch-url --unpack $linux_url)
linux_sha=$(nix-prefetch-url --unpack "$linux_url")

sed -i -E "s|https://github.com/crystal-lang/crystal/releases/download/[0-9.]+/crystal-[0-9.]+-[0-9]-linux-x86_64.tar.gz|$linux_url|" shell.nix
sed -i -E "/linux-x86_64\.tar\.gz/ {n;s|sha256:[^\"]+|sha256:$linux_sha|}" shell.nix
10 changes: 5 additions & 5 deletions scripts/update-changelog.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -35,10 +35,10 @@ branch="changelog/$VERSION"
current_changelog="CHANGELOG.$VERSION.md"

echo "Generating $current_changelog..."
scripts/github-changelog.cr $VERSION > $current_changelog
scripts/github-changelog.cr "$VERSION" > "$current_changelog"

echo "Switching to branch $branch"
git switch $branch 2>/dev/null || git switch -c $branch;
git switch "$branch" 2>/dev/null || git switch -c "$branch";

# Write release version into src/VERSION
echo "${VERSION}" > src/VERSION
Expand All @@ -49,8 +49,8 @@ sed -i -E "s/version: .*/version: ${VERSION}/" shard.yml
git add shard.yml

# Write release date into src/SOURCE_DATE_EPOCH
release_date=$(head -n1 $current_changelog | grep -o -P '(?<=\()[^)]+')
echo "$(date --utc --date="${release_date}" +%s)" > src/SOURCE_DATE_EPOCH
release_date=$(head -n1 "$current_changelog" | grep -o -P '(?<=\()[^)]+')
date --utc --date="${release_date}" +%s > src/SOURCE_DATE_EPOCH
git add src/SOURCE_DATE_EPOCH

if grep --silent -E "^## \[$VERSION\]" CHANGELOG.md; then
Expand All @@ -70,7 +70,7 @@ else

git add CHANGELOG.md
git commit -m "Add changelog for $VERSION"
git push -u upstream $branch
git push -u upstream "$branch"

gh pr create --draft --base "$base_branch" \
--body "Preview: https://github.com/crystal-lang/crystal/blob/$branch/CHANGELOG.md" \
Expand Down
4 changes: 2 additions & 2 deletions scripts/update-distribution-scripts.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -35,14 +35,14 @@ branch="${2:-"ci/update-distribution-scripts"}"
git switch -C "$branch" master

old_reference=$(sed -n "/distribution-scripts-version:/{n;n;n;p}" .circleci/config.yml | grep -o -P '(?<=default: ")[^"]+')
echo $old_reference..$reference
echo "$old_reference".."$reference"

sed -i -E "/distribution-scripts-version:/{n;n;n;s/default: \".*\"/default: \"$reference\"/}" .circleci/config.yml

git add .circleci/config.yml

message="Updates \`distribution-scripts\` dependency to https://github.com/crystal-lang/distribution-scripts/commit/$reference"
log=$($GIT_DS log $old_reference..$reference --format="%s" | sed "s/.*(/crystal-lang\/distribution-scripts/;s/^/* /;s/)$//")
log=$($GIT_DS log "$old_reference".."$reference" --format="%s" | sed "s/.*(/crystal-lang\/distribution-scripts/;s/^/* /;s/)$//")
message=$(printf "%s\n\nThis includes the following changes:\n\n%s" "$message" "$log")

git commit -m "Update distribution-scripts" -m "$message"
Expand Down
14 changes: 7 additions & 7 deletions spec/debug/test.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -36,11 +36,11 @@ BUILD_DIR=$SCRIPT_ROOT/../../.build
crystal=${CRYSTAL_SPEC_COMPILER_BIN:-$SCRIPT_ROOT/../../bin/crystal}
debugger=${1:-lldb}
driver=$BUILD_DIR/debug_driver
mkdir -p $BUILD_DIR
"$crystal" build $SCRIPT_ROOT/driver.cr -o $driver
mkdir -p "$BUILD_DIR"
"$crystal" build "$SCRIPT_ROOT"/driver.cr -o "$driver"

$driver $SCRIPT_ROOT/top_level.cr $debugger
$driver $SCRIPT_ROOT/strings.cr $debugger
$driver $SCRIPT_ROOT/arrays.cr $debugger
$driver $SCRIPT_ROOT/blocks.cr $debugger
$driver $SCRIPT_ROOT/large_enums.cr $debugger
$driver "$SCRIPT_ROOT"/top_level.cr "$debugger"
$driver "$SCRIPT_ROOT"/strings.cr "$debugger"
$driver "$SCRIPT_ROOT"/arrays.cr "$debugger"
$driver "$SCRIPT_ROOT"/blocks.cr "$debugger"
$driver "$SCRIPT_ROOT"/large_enums.cr "$debugger"
6 changes: 3 additions & 3 deletions spec/generate_wasm32_spec.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -24,7 +24,7 @@ set +x

WORK_DIR=$(mktemp -d)
function cleanup {
rm -rf $WORK_DIR
rm -rf "$WORK_DIR"
}
trap cleanup EXIT

Expand All @@ -39,8 +39,8 @@ echo

for spec in $(find "spec/std" -type f -iname "*_spec.cr" | LC_ALL=C sort); do
require="require \"./${spec##spec/}\""
target=$WORK_DIR"/"$spec".wasm"
mkdir -p $(dirname "$target")
target="$WORK_DIR/$spec.wasm"
mkdir -p "$(dirname "$target")"

if ! output=$(bin/crystal build "$spec" -o "$target" --target wasm32-wasi 2>&1); then
if [[ "$output" =~ "execution of command failed" ]]; then
Expand Down
19 changes: 10 additions & 9 deletions spec/llvm-ir/test.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -9,31 +9,32 @@ SCRIPT_PATH="$(realpath "$0")"
SCRIPT_ROOT="$(dirname "$SCRIPT_PATH")"

BUILD_DIR=$SCRIPT_ROOT/../../.build
LLVM_CONFIG="$(basename $($SCRIPT_ROOT/../../src/llvm/ext/find-llvm-config))"
LLVM_CONFIG="$(basename "$("$SCRIPT_ROOT"/../../src/llvm/ext/find-llvm-config)")"
FILE_CHECK=FileCheck-"${LLVM_CONFIG#llvm-config-}"
crystal=${CRYSTAL_SPEC_COMPILER_BIN:-$SCRIPT_ROOT/../../bin/crystal}

mkdir -p $BUILD_DIR
mkdir -p "$BUILD_DIR"

function test() {
echo "test: $@"
echo "test: $*"

input_cr="$SCRIPT_ROOT/$1"
output_ll="$BUILD_DIR/${1%.cr}.ll"
compiler_options="$2"
# FIXME: unused variable
# compiler_options="$2"
check_prefix="${3+--check-prefix $3}"

# $BUILD_DIR/test-ir is never used
# pushd $BUILD_DIR + $output_ll is a workaround due to the fact that we can't control
# the filename generated by --emit=llvm-ir
"$crystal" build --single-module --no-color --emit=llvm-ir $2 -o $BUILD_DIR/test-ir $input_cr
$FILE_CHECK $input_cr --input-file $output_ll $check_prefix
"$crystal" build --single-module --no-color --emit=llvm-ir "$2" -o "$BUILD_DIR"/test-ir "$input_cr"
$FILE_CHECK "$input_cr" --input-file "$output_ll" "$check_prefix"

rm $BUILD_DIR/test-ir.o
rm $output_ll
rm "$BUILD_DIR"/test-ir.o
rm "$output_ll"
}

pushd $BUILD_DIR >/dev/null
pushd "$BUILD_DIR" >/dev/null

test argless-initialize-debug-loc.cr "--cross-compile --target x86_64-unknown-linux-gnu --prelude=empty"
test proc-call-debug-loc.cr "--cross-compile --target x86_64-unknown-linux-gnu --prelude=empty"
Expand Down
19 changes: 10 additions & 9 deletions src/llvm/ext/find-llvm-config
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,15 +2,16 @@

if ! LLVM_CONFIG=$(command -v "$LLVM_CONFIG"); then
llvm_config_version=$(llvm-config --version 2>/dev/null)
for version in $(cat "$(dirname $0)/llvm-versions.txt"); do
# shellcheck disable=SC2013
for version in $(cat "$(dirname "$0")/llvm-versions.txt"); do
LLVM_CONFIG=$(
([ "${llvm_config_version#$version}" != "$llvm_config_version" ] && command -v llvm-config) || \
command -v llvm-config-${version%.*} || \
command -v llvm-config-$version || \
command -v llvm-config${version%.*}${version#*.} || \
command -v llvm-config${version%.*} || \
command -v llvm-config$version || \
command -v llvm${version%.*}-config)
([ "${llvm_config_version#"$version"}" != "$llvm_config_version" ] && command -v llvm-config) || \
command -v llvm-config-"${version%.*}" || \
command -v llvm-config-"$version" || \
command -v llvm-config"${version%.*}""${version#*.}" || \
straight-shoota marked this conversation as resolved.
Show resolved Hide resolved
command -v llvm-config"${version%.*}" || \
command -v llvm-config"$version" || \
command -v llvm"${version%.*}"-config)
[ "$LLVM_CONFIG" ] && break
done
fi
Expand All @@ -26,6 +27,6 @@ if [ "$LLVM_CONFIG" ]; then
esac
else
printf "Error: Could not find location of llvm-config. Please specify path in environment variable LLVM_CONFIG.\n" >&2
printf "Supported LLVM versions: $(cat "$(dirname $0)/llvm-versions.txt" | sed 's/\.0//g')\n" >&2
printf "Supported LLVM versions: %s\n" "$(sed 's/\.0//g' "$(dirname "$0")/llvm-versions.txt")" >&2
exit 1
fi
Loading