Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[ShellScript] Add ZSH #4024

Open
wants to merge 113 commits into
base: master
Choose a base branch
from
Open

Conversation

deathaxe
Copy link
Collaborator

@deathaxe deathaxe commented Aug 5, 2024

Fixes #1358
Fixes #2635
Resolves #3826
Fixes #4033

This PR refactors Bash as foundation for adding Zsh.

Used syntax specifications:

Syntax follows officiel specifications as close as possible
to reliably support most language features.

Bash highlights

  • tokenize shell words according to official specifications
  • improve brace expansions (actually add support for them, everywhere)
  • improve glob patterns
  • improve parameter expansions
  • improve tilde expansions
  • add support for arithmetic highlighting shell words
  • add support for command highlighting shell words
  • drop generic built-in command specific contexts
  • add syntax based folding
  • improve indentation rules
  • improve symbol definitions
  • add completions for keywords, built-in commands and variables
  • ...

Various scopes of keywords, punctuation, etc. have been modified
to follow latest scope naming best practices.

Those may cause syntax tests of 3rd-party packages to fail.
Syntax highlighting itself should however still work.

Zsh highlights

  • support for ZSH specific alternative control structures
  • support for expansion flags
  • support glob modifiers

Benchmarks

Parsing performance is evaluated using syntax test files from current release
and various other real world scripts.

This PR does not affect parsing performance.

This commit...

1. adds missing -f and -p options for wait built-in
2. adds dedicated tests for job built-in commands
3. refactors job variable tests by replacing `fg` command by `:` dummy
   to align them with majority of other variable tests
This commit fixes support for brace expansions according to bash syntax specs.
Former `number-range` was just a special case for what brace expansions can do.
This commit ...

1. applies structural changes, required for ZSH.
2. fixes various parameter expansion bugs related with special parameters
   e.g. `${$}`, `${?}`, ...
3. fixes advanced tilde expansions such as `~name/` or `~+/`

In general it means to introduce contexts and rules to scope expansions
more precisely and detailed.
This commit removes quote-removal from function and alias identifiers
as it is not supported by Bash. Function identifiers must be unquoted
literals.
This commit makes it easier to extend contexts in inherited syntax definitions.
This commit scopes function bodies with a commonly used meta scope
in favor of Bash's "compound" command as preparation for other shells,
which follow closer to other programming languages.

It removes redirections from context stack as those are handled globally.
This commit applies required changes to support ZSH case statements,
which may use curly braced code blocks as in C, which didn't work well
with previous meta scope structure.
This commit...

1. refactors how word boundaries and quotes are treated.

   Motivation:

   A reliably distinction between tilde-expansions and logic operators
   is required in ZSH pattern matching to support the following:

     cmd ~/path/*~*.com
                 ^ operator
         ^ tilde expansion

2. assigns `string` scope to all suitable strings and patterns to enable
   color schemes to more clearly distinguish them from variables.

   Note: Aligns Bash with "Batch File" syntax.

3. refactors command argument list termination

   Replace clunky and messy "boilerplate" contexts with proper "cmd-args-end".

   Note: Aligns Bash with "Batch File" syntax.
Everything between options and end of command is a string, regardless quotation.
This commit therefore introduces special contexts for `echo` builtin.
This commit...

1. fixes [ command termination behavior
   - terminate at \n ; && or || as normal `test` builtin
2. prefers scoping < > as comparison operator as it is more likely to be valid
   than redirections. Note: May need further refinements
3. highlight unquoted ; illegal in [[ ... ]] compound test expressions
4. fixes word boundaries of strings withing test expressions
This commit scopes all pre-defined built-in variables and distinguishes them
from special variables such as `$0` or `$*`.

Fixes an issue with `$_` vs. `$$_` highlighting.

see: https://www.gnu.org/software/bash/manual/bash.html#Shell-Variables
Move up expressions and operators contexts.
This commit moves literals, strings and pattern matching contexts up,
right after heredocs.
This commit...

1. enables support for correctly scoping literal and arithmetic
   assignment values in `declare` commands and its friends.

2. fixes unsupported nested `(...)` in value sequences.
   The following is illegal `var=(foo (bar baz))`.

3. distinguishes option assignment values and variable assignment values.
   Only the latter supports value sequences wrapped in parens.
This commit...

1. drops leading `\b` from `numbers`'s patterns as proper global word boundary
   checks are available.
2. treat numbers as words, consuming everything until word boundary and scope
   it illegal if not valid part of a number.
1. `comments` is not required
2. `line-continuations` is already provided by `eol-pop`
Only redirections are allowed after compound commands, which is already
satisfied/handled via global statements context.
1. always clear parent scope
2. scope username `constant.other`
Use already existing `invalid.illegal.unexpected-token.shell`
Numbers in command arguments are mainly guesswork. Bash/Zsh treat them string,
but this syntax scopes valid looking numbers constant.number as it is what
users might mainly expect.
@michaelblyons
Copy link
Collaborator

@okdana @Antibioticss Test drive the ZSH, maybe?

This commit ...
1. adds contexts to scope pipe operators illegal within lists, groups and tests.
2. handles the tricky part to distinguish `<1-2>` range operators at the
   beginning or within expansions, patterns or groups from redirections and
   literal scoped or illegal pipe operators, while maintaining word boundaries.
@deathaxe deathaxe marked this pull request as ready for review September 8, 2024 14:20
michaelblyons
michaelblyons previously approved these changes Sep 22, 2024
deathaxe added a commit to deathaxe/Containerfile that referenced this pull request Sep 22, 2024
This commit fixes compatibility with some breaking changes introduced by
sublimehq/Packages#4024.

CAUTION: It is not forward or backward compatible!
michaelblyons
michaelblyons previously approved these changes Oct 12, 2024
@michaelblyons
Copy link
Collaborator

Okay, so there's a test failure from the Groovy changes, but otherwise still 👌🏼 from me.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
2 participants