Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Drop dependency on bash-parser #72

Closed
mcmxcdev opened this issue Mar 3, 2023 · 15 comments
Closed

Drop dependency on bash-parser #72

mcmxcdev opened this issue Mar 3, 2023 · 15 comments
Labels
discussion Discussion

Comments

@mcmxcdev
Copy link

mcmxcdev commented Mar 3, 2023

We recently started using https://socket.dev/ as part of our dependencies maintenance and the tool has detected bash-parser as problematic due to being unmaintained for 6 years: https://socket.dev/npm/package/bash-parser

bash-parser itself also uses old and unmaintained dependencies, e.g. https://socket.dev/npm/package/deep-freeze which is a repo that has a non-existent npm account associated to it.

@webpro
Copy link
Collaborator

webpro commented Mar 3, 2023

It was difficult to even find such a package, let alone a recent or actively maintained one.

On the other hand, it provides functionality that makes Knip quite powerful in finding dependencies and entry files. This test shows quite some cases it can handle, and the scripts can come from package.json#scripts, and the GitHub Action, Husky, Lefthook, lint-staged, Nx and release-it plugins. This is exactly the extra mile Knip likes to go.

I would also be happy to replace bash-parser with something more light-weight and well-maintained, but I'm not willing to consider losing this type of functionality. Perhaps it could be some combination of bash-parser becoming an optional dependency and opt-out by configuration or something.

Just for reference, I started out with a bunch of regexes, but that's far from elegant, limited and hard to maintain.

On the other hand, we're talking Bash here, syntax doesn't really change at all. It parses strings in configuration files. I wonder how can this package can actually be a threat or exploited? If it is I should perhaps add a warning to the docs.

Very open to ideas and suggestions here, I want Knip to be a good citizen in anyone's node_modules.

@webpro webpro added the discussion Discussion label Mar 3, 2023
@mcmxcdev
Copy link
Author

mcmxcdev commented Mar 6, 2023

Thanks for the detailed explanation!

Since knip is a dev dependency, I am not worried but I simply would like to get rid of unmaintained deps as much as possible. Reporting issues to multiple repositories to move the ecosystem forward.

No need to be actionable on this right away, let's keep it open for reference and maybe a better alternative will come around for the package or someone might have a recommendation!

@camsaul
Copy link

camsaul commented Mar 21, 2023

It's causing errors for me

$ knip
Resolving plugin configuration files...
/home/linuxbrew/.linuxbrew/lib/node_modules/knip/node_modules/bash-parser/src/index.js:52
                throw new Error(err.stack || err.message);
                      ^

Error: Error: Parse error on line 1: Unexpected 'LESS'
    at parse (/home/linuxbrew/.linuxbrew/lib/node_modules/knip/node_modules/bash-parser/src/index.js:52:9)
    at setCommandExpansion (/home/linuxbrew/.linuxbrew/lib/node_modules/knip/node_modules/bash-parser/src/modes/posix/rules/command-expansion.js:17:21)
    at /home/linuxbrew/.linuxbrew/lib/node_modules/knip/node_modules/bash-parser/src/modes/posix/rules/command-expansion.js:37:12
    at Array.map (<anonymous>)
    at /home/linuxbrew/.linuxbrew/lib/node_modules/knip/node_modules/bash-parser/src/modes/posix/rules/command-expansion.js:35:59
    at Object.next (/home/linuxbrew/.linuxbrew/lib/node_modules/knip/node_modules/map-iterable/index.js:35:18)
    at Object.next (/home/linuxbrew/.linuxbrew/lib/node_modules/knip/node_modules/iterable-lookahead/index.js:51:24)
    at Object.next (/home/linuxbrew/.linuxbrew/lib/node_modules/knip/node_modules/map-iterable/index.js:33:30)
    at Object.next (/home/linuxbrew/.linuxbrew/lib/node_modules/knip/node_modules/iterable-lookahead/index.js:51:24)
    at Object.next (/home/linuxbrew/.linuxbrew/lib/node_modules/knip/node_modules/map-iterable/index.js:33:30)
    at parse (/home/linuxbrew/.linuxbrew/lib/node_modules/knip/node_modules/bash-parser/src/index.js:52:9)
    at getBinariesFromScript (file:///home/linuxbrew/.linuxbrew/lib/node_modules/knip/dist/util/binaries/bash-parser.js:55:20)
    at file:///home/linuxbrew/.linuxbrew/lib/node_modules/knip/dist/util/binaries/index.js:16:28
    at Array.flatMap (<anonymous>)
    at getReferencesFromScripts (file:///home/linuxbrew/.linuxbrew/lib/node_modules/knip/dist/util/binaries/index.js:16:10)
    at Module.findGithubActionsDependencies (file:///home/linuxbrew/.linuxbrew/lib/node_modules/knip/dist/plugins/github-actions/index.js:15:38)
    at async WorkspaceWorker.findDependenciesByPlugins (file:///home/linuxbrew/.linuxbrew/lib/node_modules/knip/dist/workspace-worker.js:213:41)
    at async main (file:///home/linuxbrew/.linuxbrew/lib/node_modules/knip/dist/index.js:187:17)
    at async run (file:///home/linuxbrew/.linuxbrew/lib/node_modules/knip/dist/cli.js:27:46)
    at async file:///home/linuxbrew/.linuxbrew/lib/node_modules/knip/dist/cli.js:59:1

@webpro
Copy link
Collaborator

webpro commented Mar 21, 2023

Any chance you could look up the script that contains the "less than" (<) character? It should be in some GitHub Action .yml file afaics.

The issue could be in Knip as well, and at least it should try to work around it.

@benasher44
Copy link

For me it errored on an unclosed ' in a package.json script. Here's the value of the script entry:

NODE_ENV=test; NODE_OPTIONS=\"--max-old-space-size=3072\" node --expose-gc \"$(yarn bin jest || kill $$)\" --reporters=\"default\" --reporters=\"github-actions\"     --forceExit --logHeapUsage --maxWorkers=\"$(node -e 'process.stdout.write(os.cpus().length.toString())')\"

Removing this script entry fixed it for me (though different issue than the reported one above)

@webpro
Copy link
Collaborator

webpro commented Mar 24, 2023

I fixed that it throws so hard on bash parser errors by swallowing them in v2.1.2. Errors still show up in --debug if debugging false positives.

@ericcornelissen
Copy link
Contributor

On a related note, bash-parser also brings in a transitive dependency that's technically protected by copyright as it lacks a license statement of any kind - see also vorpaljs/bash-parser#65. Even as a devDependency, this can make knip difficult to use in certain settings.

While not actively addressing the maintainance issue on which this issue is based, it might make sense to fork/vendor bash-parser and replace the problematic transitive dependency. What are your thoughts on this @webpro?

@webpro
Copy link
Collaborator

webpro commented Mar 28, 2023

Since the latest release of vorpaljs/bash-parser is 6 years ago I wouldn't mind depending on a fork.

Still on-topic, does anybody have an idea about possible alternatives?

@webpro
Copy link
Collaborator

webpro commented Jul 10, 2023

Thanks to @ericcornelissen, v2.15.1 should be less problematic in terms of dependencies and licenses. Thanks again, Eric.

Still thinking it could be a nice weekend project to let some model do the hard work here and build a modern alternative to bash parser.

@webpro
Copy link
Collaborator

webpro commented Jul 30, 2023

After seeing tree-sitter again the other day I tried using it in Knip with the Bash grammar to replace bash-parser. Turned out nicely and took little efforts.

Pros:

  • Works well on the existing test suite within Knip, not much logic in Knip needed (a lot less compared to bash-parser).
  • Seems widely used and plenty of activity in tree-sitter.

Cons:

  • Last publish of tree-sitter-bash is over two years ago (but recent fork: @curlconverter/tree-sitter-bash).
  • Handling of quotes is lacking (but seems manageable at first sight).
  • Not sure if pro/con, but not using tree cursor or query features, doing simple AST walking instead.

Although we may not like the age of bash-parser, it did a very decent job. So here I am asking for your opinion, security assessment, and hopefully a test run on your own project(s).

You can install v2.17.2-ts.0 with the ts tag using something like:

npm install -D knip@ts

@mcmxcdev
Copy link
Author

Looking at tree-sitter, its fork and bash-parser I am coming to the conclusion:

Probably your best bet would be to ask for a new release for tree-sitter-bash as there are somewhat frequent commits on the repo recently. That dependency pulls in way more dependencies on its own though than bash-parser ~(68 vs 28)

Forks don't make sense most of the time because they are rarely ever maintained over a longer period of time.

All 3 have CVEs, so really not a great choice here.

@webpro
Copy link
Collaborator

webpro commented Jul 31, 2023

A new release was requested in October 2022 in tree-sitter/tree-sitter-bash#134. Tried to nudge the owner now.

At this point I think tree-sitter is the best option.

@webpro webpro closed this as completed in ef3981f Aug 19, 2023
@webpro
Copy link
Collaborator

webpro commented Aug 19, 2023

🚀 This issue has been resolved in v2.19.6. See Release 2.19.6 for release notes.

@webpro
Copy link
Collaborator

webpro commented Aug 19, 2023

Since [email protected] was released the other day I've pulled the trigger here as well. Would deserve either a major bump or just a patch release. As I think it shouldn't cause friction anywhere I opted for the latter (I do expect an occasional issue around arguments w/ quotes here and there, let's see).

mournfully added a commit to mournfully/dotfiles that referenced this issue Aug 21, 2023
[Wrong parsing for `${!#}` · Issue #160 · tree-sitter/tree-sitter-bash](tree-sitter/tree-sitter-bash#160)
> Same problem here, with a similar variable replacement:
> `display="${display%|*}"`
> It makes the syntax broken for the rest of the file

[Fixes by amaanq · Pull Request #186 · tree-sitter/tree-sitter-bash](tree-sitter/tree-sitter-bash#186)

[Drop dependency on `bash-parser` · Issue #72 · webpro/knip](webpro-nl/knip#72)
> Since [email protected] was released the other day I've pulled the
> trigger here as well. Would deserve either a major bump or just a patch
> release. As I think it shouldn't cause friction anywhere I opted for the
> latter (I do expect an occasional issue around arguments w/ quotes here
> and there, let's see).
>
> 🚀 This issue has been resolved in v2.19.6. See Release 2.19.6 for
> release notes.
webpro added a commit that referenced this issue Aug 23, 2023
@webpro
Copy link
Collaborator

webpro commented Aug 23, 2023

🚀 This change has been reverted in v2.19.11. See Release 2.19.11 for release notes.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
discussion Discussion
Projects
None yet
Development

No branches or pull requests

5 participants