Clean up bangs.

[Leftium]

I am working on cleaning up the bangs for my app. Sharing my work:

• Separated into four commits so the resulting diffs are easier to understand.
• Each commit message is a log of the changes that were made.
  • The commit messages may be pasted into https://multi-launch.leftium.com (click "Edit" first) to easily confirm the new and old URLs behave the same. (I spot tested several URLs; many bangs don't even work and should just be removed...)
• The script used to make the changes: clean-bangs.ts
• To simplify the diffs between changes, all URLs were trimmed and unescaped.
  • Some URLs had been escaped, then escaped again, and again...
  • I find the unescaped URLs easier to read, and they seem to work without problems.
  • However a final pass with encodeURI() may be desired?

The changes

Trim Spaces

• All properties

Unescape URLs

• Just URL .u
• Mainly to cut down noise and make other diffs more readable.

Strip extra Google CSE url params

• Just URL .u
• The other params seem to be just for tracking.

TidyURL

• Just URL .u
• https://www.npmjs.com/package/@protontech/tidy-url

[Leftium]

Found some cases where percent encoding was required for #. There may be other classes of this situation, where a previously encoded character incorrectly becomes interpreted as part of the URL syntax.

For example, / in the path or . in the domain?

[nobodywasishere]

There may be other classes of this situation

Ah yeah that's a good point, it's the reason we have the various fmt options, as some websites need spaces to be plusses, for example. Are you able to create a separate PR for removing the CSE params? May be easier to split this up into smaller PRs

[Leftium]

Are you able to create a separate PR for removing the CSE params? May be easier to split this up into smaller PRs

This single commit isolates the diffs for just removing the CSE parameters: e412799

But I don't think the PR's will be easier/smaller if broken up (see explanation at bottom.)

Also I think most of the review comments for this PR should actually be fixed outside this PR. Usually the issues already existed in the URLs before this PR; the PR just happened to highlight them. For example, the issues pointed out with the following bangs already existed before this PR was opened:

• !analyser
• !amvidde
• !antonimo
• !apiv

If we focus only on regressions caused by this PR, I think there will be almost no issues to fix. I tried to make it as simple as possible to verify the changes don't affect the behavior of the bangs. (If you actually test the bangs, you will notice many of the bangs are actually dead and should just be deleted; both before and after URLs don't generate any useful search results.)

Each individual commit message has a detailed log of what the script did, including triggers, before/after values and list of params removed. Also there are links that can be clicked so it is very quick to verify if the behavior changed or not.

(The commit messages are actually in TOML format that can be copy-pasted into https://multi-launch.leftium.com/ for even more convenient checking.)

---

My script is kind of able to do each step individually, unfortunately the URL() constructor and/or some libraries used tend to make tons of other modifications that cannot be prevented. So I don't think the PR's will be smaller if broken up:

• The bulk of the diffs happened in the commit that unescapes the URLs. (945def2)
• However, if that step is skipped, you get a mess of unencoding/encoding diffs in the following steps. (Also I believe already percent encoded strings are being encoded again, and again each time.)
• I split the PR into multiple, focused commits. Each commit has a detailed log of the script run, and the script log output (before/ after versions, list of params removed.)
  1. First, I started with --trim and --unescape operations to minimize noise in the following commits.
     • Otherwise there are so many other changes it is difficult to see what actually changed.
  2. Then the 3rd commit removed the Google CSE params.
  3. The 4th commit used TidyUrl to remove some other common unneeded params.
  4. The following commits were manually applying review comments.

At the same time I have been applying manual changes, I have also been updating the script to fix the problems I noticed after opening the PR. For example, the script now preserves %-encoding for # and & characters. (To fix the regression I noticed.)

If I start over with a new PR (and leave out the TidyURL step) it might be a little cleaner, but we would lose many of the manual edits.

[nobodywasishere]

If you actually test the bangs, you will notice many of the bangs are actually dead and should just be deleted

This is something I'm currently working on, though it takes time (see this commit). If you come across a dead bang, don't be afraid to delete it. I appreciate the work you're putting into this.

[Leftium]

This is something I'm currently working on, though it takes time (see this commit). If you come across a dead bang, don't be afraid to delete it.

Thank you for your effort~ it is the main reason I decided to base my app's "canonical" bangs on Kagi bangs (even though DuckDuckGo invented the format).

Removing dead bangs is certainly the next step. First, I wanted to clean up the bangs to make reasoning about them easier.

I was hoping to cut down the size of the file by either focusing on updating many bangs at the same time via automation (this PR) and/or focusing on the largest bangs (#123). But both efforts barely put a dent in the (gzipped) file size 🥲

(Gzipped text is very hard to reduce. Even though I was able to reduce the bang.json file by about 50% through various means, it only resulted in a gzipped reduction of about 10-15%)

[nobodywasishere]

How big is the gzipped file size currently (and how big is ideal for your use-case?)

[Leftium]

How big is the gzipped file size currently (and how big is ideal for your use-case?)

I reduced the bangs.json file to 1.41 MB (394 KB gzipped) from 2.59 MB (442 KB).

The original 2.59 MB size of (Kagi's) bang.json file seemed really big at first, but I guess these days even 442 KB gzipped isn't that large. And most times this file will be cached.

Smaller is always better, but I guess this could just be premature optimization...

[nobodywasishere]

Ah my apologies if you pushed commits fixing the merge conflicts

[nobodywasishere]

VS Code for me is formatting fmt over multiple lines (I'm not using prettier but w/e VS Code has built-in). Is it okay if we stick with that over a single line format?

[Leftium]

VS Code for me is formatting fmt over multiple lines (I'm not using prettier but w/e VS Code has built-in). Is it okay if we stick with that over a single line format?

I have no preference. I should have probably respected this repo's formatting. VS Code just kept automatically saving like that whenever I saved the file. I was expecting a lot of review comments, and I got tired of manually removing all the formatting that was automatically added. (I think I figured out how to prevent the auto-formatting in the future...)

It would be good if there was a way for this project to specify the formatting settings so it would be the same for everyone (in VS Code). I've worked on other repos that enforce their own formatting rules, but I'm not quite sure how they did it...

[Leftium]

How big is the gzipped file size currently (and how big is ideal for your use-case?)

I reduced the bangs.json file to 1.41 MB (394 KB gzipped) from 2.59 MB (442 KB).

@nobodywasishere BTW the 1.41 MB (394 KB gzipped) size doesn't refer to this PR. Most of that reduction comes from deduplicating all the bangs that serve the same URL. Which is not possible with the current format.

This PR only results in a modest file size reduction.

[nobodywasishere]

VS Code just kept automatically saving like that whenever I saved the file.

No worries, same thing happens to me all the time haha.

I was expecting a lot of review comments, and I got tired of manually removing all the formatting

Completely fair, I should've done one big review instead of a bunch of individual comments, so you could do them all at once.

It would be good if there was a way for this project to specify the formatting settings so it would be the same for everyone

I agree, I thought maybe .editorconfig may cover it but I was wrong. I'll look into a solution

BTW the 1.41 MB (394 KB gzipped) size doesn't refer to this PR.

Okay no worries. I went ahead and reduced the size of some of the larger bangs (which won't help much but should help a little).

[nobodywasishere]

I'm probably going to merge this PR soon as everything looks good, in case you wanted to add anything else to this. Thank you!

[Leftium]

I tried running the latest bangs.json from this PR through my cleanup script again, and it unescaped a few more characters:

Any reason those need to be escaped? Should I add them to this PR?

Also my script only looks at the .u field when unescaping, but it might be good to unescape the .ad field as well?