Consider supporting regular expressions

[springcomp]

There seems to be some demand for supporting regular expressions.

• regular expression support jmespath/go-jmespath#64
• https://github.com/jmespath/jmespath.py/issues/213

This presents some challenges as there are many different dialects of regexp, although there is an effort around
specifying an interoperable regex format.

Also, regular expressions uses cases fall into three main categories:

• Matching patterns, which could be useful in filter-expressions. It either matches, or it does not.
• Replacing substrings.
• Capturing and extracting substrings in named (or numbered) groups.

So a JEP would need to come up with intuitive syntax for those use cases that would be relevant.

The interoperable regex format proposal referred to above only supports the first scenario. So maybe that’s the only scenario that requires extending the JMESPath syntax.

The last two scenarios may be supported using dedicated functions.

[springcomp]

Tu use regular expressions in a filter-expression we would need:

• A new function regex_match defined like so:

boolean regex_match(string $text, regexp $regex)

regexp could be a string or could be a new datatype that could only be used in functions.

regexp          = …
function-arg =/ regexp

Example

Consider the following JSON document

{
	"assets": [
		{ "name": "first-name" },
		{ "name": "last-name" },
		{ "name": "other" }
	]
}

To select all assets whose name matches a given pattern, the following hypothetical JMESPath expression could be used:

assets[? regex_match(name, /\-name$/i) ]

This would evaluate to:

[ { "name": "first-name"}, {"name": "last-name"} ]

[springcomp]

To support regular expressions to replace substrings, we would need a new function:

string regex_replace(string $text, regexp $regex)

For instance, using the same example as the previous comment, the following expression:

assets.{raw: regex_replace(name, /\-name$/g, '')}

would evaluate to:

[
  { "raw": "first" },
  { "raw": "last" },
  { "raw": "other" }
]

[springcomp]

Finally, a function could return the named or numbered captures from a regex.

object regex_capture(string $text, regexp $regex)

Example:

The simplest usage could look like:

regex_capure('hello, world!', /[a-z]+/)

Which evaluates to a JSON object if the match is successful:

{ "0": "hello" }

Adding capturing groups would be supported like so:

regex_capure('hello, world!', /([a-z]+).*/)

Would return:

{
  "0": "hello, world!",
  "1": "hello"
}

Likewise, this expression:

regex_capture('hello, world!', /(?<first>[a-z]+).*?(?<second>[a-z]+).*$/)

Would evaluate to:

{
   "0": "hello, world!",
   "first": "hello",
   "second": "world"
}

This is currently not very useful, but in light of the probably upcoming function let, we could have expressions like so:

let( { re: regex_capture(foo, /^([^\-]+)?[a-z]+$/) }, &\<expression-type> )

[innovate-invent]

I like the idea of regular expression support, but think that it could be quite burdensome for some implementations. How would we handle varying RE specs (POSIX vs PCRE vs RE2) that are available for each language?

I think we should have an explicitly optional spec for the function interface to improve consistency between implementations, but leave it up to the implementers as to whether they want to implement it and what RE spec it would use.

[springcomp]

The more I think about this, the less I think we need a regexp type in the grammar. I think simple strings will be sufficient.

The reason why there is a regexp type in JavaScript is because it is an object and can be compiled. This is useful for regexp literals in the JavaScript source code. In JMESPath, we do not have a compilation step as the AST is interpreted in most implementations.

The following functions will be sufficient:

boolean regex_match(string $text, string $regex)
string regex_replace(string $text, string $regex)
object regex_capture(string $text, string $regex)

Where the $regex argument would be either implementation-specific, or using an interoperable subset of the regular expression language.

[innovate-invent]

I think for the spec we should require that error messages state that the function is either not implemented if that implementation does not support RE, or for any RE parsing error it must state the expected RE standard provided by that implementation. This makes it so that end users don't have to dig up what language their application is written in when trying to figure out if RE is supported and for what RE standard.

[springcomp]

I agree we may leave the door open to implementation-specific Regex dialects.

However, I think that, if specified, those functions should become mandatory for a standards compliant implementation to include those. I mean, an implementation can choose not to implement those functions, but they would not claim to be standards compliant.

[innovate-invent]

I think that would condemn all microarchitecture implementations to be painted with the broad "not compliant" brush. Some environments simply can't provide RE support.

[suntong]

Reiterating my view

different programming languages have different regular expression implementations

Good point, but how about providing a regular expressions function among the Built-in Functions.

The regular expressions will be Go specific, but that's much better than not having any regular expression support, or supporting them all, for jp at least.

the regular expression matching part has been fully implemented:
go-jsonfile/go-jmespath@d90b399

just its exposed name need to be changed.

[springcomp]

Agreed. We are sort of coming up to the same conclusion.

[Thaina]

I felt this should be fall into custom function/filter instead

Also there was also regex replace with custom function that would be great to supported

[springcomp]

Agreed, please see the regex_replace() function specified above.