format() function

[innovate-invent]

A number of string composition functions are needed to support common use cases. We could propose a new function for each case, but this pollutes the function list and potentially complex manipulation will become verbose.

I propose we roll all of string composition into a single function that accepts a format string with syntax that allows describing all of the composition features that might be required.

The syntax would be string format(object|array data, string formatstr)

Python already has a mature syntax for this and looking at what is offered in other languages, it appears to be the most feature rich. It would be best to borrow heavily from this syntax, while also tweaking it to fit JMESPath:

Format strings contain “replacement fields” surrounded by curly braces {}. Anything that is not contained in braces is considered literal text, which is copied unchanged to the output. If you need to include a brace character in the literal text, it can be escaped by doubling: {{ and }}.

The grammar for a replacement field is as follows:

replacement_field = "{" [arg_name] ["!" conversion] [":" format_spec] "}"
arg_name =  [identifier | digit+]
conversion =  "r" | "s" | "a"
format_spec =  <described in the next section>

In less formal terms, the replacement field can start with a field_name that specifies the object whose value is to be formatted and inserted into the output instead of the replacement field. The field_name is optionally followed by a conversion field, which is preceded by an exclamation point '!', and a format_spec, which is preceded by a colon ':'. These specify a non-default format for the replacement value.

The field_name itself begins with an arg_name that is either a number or a keyword. If it’s a number, it refers to a positional argument, and if it’s a keyword, it refers to a named keyword argument. If the numerical arg_names in a format string are 0, 1, 2, … in sequence, they can all be omitted (not just some) and the numbers 0, 1, 2, … will be automatically inserted in that order. Because arg_name is not quote-delimited, it is not possible to specify arbitrary dictionary keys (e.g., the strings '10' or ':-]') within a format string.

format_spec

“Format specifications” are used within replacement fields contained within a format string to define how individual values are presented. Each formattable type may define how the format specification is to be interpreted.

A general convention is that an empty format specification produces the same result as if you had called to_string() on the value. A non-empty format specification typically modifies the result.

The general form of a standard format specifier is:

format_spec :=  [[fill]align][sign][#][0][width][grouping_option][.precision][type]
fill =  <any character>
align =  "<" | ">" | "=" | "^"
sign =  "+" | "-" | " "
width =  digit+
grouping_option =  "_" | ","
precision =  digit+
type =  "b" | "c" | "d" | "e" | "E" | "f" | "F" | "g" | "G" | "n" | "o" | "s" | "x" | "X" | "%"

If a valid align value is specified, it can be preceded by a fill character that can be any character and defaults to a space if omitted. It is not possible to use a literal curly brace (“{” or “}”) as the fill character. However, it is possible to insert a curly brace with a nested replacement field.

The meaning of the various alignment options is as follows:

Option	Meaning
'<'	Forces the field to be left-aligned within the available space (this is the default for most objects).
'>'	Forces the field to be right-aligned within the available space (this is the default for numbers).
'='	Forces the padding to be placed after the sign (if any) but before the digits. This is used for printing fields in the form ‘+000000120’. This alignment option is only valid for numeric types. It becomes the default for numbers when ‘0’ immediately precedes the field width.
'^'	Forces the field to be centered within the available space.

Note that unless a minimum field width is defined, the field width will always be the same size as the data to fill it, so that the alignment option has no meaning in this case.

The sign option is only valid for number types, and can be one of the following:

Option	Meaning
'+'	indicates that a sign should be used for both positive as well as negative numbers.
'-'	indicates that a sign should be used only for negative numbers (this is the default behavior).
space	indicates that a leading space should be used on positive numbers, and a minus sign on negative numbers.

The '#' option causes the “alternate form” to be used for the conversion. The alternate form is defined differently for different types. This option is only valid for integer, float and complex types. For integers, when binary, octal, or hexadecimal output is used, this option adds the respective prefix '0b', '0o', '0x', or '0X' to the output value. For float and complex the alternate form causes the result of the conversion to always contain a decimal-point character, even if no digits follow it. Normally, a decimal-point character appears in the result of these conversions only if a digit follows it. In addition, for 'g' and 'G' conversions, trailing zeros are not removed from the result.

The ',' option signals the use of a comma for a thousands separator. For a locale aware separator, use the 'n' integer presentation type instead.

The '_' option signals the use of an underscore for a thousands separator for floating point presentation types and for integer presentation type 'd'. For integer presentation types 'b', 'o', 'x', and 'X', underscores will be inserted every 4 digits. For other presentation types, specifying this option is an error.

width is a decimal integer defining the minimum total field width, including any prefixes, separators, and other formatting characters. If not specified, then the field width will be determined by the content.

When no explicit alignment is given, preceding the width field by a zero ('0') character enables sign-aware zero-padding for numeric types. This is equivalent to a fill character of '0' with an alignment type of '='.

The precision is a decimal integer indicating how many digits should be displayed after the decimal point for presentation types 'f' and 'F', or before and after the decimal point for presentation types 'g' or 'G'. For string presentation types the field indicates the maximum field size - in other words, how many characters will be used from the field content. The precision is not allowed for integer presentation types.

Finally, the type determines how the data should be presented.

The available string presentation types are:

Type	Meaning
's'	String format. This is the default type for strings and may be omitted.
'U'	Uppercase
'l'	Lowercase
'T'	Title Case
'c'	snake_case
None	The same as 's'.

The available number presentation types are:

Type	Meaning
'b'	Binary format. Outputs the number in base 2.
'c'	Character. Converts the integer to the corresponding unicode character before printing.
'd'	Decimal Integer. Outputs the number in base 10.
'o'	Octal format. Outputs the number in base 8.
'x'	Hex format. Outputs the number in base 16, using lower-case letters for the digits above 9.
'X'	Hex format. Outputs the number in base 16, using upper-case letters for the digits above 9. In case '#' is specified, the prefix '0x' will be upper-cased to '0X' as well.
'n'	Number. This is the same as 'd', except that it uses the current locale setting to insert the appropriate number separator characters.
'e'	Scientific notation. For a given precision p, formats the number in scientific notation with the letter ‘e’ separating the coefficient from the exponent. The coefficient has one digit before and p digits after the decimal point, for a total of p + 1 significant digits. With no precision given, uses a precision of 6 digits after the decimal point for float, and shows all coefficient digits for Decimal. If no digits follow the decimal point, the decimal point is also removed unless the # option is used.
'E'	Scientific notation. Same as 'e' except it uses an upper case ‘E’ as the separator character.
'f'	Fixed-point notation. For a given precision p, formats the number as a decimal number with exactly p digits following the decimal point. With no precision given, uses a precision of 6 digits after the decimal point for float, and uses a precision large enough to show all coefficient digits for Decimal. If no digits follow the decimal point, the decimal point is also removed unless the # option is used.
'F'	Fixed-point notation. Same as 'f', but converts nan to NAN and inf to INF.
'g'	General format. For a given precision p >= 1, this rounds the number to p significant digits and then formats the result in either fixed-point format or in scientific notation, depending on its magnitude. A precision of 0 is treated as equivalent to a precision of 1. The precise rules are as follows: suppose that the result formatted with presentation type 'e' and precision p-1 would have exponent exp. Then, if m <= exp < p, where m is -4 for floats and -6 for Decimals, the number is formatted with presentation type 'f' and precision p-1-exp. Otherwise, the number is formatted with presentation type 'e' and precision p-1. In both cases insignificant trailing zeros are removed from the significand, and the decimal point is also removed if there are no remaining digits following it, unless the '#' option is used. With no precision given, uses a precision of 6 significant digits for float. For Decimal, the coefficient of the result is formed from the coefficient digits of the value; scientific notation is used for values smaller than 1e-6 in absolute value and values where the place value of the least significant digit is larger than 1, and fixed-point notation is used otherwise. Positive and negative infinity, positive and negative zero, and nans, are formatted as inf, -inf, 0, -0 and nan respectively, regardless of the precision.
'G'	General format. Same as 'g' except switches to 'E' if the number gets too large. The representations of infinity and NaN are uppercased, too.
'n'	Number. This is the same as 'g', except that it uses the current locale setting to insert the appropriate number separator characters.
'%'	Percentage. Multiplies the number by 100 and displays in fixed ('f') format, followed by a percent sign.
None	For float this is the same as 'g', except that when fixed-point notation is used to format the result, it always includes at least one digit past the decimal point. The precision used is as large as needed to represent the given value faithfully.

The available boolean presentation types are:

Type	Meaning
'n'	"1" or "0"
't'	"true"/"false"
'T'	"T"/"F"
'g'	"True"/"False"
'G'	"TRUE"/"FALSE"
'y'	"yes"/"no"
'Y'	"Yes"/"No"

---

I am interested to know if anyone can think of any other presentation types that could be included?

[springcomp]

Thanks a lot for the detailed suggestion.

I personnaly think we should exclude "local aware" transformations capabilities from JMESPath.
As there would be no meaningful way to even write a reliable compliance test suite for thoses cases.

[innovate-invent]

I agree, I almost removed that but left it in as a conversation topic.

[innovate-invent]

What about including in the string presentation types base64 encode/decode, uri encode/decode, md5, and sha1/256?

[innovate-invent]

Looks like someone else agrees that the python format syntax is worth adopting: https://github.com/fmtlib/fmt#features
It might actually be good to approach this project and see if we can work with them to translate it to the various JMESPath languages. We can then import it rather than maintaining our own implementation.

[springcomp]

Does this conflict or could work hand in hand with a similar string-interpolation feature?
See #90.

[innovate-invent]

I am not sure we need two solutions, especially if they were to have different syntaxes.
I do have some reservations about #90 that I will add there.