[question] should "printf -v%s x" print "-vx" ?

[avih]

This is more of a question to the Austin group, but it does apply to ksh93[u+m] as well, and I am hoping the experienced people here would be able to help.

The question at the title is an example case to demonstrate an issue, and the more general question is:

Is the form printf -- "$format" ... portable, in the sense than a conformant implementation is required to support it?

And even more generically:
Is a utility in the "Shell and Utilities" section required to accept -- as "end of options" (unless explicitly stated otherwise) even when the utility has no options (like printf), i.e. that the options section of that utility spec looks like:

OPTIONS:
    None.

So far, as far as I can tell and interpret the spec, the answer is that utilities should accept -- as "end of options" even when the utility has no options (unless stated otherwise).

From Base-definitions, "12.2 Utility Syntax Guidelines" (which include the specification for --):

Some of the standard utilities do not conform to all of these guidelines; in those cases, the OPTIONS sections describe the deviations.

Which I interpret as "The standard utilities shall conform to these guidelines, unless stated otherwise at the OPTIONS section".

And indeed, for some utilities (echo, test, :) it's stated explicitly, e.g. for test:

OPTIONS:
    The test utility shall not recognize the "--" argument in the manner
    specified by Guideline 10 in XBD 12.2 Utility Syntax Guidelines .
    In addition, when the utility name used is [ the utility does not
    conform to Guidelines 1 and 2.

    No options shall be supported.

The counter argument which was raised at the #ksh irc channel on libera - on whether printf is required to support -- is, and I'm quoting:

it does not matter that what the "Utility Syntax Guidelines" says; if it is not explicitly specified, it is not required

Which likely draws from the fact that the docs for utilities which do accept options typically (always?) state that it shall conform with the syntax guidelines, for instance for date:

OPTIONS
    The date utility shall conform to XBD 12.2 Utility Syntax Guidelines .

    The following option shall be supported:

    -u
        Perform operations .....

Some data points, without claiming it proves anything:

• Of the 24 variations of printf which I tested, all of them accept -- as "end of options". Most of the variants are builtin implementations of shells (bash, ksh93[u+m], zsh (eh...), dash, free/net bsd sh, busybox ash, Schily posix Bourne sh variants, yash, and few more) with and without posix compatibility options where available, but also system /usr/bin/printf on linux, illumos, and even 1981 BSD 2.11 on PDP11. Bare none, all support -- as "end of options".
• If a conformant printf is not required to accept -- as "end of options", then I think it must print -vx for printf -v%s x, at which case, of the 24 printf variations which I tested, only 4 are conformant with this specific input and indeed print -vx (netbsd sh, busybox ash, zsh, and gnu /bin/printf). All the others (including BSD 2.11) error on invalid option.
• With other standard utilities (which don't take options) it's less unanimous. For instance basename -- foo/bar prints bar on most systems, but on illumos it prints --. or echo | dd -- bs=1 works on linux and illumos, but on freebsd and openbsd it prints dd: unknown operand --.

So, are standard utilities which have "OPTIONS: None." required to support -- as "end of options"? or not?

Should printf?

(for printf specifically there's a general solution which doesn't work with other utilities: replace a leading - at the format with \055 which is its equivalent octal literal in ASCII, and don't use --. but preferably printf -- "$fmt"... is portable...)

[McDutchie]

Is the form printf -- "$format" ... portable, in the sense than a conformant implementation is required to support it?

And even more generically: Is a utility in the "Shell and Utilities" section required to accept -- as "end of options" (unless explicitly stated otherwise) even when the utility has no options (like printf), i.e. that the options section of that utility spec looks like:

OPTIONS:
    None.

The answer to the above is simple: yes. The case in the title is not portable and fails under POSIX rules.

The reason is that many implementations support options as an extension, and POSIX allows for this by requiring an option parsing pass whether or not there actually are options. The -v varname option to printf in bash is a good example (ksh 93u+m now also has that option). Other examples include the AST self-documentation options such as --man and --help, which printf also supports.

The counter argument which was raised at the #ksh irc channel on libera - on whether printf is required to support -- is, and I'm quoting:

it does not matter that what the "Utility Syntax Guidelines" says; if it is not explicitly specified, it is not required

This is clearly incorrect. The Utility Syntax Guidelines apply to all the utilities, except where explicitly specified otherwise.

Or, as POSIX puts it, in 12.2:

“The utilities in the Shell and Utilities volume of POSIX.1-2017 that claim conformance to these guidelines shall conform completely to these guidelines as if these guidelines contained the term "shall" instead of "should". On some implementations, the utilities accept usage in violation of these guidelines for backwards-compatibility as well as accepting the required form.”

Which brings me to a compile-time option I recently added to ksh and which is now in 93u+m/1.0.9, SHOPT_PRINTF_LEGACY. Please see 1522c8d for a full explanation.

I think that approach is much better than the one the illumos people took in their fork of ksh, which is to break things like printf --man completely.

[emanuele6]

The counter argument which was raised at the #ksh irc channel on libera - on whether printf is required to support -- is, and I'm quoting:

it does not matter that what the "Utility Syntax Guidelines" says; if it is not explicitly specified, it is not required
This is clearly incorrect. The Utility Syntax Guidelines apply to all the utilities, except where explicitly specified otherwise.

They said that the "Utility Syntax Guidelines" page claims that all utilities should follow them unless otherwise specified.
And I responded with that meaning that whatever they are reading from that page, they are misinterpreting it, since that is not the correct interpretation.
If a utility should support the guidelines it must be explicitly specified in its description

    "The xyz utility shall conform to XBD Utility Syntax Guidelines."
    "The xyz special built-in shall support XBD Utility Syntax Guidelines."

When I had the time to finally read what text they were referring to, I even commented that the text:

Some of the standard utilities do not conform to all of these guidelines; in those cases, the OPTIONS sections describe the deviations.

I commented that I think that obviously does not say that "All utilities conform to guidelines unless specified otherwise", but that "Not all utilities that conform to guidelines conform to all of them exactly, in which case the difference is specified"; for example, uncompress having a name longer than 9 characters, and set supporing - for -- and + options.

Utilities without the explicit "shall conform to XBD Utility Syntax Guidelines." note need not conform to them; all utilities without POSIX options will not have that note.

shift, return, trap, eval, etc do not need to conform, and are not required to support --, and indeed they don't have it in many shells.
unlink does not conform, and indeed, in NetBSD unlink -- file is an error that will print the usage Usage: unlink file.

o/
 emanuele6

[avih]

You are saying that if there is not a note in options that says the utility should support guidelines, it should support them anyway because it is not specified otherwise. That is the same thing except being explicit about the text "none".

Correct, but still I was only speaking in the context of those which have no options, but indeed the spec does not make that distinction.

Anyway, that is indeed still my interpretation of that text:

From Base-definitions, "12.2 Utility Syntax Guidelines" (which include the specification for --):

Some of the standard utilities do not conform to all of these guidelines; in those cases, the OPTIONS sections describe the deviations.

Which I interpret as "The standard utilities shall conform to these guidelines, unless stated otherwise at the OPTIONS section".

Am I misinterpreting that statement?

If yes, how should it be interpreted instead?

Specifically, we have a problem.

Utils which do take options always/mostly say "this shall follow XBD 12.2" (e.g. date)

Some utils which don't take options say "this shall not follow XBD 12.2" (echo, test)

And the rest ("OPTIONS: None") don't say either of those.

So at the very least in the 3rd case it's not obvios whether XBD 12.2 shall or shall not be followed.

To me, XBD 12.2, does apply to it, because so far I have not seen anything which makes exceptions beyond those stated at the quote above.

And specifically, I've not seen (yet) anything along the lines of "XBD 12.2 shall be followed only when the documentation states that it is followed).

On the contrary, my quote from above says exactly the opposite (again, as far as my interpretation goes) - it shall be followed unless stated otherwise.

The fact that in some utils it's mentioned explicitly that it should be followed, does not mean that it should not be followed when it's not mentioned. It's a simple logic.

I do agree it could use a clarification.

[emanuele6]

I have never claimed that printf -v%s x should print -vx, it is not reasonable to think passing an argument that is -something to a utility without the explicit guidelines note is portable. The solution is always to not pass a -something argument; either using - and a >file/<file redirection, rewriting it as ./-something or \055something or -something or using other techniques.

I could say the same thing about evidence mentioning again the text of the eval, trap rationales.

Again, the correct interpretation should be that if there is no note, support is not required. This is not speculation, and I used to also think all utilities should follow guidelines by default before being taught this is how it works some years ago.

[avih]

I think that is the difference between POSIX simply not requiring --, and POSIX outright disallowing implementations to support it.

Wel, I think this has a consistency problem.

We both agree that test etc is outright disallowing it.

But I don't think "not requiring --" can exist portably.

What does that mean? that implementations are allowed to choose whether to accept -- or to reject it?

How can that work portably when you have first operand which begins with a dash? It means exactly that there's no way to use it portably.

And it means that implementations cannot extend it with more options, in direct contradiction to the statement that can extend it with options (and remain conformant).

So I don't think "not requiring --" can be a thing which posix says.

Which leave us either with "required to support --" or "required to not support --".

And my interpretation of XBD 12.2 is that "OPTIONS: None" is required to support it, because it states the guidelines shall be followed unless stated otehrwise, and this "OPTIONS: None" is not an explicit exception.

[emanuele6]

We both agree that test etc is outright disallowing it.

But I don't think "not requiring --" can exist portably.

What does that mean? that implementations are allowed to choose whether to accept -- or to reject it?

Yes, portability concerns exist; yes, they can and do choose.

[avih]

What does that mean? that implementations are allowed to choose whether to accept -- or to reject it?

Yes, portability concerns exist; yes, they can and do choose.

They obviously do. I gave plenty examples myself, like dd and basename.

But I didn't say that implementations don't do that.

I said posix can't say such a thing and expect the utility to remain portable.

And because the whole goal is portability, it means that it can't say "not requiring --".

And therefore it must be saying "must accept -- termination" or "must reject -- as termination".

And we need to decide which of those it says when it documents "OPTIONS: None".

[avih]

And even more generically: Is a utility in the "Shell and Utilities" section required to accept -- as "end of options" (unless explicitly stated otherwise) even when the utility has no options (like printf)

The answer to the above is simple: yes

Thanks. Much appreciated.

The reason is that many implementations support options as an extension, and POSIX allows for this by requiring an option parsing pass whether or not there actually are options.

Indeed.

I also reached a similar conclusion before I posted but didn't want to make the post even bigger.

Base definitions, Conformance, 2.1.1.4:

The system may provide non-standard extensions. These are features not required by POSIX.1-2024 and may include, but are not limited to:
...
Additional options for standard utilities

So to support additional options by an implementation (while staying conformant), utilities, even without options, must accept -- so that it can be used portably in case some non-required options are added by other implementations (except where it's disallowed, like test).

This is somewhat off topic, but I think this poses an issue for the standard. Because, as demonstrated earlier, this does not seem a unanimous interpretation. E.g. open/free bsd reject dd -- ..., and illumos prints -- for basename -- foo/bar, and I'm sure there are many more similar cases.

I think it would be good if this:

Some of the standard utilities do not conform to all of these guidelines; in those cases, the OPTIONS sections describe the deviations.

Also had a clarification, maybe like

even when the utility is not requied to accept options, and specifically support -- as "end of options".

Are there existing discussions of this subject? Specifically that standard utilities without options are required to support -- as "end of options"?

I did try to search the Austin group bug tracker, but I didn't find anything related that I could identify.

[avih]

@McDutchie thanks for posting the question at the open group. I wanted to post it at the bug tracker but discovered my account did not automatically extend to the tracker... so I asked here.

To quote from XCU 1.4 ("Shells and Utilities", "Introduction") as replied:

OPTIONS
...
Default Behavior: When this section is listed as "None.", it means that the implementation need not support any options. Standard utilities that do not accept options, but that do accept operands, shall recognize "--" as a first argument to be discarded.

The requirement for recognizing "--" is because conforming applications need a way to shield their operands from any arbitrary options that the implementation may provide as an extension. For example, if the standard utility foo is listed as taking no options, and the application needed to give it a pathname with a leading , it could safely do it as:

foo -- -myfile

and avoid any problems with -m used as an extension.

So I think this settles it.

But I still think a clarification would be useful at XBD 12.2, or at least a reference to XCU 1.4, because clearly not all implementation follow it, despite it being very easy to implement.

[avih]

eval is a special built-in utility, which are documented as:

Some of the special built-ins are described as conforming to XBD 12.2 Utility Syntax Guidelines. For those that are not, the requirement in 1.4 Utility Description Defaults that "--" be recognized as a first argument to be discarded does not apply and a conforming application shall not use that argument.

I'm guessing that due to historic reasons it has to not recognize -- as end of options.

https://pubs.opengroup.org/onlinepubs/9699919799/utilities/V3_chap02.html#eval

POSIX 2024 version

[emanuele6]

Ok, so dash builtins are not über bugged for that reason; we will have to settle with complaining only that dash's kill is bugged when you use kill -9 -- "$pid" or kill -TERM -- "$pid" (-- only works if you use -s) because -num and -name are implemented as hacks (https://git.kernel.org/pub/scm/utils/dash/dash.git/tree/src/jobs.c#n282 at commit 247fab5b).

So let's clarify:

@McDutchie

shift, return, trap, eval, etc do not need to conform, and are not required to support --, and indeed they don't have it in many shells.

Yes, these are all required to support -- (in fact, to have an option parsing loop to support the full guidelines), and in fact they do just that, in the vast majority of shells. Shells that don't are simply not compliant in this aspect, which should be reported to their authors as bugs.

Those are NOT required to support --, and indeed they technically should not even accept it, but bash, ksh93, and some other shells reasonably do because of course they want to have --help and possibly extend them with other options, so we have to deal with this portability problem.
Which is the convention I knew about.

However it is true, that this XCU 1.4 note (that exists at least as far back as POSIX Issue 6), requires unlink, printf, basename, etc. to accept --; And that NetBSD's comment on the printf source code is wrong, and that at least their unlink is not conformant.

Interesting POSIX weirdness, thank you guys for investigating.

o/
 emanuele6

[McDutchie]

Yeah, I stand corrected on that -- special builtins aren't required to accept the -- unless explicitly specified.

[McDutchie]

Thanks to both of you for the discussion. We all learned something today, myself included :)

As for compatibility with pre-POSIX printf legacy syntax, hopefully SHOPT_PRINTF_LEGACY will make those who need it happy.

[avih]

I'm just happy to now know that printf (and others) are required to support --, because it would have been a portability nightmare otherwise.

Thanks again for posting that question in the right place ;)

[avih]

From Base-definitions, "12.2 Utility Syntax Guidelines" (which include the specification for --):

Some of the standard utilities do not conform to all of these guidelines; in those cases, the OPTIONS sections describe the deviations.

Which I interpret as "The standard utilities shall conform to these guidelines, unless stated otherwise at the OPTIONS section".

For future reference, this (my) interpretation is invalid.

It simply states: "if there are exceptions to the guidelines, they'll be listed at the OPTIONS section".

And it doesn't say anything about utils which don't list exceptions, such as "OPTIONS: None.", so it was not possible to deduce that "OPTIONS: None" requires to follow the guidelines.

The other arguments that without supporting "--" it would become non-portable and non-extensible are indeed valid, but only XCU 1.4 default OPTIONS states it explicitly, and was indeed the missing link.