Implement strict decoding for JetStream API requests #5858

caspervonb · 2024-09-03T18:35:03Z

This implements optional strict JSON decoding for JetStream.

The intent of this is to minimize accidental misalignments between server and clients, we've had numerous of these across the various clients, especially for rarely used fields in the request payloads.

Signed-off-by: Casper Beyer [email protected]

derekcollison · 2024-09-03T20:05:38Z

https://app.travis-ci.com/github/nats-io/nats-server/jobs/625767923

Needs a fix

neilalexander · 2024-09-05T08:25:28Z

server/jetstream_api.go

+ return err
+ }
+
+ return json.Unmarshal(msg, v)


Not convinced by this block, in that if strict mode is disabled, we can in effect unmarshal twice. (Unmarshalling JSON is actually quite expensive on CPU so need to be careful on hot paths.)

Why this vs just having one decoder and setting DisallowUnknownFields() based on the Strict setting?

My first pass did just that, but was discussed offline with @ripienaar that we want to log marshaling failures even if strict is set to false to have a softer error path but keep the current behavior for existing deployments where it would at-least surface in the logs.

A misbehaving client is going to a) spam log lines and b) unmarshal twice for each request. I don't think that's wise, needs further discussion.

Derek suggested that for 2.11 while we are in soft mode we should still report the problem so client authors can discover and remediate issues so he suggested this approach. It's nice a nice assist before we enable this by default to reject requests but I agree with you it will spam logs until things are fixed.

I think client authors could enable this as an opt-in too, in order to avoid spamming there is rateLimitFormatWarnf which would log less

Yeah we discussed maybe a header on specific requests - but we have no api behaviours on headers so I am a bit reluctant.

Other option is something at the request level like we have action on consumer or pedantic now - big ripple effect on clients having to support it and ultimately this does not help users. The point is to identify bad clients that perhaps are not compatible or have bugs. So making it opt-in only for clients that support it is a no win

rateLimitFormatWarnf is a good idea though for sure

I think that any opt-in for the client misses the point that Derek had - it's about detecting problems with non-tier-1 and older clients, and those will not have such opt-in implemented.

Changed the log lines to use rate limited warnings.

wallyqs · 2024-09-05T10:43:31Z

server/jetstream_api.go

+ }
+
+ if err != nil {
+ s.Errorf("Invalid JetStream request '%s > %s': %s", acc, subject, err)


what is a sample line that gets reported here?

[10037] 2024/09/06 14:50:14.087465 [ERR] Invalid JetStream request '$G > $JS.API.STREAM.INFO.ORDERS': json: unknown field "stream_name"

Now with rate limit warnings instead of error it is:

[11545] 2024/09/06 15:10:33.449547 [WRN] Invalid JetStream request '$G > $JS.API.STREAM.INFO.ORDERS': json: unknown field "stream_name"

server/jetstream_api.go

caspervonb · 2024-09-06T14:27:52Z

There's one failing test, but this is failing on main as-well so I don't think that's related.

wallyqs · 2024-09-06T17:35:21Z

server/jetstream.go

@@ -453,6 +454,7 @@ func (s *Server) enableJetStream(cfg JetStreamConfig) error {
 s.Noticef("")
 }
 s.Noticef("---------------- JETSTREAM ----------------")
+ s.Noticef(" Strict: %t", cfg.Strict)


I think we can remove these from banner

I'd vote to keep it, it's a sanity notice that you got the configuration you expected?

Let's only print it if cfg.Strict == true.

server/jetstream_api.go

caspervonb · 2024-09-09T13:29:55Z

Ran the meta benchmarks on this as they're pretty request heavy:

JetStreamCreate/N=1,R=1,storage=Memory,C=12/resource=Stream-14        593.6µ ± ∞ ¹   618.6µ ± ∞ ¹       ~ (p=1.000 n=1) ²
JetStreamCreate/N=1,R=1,storage=Memory,C=12/resource=KVBucket-14      1.065m ± ∞ ¹   1.074m ± ∞ ¹       ~ (p=1.000 n=1) ²
JetStreamCreate/N=1,R=1,storage=Memory,C=12/resource=ObjStore-14      2.889m ± ∞ ¹   2.831m ± ∞ ¹       ~ (p=1.000 n=1) ²
JetStreamCreate/N=3,R=3,storage=Memory,C=12/resource=Stream-14        51.54m ± ∞ ¹   56.28m ± ∞ ¹       ~ (p=1.000 n=1) ²
JetStreamCreate/N=3,R=3,storage=Memory,C=12/resource=KVBucket-14      63.51m ± ∞ ¹   58.32m ± ∞ ¹       ~ (p=1.000 n=1) ²
JetStreamCreate/N=3,R=3,storage=Memory,C=12/resource=ObjStore-14      62.60m ± ∞ ¹   64.33m ± ∞ ¹       ~ (p=1.000 n=1) ²
JetStreamCreate/N=3,R=3,storage=File,C=12/resource=Stream-14          60.38m ± ∞ ¹   56.13m ± ∞ ¹       ~ (p=1.000 n=1) ²
JetStreamCreate/N=3,R=3,storage=File,C=12/resource=KVBucket-14        66.11m ± ∞ ¹   56.62m ± ∞ ¹       ~ (p=1.000 n=1) ²
JetStreamCreate/N=3,R=3,storage=File,C=12/resource=ObjStore-14        58.56m ± ∞ ¹   59.47m ± ∞ ¹       ~ (p=1.000 n=1) ²

neilalexander · 2024-09-10T11:15:45Z

server/opts.go

@@ -2317,6 +2318,9 @@ func parseJetStream(v any, opts *Options, errors *[]error, warnings *[]error) er
 for mk, mv := range vv {
 tk, mv = unwrapValue(mv, &lt)
 switch strings.ToLower(mk) {
+ case "strict":
+ opts.JetStreamStrict = true


Better to do a type assertion here for s, ok := vv.(bool), return an error if the type assert fails, and then set JetStreamStrict to the bool value. Otherwise setting strict: false in the config does the wrong thing.

Currently failures to unmarshal only give a vague error of "invalid JSON", this adds the original error to it as context. Before: ```` invalid JSON ```` After: ``` invalid JSON: invalid character '\"' after object key ``` Related to but **not** dependent on #5858 Signed-off-by: Casper Beyer <[email protected]>

neilalexander · 2024-09-13T11:52:34Z

Mind rebasing this & resolving conflicts please?

ripienaar · 2024-09-23T12:02:56Z

Please make sure the strict option value is shown in jsz output

neilalexander · 2024-09-27T15:04:23Z

server/jetstream.go

@@ -453,6 +454,7 @@ func (s *Server) enableJetStream(cfg JetStreamConfig) error {
 s.Noticef("")
 }
 s.Noticef("---------------- JETSTREAM ----------------")
+ s.Noticef(" Strict: %t", cfg.Strict)


Let's only print it if cfg.Strict == true.

server/jetstream_test.go

caspervonb force-pushed the optional-strict-json-decoding branch 3 times, most recently from 9beb4f6 to 6b84a9b Compare September 4, 2024 18:37

neilalexander reviewed Sep 5, 2024

View reviewed changes

wallyqs reviewed Sep 5, 2024

View reviewed changes

server/jetstream_api.go Show resolved Hide resolved

caspervonb mentioned this pull request Sep 6, 2024

Target server branch with strict mode enabled nats-io/nats.go#1712

Draft

caspervonb changed the title ~~[WIP] Implement strict decoding for JetStream API requests~~ Implement strict decoding for JetStream API requests Sep 6, 2024

caspervonb force-pushed the optional-strict-json-decoding branch 2 times, most recently from a9b2549 to c4418f4 Compare September 6, 2024 14:26

caspervonb marked this pull request as ready for review September 6, 2024 14:27

caspervonb requested a review from a team as a code owner September 6, 2024 14:27

caspervonb mentioned this pull request Sep 6, 2024

make all jetstream configurations strict on tests nats-io/nats.js#56

Draft

wallyqs reviewed Sep 6, 2024

View reviewed changes

server/jetstream_api.go Outdated Show resolved Hide resolved

caspervonb force-pushed the optional-strict-json-decoding branch from c4418f4 to 92bfdd6 Compare September 9, 2024 07:30

caspervonb mentioned this pull request Sep 9, 2024

Add error details to invalid JetStream JSON errors #5868

Merged

neilalexander self-requested a review September 10, 2024 10:21

neilalexander reviewed Sep 10, 2024

View reviewed changes

caspervonb force-pushed the optional-strict-json-decoding branch from 0f7aa53 to b0144fc Compare September 16, 2024 11:11

caspervonb requested review from neilalexander and Jarema September 17, 2024 07:26

Implement strict decoding for JetStream API requests

50f27e4

caspervonb force-pushed the optional-strict-json-decoding branch from b0144fc to 50f27e4 Compare September 23, 2024 11:47

Fetch option in server.Start as-well

b2553c0

neilalexander reviewed Sep 27, 2024

View reviewed changes

caspervonb added 3 commits October 15, 2024 18:11

Only print strict in header when enabled

7bc2273

Unmarshal to ApiResponse in test

4ec16f2

Merge branch 'main' into optional-strict-json-decoding

f50cf20

caspervonb requested a review from neilalexander October 16, 2024 19:46

Fixup merge

42abb6b

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Implement strict decoding for JetStream API requests #5858

Implement strict decoding for JetStream API requests #5858

caspervonb commented Sep 3, 2024

derekcollison commented Sep 3, 2024

neilalexander Sep 5, 2024

caspervonb Sep 5, 2024 •

edited

Loading

neilalexander Sep 5, 2024

ripienaar Sep 5, 2024

wallyqs Sep 5, 2024

ripienaar Sep 5, 2024

ripienaar Sep 5, 2024

Jarema Sep 5, 2024

caspervonb Sep 6, 2024 •

edited

Loading

wallyqs Sep 5, 2024

caspervonb Sep 6, 2024

caspervonb Sep 6, 2024

caspervonb commented Sep 6, 2024

wallyqs Sep 6, 2024

caspervonb Sep 9, 2024 •

edited

Loading

neilalexander Sep 27, 2024

caspervonb commented Sep 9, 2024

neilalexander Sep 10, 2024

neilalexander commented Sep 13, 2024

ripienaar commented Sep 23, 2024

neilalexander Sep 27, 2024

Implement strict decoding for JetStream API requests #5858

Are you sure you want to change the base?

Implement strict decoding for JetStream API requests #5858

Conversation

caspervonb commented Sep 3, 2024

derekcollison commented Sep 3, 2024

Choose a reason for hiding this comment

caspervonb Sep 5, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

caspervonb Sep 6, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

caspervonb commented Sep 6, 2024

Choose a reason for hiding this comment

caspervonb Sep 9, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

caspervonb commented Sep 9, 2024

Choose a reason for hiding this comment

neilalexander commented Sep 13, 2024

ripienaar commented Sep 23, 2024

Choose a reason for hiding this comment

caspervonb Sep 5, 2024 •

edited

Loading

caspervonb Sep 6, 2024 •

edited

Loading

caspervonb Sep 9, 2024 •

edited

Loading