Access context information in transformer

[jeengbe]

Hi!

Is there a way to read the current path/field index within the transform method of a Transformer?

The concrete use-case is for decoding gRPC update messages with respect to a field mask:

message UpdatePersonRequest {
  string id = 1;
  Person person = 2;
  google.protobuf.FieldMask update_mask = 3;
}

message Person {
  string the_name = 1;
  int32 age = 2;
}

case class PersonUpdate(
  theName: UpdateOrIgnore[String],
  age: UpdateOrIgnore[Int]
)

---

A request like:

{
  "the_name": "Thomas",
  "field_mask": [
    "the_name"
  ]
}

should result in:

PersonUpdate(
  Update("Thomas"),
  Ignore
)

In order to write a generic Transform:

given [A]: Transformer[A, UpdateOrIgnore[A]] = ???

I primarily need the path of field indices to the field that's being transformed (since ScalaPB renames fields to camel case). For theName, its path is .0 which I can map to the_name, which I then can check in the field mask.

In summary, is there a way to get the indexed path of the current transformation? Alternatively, is there an easier solution to this? 🙂

[MateuszKubuszok]

Hi,

When it comes to reading the current field path, then currently Chimney does not provide such a feature - mostly because this error-path is built in the runtime, e.g. when we are building error paths in PartialTransformers:

case class Foo(bar: Bar)
case class Bar(a: String)

case class Baz(bar: Bar2)
case class Bar2(a: Int)

implicit val bars: PartialTransformer[Bar, Bar2] = PartialTransformer { bar =>
  partial.Result
    .fromCatching(bar.a.toInt)
    .prependErrorPath(partial.PathElement.Accessor("a"))
    .map(a => Bar2(a))
}

implicit val fooBar: PartialTransformer[Foo, Baz] = PartialTransformrt { foo =>
  bars.transform(foo.bar)
    .prependErrorPath(partial.PathElement.Accessor("bar"))
    .map(bar => Baz(bar))
}

then when we would like to get the full path to the foo.bar.a inside bars code, we wouldn't get it. It would only be build in the runtime, by the prependErrorPath method. This method let us create this path lazily, so that you wouldn't pay the price for this feature if you wouldn't use it.

To make that path available we would have to change the way it is build:

• another parameter would have to be provided to the transform method
• this parameter would have to somehow be lazy (a by-name might not be enough on its own, because we might want to avoid calling this logic multiple times, so caching it in some lazy val might be necessary)
• everyone who provides transformers or partial transformers manually would have to handle this parameter and know what it is for

so while it it possible, at this point it would be quite a lot of work to make it possible.

However, you in you used PartialTransformers you WOULD get the path to each fields which failed at being converted. By that I do not mean that the right mapping couldn't be found, that's checked during the compilation, but that in the runtime it was checked that:

• you wanted to unwrap the Option and it was None
• you wanted to call a method converting some field and it threw
• you provided some custom method for conversion and you manually returned Exception or String with error message etc using partial.Result

Your use case seem to be quite exotic, I can only guess it's about PB treating integers and strings as non-nullable, and you having to manually determine if this empty value means "reset" or "ignore". I'm afraid this cannot be done OOTB with Chimney alone, and I would implement it by:

• having the Chimney conversion always convert the field into Updated
• having some custom method which would filter PersonUpdate with FieldMask value (setting the unused fields to ignore)

If you are using givens I guess you are using Scala 3, then it should be possible to write some generic logic using mirrors which would (example for PersonUpdate):

• take m: Mirror.Product[PersonUpdate]
• obtain field names with summonAll[Tuple.Map[m.MirroredElemLabels, ValueOf]]
• turn PersonUpdate into Product
• write a code which would compare field names, against FieldMask content, and for each missing name replacing Update with Ignore
• finally, calling the constructor with updated values

but such a code would not be completely autoderived, it would have to be semiautomatically derived for inputs which have FieldMasks and outputs which have this UpdateOrIgnore fields.

[jeengbe]

Sorry for the delayed response, thank you a lot for your reply! I appreciate it a lot. It's unfortunate that that doesn't work out of the box, [see below].

Your use case seem to be quite exotic

What makes you believe that, I thought using gRPC with a field mask is common practice. Representing update operations with an UpdateOrIgnore sum type felt like a pretty natural representation of the operation in Scala without having to pass field masks around. That's also what a similar operation would look like in a JSON API that distinguishes between not set/default value.

In fact, I was rather surprised to see that field masks are not handled out of the box, especially given their (or so I thought) wide spread. Do you have any insight into what the standard way of dealing with them is?

The documentation for com.google.protobuf.util.FieldMaskUtil only really provides a merge method that applies a patch onto an existing data object. There has to be a better way to do this, given "fetch from db, apply patch, overwrite in db" isn't even an atomic operation unless done inefficiently. There's a nice blog post by Netflix, but that only shows how mutation field masks are used by the client. Kinda what's proposed in https://stackoverflow.com/questions/57914586/fieldmask-update-method-implementation.

I doubt this isn't a common question for anyone writing a gRPC service with update masks in any to-JVM language.

[MateuszKubuszok]

What makes you believe that, I thought using gRPC with a field mask is common practice.

In companies I worked for these were never used, at least I haven't seen them in any gRPC defined with Protobufs (usually through Akka-gRPC). When I asked around on my local Scala Slack channels only 1 colleague used them in project some time ago.

Usually, the pattern that I saw was that we use oneof to define that one of several different possibilities:

message UpdateRequest {
  message UpdateAddress {
    string value = 1;
  }
  message UpdateCountry {
    string value = 1;
  }

  oneof value {
    UpdateAddress address = 1;
    UpdateCountry country = 2;
  }
}

message UpdateResponse {
  message Success {}
  message FailureCauseA {
    string details = 1;
  }
  message FailureCauseB {
    string details = 1;
  }

  oneof value {
    Success success = 1;
    FailureCauseA failure_a = 2;
    FailureCauseB failure_b = 3;
  }
}

service UpdateService {
  rpc Update (UpdateRequest) returns (UpdateResponse) {}
}

So basically, my experience is that we didn't treated messages as JSONs, where path-to-field would in used in runtime for business logic (only for debugging so the paths was usually converted to String and logged immediately), but we treated them as kinda domain models. If we wanted to select of of possible actions we used oneof with possible actions, if we wanted to return errors in gRPCs, we used dedicated oneof message for each cause.

Perhaps some other companies do things differently. But using FieldMask in decoding logic is outside of Chimney's current scope - any such support would most likely have to be encoded in macros and core derivation logic (and core module in general) is protocol-agnostic and dependency-free.

At best, I can imagine that we would add conversion from partial.Results ErrorPath to FieldMask - you could convert protobuf to domain object with PartialTransformer, and on failure get the failed result to convert it to e.g. Map[FieldMask, List[String]], which you would convert to the response yourself. However using FieldMask for driving the derivation logic for is currently out of scope.

[jeengbe]

That sounds very strange, considering even Google explicitly mention update masks in their API design guide, but I understand your point. Thanks for taking all the time for the replies. Similarly, it's surprising that error handling is also done differently than stated in Google's guidelines, assuming that they who made it must know best how to use it.

I see how it makes little sense to convert this to a different domain model, and do agree with that Chimney should not add this to the core if that requires a new feature.