Avro v2

[nrwiersma]

There are some breaking changes needed in the library, and this is a good time to think about the list of features for v2. Here are some of the ideas I would like to tackle for this release:

• Rework Schema and Schema Parsing to allow for better JSON serialisation.
• Support Composed Reader Schemas as a first step to Avro RPC. (Too complex, address for v3).
• Support avro.Writer as encoder writer.
• Support Context in Schema Registry.
• Recursive Structs (Likely post v2 release)

Other ideas are welcome.

Contributed Ideas:

• Struct Generator

[pedromss]

When I first started out using this lib it was also around the time I started using avro with Go. The error messages when decoding OR encoding fail aren't very user friendly for new comers, if the structs are big. After you get the hang of it they become okish, but improvements could be made - I personally love error messages like Elm (the language), but I understand if that's considered too much work for this library.

Another thing would be the fact that unknown fields are ignored during the Unmarshall process. This makes it easy for typos to go through especially when you have really big schemas and things to not be decoded into the fields without any err. This is something that wouldn't happen if #96 existed.

Nevertheless, this lib feels pretty awesome when compared to other's we tried when we were choosing one. Great work overall!

[nrwiersma]

I think an option could be added to return an error if the field is not found.

Can you give more info about how the errors are bad or how they could be improved?

[pedromss]

Continuing here the discussion around generating structs started here #96 :

Regarding multi purpose structures and tags that are already there: if the struct is generated I'd think its OK to configure the generator to generate tags for anything. Example configuration interface (first thing that came to mind):

structs:
  A:
    annotations:
      - name: avro
        style: camel_case
      - name: json
        style: snake_case
        ignore_private_fields: true

This way everytime the struct is generated the annotations are generated as well. In my mind this is fine because annotations are just key value pairs, they're not avro specific nor json specific, its just metadata.

When I use code generation the principle I follow is: if its generated, only the machine changes it. Therefore multi purpose structs would have everything generated, avro tags and other tags as well.

[nrwiersma]

Yes. That would be awesome. Thanks

[pedromss]

Have you given some thought to the CLI for the generation feature? I can work on it if you let me know what lib you'd like to use here and what folder structure you'd like to use.

You did mention an internal command here #159 (comment) I'm not sure the CLI is something you want done now.

[nrwiersma]

I was actually thinking about today. My usual goto lib for CLIs is https://github.com/urfave/cli but I wonder if it is worth the dependency and perhaps flag is enough for our needs. I think the command should go into /cmd/avrogen (avrogen is off the top of my head, open to suggestions here.

If you have the time and want to see this feature through I am happy for you to. If you dont have the time I would be happy to pick this up for you.

In terms of usage, my thoughts were along the lines of avrogen -pkg somepackage -tags json:camel,yaml:snake -o my/file.go schema.avsc, but again open to suggestion here.

[pedromss]

I can pick it up this week. avrogen seems ideal. those options seems enough for now. flag seems enough for now.

Should this next PR include a README section or would you rather come up with that on your own when you decide to release v2?

[nrwiersma]

It can include the README section. Thanks.

[Ignalina]

An addition to Marshall() could be an MarshallZeroCopy where a "large enough" byte buffer is provided.

func (c *frozenConfig) MarshalZeroCopy(schema Schema, v interface{},[]destination) ([]byte, error)
The return type is kept , to see the correct length.. and also the writing interface need to have a zerocopy version
writer.BufferZeroCopy(destination)

I will fork and try this out , se if gives a 1% or something on my current numbers (That are fantastic already due Hamba Avro!) for my https://github.com/Ignalina/shredder

Troughput bytes/s total : 761.41MB /s
Troughput lines/s total : 1.44M Lines/s
Troughput lines/s toAvro: 2.82M Lines/s
Time spent toReadChunks : 0.0215806745 s
Time spent toAvro : 0.87478208175 s
Time spent toKafka : 0.59487903675 s

[nrwiersma]

Glad to hear you are having success with the lib.

Instead of using Marshal you should be using Encoder as this gives a lot more control. I think we can do what you want by allowing the NewEncoder writer to be an avro.Writer which it should use and just reset rather then allocating a new one. So in config.go L154 we could change if the write is an avro.Writer and if so use it. As the writer can be reset and gives you access to the internal buffer, this should do what you need.

Does this do what you need in this case?

[nrwiersma]

I think I will release V2 after the inclusion of the struct generator. The composed reader schemas is too complex for the time I have available at the moment. I also think Recursive Structs can wait till post v2 release, but still in v2.