Best practices for generating dumps of 50-100k durable entities

[drdamour]

Got a pattern with entities where we have 1 per customer that act as an actor aggregate for a lot of different events coming from bunch of different event hubs. That part work real well, but kinda loss on the step where we need to generate a "dump" file (csv or xml) for ingestion into some legacy system on a every X (between 6 and 48) hour type cadence. Specifically around how to "ask" all the entities for their representation. We'll have about 100k entities representing 100k active customers at a given point in time.

Some approaches i've considrered/tried:

• standard function make a bunch of ReadEntityAsync callls custom serializer dumping the entity basically as if it were a POCO in an efficient manner
  • same but in in a parrallel linq Task.WaitAll() setup
  • same but fanning out to other function calls to partition the requests
• orchestrator function making lots of CallEntityAsync which returns an already serialized row/node
  • in a parrallel linq Task.WaitAll() setupd (i've had problesm with scale of this in the past so this is my least favorite)
  • fanning out to sub orchestrations
• orchestrator function fanning out to activities
  • possibly fanning out to sub orchestrations fanning out to activities

None of these seems to be significantly better than the other perf wise in my admittedly weak testing/benchmarking. Before i dove into setting up a bigger test harness, i wanted to ask cause...i know characteristics like this will change as this lib evolves but usually if i dont' stray to far from an expected (aka load tested) path i can stay pretty confident.

#2053 indicates maybe fanning out to activities that read a lot of entities is the best approach.
#1974 didn't seem to really apply to my scenario, but i could be missing something

gotcha: right now i'm sticking to traditional persistence as we have a pretty strong no preview stuff in prod. but i'm down to hear if the answers are meant to change if i change persistence backends to say netherite. though i do see netherite is outta preview per it's github....

[davidmrdavid]

Hi @drdamour: Indeed, the new storage providers (Netherite and MSSQL) are now GA'ed, so you should be able to test them :)

As for your question - I'll have to defer to @sebastianburckhardt for expert advice.

[XBeg9]

I have map-reduce scenario and Netherite backend reduced the time by 2-3 times, but I think with durable functions you have a huge overhead to do such operations, I am still struggling how to make it happen and keep it durable.
The task is to parse csv file and fan-out per row with a worker/activity tasks, we can have around 100k rows for single file and we need to know when the processing is finished. @davidmrdavid is there a better way to do that?

What I have tried so far:

• Netherite backend (reduced overhead / throttling on storage, but this doesn't help overall looks like)
• Chunking single orchestrator to sub-orchestrators (1k/rows per sub-orchestrator)

[davidmrdavid]

Hi @XBeg9, this appears to be a separate question. Could you please create a separate thread so it can be triaged by the team? Thanks!

[cgillum]

I don't think any of these options are going to give you good performance. You might be better off doing a scan of all your entities rather than trying to fetch the state of all of them individually.

For example, consider using ListInstancesAsync with an InstanceIdPrefix value that matches your entity IDs, something like "@myentityType@", and set ShowInput to true. It won't be super-fast, especially when using Azure Storage, but it will be much more efficient than fetching the state of individual entities.

[sebastianburckhardt]

I agree that using a query method (as opposed to fetching them individually) should give you better performance.

FYI, we now also have a query method specifically for entities:
it is called ListEntitiesAsync.

[drdamour]

Ha, I am leveraging that quite a bit to identify all instances in the first place…was unaware that input is the state itself.

Bit difficult to fan that out to many functions given continuation token structure. Unless im missing a trick. Figured scaling with a fan out was the ay to go, but sequentially it may process in acceptable time. We’ll try it out.

btw u all aware that vs current azurite version has broken paging so all the list async methods dont work right unless you hack upgrade th azurite packaged with vs?

[sebastianburckhardt]

Bit difficult to fan that out to many functions given continuation token structure. Unless im missing a trick

One trick I can think of is to use the InstanceIdPrefix that @cgillum mentioned earlier to partition the query.

For example, if you have 10000 entities named @myentityType@0000, @myentityType@0001, ... @myentityType@9999 you can partition the query into 10 subqueries with respective prefixes @myentityType@0, myentityType@1, ... @myentityType@9. Each of those queries then return 1000 entities.

Depending on how your application uses entity keys this trick can be adjusted. For example, if the keys are Guids, you could use the first hex digit. But you would probably have to be careful about exacly how the keys are formatted (e.g. lowercase vs. uppercase and such).

btw u all aware that vs current azurite version has broken paging so all the list async methods dont work right unless you hack upgrade th azurite packaged with vs?

No, I was not aware of that. Thanks for letting us know.

[davidmrdavid]

@drdamour: I'm interested to learn more about this Azurite issue. Any chance you could open an issue describing it further? That would allow us to follow up. Thanks!

[drdamour]

Ah yeah i shoulda thought about prefix hex “modulo”, that could work

[drdamour]

In between flights and we have work arounds but the bug was fixed and mentioned in release notes of 3.18 https://github.com/Azure/Azurite/releases/tag/v3.18.0 but vs packages 3.17 with no easy way to upgrade

[davidmrdavid]

Thanks!