Fragmentation efficiency - 232738 members ingested - 22574 processed

[xdxxxdx]

Hello.

I am using the low water quality observations data set for testing 1.4.0 snapshot
Full setup please see: https://github.com/xdxxxdx/ODALA_demonstrator

232738 water quality members are ingested.

In the real-time: 22574 members are processed for the fragmentations.

The gap is quite big for me, but don't know if there is an efficiency matrix set for evaluating the project.
Therefore, I make this case for future discussion.

Thanks

[Tomvbe]

Hi @xdxxxdx,

Decoupling the ingest from the fragmentation is intended behavior. We persist these in the database so a large gap won't cause memory issues and we will get eventual consistency.

However I do agree that the gap is quite big in your case and took the liberty to look into your ldes-server.yml. You did not activate auto-index-creation. I assume you aren't managing your indices manually either. This property has a very big impact on the performance of the fragmentation. Can you try adding the following property to your yml?

spring:
  data:
    mongodb:
      auto-index-creation: true

[xdxxxdx]

Thank @Tomvbe , will have a look, and give you a feedback.

[xdxxxdx]

Works perfectly for 562777 ldes members. thanks @Tomvbe