Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Workflow to Support Lazy Loading of Dataset? #75

Open
gursharan-info opened this issue May 14, 2023 · 4 comments
Open

Workflow to Support Lazy Loading of Dataset? #75

gursharan-info opened this issue May 14, 2023 · 4 comments
Labels
enhancement New feature or request

Comments

@gursharan-info
Copy link

I'm considering using the @kanaries/graphic-walker library for generic data visualisation, and I'm finding it extremely useful. However, I have a requirement where I need to handle large datasets efficiently. Currently, the library fetches the entire dataset upfront, which can be problematic when dealing with large amounts of data. I'm assuming that based on demos; please let me know if I'm wrong. We have a requirement to display spatial datasets which can go up to 50GB in size.

I wanted to inquire if it's possible to modify the workflow of the library to support the lazy loading of the dataset as a React Component. Here's the proposed workflow:

  • Initially, only fetch the measures and dimensions from the data source. JSON API is preferable, but direct SQL connection is also doable.
  • Allow the user to make chart selections such as the x-axis, y-axis, size, and color.
  • A button (e.g., "Generate Visualization") should be used to trigger the lazy loading of the dataset. It can either filter via the JSON API,or fetch the actual group/subset via SQL query, based on the selected measures and dimensions.
  • Once the dataset is fetched, generate the visualization using the selected chart options.

Please let me know if this is something that can be done using this library as an import without needing to customize the internals.

@ObservedObserver
Copy link
Member

We have a branch working on a related feature: Add a new option that the dataset will not be loaded into front-end. In this case, the query task will be sent to a customized server to handle the computation and only return the view data (which is usually much smaller than the raw data).

It can allow graphic-walker to handle much larger dataset than before. The feature is already in testing and will be published this week.

@ObservedObserver ObservedObserver added the enhancement New feature or request label May 15, 2023
@gursharan-info
Copy link
Author

Thanks for the reply. I will be waiting for the feature to try.

@maxott
Copy link

maxott commented Oct 30, 2023

Any update on that?

@LeonardoHBein
Copy link

@ObservedObserver Has the branch containing this feature been published? I'm very interested in this feature.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

No branches or pull requests

4 participants