Replies: 3 comments 8 replies
-
|
Current access time for metadata is already logarithmic.
My own experience and benchmarks have been more in the range of 10-20ms. Do you have any particular reproducible case / setup where you observe such large delays ? |
Beta Was this translation helpful? Give feedback.
-
|
I will try to set up something reproducible! If you want to play around with it in the meantime, this example comes from the MinSwap orderbook contract, transactions close to launch of the platform on mainnet. |
Beta Was this translation helpful? Give feedback.
-
|
Just to update this: I have tried the following query on mainnet And could not reproduce access times around 1-2s. However I found that tail latency might be quite high and appears to generally depend on the load on the cardano-node. Server without load on node: all requests around 0.02s = 20ms I have recently drastically reduced the load on the node (using kupo) so I am assuming that the measurement fo 1-2s was done when there was more load (i.e. permanent calls to cardano-cli fetching large parts of the UTxO). Ideally I would wish for kupo to enable taking load off the node and since metadata is already present when fetching blocks for syncing, storing the metadata in kupo would reduce the load for the node. I acknowledge though that this might also add more load on kupo/the supposedly minimal sqlite db so I understand if the metadata is not added into the db. |
Beta Was this translation helpful? Give feedback.
-
|
I have a use case for this as well. Storing metadata in the database would be an awesome feature. My use case is for a simple API that can get the assets within a user's wallet for example. Right now Kupo is great because I can get how much Cardano (and other tokens) and # of NFT's a user has, but obviously the output looks like I tried using the |
Beta Was this translation helpful? Give feedback.
-
|
Also being able to lookup metadata in the database given a specific |
Beta Was this translation helpful? Give feedback.
-
|
Right now, my solution is just to use Maestro and query the So when I need to get a user's assets, I'll query the But having Kupo store metadata in the db would eliminate the need to use the Maestro API to query metadata essentially 😃 |
Beta Was this translation helpful? Give feedback.
-
|
But I am slightly confused, you're saying it is "too slow" whereas the various benchmarks I have show that metadata can be retrieved within a dozen of milliseconds. I don't see how querying another remote service (eg Maestro) can be significantly faster here? For example, in the starter kit for Demeter, the example number 3 fetches all metadata associated with Hosky holders of the past 24h. The example (https://github.com/CardanoSolutions/kupo-js-starter-kit/tree/main/examples/3-find-hosky-hodlers) typically fetches ten of thousands of results and their metadata within seconds. Truth is, metadata are already stored in a database, but the node's one. Kupo has a direct access to them in a logarithmic time complexity. Duplicating the data in a new database, while it would possibly provide new index keys I concede, wouldn't lead to much asymptotic improvements. One idea mentioned on another discussion could be to have Kupo do some "deep fetching" of metadata and fetch them alongside the query. This wouldn't be significantly faster but at least it would be more convenient (as clients would get everything in a single request). |
Beta Was this translation helpful? Give feedback.
-
|
I take a performance first approach to everything, so I'd prefer if the data was just there ready for querying, instead of having to go and fetch it, is mostly why I suggested storing it in the database. But doing the "deep fetching" approach you suggested could also work, as long as it still works well under medium/heavy load 👍 |
Beta Was this translation helpful? Give feedback.
-
|
But the data IS there. That's what I've been kindly explaining Right now, the system can accommodate serving metadata at a rather decent rate (~hundred of metadata per second). If you need something higher than that, I'd be curious to hear about the use case. Kupo may simply not be the best fit for you. I am also keen on exploring options which remains reasonable, yet for now your claim seems more of a gut feeling than a measured observation? |
Beta Was this translation helpful? Give feedback.
-
What is your idea? Provide a use case.
When querying the transaction history of DEXs, it is interesting to see what metadata was provided with each transaction (i.e. to distinguish order placed through the main interface, trading bots and DEX Aggregators).
This requires quick access to transaction metadata.
Why is it a good idea?
By storing the metadata in a local table in kupo, access time can be reduced to logarithmic time. By pruning away metadata unrelated to the stored transactions, this wouldn't affect the size of the database too much either.
What is the current alternative and why is it not good enough?
Currently kupo offers an option to access metadata for a transaction. Access time is easily on the order of 1-2s though since the metadata is not stored in the local database, but fetched from the node. This makes analysis of DEX transactions (easily exceeding 1M unique transactions) infeasible.
Another alternative is fetching data from blockfrost or seperate instance, which however seperates sources and may lead to problems when accessing "fresh" data on chain (one instance having picked up a transaction with the other instance missing it)
Beta Was this translation helpful? Give feedback.
All reactions