feat: Implement MPT changes

[shawnxie999]

Spec is here https://github.com/XRPLF/XRPL-Standards/tree/master/XLS-0033d-multi-purpose-tokens

Builds rippled on https://github.com/gregtatcam/rippled/tree/mpt-direct-pay

• refactor Issue and STAmount related changes (will need to change if rippled has further changes)
• introduces new table mp_token_holders
• ETL that extracts holders into mp_token_holders table whenever a new MPTokenAuthorize tx is observed
• new mpt_holders API
• inject synthetic mpt_issuance_id field into MPTokenIssuanceCreate transaction metadata for APIs (eg, tx, ledger, subscribe, account_tx)
• support new MPTokenIssuance and MPToken objects types for ledger_entry
• inject synthetic mpt_issuance_id field into MPTokenIssuance object for ledger_data and account_objects APIs

[godexsoft]

Found some things to improve 👍
Another thing is that your test coverage is slightly negative - please take a look at codecov for your branch to see where it's lacking tests. Anything with red change% needs a bit more love.

[godexsoft]

There are no JS for these atm. i assume. Leaving it here for when they are added - please update here and below 👍

[shawnxie999]

@godexsoft as for test coverage, i believe most of the parts that lack coverage are ETL related

[cindyyan317]

Actually , Only this part needs to be virtual. The filter logic is same for different DB. By reducing the scope of virtual functions, you can unittest more of the logic without real DB.

[shawnxie999]

could you clarify what you mean? inside this function, there is another call to the database auto mptObjects = doFetchLedgerObjects(mptKeys, ledgerSequence, yield);

[cindyyan317]

If we create a function in CassandraBackend.hpp to just select MPT holder, then all the filter logic can move to BackendInterface. If we want to add a new database, you don't need to copy all the logic to new database implementation. How to filter the valid holder is database unrelated. Instead, new database only needs to provide how to select MPT holder.

[shawnxie999]

But aren't existing functions in CassandraBackend.hpp also do something similar? BTW fetchMPTHolders doesn't only have filter logic, it does make another call to the objects table on line 576:

auto mptObjects = doFetchLedgerObjects(mptKeys, ledgerSequence, yield);

[cindyyan317]

I understood the fetchMPTHolders does not only have filter logic. So I suggested that we can add the pure virtual function doFetchMPTHolders in BackendInterface, which only obtains the data from selectMPTHolders table. Then the filer logic can move to BackendInterface normal function.

override doFetchfetchMPTHolders in CassandraBackend.hpp

    std::pair<std::vector<ripple::uint256>, std::optional<ripple::AccountID>>
    doFetchfetchMPTHolders(
        ripple::uint192 const& mptID,
        std::uint32_t const limit,
        std::optional<ripple::AccountID> const& cursorIn,
        boost::asio::yield_context yield
    ) const override
    {
        auto const holderEntries = executor_.read(
            yield, schema_->selectMPTHolders, mptID, cursorIn.value_or(ripple::AccountID(0)), Limit{limit}
        );

        auto const& holderResults = holderEntries.value();
        if (not holderResults.hasRows()) {
            LOG(log_.debug()) << "No rows returned";
            return {};
        }

        std::vector<ripple::uint256> mptKeys;
        std::optional<ripple::AccountID> cursor;
        for (auto const [holder] : extract<ripple::AccountID>(holderResults)) {
            mptKeys.push_back(ripple::keylet::mptoken(mptID, holder).key);
            cursor = holder;
        }
        return {mptKeys, cursor};
    }

Then your fetchMPTHolders will be like:

MPTHoldersAndCursor
BackendInterface::fetchMPTHolders(
    ripple::uint192 const& mptID,
    std::uint32_t const limit,
    std::optional<ripple::AccountID> const& cursorIn,
    std::uint32_t const ledgerSequence,
    boost::asio::yield_context yield
) const
{
    auto [mptKeys, cursor] = doFetchfetchMPTHolders(mptID, limit, cursorIn, yield);

    auto mptObjects = doFetchLedgerObjects(mptKeys, ledgerSequence, yield);

    auto it = std::remove_if(mptObjects.begin(), mptObjects.end(), [](Blob const& mpt) { return mpt.empty(); });

    mptObjects.erase(it, mptObjects.end());

    ASSERT(mptKeys.size() <= limit, "Number of keys can't exceed the limit");
    if (mptKeys.size() == limit)
        return {mptObjects, cursor};

    return {mptObjects, {}};
}

I admit that current CassandraBackend.hpp has similar implementation which is not correct. We will refactor them eventually. Feel free to leave it. We can refactor it in future.

[cindyyan317]

💯 All comments are addressed. Looks like ready to go. 👍

[kuznetsss]

I think this should be reverted before merging

[shawnxie999]

Yes these will be changed when it's ready to merge. We'd need wait for the rippled one to merge first