add config parameters to allow reducing ledger transactions table size.

CHANGELOG.md

@@ -27,6 +27,13 @@ Release built: _not released yet_
 >   - `/state/entity/page/non-fungibles/` (when using `non_fungible_include_nfids` opt-in)
 >   - `/state/entity/page/non-fungible-vaults/` (when using `non_fungible_include_nfids` opt-in)
 
+### What’s new?
+- New configuration options `DataAggregator__Storage__StoreTransactionReceiptEvents`, and `DataAggregator__Storage__StoreReceiptStateUpdates` for data aggregator to configure if transaction's receipt events and receipt state updates should be stored in the database. It is meant to be used by gateway runners who want to reduce their database size. Keep in mind that when disabled `/stream/transactions` and `/stream/transactions` will not return that data.
+    - Possible values:
+        - `StoreForAllTransactions` (default) - will store data for all transactions.
+        - `StoryOnlyForUserTransactionsAndEpochChanges` - will store data for user transactions and transactions that resulted in epoch change.
+        - `StoreOnlyForUserTransactions` - will store data only for user transactions.
+        - `DoNotStore` - will not store any data.
 
 ### Bug fixes
 - Fixed `total_count` property returned for `explicit_metadata` collection. It returns now total number of metadata items for given entity. Affected endpoints:
@@ -60,6 +67,8 @@ Release built: _not released yet_
         - Renamed `entity_vault_history` to `vault_balance_history`. Holds information about vault content (amount of fungibles or count of non fungible ids inside vault) at a given state version.
     - Key value store
         - New `key_value_store_totals_history` table, which holds total count of all keys under a given store at a given state version.
+- Changed `receipt_state_updates` in the `ledger_transactions` table to be nullable.
+- Moved all `receipt_event_*` columns from `ledger_transactions` to separate `ledger_transaction_events` table.
 
 ## 1.7.2
 Release built: 17.09.2024

docs/configuration.md

@@ -73,6 +73,10 @@ The Network Gateway services can be configured in line with the [configuration i
 - `DataAggregator__Network__MaxAllowedStateVersionLagToBeConsideredSynced` (type: `int`, default value: `100`) - maximum allowed state version lag for CoreAPI node to be considered synced.
 - `DataAggregator__Network__IgnoreNonSyncedNodes` (type: `bool`, default value: `true`) - controls if nodes with a different status than healthy and synced should be used by the Data Aggregator.
 
+#### Storage
+- `DataAggregator__Storage__StoreTransactionReceiptEvents` (type: `enum`, default value: `StoreForAllTransactions`) - controls if data aggregator should store transaction receipt events in database.
+- `DataAggregator__Storage__StoreReceiptStateUpdates` (type: `enum`, default value: `StoreForAllTransactions`) - controls if data aggregator should store transaction receipt state updates in database.
+
 #### Monitoring
 `DataAggregator__Monitoring__StartupGracePeriodSeconds` (type: `int`, default value: `10`) - duration (seconds) of start-up grace period for Data Aggregator.
 `DataAggregator__Monitoring__UnhealthyCommitmentGapSeconds` (type: `int`, default value: `20`) - time window since the last committed transaction (seconds) in which Data Aggregator is considered healthy if does not commit new transactions.

src/RadixDlt.NetworkGateway.Abstractions/Configuration/StorageOptions.cs

@@ -0,0 +1,94 @@
+/* Copyright 2021 Radix Publishing Ltd incorporated in Jersey (Channel Islands).
+ *
+ * Licensed under the Radix License, Version 1.0 (the "License"); you may not use this
+ * file except in compliance with the License. You may obtain a copy of the License at:
+ *
+ * radixfoundation.org/licenses/LICENSE-v1
+ *
+ * The Licensor hereby grants permission for the Canonical version of the Work to be
+ * published, distributed and used under or by reference to the Licensor’s trademark
+ * Radix ® and use of any unregistered trade names, logos or get-up.
+ *
+ * The Licensor provides the Work (and each Contributor provides its Contributions) on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied,
+ * including, without limitation, any warranties or conditions of TITLE, NON-INFRINGEMENT,
+ * MERCHANTABILITY, or FITNESS FOR A PARTICULAR PURPOSE.
+ *
+ * Whilst the Work is capable of being deployed, used and adopted (instantiated) to create
+ * a distributed ledger it is your responsibility to test and validate the code, together
+ * with all logic and performance of that code under all foreseeable scenarios.
+ *
+ * The Licensor does not make or purport to make and hereby excludes liability for all
+ * and any representation, warranty or undertaking in any form whatsoever, whether express
+ * or implied, to any entity or person, including any representation, warranty or
+ * undertaking, as to the functionality security use, value or other characteristics of
+ * any distributed ledger nor in respect the functioning or value of any tokens which may
+ * be created stored or transferred using the Work. The Licensor does not warrant that the
+ * Work or any use of the Work complies with any law or regulation in any territory where
+ * it may be implemented or used or that it will be appropriate for any specific purpose.
+ *
+ * Neither the licensor nor any current or former employees, officers, directors, partners,
+ * trustees, representatives, agents, advisors, contractors, or volunteers of the Licensor
+ * shall be liable for any direct or indirect, special, incidental, consequential or other
+ * losses of any kind, in tort, contract or otherwise (including but not limited to loss
+ * of revenue, income or profits, or loss of use or data, or loss of reputation, or loss
+ * of any economic or other opportunity of whatsoever nature or howsoever arising), arising
+ * out of or in connection with (without limitation of any use, misuse, of any ledger system
+ * or use made or its functionality or any performance or operation of any code or protocol
+ * caused by bugs or programming or logic errors or otherwise);
+ *
+ * A. any offer, purchase, holding, use, sale, exchange or transmission of any
+ * cryptographic keys, tokens or assets created, exchanged, stored or arising from any
+ * interaction with the Work;
+ *
+ * B. any failure in a transmission or loss of any token or assets keys or other digital
+ * artefacts due to errors in transmission;
+ *
+ * C. bugs, hacks, logic errors or faults in the Work or any communication;
+ *
+ * D. system software or apparatus including but not limited to losses caused by errors
+ * in holding or transmitting tokens by any third-party;
+ *
+ * E. breaches or failure of security including hacker attacks, loss or disclosure of
+ * password, loss of private key, unauthorised use or misuse of such passwords or keys;
+ *
+ * F. any losses including loss of anticipated savings or other benefits resulting from
+ * use of the Work or any changes to the Work (however implemented).
+ *
+ * You are solely responsible for; testing, validating and evaluation of all operation
+ * logic, functionality, security and appropriateness of using the Work for any commercial
+ * or non-commercial purpose and for any reproduction or redistribution by You of the
+ * Work. You assume all risks associated with Your use of the Work and the exercise of
+ * permissions under this License.
+ */
+
+using FluentValidation;
+using Microsoft.Extensions.Configuration;
+
+namespace RadixDlt.NetworkGateway.Abstractions.Configuration;
+
+public sealed class StorageOptions
+{
+    [ConfigurationKeyName("StoreTransactionReceiptEvents")]
+    public LedgerTransactionStorageOption StoreTransactionReceiptEvents { get; set; } = LedgerTransactionStorageOption.StoreForAllTransactions;
+
+    [ConfigurationKeyName("StoreReceiptStateUpdates")]
+    public LedgerTransactionStorageOption StoreReceiptStateUpdates { get; set; } = LedgerTransactionStorageOption.StoreForAllTransactions;
+}
+
+public sealed class StorageOptionsValidator : AbstractOptionsValidator<StorageOptions>
+{
+    public StorageOptionsValidator()
+    {
+        RuleFor(x => x.StoreTransactionReceiptEvents).IsInEnum();
+        RuleFor(x => x.StoreReceiptStateUpdates).IsInEnum();
+    }
+}
+
+public enum LedgerTransactionStorageOption
+{
+    StoreForAllTransactions,
+    StoryOnlyForUserTransactionsAndEpochChanges,
+    StoreOnlyForUserTransactions,
+    DoNotStore,
+}

src/RadixDlt.NetworkGateway.DataAggregator/ServiceCollectionExtensions.cs

@@ -99,6 +99,7 @@ public static DataAggregatorBuilder AddNetworkGatewayDataAggregatorCore(this ISe
 
         services
             .AddValidatableOptionsAtSection<NetworkOptions, NetworkOptionsValidator>("DataAggregator:Network")
+            .AddValidatableOptionsAtSection<StorageOptions, StorageOptionsValidator>("DataAggregator:Storage")
             .AddValidatableOptionsAtSection<SlowQueryLoggingOptions, SlowQueryLoggingOptionsValidator>("DataAggregator:SlowQueryLogging")
             .AddValidatableOptionsAtSection<MonitoringOptions, MonitoringOptionsValidator>("DataAggregator:Monitoring")
             .AddValidatableOptionsAtSection<MempoolOptions, MempoolOptionsValidator>("DataAggregator:Mempool")

src/RadixDlt.NetworkGateway.DataAggregator/Services/ILedgerExtenderService.cs

@@ -77,9 +77,6 @@ public interface ILedgerExtenderService
 
 public sealed record TransactionSummary(
     long StateVersion,
-    string TransactionTreeHash,
-    string ReceiptTreeHash,
-    string StateTreeHash,
     DateTime RoundTimestamp,
     DateTime NormalizedRoundTimestamp,
     DateTime CreatedTimestamp,

src/RadixDlt.NetworkGateway.PostgresIntegration/CommonDbContext.cs

@@ -81,15 +81,10 @@ internal abstract class CommonDbContext : DbContext
 {
     internal const string DiscriminatorColumnName = "discriminator";
 
-    /// <summary>
-    /// Gets LedgerTransactions.
-    /// </summary>
-    /// <remarks>
-    /// A LedgerTransaction row contains large blobs, so you must SELECT the fields you need after using this, and not pull down the whole
-    /// ledger transaction row, to avoid possible performance issues.
-    /// </remarks>
     public DbSet<LedgerTransaction> LedgerTransactions => Set<LedgerTransaction>();
 
+    public DbSet<LedgerTransactionEvents> LedgerTransactionEvents => Set<LedgerTransactionEvents>();
+
     public DbSet<LedgerTransactionMarker> LedgerTransactionMarkers => Set<LedgerTransactionMarker>();
 
     public DbSet<PendingTransaction> PendingTransactions => Set<PendingTransaction>();

src/RadixDlt.NetworkGateway.PostgresIntegration/CustomTypesRegistrator.cs

@@ -94,6 +94,8 @@ public static void EnsureConfigured()
         SqlMapper.AddTypeHandler(new GenericArrayHandler<long>());
         SqlMapper.AddTypeHandler(new GenericArrayHandler<string>());
         SqlMapper.AddTypeHandler(new GenericArrayHandler<byte[]>());
+        SqlMapper.AddTypeHandler(new GenericArrayHandler<SborTypeKind>());
+        SqlMapper.AddTypeHandler(new GenericArrayHandler<LedgerTransactionManifestClass>());
 
         SqlMapper.AddTypeHandler(new GenericListHandler<int>());
         SqlMapper.AddTypeHandler(new GenericListHandler<long>());

src/RadixDlt.NetworkGateway.PostgresIntegration/GatewayModelExtensions.cs

@@ -180,75 +180,6 @@ public static GatewayModel.PublicKey ToGatewayPublicKey(this ValidatorPublicKeyH
         };
     }
 
-    public static GatewayModel.CommittedTransactionInfo ToGatewayModel(
-        this LedgerTransaction lt,
-        GatewayModel.TransactionDetailsOptIns optIns,
-        Dictionary<long, string> entityIdToAddressMap,
-        List<TransactionQuerier.Event>? events,
-        GatewayModel.TransactionBalanceChanges? transactionBalanceChanges)
-    {
-        string? payloadHash = null;
-        string? intentHash = null;
-        string? rawHex = null;
-        JRaw? message = null;
-        string? manifestInstructions = null;
-        List<GatewayModel.ManifestClass>? manifestClasses = null;
-
-        if (lt is UserLedgerTransaction ult)
-        {
-            payloadHash = ult.PayloadHash;
-            intentHash = ult.IntentHash;
-            rawHex = optIns.RawHex ? ult.RawPayload.ToHex() : null;
-            message = ult.Message != null ? new JRaw(ult.Message) : null;
-            manifestInstructions = optIns.ManifestInstructions ? ult.ManifestInstructions : null;
-            manifestClasses = ult.ManifestClasses.Select(mc => mc.ToGatewayModel()).ToList();
-        }
-
-        var receipt = new GatewayModel.TransactionReceipt
-        {
-            ErrorMessage = lt.EngineReceipt.ErrorMessage,
-            Status = ToGatewayModel(lt.EngineReceipt.Status),
-            Output = optIns.ReceiptOutput && lt.EngineReceipt.Output != null ? new JRaw(lt.EngineReceipt.Output) : null,
-            FeeSummary = optIns.ReceiptFeeSummary ? new JRaw(lt.EngineReceipt.FeeSummary) : null,
-            FeeDestination = optIns.ReceiptFeeDestination && lt.EngineReceipt.FeeDestination != null ? new JRaw(lt.EngineReceipt.FeeDestination) : null,
-            FeeSource = optIns.ReceiptFeeSource && lt.EngineReceipt.FeeSource != null ? new JRaw(lt.EngineReceipt.FeeSource) : null,
-            CostingParameters = optIns.ReceiptCostingParameters ? new JRaw(lt.EngineReceipt.CostingParameters) : null,
-            NextEpoch = lt.EngineReceipt.NextEpoch != null ? new JRaw(lt.EngineReceipt.NextEpoch) : null,
-            StateUpdates = optIns.ReceiptStateChanges ? new JRaw(lt.EngineReceipt.StateUpdates) : null,
-            Events = optIns.ReceiptEvents ? events?.Select(x => new GatewayModel.EventsItem(x.Name, new JRaw(x.Emitter), x.Data)).ToList() : null,
-        };
-
-        return new GatewayModel.CommittedTransactionInfo(
-            stateVersion: lt.StateVersion,
-            epoch: lt.Epoch,
-            round: lt.RoundInEpoch,
-            roundTimestamp: lt.RoundTimestamp.AsUtcIsoDateWithMillisString(),
-            transactionStatus: lt.EngineReceipt.Status.ToGatewayModel(),
-            affectedGlobalEntities: optIns.AffectedGlobalEntities ? lt.AffectedGlobalEntities.Select(x => entityIdToAddressMap[x]).ToList() : null,
-            payloadHash: payloadHash,
-            intentHash: intentHash,
-            feePaid: lt.FeePaid.ToString(),
-            confirmedAt: lt.RoundTimestamp,
-            errorMessage: lt.EngineReceipt.ErrorMessage,
-            rawHex: rawHex,
-            receipt: receipt,
-            message: message,
-            balanceChanges: optIns.BalanceChanges ? transactionBalanceChanges : null,
-            manifestInstructions: manifestInstructions,
-            manifestClasses: manifestClasses
-        );
-    }
-
-    public static GatewayModel.TransactionStatus ToGatewayModel(this LedgerTransactionStatus status)
-    {
-        return status switch
-        {
-            LedgerTransactionStatus.Succeeded => GatewayModel.TransactionStatus.CommittedSuccess,
-            LedgerTransactionStatus.Failed => GatewayModel.TransactionStatus.CommittedFailure,
-            _ => throw new UnreachableException($"Didn't expect {status} value"),
-        };
-    }
-
     public static GatewayModel.PackageVmType ToGatewayModel(this PackageVmType vmType)
     {
         return vmType switch

src/RadixDlt.NetworkGateway.PostgresIntegration/LedgerExtension/Extensions.cs

@@ -111,6 +111,22 @@ public static TVal AddOrUpdate<TKey, TVal>(this IDictionary<TKey, TVal> dictiona
         return value;
     }
 
+    public static TVal Update<TKey, TVal>(this IDictionary<TKey, TVal> dictionary, TKey key, Action<TVal> updateFactory)
+        where TKey : notnull
+    {
+        ArgumentNullException.ThrowIfNull(dictionary, nameof(dictionary));
+        ArgumentNullException.ThrowIfNull(key, nameof(key));
+        ArgumentNullException.ThrowIfNull(updateFactory, nameof(updateFactory));
+
+        if (!dictionary.TryGetValue(key, out var existingValue))
+        {
+            throw new ArgumentException($"Key with index: {key} not found in dictionary.");
+        }
+
+        updateFactory(existingValue);
+        return existingValue;
+    }
+
     public static void AddRange<TKey, TVal>(this IDictionary<TKey, TVal> dictionary, IDictionary<TKey, TVal> other)
     {
         ArgumentNullException.ThrowIfNull(dictionary, nameof(dictionary));