Proposal on expanding event streaming pt.2

[RostyslavAntonyshyn]

Grant category

• Open source maintenance

Description

This grant consist of two main parts: first part consist of 6 open tasks, which lead to improvements for Access node and Observers indexing while the second set of tasks is related to the Event Streaming expansion work, and encompasses tasks that address gRPC implementations along with test coverage of next endpoints: SubscribeBlocks, SubscribeBlocksHeaders, SubscribeBlocksLightweight, SendAndSubscribeTransactionStatuses and SubscribeAccountStatuses. The grant is divided into 4 (four) separate milestones.

As extra, it also includes RND work into the Websockets improvement which will be the next planned step which is still be under discussion along with new REST implementation as channels.

Impact

The expansion of the Event Streaming API with items 3-8 brings parity between Access and Observer nodes for them supporting event streaming. These roadmap features also set the foundation for community development of advanced locally optimized event indexing and streaming capabilities.
Community requested items 2, 9-13 leverage the above foundations to make event streaming the preferred form of integration for builders.

Next Step

Improvement of current WebSocket to support a single connection for all subscriptions from one client.

• The WSHandler should initiate a connection only for the initial subscription and retain it for subsequent subscriptions.
• An unsubscribe mechanism should be introduced. With multiple subscriptions on a single connection, simply closing this connection to halt updates becomes insufficient. Therefore, a dedicated unsubscribing from undesired updates is necessary.
• Each subscription response should include a message type. For instance, the SubscribeEvents endpoint should have a message with type: "Events", while SubscribeBlocksHeaders should have type: "BlockHeader", and so forth. This is essential to distinguish between various subscription results that will be delivered through a single connection.

Proposed solution

1 - [Access] Reviews on Access node open issues

This task will cover possible PR reviews on Access/Observer open issues till the end of February.

2 - [Access] Rnd work on Websockets improvement

3 - [Access] Use Local Index for Transaction Results in Access API | Fixes: onflow/flow-go#4753

The backend_transactions.go file will be reviewed. Where feasible, the local transaction result index will be used to retrieve transaction results for indexed blocks. The indexed transaction results do not contain cached error messages. To access this information, the functionality provided in issue #4850 will be utilized. The tests for the existing improved endpoints will be updated.

4 - [State Sync] Add execution state sync to public network | Fixes: onflow/flow-go#2795

The ExecutionDataRequester should be added to the observer node in the same manner as it was done for the Access Node. Additionally, a new CLI flag and supplementary functionality will be incorporated into the access node to facilitate bridging for the public/staked networks with bitswap. Furthermore, support for utilizing bitswap to synchronize the execution data on the observer node will be implemented.

5 - [Access] Add support for indexing execution data on Observers | Fixes: onflow/flow-go#4848

Add execution state checkpoint and indexing support for the Observer node. This should implement the same functionality, as it is in the Access node's DependableComponent("execution data indexer".... A Related cmd flag execution-data-indexing-enabled and all dependent functionality should also be implemented.

6 - [Access] Add standard Access API implementations on Observer | Fixes: onflow/flow-go#4847

The functionality in the apiproxy.FlowAccessAPIRouter should be modified to use locally cached execution data from observer node directly, when execution data indexer is enabled, instead of upstreaming request to access node.

7 - [Access] Add state streaming API to observers | Fixes: onflow/flow-go#4406

The state-stream-addr CLI argument will be added to enable stream listening. If the argument was set, the state stream engine should create on the observer node in the same way as it implemented in access node. The engineBuilder should be populated with created state stream.

8 - [Access] Benchmark testing and analysis of execution data indexing on Observers | Fixes: onflow/flow-go#4849

Integration tests will be added to verify the new indexing functionality. Performance measurement of the new execution data indexing on the Observer node will be conducted on the local network based on metrics and logs data.

9 - [Access] Implement gRPC streaming endpoint SubscribeBlocks:
Introduce a new SubscribeBlock endpoint within the Access Node API to facilitate the streaming of blocks (flow.Block) at the gRPC level. This new endpoint should has similar functionality of reciving new blocks as SubscribeEvents. Additionally, subscribers should be able to define the block status (BlockStatusSealed or BlockStatusFinalized) through an argument, receiving only blocks with the specified statuses. This would have a heavy response, as it contains flow.Payload.

10 - [Access] Implement gRPC streaming endpoint SendAndSubscribeTransactionStatuses:
Introduce a new endpoint named SendAndSubscribeTransactionStatuses within the Access Node API to facilitate the streaming of three transactions' fields: ID, Status, and Error at the gRPC level. This endpoint enables users to send a transaction and immediately subscribe to its status changes. The status is streamed back until the block containing the transaction becomes sealed. The functionality should be similar to one, that implemented in Access API deriveTransactionStatus to track transaction statuses.

11 - [Access] Implement gRPC streaming endpoint SubscribeBlocksHeaders:
Introduce a new endpoint named SubscribeBlocksHeaders within the Access Node API to facilitate the streaming of headers (flow.Header) at the gRPC level. This new endpoint should has similar functionality of reciving new blocks as SubscribeBlocks. Subscribers will also be able to specify the block status (BlockStatusSealed or BlockStatusFinalized) using an argument, receiving only blocks with the specified statuses. Although this endpoint is similar to SubscribeBlocks, it will yield lighter responses as it will not contain the flow.Payload part.

12 - [Access] Implement gRPC streaming endpoint SubscribeBlocksLightweight:
Introduce a new endpoint named SubscribeBlocksLightweight within the Access Node API to facilitate the streaming of three fields: ID, Height, and Timestamp at the gRPC level. This new endpoint should has similar functionality of reciving new blocks as SubscribeBlocks and SubscribeBlocksHeaders. Subscribers will also have the capability to specify the block status (BlockStatusSealed or BlockStatusFinalized) using an argument, receiving only blocks with the specified statuses. Although this endpoint is similar to SubscribeBlocks and SubscribeBlocksHeaders, it will be the lightest among them as it only provides a limited set of fields in response.

13 - [Access] Implement gRPC streaming endpoint SubscribeAccountStatuses:

Introduce a new endpoint named SubscribeAccountStatuses within the state_stream Execution Data API to facilitate the streaming of two account fields: Address and Status at the gRPC level. This endpoint allows users to subscribe to status changes (built-in account event types). It should include an argument introducing a filter array of statuses matching the client’s filter. Any statuses that match at least one of the conditions are returned. This new endpoint aims to track account status in each received sealed block and notify status changes if any of the built-in account event types are found in block events.

Impact

Proposed fixes will improve specific aspects of Access Node functionality and contributes to overall system performance and efficiency.

Milestones and funding

Milestone 1

Task	Deliverables	Timeline	Risks	USD proposal
1 - [Access] Reviews on Access node open issues		12h	-	780$
2 - [Access] Rnd work on Websockets improvement	Decision on approach made	32h	-	2080$
3 - [Access] Use Local Index for Transaction Results in Access API	Fixes: onflow/flow-go#4753	40 h	-	2600$
4 - [State Sync] Add execution state sync to public network	Fixes: [onflow/flow-go#2795] and onflow/flow-go#5186	48 h	-	3120$
5 - [Access] Add support for indexing execution data on Observers	Fixes: onflow/flow-go#4848	40 h	-	2600$

Milestone 2

Task	Deliverables	Timeline	Risks	USD proposal
6 - [Access] Add standard Access API implementations on Observer	Fixes: onflow/flow-go#4847	56 h	-	3640$
7 - [Access] Add state streaming API to observers	Fixes: onflow/flow-go#4406	48 h	-	3120$
8 - [Access] Benchmark testing and analysis of execution data indexing on Observers	Fixes: onflow/flow-go#4849	56 h	-	3640$

Milestone 3

Task	Deliverables	Timeline	Risks	USD proposal
9 - [Access] Implement gRPC streaming endpoint SubscribeBlocks	SubscribeBlocks endpoint for gRPC added and tested	64h	-	4160$
10 - [Access] Implement gRPC streaming endpoint SendAndSubscribeTransactionStatuses	endpoint for gRPC added and tested	72h	-	4680$

Milestone 4

Task	Deliverables	Timeline	Risks	USD proposal
11 - [Access] Implement gRPC streaming endpoint SubscribeBlocksHeaders	endpoint for gRPC added and covered with tests	48h	-	3120$
12 - [Access] Implement gRPC streaming endpoint SubscribeBlocksLightweight	endpoint for gRPC added and covered with tests	48h	-	3120$
13 - [Access] Implement gRPC streaming endpoint SubscribeAccountStatuses	endpoint for gRPC added and covered with tests	64h	-	4160$

Total funding proposed: 41340 USD

Team

Name	Role	Bio	Contact
Andrii	Blockchain Engineer	Software Engineer with over 8 years of technical experience. Primary skills include C++(Standards 11,14,17), Rust, and Golang.	[email protected]
Uliana	Blockchain Engineer	Software Engineer with over 5 years of technical experience. Primary skills include C++(Standards 11,14,17), Rust, and Golang.	[email protected]
Andrii	Blockchain Engineer	Software Engineer with over 6 years of technical experience. Primary skills include C++(Standards 11,14,17), Rust, and Golang.	[email protected]