Proposal for expanding event streaming and fixing Access/Observer node issues

[RostyslavAntonyshyn]

Grant category

• Open source maintenance

Description

This grant proposal consists of two parts:

Part 1 is the first part of the Event Streaming expansion set of work. Its goal is to expand event streaming within the Flow protocol. The current grant encompasses tasks that address investigation steps, community and team discussions, API definition for the whole scope of work, and FLIP's engagement. The part is divided into two separate milestones.

Part 2 focuses on 6 open issues that address technical debt for Access and Observer nodes, as well as 2 tasks to maintain the issues and usability of Access node which the core Flow team would like to resolve but currently cannot prioritize.
The part is divided into three milestones.

Proposed solution

Part 1 - Event Streaming enhancement design

1 - [Access] Make a proposal for the community in Discord to expand streaming for blocks, headers and transaction results. Discuss with community the idea overall and other possible needed subscriptions, get a feedback about current streaming approach for events and execution data, and possible ways for improvement.
2 - [Access] Investigate community applications/products to base streaming expansion on users needs.
3 - [Access] Investigate subscribing API on other platforms like Infura, Alchemy, QuickNode etc. to discover api templates and most usable endpoints.
4 - [Access] Revise the current streaming approach based on feedback from community and researches, create a basic design.
5 - [Access] Write a FLIP document based on discussions with community and investigation results.
6 - [Access] Share the FLIP document with the community for further polishing and improvements.

Part 2: Access node open issues tech debt

7 - [Access] Add endpoint to get protocol snapshot from any block #4869 ( onflow/flow-go#4869 )

Introduce two new methods for access API: GetProtocolStateSnapshotByBlockID and GetProtocolStateSnapshotByHeight that return the snapshot from the specified block if that block is finalized.

8 - [Access] Handle inconsistent protocol state in access rpc engine #4028 ( onflow/flow-go#4028 ):
Document the expected errors from all access API endpoints and implement handlers for each of them. Any unexpected errors should be forwarded and managed by irrecoverable.SignalerContext as irrecoverable.

9 - [Access] Add endpoints to Execution nodes to support getting Transaction Result error messages #4754 ( onflow/flow-go#4754 ):
For execution node add next API calls:

• Get error messages for all failed txns for block and specific collection: (blockID, collectionID)
• Get error messages for all failed txns in block and all collections: (blockID)
• Get error messages for a specific failed txn in block: (blockID, txID)

Add tests

For access node: update GRPC API, add supporting code for AN to call new EN methods.

10 - [Observer] Improve upstream gRPC failovers with AN connection pooling #2955 ( onflow/flow-go#2955 )

Adopt current reconnectingClient function implementation of access Forwarder to use existing GetAccessAPIClient functionality from access ConnectionFactory to retrieve clients.

11 - [Access] Cache Transaction Result error messages #4850 ( onflow/flow-go#4850 ):
Add support for AN to cache error messages in a separate storage in lazy fashion. Whenever we sync execution data and depending on user's query we will request messages for failed transaction and cache it in local db. Next queries for the same transaction will be served from local DB. Add tests.

12 - [Access] Get Block endpoint is missing the system collection #4584 ( onflow/flow-go#4584 ):
Implement a separate API calls to return system transaction and system transaction result. Remove special workarounds for retrieving system tx that are present in AN API calls right now. Update existing tests and add new tests.

13 - [Access] Prepare documentation for existing streaming API. Make a publication on onflow/docs portal.

14 - [Access] Maintenance of existing code.
Extra effort for possible issues that may arise on the functionality related to previously developed features.

Impact

Proposed fixes will improve the streaming capabilities of the Flow project and will have a beneficial impact on community needs.

Milestones and funding

Milestone 1

Milestone	Deliverables	Timeline	Risks	USD proposal
1 - [Access] Community proposal	Make a proposal to the community to expand streaming. Discuss the idea, get the feedback.	24h	-	1560$
2 - [Access] Analyze streaming expansion	Investigate community applications/products to base streaming expansion on users needs.	24h	-	1560$
3 - [Access] Investigate subscribing API on other platforms.	4 platforms analyzed to get the best approaches ideas.	24h	-	1560$
4 - [Access] Revise the current streaming approach based on feedback from community and researches, create a basic design.	Draft design ready	80h	-	5200$

Milestone 2

Milestone	Deliverables	Timeline	Risks	USD proposal
5 - [Access] Write a FLIP document based on discussions with community and investigation results.	FLIP ready and properly documented	120h	-	8580$
6 - [Access] Share the FLIP document with the community for further polishing and improvements.	Final version ready, discussed and published into repo	80h	-	5200$

Milestone 3

Milestone	Deliverables	Timeline	Risks	USD proposal
7 - [Access] Add endpoint to get protocol snapshot from any block #4869	Fixes onflow/flow-go#4869	32h	-	2080$
8 - [Access] Handle inconsistent protocol state in access rpc engine #4028	Fixes onflow/flow-go#4028	40h	-	2600$
9 - [Access] Add endpoints to Execution nodes to support getting Transaction Result error messages #4754	Fixes onflow/flow-go#4754	48h	-	3120$

Milestone 4

Milestone	Deliverables	Timeline	Risks	USD proposal
10 - [Observer] Improve upstream gRPC failovers with AN connection pooling #2955	Fixes onflow/flow-go#2955	40h	-	2600$
11 - [Access] Cache Transaction Result error messages #4850	Fixes onflow/flow-go#4850	48h	-	3120$
12 - [Access] Get Block endpoint is missing the system collection #4584	Fixes onflow/flow-go#4584	56h	-	3640$

Milestone 5

Milestone	Deliverables	Timeline	Risks	USD proposal
13 - [Access] Prepare documentation for existing streaming API	Provide missing API documentation for event streaming	8h	-	520$
14 - [Access] Maintenance of existing code	Maintenance done	24h	-	1560$

Total funding proposed: 42900 USD

Team

Name	Role	Bio	Contact
Andrii	Blockchain Engineer	Software Engineer with over 8 years of technical experience. Primary skills include C++(Standards 11,14,17), Rust, and Golang.	[email protected]
Uliana	Blockchain Engineer	Software Engineer with over 5 years of technical experience. Primary skills include C++(Standards 11,14,17), Rust, and Golang.	[email protected]
Andrii	Blockchain Engineer	Software Engineer with over 6 years of technical experience. Primary skills include C++(Standards 11,14,17), Rust, and Golang.	[email protected]