feat: compute piece for uploaded cars #228

vasco-santos · 2023-09-14T14:10:37Z

This is the first of a series of PRs to integrate w3up with w3filecoin.

This PR adds Filecoin Stack, where CARPARK bucket is hooked up with a lambda to compute Piece CIDs via Event Bus events. A queue is in the middle to keep the flow steady. Once a PR is computed, its details are written into a DynamoDB table. This will have all the information users might need to ask, in order to then ask Deal Tracker for information for a given deal. In other words, it maps piece, link (Car CID), aggregate and inclusionProof (TBD what it actually is, is it a CID for a stored file in a bucket?).

Also integration tests were added.

In a follow up PR, a event will be hooked up from DynamoDB table insert to trigger segment/add capability from Aggregator together with writing a claim to equivalency claims.

TODO:

add https://github.com/web3-storage/fr32-sha2-256-trunc254-padded-binary-tree-multihash/ published module

filecoin/index.js

seed-deploy · 2023-09-14T16:50:06Z

View stack outputs

pr228-w3infra-CarparkStack

Name Value

BucketName carpark-pr228-0

Region us-east-2
pr228-w3infra-RoundaboutStack

Name Value

ApiEndpoint https://rkbcbltml3.execute-api.us-east-2.amazonaws.com

CustomDomain https://pr228.roundabout.web3.storage
pr228-w3infra-SatnavStack

Name Value

BucketName satnav-pr228-0

Region us-east-2
pr228-w3infra-UploadApiStack

Name Value

ApiEndpoint https://5z10vjo281.execute-api.us-east-2.amazonaws.com

CustomDomain https://pr228.up.web3.storage
pr228-w3infra-BusStack
pr228-w3infra-FilecoinStack
pr228-w3infra-ReplicatorStack
pr228-w3infra-UcanInvocationStack
pr228-w3infra-UploadDbStack

vasco-santos · 2023-09-15T09:27:17Z

stacks/index.js

@@ -21,7 +22,7 @@ export default function (app) {
  app.setDefaultFunctionProps({
    runtime: 'nodejs16.x',
    environment: {
-      NODE_OPTIONS: "--enable-source-maps",
+      NODE_OPTIONS: "--enable-source-maps --experimental-fetch",


https://github.com/web3-storage/fr32-sha2-256-trunc254-padded-binary-tree-multihash relies on globalThis.fetch and we will need to rely on this until we can upgrade SST here to bump node version

ACK, lets get that upgrade done soon!

alanshaw

This LGTM, I left a few nits. Please can we also publish an equivalency claim?

alanshaw · 2023-09-18T15:01:16Z

filecoin/errors.js

+)
+export class DatabaseOperationFailed extends Error {
+  get reason() {
+    return this.message


Is this necessary? Why not just use the message property?

It follows the same pattern as ucanto does https://github.com/web3-storage/ucanto/blob/c8999a59852b61549d163532a83bac62290b629d/packages/core/src/result.js#L63 for failures, so that the types of failures from handlers of ucanto server are all well aligned. So, not strictly necessary, but consistent

filecoin/functions/piece-cid-compute.js

filecoin/index.js

alanshaw · 2023-09-18T15:15:11Z

filecoin/package.json

+    "@aws-sdk/client-sqs": "^3.226.0",
+    "@sentry/serverless": "^7.22.0",
+    "@web3-storage/data-segment": "^3.0.1",
+    "fr32-sha2-256-trunc254-padded-binary-tree-multihash": "github:web3-storage/fr32-sha2-256-trunc254-padded-binary-tree-multihash",


I think this can be npm now?

filecoin/tables/piece.js

alanshaw · 2023-09-18T15:26:56Z

filecoin/test/compute-piece-cid.test.js

+  })
+})
+
+test('computes piece cid from a CAR file in the bucket', async t => {


Suggested change

test('computes piece cid from a CAR file in the bucket', async t => {

test('computes piece CID from a CAR file in the bucket', async t => {

alanshaw · 2023-09-18T15:36:46Z

stacks/index.js

@@ -21,7 +22,7 @@ export default function (app) {
  app.setDefaultFunctionProps({
    runtime: 'nodejs16.x',
    environment: {
-      NODE_OPTIONS: "--enable-source-maps",
+      NODE_OPTIONS: "--enable-source-maps --experimental-fetch",


ACK, lets get that upgrade done soon!

test/helpers/table.js

alanshaw · 2023-09-18T15:39:10Z

test/helpers/table.js

+      minTimeout: 1000,
+      retries: 100
+    })
+  } catch {}


I'd let this throw - if we don't find after 100 retries it ain't coming.

vasco-santos · 2023-09-18T15:43:24Z

This LGTM, I left a few nits. Please can we also publish an equivalency claim?

Yes, it is mentioned in main comment and will come as follow up PR

Co-authored-by: Alan Shaw <[email protected]>

vasco-santos · 2023-09-18T16:37:17Z

CI for integration tests is failing given resources were destroyed for kinesis for this cloudformation

Needs: - [x] #228

seed-deploy bot had a problem deploying to pr228 September 14, 2023 14:10 Failure

vasco-santos force-pushed the feat/compute-piece-for-uploaded-cars branch from 82ee974 to 6ed7cd0 Compare September 14, 2023 14:14

seed-deploy bot had a problem deploying to pr228 September 14, 2023 14:16 Failure

vasco-santos force-pushed the feat/compute-piece-for-uploaded-cars branch from 6ed7cd0 to 48d8bb6 Compare September 14, 2023 16:35

seed-deploy bot temporarily deployed to pr228 September 14, 2023 16:35 Inactive

vasco-santos commented Sep 14, 2023

View reviewed changes

filecoin/index.js Outdated Show resolved Hide resolved

feat: compute piece for uploaded cars

b4ae896

vasco-santos force-pushed the feat/compute-piece-for-uploaded-cars branch from 48d8bb6 to b4ae896 Compare September 14, 2023 16:53

seed-deploy bot temporarily deployed to pr228 September 14, 2023 16:53 Inactive

vasco-santos requested review from Gozala and alanshaw September 14, 2023 16:59

vasco-santos marked this pull request as ready for review September 14, 2023 16:59

fix: use wasm fr32 lib

2429663

seed-deploy bot deployed to pr228 September 15, 2023 09:23 View deployment

vasco-santos commented Sep 15, 2023

View reviewed changes

vasco-santos mentioned this pull request Sep 15, 2023

feat: trigger aggregator from pieces computed #229

Merged

1 task

alanshaw approved these changes Sep 18, 2023

View reviewed changes

chore: apply suggestions from code review

bd53447

Co-authored-by: Alan Shaw <[email protected]>

seed-deploy bot had a problem deploying to pr228 September 18, 2023 15:47 Failure

chore: address review comments

93f202d

seed-deploy bot had a problem deploying to pr228 September 18, 2023 15:56 Failure

seed-deploy bot temporarily deployed to pr228 September 18, 2023 16:31 Inactive

vasco-santos merged commit 0ec8078 into main Sep 18, 2023
1 check passed

vasco-santos deleted the feat/compute-piece-for-uploaded-cars branch September 18, 2023 16:37

vasco-santos added a commit that referenced this pull request Sep 18, 2023

feat: trigger aggregator from pieces computed (#229)

12a5393

Needs: - [x] #228

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: compute piece for uploaded cars #228

feat: compute piece for uploaded cars #228

vasco-santos commented Sep 14, 2023 •

edited

Loading

seed-deploy bot commented Sep 14, 2023

vasco-santos Sep 15, 2023

alanshaw Sep 18, 2023

alanshaw left a comment

alanshaw Sep 18, 2023

vasco-santos Sep 18, 2023

alanshaw Sep 18, 2023

alanshaw Sep 18, 2023

alanshaw Sep 18, 2023

alanshaw Sep 18, 2023

vasco-santos commented Sep 18, 2023

vasco-santos commented Sep 18, 2023

	test('computes piece cid from a CAR file in the bucket', async t => {
	test('computes piece CID from a CAR file in the bucket', async t => {

feat: compute piece for uploaded cars #228

feat: compute piece for uploaded cars #228

Conversation

vasco-santos commented Sep 14, 2023 • edited Loading

seed-deploy bot commented Sep 14, 2023

Choose a reason for hiding this comment

Choose a reason for hiding this comment

alanshaw left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

vasco-santos commented Sep 18, 2023

vasco-santos commented Sep 18, 2023

vasco-santos commented Sep 14, 2023 •

edited

Loading