From 2f9b6042503f087e2754a0236c0f6e4db3a527ee Mon Sep 17 00:00:00 2001 From: Alan Shaw Date: Wed, 13 Dec 2023 14:29:16 +0000 Subject: [PATCH] fix: upload section --- src/pages/docs/concepts/car.md | 4 +- src/pages/docs/concepts/upload-vs-store.md | 46 +++++--- src/pages/docs/how-to/create-account.md | 2 +- .../{create-space.md => create-space.mdx} | 8 +- src/pages/docs/how-to/retrieve.mdx | 2 +- src/pages/docs/how-to/upload.mdx | 108 +++++++++--------- 6 files changed, 95 insertions(+), 75 deletions(-) rename src/pages/docs/how-to/{create-space.md => create-space.mdx} (90%) diff --git a/src/pages/docs/concepts/car.md b/src/pages/docs/concepts/car.md index 1846e28..064df71 100644 --- a/src/pages/docs/concepts/car.md +++ b/src/pages/docs/concepts/car.md @@ -44,13 +44,13 @@ The --pack flag will create a new CAR file from a collection of input files: ipfs-car pack path/to/files --output path/to/write/a.car ``` -Extract files from a CAR with --unpack: +Extract files from a CAR with `--unpack`: ```sh ipfs-car unpack path/to/my.car --output path/to/unpack/files/to ``` -List the contents of a CAR with --list: +List the contents of a CAR with `--list`: ```sh ipfs-car list path/to/my.car diff --git a/src/pages/docs/concepts/upload-vs-store.md b/src/pages/docs/concepts/upload-vs-store.md index 1fa2220..278c5a6 100644 --- a/src/pages/docs/concepts/upload-vs-store.md +++ b/src/pages/docs/concepts/upload-vs-store.md @@ -1,35 +1,41 @@ # Upload vs. Store capabilities in web3.storage -There are two types of content identifiers (CIDs) that web3.storage interacts with: +There are three types of content identifiers (CIDs) that web3.storage interacts with: 1. **Content CIDs**: The CIDs used to reference and access uploads in the format generally useful to users (e.g., files, directories). These CIDs are generally prefixed by `bafy…`. 2. **Shard CIDs**: The CID of the serialized shards of data itself that are produced client-side, sent to web3.storage, and stored. These CIDs are generally prefixed by `bag…`. +3. **Piece CIDs**: The primary means of referencing data stored in a sector of a Filecoin Storage Provider. You can think of them as another way to reference a shard. These CIDs are generally prefixed by `bafk…`. -One example of where you might see both is uploading a large file with the CLI: +One example of where you might see all three is uploading a large file with the CLI: ```sh -w3 up gb.file - 1 file (1GB) -bagbaierao... -bagbaieraq... -bagbaieraj... -bagbaierai... -bagbaierax... -bagbaieraf... -bagbaierac... -bagbaierax... -bagbaierax... -bagbaiera4... - +w3 up gb.file --verbose + 1 file 1.0GB + bagbaierakgivx2igjaydgkdholxohq77gsmjceb44wghn6l7rcxjnxlb2ysq 132.1MB + └── Piece CID: bafkzcibextmt6fvx6vkm56k3je76dirfpwzbqhs75agxen7yidv5lo5usnfdbetzcu + bagbaiera5hekddouwxfgue3wrbe5ddwcefr6l7z3majbqsjk3rxlruziw4nq 132.1MB + └── Piece CID: bafkzcibextmt6fujkiyd45vkvyd2gppji2wzcpsvu72y3sshzppmi2wnxp4q5pblay + bagbaierazfqaaabmnaksetr4ivqu7ytchnikygij7g26cycbjbbnfja6oxfa 132.1MB + └── Piece CID: bafkzcibextmt6ftpb5xepzfvg4ot5bv7uhuplrdbdrwnznpwnpaempif4i76gxbmdm + bagbaiera3elp7gxxbwvgyrgzrjjqsrhefofnr4t2zayktvkamqbjxfuundoa 132.1MB + └── Piece CID: bafkzcibextmt6frkjuqg6pxgqaq2zi5ktwsprflbikk2cnqshxifvsumntmqedgrba + bagbaieraamcctnhnqvwzuf7zhg4jyedg5c76fqn625wi7pico4xtnid5bgba 132.1MB + └── Piece CID: bafkzcibextmt6frloqsvau6qhwzfndrh55ahdjech52ddhspakf4gnxwksuks6uiem + bagbaiera6kel2dvxaiowgtj563kic2ftosnzqwue7etiph242w7w5s262fha 132.1MB + └── Piece CID: bafkzcibextmt6frcrmqxujkfxiy34yutheho55cwhlaqdypafjkhgusphcn5kyp3da + bagbaieranxcdfpqqqvmyugisyxmautsduivwrsm4jg2x2akys4ia3u4oyh7q 59.0MB + └── Piece CID: bafkzcibfzxlncayvf3ndiybf3hpigkfd4o27oqrfgmpr6y7ut5nhcyjdtjhqsvstwyqq + bagbaierakynqcaqedv2brd6qs772s3uqnrxwa62guoeqqjq2bi6kr76vfvla 132.1MB + └── Piece CID: bafkzcibextmt6fvobnshzubgtr4szqsztg35yjkoczy3bry74zfy2bu5fm2x5caefq ⁂ Stored 1 file -⁂ https://w3s.link/ipfs/bafybeiaxwvg4... +⁂ https://w3s.link/ipfs/bafybeidt227tuki2axtb3xcolwd5fw7relv6hfwxrlhgxjzyzwh4cowymy ``` -The CLI sharded the 1GB upload into 10 shards, each with a `bag…`-prefixed CID. The content CID of the file itself is included in the `w3s.link/ipfs/bafy…` link at the bottom. +The CLI sharded the ~1GB upload into 8 shards, each with a `bag…`-prefixed CID. The content CID of the file itself is included in the `w3s.link/ipfs/bafy…` link at the bottom. In the vast majority of cases, users should focus on content CIDs, as this is what they'll be using to fetch their content. If you stick with using the recommended client and CLI methods, then you won't really have to ever worry about the shard CIDs. -However, if you're interested in learning more about how web3.storage uses both, read on! +However, if you're interested in learning more about how web3.storage uses all three, read on! ## Upload vs. Store @@ -68,3 +74,7 @@ web3.storage tracks usage for payment (i.e., how much storage is utilized by a u Fortunately, this shouldn't make things any more complicated - we go into more detail below, but in general, when you remove a content CID from your account, you'll want to remove the shard CIDs as well (e.g., in the client calling `client.remove(contentCID, { shards: true })`). However, if you are a power user interacting with shard CIDs as well (like in the previous section), then you need to be more cautious about removing shard CIDs from your account. (This is why the default for the client and CLI methods is for shards to be maintained after removing a content CID). Learn more about how to do this in the Remove section. + +## When should I care about Piece CIDs? + +Piece CIDs are how you can reference your data when it is stored in Filecoin Storage Providers. You can think of Piece CIDs as another way to reference a shard - they are in fact calculated from shard data. Piece CIDs are used in [PoDSI (Proof of Data Segment Inclusion)](/docs/concepts/podsi/) - a proof that a piece is included in a larger piece, which allows users and third parties to prove their data is stored with a Filecoin Storage Provider. diff --git a/src/pages/docs/how-to/create-account.md b/src/pages/docs/how-to/create-account.md index a37d6e7..f1b5128 100644 --- a/src/pages/docs/how-to/create-account.md +++ b/src/pages/docs/how-to/create-account.md @@ -19,7 +19,7 @@ The easiest way to create an account is by using `w3cli`. 2. If you don't have an account, the website will ask you for your email address. Enter your email address and submit the form. 3. Check your email for a message from web3.storage including a link to confirm your intention to authenticate using the email you provided. -## Using the JS library +## Using the JS client 1. Install the client library from npm using your command line: `npm install @web3-storage/w3up-client`. 2. Call `client.login('alice@example.com')`, and wait for the promise to resolve. This will send an email to your inbox with a link for validation. diff --git a/src/pages/docs/how-to/create-space.md b/src/pages/docs/how-to/create-space.mdx similarity index 90% rename from src/pages/docs/how-to/create-space.md rename to src/pages/docs/how-to/create-space.mdx index 3640942..aec251b 100644 --- a/src/pages/docs/how-to/create-space.md +++ b/src/pages/docs/how-to/create-space.mdx @@ -1,3 +1,5 @@ +import { Callout } from 'nextra/components' + # How to create a space In this how-to guide, you'll learn how to create a web3.storage Space to organize stored data. For an overview of the various ways web3.storage can be integrated with your application, check out [Architecture Options](/docs/concepts-architecture-options/). @@ -26,9 +28,11 @@ Separately, you can visit [console.web3.storage](https://console.web3.storage/), The Space you create can be used to [upload](/docs/how-to/upload/) data using the CLI, the w3up client, or when you log into the web console. -## Using the JS library +## Using the JS client 1. Install the client library from npm using your command line: `npm install @web3-storage/w3up-client`. 2. Call `client.createSpace('Documents')` and wait for the promise to resolve. -Note: the space must be provisioned by an account before it can be used for uploads. See [our guide](/docs/w3up-client/#create-and-register-a-space) for details. + + The space must be provisioned by an account before it can be used for uploads. See [our guide](/docs/w3up-client/#create-and-provision-a-space) for details. + diff --git a/src/pages/docs/how-to/retrieve.mdx b/src/pages/docs/how-to/retrieve.mdx index db54ec3..46f14d5 100644 --- a/src/pages/docs/how-to/retrieve.mdx +++ b/src/pages/docs/how-to/retrieve.mdx @@ -92,4 +92,4 @@ Replace ``, ``, and `` with their respective v ## Next steps -Next, you'll learn about how to [list](/docs/how-to/list/) +Next, you'll learn about how to [list](/docs/how-to/list/) uploaded content. diff --git a/src/pages/docs/how-to/upload.mdx b/src/pages/docs/how-to/upload.mdx index d508d5e..d89c260 100644 --- a/src/pages/docs/how-to/upload.mdx +++ b/src/pages/docs/how-to/upload.mdx @@ -16,15 +16,15 @@ Later in this section, we also cover uploading data using the CLI or web console Removing files from w3up will remove them from the file listing for your account, but that doesn't prevent nodes on the decentralized storage network from retaining copies of the data indefinitely. web3.storage itself generally retains and charges users for any uploaded data for a minimum of 30 days. Do not use w3up for data that may need to be permanently deleted in the future. -## web3.storage API authorization +{/* ## web3.storage API authorization -In the previous section, you created a Space that has a unique DID. To use the client to upload data to this Space, you need to give it permission to do so. This is done by passing an Agent into the client and a UCAN token delegating permissions to this Agent. An Agent has a DID with an underlying private key (like a Space), but can be thought of as parallel to a login session (whereas the Space can be thought of more like a bucket or account). To learn more about how UCAN works and how web3.storage uses it, read our docs on UCAN authorization. +In the previous section, you created a Space that has a unique DID. To use the client to upload data to this Space, you need to give it permission to do so. This is done by passing an Agent into the client and a UCAN token delegating permissions to this Agent. An Agent has a DID with an underlying private key (like a Space), but can be thought of as parallel to a login session (whereas the Space can be thought of more like a bucket or account). To learn more about how UCAN works and how web3.storage uses it, read our docs on UCAN authorization. */} -## Using the Command line `w3cli` +## Using the CLI -If you followed the [Create account](create-account) and [Create Space](create-space) sections, you will already have the CLI set up with a Space. However, you might be using the CLI on a new machine, in which case you can follow these instructions: +If you followed the [Create Account](create-account) and [Create Space](create-space) sections, you will already have the CLI set up with a Space. However, you might be using the CLI on a new machine, in which case you can follow these instructions: -1. (If not yet installed) Install the CLI from npm using your command line: `npm install -g @web3-storage/w3cli`. +1. Install the CLI from npm using your command line: `npm install -g @web3-storage/w3cli`. 2. Run `w3 login alice@example.com` in the command line using your email address. Click on the validation link sent to your email. 3. After successfully running `login`, your CLI Agent has been delegated access to all Spaces associated with your email address. You can see a list of these Spaces using `w3 space ls` and select the one you'd like to upload to using `w3 space use `. @@ -38,18 +38,7 @@ There are a few useful flags (check out the reference docs or `w3 up --help` to -c, --car # File is a CAR file. ``` -## Using the Javascript client - -This section discusses using the web3.storage JavaScript client, `w3up-client`, with your (developer-owned) Space in your application. web3.storage's Javascript client provides a simple interface for storing data using syntax inspired by familiar web APIs such as [fetch](https://developer.mozilla.org/en-US/docs/Web/API/Fetch_API) and [File](https://developer.mozilla.org/en-US/docs/Web/API/File). - -```mermaid -flowchart TD -B[w3up-client instance] --\>|Automatic if specific AgentData is not passed when client object created|S(Pass local AgentData) -S --\> C(Get UCAN delegation from Space to Agent) -C --\> D(Upload to Space using Agent) -``` - -All uses of `w3up-client` to upload with web3.storage follow the flow above. +## Using the JS client ### Installing the client @@ -63,36 +52,44 @@ npm install @web3-storage/w3up-client The package provides a [static create function](https://web3-storage.github.io/w3up/functions/_web3_storage_w3up_client.create.html) that returns a [Client object](https://web3-storage.github.io/w3up/classes/_web3_storage_w3up_client.Client.html). How you initialize it depends on the environment the client is used in: **persistent** or **ephemeral**. -Examples of persistent environments: +Examples of **persistent** environments: * A browser application * A terminal application * An installed application (e.g. Electron) -Examples of ephemeral environments: +Examples of **ephemeral** environments: * AWS Lambda or server side workers * Running inside Docker instances * CI -### Claim delegation via email validation: For persistent environment only +#### Claim delegations via email validation + + + For persistent environment only + + +A new client can claim access to their existing Spaces by validating their email address. ```mermaid sequenceDiagram Client-\>\>web3.storage service: Here is my email address and Agent DID web3.storage service--\>\>Client: Please click the link to validate Client--\>\>web3.storage service: Email address validated -web3.storage service-\>\>Client: Here is a UCAN delegating permission from Space DID to Agent DID +web3.storage service-\>\>Client: Here is a UCAN attesting that your Agent DID belongs to your email ``` -You can use web3.storage's email authorization flow to give permissions to your server-side client. This can be good if your environment will be persistent (otherwise it would be prohibitive to click an email validation link every time the client is re-instantiated). +You can use web3.storage's email authorization flow to give permissions to your client. This can be good if your environment will be persistent (otherwise it would be prohibitive to click an email validation link every time the client is re-instantiated). + +When a Space is created, access permissions are delegated to your email address. We use a special kind of DID for this, a `did:mailto:`. These UCANs are stashed with the web3.storage service. When you validate your email address with a new Agent DID, web3.storage issues a UCAN attestation, that says your Agent DID is owned by your email address. It also returns the UCAN permissions you previously stashed. You can then use the returned UCANs, along with the attestation to prove you are authorized to perform actions. ```javascript import { create } from '@web3-storage/w3up-client' const client = await create() ``` -By default, constructing a client like this will re-use state persisted by other clients because `create` constructs the w3up [Client](https://web3-storage.github.io/w3up/classes/_web3_storage_w3up_client.Client.html)'s access [Agent](https://web3-storage.github.io/w3up/classes/_web3_storage_access.Agent.html) with [AgentData](https://web3-storage.github.io/w3up/classes/_web3_storage_access.AgentData.html) backed by a storage [Driver](https://web3-storage.github.io/w3up/interfaces/_web3_storage_access.unknown.Driver.html) ([IndexedDBDriver](https://web3-storage.github.io/w3up/classes/_web3_storage_access.unknown.IndexedDBDriver.html) in the browser and [ConfDriver](https://web3-storage.github.io/w3up/classes/_web3_storage_access.unknown.ConfDriver.html) otherwise) that persists the AgentData between processes and client instances. +By default, constructing a client like this will re-use state persisted by other clients because `create` constructs the client with a store that persists data between processes and client instances. Once you have created a client, you can login with your email address. Calling login will cause an email to be sent to the given address. @@ -106,9 +103,11 @@ Once a user clicks the confirmation link in the email, the login method will res await client.setCurrentSpace('did:key:...') // select the relevant Space DID that is associated with your account ``` -### Bring your own Agent +#### Bring your own delegations -For any backend (including non-persistent and/or serverless) + + For any backend (including non-persistent and/or serverless) + ```mermaid sequenceDiagram @@ -117,14 +116,15 @@ Developer-\>\>Developer: Delegate UCAN from Space to Agent Developer-\>\>Client: Here is my Agent private key and UCAN delegating permissions ``` -An option that works for any backend environment is to define an Agent and delegate a UCAN from your Space to this Agent before initializing the client. This is especially useful if you're using the client in a serverless Node environment (e.g., AWS Lambda). +An option that works for any backend environment is for a developer to create and provision a Space, and then delegate access to a different Agent DID that will be used by the client. This is especially useful if you're using the client in a serverless environment (e.g., AWS Lambda). In your command line where `w3cli` is configured with the Space you want to use (e.g., where you created the Space): ```shell # The following command returns what will be your Agent private key and DID -# store the private key in environment variable KEY -npx ucan-key ed --json +npx ucan-key ed + +# ❗️ Store the private key (starting "Mg...") in environment variable KEY # The following command creates a UCAN delegation from the w3cli agent to the # agent you generated above. @@ -137,7 +137,7 @@ npx ucan-key ed --json # just being able to upload. w3 delegation create | base64 -# Store the output in environment variable PROOF +# ❗️ Store the output in environment variable PROOF ``` Then, when you initialize and configure the client, you can pass in this Agent and UCAN. @@ -157,7 +157,7 @@ async function main () { // READY to go! } -/** @param {string} data Base64 encoded CAR file \*/ +/** @param {string} data Base64 encoded CAR file */ async function parseProof (data) { const blocks = [] const reader = await CarReader.fromBytes(Buffer.from(data, 'base64')) @@ -188,8 +188,8 @@ There are two main options to getting content into your Space: ```mermaid sequenceDiagram -User-\>\>w3up-client in backend: Upload data -w3up-client in backend-\>\>web3.storage service: Upload data +User-\>\>Application Backend: Upload data +Application Backend-\>\>web3.storage service: Upload data ``` You are already set up to upload using your client instance as data becomes available to your backend - you can call `uploadFile` or `uploadDirectory` with it. @@ -212,22 +212,23 @@ In the example above, `directoryCid` resolves to an IPFS directory. ```mermaid sequenceDiagram -participant w3up-client in user -participant w3up-client in backend +participant User +participant Application Backend participant web3.storage service -w3up-client in user-\>\>w3up-client in user: Client instantiated with default Agent -w3up-client in user-\>\>w3up-client in backend: Request delegation with user's Agent DID -w3up-client in backend-\>\>w3up-client in user: Send delegation from Space to user's Agent DID -w3up-client in user-\>\>web3.storage service: Upload data +User-\>\>User: Client instantiated with default Agent +User-\>\>Application Backend: Request delegation with user's Agent DID +Application Backend-\>\>User: Send delegation from Space to user's Agent DID +User-\>\>web3.storage service: Upload data ``` Your backend instance can also be used to delegate upload permissions directly to your user to upload. The code snippet below shows an example of how you might set up a client instance in your application frontend and how it might interact with your backend client. You can see how the frontend client Agent DID is used for the backend client to delegate permissions to; from there, it will be the frontend client that will call the `upload` method. -```javascript +**Backend** + +```js import { CarReader } from '@ipld/car' import * as DID from '@ipld/dag-ucan/did' import * as Delegation from '@ucanto/core/delegation' -import { importDAG } from '@ucanto/core/delegation' import * as Signer from '@ucanto/principal/ed25519' import * as Client from '@web3-storage/w3up-client' @@ -244,10 +245,8 @@ async function backend(did) { // Create a delegation for a specific DID const audience = DID.parse(did) const abilities = ['store/add', 'upload/add'] - const expiration = Math.floor(Date.now() / 1000) + 60 * 60 * 24 // 24 hours from now - const delegation = await client.createDelegation(audience, abilities, { - expiration, - }) + const expiration = Math.floor(Date.now() / 1000) + (60 * 60 * 24) // 24 hours from now + const delegation = await client.createDelegation(audience, abilities, { expiration }) // Serialize the delegation and send it to the client const archive = await delegation.archive() @@ -261,8 +260,15 @@ async function parseProof(data) { for await (const block of reader.blocks()) { blocks.push(block) } - return importDAG(blocks) + return Delegation.importDAG(blocks) } +``` + +**Frontend** + +```js +import * as Delegation from '@ucanto/core/delegation' +import * as Client from '@web3-storage/w3up-client' async function frontend() { // Create a new client @@ -276,7 +282,7 @@ async function frontend() { // Deserialize the delegation const delegation = await Delegation.extract(new Uint8Array(data)) if (!delegation.ok) { - throw new Error('Failed to extract delegation') + throw new Error('Failed to extract delegation', { cause: delegation.error }) } // Add proof that this agent has been delegated capabilities on the space @@ -291,11 +297,13 @@ async function frontend() { You are now ready to upload using the client! In general, the easiest way to upload data is using the `uploadFile` or `uploadDirectory` method. -`uploadFile` expects a "Blob like" input, which can be a [Blob](https://developer.mozilla.org/en-US/docs/Web/API/Blob) or [File](https://developer.mozilla.org/en-US/docs/Web/API/File) when running in a browser. On node.js, see the [filesFromPath library](https://github.com/web3-storage/files-from-path), which can load compatible objects from the local filesystem. By default, files uploaded to web3.storage will be wrapped in an IPFS directory listing. This preserves the original filename and makes links more human-friendly than CID strings, which look like random gibberish. +`uploadFile` expects a "Blob like" input, which can be a [Blob](https://developer.mozilla.org/en-US/docs/Web/API/Blob) or [File](https://developer.mozilla.org/en-US/docs/Web/API/File) when running in a browser. On Node.js, see the [`files-from-path` library](https://github.com/web3-storage/files-from-path), which can load compatible objects from the local filesystem. By default, files uploaded to web3.storage will be wrapped in an IPFS directory listing. This preserves the original filename and makes links more human-friendly than CID strings, which look like random gibberish. `uploadDirectory` requires `File`-like objects instead of `Blob`s, as the file's name property is used to build the directory hierarchy. -**Tip:** When uploading multiple files, give each file a unique name. All the files in a `uploadDirectory` request will be bundled into one content archive, and linking to the files inside is much easier if each file has a unique, human-readable name. + + When uploading multiple files, give each file a unique name. All the files in a `uploadDirectory` request will be bundled into one content archive, and linking to the files inside is much easier if each file has a unique, human-readable name. + You can control the directory layout and create nested directory structures by using / delimited paths in your filenames: @@ -317,7 +325,7 @@ In the example above, directoryCid resolves to an IPFS directory with the follow | └──example.png ├──readme.md └──src -└──main.py + └──main.py ``` There are a few different ways of creating `File` objects available, depending on your platform. @@ -356,8 +364,6 @@ In Node.js, the [`files-from-path` library](https://github.com/web3-storage/file import { filesFromPaths } from 'files-from-path' ``` -If you expect to be loading a lot of large files, you may be better served by the [filesFromPath](https://github.com/web3-storage/files-from-path#filesfrompath)[helper](https://github.com/web3-storage/files-from-path#filesfrompath). It reduces memory pressure by yielding File objects one by one as they're loaded from disk, instead of loading everything into memory. You can then issue multiple put requests to send each file to web3.storage. - You can also manually create File objects by importing a Node.js implementation of File from the web3.storage package. This is useful when you want to store data created by your application, instead of files from the user's computer. ```javascript