diff --git a/src/pages/docs/concepts/architecture-options.md b/src/pages/docs/concepts/architecture-options.md deleted file mode 100644 index 76d5e06..0000000 --- a/src/pages/docs/concepts/architecture-options.md +++ /dev/null @@ -1,136 +0,0 @@ -# Possible architectures using web3.storage to upload - -UCAN opens up a number of options in how to integrate with w3up: Should you, the developer, own the Space? Should you delegate permissions to your users? Or should your user own their own Space? Broadly, there are three ways to integrate: - -- Client-server: You (the developer) own the Space, and your user uploads to your backend infra before you upload it to the service. -- Delegated: You own the Space, but you give a delegated UCAN token to your user's Agent to upload directly to the service, rather than needing to proxy the upload through your backend (no egress from your infrastructure). -- User-owned: Your user owns the Space and registers it and they use it to upload directly with the service; if you want to instrument visibility into what they're uploading, you'll have to write separate code in your app for it. - -In the How-tos section of the docs, we focused on the first two options, as they are the most familiar today. However, you can implement each of these in a number of ways, but we talk through some considerations when implementing a given option. - -## Client-server - -```mermaid -sequenceDiagram -participant User -w3up-client in backend-\>\>w3up-client in backend: Client set with Agent with delegation from Space -User-\>\>w3up-client in backend: Upload data -w3up-client in backend-\>\>web3.storage w3up service: Upload data -``` - -- For your backend to be scalable, you might consider using serverless workers or a queue in front of a server -- In either case, you'll need a registered Space, and your client instance in your backend to have an Agent with a delegation from this Space - - (Recommended) It's likely easiest to create and register your Space using [w3cli](https://github.com/web3-storage/w3cli) rather than using `w3up-client` to do so (especially if your backend isn't persistent); you can then generate your own Agent and delegate the ability to upload to your Space using something like [this example](https://github.com/web3-storage/w3up/tree/main/packages/w3up-client#bringing-your-own-agent-and-delegation) - - If your backend is persistent, you can do this or do everything in the client directly ([create Space](https://github.com/web3-storage/w3up/tree/main/packages/w3up-client#creating-and-registering-spaces) and [get delegation](https://github.com/web3-storage/w3up/tree/main/packages/w3up-client#delegating-from-space-to-agent)) -- After this, once your user uploads data to your backend, you can run any of the upload methods - -## Delegated - -```mermaid -sequenceDiagram -participant w3up-client in user -participant w3up-client in backend -participant web3.storage w3up service -w3up-client in backend-\>\>w3up-client in backend: Client created with Agent and delegation from Space -w3up-client in user-\>\>w3up-client in user: Client instantiated with default Agent -w3up-client in user-\>\>w3up-client in backend: Request delegation with user's Agent DID -w3up-client in backend-\>\>w3up-client in user: Send delegation from Space to user's Agent DID -w3up-client in user-\>\>web3.storage w3up service: Upload data -``` - -- You will likely have `w3up-client` running in your end-user's client code, as well as backend code that's able to generate UCANs that delegate the ability to upload and pass them to your users (e.g., `w3up-client` running in a serverless worker) -- For your backend to be scalable, you might consider using serverless workers or a queue in front of a server -- As the developer, you'll need a registered Space, and your client instance in your backend to have an Agent with a delegation from this Space - - (Recommended) It's likely easiest to create and register your Space using [`w3cli`](https://github.com/web3-storage/w3cli) rather than using `w3up-client` to do so (especially if your backend isn't persistent); you can then generate your own Agent and delegate the ability to upload to your Space using something like [this example](https://github.com/web3-storage/w3up/tree/main/packages/w3up-client#bringing-your-own-agent-and-delegation) - - If your backend is persistent, you can do this or do everything in the client directly ([create Space](https://github.com/web3-storage/w3up/tree/main/packages/w3up-client#creating-and-registering-spaces) and [get delegation](https://github.com/web3-storage/w3up/tree/main/packages/w3up-client#delegating-from-space-to-agent)) -- Your user does not need a registered Space - just an Agent with a delegation from your Space - - `w3up-client` in the end user environment should have a unique Agent for each user, which should happen by default (since when `w3up-client` is instantiated it creates a new Agent anyway, or uses the one in local Store) - - From there, when your end user is ready to upload, they should request from your backend a delegation from your developer-owned Space to their Agent (which can be derived via [client.agent()](https://github.com/web3-storage/w3up/blob/main/packages/w3up-client/docs-Client#agent)) - - In your backend, you can call [client.createDelegation()](https://github.com/web3-storage/w3up/blob/main/packages/w3up-client/docs-Client#createDelegation) passing in the Agent object from `client.agent()` in your end user's instance, and passing through options? params to limit the scope of the delegation (e.g., `store/add`, `upload/add`, expiration time) - - You can serialize this using `delegation.archive()` and send it to your user - - The end user instance of the client should not need to call `client.authorize(email)`, as it is not claiming any delegations via email address (but rather getting the delegation directly from your backend) -- Once your user receives the delegation, they can deserialize it using [`ucanto.Delegation.extract()`](https://github.com/web3-storage/ucanto/blob/c8999a59852b61549d163532a83bac62290b629d/packages/core/src/delegation.js#L399) and pass it in using `client.addSpace()`, and from there they can run any of the upload methods - - Note that this alone does not give visibility into which of your end users are uploading what; to track this, you'll probably need them to send you that information separately (e.g., once they've run upload and get back a content CID, you can have them send that CID to you for tracking) -- A code example that does this can be found below - -```js -import { CarReader } from '@ipld/car' -import * as DID from '@ipld/dag-ucan/did' -import * as Delegation from '@ucanto/core/delegation' -import { importDAG } from '@ucanto/core/delegation' -import * as Signer from '@ucanto/principal/ed25519' -import * as Client from '@web3-storage/w3up-client' - -async function backend(did: string) { - // Load client with specific private key - const principal = Signer.parse(process.env.KEY) - const client = await Client.create({ principal }) - - // Add proof that this agent has been delegated capabilities on the space - const proof = await parseProof(process.env.PROOF) - const space = await client.addSpace(proof) - await client.setCurrentSpace(space.did()) - - // Create a delegation for a specific DID - const audience = DID.parse(did) - const abilities = ['store/add', 'upload/add'] - const expiration = Math.floor(Date.now() / 1000) + 60 \* 60 \* 24 // 24 hours from now - const delegation = await client.createDelegation(audience, abilities, { expiration }) - - // Serialize the delegation and send it to the client - const archive = await delegation.archive() - - return archive.ok -} - -/** @param {string} data Base64 encoded CAR file */ -async function parseProof(data) { - const blocks = [] - const reader = await CarReader.fromBytes(Buffer.from(data, 'base64')) - for await (const block of reader.blocks()) { - blocks.push(block) - } - return importDAG(blocks) -} - -async function frontend() { - // Create a new client - const client = await Client.create() - - // Fetch the delegation from the backend - const apiUrl = `/api/w3up-delegation/${client.agent().did()}` - const response = await fetch(apiUrl) - const data = await response.arrayBuffer() - - // Deserialize the delegation - const delegation = await Delegation.extract(new Uint8Array(data)) - if (!delegation.ok) { - throw new Error('Failed to extract delegation') - } - - // Add proof that this agent has been delegated capabilities on the space - const space = await client.addSpace(delegation.ok) - client.setCurrentSpace(space.did()) - - // READY to go! -} -``` - -## User-owned - -```mermaid -sequenceDiagram -participant User -participant Application backend -participant web3.storage w3up service -Application backend-\>\>User: Front end code that includes w3up-client -User-\>\>web3.storage w3up service: (If needed) Create Space and register it -User-\>\>web3.storage w3up service: (If needed) Use Agent email verification to "log in" to Space -User-\>\>web3.storage w3up service: Upload data using w3up-client -``` - -- If you want your user to own their own Space, you'll likely be relying on the w3up-client methods to create a Space, authorize the Space, and authorize the Agent on the end user-side; from there they can run any of the upload methods - - Doing this does take some of the UX out of your control; for instance, when web3.storage fully launches with w3up, your users will have to set up their payment methods with web3.storage - - Note that this alone does not give visibility into which of your end users are uploading what; to track this, you'll probably need them to send you that information separately (e.g., once they've run upload and get back a content CID, you can have them send that CID to you for tracking) -- There is a world of possibilities with your users "bringing their own identity" for their Space; you could explore how crypto wallet private keys, Apple Passkey, and more might map to Space DIDs and have the client use those -- If you have code snippet(s) that works for you, please share them in a PR or [Github issue](https://github.com/web3-storage/www/issues) and we'll link them here! diff --git a/src/pages/docs/concepts/architecture-options.mdx b/src/pages/docs/concepts/architecture-options.mdx new file mode 100644 index 0000000..93473ba --- /dev/null +++ b/src/pages/docs/concepts/architecture-options.mdx @@ -0,0 +1,173 @@ +import { Callout } from 'nextra/components' + +# Possible architectures using web3.storage to upload + +UCAN opens up a number of options in how to integrate with w3up: Should you, the developer, own the Space? Should you delegate permissions to your users? Or should your user own their own Space? Broadly, there are three ways to integrate: + +- **Client-server**: You (the developer) own the space, your users upload data to your application server, and you upload it to web3.storage from there. +- **Delegated**: You own the Space, but you provide your user with a delegation that allows them to upload directly to the service. This avoids proxying the upload through your backend (no egress from your infrastructure). +- **User-owned**: Your user owns the Space and registers it and they use it to upload directly with the service; if you want to instrument visibility into what they're uploading, you'll have to write separate code in your app for it. + +In the How-tos section of the docs, we focused on the first two options, as they are the most typical today. However, you can implement each of these in a number of ways, but we talk through some considerations when implementing a given option. + +## Client-server + +In this set up, you (the developer) own the space, your users upload data to your application server, and you upload it to web3.storage from there. + +```mermaid +sequenceDiagram +participant User +w3up-client in backend-\>\>w3up-client in backend: Client set with Agent with delegation from Space +User-\>\>w3up-client in backend: Upload data +w3up-client in backend-\>\>web3.storage w3up service: Upload data +``` + +You'll need a registered Space, and your client in the backend to have a delegation to use the Space. + + + **Recommended**: It's easiest to create and register your Space using [w3cli](https://github.com/web3-storage/w3cli), then create an identity for your server and delegate it the ability to upload to your Space. [This example](https://github.com/web3-storage/w3up/tree/main/packages/w3up-client#bringing-your-own-agent-and-delegation) shows how to load an identity and a delegation into the client on the server. + + +After this, once your user uploads data to your backend, you can run any of the upload methods. + +For your backend to be scalable, you might consider using serverless workers or a queue in front of a server. + +## Delegated + +In this set up, you own the Space, but you provide your user with a delegation that allows them to upload directly to the service. This avoids proxying the upload through your backend (no egress from your infrastructure). + +```mermaid +sequenceDiagram +participant w3up-client in user +participant w3up-client in backend +participant web3.storage w3up service +w3up-client in backend-\>\>w3up-client in backend: Client created with Agent and delegation from Space +w3up-client in user-\>\>w3up-client in user: Client instantiated with default Agent +w3up-client in user-\>\>w3up-client in backend: Request delegation with user's Agent DID +w3up-client in backend-\>\>w3up-client in user: Send delegation from Space to user's Agent DID +w3up-client in user-\>\>web3.storage w3up service: Upload data +``` + +Typically `w3up-client` will be running in your end-user's client code, as well as backend code that's able to generate UCANs that delegate the ability to upload and pass them to your users (e.g., `w3up-client` running in a serverless worker). + +You'll need a registered Space, and your client in the backend to have a delegation to use the Space. + + + **Recommended**: It's easiest to create and register your Space using [w3cli](https://github.com/web3-storage/w3cli), then create an identity for your server and delegate it the ability to upload to your Space. [This example](https://github.com/web3-storage/w3up/tree/main/packages/w3up-client#bringing-your-own-agent-and-delegation) shows how to load an identity and a delegation into the client on the server. + + +Your backend code can then re-delegate it's capabilities to your users. Your user does not need a registered Space - just an Agent with a delegation from your Space. + +When your user is ready to upload, they should request a delegation from your backend to upload to your Space. They must send their Agent DID so that the delegation is restricted to be used by only them. + +In your backend, you can call [`client.createDelegation(...)`](https://github.com/web3-storage/w3up/blob/main/packages/w3up-client/docs-Client#createDelegation) passing in the Agent object from `client.agent()` in your end user's instance, and passing params to limit the scope of the delegation (e.g., `store/add`, `upload/add`, expiration time) + +You can serialize the delegation using `delegation.archive()` and send it to your user. + +The client the user is using does not need to call `client.login(email)`, as it is not claiming any delegations via email address (but instead getting a delegation from your backend). + + +When your user receives the delegation, they can deserialize it using [`Delegation.extract()`](https://github.com/web3-storage/ucanto/blob/c8999a59852b61549d163532a83bac62290b629d/packages/core/src/delegation.js#L399) and pass it in using `client.addSpace()`, and from there they can run any of the upload methods. + + + This does not provide visibility over which users are uploading what; to track this, you'll need to share that information separately (e.g., once they've run upload and get back a content CID, you can have them send that CID to you for tracking). + + +A code example that does this can be found below: + +**Backend** + +```js +import { CarReader } from '@ipld/car' +import * as DID from '@ipld/dag-ucan/did' +import * as Delegation from '@ucanto/core/delegation' +import * as Signer from '@ucanto/principal/ed25519' +import * as Client from '@web3-storage/w3up-client' + +async function backend(did) { + // Load client with specific private key + const principal = Signer.parse(process.env.KEY) + const client = await Client.create({ principal }) + + // Add proof that this agent has been delegated capabilities on the space + const proof = await parseProof(process.env.PROOF) + const space = await client.addSpace(proof) + await client.setCurrentSpace(space.did()) + + // Create a delegation for a specific DID + const audience = DID.parse(did) + const abilities = ['store/add', 'upload/add'] + const expiration = Math.floor(Date.now() / 1000) + (60 * 60 * 24) // 24 hours from now + const delegation = await client.createDelegation(audience, abilities, { expiration }) + + // Serialize the delegation and send it to the client + const archive = await delegation.archive() + return archive.ok +} + +/** @param {string} data Base64 encoded CAR file */ +async function parseProof(data) { + const blocks = [] + const reader = await CarReader.fromBytes(Buffer.from(data, 'base64')) + for await (const block of reader.blocks()) { + blocks.push(block) + } + return Delegation.importDAG(blocks) +} +``` + +**Frontend** + +```js +import * as Delegation from '@ucanto/core/delegation' +import * as Client from '@web3-storage/w3up-client' + +async function frontend() { + // Create a new client + const client = await Client.create() + + // Fetch the delegation from the backend + const apiUrl = `/api/w3up-delegation/${client.agent().did()}` + const response = await fetch(apiUrl) + const data = await response.arrayBuffer() + + // Deserialize the delegation + const delegation = await Delegation.extract(new Uint8Array(data)) + if (!delegation.ok) { + throw new Error('Failed to extract delegation', { cause: delegation.error }) + } + + // Add proof that this agent has been delegated capabilities on the space + const space = await client.addSpace(delegation.ok) + client.setCurrentSpace(space.did()) + + // READY to go! +} +``` + +## User-owned + +In this set up your user owns the Space and registers it and they use it to upload directly with the service; if you want to instrument visibility into what they're uploading, you'll have to write separate code in your app for it. + +```mermaid +sequenceDiagram +participant User +participant Application backend +participant web3.storage w3up service +Application backend-\>\>User: Front end code that includes w3up-client +User-\>\>web3.storage w3up service: (If needed) Create Space and register it +User-\>\>web3.storage w3up service: (If needed) Use Agent email verification to "log in" to Space +User-\>\>web3.storage w3up service: Upload data using w3up-client +``` + +If you want your user to own their own Space, you'll typically be relying on the `w3up-client` methods to create a Space, authorize the Space, and authorize the Agent on the user-side; afterwards they can run any of the upload methods. + + + This does not provide visibility over which users are uploading what; to track this, you'll need to share that information separately (e.g., once they've run upload and get back a content CID, you can have them send that CID to you for tracking). + + +There is a world of possibilities when users "bring their own" Space; you could explore how crypto wallet private keys, Apple Passkey, and more might map to Space DIDs and have the client use those. Your users could delegate capabilities for their space to your backend application or share spaces between friends by delegating to each other. + +--- + +If you have code snippet(s) for any of the above architectural options, please share them in a PR or [Github issue](https://github.com/web3-storage/www/issues) and we'll link them here! diff --git a/src/pages/docs/concepts/car.md b/src/pages/docs/concepts/car.md index 2dbaacc..064df71 100644 --- a/src/pages/docs/concepts/car.md +++ b/src/pages/docs/concepts/car.md @@ -22,7 +22,7 @@ First, formatting everything on the client allows us to calculate the root Conte Another reason to use CARs is to support large files, which would otherwise hit size limits on the web3.storage backend platform. The data in a CAR is already chunked into small blocks, which makes CARs easy to shard into small pieces that can be uploaded in batches. This also enables the web3.storage platform to get larger content into Filecoin deals. -CAR files are a format that pretty much any IPFS tool or implementation can interact with. You can export data from your personal IPFS node into a CAR file and upload it to web3.storage using `w3 up --car` or `client.uploadCar`. As a result, we dive into the various ways you might interact with CAR files. +CAR files are a format that pretty much any IPFS tool or implementation can interact with. You can export data from your personal IPFS node into a CAR file and upload it to web3.storage using `w3 up --car` or `client.uploadCAR`. As a result, we dive into the various ways you might interact with CAR files. ## Command line tools @@ -44,13 +44,13 @@ The --pack flag will create a new CAR file from a collection of input files: ipfs-car pack path/to/files --output path/to/write/a.car ``` -Extract files from a CAR with --unpack: +Extract files from a CAR with `--unpack`: ```sh ipfs-car unpack path/to/my.car --output path/to/unpack/files/to ``` -List the contents of a CAR with --list: +List the contents of a CAR with `--list`: ```sh ipfs-car list path/to/my.car @@ -83,15 +83,15 @@ ipfs dag import path/to/input.car ## Javascript libraries -### ipfs-car +### `ipfs-car` -The ipfs-car package includes library functions for packing and unpacking files into CARs, using the IPFS UnixFs data model. The library includes the same functionality as the ipfs-car command line utility [described above](https://web3.storage/docs/how-tos/work-with-car-files/#ipfs-car). +The [`ipfs-car`](https://github.com/web3-storage/ipfs-car) package includes library functions for packing and unpacking files into CARs, using the IPFS UnixFs data model. The library includes the same functionality as the ipfs-car command line utility [described above](https://web3.storage/docs/how-tos/work-with-car-files/#ipfs-car). -See the [ipfs-car README](https://github.com/web3-storage/ipfs-car#api) for API documentation and usage examples. +See the `ipfs-car` [README](https://github.com/web3-storage/ipfs-car#api) for API documentation and usage examples. -### @ipld/car +### `@ipld/car` -The [@ipld/car](https://github.com/ipld/js-car)[package](https://github.com/ipld/js-car) contains the main JavaScript implementation of the CAR specification and is used by ipfs-car under the hood. If you want to store non-file data using [advanced IPLD formats](https://web3.storage/docs/how-tos/work-with-car-files/#advanced-ipld-formats), you should use @ipld/car directly. +The [`@ipld/car`](https://github.com/ipld/js-car) package contains the main JavaScript implementation of the CAR specification and is used by ipfs-car under the hood. If you want to store non-file data using [advanced IPLD formats](https://web3.storage/docs/how-tos/work-with-car-files/#advanced-ipld-formats), you should use @ipld/car directly. @ipld/car also provides the CarReader interface used by the web3.storage client's [putCar](https://web3.storage/docs/reference/js-client-library/#store-car-files)[method](https://web3.storage/docs/reference/js-client-library/#store-car-files). diff --git a/src/pages/docs/concepts/content-addressing.md b/src/pages/docs/concepts/content-addressing.md index 050e86e..e24834a 100644 --- a/src/pages/docs/concepts/content-addressing.md +++ b/src/pages/docs/concepts/content-addressing.md @@ -4,9 +4,9 @@ web3.storage's decentralized file storage relies on _content addressing_ to find ## The basic problem -Consider what happens when you resolve a link like web3.storage/docs/concepts/content-addressing. First, your operating system queries a global shared key-value store, split into many domains — you may know this as the Domain Name System (DNS). The DNS returns an IP address that your network card can use to send HTTP requests over the network, where this site's naming conventions turn the key /concepts/content-addressing into a response payload. +Consider what happens when you resolve a link like `web3.storage/docs/concepts/content-addressing`. First, your operating system queries a global shared key-value store, split into many domains — you may know this as the Domain Name System (DNS). The DNS returns an IP address that your network card can use to send HTTP requests over the network, where this site's naming conventions turn the key `/docs/concepts/content-addressing` into a response payload. -The problem is, components of an address like web3.storage/docs/concepts/content-addressing are _mutable_, meaning they can change over time. In the context of the web, where _everything_ is mutable and dynamic, this is just the way it's always been. As a result, [link rot](https://en.wikipedia-on-ipfs.org/wiki/Link_rot) is just something we've all learned to live with. +The problem is, components of an address like `web3.storage/docs/concepts/content-addressing` are _mutable_, meaning they can change over time. In the context of the web, where _everything_ is mutable and dynamic, this is just the way it's always been. As a result, [link rot](https://en.wikipedia-on-ipfs.org/wiki/Link_rot) is just something we've all learned to live with. ## CIDs: Location-independent, globally unique keys diff --git a/src/pages/docs/concepts/ipfs-gateways.md b/src/pages/docs/concepts/ipfs-gateways.md index fdb00e1..bbd38b8 100644 --- a/src/pages/docs/concepts/ipfs-gateways.md +++ b/src/pages/docs/concepts/ipfs-gateways.md @@ -8,7 +8,7 @@ To make IPFS data accessible outside of the peer-to-peer network, [special IPFS As more browsers like [Brave](https://brave.com/ipfs-support/) and [Opera](https://blogs.opera.com/tips-and-tricks/2021/02/opera-crypto-files-for-keeps-ipfs-unstoppable-domains/) adopt native IPFS support, the need for gateways will naturally lessen over time. Today, you can reach the widest audience by using HTTP gateways in your web applications, but it's a great idea to also surface the original ipfs:// URI for your content, so that IPFS-native browsers can access the content directly through Bitswap. -For more information about fetching content that you uploaded through an IPFS HTTP gateway, see the Retrieve section. +For more information about fetching content that you uploaded through an IPFS HTTP gateway, see the [Retrieve section](/docs/how-to/retrieve). ## Types of gateway @@ -36,6 +36,6 @@ A "subdomain style" gateway URL puts the CID into the host portion of the URL, a If the CID points to a directory listing, you can use the path portion of the URL to specify the filename: - +https://bafybeid4gwmvbza257a7rx52bheeplwlaogshu4rgse3eaudfkfm7tx2my.ipfs.w3s.link/hi-gateway.txt This is the preferred style for serving web assets over HTTP gateways, because web browsers provide security isolation on a per-domain basis. Using the subdomain style, every CID gets its own "namespace" for things like cookies and local storage, which isolates things from other web content stored on IPFS. diff --git a/src/pages/docs/concepts/ucans-and-web3storage.md b/src/pages/docs/concepts/ucans-and-web3storage.md index 19160e1..2048aa8 100644 --- a/src/pages/docs/concepts/ucans-and-web3storage.md +++ b/src/pages/docs/concepts/ucans-and-web3storage.md @@ -26,17 +26,18 @@ The delegation from a Space to your Agent that w3up-client needs can be passed e ### Delegation to other actors -Just like Spaces can delegate permissions to Agents you own, you can also delegate permissions to other actors' Agents. One common application of this could be you delegating permission to upload to your Space to your users. Here's a code snippet demonstrating this from the Upload section: +Just like Spaces can delegate permissions to Agents you own, you can also delegate permissions to other actors' Agents. One common application of this could be you delegating permission to upload to your Space to your users. Here's a code snippet demonstrating this: -```javascript +**Backend** + +```js import { CarReader } from '@ipld/car' import * as DID from '@ipld/dag-ucan/did' import * as Delegation from '@ucanto/core/delegation' -import { importDAG } from '@ucanto/core/delegation' import * as Signer from '@ucanto/principal/ed25519' import * as Client from '@web3-storage/w3up-client' -async function backend(did: string) { +async function backend(did) { // Load client with specific private key const principal = Signer.parse(process.env.KEY) const client = await Client.create({ principal }) @@ -49,10 +50,8 @@ async function backend(did: string) { // Create a delegation for a specific DID const audience = DID.parse(did) const abilities = ['store/add', 'upload/add'] - const expiration = Math.floor(Date.now() / 1000) + 60 * 60 * 24 // 24 hours from now - const delegation = await client.createDelegation(audience, abilities, { - expiration, - }) + const expiration = Math.floor(Date.now() / 1000) + (60 * 60 * 24) // 24 hours from now + const delegation = await client.createDelegation(audience, abilities, { expiration }) // Serialize the delegation and send it to the client const archive = await delegation.archive() @@ -66,22 +65,34 @@ async function parseProof(data) { for await (const block of reader.blocks()) { blocks.push(block) } - return importDAG(blocks) + return Delegation.importDAG(blocks) } +``` + +When the `backend` function is called in the developer's backend: +- It's passed the DID of the user's Agent +- Backend client initializes with an Agent that has permission to the developer's Space +- It then generates a UCAN delegated to the user Agent DID passed in with only the `store/add` and `upload/add` abilities (to give the user ability to upload) and set to expire in 24 hours + +**Frontend** + +```js +import * as Delegation from '@ucanto/core/delegation' +import * as Client from '@web3-storage/w3up-client' async function frontend() { // Create a new client const client = await Client.create() // Fetch the delegation from the backend - const apiUrl = `/api/w3up-delegation/${client.agent().did()}` // backend method is exposed at this API URL + const apiUrl = `/api/w3up-delegation/${client.agent().did()}` const response = await fetch(apiUrl) const data = await response.arrayBuffer() // Deserialize the delegation const delegation = await Delegation.extract(new Uint8Array(data)) if (!delegation.ok) { - throw new Error('Failed to extract delegation') + throw new Error('Failed to extract delegation', { cause: delegation.error }) } // Add proof that this agent has been delegated capabilities on the space @@ -92,16 +103,10 @@ async function frontend() { } ``` -You can see the following flow: - -- When `backend` function is called in the developer's backend: - - It's passed the DID of the user's Agent - - Backend client initializes with an Agent that has permission to the developer's Space - - It then generates a UCAN delegated to the user Agent DID passed in with only the `store/add` and `upload/add` abilities (to give the user ability to upload) and set to expire in 24 hours -- When `frontend` function is called in the user's environment: - - An Agent DID is created - - The `backend` function hosted at an API endpoint is called, passing in the Agent DID - - The client is set up with a UCAN delegating upload capabilities to the Agent - - It's now ready to upload! +When the `frontend` function is called in the user's environment: +- An Agent DID is created +- The `backend` function hosted at an API endpoint is called, passing in the Agent DID +- The client is set up with a UCAN delegating upload capabilities to the Agent +- It's now ready to upload! However, there's other interesting possibilities - for instance, you could create an app where your users make Spaces and delegate permission to your app to read their uploads. Read the [Architecture options](/docs/concepts/architecture-options/) section to explore more. diff --git a/src/pages/docs/concepts/upload-vs-store.md b/src/pages/docs/concepts/upload-vs-store.md index fec88da..278c5a6 100644 --- a/src/pages/docs/concepts/upload-vs-store.md +++ b/src/pages/docs/concepts/upload-vs-store.md @@ -1,35 +1,41 @@ # Upload vs. Store capabilities in web3.storage -There are two types of content identifiers (CIDs) that web3.storage interacts with: +There are three types of content identifiers (CIDs) that web3.storage interacts with: -1. Content CIDs: The CIDs used to reference and access uploads in the format generally useful to users (e.g., files, directories). These CIDs are generally prefixed by `bafy…`. -2. Shard CIDs: The CID of the serialized shards of data itself that are produced client-side, sent to web3.storage, and stored. These CIDs are generally prefixed by `bag…`. +1. **Content CIDs**: The CIDs used to reference and access uploads in the format generally useful to users (e.g., files, directories). These CIDs are generally prefixed by `bafy…`. +2. **Shard CIDs**: The CID of the serialized shards of data itself that are produced client-side, sent to web3.storage, and stored. These CIDs are generally prefixed by `bag…`. +3. **Piece CIDs**: The primary means of referencing data stored in a sector of a Filecoin Storage Provider. You can think of them as another way to reference a shard. These CIDs are generally prefixed by `bafk…`. -One example of where you might see both is uploading a large file with the CLI: +One example of where you might see all three is uploading a large file with the CLI: ```sh -w3 up gb.file - 1 file (1GB) -bagbaierao... -bagbaieraq... -bagbaieraj... -bagbaierai... -bagbaierax... -bagbaieraf... -bagbaierac... -bagbaierax... -bagbaierax... -bagbaiera4... - +w3 up gb.file --verbose + 1 file 1.0GB + bagbaierakgivx2igjaydgkdholxohq77gsmjceb44wghn6l7rcxjnxlb2ysq 132.1MB + └── Piece CID: bafkzcibextmt6fvx6vkm56k3je76dirfpwzbqhs75agxen7yidv5lo5usnfdbetzcu + bagbaiera5hekddouwxfgue3wrbe5ddwcefr6l7z3majbqsjk3rxlruziw4nq 132.1MB + └── Piece CID: bafkzcibextmt6fujkiyd45vkvyd2gppji2wzcpsvu72y3sshzppmi2wnxp4q5pblay + bagbaierazfqaaabmnaksetr4ivqu7ytchnikygij7g26cycbjbbnfja6oxfa 132.1MB + └── Piece CID: bafkzcibextmt6ftpb5xepzfvg4ot5bv7uhuplrdbdrwnznpwnpaempif4i76gxbmdm + bagbaiera3elp7gxxbwvgyrgzrjjqsrhefofnr4t2zayktvkamqbjxfuundoa 132.1MB + └── Piece CID: bafkzcibextmt6frkjuqg6pxgqaq2zi5ktwsprflbikk2cnqshxifvsumntmqedgrba + bagbaieraamcctnhnqvwzuf7zhg4jyedg5c76fqn625wi7pico4xtnid5bgba 132.1MB + └── Piece CID: bafkzcibextmt6frloqsvau6qhwzfndrh55ahdjech52ddhspakf4gnxwksuks6uiem + bagbaiera6kel2dvxaiowgtj563kic2ftosnzqwue7etiph242w7w5s262fha 132.1MB + └── Piece CID: bafkzcibextmt6frcrmqxujkfxiy34yutheho55cwhlaqdypafjkhgusphcn5kyp3da + bagbaieranxcdfpqqqvmyugisyxmautsduivwrsm4jg2x2akys4ia3u4oyh7q 59.0MB + └── Piece CID: bafkzcibfzxlncayvf3ndiybf3hpigkfd4o27oqrfgmpr6y7ut5nhcyjdtjhqsvstwyqq + bagbaierakynqcaqedv2brd6qs772s3uqnrxwa62guoeqqjq2bi6kr76vfvla 132.1MB + └── Piece CID: bafkzcibextmt6fvobnshzubgtr4szqsztg35yjkoczy3bry74zfy2bu5fm2x5caefq ⁂ Stored 1 file -⁂ https://w3s.link/ipfs/bafybeiaxwvg4... +⁂ https://w3s.link/ipfs/bafybeidt227tuki2axtb3xcolwd5fw7relv6hfwxrlhgxjzyzwh4cowymy ``` -The CLI sharded the 1GB upload into 10 shards, each with a `bag…`-prefixed CID. The content CID of the file itself is included in the `w3s.link/ipfs/bafy…` link at the bottom. +The CLI sharded the ~1GB upload into 8 shards, each with a `bag…`-prefixed CID. The content CID of the file itself is included in the `w3s.link/ipfs/bafy…` link at the bottom. In the vast majority of cases, users should focus on content CIDs, as this is what they'll be using to fetch their content. If you stick with using the recommended client and CLI methods, then you won't really have to ever worry about the shard CIDs. -However, if you're interested in learning more about how web3.storage uses both, read on! +However, if you're interested in learning more about how web3.storage uses all three, read on! ## Upload vs. Store @@ -68,3 +74,7 @@ web3.storage tracks usage for payment (i.e., how much storage is utilized by a u Fortunately, this shouldn't make things any more complicated - we go into more detail below, but in general, when you remove a content CID from your account, you'll want to remove the shard CIDs as well (e.g., in the client calling `client.remove(contentCID, { shards: true })`). However, if you are a power user interacting with shard CIDs as well (like in the previous section), then you need to be more cautious about removing shard CIDs from your account. (This is why the default for the client and CLI methods is for shards to be maintained after removing a content CID). Learn more about how to do this in the Remove section. + +## When should I care about Piece CIDs? + +Piece CIDs are how you can reference your data when it is stored in Filecoin Storage Providers. You can think of Piece CIDs as another way to reference a shard - they are in fact calculated from shard data. Piece CIDs are used in [PoDSI (Proof of Data Segment Inclusion)](/docs/concepts/podsi/) - a proof that a piece is included in a larger piece, which allows users and third parties to prove their data is stored with a Filecoin Storage Provider. diff --git a/src/pages/docs/how-to/create-account.md b/src/pages/docs/how-to/create-account.md index f2dc35c..f1b5128 100644 --- a/src/pages/docs/how-to/create-account.md +++ b/src/pages/docs/how-to/create-account.md @@ -2,7 +2,7 @@ In this how-to guide, you'll learn how to to create a w3up account. -Anyone can use console.web3.storage to create an account and manage their storage using a web browser. Developers can use the `w3cli` command line interface or the `w3up-client` Javascript library. We recommend that developers get familiar with the `w3cli` since it's a powerful tool for many things you might want to do. +Anyone can use [console.web3.storage](https://console.web3.storage) to create an account and manage their storage using a web browser. Developers can use the `w3cli` command line interface or the `w3up-client` Javascript library. We recommend that developers get familiar with the `w3cli` since it's a powerful tool for many things you might want to do. ## Using the CLI @@ -19,7 +19,7 @@ The easiest way to create an account is by using `w3cli`. 2. If you don't have an account, the website will ask you for your email address. Enter your email address and submit the form. 3. Check your email for a message from web3.storage including a link to confirm your intention to authenticate using the email you provided. -## Using the JS library +## Using the JS client 1. Install the client library from npm using your command line: `npm install @web3-storage/w3up-client`. 2. Call `client.login('alice@example.com')`, and wait for the promise to resolve. This will send an email to your inbox with a link for validation. diff --git a/src/pages/docs/how-to/create-space.md b/src/pages/docs/how-to/create-space.mdx similarity index 80% rename from src/pages/docs/how-to/create-space.md rename to src/pages/docs/how-to/create-space.mdx index e5110c0..aec251b 100644 --- a/src/pages/docs/how-to/create-space.md +++ b/src/pages/docs/how-to/create-space.mdx @@ -1,6 +1,8 @@ +import { Callout } from 'nextra/components' + # How to create a space -In this how-to guide, you'll learn how to create a web3.storage Space to organize stored uploads. For an overview of the various ways web3.storage can be integrated with your application, check out ["Architecture Options"](/docs/concepts-architecture-options/). +In this how-to guide, you'll learn how to create a web3.storage Space to organize stored data. For an overview of the various ways web3.storage can be integrated with your application, check out [Architecture Options](/docs/concepts-architecture-options/). A Space acts as a namespace for your uploads. It is created locally, offline, and associated with a cryptographic key pair (identified by the [`did:key`](https://w3c-ccg.github.io/did-method-key/) of the public key). You can register this Space with your [web3.storage account](/docs/how-to/create-account/) to take responsibility for the uploads in the space. Once you do this, you don't need to worry about keeping track of the Space's private key, because your web3.storage account has been authorized to use the Space. @@ -13,7 +15,7 @@ The easiest way to create and register a Space is by using the CLI. ```shell w3 space create ``` -3. w3cli will ask "What would you like to call this space?". Give the space a name that will help you distinguish it from other spaces, then press the enter key. Can't spt come up with one? Try "my first space" +3. w3cli will ask "What would you like to call this space?". Give the space a name that will help you distinguish it from other spaces, then press the enter key. Can't come up with one? Try "my first space" 4. w3cli will say "🔑 You need to save the following secret recovery key somewhere safe!…" - press the enter key to reveal the recovery phrase. - save the recovery phrase somewhere safe if you want to be able to recover control of the space in case you lose access to the computer you used to create the space @@ -26,9 +28,11 @@ Separately, you can visit [console.web3.storage](https://console.web3.storage/), The Space you create can be used to [upload](/docs/how-to/upload/) data using the CLI, the w3up client, or when you log into the web console. -## Using the JS library +## Using the JS client 1. Install the client library from npm using your command line: `npm install @web3-storage/w3up-client`. 2. Call `client.createSpace('Documents')` and wait for the promise to resolve. -Note: the space must be provisioned by an account before it can be used for uploads. See [our guide](/docs/w3up-client/#create-and-register-a-space) for details. + + The space must be provisioned by an account before it can be used for uploads. See [our guide](/docs/w3up-client/#create-and-provision-a-space) for details. + diff --git a/src/pages/docs/how-to/list.md b/src/pages/docs/how-to/list.md deleted file mode 100644 index 32f501f..0000000 --- a/src/pages/docs/how-to/list.md +++ /dev/null @@ -1,22 +0,0 @@ -# How to list files uploaded to web3.storage - -In this how-to guide, you'll learn about the different ways that you can list the files that you've uploaded to web3.storage. Once you've stored some files using web3.storage, you'll want to see a list of what you've uploaded. There are two ways you can do this: - -- Programmatically using the web3.storage client or CLI -- Using the web3.storage console - -## Using the JS client or CLI - -You can also access a listing of your uploads from your code using the web3.storage client. In the example below, this guide walks through how to use the JavaScript client library to fetch a complete listing of all the data you've uploaded using web3.storage. - -For instructions on how to set up your client instance or CLI, check out the Upload section. - -Today, like other developer object storage solutions, there is no sorting or querying by timestamp to keep things scalable. - -You can get a nested list of shard CIDs, or look up what the shard CIDs are for an individual upload. Client and CLI. If you'd like to learn more check out Upload vs. Store section - -## Using the console web UI - -You can see a list of everything you've uploaded to web3.storage in the [console](https://console.web3.storage) web app. If you don't need to work with this list programmatically, using the website may be a simpler choice. - -This console provides a convenient overview of your stored data, including links to view your files in your browser via an [IPFS gateway](https://docs.ipfs.io/concepts/ipfs-gateway/) and information about how the data is being stored on the decentralized storage networks that web3.storage uses under the hood. diff --git a/src/pages/docs/how-to/list.mdx b/src/pages/docs/how-to/list.mdx new file mode 100644 index 0000000..bff9c93 --- /dev/null +++ b/src/pages/docs/how-to/list.mdx @@ -0,0 +1,48 @@ +import { Callout } from 'nextra/components' + +# How to list files uploaded to web3.storage + +In this how-to guide, you'll learn about the different ways that you can list the files that you've uploaded to web3.storage. Once you've stored some files using web3.storage, you'll want to see a list of what you've uploaded. There are two ways you can do this: + +- Programmatically using the JS client or CLI +- Using the web3.storage console + +## Using the JS client or CLI + +You can also access a listing of your uploads from your code using the web3.storage client. In the example below, this guide walks through how to use the JavaScript client library to fetch a complete listing of all the data you've uploaded using web3.storage. + +For instructions on how to set up your client instance or CLI, check out the [Upload](/docs/how-to/upload/) section. + +Today, like other developer object storage solutions, there is no sorting or querying by timestamp to keep things scalable. + +- Client: `client.capability.upload.list({ cursor: '', size: 25 })` +- CLI: `w3 ls` + +In the client the listing is paginated. The result contains a `cursor` that can be used to continue listing uploads. Pass the `cursor` in the result as an _option_ to the next call to receive the next page of results. The `size` option allows you to change the number of items that are returned per page. + +In the CLI, you can use the `--shards` option to print for each upload the list of shards (CAR CIDs) that the uploaded data is contained within. You can learn about the relationship between uploads and shards in the [Upload vs. Store](/docs/concepts/upload-vs-store/) section. + + + The `w3 ls` command automatically pages through the listing and prints the results. + + +### Listing shards + +Each upload is comprised of one or more shards. You can get a list of all shard CIDs in a Space, or look up what the shard CIDs are for an individual upload. + +- Client: `client.capability.store.list({ cursor: '', size: 25 })` +- CLI: `w3 can store ls --cursor "" --size 25` + +The listings are paginated. The result contains a `cursor` that can be used to continue listing uploads. Pass the `cursor` in the result as an _option_ to the next call to receive the next page of results. The `size` option allows you to change the number of items that are returned per page. + +A list of shards for a given upload can be retrieved like this: + +- Client: `client.capability.upload.get(contentCID)` + +You can learn about the relationship between uploads and shards in the [Upload vs. Store](/docs/concepts/upload-vs-store/) section. + +## Using the console web UI + +You can see a list of everything you've uploaded to web3.storage in the [console](https://console.web3.storage) web app. If you don't need to work with this list programmatically, using the website may be a simpler choice. + +This console provides a convenient overview of your stored data, including links to view your files in your browser via an [IPFS gateway](https://docs.ipfs.io/concepts/ipfs-gateway/) and information about how the data is being stored on the decentralized storage networks that web3.storage uses under the hood. diff --git a/src/pages/docs/how-to/remove.mdx b/src/pages/docs/how-to/remove.mdx index 7a30956..1a0faa3 100644 --- a/src/pages/docs/how-to/remove.mdx +++ b/src/pages/docs/how-to/remove.mdx @@ -20,8 +20,8 @@ Note that there is a minimum 30 day retention period for uploaded data, and even web3.storage tracks two different things for its users to support content addressing. These concepts were first introduced in the Upload section: -- Content CIDs: The CIDs used to reference and access uploads in the format generally useful to users (e.g., files, directories). These CIDs are usually prefixed by `bafy…` or `bafk…`. -- Shard CIDs: The CID of the serialized shards of data itself (CAR files) that are produced client-side, sent to web3.storage, and stored. These CIDs are prefixed by `bag…`. +- **Content CIDs**: The CIDs used to reference and access uploads in the format generally useful to users (e.g., files, directories). These CIDs are usually prefixed by `bafy…` or `bafk…`. +- **Shard CIDs**: The CID of the serialized shards of data itself (CAR files) that are produced client-side, sent to web3.storage, and stored. These CIDs are prefixed by `bag…`. web3.storage tracks usage for payment (i.e., how much storage is utilized by a user) using the volume of data associated with shard CIDs. However, in general, most users will be interacting with content CIDs (this is how you fetch your data from the network), with shard CIDs more of an implementation detail (how data gets chunked, serialized into CAR files, and stored for uploads). @@ -29,70 +29,30 @@ Fortunately, this shouldn't make things any more complicated - we go into more d However, if you are a power user interacting with shard CIDs as well (e.g., using the client's `capability.store.*` or CLI's `w3 can store *` methods), then you need to be more cautious about removing shard CIDs from your account. (This is why the default for the client and CLI methods is for shards to be maintained after removing a content CID). You can read more about why you might want to interact with shard CIDs directly and the implications in the Upload vs. Store section. -## Using the client or CLI +## Using the JS client or CLI If you followed the Upload section, you should already have your client or CLI set up with an Agent for your Space. From there, to remove a content CID from your account, you'll generally be using: - Client: `client.remove(contentCID)` - CLI: `w3 rm ` -If you initially uploaded your content by using the recommended upload methods (e.g., used `Client.upload()` or `w3 up`) and didn't interact with CAR shards at all when uploading, we recommend removing the shard CIDs associated with the content CID from your account. Otherwise, you will still be paying for the data stored with web3.storage (as mentioned above). The easiest way to do that is to set the `shards` parameter as `True`: - -- Client: `client.remove(contentCID, shards=True)` -- CLI: `w3 rm --shards` in the CLI - -A full example of this is: - -```javascript -import * as Client from '@web3-storage/w3up-client' -import * as Signer from '@ucanto/principal/ed25519' // Agents on Node should use Ed25519 keys - -const principal = Signer.parse(process.env.KEY) // Agent private key -const client = await Client.create({ principal }) - -async function main () { - // Load client with specific private key - const principal = Signer.parse(process.env.KEY) - const client = await Client.create({ principal }) - - // Add proof that this agent has been delegated capabilities on the space - const proof = await parseProof(process.env.PROOF) - const space = await client.addSpace(proof) - await client.setCurrentSpace(space.did()) - - // remove content previously uploaded, including the underlying shards - await client.remove('bafybeidd2gyhagleh47qeg77xqndy2qy3yzn4vkxmk775bg2t5lpuy7pcu', { shards: true }) -} - -/** @param {string} data Base64 encoded CAR file */ -async function parseProof (data) { - const blocks = [] - const reader = await CarReader.fromBytes(Buffer.from(data, 'base64')) - for await (const block of reader.blocks()) { - blocks.push(block) - } - return importDAG(blocks) -} - -``` +If you initially uploaded your content by using the recommended upload methods (e.g., used `Client.upload()` or `w3 up`) and didn't interact with CAR shards at all when uploading, we recommend removing the shard CIDs associated with the content CID from your account. Otherwise, you will still be paying for the data stored with web3.storage (as mentioned above). The easiest way to do that is to set the `shards` option to `true`: + +- Client: `client.remove(contentCID, { shards: true })` +- CLI: `w3 rm --shards` ## Removing content CIDs and shard CIDs separately -If you have managed your shard CIDs and upload CIDs separately (e.g., used `client.capability.store.add()` and `client.capability.upload.add()` in the client or `w3 can store add` and `w3 can upload add` in the CLI), you'll want to remove the upload CIDs and underlying shard CIDs separately as well. You can read more about why you might want to interact with shard CIDs directly and the implications in the Upload vs. Store section. +If you have managed your shard CIDs and upload CIDs separately (e.g., used `client.capability.store.add()` and `client.capability.upload.add()` in the client or `w3 can store add` and `w3 can upload add` in the CLI), you might want to remove the upload CIDs and underlying shard CIDs separately as well. You can read more about why you might want to interact with shard CIDs directly and the implications in the Upload vs. Store section. To remove shard CIDs and upload CIDs separately, you'll generally do this by: -- Client: - - If you registered a content CID you want to remove using `client.capability.upload.add(contentCID)`… - - (If you don't know which shard CIDs are associated with the content CID) Run `client.capability.upload.listShards(contentCID)`, which returns a list of shard CIDs - - Remove it using `client.capability.upload.remove(contentCID)` - - Remove the shard CIDs that you'd like to - - For each shard CID, ensure no other uploaded content CIDs share the same shard (otherwise, the other content CIDs will no longer be fetchable) - - Remove the shard CIDs one-by-one using `client.capability.store.remove(shardCID)` -- CLI: - - If you registered a content CID you want to remove using `w3 can upload add `… - - (If you don't know which shard CIDs are associated with the content CID) Run `w3 can upload ls --shards`, which returns a list of shard CIDs - - Remove it using `w3 can upload rm ` - - Remove the shard CIDs that you'd like to - - For each shard CID, ensure no other uploaded content CIDs share the same shard (otherwise, the other content CIDs will no longer be fetchable) - - Remove the shard CIDs one-by-one using `w3 can store rm ` +1. Determine shards to remove (skip this step if you already know!): + - Client: `client.capability.upload.list(contentCID)` + - CLI: `w3 can upload ls --shards` +1. Remove the upload: + - Client: `client.capability.upload.remove(contentCID)` + - CLI: `w3 can upload rm ` +1. Remove each of the shards (ensure first that no other content is using that shard!): + - Client: `client.capability.store.remove(shardCID)` + - CLI: `w3 can store rm ` diff --git a/src/pages/docs/how-to/retrieve.mdx b/src/pages/docs/how-to/retrieve.mdx index e155d0f..46f14d5 100644 --- a/src/pages/docs/how-to/retrieve.mdx +++ b/src/pages/docs/how-to/retrieve.mdx @@ -76,20 +76,20 @@ Sometimes you may need to just download a specific file to your computer using t 2. Use curl to download your file: ```shell -curl https://.ipfs.w3s.link/ -o +curl https://.ipfs.w3s.link/ -o # example curl https://bafybeie2bjap32zi2yqh5jmpve5njwulnkualcbiszvwfu36jzjyqskceq.ipfs.w3s.link/example.txt -o ./output-file.txt ``` -Replace ``, ``, and `` with their respective values. +Replace ``, ``, and `` with their respective values. | Variable | Replace with | Example | | --- | --- | --- | -| `` | The CID of the file you want to download. | bafybeie2bjap32zi2yqh5jmpve5njwulnkualcbiszvwfu36jzjyqskceq | -| `` | The _name_ of the file that you originally uploaded to web3.storage. | example.txt | -| `` | The path and filename that you want curl to save the file to. This can be different to ``. | `./output-file.txt` | +| `` | The CID of the file you want to download. | `bafybeie2bjap32zi2yqh5jmpve5n jwulnkualcbiszvwfu36jzjyqskceq` | +| `` | The _name_ of the file that you originally uploaded to web3.storage. | `example.txt` | +| `` | The path and filename that you want curl to save the file to. This can be different to ``. | `./output-file.txt` | ## Next steps -Next, you'll learn about how to [list](/docs/how-to/list/) +Next, you'll learn about how to [list](/docs/how-to/list/) uploaded content. diff --git a/src/pages/docs/how-to/upload.mdx b/src/pages/docs/how-to/upload.mdx index 929c0a5..6294ffa 100644 --- a/src/pages/docs/how-to/upload.mdx +++ b/src/pages/docs/how-to/upload.mdx @@ -16,15 +16,11 @@ Later in this section, we also cover uploading data using the CLI or web console Removing files from w3up will remove them from the file listing for your account, but that doesn't prevent nodes on the decentralized storage network from retaining copies of the data indefinitely. web3.storage itself generally retains and charges users for any uploaded data for a minimum of 30 days. Do not use w3up for data that may need to be permanently deleted in the future. -## web3.storage API authorization +## Using the CLI -In the previous section, you created a Space that has a unique DID. To use the client to upload data to this Space, you need to give it permission to do so. This is done by passing an Agent into the client and a UCAN token delegating permissions to this Agent. An Agent has a DID with an underlying private key (like a Space), but can be thought of as parallel to a login session (whereas the Space can be thought of more like a bucket or account). To learn more about how UCAN works and how web3.storage uses it, read our docs on UCAN authorization. +If you followed the [Create Account](create-account) and [Create Space](create-space) sections, you will already have the CLI set up with a Space. However, you might be using the CLI on a new machine, in which case you can follow these instructions: -## Using the Command line `w3cli` - -If you followed the [Create account](create-account) and [Create Space](create-space) sections, you will already have the CLI set up with a Space. However, you might be using the CLI on a new machine, in which case you can follow these instructions: - -1. (If not yet installed) Install the CLI from npm using your command line: `npm install -g @web3-storage/w3cli`. +1. Install the CLI from npm using your command line: `npm install -g @web3-storage/w3cli`. 2. Run `w3 login alice@example.com` in the command line using your email address. Click on the validation link sent to your email. 3. After successfully running `login`, your CLI Agent has been delegated access to all Spaces associated with your email address. You can see a list of these Spaces using `w3 space ls` and select the one you'd like to upload to using `w3 space use `. @@ -38,18 +34,7 @@ There are a few useful flags (check out the reference docs or `w3 up --help` to -c, --car # File is a CAR file. ``` -## Using the Javascript client - -This section discusses using the web3.storage JavaScript client, `w3up-client`, with your (developer-owned) Space in your application. web3.storage's Javascript client provides a simple interface for storing data using syntax inspired by familiar web APIs such as [fetch](https://developer.mozilla.org/en-US/docs/Web/API/Fetch_API) and [File](https://developer.mozilla.org/en-US/docs/Web/API/File). - -```mermaid -flowchart TD -B[w3up-client instance] --\>|Automatic if specific AgentData is not passed when client object created|B(Pass local AgentData) -S --\> C(Get UCAN delegation from Space to Agent) -C --\> D(Upload to Space using Agent) -``` - -All uses of `w3up-client` to upload with web3.storage follow the flow above. +## Using the JS client ### Installing the client @@ -63,36 +48,44 @@ npm install @web3-storage/w3up-client The package provides a [static create function](https://web3-storage.github.io/w3up/functions/_web3_storage_w3up_client.create.html) that returns a [Client object](https://web3-storage.github.io/w3up/classes/_web3_storage_w3up_client.Client.html). How you initialize it depends on the environment the client is used in: **persistent** or **ephemeral**. -Examples of persistent environments: +Examples of **persistent** environments: * A browser application * A terminal application * An installed application (e.g. Electron) -Examples of ephemeral environments: +Examples of **ephemeral** environments: * AWS Lambda or server side workers * Running inside Docker instances * CI -### Claim delegation via email validation: For persistent environment only +#### Claim delegations via email validation + + + For persistent environment only + + +A new client can claim access to their existing Spaces by validating their email address. ```mermaid sequenceDiagram Client-\>\>web3.storage service: Here is my email address and Agent DID web3.storage service--\>\>Client: Please click the link to validate Client--\>\>web3.storage service: Email address validated -web3.storage service-\>\>Client: Here is a UCAN delegating permission from Space DID to Agent DID +web3.storage service-\>\>Client: Here is a UCAN attesting that your Agent DID belongs to your email ``` -You can use web3.storage's email authorization flow to give permissions to your server-side client. This can be good if your environment will be persistent (otherwise it would be prohibitive to click an email validation link every time the client is re-instantiated). +You can use web3.storage's email authorization flow to give permissions to your client. This can be good if your environment will be persistent (otherwise it would be prohibitive to click an email validation link every time the client is re-instantiated). + +When a Space is created, access permissions are delegated to your email address. We use a special kind of DID for this, a `did:mailto:`. These UCANs are stashed with the web3.storage service. When you validate your email address with a new Agent DID, web3.storage issues a UCAN attestation, that says your Agent DID is owned by your email address. It also returns the UCAN permissions you previously stashed. You can then use the returned UCANs, along with the attestation to prove you are authorized to perform actions. ```javascript import { create } from '@web3-storage/w3up-client' const client = await create() ``` -By default, constructing a client like this will re-use state persisted by other clients because `create` constructs the w3up [Client](https://web3-storage.github.io/w3up/classes/_web3_storage_w3up_client.Client.html)'s access [Agent](https://web3-storage.github.io/w3up/classes/_web3_storage_access.Agent.html) with [AgentData](https://web3-storage.github.io/w3up/classes/_web3_storage_access.AgentData.html) backed by a storage [Driver](https://web3-storage.github.io/w3up/interfaces/_web3_storage_access.unknown.Driver.html) ([IndexedDBDriver](https://web3-storage.github.io/w3up/classes/_web3_storage_access.unknown.IndexedDBDriver.html) in the browser and [ConfDriver](https://web3-storage.github.io/w3up/classes/_web3_storage_access.unknown.ConfDriver.html) otherwise) that persists the AgentData between processes and client instances. +By default, constructing a client like this will re-use state persisted by other clients because `create` constructs the client with a store that persists data between processes and client instances. Once you have created a client, you can login with your email address. Calling login will cause an email to be sent to the given address. @@ -106,9 +99,11 @@ Once a user clicks the confirmation link in the email, the login method will res await client.setCurrentSpace('did:key:...') // select the relevant Space DID that is associated with your account ``` -### Bring your own Agent +#### Bring your own delegations -For any backend (including non-persistent and/or serverless) + + For any backend (including non-persistent and/or serverless) + ```mermaid sequenceDiagram @@ -117,14 +112,15 @@ Developer-\>\>Developer: Delegate UCAN from Space to Agent Developer-\>\>Client: Here is my Agent private key and UCAN delegating permissions ``` -An option that works for any backend environment is to define an Agent and delegate a UCAN from your Space to this Agent before initializing the client. This is especially useful if you're using the client in a serverless Node environment (e.g., AWS Lambda). +An option that works for any backend environment is for a developer to create and provision a Space, and then delegate access to a different Agent DID that will be used by the client. This is especially useful if you're using the client in a serverless environment (e.g., AWS Lambda). In your command line where `w3cli` is configured with the Space you want to use (e.g., where you created the Space): ```shell # The following command returns what will be your Agent private key and DID -# store the private key in environment variable KEY -npx ucan-key ed --json +npx ucan-key ed + +# ❗️ Store the private key (starting "Mg...") in environment variable KEY # The following command creates a UCAN delegation from the w3cli agent to the # agent you generated above. @@ -137,7 +133,7 @@ npx ucan-key ed --json # just being able to upload. w3 delegation create | base64 -# Store the output in environment variable PROOF +# ❗️ Store the output in environment variable PROOF ``` Then, when you initialize and configure the client, you can pass in this Agent and UCAN. @@ -157,7 +153,7 @@ async function main () { // READY to go! } -/** @param {string} data Base64 encoded CAR file \*/ +/** @param {string} data Base64 encoded CAR file */ async function parseProof (data) { const blocks = [] const reader = await CarReader.fromBytes(Buffer.from(data, 'base64')) @@ -188,8 +184,8 @@ There are two main options to getting content into your Space: ```mermaid sequenceDiagram -User-\>\>w3up-client in backend: Upload data -w3up-client in backend-\>\>web3.storage service: Upload data +User-\>\>Application Backend: Upload data +Application Backend-\>\>web3.storage service: Upload data ``` You are already set up to upload using your client instance as data becomes available to your backend - you can call `uploadFile` or `uploadDirectory` with it. @@ -203,7 +199,7 @@ const path = process.env.PATH_TO_FILES const files = await filesFromPaths([path]) const client = await create() -const directoryCid = await client.storeDirectory(files) +const directoryCid = await client.uploadDirectory(files) ``` In the example above, `directoryCid` resolves to an IPFS directory. @@ -212,22 +208,23 @@ In the example above, `directoryCid` resolves to an IPFS directory. ```mermaid sequenceDiagram -participant w3up-client in user -participant w3up-client in backend +participant User +participant Application Backend participant web3.storage service -w3up-client in user-\>\>w3up-client in user: Client instantiated with default Agent -w3up-client in user-\>\>w3up-client in backend: Request delegation with user's Agent DID -w3up-client in backend-\>\>w3up-client in user: Send delegation from Space to user's Agent DID -w3up-client in user-\>\>web3.storage service: Upload data +User-\>\>User: Client instantiated with default Agent +User-\>\>Application Backend: Request delegation with user's Agent DID +Application Backend-\>\>User: Send delegation from Space to user's Agent DID +User-\>\>web3.storage service: Upload data ``` Your backend instance can also be used to delegate upload permissions directly to your user to upload. The code snippet below shows an example of how you might set up a client instance in your application frontend and how it might interact with your backend client. You can see how the frontend client Agent DID is used for the backend client to delegate permissions to; from there, it will be the frontend client that will call the `upload` method. -```javascript +**Backend** + +```js import { CarReader } from '@ipld/car' import * as DID from '@ipld/dag-ucan/did' import * as Delegation from '@ucanto/core/delegation' -import { importDAG } from '@ucanto/core/delegation' import * as Signer from '@ucanto/principal/ed25519' import * as Client from '@web3-storage/w3up-client' @@ -244,10 +241,8 @@ async function backend(did) { // Create a delegation for a specific DID const audience = DID.parse(did) const abilities = ['store/add', 'upload/add'] - const expiration = Math.floor(Date.now() / 1000) + 60 * 60 * 24 // 24 hours from now - const delegation = await client.createDelegation(audience, abilities, { - expiration, - }) + const expiration = Math.floor(Date.now() / 1000) + (60 * 60 * 24) // 24 hours from now + const delegation = await client.createDelegation(audience, abilities, { expiration }) // Serialize the delegation and send it to the client const archive = await delegation.archive() @@ -261,8 +256,15 @@ async function parseProof(data) { for await (const block of reader.blocks()) { blocks.push(block) } - return importDAG(blocks) + return Delegation.importDAG(blocks) } +``` + +**Frontend** + +```js +import * as Delegation from '@ucanto/core/delegation' +import * as Client from '@web3-storage/w3up-client' async function frontend() { // Create a new client @@ -276,7 +278,7 @@ async function frontend() { // Deserialize the delegation const delegation = await Delegation.extract(new Uint8Array(data)) if (!delegation.ok) { - throw new Error('Failed to extract delegation') + throw new Error('Failed to extract delegation', { cause: delegation.error }) } // Add proof that this agent has been delegated capabilities on the space @@ -291,11 +293,13 @@ async function frontend() { You are now ready to upload using the client! In general, the easiest way to upload data is using the `uploadFile` or `uploadDirectory` method. -`uploadFile` expects a "Blob like" input, which can be a [Blob](https://developer.mozilla.org/en-US/docs/Web/API/Blob) or [File](https://developer.mozilla.org/en-US/docs/Web/API/File) when running in a browser. On node.js, see the [filesFromPath library](https://github.com/web3-storage/files-from-path), which can load compatible objects from the local filesystem. By default, files uploaded to web3.storage will be wrapped in an IPFS directory listing. This preserves the original filename and makes links more human-friendly than CID strings, which look like random gibberish. +`uploadFile` expects a "Blob like" input, which can be a [Blob](https://developer.mozilla.org/en-US/docs/Web/API/Blob) or [File](https://developer.mozilla.org/en-US/docs/Web/API/File) when running in a browser. On Node.js, see the [`files-from-path` library](https://github.com/web3-storage/files-from-path), which can load compatible objects from the local filesystem. By default, files uploaded to web3.storage will be wrapped in an IPFS directory listing. This preserves the original filename and makes links more human-friendly than CID strings, which look like random gibberish. `uploadDirectory` requires `File`-like objects instead of `Blob`s, as the file's name property is used to build the directory hierarchy. -**Tip:** When uploading multiple files, give each file a unique name. All the files in a `storeDirectory` request will be bundled into one content archive, and linking to the files inside is much easier if each file has a unique, human-readable name. + + When uploading multiple files, give each file a unique name. All the files in a `uploadDirectory` request will be bundled into one content archive, and linking to the files inside is much easier if each file has a unique, human-readable name. + You can control the directory layout and create nested directory structures by using / delimited paths in your filenames: @@ -306,7 +310,7 @@ const files = [ new File([someBinaryData], 'images/example.png'), ] -const directoryCid = await client.storeDirectory(files) +const directoryCid = await client.uploadDirectory(files) ``` In the example above, directoryCid resolves to an IPFS directory with the following layout: @@ -317,7 +321,7 @@ In the example above, directoryCid resolves to an IPFS directory with the follow | └──example.png ├──readme.md └──src -└──main.py + └──main.py ``` There are a few different ways of creating `File` objects available, depending on your platform. @@ -350,19 +354,17 @@ function makeFileObjects () { } ``` -In Node.js, the [filesFromPath library](https://github.com/web3-storage/files-from-path) reads File objects from the local file system. The `getFilesFromPaths` helper asynchronously returns an array of Files that you can use directly with the `uploadDirectory` client method: +In Node.js, the [`files-from-path` library](https://github.com/web3-storage/files-from-path) reads File objects from the local file system. The `filesFromPaths` helper asynchronously returns an array of Files that you can use directly with the `uploadDirectory` client method: ```javascript -import { getFilesFromPaths } from 'files-from-path' +import { filesFromPaths } from 'files-from-path' ``` -If you expect to be loading a lot of large files, you may be better served by the [filesFromPath](https://github.com/web3-storage/files-from-path#filesfrompath)[helper](https://github.com/web3-storage/files-from-path#filesfrompath). It reduces memory pressure by yielding File objects one by one as they're loaded from disk, instead of loading everything into memory. You can then issue multiple put requests to send each file to web3.storage. - You can also manually create File objects by importing a Node.js implementation of File from the web3.storage package. This is useful when you want to store data created by your application, instead of files from the user's computer. ```javascript async function getFiles (path) { - const files = await getFilesFromPath(path) + const files = await filesFromPaths([path]) console.log(`read ${files.length} file(s) from ${path}`) return files } diff --git a/src/pages/docs/index.mdx b/src/pages/docs/index.mdx index 1f48b69..9e68a17 100644 --- a/src/pages/docs/index.mdx +++ b/src/pages/docs/index.mdx @@ -1,5 +1,4 @@ import { Cards, Card } from 'nextra/components' -import { Callout } from 'nextra/components' **Under the hood** web3.storage uses [UCANs](https://github.com/ucan-wg/spec) for trustless, local-first authorization, and is backed by the provable storage of [IPFS](https://docs.ipfs.tech/) and [Filecoin](https://docs.filecoin.io/basics/what-is-filecoin). @@ -9,7 +8,7 @@ Building a **JavaScript** or **TypeScript** app? see our [guide to using w3up-cl Feeling terminal? Jump in to our **command line adventure** to learn you the w3up for much win! -Get started by installing it from npm +Get started by installing it from npm: ```sh $ npm install -g learnyouw3up @@ -22,4 +21,5 @@ $ learnyouw3up + diff --git a/src/pages/docs/quickstart.md b/src/pages/docs/quickstart.md index 2e68d2c..6925ccf 100644 --- a/src/pages/docs/quickstart.md +++ b/src/pages/docs/quickstart.md @@ -30,7 +30,7 @@ npm install -g @web3-storage/w3cli You need to create a web3.storage account associated with an email address and set it up so you can start uploading to a Space. The Space is created locally and associated with a private key, and is then registered with web3.storage and associated with your email address. But don't worry about keeping track of the Space's private key! web3.storage's email authorization system allows this private key to be treated as a throwaway key. -1. Run `w3 login alice@example.com` in the command line using your email address. This will sent an email to your inbox with a link for validation. +1. Run `w3 login alice@example.com` in the command line using your email address. This will send an email to your inbox with a link for validation. 2. Once you click on the validation link, you'll be taken to a webpage where you can select a plan (like our Starter tier). 3. Create a new Space for storing your data and register it: @@ -43,13 +43,13 @@ w3 space create Documents # pick a good name! You can now upload a file or directory using the command line: ```sh -$ w3 up lets-go.txt +w3 up lets-go.txt 1 file 0.6KB ⁂ Stored 1 file ⁂ https://w3s.link/ipfs/bafybeib4ht2a53pttgipw6mgckqqhmgkifnmh2glzsju2c6ait5ibnkow4 ``` -The CLI content-addresses your files, packs them into 1 or more CAR files, and uploads them to web3.storage for indexing and inclusion in Filecoin storage deals. It will show an http gateway URL that includes the content CID (content identifier) of your upload e.g: +The CLI content-addresses your files, packs them into 1 or more CAR files, and uploads them to web3.storage for indexing and inclusion in Filecoin storage deals. It will show an HTTP gateway URL that includes the content CID (content identifier) of your upload e.g: https://w3s.link/ipfs/bafybeib4ht2a53pttgipw6mgckqqhmgkifnmh2glzsju2c6ait5ibnkow4 @@ -59,7 +59,7 @@ https://w3s.link/ipfs/bafybeib4ht2a53pttgipw6mgckqqhmgkifnmh2glzsju2c6ait5ibnkow ## Get your file -Your upload is now available over the public IPFS network using the content CID of your upload. The easiest way to fetch it is using the the link that `w3 up` provided to the w3s.link gateway. w3s.link is optimized for content uploaded to web3.storage. +Your upload is now available over the public IPFS network using the content CID of your upload. The easiest way to fetch it is using the link that `w3 up` provided to the w3s.link gateway. w3s.link is optimized for content uploaded to web3.storage. ```sh curl -L 'https://w3s.link/ipfs/bafybeib4ht2a53pttgipw6mgckqqhmgkifnmh2glzsju2c6ait5ibnkow4/lets-go.txt' @@ -90,6 +90,6 @@ __ _ __ ____ \_ |__ \_____ \ _______/ |_ ____ _______ _____ Congratulations! You've just covered the basics of web3.storage. To learn more, take a look at these useful resources: - For a deep dive into storing files, including using the Javascript client to do so, visit the [Upload how-to guide](/docs/how-to/upload). -- Read more about the power of UCANs and IPFS, and learn about the various options to integrate web3.storage with your application. -- Try out our image gallery example to see how easy it is to take advantage of these decentralized protocols using web3.storage. -- Visit the reference API section for more details on what else you can do with the web3.storage client and how to integrate it into your own projects. +- Read more about the power of [UCANs and IPFS](/docs/concepts/ucans-and-web3storage), and learn about the various options to integrate web3.storage with your application. + diff --git a/src/pages/docs/w3cli.mdx b/src/pages/docs/w3cli.mdx index c83e900..8d11d4c 100644 --- a/src/pages/docs/w3cli.mdx +++ b/src/pages/docs/w3cli.mdx @@ -7,10 +7,9 @@ You can get started using web3.storage right away from your command line using ` In this guide, we'll walk through the following steps: 1. [Installing the `w3` tool](#install) -2. [Standalone binaries](#standalone-binaries) -3. [Creating and provisioning your first space](#create-your-first-space) -4. [Uploading a file or directory](#upload-files) -5. [Viewing your file with IPFS](#view-your-file-with-ipfs) +2. [Creating and provisioning your first space](#create-your-first-space) +3. [Uploading a file or directory](#upload-files) +4. [Viewing your file with IPFS](#view-your-file-with-ipfs) ## Install @@ -30,7 +29,7 @@ npm install -g @web3-storage/w3cli Once the install is complete, you'll have a `w3` command available. Try running `w3 --help` to get an idea of what's possible. -## Standalone binaries +### Standalone binaries These binaries are built from the Go client codebase which is not as fully featured as the JS client. @@ -42,11 +41,11 @@ Please ensure you [read the documentation for using the Go CLI](https://github.c ## Create your first space -When you upload things to web3.storage, each upload is associated with a "space," which is a unique identifier that acts as a namespace for your content. +When you upload things to web3.storage, each upload is associated with a "space", which is a unique identifier that acts as a namespace for your content. Spaces are identified by DID using keys created locally on your devices. To use a space for uploads, it needs to be registered with the storage service by providing an email address. -To create a space using the `w3` command line tool, use the `w3 space create` command. You can optionally give your space a "friendly" name, which acts like an alias for the space's DID and can make it easier to tell your spaces apart. In the example below, we'll use the name `Documents`: +To create a space using the `w3` command line tool, use the `w3 space create` command. You should give your space a "friendly" name, which acts like an alias for the space's DID and can make it easier to tell your spaces apart. In the example below, we'll use the name `Documents`: ```bash w3 space create Documents @@ -84,4 +83,4 @@ When uploading directories, files beginning with a `.` character are ignored by ## View your file with IPFS -When your upload completes, you should see a link to your files on the `w3s.link` IPFS gateway. Just click the link to see your files! +When your upload completes, you should see a link to your files on the `w3s.link` IPFS gateway. Just click the link to see your file! diff --git a/src/pages/docs/w3up-client.md b/src/pages/docs/w3up-client.md index 54585ee..dfa4b11 100644 --- a/src/pages/docs/w3up-client.md +++ b/src/pages/docs/w3up-client.md @@ -52,7 +52,7 @@ To use a space for uploads, it needs to be provisioned with the storage service, const myAccount = await client.login('zaphod@beeblebrox.galaxy') ``` -Calling `login` cause an email to be sent to the given address, unless the client is already logged in. Once a user clicks the confirmation link in the email, the promise returned by the `login` method will resolve. Make sure to check for errors, as `login` will fail if the email is not confirmed within the expiration timeout. +Calling `login` causes an email to be sent to the given address, unless the client is already logged in. Once a user clicks the confirmation link in the email, the promise returned by the `login` method will resolve. Make sure to check for errors, as `login` will fail if the email is not confirmed within the expiration timeout. If your account does not yet have a payment plan, you'll be prompted to choose one after your email address has been verified. You will need a payment plan in order to provision your space. You can use the following code to wait for a payment plan to be selected: @@ -98,7 +98,7 @@ Now that you've created and provisioned a space, you're ready to upload files to Call `uploadFile` to upload a single file, or `uploadDirectory` to upload multiple files. -`uploadFile` expects a "Blob like" input, which can be a [`Blob`](https://developer.mozilla.org/en-US/docs/Web/API/Blob) or [`File`](https://developer.mozilla.org/en-US/docs/Web/API/File) when running in a browser. On node.js, see the [`filesFromPath` library](https://github.com/web3-storage/files-from-path), which can load compatible objects from the local filesystem. +`uploadFile` expects a "Blob like" input, which can be a [`Blob`](https://developer.mozilla.org/en-US/docs/Web/API/Blob) or [`File`](https://developer.mozilla.org/en-US/docs/Web/API/File) when running in a browser. On Node.js, see the [`files-from-path` library](https://github.com/web3-storage/files-from-path), which can load compatible objects from the local filesystem. `uploadDirectory` requires `File`-like objects instead of `Blob`s, as the file's `name` property is used to build the directory hierarchy. @@ -127,9 +127,9 @@ In the example above, `directoryCid` resolves to an IPFS directory with the foll ## View your file on an IPFS gateway -The `uploadFile` and `uploadDirectory` methods described in the previous step both return a CID, or Content Identifier, encoded as a string. +The `uploadFile` and `uploadDirectory` methods described in the previous step both return a CID, or Content Identifier - a unique hash of the data. -To create a link to view your file on an IPFS gateway, create a URL of the form `https://${cid}.ipfs.${gatewayHost}`, where `${cid}` is the CID of the content you want to view, and `${gatewayHost}` is the domain of the gateway. To use our own gateway at `w3s.link`, your url would be `https://${cid}.ipfs.w3s.link`. +To create a link to view your file on an IPFS gateway, create a URL of the form `https://${cid}.ipfs.${gatewayHost}`, where `${cid}` is the CID of the content you want to view, and `${gatewayHost}` is the domain of the gateway. To use our own gateway at `w3s.link`, your URL would be `https://${cid}.ipfs.w3s.link`. Opening the gateway URL in a browser will take you to your uploaded file, or a directory listing of files, depending on what you uploaded.