feat: add docs for standalone binaries and Go client

src/pages/docs/_meta.json

@@ -3,6 +3,7 @@
   "quickstart": "Quickstart",
   "w3cli": "Command line",
   "w3up-client": "JS Client",
+  "go-w3up": "Go Client",
   "how-to": "How to",
   "concepts": "Concepts",
   "faq": "FAQ",

src/pages/docs/faq.md

@@ -23,7 +23,7 @@ However, once a file is uploaded to web3.storage, there cannot be a guarantee th
 
 ## Are there client libraries other than Javascript?
 
-Unfortunately today there is only a Javascript client library. You can, however, use the CLI for many programmatic use cases.
+Yes! There is now a [client in Go](/docs/go-w3up). However, it is heavily under development and not yet fully featured. Alternatively, you can use the CLI for many programmatic use cases.
 
 ## How can I edit a file or add files to a folder?
 

src/pages/docs/go-w3up.mdx

@@ -0,0 +1,181 @@
+import { Callout } from 'nextra/components'
+import { Steps } from 'nextra/components'
+
+# `go-w3up`
+
+<Callout type="warning">
+  The Go client is under heavily development and is not as fully featured as the [JS client](/docs/w3up-client).
+</Callout>
+
+You can easily integrate web3.storage into your Go apps using `go-w3up`, our Go client for the w3up platform.
+
+In this guide, we'll walk through the following steps:
+
+1. [Installing the client library](#install)
+1. [Generating a DID](#generate-a-did)
+1. [Obtaining proofs](#obtain-proofs)
+1. [Loading your private key and proofs](#load-priviate-key-and-proofs)
+1. [Uploading a CAR file](#upload-a-car)
+
+## Install
+
+You'll need [Go](https://go.dev/) version 1.21.4 or higher.
+
+In addition to the w3up library you're also likely to need elements of `go-ucanto` - a library for performing UCAN RPC calls. Add the libraries to your project's dependencies:
+
+```bash
+go get github.com/web3.storage/go-w3up
+go get github.com/web3.storage/go-ucanto
+```
+
+## Generate a DID
+
+You can use `ucan-key` to generate a private key and DID for use with the library. Install Node.js and then use the `ucan-key` module:
+
+```sh
+npx ucan-key ed
+```
+
+Output should look something like:
+
+```sh
+# did:key:z6Mkh9TtUbFJcUHhMmS9dEbqpBbHPbL9oxg1zziWn1CYCNZ2
+MgCb+bRGl02JqlWMPUxCyntxlYj0T/zLtR2tn8LFvw6+Yke0BKAP/OUu2tXpd+tniEoOzB3pxqxHZpRhrZl1UYUeraT0=
+```
+
+<Callout>
+  Save the private key (starting `Mg...`) to a file `private.key`.
+</Callout>
+
+## Obtain proofs
+
+Proofs are delegations to your DID enabling it to perform tasks. Currently the best way to obtain proofs that will allow you to interact with the web3.storage API is to use the w3up JS CLI:
+
+<Steps>
+### Generate a DID
+
+[Generate a DID](#generate-a-did) and make a note of it (the string starting with `did:key:...`)
+
+### Install w3 CLI
+
+```sh
+npm install -g @web3-storage/w3cli
+```
+
+### Create a space
+
+```sh
+w3 space create [NAME]
+```
+
+### Delegate capabilities to your DID
+
+```sh
+w3 delegation create -c 'store/*' -c 'upload/*' [DID] -o proof.ucan
+```
+</Steps>
+
+<Callout>
+  Make a note of the space DID from above. You'll need it later.
+</Callout>
+
+## Load private key and proofs
+
+To interact with the web3.storage API you need your private key to sign UCAN invocations and a proof that your key has been delegated capabilities to perform tasks:
+
+```go
+package main
+
+import (
+  "ioutil"
+
+  "github.com/web3-storage/go-ucanto/did"
+  "github.com/web3-storage/go-ucanto/principal/ed25519/signer"
+  "github.com/web3-storage/go-w3up/delegation"
+)
+
+// space that the client will interact with
+space, _ := did.Parse("did:key:z6MkwDuRThQcyWjqNsK54yKAmzfsiH6BTkASyiucThMtHt1y")
+
+// private key to sign UCAN invocations with
+priv, _ := ioutil.ReadFile("path/to/private.key")
+issuer, _ := signer.Parse(priv)
+
+// UCAN proof(s) that the signer can perform tasks in this space (a delegation chain)
+prfbytes, _ := ioutil.ReadFile("path/to/proof.ucan")
+proof, _ := delegation.ExtractProof(b)
+```
+
+## Upload a CAR
+
+Once you have loaded your space DID, your private key and your delegation proofs, you can upload a CAR to web3.storage.
+
+```go
+package main
+
+import (
+	"bytes"
+	"fmt"
+	"net/http"
+
+	"github.com/ipfs/go-cid"
+	cidlink "github.com/ipld/go-ipld-prime/linking/cid"
+	"github.com/multiformats/go-multihash"
+	"github.com/web3-storage/go-w3up/capability/storeadd"
+	"github.com/web3-storage/go-w3up/client"
+)
+
+func main() {
+  data, _ := ioutil.ReadFile("path/to/my.car")
+
+  // generate the CID for the CAR
+  mh, _ := multihash.Sum(data, multihash.SHA2_256, -1)
+  link := cidlink.Link{Cid: cid.NewCidV1(0x0202, mh)}
+
+  rcpt, _ := client.StoreAdd(
+    issuer,
+    space,
+    &storeadd.Caveat{Link: link, Size: len(data)},
+    client.WithProofs(proofs),
+  )
+
+  // "upload" means it needs to be uploaded, "done" means it is already done!
+  if rcpt.Out().Ok().Status == "upload" {
+    hr, _ := http.NewRequest("PUT", *rcpt.Out().Ok().Url, bytes.NewReader(data))
+
+    hdr := map[string][]string{}
+    for k, v := range rcpt.Out().Ok().Headers.Values {
+      hdr[k] = []string{v}
+    }
+
+    hr.Header = hdr
+    hr.ContentLength = len(data)
+    httpClient := http.Client{}
+    res, _ := httpClient.Do(hr)
+    res.Body.Close()
+  }
+
+  fmt.Println(link.String())
+}
+```
+
+<Callout type="warning">
+  Maximum upload size is 4,261,412,864 bytes (around 4GB). To upload CAR files larger than this, please use the [sharding utility](https://github.com/web3-storage/go-w3up/blob/main/car/sharding/sharding.go) to split the CAR into multiple shards.
+</Callout>
+
+A DAG can be sharded amongst multiple CAR files (see maximum upload size above). To tie together multiple stored CAR files to a content root CID you can register an "upload". An "upload" is a content root CID + a set of CAR shards that it is contained within.
+
+<Callout type="info">
+  You can register an upload with just one "shard". It is best practice to always register an upload even if there is only 1 shard.
+</Callout>
+
+Registering an upload is simple:
+
+```go
+rcpt, _ := client.UploadAdd(
+  issuer,
+  space,
+  &uploadadd.Caveat{Root: root, Shards: shards},
+  client.WithProofs(proofs),
+)
+```

src/pages/docs/how-to/remove.md → src/pages/docs/how-to/remove.mdx

@@ -1,14 +1,20 @@
+import { Callout } from 'nextra/components'
+
 # Removing data from your account
 
 You might want to remove data from being associated with your account. You can do so using the JS client, CLI, or web console.
 
 Note that there is a minimum 30 day retention period for uploaded data, and even once removed, the data might persist on the public IPFS network.
 
-**CAUTION**
-
-⚠️❗ Public Data 🌎: All data uploaded to w3up is available to anyone who requests it using the correct CID. Do not store any private or sensitive information in an unencrypted form using w3up.
+<Callout type="error" emoji="⚠️❗">
+  **Public Data 🌎**<br/>
+  All data uploaded to w3up is available to anyone who requests it using the correct CID. Do not store any private or sensitive information in an unencrypted form using w3up.
+</Callout>
 
-⚠️❗ Permanent Data ♾️: Removing files from w3up will remove them from the file listing for your account, but that doesn't prevent nodes on the decentralized storage network from retaining copies of the data indefinitely. web3.storage itself generally retains and charges users for any uploaded data for a minimum of 30 days. Do not use w3up for data that may need to be permanently deleted in the future.
+<Callout type="error" emoji="⚠️❗">
+  **Permanent Data ♾️**<br/>
+  Removing files from w3up will remove them from the file listing for your account, but that doesn't prevent nodes on the decentralized storage network from retaining copies of the data indefinitely. web3.storage itself generally retains and charges users for any uploaded data for a minimum of 30 days. Do not use w3up for data that may need to be permanently deleted in the future.
+</Callout>
 
 ## Removing uploads (content CIDs) vs. stores (shard CIDs)
 

src/pages/docs/how-to/retrieve.md → src/pages/docs/how-to/retrieve.mdx

@@ -1,3 +1,5 @@
+import { Callout } from 'nextra/components'
+
 # How to retrieve data from web3.storage
 
 In this how-to guide, you'll learn several methods for retrieving data from web3.storage.
@@ -15,36 +17,40 @@ When retrieving any data, you'll be using the content CID of the upload (prefixe
 
 You can easily fetch any data stored using web3.storage using an IPFS HTTP gateway. Because IPFS is a peer-to-peer, decentralized network, you can use any public HTTP gateway to fetch your data. In this guide, we'll use the gateway at w3s.link (which is optimized for data stored with web3.storage), but you can see more worldwide gateways on the [IPFS Public Gateway Checker](https://ipfs.github.io/public-gateway-checker/).
 
-You can use an IPFS gateway to view a list of all the files in that directory from your browser. To do so, simply create a gateway URL. For example, if your CID is `bafybeidd2gyhagleh47qeg77xqndy2qy3yzn4vkxmk775bg2t5lpuy7pcu`, you can make a URL for the w3s.link gateway as follows: `https://bafybeidd2gyhagleh47qeg77xqndy2qy3yzn4vkxmk775bg2t5lpuy7pcu.ipfs.w3s.link/`. Follow that link, and you'll see a page similar to this:
+You can use an IPFS gateway to view a list of all the files in that directory from your browser. To do so, simply create a gateway URL. For example, if your CID is `bafybeidd2gyhagleh47qeg77xqndy2qy3yzn4vkxmk775bg2t5lpuy7pcu`, you can make a URL for the w3s.link gateway as follows:
 
-If you want to link directly to a file within that directory, just add the file path after the CID portion of the link. For example: [bafybeidd2gyhagleh47qeg77xqndy2qy3yzn4vkxmk775bg2t5lpuy7pcu.ipfs.w3s.link/not-distributed.jpg](https://bafybeidd2gyhagleh47qeg77xqndy2qy3yzn4vkxmk775bg2t5lpuy7pcu.ipfs.w3s.link/not-distributed.jpg) could be used as a shareable link for your new favorite wallpaper.
+```sh
+https://bafybeidd2gyhagleh47qeg77xqndy2qy3yzn4vkxmk775bg2t5lpuy7pcu.ipfs.w3s.link/
+```
 
-**Tip**
+If you want to link directly to a file within that directory, just add the file path after the CID portion of the link. For example: [bafybeidd2gyhagleh47qeg77xqndy2qy3yzn4vkxmk775bg2t5lpuy7pcu.ipfs.w3s.link/not-distributed.jpg](https://bafybeidd2gyhagleh47qeg77xqndy2qy3yzn4vkxmk775bg2t5lpuy7pcu.ipfs.w3s.link/not-distributed.jpg) could be used as a shareable link for your new favorite wallpaper.
 
-Your web3.storage console page includes IPFS gateway links to all the content you've uploaded.
+<Callout>
+  Your web3.storage console page includes IPFS gateway links to all the content you've uploaded.
+</Callout>
 
 ### Setting the filename
 
 When downloading files from an HTTP gateway, web browsers will set the default filename for the downloaded file based on the path component of the gateway link. For example, if you use your browser's "Save link as..." feature on the following link, it should prompt you to save a file named treehouse.jpeg:
 
-<https://bafybeicfnbaeigdtklwkrj35r4wtfppix732zromsadvgiu33mowah74yq.ipfs.w3s.link/treehouse.jpeg>
+https://bafybeicfnbaeigdtklwkrj35r4wtfppix732zromsadvgiu33mowah74yq.ipfs.w3s.link/treehouse.jpeg
 
 In the link above, the CID `bafybeicfnbaeigdtklwkrj35r4wtfppix732zromsadvgiu33mowah74yq` points to an IPFS directory listing, which maps from the filename treehouse.jpeg to the CID for the image itself.
 
 Since the web3.storage client wraps your uploaded files in a directory by default, this is the most common kind of gateway link you're likely to need, and your users should get nice filenames when they download their content.
 
 However, the behavior is a bit different if you make a gateway link directly to the image CID:
 
-- <https://bafkreifvallbyfxnedeseuvkkswt5u3hbdb2fexcygbyjqy5a5rzmhrzei.ipfs.w3s.link/>
-- <https://ipfs.io/ipfs/bafkreifvallbyfxnedeseuvkkswt5u3hbdb2fexcygbyjqy5a5rzmhrzei>
+- https://bafkreifvallbyfxnedeseuvkkswt5u3hbdb2fexcygbyjqy5a5rzmhrzei.ipfs.w3s.link
+- https://ipfs.io/ipfs/bafkreifvallbyfxnedeseuvkkswt5u3hbdb2fexcygbyjqy5a5rzmhrzei
 
 Both of the URLs above link directly to the CID of the image, without an associated filename. The first URL uses the recommended "subdomain" URL format for gateway links, while the second form uses a "path prefix" format that you may see in use elsewhere in the IPFS ecosystem.
 
 Depending on which style of link you use, your browser will prompt you to save a file with a generic name like download, or with the CID as the filename.
 
 If you have such a link, you can override the default filename by adding a query string parameter to your link of the form `?filename=<desired-filename>`. For example, the following link will save as treehouse.jpeg, even though it links directly to the image by CID:
 
-<https://bafkreifvallbyfxnedeseuvkkswt5u3hbdb2fexcygbyjqy5a5rzmhrzei.ipfs.w3s.link/?filename=treehouse.jpeg>
+https://bafkreifvallbyfxnedeseuvkkswt5u3hbdb2fexcygbyjqy5a5rzmhrzei.ipfs.w3s.link/?filename=treehouse.jpeg
 
 ## Using the `ipfs` command
 

src/pages/docs/how-to/upload.md → src/pages/docs/how-to/upload.mdx

@@ -1,14 +1,20 @@
+import { Callout } from 'nextra/components'
+
 # How to upload data using web3.storage
 
 In this how-to guide, you'll learn how to store data programmatically for your development projects using the web3.storage client library in JavaScript using your (developer-owned) Space. This includes various architecture options for the data pipeline for your users to upload to web3.storage, which then makes your data available on the decentralized IPFS network with persistent long-term storage provided by Filecoin.
 
 Later in this section, we also cover uploading data using the CLI or web console. If you just want to quickly store a few files using web3.storage rather than include upload functionality in an app or service you're building, you may want to hop down there.
 
-**CAUTION**
-
-⚠️❗ Public Data 🌎: All data uploaded to w3up is available to anyone who requests it using the correct CID. Do not store any private or sensitive information in an unencrypted form using w3up.
+<Callout type="error" emoji="⚠️❗">
+  **Public Data 🌎**<br/>
+  All data uploaded to w3up is available to anyone who requests it using the correct CID. Do not store any private or sensitive information in an unencrypted form using w3up.
+</Callout>
 
-⚠️❗ Permanent Data ♾️: Removing files from w3up will remove them from the file listing for your account, but that doesn't prevent nodes on the decentralized storage network from retaining copies of the data indefinitely. web3.storage itself generally retains and charges users for any uploaded data for a minimum of 30 days. Do not use w3up for data that may need to be permanently deleted in the future.
+<Callout type="error" emoji="⚠️❗">
+  **Permanent Data ♾️**<br/>
+  Removing files from w3up will remove them from the file listing for your account, but that doesn't prevent nodes on the decentralized storage network from retaining copies of the data indefinitely. web3.storage itself generally retains and charges users for any uploaded data for a minimum of 30 days. Do not use w3up for data that may need to be permanently deleted in the future.
+</Callout>
 
 ## web3.storage API authorization
 

src/pages/docs/w3cli.md → src/pages/docs/w3cli.mdx

@@ -1,18 +1,22 @@
+import { Callout } from 'nextra/components'
+
 # `w3cli`
 
 You can get started using web3.storage right away from your command line using `w3`, our command line interface tool.
 
 In this guide, we'll walk through the following steps:
 
 1. [Installing the `w3` tool](#install)
-2. [Creating and provisioning your first space](#create-your-first-space)
-3. [Uploading a file or directory](#upload-files)
-4. [Viewing your file with IPFS](#view-your-file-with-ipfs)
+2. [Standalone binaries](#standalone-binaries)
+3. [Creating and provisioning your first space](#create-your-first-space)
+4. [Uploading a file or directory](#upload-files)
+5. [Viewing your file with IPFS](#view-your-file-with-ipfs)
 
 ## Install
 
 You'll need [Node](https://nodejs.com) version 18 or higher, with NPM version 7 or higher to complete this guide.
-You can check your local versions like this
+
+You can check your local versions like this:
 
 ```bash
 node --version && npm --version
@@ -26,6 +30,16 @@ npm install -g @web3-storage/w3cli
 
 Once the install is complete, you'll have a `w3` command available. Try running `w3 --help` to get an idea of what's possible.
 
+## Standalone binaries
+
+<Callout type="warning">
+  These binaries are built from the Go client codebase which is not as fully featured as the JS client.
+</Callout>
+
+Pre-compiled, standalone binaries are now available for MacOS, Linux and Windows: [Download latest release](https://github.com/web3-storage/go-w3up/releases/latest).
+
+Please ensure you [read the documentation for using the Go CLI](https://github.com/web3-storage/go-w3up#cli). These binaries are built from the Go client codebase which is not as fully featured as the JS client. Currently to use the Go CLI tool you will need to obtain a delegation created by the JS CLI or [console](https://console.web3.storage).
+
 ## Create your first space
 
 When you upload things to web3.storage, each upload is associated with a <abbr id="space">"space,"</abbr> which is a unique identifier that acts as a namespace for your content.