Refactor + Block store

[n8maninger]

Basically a huge refactor that also happens to add a chunk database to store each chunk and offset 😅

You can use [POST] /api/cid/calculate to test the chunk calculation and compare it with IPFS desktop. [0] is always the root CID and should have offset = 0, length = upload length.

A major change to the API is that you no longer have to load the whole file into memory. The downside is it can't automatically use the CID as the renterd object key. You can either precalculate it or use an arbitrary key. The local db maps the key back to the CID after upload. Maybe in the future, the file should be uploaded to the renterd first (S3/regularly) then manually pinned to the gateway.

[n8maninger]

@alexfreska all of the necessary IPFS config has been moved into the yaml file. You can delete the ipfs folder and there's no need to run init.

api:
  address: :8081
  password: sia is cool
renterd:
  address: http://localhost:9980/api/worker
  password: sia is extra cool
  bucket: ipfs
ipfs:
  gatewayAddress: :8080 # gateway URL
  listenAddresses:
    - /ip4/0.0.0.0/tcp/4001 # the listen address for bitswap. Since no announce addresses are configured, no bits will be swapped
  fetchRemote: true # true if you want to download files that aren't pinned

Gateway Endpoints:

[GET] /ipfs/:cid (if fetchRemote is true, will try to download from IPFS)

API Endpoints:

[POST] /api/upload/:cid (request body is the raw file to upload, CID is technically arbitrary)
[POST] /api/pin/:cid (request body is empty, file will be downloaded from IPFS and pinned to the node)
[POST] /api/cid/calculate (request body is the raw file to upload, calculates the blocks and root CID)

[alexfreska]

@n8maninger are you able to upload? I get a status 308 from the renterd worker client here: https://github.com/SiaFoundation/renterd/blob/master/worker/client.go#L208

[n8maninger]

@n8maninger are you able to upload? I get a status 308 from the renterd worker client here: https://github.com/SiaFoundation/renterd/blob/master/worker/client.go#L208

308 is a permanent redirect. What are your renterd settings?

[alexfreska]

@n8maninger

308 is a permanent redirect. What are your renterd settings?

api:
  address: :8081
  password: <password>
renterd:
  address: http://arequipa.sia.tech/api/worker
  password: <password>
  bucket: ipfs
ipfs:
  gatewayAddress: :8080
  listenAddresses:
    - /ip4/0.0.0.0/tcp/4001
  fetchRemote: true

[n8maninger]

@n8maninger

308 is a permanent redirect. What are your renterd settings?

api:
  address: :8081
  password: <password>
renterd:
  address: http://arequipa.sia.tech/api/worker
  password: <password>
  bucket: ipfs
ipfs:
  gatewayAddress: :8080
  listenAddresses:
    - /ip4/0.0.0.0/tcp/4001
  fetchRemote: true

Shouldn't that be https://arequipa.sia.tech/api/worker?

[alexfreska]

🚀