Skip to content

Latest commit

 

History

History
1195 lines (1084 loc) · 95.3 KB

v0.13.md

File metadata and controls

1195 lines (1084 loc) · 95.3 KB
Raw

go-ipfs changelog 2022

v0.13.1 2022-07-06

This release includes security fixes for various DOS vectors when importing untrusted user input with ipfs dag import and the v0/dag/import endpoint.

View the linked security advisory for more information.

Changelog

Full Changelog
  • github.com/ipfs/go-ipfs:
    • chore: update car
  • github.com/ipld/go-car (v0.3.2 -> v0.4.0) & (v2.1.1 -> v2.4.0):
    • Bump version in prep for releasing go-car v0
    • Revert changes to insertionindex
    • Revert changes to index.Index while keeping most of security fixes
    • Return error when section length is invalid varint
    • Drop repeated package name from CarStats
    • Benchmark Reader.Inspect with and without hash validation
    • Use consistent CID mismatch error in Inspect and BlockReader.Next
    • Use streaming APIs to verify the hash of blocks in CAR Inspect
    • test: add fuzzing for reader#Inspect
    • feat: add block hash validation to Inspect()
    • feat: add Reader#Inspect() function to check basic validity of a CAR and return stats
    • Remove support for ForEach enumeration from car-index-sorted
    • Use a fix code as the multihash code for CarIndexSorted
    • Fix testutil assertion logic and update index generation tests
    • fix: tighter constraint of singleWidthIndex width, add index recommendation docs
    • fix: explicitly disable serialization of insertionindex
    • feat: MaxAllowed{Header,Section}Size option
    • feat: MaxAllowedSectionSize default to 32M
    • fix: use CidFromReader() which has overread and OOM protection
    • fix: staticcheck catches
    • fix: revert to internalio.NewOffsetReadSeeker in Reader#IndexReader
    • fix index comparisons
    • feat: Refactor indexes to put storage considerations on consumers
    • test: v2 add fuzzing of the index
    • fix: v2 don't divide by zero in width indexes
    • fix: v2 don't allocate indexes too big
    • test: v2 add fuzzing to Reader
    • fix: v2 don't accept overflowing offsets while reading v2 headers
    • test: v2 add fuzzing to BlockReader
    • fix: v2 don't OOM if the header size is too big
    • test: add fuzzing of NewCarReader
    • fix: do bound check while checking for CIDv0
    • fix: don't OOM if the header size is too big
    • Add API to regenerate index from CARv1 or CARv2
    • PrototypeChooser support (#305) (ipld/go-car#305)
    • bump to newer blockstore err not found (#301) (ipld/go-car#301)
    • Car command supports for largebytes nodes (#296) (ipld/go-car#296)
    • fix(test): rootless fixture should have no roots, not null roots
    • Allow extracton of a raw unixfs file (#284) (ipld/go-car#284)
    • cmd/car: use a better install command in the README
    • feat: --version selector for car create & update deps
    • feat: add option to create blockstore that writes a plain CARv1 (#288) (ipld/go-car#288)
    • add car detach-index list to list detached index contents (#287) (ipld/go-car#287)
    • add car root command (#283) (ipld/go-car#283)
    • make specification of root cid in get-dag command optional (#281) (ipld/go-car#281)
    • Update version.json after manual tag push
    • Update v2 to context datastores (#275) (ipld/go-car#275)
    • update context datastore (ipld/go-car#273)
    • Traversal-based car creation (#269) (ipld/go-car#269)
    • Seek to start before index generation in ReadOnly blockstore
    • support extraction of unixfs content stored in car files (#263) (ipld/go-car#263)
    • Add a bare bones readme to the car CLI (#262) (ipld/go-car#262)
    • sync: update CI config files (#261) (ipld/go-car#261)
    • fix!: use -version=n instead of -v1 for index command
    • feat: fix get-dag and add version=1 option
    • creation of car from file / directory (#246) (ipld/go-car#246)
    • forEach iterates over index in stable order (#258) (ipld/go-car#258)
  • github.com/multiformats/go-multicodec (v0.4.1 -> v0.5.0):
    • Bump version to 0.5.0
    • Bump version to 0.4.2
    • deps: update stringer version in go generate command
    • docs(readme): improved usage examples (#66) (multiformats/go-multicodec#66)

❤ Contributors

Contributor Commits Lines ± Files Changed
Masih H. Derkani 27 +1494/-1446 100
Rod Vagg 31 +2021/-606 105
Will 19 +1898/-151 69
Jorropo 27 +1638/-248 76
Aayush Rajasekaran 1 +130/-100 10
whyrusleeping 1 +24/-22 4
Marcin Rataj 1 +27/-1 1

v0.13.0 2022-05-04

We're happy to announce go-ipfs 0.13.0, packed full of changes and improvements!

As usual, this release includes important fixes, some of which may be critical for security. Unless the fix addresses a bug being exploited in the wild, the fix will not be called out in the release notes. Please make sure to update ASAP. See our release process for details.

Overview

Below is an outline of all that is in this release, so you get a sense of all that's included.

🛠 BREAKING CHANGES

ipfs block put command

ipfs block put command returns a CIDv1 with raw codec by default now.

  • ipfs block put --cid-codec makes block put return CID with alternative codec
    • This impacts only the returned CID; it does not trigger any validation or data transformation.
    • Retrieving a block with a different codec or CID version than it was put with is valid.
    • Codec names are validated against tables from go-multicodec library.
  • ipfs block put --format is deprecated. It used incorrect codec names and should be avoided for new deployments. Use it only if you need the old, invalid behavior, namely:
    • ipfs block put --format=v0 will produce CIDv0 (implicit dag-pb)
    • ipfs block put --format=cbor will produce CIDv1 with dag-cbor (!)
    • ipfs block put --format=protobuf will produce CIDv1 with dag-pb (!)

ipfs cid codecs command

  • Now lists codecs from go-multicodec library.
  • ipfs cid codecs --supported can be passed to only show codecs supported in various go-ipfs commands.

ipfs cid format command

  • --codec was removed and replaced with --mc to ensure existing users are aware of the following changes:
    • --mc protobuf now correctly points to code 0x50 (was 0x70, which is dab-pg)
    • --mc cbor now correctly points to code 0x51 (was 0x71, which is dag-cbor)

Swarm configuration

  • Daemon will refuse to start if long-deprecated RelayV1 config key Swarm.EnableAutoRelay or Swarm.DisableRelay is set to true.
  • If Swarm.Transports.Network.Relay is disabled, then Swarm.RelayService and Swarm.RelayClient are also disabled (unless they have been explicitly enabled).

Circuit Relay V1 is deprecated

  • By default, Swarm.RelayClient does not use Circuit Relay V1. Circuit V1 support is only enabled when Swarm.RelayClient.StaticRelays are specified.

ls requests for /multistream/1.0.0 are removed

  • go-libp2p 0.19 removed support for undocumented ls command (PR). If you are still using it for internal testing, it is time to refactor (example)

Gateway Behavior

Directory listings returned by the HTTP Gateway won't have size column if the directory is bigger than Gateway.FastDirIndexThreshold config (default is 100).

To understand the wider context why we made these changes, read Highlights below.

🔦 Highlights

🧑‍💼 libp2p Network Resource Manager (Swarm.ResourceMgr)

You can now easily bound how much resource usage libp2p consumes! This aids in protecting nodes from consuming more resources then are available to them.

The libp2p Network Resource Manager is disabled by default, but can be enabled via:

ipfs config --json Swarm.ResourceMgr.Enabled true

When enabled, it applies some safe defaults that can be inspected and adjusted with:

  • ipfs swarm stats --help
  • ipfs swarm limit --help

User changes persist to config at Swarm.ResourceMgr.

The Resource Manager will be enabled by default in a future release.

🔃 Relay V2 client with auto discovery (Swarm.RelayClient)

All the pieces are enabled for hole-punching by default, improving connecting with nodes behind NATs and Firewalls!

This release enables Swarm.RelayClient by default, along with circuit v2 relay discovery provided by go-libp2p v0.19.0. This means:

  1. go-ipfs will coordinate with the counterparty using a relayed connection, to upgrade to a direct connection through a NAT/firewall whenever possible.
  2. go-ipfs daemon will automatically use public relays if it detects that it cannot be reached from the public internet (e.g., it's behind a firewall). This results in a /p2p-circuit address from a public relay.

Notes:

🌉 HTTP Gateway improvements

HTTP Gateway enables seamless interop with the existing Web, clients, user agents, tools, frameworks and libraries.

This release ships the first batch of improvements that enable creation of faster and smarter CDNs, and unblocks creation of light clients for Mobile and IoT.

Details below.

🍱 Support for Block and CAR response formats

Alternative response formats from Gateway can be requested to avoid needing to trust a gateway.

For now, {format} is limited to two options:

  • raw – fetching single block
  • car – fetching entire DAG behind a CID as a CARv1 stream

When not set, the default UnixFS response is returned.

Why these two formats? Requesting Block or CAR for /ipfs/{cid} allows a client to use gateways in a trustless fashion. These types of gateway responses can be verified locally and rejected if digest inside of requested CID does not match received bytes. This enables creation of "light IPFS clients" which use HTTP Gateways as inexpensive transport for content-addressed data, unlocking use in Mobile and IoT contexts.

Future releases will add support for dag-json and dag-cbor responses.

There are two ways for requesting CID specific response format:

  1. HTTP header: Accept: application/vnd.ipld.{format}
  1. URL parameter: ?format=
  • Useful for creating "Download CAR" links.

Usage examples:

  1. Downloading a single raw Block and manually importing it to the local datastore:
$ curl  -H 'Accept: application/vnd.ipld.raw' "http://127.0.0.1:8080/ipfs/QmZULkCELmmk5XNfCgTnCyFgAVxBRBXyDHGGMVoLFLiXEN" --output block.bin
$ cat block.bin | ipfs block put
$ ipfs cat QmZULkCELmmk5XNfCgTnCyFgAVxBRBXyDHGGMVoLFLiXEN
hello
  1. Downloading entire DAG as a CAR file and importing it:
$ ipfs resolve -r  /ipns/webui.ipfs.io
/ipfs/bafybeiednzu62vskme5wpoj4bjjikeg3xovfpp4t7vxk5ty2jxdi4mv4bu
$ curl  -H 'Accept: application/vnd.ipld.car' "http://127.0.0.1:8080/ipfs/bafybeiednzu62vskme5wpoj4bjjikeg3xovfpp4t7vxk5ty2jxdi4mv4bu" --output webui.car
$ ipfs dag import webui.car
$ ipfs dag stat bafybeiednzu62vskme5wpoj4bjjikeg3xovfpp4t7vxk5ty2jxdi4mv4bu --offline
Size: 27684934, NumBlocks: 394

See also:

🐎 Fast listing generation for huge directories

Added Gateway.FastDirIndexThreshold configuration, which allows for fast listings of big directories, without the linear slowdown caused by reading size metadata from child nodes.

As an example, the CID bafybeiggvykl7skb2ndlmacg2k5modvudocffxjesexlod2pfvg5yhwrqm represents UnixFS directory with over 10k (10100) of files.

Opening it with go-ipfs 0.12 would require fetching size information of each file, which would take a long long time, most likely causing timeout in the browser or CDN, and introducing unnecessary burden on the gateway node.

go-ipfs 0.13 opens it instantly, because the number of items is bigger than the default Gateway.FastDirIndexThreshold and only the root UnixFS node needs to be resolved before the HTML Dir Index is returned to the user.

Notes:

  • The default threshold is 100 items.
  • Setting to 0 will enable fast listings for all directories.
  • CLI users will note that this is equivalent to running ipfs ls -s --size=false --resolve-type=false /ipfs/bafybeiggvykl7skb2ndlmacg2k5modvudocffxjesexlod2pfvg5yhwrqm. Now the same speed is available on the gateways.
🎫 Improved Etag and If-None-Match for bandwidth savings

Every response type has an unique Etag which can be used by the client or CDN to save bandwidth, as a gateway does not need to resend a full response if the content was not changed.

Gateway evaluates Etags sent by a client in If-None-Match and returns status code 304 (Not Modified) on strong or weak match (RFC 7232, 2.3).

⛓️ Added X-Ipfs-Roots for smarter HTTP caches

X-Ipfs-Roots is now returned with every Gateway response. It is a way to indicate all CIDs required for resolving path segments from X-Ipfs-Path. Together, these two headers are meant to improve interop with existing HTTP software (load-balancers, caches, CDNs).

This additional information allows HTTP caches and CDNs to make better decisions around cache invalidation: not just invalidate everything under specific IPNS website when the root changes, but do more fine-grained cache invalidation by detecting when only a specific subdirectory (branch of a DAG) changes.

🌡️ Added metrics per response type

New metrics can be found at /debug/metrics/prometheus on the RPC API port (127.0.0.1:5001 is the default):

  • gw_first_content_block_get_latency_seconds – the time until the first content block is received on GET from the gateway (no matter the content or response types)
  • gw_unixfs_file_get_duration_seconds – the time to serve an entire UnixFS file from the gateway
  • gw_unixfs_gen_dir_listing_get_duration_seconds – the time to serve a generated UnixFS HTML directory listing from the gateway
  • gw_car_stream_get_duration_seconds – the time to GET an entire CAR stream from the gateway
  • gw_raw_block_get_duration_seconds – The time to GET an entire raw Block from the gateway

🕵️ OpenTelemetry tracing

Opt-in tracing support with many spans for tracing the duration of specific tasks performed by go-ipfs.

See Tracing for details.

We will continue to add tracing instrumentation throughout IPFS subcomponents over time.

How to use Jaeger UI for visual tracing?

One can use the jaegertracing/all-in-one Docker image to run a full Jaeger stack and configure go-ipfs to publish traces to it (here, in an ephemeral container):

$ docker run --rm -it --name jaeger \
    -e COLLECTOR_ZIPKIN_HOST_PORT=:9411 \
    -p 5775:5775/udp \
    -p 6831:6831/udp \
    -p 6832:6832/udp \
    -p 5778:5778 \
    -p 16686:16686 \
    -p 14268:14268 \
    -p 14250:14250 \
    -p 9411:9411 \
    jaegertracing/all-in-one

Then, in other terminal, start go-ipfs with Jaeger tracing enabled:

$ OTEL_TRACES_EXPORTER=jaeger ipfs daemon

Finally, the Jaeger UI is available at http://localhost:16686

Below are examples of visual tracing for Gateway requests. (Note: this a preview how useful this insight is. Details may look different now, as we are constantly improving tracing annotations across the go-ipfs codebase.)

CAR Block File Directory
2022-04-01_01-46 block_2022-04-01_01-47 2022-04-01_01-49 dir_2022-04-01_01-48

🩺 Built-in ipfs diag profile to ease debugging

The diag profile command has been expanded to include all information that was previously included in the collect-profiles.sh script, and the script has been removed. Profiles are now collected in parallel, so that profile collection is much faster. Specific profiles can also be selected for targeted debugging.

See ipfs diag profile --help for more details.

For general debugging information, see the debug guide.

🔑 Support for PEM/PKCS8 for key import/export

It is now possible to import or export private keys wrapped in interoperable PEM PKCS8 by passing --format=pem-pkcs8-cleartext to ipfs key import and export commands.

This improved interop allows for key generation outside of the IPFS node:

$ openssl genpkey -algorithm ED25519 > ed25519.pem
$ ipfs key import test-openssl -f pem-pkcs8-cleartext ed25519.pem

Or using external tools like the standard openssl to get a PEM file with the public key:

  $ ipfs key export testkey --format=pem-pkcs8-cleartext -o privkey.pem
  $ openssl pkey -in privkey.pem -pubout > pubkey.pem

🧹 Using standard IPLD codec names across the CLI/HTTP API

This release makes necessary (breaking) changes in effort to use canonical codec names from multicodec/table.csv. We also switched to CIDv1 in block put. The breaking changes are discussed above.

🐳 Custom initialization for Docker

Docker images published at https://hub.docker.com/r/ipfs/go-ipfs/ now support custom initialization by mounting scripts in the /container-init.d directory in the container. Scripts can set custom configuration using ipfs config, or otherwise customize the container before the daemon is started.

Scripts are executed sequentially and in lexicographic order, before the IPFS daemon is started and after ipfs init is run and the swarm keys are copied (if the IPFS repo needs initialization).

For more information, see:

RPC API docs for experimental and deprecated commands

https://docs.ipfs.tech/reference/kubo/rpc/ now includes separate sections for experimental and deprecated commands.

We also display a warning in the command line:

$ ipfs name pubsub state --help
WARNING:   EXPERIMENTAL, command may change in future releases

Yamux over Mplex

The more fully featured yamux stream multiplexer is now prioritized over mplex for outgoing connections.

Changelog

Full Changelog

❤ Contributors

Contributor Commits Lines ± Files Changed
Marten Seemann 347 +14453/-12552 842
Rod Vagg 28 +8848/-4033 214
vyzo 133 +7959/-1783 231
hannahhoward 40 +3761/-1652 175
Will Scott 39 +2885/-1784 93
Daniel Martí 32 +3043/-969 103
Adin Schmahmann 48 +3439/-536 121
Gus Eggert 29 +2644/-788 123
Steven Allen 87 +2417/-840 135
Marcin Rataj 29 +2312/-942 75
Will 11 +2520/-62 56
Lucas Molas 28 +1602/-578 90
Raúl Kripalani 18 +1519/-271 38
Brian Tiger Chow 20 +833/-379 40
Masih H. Derkani 5 +514/-460 8
Jeromy Johnson 53 +646/-302 83
Łukasz Magiera 26 +592/-245 43
Artem Mikheev 2 +616/-120 5
Franky W 2 +49/-525 9
Laurent Senta 3 +468/-82 52
Hector Sanjuan 32 +253/-187 62
Juan Batiz-Benet 8 +285/-80 18
Justin Johnson 2 +181/-88 2
Thibault Meunier 5 +216/-28 8
James Wetter 2 +234/-1 2
web3-bot 36 +158/-66 62
gammazero 7 +140/-84 12
Rachel Chen 2 +165/-57 17
Jorropo 18 +108/-99 26
Toby 2 +107/-86 11
Antonio Navarro Perez 4 +82/-103 9
Dominic Della Valle 4 +148/-33 6
Ian Davis 2 +152/-28 6
Kyle Huntsman 2 +172/-6 5
huoju 4 +127/-41 6
Jeromy 19 +71/-58 31
Lars Gierth 12 +63/-54 20
Eric Myhre 3 +95/-15 8
Caian Benedicto 1 +69/-12 6
Raúl Kripalani 2 +63/-13 2
Anton Petrov 1 +73/-0 1
hunjixin 2 +67/-2 5
odanado 1 +61/-0 1
Andrew Gillis 2 +61/-0 3
Kevin Atkinson 6 +21/-34 7
Richard Ramos 1 +51/-0 2
Manuel Alonso 1 +42/-9 2
Jakub Sztandera 10 +37/-13 13
Aarsh Shah 1 +39/-5 2
Dave Justice 1 +32/-4 2
Tommi Virtanen 3 +23/-9 4
tarekbadr 1 +30/-1 1
whyrusleeping 1 +26/-4 3
Petar Maymounkov 2 +30/-0 4
rht 3 +17/-10 4
Miguel Mota 1 +23/-0 1
Manfred Touron 1 +21/-2 2
watjurk 1 +17/-5 1
SukkaW 1 +11/-11 5
Nicholas Bollweg 1 +21/-0 1
Ho-Sheng Hsiao 2 +11/-10 6
chblodg 1 +18/-2 1
Friedel Ziegelmayer 2 +18/-0 2
Shu Shen 2 +15/-2 3
Peter Rabbitson 1 +15/-2 1
galargh 2 +15/-0 2
ᴍᴀᴛᴛ ʙᴇʟʟ 3 +13/-1 4
aarshkshah1992 3 +12/-2 3
RubenKelevra 4 +5/-8 5
Feiran Yang 1 +11/-0 2
zramsay 2 +0/-10 2
Teran McKinney 1 +8/-2 1
Richard Littauer 2 +5/-5 5
Elijah 1 +10/-0 1
Dimitris Apostolou 2 +5/-5 5
Michael Avila 3 +8/-1 4
siiky 3 +4/-4 3
Somajit 1 +4/-4 1
Sherod Taylor 1 +0/-8 2
Eclésio Junior 1 +8/-0 1
godcong 3 +4/-3 3
Piotr Galar 3 +3/-4 3
jwh 1 +6/-0 2
dependabot[bot] 1 +3/-3 1
Volker Mische 1 +4/-2 1
Aayush Rajasekaran 1 +3/-3 1
rene 2 +3/-2 2
keks 1 +5/-0 1
Hlib 1 +4/-1 2
Arash Payan 1 +5/-0 1
Wayback Archiver 1 +2/-2 1
T Mo 1 +2/-2 1
Ju Huo 1 +2/-2 1
Ivan 2 +2/-2 2
Ettore Di Giacinto 2 +3/-1 2
Christian Couder 1 +3/-1 1
ningmingxiao 1 +0/-3 1
市川恭佑 (ebi) 1 +1/-1 1
star 1 +0/-2 1
alliswell 1 +0/-2 1
Preston Van Loon 1 +2/-0 1
Nguyễn Gia Phong 1 +1/-1 1
Nato Boram 1 +1/-1 1
Mildred Ki'Lya 1 +2/-0 2
Michael Burns 1 +1/-1 1
Glenn 1 +1/-1 1
George Antoniadis 1 +1/-1 1
David Florness 1 +1/-1 1
Daniel Norman 1 +1/-1 1
Coenie Beyers 1 +1/-1 1
Benedikt Spies 1 +1/-1 1
Abdul Rauf 1 +1/-1 1
makeworld 1 +1/-0 1
ignoramous 1 +0/-1 1
Omicron166 1 +0/-1 1
Jan Winkelmann 1 +1/-0 1
Dr Ian Preston 1 +1/-0 1
Baptiste Jonglez 1 +1/-0 1