A hash-friendly Haskell Package Repository.
Foliage is a tool to create custom or private Haskell package repositories, in a fully reproducible way.
Like GitHub Pages but for Haskell Packages
A "Hackage repository" is collection of source distributions and cabal files. In addition, Hackage has implemented The Update Framework (TUF) and the repository also includes cryptographic metadata (public keys and signatures).
These files are commonly served by Hackage proper, that is the central deployment of hackage-server.
Foliage explores the idea of creating and serving this content as a static website, generated programmatically a deterministically from textual input files.
Company XYZ has developed many Haskell packages, some of which are forks of public Haskell libraries. For various reasons XYZ might not be inclined in publishing their packages to Hackage. If XYZ is using multiple repositories for version control, keeping all the company's packages in sync will become a problem.
Currently XYZ's best option is to use cabal.project files. Each cabal
package can declare its non-Hackage dependencies using the
source-repository-package stanza.
Note: packages can be grouped together into a cabal project, in which case
source-repository-package stanzas become the project dependencies; this
distintion is inconsequential to our example.
E.g. if packageA needs packageB, hosted on GitHub; packageA's
cabal.project will include:
source-repository-package
type: git
location: https://github.com/Company-XYZ/packageB
tag: e70cf0c171c9a586b62b3f75d72f1591e4e6aaa1
While the presence of a git tag makes this quite reproducible; a problem arises in that these dependencies are not transitive. Without any versioning to help, one has to manually pick a working set of dependencies.
E.g. if packageC depends on packageA, packageC cabal.project will have to
include something like:
-- Direct dependency
source-repository-package
type: git
location: https://github.com/Company-XYZ/packageA
tag: e76fdc753e660dfa615af6c8b6a2ad9ddf6afe70
-- From packageA
source-repository-package
type: git
location: https://github.com/Company-XYZ/packageB
tag: e70cf0c171c9a586b62b3f75d72f1591e4e6aaa1
Having an internal company Hackage, solves the above problem by reintroducing versioning and a familiar workflow for handling Hackage dependencies; while maintaining absolute control and flexibility over versioning policies and dependency management.
When the team behind packageA wants to push out a new version, say version
1.2.3.4, all they have to do is to update the foliage repository with a
file packageA/1.2.3.4/meta.toml with content:
timestamp = 2022-03-29T06:19:50+00:00
github = { repo = "Company-XYZ/packageA", rev = "e76fdc753e660dfa615af6c8b6a2ad9ddf6afe70" }Note: Rather than github = one che use the more generic url = , e.g.
url = "https://hackage.haskell.org/package/shake-0.19.6/shake-0.19.6.tar.gz"ℹ️ Although the timestamp field in the package source metadata is
optional, it is highly reccomended if you intend your repository users to
be able to use cabal's index-state functionality. Adding a timestamp
every time you add a package version ensures the newly created index is
"compatible" with what the users have already fetched.
It's reccomended to create a working directory first.
Let's add a package (say typed-protocols-0.1.0.0 from https://github.com/input-output-hk/ouroboros-network/tree/master/typed-protocols, at commit hash fa10cb4eef1e7d3e095cec3c2bb1210774b7e5fa).
$ mkdir -p _sources/typed-protocols/0.1.0.0
$ cat _sources/typed-protocols/0.1.0.0/meta.toml
github = { repo = "input-output-hk/ouroboros-network", rev = "fa10cb4eef1e7d3e095cec3c2bb1210774b7e5fa" }
subdir = "typed-protocols"Run foliage build.
$ foliage build
🌿 Foliage
You don't seem to have created a set of TUF keys. I will create one in _keys
Current time set to 2022-05-16T09:33:26Z. You can set a fixed time using the --current-time option.
Expiry time set to 2023-05-16T09:33:26Z (a year from now).
# curl (for RemoteAsset "https://github.com/input-output-hk/ouroboros-network/tarball/fa10cb4eef1e7d3e095cec3c2bb1210774b7e5fa")
# tar (for OracleQ (PreparePackageSource (PackageId {pkgName = "typed-protocols", pkgVersion = "0.1.0.0"})))
# cp (for OracleQ (PreparePackageSource (PackageId {pkgName = "typed-protocols", pkgVersion = "0.1.0.0"})))
# writing (for _repo/root.json)
# writing (for _repo/mirrors.json)
_cache/packages/typed-protocols/0.1.0.0
Creating source distribution for typed-protocols-0.1.0.0
# cabal (for _repo/package/typed-protocols-0.1.0.0.tar.gz)
# mv (for _repo/package/typed-protocols-0.1.0.0.tar.gz)
# writing (for _repo/01-index.tar)
# writing (for _repo/01-index.tar.gz)
# writing (for _repo/snapshot.json)
# writing (for _repo/timestamp.json)
All done. The repository is now available in _repo.
If you want to rely on the cabal index-state feature you need to specify a
timestamp in the meta.toml file.
E.g.
$ cat _sources/typed-protocols/0.1.0.0/meta.toml
github = { repo = "input-output-hk/ouroboros-network", rev = "fa10cb4eef1e7d3e095cec3c2bb1210774b7e5fa" }
subdir = "typed-protocols"
timestamp = 2022-03-29T06:19:50+00:00
ℹ️ Foliage uses the metadata timestamps to determine the
order of the entries in 01-index. This allows you to create an index that
can be updated incrementally and can be used with cabal's
index-state
feature.
With the input above foliage will produce the following:
_repo
├── 01-index.tar
├── 01-index.tar.gz
├── index
│ └── typed-protocols
│ └── 0.1.0.0
│ ├── package.json
│ └── typed-protocols.cabal
├── mirrors.json
├── package
│ └── typed-protocols-0.1.0.0.tar.gz
├── root.json
├── snapshot.json
└── timestamp.json
The content of _repo can be served over HTTP(s) to cabal, e.g. from a
cabal.project file.
repository packages.example.org
url: http://packages.example.org/
secure: True
root-keys:
-- root-keys ids, see below
144d97d34d0a86adb1ca7d6bdc1b2d9f0c9123e3c29e3765f5a9eece345ce4f9
a15f6ae88a26638934d90eff28da29990a4b12c8bb0b2c12f07e9a510e839a97
fde23c79a14bcbef6ccf198b4ad94ded4092784fcaed17c3d184008e9bf6f722
key-threshold: 3
Foliage creates a set of private keys to sign the TUF metadata at first
use. By default, the keys are created in _keys:
_keys/
├── mirrors
│ ├── 105369fb9cb1555cf4517be3e885215a7bc730bd59cf3084ea7140f9692ae847.json
│ ├── 2178cff7b2a3a6edd396371c03bc8fddb77b77b104a9fd97f6291f2c49285946.json
│ └── 4689f8a6d905a59536213a27ea577b34ff3e6d79d5a7b375458d3bb6026a5e13.json
├── root
│ ├── 144d97d34d0a86adb1ca7d6bdc1b2d9f0c9123e3c29e3765f5a9eece345ce4f9.json
│ ├── a15f6ae88a26638934d90eff28da29990a4b12c8bb0b2c12f07e9a510e839a97.json
│ └── fde23c79a14bcbef6ccf198b4ad94ded4092784fcaed17c3d184008e9bf6f722.json
├── snapshot
│ └── 07a918ccdb3ac0600a8b65f3bc96da18dfc5be65c023c64ccd0fb6a04692b64d.json
├── target
│ ├── 793b1b2730db6ec5247934ad0cc7b28ed3b05eae4ffec7e28e7b1739f1cb65b4.json
│ ├── 908041deaae700390dfd7b4c8d3eca7049c8172a205bea19de8314c4fd9ca56f.json
│ └── f102c7035da2a3b5fa82b90d8afe3e8892d13a8c6b7a4281403c340317a35014.json
└── timestamp
└── 141da8eb2ccba61c2f6bb656b2292970d086770f5bf7d53802d2bc0ec1defa26.json
The root-keys ids are simply the names of they key files in
_keys/roots/*.json.
These keys are small enough you can store them in an environment variable. E.g.
Save:
$ KEYS=$(tar cz -C _keys . | base64)Restore:
$ mkdir _keys
$ echo "$KEYS" | base64 -d | tar xz -C _keysFoliage supports cabal file revisions. Adding the following snippet to a
package's meta.toml, will make foliage look for a cabal file in
<package>/<version>/revisions/1.cabal.
[[revisions]]
number = 1
timestamp = 2022-03-22T14:15:00+00:00
The revised cabal file will enter the index with the timestamp provided in
meta.toml.
Foliage also supports patches. Any file with .patch extension in
<package>/<version>/patches will be applied as a patch to the package
sources.
Patches are applied very early in the process, and they can modify any file in the source tree.
Revisions are independent of patches, a patched cabal file will be subject to revisions just like the original cabal file.
You can introduce new versions of existing packages by patching the cabal
file. Foliage supports a short cut for this use case. If the package
metadata file <package>/<version>/meta.toml includes the option
force-version = true, then the version in the package cabal file will be
overwritten with <version>. Notice this is done after applying patches.
Foliage uses curl to fetch remote assets. If you need to access private repositories you can use one of the following methods.
- You can instruct curl to use a netrc file.
- Add netrc-optional to ~/.curlrc (see here for the places curl looks in).
- Add the relevant credentials to ~/.netrc
- You can also add the credentials directly to curl's configuration, e.g.
- Add
user = "USERNAME:PASSWORD"to .curlrc in the current directly - Set
CURL_HOME=$PWD
- Add
- Andrea Bedini (@andreabedini)