From 7ceb947eb27abaa947129117415be45e70d95054 Mon Sep 17 00:00:00 2001 From: Benjamin Porter Date: Sat, 12 Aug 2023 00:48:33 -0500 Subject: [PATCH] Update readme --- README.md | 36 +++++++++++++++++++++++++++++++++++- 1 file changed, 35 insertions(+), 1 deletion(-) diff --git a/README.md b/README.md index bb871fb..944546f 100644 --- a/README.md +++ b/README.md @@ -1 +1,35 @@ -# issilksongoutyet.com \ No newline at end of file +# issilksongoutyet.com + +A website that tracks the release date of [Hollow Knight: Silksong](https://store.steampowered.com/app/1030300/Hollow_Knight_Silksong/). + +To get the site to update dynamically, I looked into options for how to fetch data from Steam. +Steam's website hits an underlying API that can be queried directly: +```sh +$ curl 'https://store.steampowered.com/api/appdetails?appids=1030300' | jq '.["1030300"].data.release_date' +{ + "coming_soon": true, + "date": "To be announced" +} +``` + +My first thought was to fetch this data client-side in the browser, but this didn't work due to CORS (the API doesn't return an `Access-Control-Allow-Origin` header). + +Next, I tried to fetch the data in a Cloudflare Worker, still processing it client-side. When I started making the request in the Worker, I began getting 403 responses back. +I suspected the Steam API was blocking Cloudflare Workers, and confirmed this by making a test request with one of the headers that Cloudflare injects into outbound requests from Workers: +```sh +$ curl -I 'https://store.steampowered.com/api/appdetails?appids=1030300' +HTTP/1.1 200 OK +... + +$ curl -I -H 'cf-worker: blah' 'https://store.steampowered.com/api/appdetails?appids=1030300' +HTTP/1.1 403 Forbidden +... +``` + +OK, looks like Steam doesn't want Cloudflare Workers to scrape their API. Fair enough. Not wanting to give up so easily, I rewrote the Worker script in an AWS Lambda function. +This worked, but still had a couple of undesirable characteristics: +* Data fetching on page load takes time, and there is a noticable lag when the page loads before the content appears. +* The Lambda function is public, so I can rack up costs from bots hitting it, etc. It's just not very elegant. + +To overcome these issues, I changed the site to build hourly, producing a static site with the data from the Steam API injected at build time. +The finished site is built with Jekyll on GitHub Pages and updates via a scheduled GitHub Actions workflow.