Skip to content

Commit

Permalink
Readme edits
Browse files Browse the repository at this point in the history
  • Loading branch information
bporter816 committed Aug 12, 2023
1 parent 7eca132 commit 49e8054
Showing 1 changed file with 8 additions and 8 deletions.
16 changes: 8 additions & 8 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,8 +2,8 @@

A website that tracks the release date of [Hollow Knight: Silksong](https://store.steampowered.com/app/1030300/Hollow_Knight_Silksong/).

To get the site to update dynamically, I looked into options for how to fetch data from Steam.
Steam's website hits an underlying API that can be queried directly:
To get the site to update dynamically, I looked into options for fetching data from Steam.
Their website hits an underlying API that can be queried directly:
```sh
$ curl 'https://store.steampowered.com/api/appdetails?appids=1030300' | jq '.["1030300"].data.release_date'
{
Expand All @@ -12,10 +12,10 @@ $ curl 'https://store.steampowered.com/api/appdetails?appids=1030300' | jq '.["1
}
```

My first thought was to fetch this data client-side in the browser, but this didn't work due to CORS (the API doesn't return an `Access-Control-Allow-Origin` header).
My first thought was to fetch the data client-side in the browser, but this didn't work due to CORS (the API doesn't return an `Access-Control-Allow-Origin` header).

Next, I tried to fetch the data in a Cloudflare Worker, still processing it client-side. When I started making the request in the Worker, I began getting 403 responses back.
I suspected the Steam API was blocking Cloudflare Workers, and confirmed this by making a test request with one of the headers that Cloudflare injects into outbound requests from Workers:
Next, I tried fetching the data in a Cloudflare Worker, still processing it client-side. When I started making the request in the Worker, I began getting 403 responses back.
I suspected the Steam API was blocking Cloudflare Workers, and confirmed this by making a test request with a header that Cloudflare injects into outbound requests from Workers:
```sh
$ curl -I 'https://store.steampowered.com/api/appdetails?appids=1030300'
HTTP/1.1 200 OK
Expand All @@ -26,10 +26,10 @@ HTTP/1.1 403 Forbidden
...
```

OK, looks like Steam doesn't want Cloudflare Workers to scrape their API. Fair enough. Not wanting to give up so easily, I rewrote the Worker script in an AWS Lambda function.
OK, looks like Steam doesn't want Cloudflare Workers scraping their API. Fair enough. Not wanting to give up so easily, I rewrote the Worker script in an AWS Lambda function.
This worked, but still had a couple of undesirable characteristics:
* Data fetching on page load takes time, and there is a noticable lag when the page loads before the content appears.
* Data fetching on page load takes time, and there is a noticable lag before the content appears.
* The Lambda function is public, so I can rack up costs from bots hitting it, etc. It's just not very elegant.

To overcome these issues, I changed the site to build hourly, producing a static site with the data from the Steam API injected at build time.
To overcome these issues, I changed the site to build hourly, producing a static site with the Steam API data injected at build time.
The finished site is built with Jekyll on GitHub Pages and updates via a scheduled GitHub Actions workflow.

0 comments on commit 49e8054

Please sign in to comment.