Add Trello webhook listener (#49)

* implement trello webhook request handler

* add debug logs

* use sha1 instead of sha256

* remove debug logs

* parse webhook request body & parse archivedcard ID

* define new route for trello webhooks

* fetch archived card details via trello api

* get rid of useless test

* fix bug

* polish

* add debug logs

* rename "(data) source" as "service"

* introduce the new SERVICES envar for server mode

* bump go version in CI & CD configs

* add hack to skip envar checks during dev/testing

* split the config package into 2 files

* make post request to matching service(s) with archived card data

* update readme

* improve error handling

.env.example

@@ -0,0 +1,10 @@
+APP_ENV=development
+PORT=XXXX
+USERNAME=user
+PASSWORD=pwd
+SERVICES=<label_id_1>@<endpoint_url_1>,<label_id_2>@<endpoint_url_2>
+TRELLO_API_KEY=xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
+TRELLO_API_TOKEN=xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
+TRELLO_BOARD_ID=xxxxxxxxxxxxxxxxxxxxxxxx
+TRELLO_SECRET=xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
+TRELLO_WEBHOOK_CALLBACK_URL=<url>

.github/workflows/ci.yml

@@ -11,7 +11,7 @@ jobs:
     - name: Setup Go
       uses: actions/setup-go@v1
       with:
-        go-version: 1.16
+        go-version: 1.18
 
     - name: Check gofmt
       run: test -z "$(gofmt -s -d .)"

.github/workflows/release.yml

@@ -18,7 +18,7 @@ jobs:
       - name: Setup Go
         uses: actions/setup-go@v1
         with:
-          go-version: 1.16.x
+          go-version: 1.18.x
 
       - name: GoReleaser
         uses: goreleaser/goreleaser-action@v1

.gitignore

@@ -1,2 +1,4 @@
+.env
 *.json
+webhook.sh
 !config.example.json

README.md

@@ -4,52 +4,53 @@
 [![Go Report Card](https://goreportcard.com/badge/github.com/utkuufuk/entrello)](https://goreportcard.com/report/github.com/utkuufuk/entrello)
 [![Coverage Status](https://coveralls.io/repos/github/utkuufuk/entrello/badge.svg)](https://coveralls.io/github/utkuufuk/entrello)
 
-Polls compatible data sources and keeps your Trello cards synchronized with fresh data.
+Minimum Go version required: `1.18`
 
-Let's say you have an HTTP endpoint that returns GitHub issues assigned to you upon a `GET` request.
-You can point `entrello` to that endpoint to keep your GitHub issues synchronized in your Trello board.
-
-Each data source must return a JSON array of Trello card objects upon a `GET` request. You can import and use the `NewCard` function from `pkg/trello/trello.go` in order to construct Trello card objects.
-
-- Can be run as a scheduled job:
+## Usage
+- Polls compatible services and keeps your Trello cards synchronized with fresh data.
+- Listens for events from your Trello board and forwards "archived card" events to the matching service, if any.
+- Can be run as a scheduled job, or as an HTTP server:
     ```sh
+    # 1. CRON JOB
     go run ./cmd/runner
-    ```
-- Can be run as an HTTP server:
-    ```sh
-    PORT=<port> USERNAME=<user> PASSWORD=<password> go run ./cmd/server
-    ```
 
-    In this case, the runner can be triggered by a `POST` request to the server like this:
-    ```sh
+
+    # 2. HTTP SERVER
+    go run ./cmd/server
+
+    # make a `POST` request to the HTTP server to trigger polling
     curl -d @config.json <SERVER_URL> -H "Authorization: Basic <base64(<user>:<password>)>"
     ```
 
-## Configuration
+### Example Use Case
+Let's say you have an HTTP service that returns GitHub issues that are assigned to you upon a `GET` request.
+Then `entrello` can use it as a data source to keep your GitHub issues synchronized in your Trello board.
+
+Moreover, when you use `entrello` as a server, it can forward "archived card" events to your GitHub service.
+This means that whenever one of your "GitHub" cards is archived, your GitHub service can be notified and take an action of your choosing, e.g. it could close the corresponding GitHub issue. 
+
+## Runner Configuration
 Copy and rename `config.example.json` as `config.json` (default), then set your own values in `config.json`.
 
 You can also use a custom config file path using the `-c` flag:
 ```sh
 go run ./cmd/runner -c /path/to/config/file
 ```
 
-### Trello
-You need to set your [Trello API key & token](https://trello.com/app-key) in the configuraiton file, as well as the Trello board ID.
-
-### Data Sources
-For each data source, the following parameters have to be specified. (See `config.example.json`)
+### Configuring Services in Runner Mode
+Each configured service must return a JSON array of Trello card objects upon a `GET` request. See `pkg/trello/trello.go` for reference. For each service, the following configuration parameters have to be specified:
 
-- `name` &mdash; Data source name.
+- `name` &mdash; Service name.
 
-- `endpoint` &mdash; Data source endpoint. `entrello` will make a `GET` request to this endpoint to fetch fresh cards from the data source.
+- `endpoint` &mdash; Service endpoint. `entrello` will make a `GET` request to this endpoint to fetch fresh cards from the service.
 
-- `strict` &mdash; When strict mode is enabled, previously auto-generated cards that are no longer present in the fresh data will be deleted. For instance, with a GitHub data source, strict mode can be useful for automatically removing previously auto-generated cards for issues/PRs from the board when the corresponding issues/PRs are closed/merged.
+- `strict` &mdash; When strict mode is enabled, previously auto-generated cards that are no longer present in the fresh data will be deleted. For instance, with a GitHub service, strict mode can be useful for automatically removing previously auto-generated cards for issues/PRs from the board when the corresponding issues/PRs are closed/merged.
 
-- `label_id` &mdash; **Distinct** Trello label ID associated with the data source.
+- `label_id` &mdash; **Distinct** Trello label ID associated with the service.
 
-- `list_id` &mdash; Trello list ID for the data source to determine where to insert new cards. The selected list must be in the same board as configured by the `board_id` parameter.
+- `list_id` &mdash; Trello list ID for the service to determine where to insert new cards. The selected list must be in the same board as configured by the `board_id` parameter.
 
-- `period` &mdash; Polling period for the data source. Some examples:
+- `period` &mdash; Polling period for the service. Some examples:
     ```json
     // query at 3rd, 6th, 9th, ... of each month
     "period": {
@@ -76,10 +77,44 @@ For each data source, the following parameters have to be specified. (See `confi
     }
     ```
 
-## Example Cron Job
+### Example Runner Cron Job
 Assuming `config.json` is located in the current working directory:
 ``` sh
 0 * * * * cd /home/you/git/entrello && /usr/local/go/bin/go run ./cmd/runner
 ```
 
 Make sure that the cron job runs frequently enough to keep up with the most frequent custom interval in your configuration. For instance, it wouldn't make sense to define a custom period of 15 minutes while the cron job only runs every hour.
+
+## Server Configuration
+Copy and rename `.env.example` as `.env`, then set your own values in `.env`.
+
+You can trigger the runner by making a `POST` request to the root URL of your server with the runner configuration in the request body:
+```sh
+curl -d @config.json <SERVER_URL> -H "Authorization: Basic <base64(<USERNAME>:<PASSWORD>)>"
+```
+
+You can create a Trello webhook pointed at `<SERVER_URL>/trello-webhook` in order to listen to events from your Trello board.
+
+### Creating Trello Webhooks
+You can create a Trello webhook using the following command:
+
+```sh
+curl -X POST -H "Content-Type: application/json" \
+https://api.trello.com/1/tokens/<api_token>/webhooks/ \
+-d '{
+  "key": "<api_key>",
+  "callbackURL": "<url>",
+  "idModel": "<id_model>",
+  "description": "<desc>"
+}'
+```
+
+* `api_token` &mdash; Trello API token
+* `api_key` &mdash; Trello API key
+* `url` &mdash; Entrello Server URL
+* `id_model` &mdash; Trello Board ID
+* `desc` &mdash; Arbitrary description string
+
+For more information, see
+* [Trello Webhooks Guide](https://developer.atlassian.com/cloud/trello/guides/rest-api/webhooks/)
+* [Trello Webhooks Reference](https://developer.atlassian.com/cloud/trello/rest/#api-group-Webhooks)

cmd/runner/main.go

@@ -6,20 +6,20 @@ import (
 
 	"github.com/utkuufuk/entrello/internal/config"
 	"github.com/utkuufuk/entrello/internal/logger"
-	"github.com/utkuufuk/entrello/internal/service"
+	"github.com/utkuufuk/entrello/internal/services"
 )
 
 func main() {
 	var configFile string
 	flag.StringVar(&configFile, "c", "config.json", "config file path")
 	flag.Parse()
 
-	cfg, err := config.ReadConfig(configFile)
+	cfg, err := config.ReadRunnerConfig(configFile)
 	if err != nil {
 		log.Fatalf("Could not read configuration: %v", err)
 	}
 
-	if err = service.Poll(cfg); err != nil {
+	if err = services.Poll(cfg); err != nil {
 		logger.Error(err.Error())
 	}
 }

cmd/server/main.go

@@ -5,21 +5,30 @@ import (
 	"fmt"
 	"io/ioutil"
 	"net/http"
-	"os"
 
 	"github.com/utkuufuk/entrello/internal/config"
 	"github.com/utkuufuk/entrello/internal/logger"
-	"github.com/utkuufuk/entrello/internal/service"
+	"github.com/utkuufuk/entrello/internal/services"
+	"github.com/utkuufuk/entrello/pkg/trello"
 )
 
+var client trello.Client
+
 func main() {
-	http.HandleFunc("/", handler)
-	http.ListenAndServe(fmt.Sprintf(":%s", os.Getenv("PORT")), nil)
+	client = trello.NewClient(config.Trello{
+		ApiKey:   config.ServerCfg.TrelloApiKey,
+		ApiToken: config.ServerCfg.TrelloApiToken,
+		BoardId:  config.ServerCfg.TrelloBoardId,
+	})
+
+	http.HandleFunc("/", handlePollRequest)
+	http.HandleFunc("/trello-webhook", handleTrelloWebhookRequest)
+	http.ListenAndServe(fmt.Sprintf(":%s", config.ServerCfg.Port), nil)
 }
 
-func handler(w http.ResponseWriter, req *http.Request) {
+func handlePollRequest(w http.ResponseWriter, req *http.Request) {
 	if req.Method != http.MethodPost {
-		logger.Warn("Method %s not allowed", req.Method)
+		logger.Warn("Method %s not allowed for %s", req.Method, req.URL.Path)
 		w.WriteHeader(http.StatusMethodNotAllowed)
 		return
 	}
@@ -30,12 +39,12 @@ func handler(w http.ResponseWriter, req *http.Request) {
 		w.WriteHeader(http.StatusUnauthorized)
 		return
 	}
-	if user != os.Getenv("USERNAME") {
+	if user != config.ServerCfg.Username {
 		logger.Warn("Invalid user name: %s", user)
 		w.WriteHeader(http.StatusUnauthorized)
 		return
 	}
-	if pwd != os.Getenv("PASSWORD") {
+	if pwd != config.ServerCfg.Password {
 		logger.Warn("Invalid password: %s", pwd)
 		w.WriteHeader(http.StatusUnauthorized)
 		return
@@ -44,22 +53,88 @@ func handler(w http.ResponseWriter, req *http.Request) {
 	body, err := ioutil.ReadAll(req.Body)
 	if err != nil {
 		logger.Error("Could not read request body: %v", err)
-		w.WriteHeader(http.StatusInternalServerError)
+		w.WriteHeader(http.StatusBadRequest)
 		return
 	}
 
-	var cfg config.Config
+	var cfg config.RunnerConfig
 	if err = json.Unmarshal(body, &cfg); err != nil {
 		logger.Warn("Invalid request body: %v", err)
 		w.WriteHeader(http.StatusBadRequest)
 		return
 	}
 
-	if err = service.Poll(cfg); err != nil {
+	if err = services.Poll(cfg); err != nil {
 		logger.Error(err.Error())
 		w.WriteHeader(http.StatusInternalServerError)
 		return
 	}
 
 	w.WriteHeader(http.StatusOK)
 }
+
+func handleTrelloWebhookRequest(w http.ResponseWriter, req *http.Request) {
+	if req.Method == http.MethodHead {
+		w.WriteHeader(http.StatusOK)
+		return
+	}
+
+	if req.Method != http.MethodPost {
+		logger.Warn("Method %s not allowed for %s", req.Method, req.URL.Path)
+		w.WriteHeader(http.StatusMethodNotAllowed)
+		return
+	}
+
+	hash := req.Header.Get("x-trello-webhook")
+	if hash == "" {
+		logger.Warn("Missing 'x-trello-webhook' header")
+		w.WriteHeader(http.StatusUnauthorized)
+		return
+	}
+
+	body, err := ioutil.ReadAll(req.Body)
+	if err != nil {
+		logger.Error("Could not read Trello webhook request body: %v", err)
+		w.WriteHeader(http.StatusBadRequest)
+		return
+	}
+
+	if !trello.VerifyWebhookSignature(
+		config.ServerCfg.TrelloWebhookCallbackUrl,
+		config.ServerCfg.TrelloSecret,
+		hash,
+		body,
+	) {
+		logger.Warn("Invalid Trello webhook signature")
+		w.WriteHeader(http.StatusUnauthorized)
+		return
+	}
+
+	var wrb trello.WebhookRequestBody
+	if err = json.Unmarshal(body, &wrb); err != nil {
+		logger.Warn("Invalid Trello webhook request body: %v", err)
+		w.WriteHeader(http.StatusBadRequest)
+		return
+	}
+
+	archivedCardId := trello.ParseArchivedCardId(wrb)
+	if archivedCardId == "" {
+		w.WriteHeader(http.StatusAccepted)
+		return
+	}
+
+	archivedCard, err := client.GetCard(archivedCardId)
+	if err != nil {
+		logger.Error("Could not fetch archived Trello card: %v", err)
+		w.WriteHeader(http.StatusInternalServerError)
+		return
+	}
+
+	if err = services.Notify(archivedCard, config.ServerCfg.Services); err != nil {
+		logger.Error("Could not notify service(s) with the archived card data: %v", err)
+		w.WriteHeader(http.StatusInternalServerError)
+		return
+	}
+
+	w.WriteHeader(http.StatusOK)
+}

config.example.json

@@ -5,7 +5,7 @@
     "api_token": "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx",
     "board_id": "xxxxxxxxxxxxxxxxxxxxxxxx"
   },
-  "sources": [
+  "services": [
     {
       "name": "Github Issues",
       "endpoint": "http://<github-issues-endpoint>",

go.mod

@@ -1,11 +1,13 @@
 module github.com/utkuufuk/entrello
 
-// +heroku goVersion go1.16
-go 1.16
+// +heroku goVersion go1.18
+go 1.18
 
 require (
 	github.com/adlio/trello v1.8.0
 	github.com/google/go-cmp v0.5.4
-	github.com/pkg/errors v0.9.1 // indirect
-	golang.org/x/xerrors v0.0.0-20200804184101-5ec99f83aff1 // indirect
+	github.com/joho/godotenv v1.4.0
+	golang.org/x/exp v0.0.0-20220328175248-053ad81199eb
 )
+
+require github.com/pkg/errors v0.9.1 // indirect

go.sum

@@ -2,9 +2,12 @@ github.com/adlio/trello v1.8.0 h1:VU/1zwzuRzATsFC8WiK4f8R0HHQPWpf2H658KEchsmA=
 github.com/adlio/trello v1.8.0/go.mod h1:l2068AhUuUuQ9Vsb95ECMueHThYyAj4e85lWPmr2/LE=
 github.com/google/go-cmp v0.5.4 h1:L8R9j+yAqZuZjsqh/z+F1NCffTKKLShY6zXTItVIZ8M=
 github.com/google/go-cmp v0.5.4/go.mod h1:v8dTdLbMG2kIc/vJvl+f65V22dbkXbowE6jgT/gNBxE=
+github.com/joho/godotenv v1.4.0 h1:3l4+N6zfMWnkbPEXKng2o2/MR5mSwTrBih4ZEkkz1lg=
+github.com/joho/godotenv v1.4.0/go.mod h1:f4LDr5Voq0i2e/R5DDNOoa2zzDfwtkZa6DnEwAbqwq4=
 github.com/pkg/errors v0.8.1/go.mod h1:bwawxfHBFNV+L2hUp1rHADufV3IMtnDRdf1r5NINEl0=
 github.com/pkg/errors v0.9.1 h1:FEBLx1zS214owpjy7qsBeixbURkuhQAwrK5UwLGTwt4=
 github.com/pkg/errors v0.9.1/go.mod h1:bwawxfHBFNV+L2hUp1rHADufV3IMtnDRdf1r5NINEl0=
+golang.org/x/exp v0.0.0-20220328175248-053ad81199eb h1:pC9Okm6BVmxEw76PUu0XUbOTQ92JX11hfvqTjAV3qxM=
+golang.org/x/exp v0.0.0-20220328175248-053ad81199eb/go.mod h1:lgLbSvA5ygNOMpwM/9anMpWVlVJ7Z+cHWq/eFuinpGE=
 golang.org/x/xerrors v0.0.0-20191204190536-9bdfabe68543/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0=
 golang.org/x/xerrors v0.0.0-20200804184101-5ec99f83aff1 h1:go1bK/D/BFZV2I8cIQd1NKEZ+0owSTG1fDTci4IqFcE=
-golang.org/x/xerrors v0.0.0-20200804184101-5ec99f83aff1/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0=