Skip to content

Commit

Permalink
Add Trello webhook listener (#49)
Browse files Browse the repository at this point in the history
* implement trello webhook request handler

* add debug logs

* use sha1 instead of sha256

* remove debug logs

* parse webhook request body & parse archivedcard ID

* define new route for trello webhooks

* fetch archived card details via trello api

* get rid of useless test

* fix bug

* polish

* add debug logs

* rename "(data) source" as "service"

* introduce the new SERVICES envar for server mode

* bump go version in CI & CD configs

* add hack to skip envar checks during dev/testing

* split the config package into 2 files

* make post request to matching service(s) with archived card data

* update readme

* improve error handling
  • Loading branch information
utkuufuk authored Apr 3, 2022
1 parent 3454797 commit 09cfd8e
Show file tree
Hide file tree
Showing 21 changed files with 429 additions and 288 deletions.
10 changes: 10 additions & 0 deletions .env.example
Original file line number Diff line number Diff line change
@@ -0,0 +1,10 @@
APP_ENV=development
PORT=XXXX
USERNAME=user
PASSWORD=pwd
SERVICES=<label_id_1>@<endpoint_url_1>,<label_id_2>@<endpoint_url_2>
TRELLO_API_KEY=xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
TRELLO_API_TOKEN=xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
TRELLO_BOARD_ID=xxxxxxxxxxxxxxxxxxxxxxxx
TRELLO_SECRET=xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
TRELLO_WEBHOOK_CALLBACK_URL=<url>
2 changes: 1 addition & 1 deletion .github/workflows/ci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -11,7 +11,7 @@ jobs:
- name: Setup Go
uses: actions/setup-go@v1
with:
go-version: 1.16
go-version: 1.18

- name: Check gofmt
run: test -z "$(gofmt -s -d .)"
Expand Down
2 changes: 1 addition & 1 deletion .github/workflows/release.yml
Original file line number Diff line number Diff line change
Expand Up @@ -18,7 +18,7 @@ jobs:
- name: Setup Go
uses: actions/setup-go@v1
with:
go-version: 1.16.x
go-version: 1.18.x

- name: GoReleaser
uses: goreleaser/goreleaser-action@v1
Expand Down
2 changes: 2 additions & 0 deletions .gitignore
Original file line number Diff line number Diff line change
@@ -1,2 +1,4 @@
.env
*.json
webhook.sh
!config.example.json
89 changes: 62 additions & 27 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,52 +4,53 @@
[![Go Report Card](https://goreportcard.com/badge/github.com/utkuufuk/entrello)](https://goreportcard.com/report/github.com/utkuufuk/entrello)
[![Coverage Status](https://coveralls.io/repos/github/utkuufuk/entrello/badge.svg)](https://coveralls.io/github/utkuufuk/entrello)

Polls compatible data sources and keeps your Trello cards synchronized with fresh data.
Minimum Go version required: `1.18`

Let's say you have an HTTP endpoint that returns GitHub issues assigned to you upon a `GET` request.
You can point `entrello` to that endpoint to keep your GitHub issues synchronized in your Trello board.

Each data source must return a JSON array of Trello card objects upon a `GET` request. You can import and use the `NewCard` function from `pkg/trello/trello.go` in order to construct Trello card objects.

- Can be run as a scheduled job:
## Usage
- Polls compatible services and keeps your Trello cards synchronized with fresh data.
- Listens for events from your Trello board and forwards "archived card" events to the matching service, if any.
- Can be run as a scheduled job, or as an HTTP server:
```sh
# 1. CRON JOB
go run ./cmd/runner
```
- Can be run as an HTTP server:
```sh
PORT=<port> USERNAME=<user> PASSWORD=<password> go run ./cmd/server
```

In this case, the runner can be triggered by a `POST` request to the server like this:
```sh

# 2. HTTP SERVER
go run ./cmd/server

# make a `POST` request to the HTTP server to trigger polling
curl -d @config.json <SERVER_URL> -H "Authorization: Basic <base64(<user>:<password>)>"
```

## Configuration
### Example Use Case
Let's say you have an HTTP service that returns GitHub issues that are assigned to you upon a `GET` request.
Then `entrello` can use it as a data source to keep your GitHub issues synchronized in your Trello board.
Moreover, when you use `entrello` as a server, it can forward "archived card" events to your GitHub service.
This means that whenever one of your "GitHub" cards is archived, your GitHub service can be notified and take an action of your choosing, e.g. it could close the corresponding GitHub issue.
## Runner Configuration
Copy and rename `config.example.json` as `config.json` (default), then set your own values in `config.json`.
You can also use a custom config file path using the `-c` flag:
```sh
go run ./cmd/runner -c /path/to/config/file
```
### Trello
You need to set your [Trello API key & token](https://trello.com/app-key) in the configuraiton file, as well as the Trello board ID.

### Data Sources
For each data source, the following parameters have to be specified. (See `config.example.json`)
### Configuring Services in Runner Mode
Each configured service must return a JSON array of Trello card objects upon a `GET` request. See `pkg/trello/trello.go` for reference. For each service, the following configuration parameters have to be specified:
- `name` &mdash; Data source name.
- `name` &mdash; Service name.
- `endpoint` &mdash; Data source endpoint. `entrello` will make a `GET` request to this endpoint to fetch fresh cards from the data source.
- `endpoint` &mdash; Service endpoint. `entrello` will make a `GET` request to this endpoint to fetch fresh cards from the service.
- `strict` &mdash; When strict mode is enabled, previously auto-generated cards that are no longer present in the fresh data will be deleted. For instance, with a GitHub data source, strict mode can be useful for automatically removing previously auto-generated cards for issues/PRs from the board when the corresponding issues/PRs are closed/merged.
- `strict` &mdash; When strict mode is enabled, previously auto-generated cards that are no longer present in the fresh data will be deleted. For instance, with a GitHub service, strict mode can be useful for automatically removing previously auto-generated cards for issues/PRs from the board when the corresponding issues/PRs are closed/merged.
- `label_id` &mdash; **Distinct** Trello label ID associated with the data source.
- `label_id` &mdash; **Distinct** Trello label ID associated with the service.
- `list_id` &mdash; Trello list ID for the data source to determine where to insert new cards. The selected list must be in the same board as configured by the `board_id` parameter.
- `list_id` &mdash; Trello list ID for the service to determine where to insert new cards. The selected list must be in the same board as configured by the `board_id` parameter.
- `period` &mdash; Polling period for the data source. Some examples:
- `period` &mdash; Polling period for the service. Some examples:
```json
// query at 3rd, 6th, 9th, ... of each month
"period": {
Expand All @@ -76,10 +77,44 @@ For each data source, the following parameters have to be specified. (See `confi
}
```
## Example Cron Job
### Example Runner Cron Job
Assuming `config.json` is located in the current working directory:
``` sh
0 * * * * cd /home/you/git/entrello && /usr/local/go/bin/go run ./cmd/runner
```
Make sure that the cron job runs frequently enough to keep up with the most frequent custom interval in your configuration. For instance, it wouldn't make sense to define a custom period of 15 minutes while the cron job only runs every hour.

## Server Configuration
Copy and rename `.env.example` as `.env`, then set your own values in `.env`.

You can trigger the runner by making a `POST` request to the root URL of your server with the runner configuration in the request body:
```sh
curl -d @config.json <SERVER_URL> -H "Authorization: Basic <base64(<USERNAME>:<PASSWORD>)>"
```

You can create a Trello webhook pointed at `<SERVER_URL>/trello-webhook` in order to listen to events from your Trello board.

### Creating Trello Webhooks
You can create a Trello webhook using the following command:

```sh
curl -X POST -H "Content-Type: application/json" \
https://api.trello.com/1/tokens/<api_token>/webhooks/ \
-d '{
"key": "<api_key>",
"callbackURL": "<url>",
"idModel": "<id_model>",
"description": "<desc>"
}'
```

* `api_token` &mdash; Trello API token
* `api_key` &mdash; Trello API key
* `url` &mdash; Entrello Server URL
* `id_model` &mdash; Trello Board ID
* `desc` &mdash; Arbitrary description string

For more information, see
* [Trello Webhooks Guide](https://developer.atlassian.com/cloud/trello/guides/rest-api/webhooks/)
* [Trello Webhooks Reference](https://developer.atlassian.com/cloud/trello/rest/#api-group-Webhooks)
6 changes: 3 additions & 3 deletions cmd/runner/main.go
Original file line number Diff line number Diff line change
Expand Up @@ -6,20 +6,20 @@ import (

"github.com/utkuufuk/entrello/internal/config"
"github.com/utkuufuk/entrello/internal/logger"
"github.com/utkuufuk/entrello/internal/service"
"github.com/utkuufuk/entrello/internal/services"
)

func main() {
var configFile string
flag.StringVar(&configFile, "c", "config.json", "config file path")
flag.Parse()

cfg, err := config.ReadConfig(configFile)
cfg, err := config.ReadRunnerConfig(configFile)
if err != nil {
log.Fatalf("Could not read configuration: %v", err)
}

if err = service.Poll(cfg); err != nil {
if err = services.Poll(cfg); err != nil {
logger.Error(err.Error())
}
}
97 changes: 86 additions & 11 deletions cmd/server/main.go
Original file line number Diff line number Diff line change
Expand Up @@ -5,21 +5,30 @@ import (
"fmt"
"io/ioutil"
"net/http"
"os"

"github.com/utkuufuk/entrello/internal/config"
"github.com/utkuufuk/entrello/internal/logger"
"github.com/utkuufuk/entrello/internal/service"
"github.com/utkuufuk/entrello/internal/services"
"github.com/utkuufuk/entrello/pkg/trello"
)

var client trello.Client

func main() {
http.HandleFunc("/", handler)
http.ListenAndServe(fmt.Sprintf(":%s", os.Getenv("PORT")), nil)
client = trello.NewClient(config.Trello{
ApiKey: config.ServerCfg.TrelloApiKey,
ApiToken: config.ServerCfg.TrelloApiToken,
BoardId: config.ServerCfg.TrelloBoardId,
})

http.HandleFunc("/", handlePollRequest)
http.HandleFunc("/trello-webhook", handleTrelloWebhookRequest)
http.ListenAndServe(fmt.Sprintf(":%s", config.ServerCfg.Port), nil)
}

func handler(w http.ResponseWriter, req *http.Request) {
func handlePollRequest(w http.ResponseWriter, req *http.Request) {
if req.Method != http.MethodPost {
logger.Warn("Method %s not allowed", req.Method)
logger.Warn("Method %s not allowed for %s", req.Method, req.URL.Path)
w.WriteHeader(http.StatusMethodNotAllowed)
return
}
Expand All @@ -30,12 +39,12 @@ func handler(w http.ResponseWriter, req *http.Request) {
w.WriteHeader(http.StatusUnauthorized)
return
}
if user != os.Getenv("USERNAME") {
if user != config.ServerCfg.Username {
logger.Warn("Invalid user name: %s", user)
w.WriteHeader(http.StatusUnauthorized)
return
}
if pwd != os.Getenv("PASSWORD") {
if pwd != config.ServerCfg.Password {
logger.Warn("Invalid password: %s", pwd)
w.WriteHeader(http.StatusUnauthorized)
return
Expand All @@ -44,22 +53,88 @@ func handler(w http.ResponseWriter, req *http.Request) {
body, err := ioutil.ReadAll(req.Body)
if err != nil {
logger.Error("Could not read request body: %v", err)
w.WriteHeader(http.StatusInternalServerError)
w.WriteHeader(http.StatusBadRequest)
return
}

var cfg config.Config
var cfg config.RunnerConfig
if err = json.Unmarshal(body, &cfg); err != nil {
logger.Warn("Invalid request body: %v", err)
w.WriteHeader(http.StatusBadRequest)
return
}

if err = service.Poll(cfg); err != nil {
if err = services.Poll(cfg); err != nil {
logger.Error(err.Error())
w.WriteHeader(http.StatusInternalServerError)
return
}

w.WriteHeader(http.StatusOK)
}

func handleTrelloWebhookRequest(w http.ResponseWriter, req *http.Request) {
if req.Method == http.MethodHead {
w.WriteHeader(http.StatusOK)
return
}

if req.Method != http.MethodPost {
logger.Warn("Method %s not allowed for %s", req.Method, req.URL.Path)
w.WriteHeader(http.StatusMethodNotAllowed)
return
}

hash := req.Header.Get("x-trello-webhook")
if hash == "" {
logger.Warn("Missing 'x-trello-webhook' header")
w.WriteHeader(http.StatusUnauthorized)
return
}

body, err := ioutil.ReadAll(req.Body)
if err != nil {
logger.Error("Could not read Trello webhook request body: %v", err)
w.WriteHeader(http.StatusBadRequest)
return
}

if !trello.VerifyWebhookSignature(
config.ServerCfg.TrelloWebhookCallbackUrl,
config.ServerCfg.TrelloSecret,
hash,
body,
) {
logger.Warn("Invalid Trello webhook signature")
w.WriteHeader(http.StatusUnauthorized)
return
}

var wrb trello.WebhookRequestBody
if err = json.Unmarshal(body, &wrb); err != nil {
logger.Warn("Invalid Trello webhook request body: %v", err)
w.WriteHeader(http.StatusBadRequest)
return
}

archivedCardId := trello.ParseArchivedCardId(wrb)
if archivedCardId == "" {
w.WriteHeader(http.StatusAccepted)
return
}

archivedCard, err := client.GetCard(archivedCardId)
if err != nil {
logger.Error("Could not fetch archived Trello card: %v", err)
w.WriteHeader(http.StatusInternalServerError)
return
}

if err = services.Notify(archivedCard, config.ServerCfg.Services); err != nil {
logger.Error("Could not notify service(s) with the archived card data: %v", err)
w.WriteHeader(http.StatusInternalServerError)
return
}

w.WriteHeader(http.StatusOK)
}
2 changes: 1 addition & 1 deletion config.example.json
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@
"api_token": "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx",
"board_id": "xxxxxxxxxxxxxxxxxxxxxxxx"
},
"sources": [
"services": [
{
"name": "Github Issues",
"endpoint": "http://<github-issues-endpoint>",
Expand Down
10 changes: 6 additions & 4 deletions go.mod
Original file line number Diff line number Diff line change
@@ -1,11 +1,13 @@
module github.com/utkuufuk/entrello

// +heroku goVersion go1.16
go 1.16
// +heroku goVersion go1.18
go 1.18

require (
github.com/adlio/trello v1.8.0
github.com/google/go-cmp v0.5.4
github.com/pkg/errors v0.9.1 // indirect
golang.org/x/xerrors v0.0.0-20200804184101-5ec99f83aff1 // indirect
github.com/joho/godotenv v1.4.0
golang.org/x/exp v0.0.0-20220328175248-053ad81199eb
)

require github.com/pkg/errors v0.9.1 // indirect
5 changes: 4 additions & 1 deletion go.sum
Original file line number Diff line number Diff line change
Expand Up @@ -2,9 +2,12 @@ github.com/adlio/trello v1.8.0 h1:VU/1zwzuRzATsFC8WiK4f8R0HHQPWpf2H658KEchsmA=
github.com/adlio/trello v1.8.0/go.mod h1:l2068AhUuUuQ9Vsb95ECMueHThYyAj4e85lWPmr2/LE=
github.com/google/go-cmp v0.5.4 h1:L8R9j+yAqZuZjsqh/z+F1NCffTKKLShY6zXTItVIZ8M=
github.com/google/go-cmp v0.5.4/go.mod h1:v8dTdLbMG2kIc/vJvl+f65V22dbkXbowE6jgT/gNBxE=
github.com/joho/godotenv v1.4.0 h1:3l4+N6zfMWnkbPEXKng2o2/MR5mSwTrBih4ZEkkz1lg=
github.com/joho/godotenv v1.4.0/go.mod h1:f4LDr5Voq0i2e/R5DDNOoa2zzDfwtkZa6DnEwAbqwq4=
github.com/pkg/errors v0.8.1/go.mod h1:bwawxfHBFNV+L2hUp1rHADufV3IMtnDRdf1r5NINEl0=
github.com/pkg/errors v0.9.1 h1:FEBLx1zS214owpjy7qsBeixbURkuhQAwrK5UwLGTwt4=
github.com/pkg/errors v0.9.1/go.mod h1:bwawxfHBFNV+L2hUp1rHADufV3IMtnDRdf1r5NINEl0=
golang.org/x/exp v0.0.0-20220328175248-053ad81199eb h1:pC9Okm6BVmxEw76PUu0XUbOTQ92JX11hfvqTjAV3qxM=
golang.org/x/exp v0.0.0-20220328175248-053ad81199eb/go.mod h1:lgLbSvA5ygNOMpwM/9anMpWVlVJ7Z+cHWq/eFuinpGE=
golang.org/x/xerrors v0.0.0-20191204190536-9bdfabe68543/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0=
golang.org/x/xerrors v0.0.0-20200804184101-5ec99f83aff1 h1:go1bK/D/BFZV2I8cIQd1NKEZ+0owSTG1fDTci4IqFcE=
golang.org/x/xerrors v0.0.0-20200804184101-5ec99f83aff1/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0=
Loading

0 comments on commit 09cfd8e

Please sign in to comment.