Jshearer/materialize slack #722

jshearer · 2023-05-19T17:30:07Z

Description:

This fleshes out the Slack materialization connector that @willdonnelly originally wrote for one of our hackathons.

It automatically deals with joining channels when needed, and lets you specify a username and profile picture to communicate as. It also supports rich messages via Slack's blocks API

Note:
One thing I don't love is the config schema, which results in the following UI:

By default, Sender Config is collapsed. This comes from

type config struct {
	SenderConfig SlackSenderConfig `json:"sender_config" jsonschema:"title=Slack Config"`
	Credentials  CredentialConfig  `json:"credentials" jsonschema:"title=Authentication" jsonschema_extras:"x-oauth2-provider=slack"`
}

I had to extract SlackSenderConfig because it was used in a few different places. What I really want is the equivalent to Serde's flatten attribute, but I can't figure out a way to get Go to do the same thing. Any ideas?

This change is

materialize-slack/main.go

williamhbaker

Left some comments as an initial review. You will need to add this to CI in the github workflow so it actually gets built - there might be some changes needed to actually get it building (I would suggest rebasing on the last from the main branch).

I'll plan on doing another detailed review after that is done.

williamhbaker · 2023-06-08T01:16:14Z

materialize-slack/Dockerfile

+
+# Build Stage
+################################################################################
+FROM golang:1.17-bullseye as builder


This dockerfile will need updated to be like the current ones on the main branch, which use go 1.20, and the directory structure of the imports ("Build the connector projects we depend on.") has changed. See

connectors/materialize-pinecone/Dockerfile

Lines 1 to 16 in 60728e3

ARG BASE_IMAGE=ghcr.io/estuary/base-image:v1

# Build Stage

################################################################################

FROM --platform=linux/amd64 golang:1.20-bullseye as builder

WORKDIR /builder

# Download & compile dependencies early. Doing this separately allows for layer

# caching opportunities when no dependencies are updated.

COPY go.* ./

RUN go mod download

COPY go ./go

COPY materialize-boilerplate ./materialize-boilerplate

COPY materialize-pinecone ./materialize-pinecone

.

I think I had already updated this. The change you're talking about is moving things from e.g go-materialize-boilerplate to go/materialize-boilerplate right? I did push one change to just include the whole go/ subdir, which seems to be what the other dockerfiles do

materialize-slack/main.go

williamhbaker

Thanks for updating the transactor! I left some more comments. You will also need to update .github/workflows/ci.yaml in order for the connector to be built automatically.

materialize-slack/slack.go

materialize-slack/main.go

williamhbaker · 2023-06-14T21:10:47Z

materialize-slack/main.go

+				if marshal_err == nil {
+					log.Warn(fmt.Sprintf("Parse Blocks: %+v", string(serializedBlocks)))
+				}
+				log.Warn(fmt.Errorf("error sending message: %w", err))


This may be something where we'd want to retry with rate limiting, but it doesn't look like that is implemented here, although I see the partial comment below and the sleep.

Yeah, retries would be nice. Do we have any common way of implementing that with e.g exponential backoff and all that?

You could probably leverage something like this, from the google sheets materialization:

connectors/materialize-google-sheets/util.go

Lines 305 to 327 in 0f94d07

for attempt := 1; true; attempt++ {

var _, err = client.Spreadsheets.BatchUpdate(spreadsheetID,

&sheets.BatchUpdateSpreadsheetRequest{Requests: requests}).

Context(ctx).

Do()

if err == nil {

return nil

}

// If we encounter a rate limit error, retry after exponential back-off.

if e, ok := err.(*googleapi.Error); ok && retryableErrorCodes[e.Code] {

var dur = time.Duration(1<<attempt) * time.Second

log.Println("received error ", e.Code, http.StatusText(e.Code), ", retrying after ", dur)

time.Sleep(dur)

continue

}

// All other errors bail out.

return err

}

panic("not reached")

}

williamhbaker · 2023-06-14T21:14:32Z

materialize-slack/main.go

+		}
+
+		// Accept messages from at most 10 minutes in the past
+		if time.Since(ts).Minutes() < 10 {


I'm not really following this. What happens if a message is outside of this time window? Does it just get dropped? I would expect a connector like this to provide at-least-once materialization, but if it is possibly dropping messages it would not do that. It feels like there is some kind of filtering happening here that may not generally be desirable.

This should probably be tunable, but the idea is that Slack is different from a regular database in that you can't send a message with a timestamp other than right now. So given that, the concept of "backfilling" slack messages doesn't make sense (this would look like sending a message to slack and having it get slotted in somewhere in the conversation's history, which you can't do), and we really do only want to deliver messages that happen in roughly real time. 10 minutes was picked as a reasonable window for "real time" slack messages to show up.

williamhbaker · 2023-06-29T12:48:48Z

materialize-slack/main.go

+		var parsed = buildDocument(b, it.Key, it.Values)
+		var tsStr, tsOk = parsed["ts"].(string)
+		var text, textOk = parsed["text"].(string)
+		var blocks = parsed["blocks"].(json.RawMessage)


I think this is ok if blocks isn't present in the output from buildDocument? nil can be coerved into a json.RawMessage (which is a []byte)...and later string([]byte(nil)) I assume is ""? I'm not 100% sure on this so if you haven't tried it, it might not be a bad idea to test it out.

narrator: it was not okay. fixed it

williamhbaker

Overall LGTM. Left a few minor comments.

My main uncertainty is with the filtering that the materialization is doing. I understand the reasoning and it sounds like it has been hashed out elsewhere already, so my uncertainty is not blocking. Perhaps the ideal solution would be for the kind of notBefore / notAfter type of general feature we have discussed elsewhere, and the behavior here will suffice until we have that.

williamhbaker · 2023-06-29T12:50:54Z

materialize-slack/main.go

+			if err := t.api.PostMessage(b.resource.Channel, text, blocksParsed.BlockSet, b.resource.SenderConfig); err != nil {
+				return nil, fmt.Errorf("error sending message: %w", err)
+			}
+			time.Sleep(time.Second * 10)


I'm not sure about this 10 second sleep? It seems like if there are messages to be posted they should be posted without delay.

This is to avoid slamming Slack and hitting the rate-limiter

materialize-slack/main.go

materialize-slack/slack.go

williamhbaker · 2023-06-29T12:56:29Z

materialize-slack/main.go

+				if marshal_err == nil {
+					log.Warn(fmt.Sprintf("Parse Blocks: %+v", string(serializedBlocks)))
+				}
+				log.Warn(fmt.Errorf("error sending message: %w", err))


You could probably leverage something like this, from the google sheets materialization:

connectors/materialize-google-sheets/util.go

Lines 305 to 327 in 0f94d07

for attempt := 1; true; attempt++ {

var _, err = client.Spreadsheets.BatchUpdate(spreadsheetID,

&sheets.BatchUpdateSpreadsheetRequest{Requests: requests}).

Context(ctx).

Do()

if err == nil {

return nil

}

// If we encounter a rate limit error, retry after exponential back-off.

if e, ok := err.(*googleapi.Error); ok && retryableErrorCodes[e.Code] {

var dur = time.Duration(1<<attempt) * time.Second

log.Println("received error ", e.Code, http.StatusText(e.Code), ", retrying after ", dur)

time.Sleep(dur)

continue

}

// All other errors bail out.

return err

}

panic("not reached")

}

jgraettinger · 2024-01-16T19:53:47Z

👋 @jshearer We need to land this PR, because it's currently live in our UI but isn't getting updates until it does.

williamhbaker · 2024-04-23T19:53:12Z

👀

This actually works pretty well, the main limitation is that the config requires a concrete bearer token. I think it ought to be possible to make it work via OAuth, so that might be a route for future improvement if we want to actually mainline this materialization. The channel ID is part of the binding resource config, and can be either a name or an explicit ID. Channel names will be looked up using the `conversations.list` API method, while if you only use explicit IDs that permission won't be needed. Explicit IDs are specified with a name like `id:C12345` for Slack channel ID `C12345`.

* Escape slack-specific strings * Rate-limit API calls * Fix `blocks` formatting

Also add more resiliency

…mposite keys

jshearer · 2024-10-09T14:51:56Z

Ok, I patched this up and it's now working for the task-failures channel again. Gonna merge

willdonnelly reviewed May 23, 2023

View reviewed changes

materialize-slack/main.go Outdated Show resolved Hide resolved

williamhbaker reviewed Jun 8, 2023

View reviewed changes

jshearer force-pushed the jshearer/materialize-slack branch from 25847d6 to d288940 Compare June 13, 2023 20:29

williamhbaker reviewed Jun 14, 2023

View reviewed changes

jshearer requested a review from williamhbaker June 28, 2023 21:01

williamhbaker reviewed Jun 29, 2023

View reviewed changes

williamhbaker approved these changes Jun 29, 2023

View reviewed changes

willdonnelly and others added 15 commits October 8, 2024 14:39

feature: Flesh out materialize-slack connector

3b580af

refactor: Use an actual Slack SDK instead of making manual calls

c01b28f

* Keep track of last timestamp per binding

69826ff

* Escape slack-specific strings * Rate-limit API calls * Fix `blocks` formatting

refactor: Move username/logo config to a per-binding basis

ca7c6e9

Also add more resiliency

fix: Reccomend all keys otherwise ordering will get messed up with co…

e65ab14

…mposite keys

refactor: Use buildDocument instead of indices

0f27b55

fix: handle slice indexing issue

5c14129

fix: require key fields

7bae65a

fix: first round of PR reviews

c7698c5

fix: match correct transactor interface

7db1f22

remove checkpoints

bcd806b

chore: remove some comments and clean up some unused structs

c29fa6c

remove unneccesary (potentially crashing) logic

fbdf36b

refactor: Move channel joining to Apple

7ee91ec

jshearer force-pushed the jshearer/materialize-slack branch from 54e48da to 6200acf Compare October 8, 2024 18:43

jshearer added 4 commits October 8, 2024 16:44

more PR review

d1c895a

materialize-slack: Add to CI

1610af4

materialize-slack: update Dockerfile and Go dependencies

d99afd7

materialize-slack: Update to satisfy new interface requirements

cc07176

jshearer force-pushed the jshearer/materialize-slack branch from a9b6a38 to cc07176 Compare October 8, 2024 20:44

materialize-slack: handle null blocks

6843376

jshearer merged commit ec7f2c3 into main Oct 9, 2024
50 of 53 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Jshearer/materialize slack #722

Jshearer/materialize slack #722

jshearer commented May 19, 2023 •

edited

Loading

williamhbaker left a comment

williamhbaker Jun 8, 2023

jshearer Jun 13, 2023 •

edited

Loading

williamhbaker left a comment

williamhbaker Jun 14, 2023

jshearer Jun 28, 2023

williamhbaker Jun 29, 2023

williamhbaker Jun 14, 2023

jshearer Jun 28, 2023 •

edited

Loading

williamhbaker Jun 29, 2023 •

edited

Loading

jshearer Oct 9, 2024

williamhbaker left a comment

williamhbaker Jun 29, 2023

jshearer Oct 9, 2024

williamhbaker Jun 29, 2023

jgraettinger commented Jan 16, 2024

williamhbaker commented Apr 23, 2024

jshearer commented Oct 9, 2024

	ARG BASE_IMAGE=ghcr.io/estuary/base-image:v1

	# Build Stage
	################################################################################
	FROM --platform=linux/amd64 golang:1.20-bullseye as builder

	WORKDIR /builder

	# Download & compile dependencies early. Doing this separately allows for layer
	# caching opportunities when no dependencies are updated.
	COPY go.* ./
	RUN go mod download

	COPY go ./go
	COPY materialize-boilerplate ./materialize-boilerplate
	COPY materialize-pinecone ./materialize-pinecone

	for attempt := 1; true; attempt++ {
	var _, err = client.Spreadsheets.BatchUpdate(spreadsheetID,
	&sheets.BatchUpdateSpreadsheetRequest{Requests: requests}).
	Context(ctx).
	Do()

	if err == nil {
	return nil
	}

	// If we encounter a rate limit error, retry after exponential back-off.
	if e, ok := err.(*googleapi.Error); ok && retryableErrorCodes[e.Code] {
	var dur = time.Duration(1<<attempt) * time.Second
	log.Println("received error ", e.Code, http.StatusText(e.Code), ", retrying after ", dur)
	time.Sleep(dur)
	continue
	}

	// All other errors bail out.
	return err
	}
	panic("not reached")
	}

Jshearer/materialize slack #722

Jshearer/materialize slack #722

Conversation

jshearer commented May 19, 2023 • edited Loading

williamhbaker left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jshearer Jun 13, 2023 • edited Loading

Choose a reason for hiding this comment

williamhbaker left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jshearer Jun 28, 2023 • edited Loading

Choose a reason for hiding this comment

williamhbaker Jun 29, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

williamhbaker left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jgraettinger commented Jan 16, 2024

williamhbaker commented Apr 23, 2024

jshearer commented Oct 9, 2024

jshearer commented May 19, 2023 •

edited

Loading

jshearer Jun 13, 2023 •

edited

Loading

jshearer Jun 28, 2023 •

edited

Loading

williamhbaker Jun 29, 2023 •

edited

Loading