DynamoDB: DuplicateKeyException after resuming CDC

[amotl]

About

We are observing a problem with the sync lambda.

Problem

Every once in a while a task will time out, which leads to the sync lambda retrying. Now, these retries fail, because, even though the original task had timed out, the data has been correctly stored into CrateDB, for all the items that are part of the batch that has timed out. Do you have any idea what could be causing this behavior?

/cc @dfeokti

[amotl]

a) We will look into the resume logic if we can spot any bugs.
b) Other strategies to compensate for those situations are using upserts / on conflict ignore clauses on the insert statements. We might just employ this strategy here as well.

/cc @wierdvanderhaar, @hammerhead

[amotl]

a) Relating to that comment,

cratedb-toolkit/cratedb_toolkit/io/processor/kinesis_lambda.py

Lines 106 to 110 in db67cd1

	"""
	Implement partial batch response for Lambda functions that receive events from
	a Kinesis stream. The function reports the batch item failures in the response,
	signaling to Lambda to retry those messages later.
	"""

and how error handling is taking place,

cratedb-toolkit/cratedb_toolkit/io/processor/kinesis_lambda.py

Lines 142 to 154 in db67cd1

	except Exception as ex:
	error_message = f"An error occurred processing event: {event_id}"
	logger.exception(error_message)
	if USE_BATCH_PROCESSING:
	# Return failed record's sequence number.
	return {"batchItemFailures": [{"itemIdentifier": cur_record_sequence_number}]}
	if ON_ERROR == "exit":
	# Signal "Input/output error" when error happens while processing data.
	sys.exit(5)
	elif ON_ERROR == "ignore":
	pass
	elif ON_ERROR == "raise":
	raise ex

I guess the regular modus operandi for a Lambda that receives events from a Kinesis stream is that if the Lambda fails for whatever reason, recent events will be re-delivered. If it's multiple records, it is probably normal that some of them may be redundant, because they have been relayed to CrateDB successfully already.

b) I guess using ON CONFLICT IGNORE/UPDATE instead will be the right choice.

[amotl]

That's it, @lservini?

• DynamoDB: Use ON CONFLICT DO NOTHING clause on INSERT CDC operations commons-codec#77