We currently need to solve a couple of problems in the mailservice:
- We need to react to account deletions (ie terminating ongoing processes, and preventing retries of failed processes)
- We need to increase processing capacity as the number of accounts being handled increases
- To reduce the probability of k8s scheduling conflicts, we need to minimize the resources requested by mailservice pods
The current approach to event notification is to write SQS events to signal state changes, for instance when a new account is created, the process creating the account writes an event to one of 64 "created" queues, which is consumed by the sync process. We could create a partitioned set of "deleted" queues and write deletion events into them. This approach feels brittle, as it relies on SQS events being created anywhere accounts are modified, and we're already doing this in at least two places (the API and the CIO kafka topic watcher).