We built sched.do for Yammer as an example open-sourced Rails application using the Yammer API. Like many services, Yammer API calls are subject to rate-limitations.
At times, the application would hit the Yammer API rate limit and encounter a 429 status code. To solve this, we leveraged delayed_job
's functionality of retrying failed jobs. Our goal was to set the retry time to just outside the rate-limit time window.
To accomplish this, we overrode the reschedule_at
and max_attempts
methods, which delayed_job
exposes for custom jobs. We also wanted to retry the job until success, so max_attempts
was also overridden.
Code example:
module YammerRateLimited
RATE_LIMIT_RETRIES = 50
def error(job, exception)
if ExceptionSilencer.is_rate_limit?(exception)
@rate_limited = true
else
@rate_limited = false
Airbrake.notify(exception)
end
end
def failure(job)
Airbrake.notify(error_message: "Job failure: #{job.last_error}")
end
def reschedule_at(attempts, time)
if @rate_limited
30.seconds.from_now
end
end
def max_attempts
if @rate_limited
RATE_LIMIT_RETRIES
else
Delayed::Worker.max_attempts
end
end
end
Then we include this module into our custom job, like so:
class ReminderCreatedJob < Struct.new(:reminder_id)
include YammerRateLimited
def self.enqueue(reminder)
job = new(reminder.id)
Delayed::Job.enqueue(job)
end
def perform
Reminder.find(reminder_id).deliver
end
end
When a job fails due to rate-limitations, we retry it every thirty seconds, fifty times in a row until it works. The job can still fail after 50 swings, at which point we will get notified via Airbrake to take a look manually.