Ent Periodic Jobs

Sidekiq Enterprise supports periodic jobs, aka cron or recurring jobs. You register your periodic jobs with a schedule upon startup and Sidekiq will fire off corresponding jobs on that schedule. Sidekiq will ensure that only a single process creates jobs so if you are running 1 or 100 processes, you don't need to worry about duplicate job creation. Note: periodic jobs are global to the entire Sidekiq cluster, you cannot have different cron config per machine

See periodic jobs in action here:

Definition

Periodic Jobs should be defined in your Sidekiq initializer, like so:

Sidekiq.configure_server do |config|
  config.periodic do |mgr|
    # see any crontab reference for the first argument
    # e.g. http://www.adminschoice.com/crontab-quick-reference
    # or   https://crontab.guru/ 
    mgr.register('0 * * * *', "SomeHourlyWorkerClass")
    mgr.register('* * * * *', "SomeWorkerClass", retry: 2, queue: 'foo')
    mgr.register(cron_expression, worker_class.to_s, job_options={})
  end
end

For example, with the schedule above Sidekiq will create a SomeWorkerClass job every minute in the foo queue to be processed like any normal job. If the job raises an error, Sidekiq will retry it like any normal job. Periodic job options will override any options set with sidekiq_options within the Worker class.

Time Zones

The cron expressions above implicitly use the Ruby process's Time Zone, which defaults to the system time zone. You can set the TZ manually with something like TZ=America/Los_Angeles bundle exec .... As of Sidekiq Enterprise 2.2.1, you can configure a specific timezone:

Sidekiq.configure_server do |config|
  config.periodic do |mgr|
    # default, local timezone
    mgr.register("* 4 * * *", "LocalWorker") # 4AM system time

    # per-job custom timezome
    # NB: tz is an option but will not be included in the job's payload
    mgr.register("* 4 * * *", "TokyoWorker", tz: ActiveSupport::TimeZone.new("Tokyo"))

    # "global" custom timezome, will affect any further registered jobs
    mgr.tz = ActiveSupport::TimeZone.new("Kathmandu")
    mgr.register("* 4 * * *", "KathmanduWorker") # 4AM in Kathmandu
    mgr.register("* 5 * * *", "KathmanduWorker") # 5AM in Kathmandu

With many timezones, jobs become very difficult to mentally track. Use sparingly.

Dynamic Jobs

Sidekiq Enterprise does not support end-user-managed cron jobs out of the box. Sidekiq never touches your database and anything user-managed should be controlled by your database. Your application should have a user interface to manage dynamic jobs like any other Rails resource, allowing the user to create, list, delete, etc their jobs as normal.

You'd then have a Periodic Job running every minute which looks for database records representing jobs which need to run now. The static job would enqueue each dynamic job and update the database record with the next timestamp.

create_table :dynamic_jobs do |t|
  # these are totally optional, might want to track them for multi-tenancy or security purposes
  t.references :account_id, null: false, index: true
  t.references :user_id, null: false, index: true

  t.string :klass, null: false
  t.string :cron_expression, null: false
  t.timestamp :next_run_at, null: false, index: true
end

class DynamicJobWorker
  include Sidekiq::Job
  sidekiq_options retry: false

  def perform
    DynamicJob.where("next_run_at <= ?", Time.now).find_each do |job|
      # Multi-tenant apps can use a server-side middleware to set DB connection based on account_id and/or user_id.
      Sidekiq::Client.push('class' => job.klass.constantize, 'args' => [],
                           'account_id' => job.account_id, 'user_id' => job.user_id)
      x = Sidekiq::CronParser.new(job.cron_expression)
      job.update_attribute(:next_run_at, x.next.to_time)
    end
  end
end

Sidekiq.configure_server do |config|
  config.periodic do |mgr|
    mgr.register("* * * * *", "DynamicJobWorker")
  end
end

Extra credit: support a job argument by adding it to the model and passing it in the push method.

Limitations

The most frequently a job can be scheduled with the crontab format is every minute. If you want a job to run every 15 seconds, you'll need to build it yourself (perhaps creating a custom thread plus the Leader Election feature) or use a meta-job to schedule jobs to run in 0,15,30,45 seconds. Note that Sidekiq's scheduler is not very precise out of the box, you can adjust the precision as explained here.

This implementation does not support backfill. If Sidekiq is shutdown, it will not create jobs for the times missed on restart. I advise people to make their cron jobs resilient to timing. Instead of having an hourly job which processes only the last hour of data, make it process the last N hours. 99.9% of the time, it will only need to process one hour's worth of data anyways but if a job is missed, the next hour's job will process it.

API

The Periodic API allows you to list registered periodic jobs and see enqueue history:

loops = Sidekiq::Periodic::LoopSet.new
loops.each do |lop|
  p [lop.schedule, lop.klass, lop.lid, lop.options, lop.history]
end

Testing

"What happens if we misspell a class name in the register call or use invalid crontab syntax? How do we test it?"

Extract the registration block and call it in an integration test:

# config/initializers/sidekiq.rb
Sidekiq.configure_server do |config|
  require 'periodic_jobs'
  config.periodic(&PERIODIC_JOBS)
end

# lib/periodic_jobs.rb
PERIODIC_JOBS = ->(mgr) {
  mgr.register "* * * * *", ScheduledWorker, retry: 1
  mgr.register "*/4 * 10 * *", "ScheduledWorker", retry: 2
  mgr.register "*/4 * 10 * *", "Missing", retry: 2
}

# test/integration/periodic_jobs_test.rb
require 'test_helper'

class PeriodicJobsTest < ActionDispatch::IntegrationTest
  test "periodic jobs exist" do
    require 'periodic_jobs'
    require "sidekiq-ent/periodic/testing"
    ct = Sidekiq::Periodic::ConfigTester.new
    ct.verify(&PERIODIC_JOBS)
  end
end

The result:

NameError: uninitialized constant Missing

Web UI

The Web UI exposes a "Cron" tab with any registered jobs. You can see an overview of all registered jobs and see job execution history. Make sure you require 'sidekiq-ent/web' in config/routes.rb.

Notes

• The leader process enqueues cron jobs. A quiet leader still enqueues cron jobs; it only stops when shutdown is signaled.
• As of 7.1.0, Cron jobs may use ActiveJob.
• The perform method for periodic workers should take NO parameters, have default values for all parameters, or you can register a static set of arguments like this:

mgr.register('0 * * * *', "PostalEventBatchJob", 'args' => ["zero"])
mgr.register('1 * * * *', "PostalEventBatchJob", 'args' => ["one"])

Args must be static since they are evaluated at boot time.

• If you need a job to run in a local timezone, you can use a tool like this to convert the cron expression into UTC.
• DEPRECATION Before Sidekiq Enterprise v2.0.1, you could register a Class. Starting in Rails 6.0, initializers are not allowed to load application classes so the API was adjusted to pass class name as a String instead.

Before:  mgr.register('0 * * * *', SomeHourlyWorkerClass)
After:   mgr.register('0 * * * *', "SomeHourlyWorkerClass")