Migrating from Sidekiq & Redis to Rails 8 Solid Queue: Real-World Production Teardown

Direct Answer: Migrating from Sidekiq and Redis to Rails 8 Solid Queue eliminates external Redis dependencies and cron daemons by leveraging your existing relational database. By moving to database-backed job processing, you simplify production infrastructure, cut memory overhead by up to 60%, and streamline deployments using native Rails primitives.

For over a decade, the default architecture for high-throughput Ruby on Rails applications relied on a tripartite infrastructure stack: Rails for application processing, Sidekiq for background workers, and Redis as the in-memory broker. While performant, this setup introduces operational overhead. You must provision, monitor, and scale an entirely separate data store (Redis), manage separate memory limits, and handle complex failure modes when Redis evicts keys under memory pressure or network partitions occur.

Rails 8 changes the game by introducing native database-backed components: Solid Queue, Solid Cache, and Solid Cable. By capitalizing on modern relational database performance, connection pooling, and multi-statement transactions, Solid Queue allows engineering teams to drop Redis entirely for queue management and eliminate external cron daemons (like whenever or systemd timers) by using its built-in recurring tasks engine.

In this technical teardown, we will walk through a zero-downtime production migration from Sidekiq/Redis to Rails 8 Solid Queue, covering database partitioning, configuration adjustments, deployment topology using Kamal 2, and real-world performance tuning.

Architectural Paradigm Shift: Why Migrate Away from Redis?

In legacy stacks, background jobs are pushed to Redis lists or sorted sets. Sidekiq pollers fetch these jobs using blocking Redis commands (BRPOPLPUSH). While incredibly fast for in-memory operations, this design introduces several production pain points:

• Dual Data Stores: Your application data lives in PostgreSQL or MySQL, but your job state lives in Redis. Ensuring data consistency between relational records and job arguments often requires complex database transaction outbox patterns.
• Memory Costs: Redis must keep your entire active job payload, retry sets, and scheduled queues in RAM. For high-volume systems handling millions of jobs daily, Redis memory instances scale quickly and become expensive.
• Infrastructure Complexity: Managing Redis replication, persistence (RDB/AOF), and failover (Redis Sentinel or Cluster) adds unnecessary cognitive load to your DevOps pipeline.

Solid Queue replaces Redis with your primary database (or a dedicated secondary database instance). It uses database row locking, SKIP LOCKED patterns, and optimized indexing to process millions of jobs efficiently without requiring external caching infrastructure.

Step 1: Gemfile Restructuring and Dependency Cleanup

To begin the migration, you must remove Sidekiq and Redis-dependent gems from your application and introduce the Solid Queue engine. Open your Gemfile and update your background processing dependencies:

# Remove these gems
# gem "sidekiq"
# gem "redis"

# Add Solid Queue (comes bundled with Rails 8, but ensure explicit requirement if needed)
gem "solid_queue"

Run bundle install to update your Gemfile.lock. Next, remove any Sidekiq-specific initializer files (e.g., config/initializers/sidekiq.rb) and redis configuration files.

Step 2: Database Configuration and Multi-Database Setup for Solid Queue

While Solid Queue can share your primary database, high-throughput production environments should isolate background job execution to a dedicated database cluster or a separate database within the same PostgreSQL instance to prevent job polling queries from starving user-facing HTTP requests.

Configure your config/database.yml to include a dedicated solid_queue database:

production:
  primary: &primary_production
    <<: *default
    database: techvinta_production
    username: techvinta
    password: <%= ENV["DATABASE_PASSWORD"] %>
    pool: <%= ENV.fetch("RAILS_MAX_THREADS") { 10 } %>

  solid_queue:
    <<: *default
    database: techvinta_production_solid_queue
    username: techvinta
    password: <%= ENV["DATABASE_PASSWORD"] %>
    pool: <%= ENV.fetch("SOLID_QUEUE_MAX_THREADS") { 25 } %>
    migrations_paths: db/solid_queue_migrate

Next, install the Solid Queue migrations into your isolated migration path:

bin/rails solid_queue:install:migrations

Run the migrations against your target database:

bin/rails db:migrate:solid_queue

Step 3: Configuring Solid Queue and Replacing Cron Daemons

Solid Queue configuration lives in config/solid_queue.yml. This file defines dispatchers, workers, queues, and concurrency limits. It also replaces traditional cron daemons by handling recurring tasks natively.

production:
  dispatchers:
    - polling_interval: 1
      batch_size: 500
      concurrency_limit: 5
  workers:
    - queues: "*"
      threads: 5
      processes: 4
      polling_interval: 0.1
    - queues: [mailers, high_priority]
      threads: 10
      processes: 2
      polling_interval: 0.05
  recurring:
    cleanup_audit_logs:
      class: CleanupAuditLogsJob
      cron: "0 2 * * *"
      queue: maintenance
    sync_analytics:
      class: SyncAnalyticsJob
      every: 15.minutes
      queue: default

Update your config/environments/production.rb to configure Active Job to use Solid Queue:

Rails.application.configure do
  # Switch Active Job adapter from :sidekiq to :solid_queue
  config.active_job.queue_adapter = :solid_queue
  config.solid_queue.connects_to = { database: { writing: :solid_queue, reading: :solid_queue } }
end

Step 4: Adapting Job Classes and Retries

If you were using Sidekiq-specific DSLs (like include Sidekiq::Job or sidekiq_options retry: 5), you must refactor your jobs to use standard Rails Active Job APIs. Solid Queue relies fully on Active Job abstractions.

class ProcessStripeWebhookJob < ApplicationJob
  queue_as :high_priority
  
  # Configure retries natively through Active Job
  retry_on Stripe::NetworkError, wait: :exponentially_longer, attempts: 10
  discard_on Stripe::InvalidRequestError

  def perform(webhook_event_id)
    event = StripeEvent.find(webhook_event_id)
    StripeWebhookProcessor.new(event).call
  end
end

Step 5: Production Deployment Topology with Kamal 2 & Docker

When deploying with Kamal 2, you no longer need to manage separate systemd service files for Sidekiq. Instead, you configure your Kamal config/deploy.yml to run the Solid Queue supervisor alongside your web servers, or as dedicated worker containers.

# config/deploy.yml snippet
service: techvinta-core
image: registry.digitalocean.com/techvinta/core

servers:
  web:
    hosts:
      - 192.168.1.10
      - 192.168.1.11
  job:
    hosts:
      - 192.168.1.12
    cmd: bundle exec bin/jobs

The bin/jobs executable is automatically generated by Rails 8 and boots the Solid Queue supervisor, managing dispatchers and workers within the container lifecycle cleanly.

Production Teardown: 2026 Architecture, Cost & Timeline Comparison

At TechVinta, our senior engineering teams specialize in high-stakes Rails migrations, performance optimization, and scalable cloud architectures. We maintain a reliable 4-6 hour US timezone overlap for seamless collaboration, code reviews, and zero-downtime cutovers.

Frequently Asked Questions

Will Solid Queue slow down my primary database under heavy background job polling?

No, provided you configure database connection pooling properly and place Solid Queue on a dedicated database or schema. Solid Queue uses optimized SQL queries with SKIP LOCKED clauses, which prevent row-locking contention and allow concurrent workers to poll efficiently without blocking user web traffic.

How do I handle recurring tasks without a cron daemon like Whenever?

Solid Queue includes a built-in recurring task dispatcher defined directly in your config/solid_queue.yml file. The supervisor process reads this configuration and enqueues tasks automatically based on cron expressions or fixed time intervals, completely eliminating external system cron dependencies.

Can Solid Queue handle millions of jobs per day as effectively as Sidekiq?

Yes. While Redis is strictly in-memory, modern PostgreSQL and MySQL instances handle high-volume write and update workloads exceptionally well when indexed correctly. For extreme scale (tens of millions of daily jobs), partitioning the solid_queue_jobs table by date or status ensures B-Tree index sizes remain small and execution speeds stay lightning fast.