DataDrip

DataDrip is a Rails engine that provides a robust framework for running data backfills and migrations in your Rails application. It offers a web interface to monitor, schedule and manage batch processing jobs with built-in error handling and progress tracking.

Features

• 🔄 Batch Processing: Process large datasets in configurable batches to avoid memory issues
• 📊 Web Interface: Monitor backfill progress with a built-in dashboard
• ⏰ Scheduled Execution: Schedule backfills to run at specific times
• 🛡️ Error Handling: Comprehensive error tracking and recovery
• 🔧 Flexible Processing: Choose between batch-level or element-level processing
• 📈 Progress Tracking: Real-time progress updates and batch monitoring
• 🎯 Scoped Processing: Define custom scopes for targeted data processing

Installation

Add DataDrip to your application's Gemfile:

gem 'data_drip'

Then execute:

bundle install

Run the installation generator:

rails generate data_drip:install

This will:

• Mount the DataDrip engine at /data_drip in your routes
• Create the app/backfills directory
• Generate and run migrations for data_drip_backfill_runs and data_drip_backfill_run_batches tables

Upgrading

If you installed DataDrip before the backfiller_name column existed, add it with:

rails generate data_drip:add_backfiller_name

This snapshots the backfiller's display name onto each run so it is preserved even if the backfiller record is later deleted.

To add the performance indexes to an older install (composite indexes on the runs list and per-run batch queries):

rails generate data_drip:add_performance_indexes

Requirements

• Ruby: >= 3.1.0
• Rails: >= 7.1
• Dependencies:
  • importmap-rails >= 1.2.1
  • stimulus-rails >= 1.0
  • turbo-rails

Configuration

DataDrip can be configured to work with your application's authentication and authorization system. While the gem works out of the box with standard Rails conventions, you may need to customize these settings to match your application's architecture.

Setting Up the Initializer (Optional)

Create a DataDrip initializer file to customize the gem's behavior:

# config/initializers/data_drip.rb

Rails.application.configure do
  # The model class that represents users who can run backfills
  DataDrip.backfiller_class = 'Access'  # default: "::User"

  # The attribute on the backfiller model to display the user's name
  DataDrip.backfiller_name_attribute = 'first_name'  # default: :name

  # The controller method that returns the current authenticated user
  DataDrip.current_backfiller_method = :current_access  # default: :current_user

  # The base controller class that DataDrip controllers should inherit from
  DataDrip.base_controller_class = 'ApiController'  # default: "::ApplicationController"

  # The base job class that DataDrip jobs should inherit from
  DataDrip.base_job_class = '::ApplicationJob'  # default: "ActiveJob::Base"

  # The Active Job queue for the parent Dripper job (default: ENV["DATA_DRIP_QUEUE"] or :data_drip)
  DataDrip.queue_name = :backfills

  # The Active Job queue for the child DripperChild job (default: ENV["DATA_DRIP_CHILD_QUEUE"] or :data_drip_child)
  DataDrip.child_queue_name = :backfills_child

  # The class that handles lifecycle hooks for backfill runs and batches
  DataDrip.hooks_handler_class_name = 'HookHandler'  # default: nil

  # Optional: Sleep time between batches in seconds (default: 5)
  DataDrip.sleep_time = 0.5
end

Configuration Options Explained

• backfiller_class: The ActiveRecord model that represents the user running backfills. This model will be referenced in the data_drip_backfill_runs table to track who initiated each backfill.

• backfiller_name_attribute: The attribute on your backfiller model that contains the user's display name. This is used in the web interface to show who is running or has run backfills.

• current_backfiller_method: The method name that your base controller uses to get the currently authenticated user. DataDrip controllers will call this method to determine who is creating backfills.

• base_controller_class: The controller class that DataDrip's controllers should inherit from. This is crucial for ensuring that DataDrip controllers have access to your application's authentication, authorization, and other controller concerns.

• base_job_class: The base job class that DataDrip's job classes (Dripper and DripperChild) should inherit from. This allows you to use a custom job class instead of the default ActiveJob::Base. Useful if your application has a custom job base class with additional functionality or configuration. The class must be a subclass of ActiveJob::Base.

• queue_name: The Active Job queue for the parent Dripper job. Defaults to ENV["DATA_DRIP_QUEUE"] if set, otherwise :data_drip. Resolved dynamically at enqueue time, so changes take effect immediately without restarting workers. Can also be configured via the DATA_DRIP_QUEUE environment variable.

• child_queue_name: The Active Job queue for the DripperChild batch job. Defaults to ENV["DATA_DRIP_CHILD_QUEUE"] if set, otherwise :data_drip_child. This is separate from queue_name so you can route parent and child jobs to different queues for priority or resource management.

• hooks_handler_class_name: The name of the class that handles lifecycle hooks for backfill runs and batches. When configured, this class will receive callbacks when backfills change status (e.g., on_run_completed, on_batch_failed). This is useful for sending notifications, tracking metrics, or integrating with external systems. See the Hooks section for more details.

This configuration is particularly useful when your application uses custom authentication systems, non-standard naming conventions, or when you need DataDrip to integrate with existing API controllers or admin interfaces.

Hooks

DataDrip provides a powerful hooks system that allows you to respond to lifecycle events during backfill execution. Hooks are triggered when backfill runs or batches change status, enabling you to integrate with external systems, send notifications, track metrics, or perform any custom logic.

Setting Up a Global Hook Handler

The recommended approach is to create a global hook handler class that responds to lifecycle events across all backfills.

First, configure the hook handler in your initializer:

# config/initializers/data_drip.rb
DataDrip.hooks_handler_class_name = 'HookHandler'

Then create your hook handler class:

# app/services/hook_handler.rb
class HookHandler
  # Run hooks - triggered when a BackfillRun changes status
  def self.on_run_pending(run)
    # Called when a run is created
  end

  def self.on_run_enqueued(run)
    # Called when a run is enqueued for execution
    # Example: Track metrics
    Metrics.increment('backfill.enqueued', tags: { backfill: run.backfill_class_name })
  end

  def self.on_run_running(run)
    # Called when a run starts processing
  end

  def self.on_run_completed(run)
    # Called when a run completes successfully
    # Example: Send notification
    SlackNotifier.notify("Backfill #{run.backfill_class_name} completed!")
  end

  def self.on_run_failed(run)
    # Called when a run fails
    # Example: Send error alert
    ErrorTracker.notify("Backfill failed: #{run.error_message}")
  end

  def self.on_run_stopped(run)
    # Called when a run is manually stopped
  end

  # Batch hooks - triggered when a BackfillRunBatch changes status
  def self.on_batch_pending(batch)
    # Called when a batch is created
  end

  def self.on_batch_enqueued(batch)
    # Called when a batch is enqueued
  end

  def self.on_batch_running(batch)
    # Called when a batch starts processing
  end

  def self.on_batch_completed(batch)
    # Called when a batch completes
    # Example: Update progress tracking
    ProgressTracker.update(batch.backfill_run_id, batch.id)
  end

  def self.on_batch_failed(batch)
    # Called when a batch fails
  end

  def self.on_batch_stopped(batch)
    # Called when a batch is stopped
  end
end

Note: You can implement any logic inside these hooks, such as sending Slack messages, tracking metrics in DataDog, updating external systems, or logging to custom services.

Available Hooks

DataDrip provides hooks for both backfill runs (entire backfill execution) and batches (individual batch processing):

Run Hooks

These hooks receive a BackfillRun object as a parameter:

Hook	Triggered When
on_run_pending	Run is created
on_run_enqueued	Run is enqueued for execution
on_run_running	Run starts processing batches
on_run_completed	All batches complete successfully
on_run_failed	Run encounters an error
on_run_stopped	Run is manually stopped

Batch Hooks

These hooks receive a BackfillRunBatch object as a parameter:

Hook	Triggered When
on_batch_pending	Batch is created
on_batch_enqueued	Batch is enqueued for processing
on_batch_running	Batch starts processing records
on_batch_completed	Batch completes successfully
on_batch_failed	Batch encounters an error
on_batch_stopped	Batch is stopped

Per-Backfill Hooks

You can also define hooks directly in your backfill classes for backfill-specific behavior. These hooks take precedence over the global handler:

class SendWelcomeEmails < DataDrip::Backfill
  def scope
    User.where(welcome_email_sent: false)
  end

  def process_batch(batch)
    batch.each { |user| WelcomeMailer.send_welcome(user).deliver_later }
    batch.update_all(welcome_email_sent: true)
  end

  # Backfill-specific hook
  def self.on_run_completed(run)
    AdminMailer.backfill_summary(run).deliver_now
  end

  def self.on_batch_completed(batch)
    # Track progress for this specific backfill
    Rails.logger.info("Sent #{batch.batch_size} welcome emails")
  end
end

Hook Precedence

When a status change occurs, DataDrip checks for hooks in the following order:

1. Backfill class hooks - If the backfill class defines the hook method, it will be called
2. Global handler hooks - If the backfill class doesn't define the hook and a global handler is configured, the handler's method will be called

This allows you to provide default behavior in the global handler while still allowing individual backfills to override specific hooks when needed.

Creating Backfills

Generate a new backfill:

rails generate data_drip:backfill FixUserEmails

You can add the --sorbet flag to implement sorbet in your backfills.

This creates a backfill class in app/backfills/fix_user_emails.rb:

class FixUserEmails < DataDrip::Backfill
  # Define the scope of records to process
  def scope
    User.where(email_verified: false)
  end

  # Option 1: Process entire batches (recommended for simple updates)
  def process_batch(batch)
    batch.update_all(email_verified: true)
  end

  # Option 2: Process individual elements (for complex logic)
  def process_element(element)
    element.update!(email_verified: true)
  end
end

Adding Options to Backfills

You can make your backfills configurable by adding attributes that users can set when creating a backfill run. This allows for dynamic filtering and lets you customize your backfill runs:

class AddRoleToEmployee < DataDrip::Backfill
  # Define configurable attributes
  attribute :age, :integer
  attribute :name, :string
  attribute :department, :string, required: true
  attribute :active_only, :boolean, default: true

  def scope
    scope = Employee.where(role: nil)

    # Now you can use the attributes to filter the scope dynamically
    scope = scope.where(age: age) if age.present?
    scope = scope.where(name: name) if name.present?
    scope = scope.where(department: department) if department.present?
    scope = scope.where(active: true) if active_only

    scope
  end

  def process_batch(batch)
    batch.update_all(role: 'intern')
  end
end

When creating a backfill run through the form in the UI, you will see form fields for each attribute, allowing you to customize the backfill's behavior without modifying code. This way, it's easy to create a backfill run for different scenarios with different scopes.

Required Options

By default every option is optional. Pass required: true to make an option mandatory:

attribute :department, :string, required: true

A run cannot be created while a required option is blank — the form marks the field as required (both visually and with the native required attribute), and the server rejects the run with a validation error, so runs created programmatically are covered too. For :boolean attributes, false counts as answered; only a missing value is rejected.

Supported Attribute Types

DataDrip supports various attribute types that automatically generate appropriate form fields:

• :string - Text input field
• :integer - Number input (whole numbers)
• :decimal / :float - Number input with decimal support
• :boolean - Checkbox
• :date - Date picker
• :time - Time picker
• :datetime - Date and time picker

Backfill Structure

Every backfill must inherit from DataDrip::Backfill and implement:

1. scope - Returns an ActiveRecord relation of records to process
2. Either process_batch OR process_element (not both):
   • process_batch(batch) - Processes entire batches at once (more efficient)
   • process_element(element) - Processes individual records (more flexible)

Examples

Simple Update (Batch Processing)

class AddDefaultRoleToUsers < DataDrip::Backfill
  def scope
    User.where(role: nil)
  end

  def process_batch(batch)
    batch.update_all(role: 'member')
  end
end

Using DataDrip

Web Interface

Navigate to /data_drip/backfill_runs in your application to access the DataDrip dashboard where you can:

• View all available backfills
• Create new backfill runs with configurable options
• Monitor progress live (progress bars, throughput, estimated time remaining)
• Filter runs by class name and status
• Inspect per-batch errors and retry failed batches
• Stop running backfills
• Schedule backfills for future execution

When creating a new backfill run, the interface dynamically generates form fields based on the attributes defined in your backfill class, making it easy to customize each run without code changes.

The interface supports light and dark mode (following the OS preference) and ships as precompiled CSS inside the gem — host applications need no Node, Tailwind, or any other frontend tooling.

The compiled stylesheet is served directly by the engine (GET /data_drip/tailwind.css, handled by DataDrip::AssetsController) and is deliberately not routed through the host's asset pipeline. Tailwind v4 output uses modern CSS (cascade layers, oklch(), media query range syntax) that a libsass/SassC-based Sprockets pipeline cannot parse — and such hosts recompress every CSS asset through SassC — so keeping the CSS out of the pipeline makes DataDrip work identically under Sprockets, Propshaft, or no asset pipeline at all. No config.assets.precompile entry is required on the host.

Development

The UI is styled with Tailwind CSS v4, compiled at development time and checked in as app/assets/stylesheets/data_drip/tailwind.css:

bin/tailwind                     # watch mode while working on views
bundle exec rake data_drip:css   # one-shot build

Both commands install the pinned toolchain from package.json on first use (Node is only needed to develop the gem itself). CI fails if the compiled CSS is out of date with the views.

To play with the UI locally, boot the dummy app and visit http://localhost:3000/data_drip/backfill_runs:

bin/dev

Scripts

Besides backfills, DataDrip can run scripts: arbitrary one-shot pieces of code with typed inputs, triggered from the web UI. Unlike backfills there is no collection to iterate — a script runs once, in a background job, and everything it logs is stored and shown live in the UI.

Every script run is recorded permanently in the data_drip_script_runs table: who triggered it, the inputs given, the full log output, any error (message and backtrace), and created/started/finished timestamps.

Installing

Fresh installs of DataDrip get everything through rails generate data_drip:install. If you installed DataDrip before scripts existed, run the upgrade generator:

rails generate data_drip:install_scripts

This creates the app/scripts directory and the data_drip_script_runs migration.

Creating Scripts

Generate a new script:

rails generate data_drip:script BackfillCompanyTimezones

A script declares its inputs with the input DSL and implements a single call method:

# app/scripts/backfill_company_timezones.rb
class BackfillCompanyTimezones < DataDrip::Script
  description "Recomputes the timezone of a company from its address."

  input :company_id, :integer, required: true
  input :mode, :enum, values: %w[fast thorough], required: true
  input :dry_run, :boolean, default: true
  input :effective_date, :date

  def call
    company = Company.find(company_id)
    log "Recomputing timezone for #{company.name} (mode: #{mode})"

    timezone = TimezoneResolver.call(company.address, mode: mode)

    if dry_run
      log "[dry run] would set timezone to #{timezone}"
    else
      company.update!(timezone: timezone)
      log "Timezone updated to #{timezone}"
    end
  end
end

• description is shown in the UI so other developers know what the script does.
• input name, type, default:, required: declares a typed input. The UI form is generated automatically from these declarations, values are coerced to the declared type, and required: true inputs are validated before the run is created (required booleans accept false but not missing). Supported types are the same as backfill attributes: :string, :integer, :decimal, :float, :boolean, :date, :time, :datetime and :enum.
• log(message) appends a timestamped line to the run's output, persisted immediately and streamed to the show page. Note each log call issues one database update — log sparsely inside hot loops.

Running Scripts

Navigate to /data_drip/script_runs, pick a script, fill in its inputs and trigger it — immediately or scheduled for later. The show page displays live status, output, and the error with backtrace if the script failed. Runs can only be deleted while they are still enqueued; anything that executed stays in the history.

Script Hooks

Scripts fire lifecycle hooks with the same precedence rules as backfills (script class first, then the global handler): on_script_run_pending, on_script_run_enqueued, on_script_run_running, on_script_run_completed, on_script_run_failed. Each receives the ScriptRun.

Notes

• Script runs are enqueued on the shared DataDrip.queue_name queue (default: ENV["DATA_DRIP_QUEUE"] or :data_drip), the same one backfills use — make sure your worker processes it.
• Datetime inputs are interpreted without timezone conversion (only the run's top-level "start at" field is converted from your browser timezone).
• If your base_job_class retries on errors, a failed script will re-run from scratch. Make scripts idempotent, or configure discard_on in your base job class.

Contributing

Bug reports and pull requests are welcome on GitHub at https://github.com/factorialco/data_drip.

License

The gem is available as open source under the terms of the MIT License.