Skip to content

A Rails plugin/gem to run asynchronous processes in a configurable pipeline

License

Notifications You must be signed in to change notification settings

dtsato/pipeline

Repository files navigation

Pipeline

Description

Pipeline is a Rails plugin/gem to run asynchronous processes in a configurable pipeline.

Documentation

rdoc.info/projects/dtsato/pipeline

Features

  • Execution of sequential user-defined stages in an asynchronous pipeline

  • Persistence of pipeline instances and stages

  • Error recovery strategies:

    • Irrecoverable errors fail the entire pipeline

    • Recoverable errors are automatically retried (using dj’s exponential retry strategy)

    • Recoverable errors that require user input pause the pipeline for further retry

  • Cancelling/Resuming of a paused pipeline

  • Callbacks before and after executing stages and pipeline

Installation and Use

pipeline can be installed as either a RubyGem (recommended) or as a plugin.

Gem

To install pipeline as a RubyGem, add the following lines to your config/environment.rb file:

config.gem "pipeline"
config.gem "delayed_job"

And execute:

rake gems:install
rake gems:unpack # Optional, if you want to vendor the gem

Plugin

To install it as a plugin, run:

script/plugin install git://github.com/dtsato/pipeline.git

Generating the required tables

In order to persist your pipelines and stages, execute the following:

script/generate pipeline # To generate the migration scripts that will store pipelines
script/generate delayed_job # To generate the migration scripts for delayed_job
rake db:migrate

Starting your workers

You will also need to run your Delayed Job workers that will process the pipeline jobs in the background:

rake jobs:work

Dependencies

Usage

Check examples for more examples (including error-recovery and cancelling)

class Step1 < Pipeline::Stage::Base
  def perform
    puts("Started step 1")
    sleep 2
    puts("Finished step 1")
  end
end

class Step2 < Pipeline::Stage::Base
  def perform
    puts("Started step 2")
    sleep 3
    puts("Finished step 2")
  end
end

class TwoStepPipeline < Pipeline::Base
  define_stages Step1 >> Step2
end

Pipeline.start(TwoStepPipeline.new)

Copyright © 2009 Danilo Sato. See LICENSE for details.

About

A Rails plugin/gem to run asynchronous processes in a configurable pipeline

Resources

License

Stars

Watchers

Forks

Packages

No packages published

Languages