edamame :: usage

Design

Jobs are persisted to a tokyo tyrant backing store. Beanstalkd

(No failover or discovery, but yes restarting and reloading.)

Caveats

Weaknesses? Mainly that it will make an Erlang’er cry for its lack of concurrency correctness. Its goal is to work pretty well and to recover gracefully, but its design limits .

• We store jobs in two places: the central DB and the distributed queue.

• As always, your jobs must either be idempotent, or harmless if re-run: a job could start and do some or all of its job — but lose contact with the queue, causing the job to be re-run. This is inherent in beanstalkd (and most comparable solutions), not just edamame.

• Although God

TODOs

• Restarting is still manual: you have to run bin/sync.rb to reload the queue from the database
• Right now each jobs lives in full in the beanstalkd. This carries a heavy memory cost but a performance gain (the alternative is to do a second query to the DB once the job has been retrieved). I’m going to make both implementations available.
• The sinatra queue viewer doesn’t work at the moment.
• Tests :-/

Performance

From a production server:

 %CPU %MEM     VSZ    RSS    Command
  0.2  0.5   231044  67424  /usr/local/bin/ttserver -host 0.0.0.0 -port 11212 -log /var/log/god/ttyrant_11212.log /data/distdb/flat_delay_queue.tct
  0.1  6.4   798440 783580  /usr/local/bin/beanstalkd -l 0.0.0.0 -p 11210 -z 65535

With about 1M jobs in the db+queue, beanstalkd is taking up about 800MB memory and tyrant less than 100MB. (This is with the full job object stored in the queue: memory pig but faster)

Note on Patches/Pull Requests

• Fork the project.
• Make your feature addition or bug fix.
• Add tests for it. This is important so I don’t break it in a future version unintentionally.
• Commit, do not mess with rakefile, version, or history. (if you want to have your own version, that is fine but bump version in a commit by itself I can ignore when I pull)
• Send me a pull request. Bonus points for topic branches.

Endnotes

• If it’s not obvious: beanstalk + tokyo = edamame, a tasty Japanese appetizer.
• You might enjoy the Monkeyshines distributed API scraper, which this library was written to support.
• Beanstalk:
  • Beanstalk, a fast, distributed, in-memory workqueue service
  • Beanstalkd code
  • FAQ
  • Beanstalk Ruby Client
  • Tutorial from nubyonrails
  • Mailing list
  • Some beanstalk utilities — edamame has its own take on some of these.

• Tokyo Tyrant:
  • Tokyo Tyrant
  • Tokyo Tyrant Ruby libs
  • You’ll need the Tokyo Cabinet libs and the Tokyo Cabinet Ruby libs

• God process monitoring framework
  • http://railscasts.com/episodes/130-monitoring-with-god
  • Some code for the god conf is inspired by that railscast, this pastie, the one from the god docs, and Configuring GMail notifiers in God
  • Alternatives to God include (in order of complexity): Monit, perhaps with Munin; Cacti and Hyperic