Delayed job, a real pain in production

There are so many ways to handle background jobs in Ruby on Rails. But the most simplest one is delayed job (DJ) as I showed you in the following post Handling delayed jobs. Simple in the sense that it requires very less effort to set up and in uses single table to handle the jobs.

It's a very good solution for development environment, but in production people starts complaining of problems such as:

• DJ is eating up my server resources.
• DJ is firing new daemon processes every now and then.
• Many DJ instances are running at a time.

And many more similar kind of issues.

To run delayed job daemon along with rails server, people often end up creating config/initializers/start_worker.rb, and put codes something like

Thread.new do
  system("rake jobs:work")
end

I don't encourage this process.
Better solution will be to start delayed job using cron jobs.
There's a gem available for this purpose: whenever.

• Add this line to your gemfile: gem "whenever", :require => false
• $ bundle install
• $ wheneverize .
• This will create config/schedule.rb.
• Open up that file and put the following codes in it:

       set :output, "/path_to_your_project/log/cron_log.log"

       every :reboot do

         command "cd /path_to_your_project && RAILS_ENV=production script/delayed_job start"

       end

• This will start delayed job on every server boot. You may also write 

       every 10.hours do

         command "cd /path_to_your_project && RAILS_ENV=production script/delayed_job restart"

       end
          This will restart delayed job after every 10 hours. Although point number 6 is not a very feasible    
          solution, but it surely helps in some cases.

• $ whenever --update-crontab your_project_name.
• $ crontab -l

          To check whether the jobs you just made are listed.

• To monitor whether delayed job is running you can try God or monit. You can get some ideas from the Railscasts / a blog on delayed job/ monit example