Turning off the SQL cache in rails background jobs
to improve garbage collection
Here's something about memory in Rails background jobs that I sort of encountered over the years but only recently fully figured out. I'll jump straight to the point.
The rails query cache causes ruby to be less effective at garbage collection. This is an acceptable tradeoff in web processes which do not run for a long time dealing with a lot of objects. This was remediated at some point by it being disabled for the queries involved in for_each and friends. However, it's still used in the code in the block passed to these methods.
For some types of background jobs this can be a big problem. The solution (here for sidekiq, but easy to adapt to other scenarios) is to disable it for all background jobs:
app/lib/sidekiq_ar_cache.rb
class SidekiqArCache
def call(worker, msg, queue)
ActiveRecord::Base.uncached do
yield
end
end
end
config/initializers/sidekiq.rb
Sidekiq.configure_server do |config|
config.server_middleware do |chain|
chain.add SidekiqArCache
end
end
After enabling this, you might see the memory usage of your background jobs processes significantly decrease and/or become more consistent. The performance impact will probably be minimal, depending on your workload of course.
If you're interested in learning more about this, you can start here: