Here's something about memory in Rails background jobs that I sort of encountered over the years but only recently fully figured out. I'll jump straight to the point.
The rails query cache causes ruby to be less effective at garbage collection. This is an acceptable tradeoff in web processes which do not run for a long time dealing with a lot of objects. This was remediated at some point by it being disabled for the queries involved in
for_each and friends. However, it's still used in the code in the block passed to these methods.
For some types of background jobs this can be a big problem. The solution (here for sidekiq, but easy to adapt to other scenarios) is to disable it for all background jobs:
def call(worker, msg, queue)
Sidekiq.configure_server do |config|
config.server_middleware do |chain|
After enabling this, you might see the memory usage of your background jobs processes significantly decrease and/or become more consistent. The performance impact will probably be minimal, depending on your workload of course.
If you're interested in learning more about this, you can start here:
John Bachir's Code Blog