How to figure out what processes are using IO on MacOS / debugging mysterious performance problems
This blog post is a loose documentation of some problems I've had with my mac lately and how I maybe diagnosed them, for my own documentation and maybe some of it will prove useful to people who find it when searching.
Lately I've been having mysterious problems on and off with my Mac laptop. Sometimes it almost seems like a common kernel_task situation, but the cause there is supposed to be overheating, and I'm not in any particularly out of the ordinary or changed heat situation when it happens. Other times, kernel_task doesn't seem to be the issue but the issue seems like an IO issue - something is doing a lot of reads and writes, inhibiting other things, so even thought CPU isn't pegged, the mac is slow.
One thing that might be the issue is an interaction between some combination of these things:
- the "deleted" process
- CleanMyMac's background task monitoring file changes for some reason
- while doing frontend dev, my node/npm setup, which doesn't have fsevent installed because of other missing dependencies, doing a lot of file reads (and writes?)
- maybe other things I removed over the months while experimenting - i'll add things to this list as I remember/discover them
How to figure out what is using CPU and/or IO
For simple ongoing CPU usage monitoring, you probably already know how to use Activity Monitor.app or top/htop.
There might be processes that run often and then exit very quickly. These won't show up in Activity Monitor or top/htop. To find these, you can run
sudo execsnoop -va
To use this, you need to have SIP disabled.
As you watch the output, look for patterns. Maybe you have a previous clue/hunch about what's causing the problem. You can grep the output based on this. The output will, crucially, include the PPID column - the PID of the parent process (the process which launched the process) in the log. This might stay the same while the short-lived processes in the log fly by. This might be the problematic process.
To monitor every process that is reading or writing to disk:
This might give clues. The process of finding the problem might involve going back and forth between fs_usage and execsnoop, greping the output of one based on a clue from the other.
I found fs_usage when googling for using iotop on mac. If you have SIP turned off you can use mac's built-in iotop, but on mac it's not as useful as it is on linux. There's no way to turn off commutative disk usage to figure out which processes are the biggest ongoing users. But still, turning it on and watching the output for a while might offer some clues.
Using these methods, I uninstalled a variety of software. I wish I wrote down everything I did along the way. None seemed to be the smoking gun. But, knock on wood, my machine hasn't had problems for a while.
Chrome, Launchbar, Spotlight, mds
Based on this, I eventually uninstalled Chrome, which I'm fairly certain solved 80% of my mysterious performance problems.
Something else I've noticed throughout this process is Launchbar frequently reindexing (little window comes up for several seconds telling me it's scanning my files). Also, it seems to reindex more frequently when I have my node frontend build server running. I can't find a way to configure its reindexing frequency. I think it might be based on total disk change events? I've often suspected that this was either a cause or a consequence of whatever my problem is. Recently I uninstalled Launchbar and installed Alfred instead. After installing Alfred I noticed that some of my Applications were not findable. Alfred has a fantastic built-in debugger which will tell you why a file isn't indexable. Through this I found that the missing apps didn't have proper metadata which is built by Spotlight (not sure if it's directly associated with the file, or in a Spotlight database). I've often vaguely suspected that the spotlight indexer was either causing problems or behaving badly because of other problems, so this was very interesting to me. I followed their instructions to rebuild this. It didn't work, the metadata was still broken! After some searching on the Alfred forums, I found someone recommending
sudo mdutil -a -i off,
sudo mdutil -a -i on, wait for reindex to finish (do a search for "a" in spotlight to view progress). It worked! Now the applications can be found by Alfred.
And also, my computer is maybe running better than ever?
So what was the cause? Chrome, Launchbar, mds, a broken spotlight database, or some old services and launchers I removed and forgot to document? I may never know.