High CPU and rising memory are only symptoms. What actually tells you what's going on inside the database is a handful of status commands MongoDB ships with. This post picks the five best suited to drive alerts, and explains what each one shows and what rule it can become.
One command, one kind of signal, one rule you can actually deploy
Host-level CPU, memory and disk metrics are essential, but they can't answer "which query is eating resources" or "how far behind is that node". Bringing the database's internal state into your alerting makes rules specific and troubleshooting shorter.
Broad coverage, collectable on any system — but they only say "resources are tight", not why.
Straight from MongoDB itself, pointing to whether the cause is replication lag, a stuck operation or a backed-up queue.
First-hand health information for a replica set lives here. Each member's stateStr (PRIMARY / SECONDARY / RECOVERING and so on), health and optimeDate can all be read from it.
Subtract each secondary's optimeDate from the primary's and you get the replication lag. For a quick look, you can also use rs.printSecondaryReplicationInfo().
To find out "what is the database busy with right now", use currentOp. It returns every operation currently executing, including how many seconds it has been running, which namespace it touches and which client it came from.
Filtering on secs_running pulls out the operations that have been running unusually long — often long transactions, queries missing an index, or something a client forgot to finish.
currentOp requires the right privileges. After spotting a long operation, work out what it is before deciding to end it — don't killOp on sight.The profiler records operations that exceed a time threshold into the system.profile collection of the current database. Set the level to 1 and give it a slowms value, and only operations slower than that many milliseconds are recorded.
That turns "slow queries" from a vague feeling into data you can count: how many in a period, which collections they cluster on, how many documents they scanned.
system.profile is a fixed-size capped collection, so older records get overwritten; the profiler itself also has overhead, so don't set the threshold too low in production.serverStatus is where MongoDB's status information is most concentrated. Two parts of it are especially good as alert inputs.
globalLock.currentQueue: how many reads and writes are queued, waiting to run. An occasional non-zero queue is normal; a queue that is persistently non-zero means the database can't keep up.
connections: current is the number of open connections and available is how many more can be accepted. Add them for the total capacity — when usage approaches the limit, act early.
The thresholds below are starting-point examples only. Every cluster has its own baseline — observe normal levels for a while first, then adjust.
Signals like replication lag, connection count and slow-query ratio are exactly what WAP collects continuously, so they can feed alert rules directly — no need to log in and run commands by hand every time. The commands above are best used after an alert fires, to confirm what's happening on the scene.
How good a rule is depends on how well you understand MongoDB's internal state. Get these five commands down and most common problems can be spotted before they affect the business.