MongoDB: How Crash Recovery Works

MongoDB: How Crash Recovery Works

Table of Contents



1. Oracle Replays Uncommitted Data. MongoDB Does Not.

After a crash, Oracle instance recovery has two phases: roll forward (replay all redo, committed or not) and roll back (use undo to remove the uncommitted changes). MongoDB needs only one phase. A change is written to the journal only when it is committed, so there is nothing to undo.

MongoDBOracle
One phase: replay the committed journal records after the last checkpoint.Two phases: roll forward with redo, then roll back with undo.
Uncommitted transaction changes were never journaled, so they are just gone.Uncommitted changes are in the redo, so they are replayed and then undone by SMON.
No undo tablespace is used in recovery.Undo segments are needed for the roll back phase.
Runs automatically at startup after an unclean shutdown.Runs automatically at startup after an instance crash.

Why is it simpler? A single-document write is committed as soon as it finishes, and a multi-document transaction is written to the journal only at commitTransaction(), all together as one unit. So at crash time, every journal record is either fully committed or not there at all.

2. What Happens When mongod Restarts After a Crash

You do not run any recovery command. When mongod starts and finds that the last shutdown was not clean (kill -9, power loss, OOM kill), it recovers on its own:


+----------------------------------------------+
| 1. CRASH                                     |    kill -9, power loss, OOM killer
|    memory (WiredTiger cache) is lost         |    cache = Oracle buffer cache
|    data files and journal on disk survive    |
+-----------------------+----------------------+
                        |
                        v
+----------------------------------------------+
| 2. mongod STARTS                             |
|    sees that the last shutdown was unclean   |
+-----------------------+----------------------+
                        |
                        v
+----------------------------------------------+
| 3. OPEN THE LAST CHECKPOINT                  |    checkpoints are taken every 60 s
|    data files are consistent as of that time |    this is the starting point
+-----------------------+----------------------+
                        |
                        v
+----------------------------------------------+
| 4. READ THE JOURNAL                          |    like reading Oracle online redo
|    find the records after the checkpoint     |
+-----------------------+----------------------+
                        |
                        v
+----------------------------------------------+
| 5. REPLAY COMMITTED RECORDS, IN ORDER        |    only committed work is in the journal
|    the changes are applied to the data       |    NO undo phase is needed
+-----------------------+----------------------+
                        |
                        v
+----------------------------------------------+
| 6. REPLICA SET MEMBER ONLY                   |    see Section 4
|    fix the oplog and replay it if needed     |
+-----------------------+----------------------+
                        |
                        v
+----------------------------------------------+
| 7. READY                                     |
|    mongod opens the port for clients         |
+-----------------------+----------------------+
                        |
                        v
+----------------------------------------------+
| 8. REJOIN THE REPLICA SET (if any)           |    catches up from other members
|    a rollback may follow if the node had     |    see the Rollback post
|    writes the new primary does not have      |
+----------------------------------------------+

The time recovery takes depends on how much journal has to be replayed. Since a checkpoint runs every 60 seconds, that is normally small, so MongoDB usually restarts in seconds. Oracle behaves the same way: shorter checkpoint intervals mean faster instance recovery.

MongoDBOracle
Last checkpoint (every 60 s)Last checkpoint (checkpoint SCN in the control file)
Journal after the checkpointOnline redo after the checkpoint
Replay of committed recordsRoll forward
Not neededRoll back of uncommitted work (SMON and undo)
Automatic at startupAutomatic at STARTUP

3. Which Writes Survive a Crash?

It depends on where the write was when the crash happened. A write in the journal on disk is safe. A write that was only in memory is lost.

Situation at crash timeAfter recoveryWhy
Write already flushed to the journal (older than about 100 ms)✅ RecoveredIt is replayed from the journal.
Write with j: true that returned OK✅ RecoveredOK was sent only after the journal flush.
w: 1 write in the last 100 ms, not yet flushed❌ Can be lostIt was only in memory. This is the COMMIT WRITE NOWAIT risk.
w: "majority" write that returned OK✅ SafeMost members have it in their journal.
Transaction that had not called commitTransaction()❌ Gone (aborted)It was never journaled as committed. The client must retry.
Transaction that was committed✅ All or nothingIt is written as one unit, so you never get half of it.

Special case: in a sharded cluster, a transaction that already reached the prepared state (MongoDB 4.2 and above) is not dropped. It comes back in the prepared state and waits for the coordinator to say commit or abort. This is similar to an in-doubt distributed transaction in Oracle, which RECO resolves.

Key point: If the client never got OK for a write, you do not know whether it was saved. After a crash, check the data or make the write safe to repeat (for example, use a unique _id).

4. Extra Step on a Replica Set

A replica set member has one more thing to fix, because of the oplog. The oplog is written by many threads at the same time, so after a crash it can have a small “hole”: a newer entry saved while an older one was not. MongoDB handles it like this:

  1. It cuts the oplog back to the last point that is complete (the oplog truncate point), so there are no gaps.
  2. It replays oplog entries from the checkpoint’s stable timestamp up to the end of the oplog, so the data matches the oplog.
  3. The node starts and, as a secondary, copies anything newer from the other members.
  4. If the node had writes that the new primary does not have (for example, w:1 writes before a failover), it rolls them back. See the Rollback post.

The details of this step differ a little between MongoDB versions. The idea is the same: recovery makes the node match its own oplog, and replication then brings it up to date with the rest of the set.

MongoDBOracle
Node recovers itself first, then catches up from the other members.Standby recovers from its own redo, then gets the missing redo from the primary (redo gap fetch).
Writes only on the crashed node can be rolled back.The old primary may need FLASHBACK DATABASE to reinstate.
If the node was down too long, it needs a full resync (initial sync).If the archived logs are gone, the standby must be rebuilt.

5. Try It Yourself (Test Server Only)

Test A: a committed write survives a crash. Use a test replica set or a test mongod. Never run kill -9 on a production server.


// In mongosh: write with j:true, so OK is returned after the journal flush
db.getSiblingDB("shop").orders.insertOne(
  { _id: "crash-test-1", item: "laptop" },
  { writeConcern: { w: 1, j: true } }
)

# In the Linux shell: crash the server (not a clean shutdown)
kill -9 $(pidof mongod)

# Start it again (use your own service name or command)
sudo systemctl start mongod

// After the restart: the document is there
db.getSiblingDB("shop").orders.find({ _id: "crash-test-1" })

Test B: an uncommitted transaction is gone after a crash. Transactions need a replica set, so use a test replica set (a single-node one is enough).


// The collection must exist before the transaction starts
db.getSiblingDB("shop").createCollection("orders")

const session = db.getMongo().startSession()
session.startTransaction()
session.getDatabase("shop").orders.insertOne({ _id: "crash-test-2", item: "phone" })

// Do NOT commit. Now crash the server from another terminal:
//   kill -9 $(pidof mongod)

// After the restart: the document is NOT there
db.getSiblingDB("shop").orders.find({ _id: "crash-test-2" })

See what recovery did in the log (log file path depends on your install):


# WiredTiger recovery messages (JSON log format, MongoDB 4.4 and above)
grep '"c":"WTRECOV"' /var/log/mongodb/mongod.log

# Startup and storage messages around the restart
grep '"c":"STORAGE"' /var/log/mongodb/mongod.log | tail -20

Check that the node is healthy again:


rs.status().members.map(m => ({ name: m.name, state: m.stateStr, optime: m.optimeDate }))
// State should return to PRIMARY or SECONDARY
// (a node can show STARTUP2 or ROLLBACK for a while)

Bottom line for Oracle DBAs: MongoDB crash recovery is Oracle instance recovery without the roll back phase. It starts from the last checkpoint and replays only committed journal records, so an uncommitted transaction vanishes by itself. The one real risk is a w:1 write in the last 100 ms. Use j: true or w: "majority" for data you cannot lose.

Caution: Your use of any information or materials on this website is entirely at your own risk. It is provided for educational purposes only. It has been tested internally, however, we do not guarantee that it will work for you. Ensure that you run it in your test environment before using.
Thank you
Rajasekhar Amudala