Table of Contents

1. Overview and Golden Rules

In Oracle Data Guard, a failover happens when the primary is lost and a standby takes over. In MongoDB this is automatic. If the primary becomes unreachable, the secondaries elect a new primary by themselves. There is no command to run. Your job is to watch it, find the cause, and bring the failed node back.

Replica set name: rs0. The service is mongod.service, managed with systemctl only. This post covers the primary (mongo01) fails.

💡 Golden rules:
• A 3-member set needs 2 members up to elect a primary. After the primary fails, 2 of 3 are still up, so the failover works.
• During the failover itself, only watch. Do not run rs.stepDown() or a forced reconfig.
• Use w: "majority" for important writes. Writes acknowledged by a majority survive a failover.
• Start the failed node with systemctl start mongod only. Test a failover in a test environment only.

2. How the Automatic Failover Works

   Timeline of an automatic failover:

   1. Primary stops answering         (members send heartbeats every 2 seconds)
   2. Secondaries wait                (electionTimeoutMillis, default 10 seconds)
   3. An eligible secondary           calls an election
   4. It gets 2 of 3 votes            --> becomes PRIMARY
   5. New primary accepts writes      --> total: usually about 12 seconds

Before a test, check the priority and the lag on the primary:

Enterprise rs0 [primary] test> rs.conf().members.map(m => ({ name: m.host, priority: m.priority, votes: m.votes }))
[
  { name: 'mongo01:27017', priority: 2, votes: 1 },
  { name: 'mongo02:27017', priority: 1, votes: 1 },
  { name: 'mongo03:27017', priority: 1, votes: 1 }
]

Enterprise rs0 [primary] test> rs.printSecondaryReplicationInfo()
source: mongo02:27017
{
  replLag: '0 secs (0 hrs) behind the primary '
}
---
source: mongo03:27017
{
  replLag: '0 secs (0 hrs) behind the primary '
}

⚠️ Application side: use the full replica set URI with retryWrites=true, so the driver finds the new primary and retries by itself: mongodb://mongo01:27017,mongo02:27017,mongo03:27017/?replicaSet=rs0&retryWrites=true. Clients see an error or a pause for a few seconds during the election.

3. Failover: The Primary Fails

Step 1 – (Test only) Simulate the failure. In a test environment, on the primary (mongo01), use one of these:

DISCLAIMER: This is test case for learning purposes and should only be used on testing systems. I will not take any responsibility of any consequences or loss of data caused by this command.

# Option 1: clean stop. The primary steps down first, so the election is quick.
[root@mongo01 ~]# systemctl stop mongod

# Option 2: hard crash. Secondaries wait the full election timeout (about 10 seconds).
[root@mongo01 ~]# pkill -9 mongod

# Option 3: shut down the server.
[root@mongo01 ~]# shutdown -h now

In a real incident you skip this step. The failure has already happened.

Step 2 – Connect to a surviving node and watch the election. Use a direct connection, because mongo01 is down:

On mongo02:

[mongop@mongo02 ~]$ /app/mongodb/7.0.43/bin/mongosh --host localhost -u admin --authenticationDatabase admin
Enter password: ******
Using MongoDB:          7.0.43
Using Mongosh:          2.12.0

Enterprise rs0 [direct: primary] test>

Enterprise rs0 [direct: primary] test> rs.status().members.map(m => m.name + "  " + m.stateStr + "  health=" + m.health)
[
  'mongo01:27017  (not reachable/healthy)  health=0',
  'mongo02:27017  PRIMARY  health=1',
  'mongo03:27017  SECONDARY  health=1'
]

If you see SECONDARY on both survivors, wait a few seconds and run it again. The election takes time.

Step 3 – Confirm why and when the election happened. Run this on the new primary:

Enterprise rs0 [direct: primary] test> rs.status().electionCandidateMetrics
{
  lastElectionReason: 'stepUpRequestSkipDryRun',
  lastElectionDate: ISODate('2026-11-08T10:47:29.182Z'),
  electionTerm: Long('19'),
  numVotesNeeded: 2,
  priorityAtElection: 1,
  electionTimeoutMillis: Long('10000'),
  priorPrimaryMemberId: 0,
  numCatchUpOps: Long('0'),
  newTermStartDate: ISODate('2026-11-08T10:47:29.205Z'),
  wMajorityWriteAvailabilityDate: ISODate('2026-11-08T10:47:29.230Z')
}

The reason depends on how the primary went down. After a clean stop, the old primary asks a secondary to take over, so you see stepUpRequestSkipDryRun (as above). After a hard crash or a server shutdown, the secondaries lose contact and you see electionTimeout. You can also check the log:

[mongop@mongo02 ~]$ egrep -i 'starting an election|election succeeded|transition to primary' /logs/mongodb/mongod.log | tail -5
{"t":{"$date":"2026-11-08T05:47:29.182-05:00"},"s":"I", "c":"ELECTION", "id":4615661, "ctx":"conn172","msg":"Starting an election due to step up request"}
{"t":{"$date":"2026-11-08T05:47:29.188-05:00"},"s":"I", "c":"ELECTION", "id":21450, "ctx":"ReplCoord-86","msg":"Election succeeded, assuming primary role","attr":{"term":19}}
{"t":{"$date":"2026-11-08T05:47:29.217-05:00"},"s":"I", "c":"REPL", "id":21331, "ctx":"OplogApplier-0","msg":"Transition to primary complete; database writes are now permitted","attr":{"term":19}}

Step 4 – Check that the application works. Run a test write with a majority acknowledgement on the new primary:

Enterprise rs0 [direct: primary] test> db.failover_test.insertOne({ ts: new Date() }, { writeConcern: { w: "majority" } })
{
  acknowledged: true,
  insertedId: ObjectId('6af054ba70cb0d4e9e0ec520')
}

Enterprise rs0 [direct: primary] test> db.failover_test.drop()
true

With two members up, a majority write still succeeds (2 of 3).

Step 5 – Check and change the priority (before you bring mongo01 back). Right now mongo01 has priority: 2, and mongo02 and mongo03 have priority: 1. This causes two problems:
• When mongo01 starts again and catches up, it takes the primary role back, with one more short interruption.
• If mongo02 restarts later, mongo03 (in the other data center) can become primary.

For now we want mongo02 to stay primary, and a data center 1 node to be preferred over mongo03. So raise mongo02 to priority 3 while mongo01 is still down. This is a temporary layout until mongo01 is fixed (see Section 4, Step 4). Run these commands on the current primary (mongo02).

Enterprise rs0 [direct: primary] test> cfg = rs.conf()
Enterprise rs0 [direct: primary] test> cfg.members.find(m => m.host == "mongo02:27017").priority = 3
Enterprise rs0 [direct: primary] test> rs.reconfig(cfg)
{ ok: 1 }

Enterprise rs0 [direct: primary] test> rs.conf().members.map(m => ({ name: m.host, priority: m.priority }))
[
  { name: 'mongo01:27017', priority: 2 },
  { name: 'mongo02:27017', priority: 3 },  <---- mongo02 has the highest priority
  { name: 'mongo03:27017', priority: 1 }
]

The reconfig works while mongo01 is down, because mongo02 and mongo03 are 2 of 3 members. The priority is stored in the replica set configuration, so mongo01 gets it automatically when it rejoins. You do not need to change anything on mongo01.

💡 Why this works: MongoDB moves the primary role by priority only to a node with a higher priority than the current primary. mongo01 (2) is lower than mongo02 (3), so nothing moves when mongo01 comes back. Both data center 1 nodes are higher than mongo03 (1), so mongo03 only becomes primary if both data center 1 nodes are down. If mongo03 became primary during the failover, mongo02 takes the role by priority within a few seconds.

✅ Failover complete. The replica set has a new primary (mongo02) and runs with 2 of 3 members. Now find the cause of the failure and bring mongo01 back (Section 4).

⚠️ Until mongo01 is back, you have no spare. If another member fails now, only 1 of 3 is left and there is no primary. Do not do any maintenance on the other two nodes meanwhile.

4. Bring the Failed Old Primary Back

Step 1 – Find the cause. Before you start it, check why the node failed:

# if node is down, please check and then start

[root@mongo01 ~]# systemctl status mongod --no-pager
[root@mongo01 ~]# journalctl -u mongod -n 50
[root@mongo01 ~]# tail -50 /logs/mongodb/mongod.log
[root@mongo01 ~]# df -h; free -m          # disk full or out of memory?

Step 2 – Start it, if down

[root@mongo01 ~]# systemctl start mongod
[root@mongo01 ~]# systemctl status mongod --no-pager
   Active: active (running)
[root@mongo01 ~]# ss -tlnp | grep 27017

Step 3 – Watch it rejoin and check the roles. It goes through STARTUP2 or RECOVERING and then becomes SECONDARY. Because of the priority change in Section 3 (Step 5), mongo01 does not take the primary role back. mongo02 stays PRIMARY.

Enterprise rs0 [direct: primary] test> rs.status().members.map(m => m.name + "  " + m.stateStr + "  health=" + m.health)
[
  'mongo01:27017  SECONDARY  health=1',
  'mongo02:27017  PRIMARY  health=1',
  'mongo03:27017  SECONDARY  health=1'
]
Enterprise rs0 [direct: primary] test>

Enterprise rs0 [direct: primary] test> rs.printSecondaryReplicationInfo()
source: mongo01:27017
{
syncedTo: 'Sun Nov 08 2026 06:47:19 GMT-0500 (Eastern Standard Time)',
replLag: '0 secs (0 hrs) behind the primary '   <-------------------- no lag
}
---
source: mongo03:27017
{
syncedTo: 'Sun Nov 08 2026 06:47:19 GMT-0500 (Eastern Standard Time)',
replLag: '0 secs (0 hrs) behind the primary '   <-------------------- no lag
}
Enterprise rs0 [direct: primary] test>

All Tasks Completed here. All nodes are healthy. No lag. ALL GOOD.

Do not proceed further until you know clearly.

⚠️ OPTIONAL: DO NOT CONTINUE WITH BELOW STEPS, It needs Little Downtime to make mong01 as NEW primary. Current primary is mongo01

Step 4 – Switch the primary back to mongo01 (when you are ready) — Optional , it needs short downtime to make mongo01 as primary

Continue only when mongo01 shows SECONDARY with about 0 seconds of lag.

. Choose a quiet window. Make mongo01 the highest priority and mongo02 the second. Run this on the current primary (mongo02):

Enterprise rs0 [direct: primary] test> cfg = rs.conf()
Enterprise rs0 [direct: primary] test> cfg.members.find(m => m.host == "mongo01:27017").priority = 3
Enterprise rs0 [direct: primary] test> cfg.members.find(m => m.host == "mongo02:27017").priority = 2
Enterprise rs0 [direct: primary] test> rs.reconfig(cfg)
{ ok: 1 }

mongo01 now has the highest priority and is caught up, so it takes the primary role within a few seconds. To move it at once, run db.adminCommand({ replSetStepUp: 1 }) on mongo01. Verify:

Enterprise rs0 [direct: primary] test> rs.status().members.map(m => m.name + "  " + m.stateStr)
[
  'mongo01:27017  PRIMARY',
  'mongo02:27017  SECONDARY',
  'mongo03:27017  SECONDARY'
]
Enterprise rs0 [direct: primary] test> rs.conf().members.map(m => ({ name: m.host, priority: m.priority }))
[
  { name: 'mongo01:27017', priority: 3 },
  { name: 'mongo02:27017', priority: 2 },
  { name: 'mongo03:27017', priority: 1 }
]

✅ Final layout: mongo01 is always the preferred primary, mongo02 is next (same data center), and mongo03 (other data center) is only the fallback. After a restart of any node, the primary goes back to mongo01 by itself.

💡 Rollback: if the old primary had writes that were not yet replicated when it failed, MongoDB rolls them back when it rejoins and saves them as BSON files in the rollback folder inside the dbPath. Review them and re-apply them if needed. Writes acknowledged with w: "majority" are never rolled back.

⚠️ OPTIONAL: DO NOT CONTINUE WITH BELOW STEPS, It’s require only when Full resync requires.

Full resync (only if the node is too stale). If the node was down longer than the oplog window, it stays in RECOVERING and the log says “too stale to catch up”. Check the window with rs.printReplicationInfo() on the primary. The fix is an initial sync. Use the storage.dbPath from your mongod.conf instead of /data/mongodb:

[root@mongo01 ~]# systemctl stop mongod
[root@mongo01 ~]# mv /data/mongodb /data/mongodb_old_$(date +%F)
[root@mongo01 ~]# mkdir /data/mongodb && chown mongop:mongop /data/mongodb
[root@mongo01 ~]# systemctl start mongod

The node goes to STARTUP2, copies all data from another member, and then becomes SECONDARY. Keep the keyfile and mongod.conf in place.

5. Quick Reference

💡 Tip: during an incident, keep one shell open on a surviving node and re-run the member status command after each step. It is the quickest way to see where the set is.