Table of Contents

1. Overview and Golden Rules

This guide covers Situation A: the primary survives and both secondaries fail. mongo01 is the primary. mongo02 and mongo03 go down, for example because of a server problem in data center 1 and a network break to data center 2. There is no automatic failover, because the survivor has only 1 of 3 votes. The primary cannot see a majority, so it steps down by itself and becomes SECONDARY. Writes fail until a second node is back. This is on purpose. It prevents two nodes from accepting writes at the same time (split brain).

Replica set name: rs0. The service is mongod.service, managed with systemctl only.

💡 Golden rules:
• First find out if the two nodes are only stopped or permanently lost. The fix is different.
• If they are only stopped, start them. As soon as 2 of 3 are up, a primary is elected by itself.
• A forced reconfig is the last resort, only for a permanent loss.
• No data loss in this situation. mongo01 was the primary, so it has every write, and the other nodes can only have the same or older data.
• Connect to the survivor with directConnection=true, because the replica set URI hangs without a primary.

2. What Happens

Step 1 – (Test only) Simulate the failure. In a test environment, on mongo02 and mongo03:

Enterprise rs0 [direct: primary] test> rs.status().members.map(m => m.name + " " + m.stateStr + " health=" + m.health)
[
'mongo01:27017 PRIMARY health=1',
'mongo02:27017 SECONDARY health=1',
'mongo03:27017 SECONDARY health=1'
]
Enterprise rs0 [direct: primary] test> 

# Start Simulation only on LAB systems

DISCLAIMER: This is test case for learning purposes and should only be used on testing systems. I will not take any responsibility of any consequences or loss of data caused by this command.

[root@mongo03 ~]# systemctl stop mongod
[root@mongo02 ~]# systemctl stop mongod

In a real incident you skip this step.

Step 2 – Connect to mongo01 and check the state. The step down takes about 10 seconds. Run the command again if you still see PRIMARY.

[mongop@mongo01 ~]$ /app/mongodb/7.0.43/bin/mongosh --host localhost -u admin --authenticationDatabase admin
Enter password: ******
Current Mongosh Log ID: 6af06e8325a22e7dbeda794a
Connecting to:          mongodb://@localhost:27017/?directConnection=true&serverSelectionTimeoutMS=2000&authSource=admin&appName=mongosh+2.12.0
Using MongoDB:          7.0.43
Using Mongosh:          2.12.0
mongosh 2.14.1 is available for download: https://www.mongodb.com/try/download/shell

For mongosh info see: https://www.mongodb.com/docs/mongodb-shell/

Enterprise rs0 [direct: secondary] test>

Enterprise rs0 [direct: secondary] test> rs.status().members.map(m => m.name + "  " + m.stateStr + "  health=" + m.health)
[
  'mongo01:27017  SECONDARY  health=1',
  'mongo02:27017  (not reachable/healthy)  health=0',
  'mongo03:27017  (not reachable/healthy)  health=0'
]

The old primary shows SECONDARY. This is correct. There is no primary, so writes fail and the application gets errors such as “not primary”.

Stopped or permanently lost? Find out which one it is before you act:

If you are not sure, treat it as “stopped” and try to start the nodes first. A forced reconfig cannot be undone.

3. Normal Fix: Start a Failed Node

You need only one of the two failed nodes. Then 2 of 3 are up and an election runs. Start the node in data center 1 (mongo02) first if you can.

Step 1 – Find the cause and start one node (here mongo02):

[root@mongo02 ~]# journalctl -u mongod -n 50
[root@mongo02 ~]# tail -50 /logs/mongodb/mongod.log
[root@mongo02 ~]# df -h; free -m          # disk full or out of memory?

[root@mongo02 ~]# systemctl start mongod
[root@mongo02 ~]# systemctl status mongod --no-pager
● mongod.service - MongoDB Enterprise Database Server
   Loaded: loaded (/etc/systemd/system/mongod.service; enabled; vendor preset: disabled)
   Active: active (running) since Sun 2026-11-08 07:46:02 EST; 12s ago
     Docs: https://docs.mongodb.com/manual
  Process: 8134 ExecStop=/app/mongodb/7.0.43/bin/mongod --config /app/mongodb/config/mongod.conf --shutdown (code=exited, status=0/SUCCESS)
  Process: 8162 ExecStart=/app/mongodb/7.0.43/bin/mongod --config /app/mongodb/config/mongod.conf (code=exited, status=0/SUCCESS)
 Main PID: 8164 (mongod)
    Tasks: 83 (limit: 17684)
   Memory: 189.6M
   CGroup: /system.slice/mongod.service
           └─8164 /app/mongodb/7.0.43/bin/mongod --config /app/mongodb/config/mongod.conf

Nov 08 07:46:00 mongo02 systemd[1]: Starting MongoDB Enterprise Database Server...
Nov 08 07:46:00 mongo02 mongod[8162]: {"t":{"$date":"2026-11-08T12:46:00.879Z"},"s":"I",  "c":"CONTROL",  "id":5760901, "ctx":"main","msg":"Applied --setParameter options","attr":{"serv…"value":true}}}}
Nov 08 07:46:00 mongo02 mongod[8162]: about to fork child process, waiting until server is ready for connections.
Nov 08 07:46:00 mongo02 mongod[8164]: forked process: 8164
Nov 08 07:46:02 mongo02 mongod[8162]: child process started successfully, parent exiting
Nov 08 07:46:02 mongo02 systemd[1]: Started MongoDB Enterprise Database Server.
Hint: Some lines were ellipsized, use -l to show in full.
[root@mongo02 ~]#

Step 2 – Check that a primary appears. It takes a few seconds. mongo01 has priority: 2 and the newest data, so it becomes PRIMARY again:

Enterprise rs0 [direct: secondary] test> rs.status().members.map(m => m.name + " " + m.stateStr + " health=" + m.health)
[
'mongo01:27017 PRIMARY health=1',   <----- mongo01 became as primary
'mongo02:27017 SECONDARY health=1',
'mongo03:27017 (not reachable/healthy) health=0'
]
Enterprise rs0 [direct: primary] test>

Writes work again with 2 of 3 members.

Step 3 – Bring the third node back. Find the cause on mongo03 (check the link between the data centers too), start it, and wait for SECONDARY with about 0 seconds of lag:

[root@mongo03 ~]# systemctl start mongod

On Primary (mongo01) # Check cluster health status

Enterprise rs0 [direct: primary] test> rs.status().members.map(m => m.name + " " + m.stateStr + " health=" + m.health)
[
  'mongo01:27017 PRIMARY health=1',
  'mongo02:27017 SECONDARY health=1',
  'mongo03:27017 SECONDARY health=1'    <----- mongo03 also came back online
]
Enterprise rs0 [direct: primary] test>

# Check the replication status 
Enterprise rs0 [direct: primary] test> rs.printSecondaryReplicationInfo()
source: mongo02:27017
{
  syncedTo: 'Sun Nov 08 2026 07:49:08 GMT-0500 (Eastern Standard Time)',
  replLag: '0 secs (0 hrs) behind the primary '    <----------- no lag
}
---
source: mongo03:27017
{
  syncedTo: 'Sun Nov 08 2026 07:49:08 GMT-0500 (Eastern Standard Time)',
  replLag: '0 secs (0 hrs) behind the primary '   <------------ no lag
}
Enterprise rs0 [direct: primary] test>

All Tasks Completed here. All nodes are healthy. No lag. ALL GOOD.

Do not proceed further until you know clearly.

⚠️ OPTIONAL: DO NOT RUN THIS SECTION (Forced Reconfig & Add nodes back to replica set).
Use it ONLY IF Section 3 fails AND both secondary nodes (mongo02 and mongo03) are PERMANENTLY LOST. Only then, proceed.
It cannot be undone. There is no rollback.

After the forced reconfig, mongo01 is the only member left in the replica set (rs0). It becomes PRIMARY and accepts writes on its own until you add the other nodes back.

4. Final Option: Forced Reconfig (Permanent Loss Only)

Use this only when mongo02 and mongo03 and their data are permanently lost. It is the same as a Data Guard failover. There is no data loss here, because mongo01 has all the writes. Only the two lost nodes are removed from the configuration.

Step 1 – Take a backup of mongo01 before you change anything, if you can (for example a mongodump).

Step 2 – Force mongo01 to be a one-member set. On mongo01:

Enterprise rs0 [direct: secondary] test> cfg = rs.conf()
Enterprise rs0 [direct: secondary] test> cfg.members = cfg.members.filter(m => m.host == "mongo01:27017")
[
{
_id: 0,
host: 'mongo01:27017',
arbiterOnly: false,
buildIndexes: true,
hidden: false,
priority: 2,
tags: {},
secondaryDelaySecs: Long('0'),
votes: 1
}
]
Enterprise rs0 [direct: secondary] test> rs.reconfig(cfg, { force: true })
{
ok: 1,
'$clusterTime': {
clusterTime: Timestamp({ t: 1794148829, i: 1 }),
signature: {
hash: Binary.createFromBase64('TEGk4cDcscC6BgzPd/nAsFqBUf0=', 0),
keyId: Long('7705068198269288454')
}
},
operationTime: Timestamp({ t: 1794148829, i: 1 })
}
Enterprise rs0 [direct: secondary] test> rs.status().members.map(m => m.name + " " + m.stateStr + " health=" + m.health)
[ 'mongo01:27017 PRIMARY health=1' ]
Enterprise rs0 [direct: primary] test>

Step 3 – Add the lost servers back. After the forced reconfig, mongo01 runs alone as a one-member replica set (rs0). Follow Section 5 to add mongo02 and mongo03 back.

5. Add the Lost Nodes Back to the Replica Set

After the forced reconfig, mongo01 is the only member left in rs0. It is not a standalone server. It is still a replica set, so you can add mongo02 and mongo03 back with rs.add(), and you do not need to restart mongo01. Each new node does an initial sync: it copies all the data from mongo01 and then becomes SECONDARY. Add the nodes one at a time, starting with mongo02 (same data center as mongo01).

⚠️ Danger: the new servers must have an empty dbPath. The lost nodes must never come back with their old data, because their old configuration conflicts with the new set. Always wipe them before you add them. Never use a forced reconfig when the nodes are simply stopped.

Step 1 – Prepare the new server (mongo02). It needs the same MongoDB version, the same mongod.conf, the same keyfile, and an empty dbPath:

[mongop@mongo02 ~]$ /app/mongodb/7.0.43/bin/mongod --version | head -1
db version v7.0.43
[mongop@mongo02 ~]$
[mongop@mongo02 ~]$ egrep -i 'dbPath|replSetName|keyFile|bindIp' /app/mongodb/config/mongod.conf
dbPath: /data/mongodb
bindIp: 127.0.0.1,192.168.2.102 # change per node
keyFile: /app/mongodb/keyfile/mongodb-keyfile
replSetName: rs0
[mongop@mongo02 ~]$

# Backup

[mongop@mongo02 ~]$ mkdir -p /backup/mongodb/backup_old
[mongop@mongo02 ~]$ cp -rp /data/mongodb /backup/mongodb/backup_old
[mongop@mongo02 ~]$ ls -l /backup/mongodb/backup_old
total 4
drwx------ 4 mongop mongogrp 4096 Nov 8 09:40 mongodb
[mongop@mongo02 ~]$

# Remove data from data path

[mongop@mongo02 ~]$ hostname
mongo02

[mongop@mongo02 ~]$ rm -rf /data/mongodb/*   (or)  move it safe side instead of remove
[mongop@mongo02 ~]$

[mongop@mongo02 ~]$ ls -la /data/mongodb                # the dbPath must be EMPTY (use your own dbPath)
total 0
drwx------. 2 mongop mongogrp 6 Nov 8 09:48 .
drwxr-xr-x. 3 root root 21 Oct 1 02:17 ..
[mongop@mongo02 ~]$

# all names must resolve 
# address hosts, IPv4 only”. (a= address hosts=/etc/hosts v4=IPv4 only)
[mongop@mongo02 ~]$ getent ahostsv4 mongo01 mongo02 mongo03   
192.168.2.101 STREAM mongo01
192.168.2.101 DGRAM
192.168.2.101 RAW
192.168.2.102 STREAM mongo02
192.168.2.102 DGRAM
192.168.2.102 RAW
192.168.3.103 STREAM mongo03
192.168.3.103 DGRAM
192.168.3.103 RAW
[mongop@mongo02 ~]$

• replSetName must be rs0, and bindIp must include the server’s own IP.
• The keyfile must have the same content as on mongo01, permissions 400, and owner mongop.
• Port 27017 must be open between all nodes (check the firewall between the data centers too).

Step 2 – Start mongod on mongo02. It starts with empty data and is not yet part of the replica set. This is normal.

[root@mongo02 ~]# systemctl start mongod
[root@mongo02 ~]# systemctl status mongod --no-pager
● mongod.service - MongoDB Enterprise Database Server
   Loaded: loaded (/etc/systemd/system/mongod.service; enabled; vendor preset: disabled)
   Active: active (running) since Sun 2026-11-08 10:01:54 EST; 4s ago
     Docs: https://docs.mongodb.com/manual
  Process: 9241 ExecStop=/app/mongodb/7.0.43/bin/mongod --config /app/mongodb/config/mongod.conf --shutdown (code=exited, status=0/SUCCESS)
  Process: 9514 ExecStart=/app/mongodb/7.0.43/bin/mongod --config /app/mongodb/config/mongod.conf (code=exited, status=0/SUCCESS)
 Main PID: 9516 (mongod)
    Tasks: 56 (limit: 17684)
   Memory: 74.6M
   CGroup: /system.slice/mongod.service
           └─9516 /app/mongodb/7.0.43/bin/mongod --config /app/mongodb/config/mongod.conf

Nov 08 10:01:53 mongo02 systemd[1]: Starting MongoDB Enterprise Database Server...
Nov 08 10:01:53 mongo02 mongod[9514]: {"t":{"$date":"2026-11-08T15:01:53.261Z"},"s":"I",  "c":"CONTROL",  "id":5760901, "ctx":"main","msg":"Applied --setParameter options","attr":{"serv…"value":true}}}}
Nov 08 10:01:53 mongo02 mongod[9514]: about to fork child process, waiting until server is ready for connections.
Nov 08 10:01:53 mongo02 mongod[9516]: forked process: 9516
Nov 08 10:01:54 mongo02 mongod[9514]: child process started successfully, parent exiting
Nov 08 10:01:54 mongo02 systemd[1]: Started MongoDB Enterprise Database Server.
Hint: Some lines were ellipsized, use -l to show in full.
[root@mongo02 ~]# ss -tlnp | grep 27017
LISTEN 0      128    192.168.2.102:27017      0.0.0.0:*    users:(("mongod",pid=9516,fd=16))
LISTEN 0      128        127.0.0.1:27017      0.0.0.0:*    users:(("mongod",pid=9516,fd=15))
[root@mongo02 ~]#

Step 3 – Add it from mongo01 (the primary).

Enterprise rs0 [direct: primary] test> rs.add("mongo02:27017")
{
ok: 1,
'$clusterTime': {
clusterTime: Timestamp({ t: 1794150194, i: 1 }),
signature: {
hash: Binary.createFromBase64('hDctmF9+9vj8O+aC16vDdV7iDck=', 0),
keyId: Long('7705068198269288454')
}
},
operationTime: Timestamp({ t: 1794150194, i: 1 })
}
Enterprise rs0 [direct: primary] test>

The new member gets the default priority: 1 and votes: 1. To set the priority when you add it, use rs.add({ host: "mongo02:27017", priority: 1 }).

Step 4 – Watch the initial sync. The new node goes to STARTUP2 while it copies the data, and then becomes SECONDARY:

Enterprise rs0 [direct: primary] test> rs.status().members.map(m => m.name + " " + m.stateStr + " health=" + m.health)
[
'mongo01:27017 PRIMARY health=1',
'mongo02:27017 STARTUP2 health=1'     <----- initial sync in progress
]

Enterprise rs0 [direct: primary] test> rs.status().members.map(m => m.name + " " + m.stateStr + " health=" + m.health)
[
'mongo01:27017 PRIMARY health=1',
'mongo02:27017 SECONDARY health=1'
]
Enterprise rs0 [direct: primary] test>

You can also follow it in the log on mongo02:

[root@mongo02 ~]# egrep -i 'initial sync' /logs/mongodb/mongod.log | tail -5
{"t":{"$date":"2026-11-08T10:03:15.738-05:00"},"s":"I",  "c":"-",        "id":4939300, "ctx":"monitoring-keys-for-HMAC","msg":"Failed to refresh key cache","attr":{"error":"InitialSyncActive: Cannot refresh keys collection cache during initial sync","nextWakeupMillis":5800}}    <---- normal while the sync runs, ignore
{"t":{"$date":"2026-11-08T10:03:15.783-05:00"},"s":"I",  "c":"INITSYNC", "id":21181,   "ctx":"ReplCoordExtern-1","msg":"Finished fetching oplog during initial sync","attr":{"oplogFetcherFinishStatus":"CallbackCanceled: oplog fetcher shutting down","lastFetched":"{ ts: Timestamp(0, 0), t: -1 }"}}    <---- normal at the end of the sync, not an error
{"t":{"$date":"2026-11-08T10:03:15.783-05:00"},"s":"I",  "c":"INITSYNC", "id":21191,   "ctx":"ReplCoordExtern-1","msg":"Initial sync attempt finishing up"}
{"t":{"$date":"2026-11-08T10:03:15.783-05:00"},"s":"I",  "c":"INITSYNC", "id":21192,   "ctx":"ReplCoordExtern-1","msg":"Initial sync status and statistics","attr":{"status":"successful","statistics":{"method":"logical","failedInitialSyncAttempts":0,"maxFailedInitialSyncAttempts":10,"initialSyncStart":{"$date":"2026-11-08T15:03:14.977Z"},"totalInitialSyncElapsedMillis":806,"initialSyncAttempts":[{"durationMillis":725,"status":"OK","syncSource":"mongo01:27017","rollBackId":4,"operationsRetried":1,"totalTimeUnreachableMillis":0}],"approxTotalDataSize":980,"approxTotalBytesCopied":980,"remainingInitialSyncEstimatedMillis":0,"appliedOps":0,"initialSyncOplogStart":{"$timestamp":{"t":1794150194,"i":1}},"initialSyncOplogEnd":{"$timestamp":{"t":1794150194,"i":1}},"totalTimeUnreachableMillis":0,"databases":{"databasesToClone":0,"databasesCloned":3,"admin":{"collections":3,"clonedCollections":3,"start":{"$date":"2026-11-08T15:03:15.072Z"},"end":{"$date":"2026-11-08T15:03:15.204Z"},"elapsedMillis":132,"admin.system.version":{"documentsToCopy":2,"documentsCopied":2,"indexes":1,"fetchedBatches":1,"bytesToCopy":104,"approxBytesCopied":104,"start":{"$date":"2026-11-08T15:03:15.072Z"},"end":{"$date":"2026-11-08T15:03:15.112Z"},"elapsedMillis":40,"receivedBatches":1},"admin.system.keys":{"documentsToCopy":2,"documentsCopied":2,"indexes":1,"fetchedBatches":1,"bytesToCopy":170,"approxBytesCopied":170,"start":{"$date":"2026-11-08T15:03:15.112Z"},"end":{"$date":"2026-11-08T15:03:15.150Z"},"elapsedMillis":38,"receivedBatches":1},"admin.system.users":{"documentsToCopy":1,"documentsCopied":1,"indexes":2,"fetchedBatches":1,"bytesToCopy":524,"approxBytesCopied":524,"start":{"$date":"2026-11-08T15:03:15.150Z"},"end":{"$date":"2026-11-08T15:03:15.204Z"},"elapsedMillis":54,"receivedBatches":1}},"config":{"collections":11,"clonedCollections":11,"start":{"$date":"2026-11-08T15:03:15.207Z"},"end":{"$date":"2026-11-08T15:03:15.735Z"},"elapsedMillis":528,"config.tenantMigrationRecipients":{"documentsToCopy":0,"documentsCopied":0,"indexes":2,"fetchedBatches":0,"bytesToCopy":0,"start":{"$date":"2026-11-08T15:03:15.207Z"},"end":{"$date":"2026-11-08T15:03:15.260Z"},"elapsedMillis":53,"receivedBatches":0},"config.tenantMigrationDonors":{"documentsToCopy":0,"documentsCopied":0,"indexes":2,"fetchedBatches":0,"bytesToCopy":0,"start":{"$date":"2026-11-08T15:03:15.260Z"},"end":{"$date":"2026-11-08T15:03:15.314Z"},"elapsedMillis":54,"receivedBatches":0},"config.system.preimages":{"documentsToCopy":0,"documentsCopied":0,"indexes":0,"fetchedBatches":0,"bytesToCopy":0,"start":{"$date":"2026-11-08T15:03:15.314Z"},"end":{"$date":"2026-11-08T15:03:15.329Z"},"elapsedMillis":15,"receivedBatches":0},"config.image_collection":{"documentsToCopy":0,"documentsCopied":0,"indexes":1,"fetchedBatches":0,"bytesToCopy":0,"start":{"$date":"2026-11-08T15:03:15.329Z"},"end":{"$date":"2026-11-08T15:03:15.368Z"},"elapsedMillis":39,"receivedBatches":0},"config.transactions":{"documentsToCopy":0,"documentsCopied":0,"indexes":2,"fetchedBatches":0,"bytesToCopy":0,"start":{"$date":"2026-11-08T15:03:15.368Z"},"end":{"$date":"2026-11-08T15:03:15.419Z"},"elapsedMillis":51,"receivedBatches":0},"config.sampledQueriesDiff":{"documentsToCopy":0,"documentsCopied":0,"indexes":2,"fetchedBatches":0,"bytesToCopy":0,"start":{"$date":"2026-11-08T15:03:15.419Z"},"end":{"$date":"2026-11-08T15:03:15.478Z"},"elapsedMillis":59,"receivedBatches":0},"config.external_validation_keys":{"documentsToCopy":0,"documentsCopied":0,"indexes":2,"fetchedBatches":0,"bytesToCopy":0,"start":{"$date":"2026-11-08T15:03:15.478Z"},"end":{"$date":"2026-11-08T15:03:15.531Z"},"elapsedMillis":53,"receivedBatches":0},"config.system.sessions":{"documentsToCopy":1,"documentsCopied":1,"indexes":2,"fetchedBatches":1,"bytesToCopy":132,"approxBytesCopied":132,"start":{"$date":"2026-11-08T15:03:15.531Z"},"end":{"$date":"2026-11-08T15:03:15.585Z"},"elapsedMillis":54,"receivedBatches":1},"config.sampledQueries":{"documentsToCopy":0,"documentsCopied":0,"indexes":2,"fetchedBatches":0,"bytesToCopy":0,"start":{"$date":"2026-11-08T15:03:15.585Z"},"end":{"$date":"2026-11-08T15:03:15.640Z"},"elapsedMillis":55,"receivedBatches":0},"config.analyzeShardKeySplitPoints":{"documentsToCopy":0,"documentsCopied":0,"indexes":2,"fetchedBatches":0,"bytesToCopy":0,"start":{"$date":"2026-11-08T15:03:15.640Z"},"end":{"$date":"2026-11-08T15:03:15.696Z"},"elapsedMillis":56,"receivedBatches":0},"config.system.indexBuilds":{"documentsToCopy":0,"documentsCopied":0,"indexes":1,"fetchedBatches":0,"bytesToCopy":0,"start":{"$date":"2026-11-08T15:03:15.696Z"},"end":{"$date":"2026-11-08T15:03:15.735Z"},"elapsedMillis":39,"receivedBatches":0}},"testdb":{"collections":1,"clonedCollections":1,"start":{"$date":"2026-11-08T15:03:15.735Z"},"end":{"$date":"2026-11-08T15:03:15.779Z"},"elapsedMillis":44,"testdb.test":{"documentsToCopy":1,"documentsCopied":1,"indexes":1,"fetchedBatches":1,"bytesToCopy":50,"approxBytesCopied":50,"start":{"$date":"2026-11-08T15:03:15.735Z"},"end":{"$date":"2026-11-08T15:03:15.779Z"},"elapsedMillis":44,"receivedBatches":1}}}}}}
{"t":{"$date":"2026-11-08T10:03:15.809-05:00"},"s":"I",  "c":"INITSYNC", "id":21163,   "ctx":"ReplCoordExtern-1","msg":"Initial sync done","attr":{"durationSeconds":0}}    <---- the sync finished
[root@mongo02 ~]#

How to read it:
• Green: success. "status":"successful", no failed attempts, all the bytes copied, and Initial sync done.
• Yellow: values to check. The data came from mongo01:27017 (the primary), and the sync took 806 ms.
• Orange: admin.system.users was copied from mongo01, so the admin user exists on the new node without creating it.
• The two lines marked “ignore” are normal during an initial sync. They are not errors.
Wait until mongo02 shows SECONDARY and the lag is about 0 seconds. The time depends on the size of the database.

💡 During the initial sync: mongo01 is the only source of the data, so expect extra disk and network load on it. Do this in a quiet window. In MongoDB 5.0 and later, a newly added member is not counted for the majority until its initial sync is done, so writes keep working on mongo01.

Step 5 – Add mongo03. Only after mongo02 is SECONDARY, repeat Steps 1 to 4 for mongo03. Do the same preparation as for mongo02 first: check that mongod is stopped, back up the old data, empty the dbPath, and check the configuration and the names. Start mongod only after that.

# mongod must be stopped on mongo03 before you touch the data path
[root@mongo03 ~]# systemctl status mongod --no-pager
   Active: inactive (dead)
[root@mongo03 ~]# pgrep -a mongod
[root@mongo03 ~]#

# Check the version and the configuration
[mongop@mongo03 ~]$ /app/mongodb/7.0.43/bin/mongod --version | head -1
db version v7.0.43
[mongop@mongo03 ~]$ egrep -i 'dbPath|replSetName|keyFile|bindIp' /app/mongodb/config/mongod.conf
dbPath: /data/mongodb
bindIp: 127.0.0.1,192.168.3.103 # change per node
keyFile: /app/mongodb/keyfile/mongodb-keyfile
replSetName: rs0
[mongop@mongo03 ~]$

# Backup
[mongop@mongo03 ~]$ mkdir -p /backup/mongodb/backup_old
[mongop@mongo03 ~]$ cp -rp /data/mongodb /backup/mongodb/backup_old
[mongop@mongo03 ~]$ ls -l /backup/mongodb/backup_old

# Remove data from data path (run this on mongo03 ONLY, check the hostname first)
[mongop@mongo03 ~]$ hostname
mongo03

[mongop@mongo03 ~]$ rm -rf /data/mongodb/*   (or)  move it safe side instead of remove
[mongop@mongo03 ~]$

[mongop@mongo03 ~]$ ls -la /data/mongodb                # the dbPath must be EMPTY (use your own dbPath)
total 0

# all names must resolve to IPv4
[mongop@mongo03 ~]$ getent ahostsv4 mongo01 mongo02 mongo03
192.168.2.101 STREAM mongo01
192.168.2.101 DGRAM
192.168.2.101 RAW
192.168.2.102 STREAM mongo02
192.168.2.102 DGRAM
192.168.2.102 RAW
192.168.3.103 STREAM mongo03
192.168.3.103 DGRAM
192.168.3.103 RAW
[mongop@mongo03 ~]$

⚠️ Check the hostname before you run rm -rf. Run it on mongo03 only. Never run it on mongo01, because that is the primary and the only copy of your data.

[root@mongo03 ~]# systemctl start mongod
[root@mongo03 ~]# systemctl status mongod --no-pager
● mongod.service - MongoDB Enterprise Database Server
   Loaded: loaded (/etc/systemd/system/mongod.service; enabled; vendor preset: disabled)
   Active: active (running) since Sun 2026-11-08 10:08:48 EST; 8s ago
     Docs: https://docs.mongodb.com/manual
  Process: 9113 ExecStop=/app/mongodb/7.0.43/bin/mongod --config /app/mongodb/config/mongod.conf --shutdown (code=exited, status=0/SUCCESS)
  Process: 9389 ExecStart=/app/mongodb/7.0.43/bin/mongod --config /app/mongodb/config/mongod.conf (code=exited, status=0/SUCCESS)
 Main PID: 9391 (mongod)
    Tasks: 72 (limit: 17684)
   Memory: 190.7M
   CGroup: /system.slice/mongod.service
           └─9391 /app/mongodb/7.0.43/bin/mongod --config /app/mongodb/config/mongod.conf

Nov 08 10:08:46 mongo03 systemd[1]: Starting MongoDB Enterprise Database Server...
Nov 08 10:08:46 mongo03 mongod[9389]: {"t":{"$date":"2026-11-08T15:08:46.405Z"},"s":"I",  "c":"CONTROL",  "id":5760901, "ctx":"main","msg":"Applied --setParameter options","attr":{"serv…"value":true}}}}
Nov 08 10:08:46 mongo03 mongod[9389]: about to fork child process, waiting until server is ready for connections.
Nov 08 10:08:46 mongo03 mongod[9391]: forked process: 9391
Nov 08 10:08:48 mongo03 mongod[9389]: child process started successfully, parent exiting
Nov 08 10:08:48 mongo03 systemd[1]: Started MongoDB Enterprise Database Server.
Hint: Some lines were ellipsized, use -l to show in full.
[root@mongo03 ~]# ss -tlnp | grep 27017
LISTEN 0      128    192.168.3.103:27017      0.0.0.0:*    users:(("mongod",pid=9391,fd=16))
LISTEN 0      128        127.0.0.1:27017      0.0.0.0:*    users:(("mongod",pid=9391,fd=15))
[root@mongo03 ~]#

Add it from mongo01 (the primary).

Enterprise rs0 [direct: primary] test> rs.add("mongo03:27017")
{
ok: 1,
'$clusterTime': {
clusterTime: Timestamp({ t: 1794150634, i: 1 }),
signature: {
hash: Binary.createFromBase64('xpg4QLBs/YgNME60AYFpDvbYW/s=', 0),
keyId: Long('7705068198269288454')
}
},
operationTime: Timestamp({ t: 1794150634, i: 1 })
}
Enterprise rs0 [direct: primary] test> 

mongo03 is in data center 2, so the initial sync goes over the link between the data centers and can take longer. Check the network bandwidth if it is slow.

Step 6 – Verify the replica set.

Enterprise rs0 [direct: primary] test> rs.status().members.map(m => m.name + " " + m.stateStr + " health=" + m.health)
[
'mongo01:27017 PRIMARY health=1',
'mongo02:27017 SECONDARY health=1',
'mongo03:27017 SECONDARY health=1'
]
Enterprise rs0 [direct: primary] test>

Enterprise rs0 [direct: primary] test> rs.printSecondaryReplicationInfo()
source: mongo02:27017
{
syncedTo: 'Sun Nov 08 2026 10:12:03 GMT-0500 (Eastern Standard Time)',
replLag: '0 secs (0 hrs) behind the primary '   <------------------ no lag
}
---
source: mongo03:27017
{
syncedTo: 'Sun Nov 08 2026 10:12:03 GMT-0500 (Eastern Standard Time)',
replLag: '0 secs (0 hrs) behind the primary '    <------------------ no lag
}
Enterprise rs0 [direct: primary] test>
Enterprise rs0 [direct: primary] test> rs.conf().members.map(m => ({ name: m.host, priority: m.priority }))
[
{ name: 'mongo01:27017', priority: 2 },
{ name: 'mongo02:27017', priority: 1 },
{ name: 'mongo03:27017', priority: 1 }
]
Enterprise rs0 [direct: primary] test>

✅ The replica set is rebuilt with one primary and two secondaries. mongo01 keeps priority: 2, so it stays the preferred primary.

6. Quick Reference