Table of Contents

1. Overview and Golden Rules

This guide covers Situation B: the primary and one secondary fail. mongo01 (the primary) and mongo02 (a secondary) are both in data center 1, so this is what you see when data center 1 goes down, for example because of a power or network failure. Only mongo03, a secondary in data center 2, is left. There is no automatic failover, because the survivor has only 1 of 3 votes. It cannot become primary, so there is no primary at all and writes fail. This is on purpose. It prevents two nodes from accepting writes at the same time (split brain).

Replica set name: rs0. The service is mongod.service, managed with systemctl only.

How much data can be lost?

💡 Golden rules:
• First find out if data center 1 is only down for a while or permanently lost. The fix is different.
• If it is only down, start the nodes. As soon as 2 of 3 are up, a primary is elected by itself.
• Start the old primary (mongo01) first when you can. It has the newest data.
• A forced reconfig is the last resort, only for a permanent loss. It can lose data.
• Connect to the survivor with directConnection=true, because the replica set URI hangs without a primary.

2. What Happens

Step 1 – (Test only) Simulate the failure. In a test environment, on mongo01 and mongo02 (data center 1):

# Start Simulation only on LAB systems

DISCLAIMER: This is test case for learning purposes and should only be used on testing systems. I will not take any responsibility of any consequences or loss of data caused by this command.

[root@mongo01 ~]# systemctl stop mongod
[root@mongo01 ~]#
[root@mongo02 ~]# systemctl stop mongod
[root@mongo02 ~]#

In a real incident you skip this step.

Step 2 – Connect to the survivor and check the state.

[mongop@mongo03 ~]$ /app/mongodb/7.0.43/bin/mongosh --host localhost -u admin --authenticationDatabase admin
Enter password: ******
Current Mongosh Log ID: 6af0952b739b3d669d59ab6d
Connecting to:          mongodb://@localhost:27017/?directConnection=true&serverSelectionTimeoutMS=2000&authSource=admin&appName=mongosh+2.12.0
Using MongoDB:          7.0.43
Using Mongosh:          2.12.0
mongosh 2.14.1 is available for download: https://www.mongodb.com/try/download/shell

For mongosh info see: https://www.mongodb.com/docs/mongodb-shell/

Enterprise rs0 [direct: secondary] test>

Enterprise rs0 [direct: secondary] test> rs.status().members.map(m => m.name + "  " + m.stateStr + "  health=" + m.health)
[
  'mongo01:27017  (not reachable/healthy)  health=0',
  'mongo02:27017  (not reachable/healthy)  health=0',
  'mongo03:27017  SECONDARY  health=1'
]

There is no primary, so writes fail and the application gets errors such as “not primary” or a server selection timeout.

Step 3 – Note how far the survivor got. This is the last operation mongo03 applied. Write it down.

If you have to run the (Final Option) forced reconfig, everything newer than this time is lost, and you can tell the application team the size of the gap:

Enterprise rs0 [direct: secondary] test> rs.status().members.filter(m => m.self).map(m => ({ name: m.name, optimeDate: m.optimeDate }))
[
{
name: 'mongo03:27017',
optimeDate: ISODate('2026-11-08T15:27:52.000Z')
}
]
Enterprise rs0 [direct: secondary] test>

Data center 1 is down for a while, or lost for good? Find out which one it is before you act:

If you are not sure, treat it as “down for a while” and try to start the nodes first. A forced reconfig cannot be undone.

3. Normal Fix: Start a Failed Node

You need only one of the two failed nodes. Then 2 of 3 are up and an election runs.

Step 1 – Start the old primary (mongo01) first. It has the newest data, because mongo03 copies data from the primary a little later and may miss the last writes. Find the cause, then start it:

[root@mongo01 ~]# journalctl -u mongod -n 50
[root@mongo01 ~]# tail -50 /logs/mongodb/mongod.log
[root@mongo01 ~]# df -h; free -m          # disk full or out of memory?

[root@mongo01 ~]# systemctl start mongod
[root@mongo01 ~]# systemctl status mongod --no-pager
   Active: active (running)

If mongo01 cannot start, start mongo02 instead. A primary can still be elected, but the writes that only mongo01 had are not in the new primary (see the warning below).

Step 2 – Check that a primary appears. It takes a few seconds. mongo01 has the newest data and priority: 2, so it becomes PRIMARY:

Enterprise rs0 [direct: primary] test> rs.status().members.map(m => m.name + "  " + m.stateStr + "  health=" + m.health)
[
  'mongo01:27017  PRIMARY  health=1',
  'mongo02:27017  (not reachable/healthy)  health=0',
  'mongo03:27017  SECONDARY  health=1'
]

Writes work again with 2 of 3 members.

Step 3 – Bring the third node back. Find the cause on mongo02, start it, and wait for SECONDARY with about 0 seconds of lag:

[root@mongo02 ~]# systemctl start mongod
[root@mongo02 ~]# systemctl status mongod --no-pager
Active: active (running)

On mongo03:

Enterprise rs0 [direct: secondary] test> rs.status().members.map(m => m.name + " " + m.stateStr + " health=" + m.health)
[
'mongo01:27017 PRIMARY health=1',
'mongo02:27017 SECONDARY health=1',
'mongo03:27017 SECONDARY health=1'
]
Enterprise rs0 [direct: secondary] test>

Enterprise rs0 [direct: secondary] test> rs.printSecondaryReplicationInfo()
source: mongo02:27017
{
syncedTo: 'Sun Nov 08 2026 10:35:47 GMT-0500 (Eastern Standard Time)',
replLag: '0 secs (0 hrs) behind the primary '    <---------------- no lag
}
---
source: mongo03:27017
{
syncedTo: 'Sun Nov 08 2026 10:35:47 GMT-0500 (Eastern Standard Time)',
replLag: '0 secs (0 hrs) behind the primary '    <---------------- no lag
}
Enterprise rs0 [direct: secondary] test>

✅ Normal Fix is fixed. One primary and two secondaries again, and no data was lost.

⚠️ If a different node becomes primary first: if mongo02 and mongo03 form the majority without mongo01, the new primary may not have the last writes of mongo01. When mongo01 rejoins, those writes are rolled back and saved as BSON files in the rollback folder inside its dbPath, for example <dbPath>/rollback/<db.collection>/removed.<timestamp>.bson. Review them and re-apply them if needed. This is why you start the old primary first when you can.

💡 Oplog window: a node that was down longer than the oplog window (rs.printReplicationInfo()) cannot catch up and needs a full resync: empty its dbPath and start it again.

All Tasks Completed here. All nodes are healthy. No lag. ALL GOOD.

Do not proceed further until you know clearly.

⚠️ OPTIONAL: DO NOT RUN THIS SECTION (Forced Reconfig).
Use it ONLY IF Section 3 fails AND both nodes in data center 1 (mongo01 and mongo02) are PERMANENTLY LOST. Only then, proceed.
It cannot be undone. There is no rollback.

DATA WILL BE LOST. Every write that mongo03 did not have will be missing from the replica set. The only way to get it back is from the old disks of data center 1, if they still exist (Section 6). After the forced reconfig, mongo03 is the only member left in the replica set (rs0). It becomes PRIMARY and accepts writes on its own until you add the other nodes back.

4. Final Option: Forced Reconfig (Permanent Loss Only)

Use this only when mongo01 and mongo02 and their data are permanently lost. It is the same as a Data Guard failover with data loss.

⚠️ Data loss in this situation: the survivor mongo03 was a secondary, so recent writes that only the lost nodes had are gone, including writes acknowledged with w: "majority" if mongo01 and mongo02 were the ones that acknowledged them. Use the optimeDate from Section 2 (Step 3) to tell the application team from which time the data is missing. Back up the survivor first.

Step 1 – Take a backup of the survivor before you change anything, if you can (for example a mongodump or a file copy while mongod is stopped).

Step 2 – Force the survivor to be a one-member set. On mongo03:

Enterprise rs0 [direct: secondary] test> rs.status().members.map(m => m.name + " " + m.stateStr + " health=" + m.health)
[
'mongo01:27017 (not reachable/healthy) health=0',
'mongo02:27017 (not reachable/healthy) health=0',
'mongo03:27017 SECONDARY health=1'
]
Enterprise rs0 [direct: secondary] test>
Enterprise rs0 [direct: secondary] test> cfg = rs.conf()

Enterprise rs0 [direct: secondary] test> cfg.members = cfg.members.filter(m => m.host == "mongo03:27017")
[
{
_id: 2,
host: 'mongo03:27017',
arbiterOnly: false,
buildIndexes: true,
hidden: false,
priority: 1,
tags: {},
secondaryDelaySecs: Long('0'),
votes: 1
}
]
Enterprise rs0 [direct: secondary] test> rs.reconfig(cfg, { force: true })
{
ok: 1,
'$clusterTime': {
clusterTime: Timestamp({ t: 1794153027, i: 1 }),
signature: {
hash: Binary.createFromBase64('Hh5wob+HHCA/sbvJ7ux7WU0YAjM=', 0),
keyId: Long('7705068198269288454')
}
},
operationTime: Timestamp({ t: 1794153027, i: 1 })
}
Enterprise rs0 [direct: secondary] test>

Step 3 – Check that it became primary. mongo03 now runs alone as a one-member replica set (rs0) and accepts writes:

Enterprise rs0 [direct: secondary] test> rs.status().members.map(m => m.name + " " + m.stateStr + " health=" + m.health)
[ 'mongo03:27017 PRIMARY health=1' ]
Enterprise rs0 [direct: primary] test>

Step 4 – Hold the primary on mongo03 (before you add any node). mongo03 is alone now, so give it the highest priority. Then mongo01 and mongo02 can join, but they can never take the primary role by themselves. Run this on mongo03:

Enterprise rs0 [direct: primary] test> cfg = rs.conf()
Enterprise rs0 [direct: primary] test> cfg.members.find(m => m.host == "mongo03:27017").priority = 3
Enterprise rs0 [direct: primary] test> rs.reconfig(cfg)
{ ok: 1 }

Enterprise rs0 [direct: primary] test> rs.conf().members.map(m => ({ name: m.host, priority: m.priority }))
[ { name: 'mongo03:27017', priority: 3 } ]

💡 Why this step matters: MongoDB moves the primary role by priority (a priority takeover) to any caught-up member with a higher priority than the current primary. It does this automatically within a few seconds, and nobody asks you. With mongo03 at priority 3, the nodes you add (mongo01 with priority 2, mongo02 with priority 1) stay SECONDARY, and you decide when to move the primary (Section 5, Step 7). This is a normal reconfig, not a forced one. mongo01 and mongo02 are not in the configuration any more, so their old priorities are gone. You set them again when you add the nodes in Section 5.

Step 5 – Add the lost servers back. Follow Section 5 to add mongo01 and mongo02 back. Until you do, there is no spare node. If mongo03 fails now, there is no copy of the data.

5. Add the Lost Nodes Back to the Replica Set

After the forced reconfig, mongo03 is the only member left in rs0. It is not a standalone server. It is still a replica set, so you can add mongo01 and mongo02 back with rs.add(), and you do not need to restart mongo03. Each new node does an initial sync: it copies all the data from the primary and then becomes SECONDARY. Add the nodes one at a time, starting with mongo01. Both new nodes are in data center 1, so the first initial sync goes over the link between the data centers and can take a long time. mongo02 may then sync from mongo01 in the same data center.

⚠️ Danger: the new servers must have an empty dbPath. The lost nodes must never come back with their old data, because their old configuration conflicts with the new set. Never use a forced reconfig when the nodes are simply stopped. Empty the dbPath on mongo01 and mongo02 only. Never run rm -rf on mongo03. It is the primary and the only copy of your data. Check the hostname before you run it.

⚠️ Do Section 4, Step 4 first. If mongo03 does not have the highest priority before you add the nodes, a node you add can take the primary role by itself as soon as it has synced.

💡 Keep the backup of the old data. The backup you take in Step 1 is the only place where the last writes (the ones mongo03 did not have) still exist. Do not delete it. After the replica set is rebuilt, you can use it to recover that data in three steps:
1. Restore a copy of the old primary’s backup as a standalone server, on the same server with a different port (for example 27018).
2. Export the missing data from it.
3. Import that data into the new primary.
See Section 6 (optional).

Step 1 – Prepare the new server (mongo01). It needs the same MongoDB version, the same mongod.conf, the same keyfile, and an empty dbPath:

# mongod must be stopped on mongo01 before you touch the data path
[root@mongo01 ~]# systemctl status mongod --no-pager
   Active: inactive (dead)
[root@mongo01 ~]# pgrep -a mongod
[root@mongo01 ~]#

[mongop@mongo01 ~]$ /app/mongodb/7.0.43/bin/mongod --version | head -1
db version v7.0.43
[mongop@mongo01 ~]$ 
[mongop@mongo01 ~]$ egrep -i 'dbPath|replSetName|keyFile|bindIp' /app/mongodb/config/mongod.conf
dbPath: /data/mongodb
bindIp: 127.0.0.1,192.168.2.101 # change per node
keyFile: /app/mongodb/keyfile/mongodb-keyfile
replSetName: rs0
[mongop@mongo01 ~]$

# Take backup # keep the old data, use your own dbPath
[mongop@mongo01 ~]$ cp -rp /data/mongodb /backup/mongodb/mongodb_old_$(date +%F)
[mongop@mongo01 ~]$
[mongop@mongo01 ~]$ ls -ld /backup/mongodb/mongodb_old*
drwx------ 4 mongop mongogrp 4096 Nov 8 10:48 /backup/mongodb/mongodb_old_2026-11-08
[mongop@mongo01 ~]$

# Remove data from Data path carefully (run on mongo01 ONLY, check the hostname first)
[mongop@mongo01 ~]$ hostname
mongo01
[mongop@mongo01 ~]$ rm -rf /data/mongodb/*
[mongop@mongo01 ~]$

[mongop@mongo01 ~]$ ls -la /data/mongodb/
total 0
drwx------. 2 mongop mongogrp 6 Nov 8 10:59 .
drwxr-xr-x. 3 root root 21 Oct 1 02:17 ..
[mongop@mongo01 ~]$

# all names must resolve 
# address hosts, IPv4 only”. (a= address hosts=/etc/hosts v4=IPv4 only)
[mongop@mongo01 ~]$ getent ahostsv4 mongo01 mongo02 mongo03   
192.168.2.101 STREAM mongo01
192.168.2.101 DGRAM
192.168.2.101 RAW
192.168.2.102 STREAM mongo02
192.168.2.102 DGRAM
192.168.2.102 RAW
192.168.3.103 STREAM mongo03
192.168.3.103 DGRAM
192.168.3.103 RAW
[mongop@mongo01 ~]$

• replSetName must be rs0, and bindIp must include the server’s own IP.
• The keyfile must have the same content as on mongo03, permissions 400, and owner mongop.
• Port 27017 must be open between all nodes (check the firewall between the data centers too).
• ahostsv4 means “address hosts, IPv4 only”. All three names must show the IPv4 addresses from the environment table.

Step 2 – Start mongod on mongo01. It starts with empty data and is not yet part of the replica set. This is normal.

[root@mongo01 ~]# systemctl start mongod
[root@mongo01 ~]# systemctl status mongod --no-pager
   Active: active (running)
[root@mongo01 ~]# ss -tlnp | grep 27017
LISTEN 0 128 192.168.2.101:27017 0.0.0.0:* users:(("mongod",pid=6433,fd=16))
LISTEN 0 128 127.0.0.1:27017 0.0.0.0:* users:(("mongod",pid=6433,fd=15))
[root@mongo01 ~]#

Step 3 – Add it from mongo03 (the primary).

Enterprise rs0 [direct: primary] test> rs.add({ host: "mongo01:27017", priority: 2 })
{
ok: 1,
'$clusterTime': {
clusterTime: Timestamp({ t: 1794153755, i: 1 }),
signature: {
hash: Binary.createFromBase64('FyHqGVMG84hZFAiMK+0kotp0JOY=', 0),
keyId: Long('7705068198269288454')
}
},
operationTime: Timestamp({ t: 1794153755, i: 1 })
}
Enterprise rs0 [direct: primary] test> 

The new member gets its original priority: 2. A plain rs.add("mongo01:27017") would give the default priority: 1, because the forced reconfig removed the old setting. mongo03 has priority 3 (Section 4, Step 4), which is higher, so it stays PRIMARY and mongo01 cannot take over by itself.

Step 4 – Watch the initial sync. The new node goes to STARTUP2 while it copies the data, and then becomes SECONDARY:

Enterprise rs0 [direct: primary] test> rs.status().members.map(m => m.name + " " + m.stateStr + " health=" + m.health)
[
'mongo01:27017 STARTUP2 health=1',     <----- initial sync in progress
'mongo03:27017 PRIMARY health=1'
]

Enterprise rs0 [direct: primary] test> rs.status().members.map(m => m.name + " " + m.stateStr + " health=" + m.health)
[
'mongo03:27017 PRIMARY health=1',
'mongo01:27017 SECONDARY health=1'
]
Enterprise rs0 [direct: primary] test>

When it is done, the log on mongo01 shows a line like this:

[root@mongo01 ~]# grep 'Initial sync done' /logs/mongodb/mongod.log
{"t":{"$date":"2026-11-08T11:02:36.907-05:00"},"s":"I", "c":"INITSYNC", "id":21163, "ctx":"ReplCoordExtern-1","msg":"Initial sync done","attr":{"durationSeconds":0}}
[root@mongo01 ~]#

Wait until mongo01 shows SECONDARY and the lag is about 0 seconds. The time depends on the size of the database and the speed of the link between the data centers.

💡 During the initial sync: mongo03 is the only source of the data, so expect extra disk and network load on it. Do this in a quiet window. In MongoDB 5.0 and later, a newly added member is not counted for the majority until its initial sync is done, so writes keep working on mongo03.

Step 5 – Add mongo02. Only after mongo01 is SECONDARY, repeat Steps 1 to 4 for mongo02. Do the same preparation as for mongo01 first: check that mongod is stopped, back up the old data, empty the dbPath, and check the configuration and the names. Start mongod only after that.

# mongod must be stopped on mongo02 before you touch the data path
[root@mongo02 ~]# systemctl status mongod --no-pager
   Active: inactive (dead)
[root@mongo02 ~]# pgrep -a mongod
[root@mongo02 ~]#

[mongop@mongo02 ~]$ /app/mongodb/7.0.43/bin/mongod --version | head -1
db version v7.0.43
[mongop@mongo02 ~]$
[mongop@mongo02 ~]$ egrep -i 'dbPath|replSetName|keyFile|bindIp' /app/mongodb/config/mongod.conf
dbPath: /data/mongodb
bindIp: 127.0.0.1,192.168.2.102 # change per node
keyFile: /app/mongodb/keyfile/mongodb-keyfile
replSetName: rs0
[mongop@mongo02 ~]$

# Take backup # keep the old data, use your own dbPath
[mongop@mongo02 ~]$ cp -rp /data/mongodb /backup/mongodb/mongodb_old_$(date +%F)
[mongop@mongo02 ~]$
[mongop@mongo02 ~]$ ls -ld /backup/mongodb/mongodb_old*
drwx------ 4 mongop mongogrp 4096 Nov 8 11:12 /backup/mongodb/mongodb_old_2026-11-08
[mongop@mongo02 ~]$

# Remove data from Data path carefully (run on mongo02 ONLY, check the hostname first)
[mongop@mongo02 ~]$ hostname
mongo02
[mongop@mongo02 ~]$ rm -rf /data/mongodb/*
[mongop@mongo02 ~]$

[mongop@mongo02 ~]$ ls -la /data/mongodb/
total 0
drwx------. 2 mongop mongogrp 6 Nov 8 11:13 .
drwxr-xr-x. 3 root root 21 Oct 1 02:17 ..
[mongop@mongo02 ~]$

# all names must resolve to IPv4
[mongop@mongo02 ~]$ getent ahostsv4 mongo01 mongo02 mongo03   
192.168.2.101 STREAM mongo01
192.168.2.101 DGRAM
192.168.2.101 RAW
192.168.2.102 STREAM mongo02
192.168.2.102 DGRAM
192.168.2.102 RAW
192.168.3.103 STREAM mongo03
192.168.3.103 DGRAM
192.168.3.103 RAW
[mongop@mongo02 ~]$

Now start mongod on mongo02:

[root@mongo02 ~]# systemctl start mongod
[root@mongo02 ~]# systemctl status mongod --no-pager
   Active: active (running)
[root@mongo02 ~]# ss -tlnp | grep 27017
LISTEN 0 128 192.168.2.102:27017 0.0.0.0:* users:(("mongod",pid=10357,fd=16))
LISTEN 0 128 127.0.0.1:27017 0.0.0.0:* users:(("mongod",pid=10357,fd=15))
[root@mongo02 ~]#

Add it from mongo03 (the primary). mongo02 gets the default priority: 1, which is what you want.

Enterprise rs0 [direct: primary] test> rs.add("mongo02:27017")
{
ok: 1,
'$clusterTime': {
clusterTime: Timestamp({ t: 1794154045, i: 1 }),
signature: {
hash: Binary.createFromBase64('ZjSS/Yi+Fh8cDQLVQNYMcpK5z6s=', 0),
keyId: Long('7705068198269288454')
}
},
operationTime: Timestamp({ t: 1794154045, i: 1 })
}
Enterprise rs0 [direct: primary] test>

Wait until mongo02 shows SECONDARY (check it the same way as in Step 4) before you go to Step 6.

Step 6 – Verify the replica set. All three nodes are healthy, and mongo03 is still the primary:

Enterprise rs0 [direct: primary] test> rs.status().members.map(m => m.name + " " + m.stateStr + " health=" + m.health)
[
'mongo01:27017 SECONDARY health=1',
'mongo02:27017 SECONDARY health=1',
'mongo03:27017 PRIMARY health=1'
]

Enterprise rs0 [direct: primary] test> rs.printSecondaryReplicationInfo()

Enterprise rs0 [direct: primary] test> rs.conf().members.map(m => ({ name: m.host, priority: m.priority }))
[
  { name: 'mongo03:27017', priority: 3 },
  { name: 'mongo01:27017', priority: 2 },
  { name: 'mongo02:27017', priority: 1 }
]

mongo01 shows its original priority: 2. mongo03 in data center 2 is still the primary, because 3 is the highest priority (it is held there on purpose). Nothing moves until you decide.

Step 7 – Move the primary back to data center 1 (only when you are ready). Choose a quiet window. Check that mongo01 and mongo02 are SECONDARY with about 0 seconds of lag (rs.printSecondaryReplicationInfo()). Then run one reconfig on the current primary (mongo03). It lowers mongo03 back to priority 1, so mongo01 (priority 2) becomes the highest:

Enterprise rs0 [direct: primary] test> cfg = rs.conf()
Enterprise rs0 [direct: primary] test> cfg.members.find(m => m.host == "mongo03:27017").priority = 1
Enterprise rs0 [direct: primary] test> rs.reconfig(cfg)
{ ok: 1 }

mongo01 now has the highest priority and is caught up, so it takes the primary role within a few seconds. This is the moment you control: nothing moves until you run this reconfig. Clients see a short interruption. Verify:

Enterprise rs0 [direct: primary] test> rs.status().members.map(m => m.name + " " + m.stateStr)
[
  'mongo01:27017 PRIMARY',
  'mongo02:27017 SECONDARY',
  'mongo03:27017 SECONDARY'
]
Enterprise rs0 [direct: primary] test> rs.conf().members.map(m => ({ name: m.host, priority: m.priority }))
[
  { name: 'mongo01:27017', priority: 2 },
  { name: 'mongo02:27017', priority: 1 },
  { name: 'mongo03:27017', priority: 1 }
]

✅ The replica set is rebuilt with one primary and two secondaries, and mongo01 in data center 1 is the preferred primary again. The data written after the forced reconfig is safe. Data that mongo03 did not have before the failure is lost; see Section 6 for how to try to recover it from the old backup.

💡 Want both data center 1 nodes to be preferred over mongo03? Also set mongo02 to priority: 2 in the same cfg before rs.reconfig(cfg). Then mongo03 only becomes primary if both data center 1 nodes are down.

6. Optional: Get Back the Data That mongo03 Did Not Have

⚠️ When to do this section:
• Only after Section 5 is finished: the replica set is healthy and the primary is back on mongo01.
• Only if you still have the backup of the old data folder from Section 5, Step 1.
• If the old disks were destroyed and there is no backup, skip this section. The missing data cannot be recovered.

💡 Optional: get back the data that mongo03 did not have

What is the problem?
• Data center 1 (mongo01 and mongo02) went down. mongo03 copies data a few seconds after the primary, so it is missing the last few writes.
• Those last writes exist only on the old disks of mongo01 and mongo02.
• In Step 1 you copied the old data folder to a backup before you emptied it. That backup is now the only place where the missing data still lives.
• So do not delete the backup, and do not use it directly.

What is the plan, in simple words?
1. Make a copy of the backup.
2. Start the copy as a small separate MongoDB server (a “standalone”). It is not part of the replica set.
3. Find the data that was written after the time you wrote down in Section 2 (Step 3).
4. Export only that missing data.
5. Import it into the current primary.
6. Check the result and clean up.

Step A – Make a working copy of the backup. Never work on the backup itself. Do this on a spare server, or on mongo01 or mongo02 in a different folder. Do not do it on mongo03, because that is the primary and it is busy. The backup from Section 5, Step 1 is already a copy of the data folder:
Use the backup from mongo01 (the old primary), because it has the newest data. Use the backup from mongo02 only if the one from mongo01 is missing. Check the free space first with df -h /data. You need about as much space as the data folder.

[mongop@mongo01 ~]$ mkdir -p /restore
[mongop@mongo01 ~]$ cp -rp /backup/mongodb/mongodb_old_2026-11-08 /restore/mongodb_copy
[mongop@mongo01 ~]$ ls -l /restore/mongodb_copy

Step B – Start the copy as a standalone server on another port. Use a different port (27018) so it does not clash with the real mongod (27017). Do not use the replica set name. --bind_ip 127.0.0.1 means that only this server can connect to it:

[mongop@mongo01 ~]$ /app/mongodb/7.0.43/bin/mongod --dbpath /restore/mongodb_copy --port 27018 --bind_ip 127.0.0.1 --logpath /restore/mongod_27018.log --fork
[mongop@mongo01 ~]$ /app/mongodb/7.0.43/bin/mongosh --port 27018

It starts without login (no --auth), so you can look at the old data. If it does not start, read /restore/mongod_27018.log.

Step C – Find what is missing. Use the time you wrote down in Section 2 (Step 3), the optimeDate of mongo03. Everything written after that time is missing on mongo03. The old data has an operations log (the oplog). Count the changes after that time for each collection:

test> use local
local> t = Math.floor(ISODate("2026-11-08T15:27:52.000Z").getTime() / 1000)
local> db.oplog.rs.aggregate([
  { $match: { ts: { $gt: Timestamp({ t: t, i: 0 }) }, op: { $in: ["i", "u", "d"] } } },
  { $group: { _id: { namespace: "$ns", operation: "$op" }, count: { $sum: 1 } } }
])
[
  { _id: { namespace: 'mydb.orders', operation: 'i' }, count: 12 },
  { _id: { namespace: 'mydb.orders', operation: 'u' }, count: 3 }
]

How to read it: i = new documents (inserts), u = changed documents (updates), d = deleted documents. In this example, 12 new orders and 3 changes were written after the time and are missing on mongo03. The names mydb.orders are examples. Use your own database and collection names.

Step D – Export only the missing data. For the new documents, export by date. This works if the documents have a date field such as createdAt (use your own field name). Do this for each collection from Step C:
mongodump and mongorestore are part of the MongoDB Database Tools. If the command is not found, use the full path (check it with which mongodump), for example /app/mongodb/7.0.43/bin/mongodump.

[mongop@mongo01 ~]$ mongodump --port 27018 --db mydb --collection orders --query '{ "createdAt": { "$gt": { "$date": "2026-11-08T15:27:52.000Z" } } }' --out /backup/mongodb/missing_data

mongodump keeps the data types exactly. If you prefer a readable text file, use mongoexport with the same --query and import it later with mongoimport.

Step E – Import it into the current primary. Use the replica set address, so the tool always finds the primary (here mongo03, or mongo01 after Step 7). mongorestore adds the documents that do not exist yet and skips the ones that are already there:

[mongop@mongo01 ~]$ mongorestore --uri "mongodb://mongo01:27017,mongo02:27017,mongo03:27017/?replicaSet=rs0" -u admin --authenticationDatabase admin --dir /backup/mongodb/missing_data

Step F – Check and clean up.
• Count the documents on the primary and ask the application team to check that the missing records are back.
• Stop the standalone server: /app/mongodb/7.0.43/bin/mongod --dbpath /restore/mongodb_copy --shutdown
• Keep the original backup until everyone confirms that the data is correct. Only then delete the working copy.

⚠️ Be careful:
• Never use mongorestore --drop here. It deletes the good data on the primary.
• This works well for new documents. Changed or deleted documents (u and d in Step C) cannot be re-applied automatically. Ask the application team to check them by hand.
• Test the steps on a test system first, and tell the application team before you import.
• The standalone server has no login. Keep it on 127.0.0.1 and shut it down when you are done.

7. Quick Reference