Table of Contents

1. Overview and Golden Rules

Environment used in this guide:

Replica set name: rs0. The service is mongod.service, managed with systemctl only. Its stop action runs mongod --shutdown, which is a clean shutdown.

💡 Golden rules:
• Work on one node at a time. Do not move on until the previous node is back to SECONDARY with about 0 seconds of lag.
• The primary is done last, and only after rs.stepDown().
• A 3-member set needs 2 members up to elect a primary. Never have two members down at the same time while the application must stay online.
• Always use a clean stop (systemctl stop mongod). Never use kill -9.
• Use systemctl only. Do not mix it with mongoctl.sh or manual mongod starts.

2. Pre-Checks Before Any Shutdown

Step 1 – Connect and check the replica set health. All members must be healthy before you start maintenance. Otherwise, a planned stop can leave you without a majority.

/app/mongodb/7.0.43/bin/mongosh --host localhost -u admin --authenticationDatabase admin

Enter password: ******
Using MongoDB:          7.0.43
Using Mongosh:          2.12.0

Enterprise rs0 [primary] test>

Enterprise rs0 [primary] test> rs.status().members.map(m => m.name + " " + m.stateStr + " health=" + m.health)
[
'mongo01:27017 PRIMARY health=1',  <----------
'mongo02:27017 SECONDARY health=1',
'mongo03:27017 SECONDARY health=1'
]

Step 2 – Check replication lag. Each secondary should be at or close to 0 seconds behind the primary.

Enterprise rs0 [direct: primary] test> rs.printSecondaryReplicationInfo()
source: mongo02:27017
{
syncedTo: 'Sun Nov 08 2026 01:38:41 GMT-0500 (Eastern Standard Time)',
replLag: '0 secs (0 hrs) behind the primary '   <---------------
}
---
source: mongo03:27017
{
syncedTo: 'Sun Nov 08 2026 01:38:41 GMT-0500 (Eastern Standard Time)',
replLag: '0 secs (0 hrs) behind the primary '  <------------------
}

Step 3 – Check the oplog window. This is how long a stopped member can stay down and still catch up without a full resync.

Enterprise rs0 [direct: primary] test> rs.printReplicationInfo()
actual oplog size
'10240 MB'
---
configured oplog size
'10240 MB'
---
log length start to end
'144034 secs (40.01 hrs)'
---
oplog first event time
'Fri Nov 06 2026 09:39:37 GMT-0500 (Eastern Standard Time)'
---
oplog last event time
'Sun Nov 08 2026 01:40:11 GMT-0500 (Eastern Standard Time)'
---
now
'Sun Nov 08 2026 01:40:12 GMT-0500 (Eastern Standard Time)'

• Oplog size: 10 GB. The configured and actual sizes match, so this is fine.
• Log length (40 hours): This is the replication window, meaning how long a secondary can be offline and still catch up by replaying the oplog.
• Oplog first event: Fri Nov 06, 09:39. Any secondary whose last synced operation is older than this cannot catch up and needs a full initial sync.
• Oplog last event: Sun Nov 08, 01:40. This is essentially “now”, which shows the primary is actively writing.

Note: the window is not fixed. A heavier write load shortens it, so 40 hours today does not guarantee 40 hours tomorrow.

Step 4 – (Optional) Check long-running operations that a shutdown would interrupt:

Enterprise rs0 [primary] test> db.currentOp({ active: true, secs_running: { $gt: 5 } }).inprog.length
3
Enterprise rs0 [primary] test>

-- 3 operations have been actively running for more than 5 seconds, run below command to find out and then inform to respective team.

db.currentOp({ active: true, secs_running: { $gt: 5 } }).inprog.map(op => ({
opid: op.opid,
op: op.op,
ns: op.ns,
secs: op.secs_running,
client: op.client,
desc: op.desc,
msg: op.msg
}))

⚠️ Before maintenance: take a backup if the work is risky (OS patching, upgrades, storage changes). Tell the application team about the window, because a primary change is a short interruption for clients that do not use retryable writes.

3. Rolling Restart (No Downtime)

Use this for patching or configuration changes while the application keeps running. Work on one node at a time. The primary is done last.

   1. mongo03: stop, patch, start, wait until SECONDARY
   2. mongo02: stop, patch, start, wait until SECONDARY
   3. mongo01: rs.stepDown(), wait for new primary, stop, patch, start

Step 1 – Secondary mongo03.

[root@mongo03 ~]# systemctl stop mongod
# ... patch, reboot, or change configuration ...#... maintenance ...
[root@mongo03 ~]# systemctl start mongod

Wait until it is back and caught up (run on the primary):

Enterprise rs0 [primary] test> rs.status().members.map(m => m.name + " " + m.stateStr + " health=" + m.health)
[
'mongo01:27017 PRIMARY health=1',
'mongo02:27017 SECONDARY health=1',
'mongo03:27017 SECONDARY health=1'
]
Enterprise rs0 [primary] test>
Enterprise rs0 [primary] test> rs.printSecondaryReplicationInfo()
source: mongo02:27017
{
syncedTo: 'Sun Nov 08 2026 02:36:51 GMT-0500 (Eastern Standard Time)',
replLag: '0 secs (0 hrs) behind the primary '
}
---
source: mongo03:27017
{
syncedTo: 'Sun Nov 08 2026 02:36:51 GMT-0500 (Eastern Standard Time)',
replLag: '0 secs (0 hrs) behind the primary '            <--------------------------- 0 seconds lag
}
Enterprise rs0 [primary] test>

Only continue with next step when mongo03 shows SECONDARY with about 0 seconds of lag.

Step 2 – Secondary mongo02. Same steps as above:

[root@mongo02 ~]# systemctl stop mongod
# ... patch, reboot, or change configuration ...#... maintenance ...
[root@mongo02 ~]# systemctl start mongod

Wait for SECONDARY and low lag again. Only continue with next step when mongo02 shows SECONDARY with about 0 seconds of lag.

Wait until it is back and caught up (run on the primary):

Enterprise rs0 [primary] test> rs.status().members.map(m => m.name + " " + m.stateStr + " health=" + m.health)
[
'mongo01:27017 PRIMARY health=1',
'mongo02:27017 SECONDARY health=1',
'mongo03:27017 SECONDARY health=1'
]
Enterprise rs0 [primary] test>
Enterprise rs0 [primary] test> rs.printSecondaryReplicationInfo()
source: mongo02:27017
{
syncedTo: 'Sun Nov 08 2026 02:44:11 GMT-0500 (Eastern Standard Time)',
replLag: '0 secs (0 hrs) behind the primary '     <---------------------  0 seconds lag
}
---
source: mongo03:27017
{
syncedTo: 'Sun Nov 08 2026 02:44:11 GMT-0500 (Eastern Standard Time)',
replLag: '0 secs (0 hrs) behind the primary '     <---------------------  0 seconds lag
}
Enterprise rs0 [primary] test>

Step 3 – Step down the primary (mongo01). Connect to mongo01 and run:

# rs.stepDown(120) tells the current primary to give up its role and not try to become primary again for 120 seconds. Enterprise rs0 [primary] test> rs.stepDown(120) { ok: 1, '$clusterTime': { clusterTime: Timestamp({ t: 1794123961, i: 1 }), signature: { hash: Binary.createFromBase64('Ntv06PS5aoP6m8pv996GtHO6n0s=', 0), keyId: Long('7705068198269288454') } }, operationTime: Timestamp({ t: 1794123961, i: 1 }) } Enterprise rs0 [secondary] test> 

The number is the time in seconds during which mongo01 will not try to become primary again. The shell connection drops and reconnects, which is normal. Confirm a new primary before you continue:

Enterprise rs0 [direct: secondary] test> rs.status().members.map(m => m.name + " " + m.stateStr + " health=" + m.health)
[
'mongo01:27017 SECONDARY health=1',
'mongo02:27017 PRIMARY health=1',   <---- Now mongo02 became primary
'mongo03:27017 SECONDARY health=1'
]Enterprise rs0 [direct: secondary] test>

Step 4 – Stop mongo01, do the maintenance, and start it.

[root@mongo01 ~]# systemctl stop mongod
# ... patch, reboot, or change configuration ...#... maintenance ...
[root@mongo01 ~]# systemctl start mongod

[mongop@mongo01 ~]$ /app/mongodb/7.0.43/bin/mongosh --host localhost -u admin --authenticationDatabase admin
Enter password: ******
Current Mongosh Log ID: 6af02b2f9daf846cbd2d3806
Connecting to:          mongodb://@localhost:27017/?directConnection=true&serverSelectionTimeoutMS=2000&authSource=admin&appName=mongosh+2.12.0
Using MongoDB:          7.0.43
Using Mongosh:          2.12.0
mongosh 2.14.1 is available for download: https://www.mongodb.com/try/download/shell

For mongosh info see: https://www.mongodb.com/docs/mongodb-shell/

Enterprise rs0 [direct: primary] test> rs.status().members.map(m => m.name + " " + m.stateStr + " health=" + m.health)
[
  'mongo01:27017 PRIMARY health=1',    <--------------- mongo01 became primary because it has priority 2
  'mongo02:27017 SECONDARY health=1',
  'mongo03:27017 SECONDARY health=1'
]
Enterprise rs0 [direct: primary] test>

Enterprise rs0 [direct: primary] test> rs.conf().members.map(m => ({ name: m.host, priority: m.priority, votes: m.votes }))
[
{ name: 'mongo01:27017', priority: 2, votes: 1 },
{ name: 'mongo02:27017', priority: 1, votes: 1 },
{ name: 'mongo03:27017', priority: 1, votes: 1 }
]
Enterprise rs0 [direct: primary] test>

When mongo01 has caught up, it takes the primary role back because of its higher priority (priority: 2).

⚠️ Warning: never take down the second node while the first is still recovering. With two of three members down there is no majority and no primary, and the application loses write access.

✅ All three nodes are restarted and the application stayed online.

4. Quick Reference

💡 Tip: keep a shell open on a node that stays up during the work, and re-run the member status command after each step. It is the quickest way to see where the set is.

5. Common Mistakes and Troubleshooting

• Stopping two nodes at the same time while the application must stay online. You lose the majority and there is no primary.
• Starting work on the next node before the previous one is back to SECONDARY.
• Stopping the primary without rs.stepDown() first. Clients see a longer interruption while the election runs.
• Using kill -9 or powering off instead of a clean stop. The node has to run crash recovery at the next start.
• Leaving a member down longer than the oplog window. It cannot catch up and needs a full resync. Check rs.printReplicationInfo() before long maintenance.