Table of Contents

1. Overview and Golden Rules

Environment used in this guide:

Replica set name: rs0. The service is mongod.service, managed with systemctl only. Its stop action runs mongod --shutdown, which is a clean shutdown.

💡 Golden rules:
• Stop secondaries first and the primary last. The primary is the node with the newest data, and it avoids pointless elections while you shut everything down.
• Start the node that was stopped last (the primary) first.
• A 3-member set needs 2 members up to elect a primary.
• Always use a clean stop (systemctl stop mongod). Never use kill -9.
• Use systemctl only. Do not mix it with mongoctl.sh or manual mongod starts.

2. Pre-Checks Before Any Shutdown

Step 1 – Connect and check the replica set health. All members must be healthy before you start maintenance. Otherwise, a planned stop can leave you without a majority.

/app/mongodb/7.0.43/bin/mongosh --host localhost -u admin --authenticationDatabase admin

Enter password: ******
Using MongoDB:          7.0.43
Using Mongosh:          2.12.0

Enterprise rs0 [primary] test>

Enterprise rs0 [primary] test> rs.status().members.map(m => m.name + " " + m.stateStr + " health=" + m.health)
[
'mongo01:27017 PRIMARY health=1',  <----------
'mongo02:27017 SECONDARY health=1',
'mongo03:27017 SECONDARY health=1'
]

Step 2 – Check replication lag. Each secondary should be at or close to 0 seconds behind the primary.

Enterprise rs0 [direct: primary] test> rs.printSecondaryReplicationInfo()
source: mongo02:27017
{
syncedTo: 'Sun Nov 08 2026 01:38:41 GMT-0500 (Eastern Standard Time)',
replLag: '0 secs (0 hrs) behind the primary '   <---------------
}
---
source: mongo03:27017
{
syncedTo: 'Sun Nov 08 2026 01:38:41 GMT-0500 (Eastern Standard Time)',
replLag: '0 secs (0 hrs) behind the primary '  <------------------
}

Step 3 – Check the oplog window. This is how long a stopped member can stay down and still catch up without a full resync.

Enterprise rs0 [direct: primary] test> rs.printReplicationInfo()
actual oplog size
'10240 MB'
---
configured oplog size
'10240 MB'
---
log length start to end
'144034 secs (40.01 hrs)'
---
oplog first event time
'Fri Nov 06 2026 09:39:37 GMT-0500 (Eastern Standard Time)'
---
oplog last event time
'Sun Nov 08 2026 01:40:11 GMT-0500 (Eastern Standard Time)'
---
now
'Sun Nov 08 2026 01:40:12 GMT-0500 (Eastern Standard Time)'

• Oplog size: 10 GB. The configured and actual sizes match, so this is fine.
• Log length (40 hours): This is the replication window, meaning how long a secondary can be offline and still catch up by replaying the oplog.
• Oplog first event: Fri Nov 06, 09:39. Any secondary whose last synced operation is older than this cannot catch up and needs a full initial sync.
• Oplog last event: Sun Nov 08, 01:40. This is essentially “now”, which shows the primary is actively writing.

Note: the window is not fixed. A heavier write load shortens it, so 40 hours today does not guarantee 40 hours tomorrow.

Step 4 – (Optional) Check long-running operations that a shutdown would interrupt:

Enterprise rs0 [direct: primary] test> db.currentOp({ active: true, secs_running: { $gt: 5 } }).inprog.length
5
Enterprise rs0 [direct: primary] test>

-- 5 operations have been actively running for more than 5 seconds, run below command to find out and then inform to respective team.

db.currentOp({ active: true, secs_running: { $gt: 5 } }).inprog.map(op => ({
opid: op.opid,
op: op.op,
ns: op.ns,
secs: op.secs_running,
client: op.client,
desc: op.desc,
msg: op.msg
}))

⚠️ Before maintenance: take a backup if the work is risky (OS patching, upgrades, storage changes). Tell the application team about the window, because a primary change is a short interruption for clients that do not use retryable writes.

3. Planned Shutdown of the Whole Replica Set

Use this when all three servers must be down, for example for OS patching or a full reboot.

   Order of shutdown:

   1. mongo03 (secondary)  --> stop
   2. mongo02 (secondary)  --> stop
   3. mongo01 (primary)    --> stop LAST

On mongo03

Step 1 – Stop the first secondary (mongo03).

[root@mongo03 ~]# systemctl stop mongod
[root@mongo03 ~]# systemctl status mongod --no-pager
● mongod.service - MongoDB Enterprise Database Server
   Loaded: loaded (/etc/systemd/system/mongod.service; enabled; vendor preset: disabled)
   Active: inactive (dead) since Sun 2026-11-08 02:01:36 EST; 1min 28s ago
     Docs: https://docs.mongodb.com/manual
  Process: 4525 ExecStop=/app/mongodb/7.0.43/bin/mongod --config /app/mongodb/config/mongod.conf --shutdown (code=exited, status=0/SUCCESS)
  Process: 3996 ExecStart=/app/mongodb/7.0.43/bin/mongod --config /app/mongodb/config/mongod.conf (code=exited, status=0/SUCCESS)
 Main PID: 4009 (code=exited, status=0/SUCCESS)

Nov 08 01:22:28 mongo03 mongod[3996]: {"t":{"$date":"2026-11-08T06:22:28.132Z"},"s":"I",  "c":"CONTROL",  "id":5760901, "ctx":"main","msg":"Applied --setParameter options","attr":{"serv…"value":true}}}}
Nov 08 01:22:28 mongo03 mongod[3996]: about to fork child process, waiting until server is ready for connections.
Nov 08 01:22:28 mongo03 mongod[4009]: forked process: 4009
Nov 08 01:22:29 mongo03 mongod[3996]: child process started successfully, parent exiting
Nov 08 01:22:29 mongo03 systemd[1]: Started MongoDB Enterprise Database Server.
Nov 08 02:01:20 mongo03 systemd[1]: Stopping MongoDB Enterprise Database Server...
Nov 08 02:01:20 mongo03 mongod[4525]: {"t":{"$date":"2026-11-08T07:01:20.229Z"},"s":"I",  "c":"CONTROL",  "id":5760901, "ctx":"main","msg":"Applied --setParameter options","attr":{"serv…"value":true}}}}
Nov 08 02:01:20 mongo03 mongod[4525]: Killing process with pid: 4009
Nov 08 02:01:36 mongo03 systemd[1]: mongod.service: Succeeded.
Nov 08 02:01:36 mongo03 systemd[1]: Stopped MongoDB Enterprise Database Server.
Hint: Some lines were ellipsized, use -l to show in full.
[root@mongo03 ~]#

[root@mongo03 ~]# pgrep -a mongod
[root@mongo03 ~]# 

No output from pgrep means the process is gone. Check the log for a clean exit:

[root@mongo03 ~]# egrep -i 'shutting down|now exiting|shutdown' /logs/mongodb/mongod.log | tail -5

{"t":{"$date":"2026-11-08T02:01:36.040-05:00"},"s":"I",  "c":"FTDC",     "id":20626,   "ctx":"SignalHandler","msg":"Shutting down full-time diagnostic data capture"}
{"t":{"$date":"2026-11-08T02:01:36.044-05:00"},"s":"I",  "c":"-",        "id":10175800,"ctx":"SignalHandler","msg":"Shutting down the standalone executor"}
{"t":{"$date":"2026-11-08T02:01:36.045-05:00"},"s":"I",  "c":"CONTROL",  "id":20565,   "ctx":"SignalHandler","msg":"Now exiting"}
{"t":{"$date":"2026-11-08T02:01:36.045-05:00"},"s":"I",  "c":"CONTROL",  "id":8423404, "ctx":"SignalHandler","msg":"mongod shutdown complete","attr":{"Summary of time elapsed":{"Statistics":{"Enter terminal shutdown":"0 ms","Step down the replication coordinator for shutdown":"1 ms","Time spent in quiesce mode":"15002 ms","Shut down FLE Crud subsystem":"1 ms","Shut down MirrorMaestro":"0 ms","Shut down WaitForMajorityService":"2 ms","Shut down the logical session cache":"1 ms","Shut down the Query Analysis Sampler":"0 ms","Shut down the transport layer":"24 ms","Shut down the global connection pool":"0 ms","Shut down the flow control ticket holder":"0 ms","Shut down the replica set node executor":"0 ms","Shut down the replica set aware services":"1 ms","Shut down replication":"0 ms","Shut down external state":"532 ms","Shut down replication executor":"0 ms","Join replication executor":"3 ms","Kill all operations for shutdown":"1 ms","Shut down all tenant migration access blockers on global shutdown":"1 ms","Shut down all open transactions":"0 ms","Acquire the RSTL for shutdown":"1 ms","Shut down the IndexBuildsCoordinator and wait for index builds to finish":"0 ms","Shut down the replica set monitor":"3 ms","Shut down the logical time validator":"0 ms","Shut down the migration util executor":"1 ms","Shut down the health log":"0 ms","Shut down the TTL monitor":"1 ms","Shut down expired pre-images and documents removers":"0 ms","Shut down the storage engine":"53 ms","Wait for the oplog cap maintainer thread to stop":"0 ms","Shut down full-time data capture":"0 ms","Shut down standalone executor":"4 ms","shutdownTask total elapsed time":"15637 ms"}}}}
{"t":{"$date":"2026-11-08T02:01:36.046-05:00"},"s":"I",  "c":"CONTROL",  "id":23138,   "ctx":"SignalHandler","msg":"Shutting down","attr":{"exitCode":0}}
[root@mongo03 ~]#

Step 2 – Confirm the remaining members, then stop the second secondary (mongo02).

From the primary:

Enterprise rs0 [direct: primary] test> rs.status().members.map(m => ({ name: m.name, state: m.stateStr, health: m.health }))
[
{ name: 'mongo01:27017', state: 'PRIMARY', health: 1 },
{ name: 'mongo02:27017', state: 'SECONDARY', health: 1 },
{
name: 'mongo03:27017',
state: '(not reachable/healthy)',   <------------
health: 0
}
]
Enterprise rs0 [direct: primary] test> 

On mongo02

[root@mongo02 ~]# systemctl stop mongod
[root@mongo02 ~]# pgrep -a mongod
[root@mongo02 ~]#

⚠️ Expected behaviour: once mongo02 is down, mongo01 is alone and has lost its majority (1 of 3). It steps down and becomes a SECONDARY, so writes are rejected. This is normal during a full shutdown. Do not try to fix it.

Step 3 – Stop the primary last (mongo01).

Enterprise rs0 [direct: secondary] test> rs.status().members.map(m => ({ name: m.name, state: m.stateStr, health: m.health }))
[
{ name: 'mongo01:27017', state: 'SECONDARY', health: 1 },  <------------- It steps down and becomes a SECONDARY, so writes are rejected. This is normal during a full shutdown { name: 'mongo02:27017', state: '(not reachable/healthy)', <-------------------- health: 0 }, { name: 'mongo03:27017', state: '(not reachable/healthy)', <-------------------- health: 0 } ] Enterprise rs0 [direct: secondary] test> [root@mongo01 ~]# systemctl stop mongod [root@mongo01 ~]# pgrep -a mongod [root@mongo01 ~]# 

Step 4 – Do the maintenance or reboot. If you reboot, mongod starts automatically at boot because the service is enabled. Check this on each server:

[root@mongo01 ~]# systemctl is-enabled mongod
enabled   <----------------------------
[root@mongo01 ~]#

⚠️ Note about reboots: if all three servers boot at the same time, the replica set forms by itself once two members are up. If you want a controlled order, shut the servers down in the order above, then power them on starting with mongo01.

4. Start the Whole Replica Set

   Order of startup:

   1. mongo01 (previous primary) --> start FIRST
   2. mongo02                    --> start
   3. mongo03                    --> start

Step 1 – Start mongo01 and check that it is running.

[root@mongo01 ~]# systemctl start mongod
[root@mongo01 ~]# systemctl status mongod --no-pager
● mongod.service - MongoDB Enterprise Database Server
   Loaded: loaded (/etc/systemd/system/mongod.service; enabled; vendor preset: disabled)
   Active: active (running) since Sun 2026-11-08 02:18:55 EST; 9s ago   <----------------
     Docs: https://docs.mongodb.com/manual 
  Process: 5112 ExecStop=/app/mongodb/7.0.43/bin/mongod --config /app/mongodb/config/mongod.conf --shutdown (code=exited, status=0/SUCCESS)
  Process: 5142 ExecStart=/app/mongodb/7.0.43/bin/mongod --config /app/mongodb/config/mongod.conf (code=exited, status=0/SUCCESS)
 Main PID: 5144 (mongod)
    Tasks: 72 (limit: 17684)
   Memory: 178.3M
   CGroup: /system.slice/mongod.service
           └─5144 /app/mongodb/7.0.43/bin/mongod --config /app/mongodb/config/mongod.conf

Nov 08 02:18:53 mongo01 systemd[1]: Starting MongoDB Enterprise Database Server...
Nov 08 02:18:53 mongo01 mongod[5142]: {"t":{"$date":"2026-11-08T07:18:53.406Z"},"s":"I",  "c":"CONTROL",  "id":5760901, "ctx":"main","msg":"Applied --setParameter options","attr":{"serv…"value":true}}}}
Nov 08 02:18:53 mongo01 mongod[5142]: about to fork child process, waiting until server is ready for connections.
Nov 08 02:18:53 mongo01 mongod[5144]: forked process: 5144
Nov 08 02:18:55 mongo01 mongod[5142]: child process started successfully, parent exiting
Nov 08 02:18:55 mongo01 systemd[1]: Started MongoDB Enterprise Database Server.
Hint: Some lines were ellipsized, use -l to show in full.
[root@mongo01 ~]#
[root@mongo01 ~]# ss -tlnp | grep 27017
LISTEN 0      128    192.168.2.101:27017      0.0.0.0:*    users:(("mongod",pid=5144,fd=16))
LISTEN 0      128        127.0.0.1:27017      0.0.0.0:*    users:(("mongod",pid=5144,fd=15))
[root@mongo01 ~]#

At this point mongo01 is up but there is no primary, because 1 of 3 members is not a majority. It shows as SECONDARY, and this is expected.

Step 2 – Start mongo02. As soon as two members are up, they hold an election.

[root@mongo02 ~]# systemctl start mongod
[root@mongo02 ~]# systemctl status mongod --no-pager
● mongod.service - MongoDB Enterprise Database Server
   Loaded: loaded (/etc/systemd/system/mongod.service; enabled; vendor preset: disabled)
   Active: active (running) since Sun 2026-11-08 02:21:41 EST; 5s ago  <----------------
     Docs: https://docs.mongodb.com/manual
  Process: 4486 ExecStop=/app/mongodb/7.0.43/bin/mongod --config /app/mongodb/config/mongod.conf --shutdown (code=exited, status=0/SUCCESS)
  Process: 4655 ExecStart=/app/mongodb/7.0.43/bin/mongod --config /app/mongodb/config/mongod.conf (code=exited, status=0/SUCCESS)
 Main PID: 4657 (mongod)
    Tasks: 79 (limit: 17684)
   Memory: 186.4M
   CGroup: /system.slice/mongod.service
           └─4657 /app/mongodb/7.0.43/bin/mongod --config /app/mongodb/config/mongod.conf

Nov 08 02:21:39 mongo02 systemd[1]: Starting MongoDB Enterprise Database Server...
Nov 08 02:21:40 mongo02 mongod[4655]: {"t":{"$date":"2026-11-08T07:21:40.023Z"},"s":"I",  "c":"CONTROL",  "id":5760901, "ctx":"main","msg":"Applied --setParameter options","attr":{"serv…"value":true}}}}
Nov 08 02:21:40 mongo02 mongod[4655]: about to fork child process, waiting until server is ready for connections.
Nov 08 02:21:40 mongo02 mongod[4657]: forked process: 4657
Nov 08 02:21:41 mongo02 mongod[4655]: child process started successfully, parent exiting
Nov 08 02:21:41 mongo02 systemd[1]: Started MongoDB Enterprise Database Server.
Hint: Some lines were ellipsized, use -l to show in full.
[root@mongo02 ~]# ss -tlnp | grep 27017
LISTEN 0      128    192.168.2.102:27017      0.0.0.0:*    users:(("mongod",pid=4657,fd=16))
LISTEN 0      128        127.0.0.1:27017      0.0.0.0:*    users:(("mongod",pid=4657,fd=15))
[root@mongo02 ~]#

Step 3 – Start mongo03.

[root@mongo03 ~]# systemctl start mongod
[root@mongo03 ~]# systemctl status mongod --no-pager
● mongod.service - MongoDB Enterprise Database Server
   Loaded: loaded (/etc/systemd/system/mongod.service; enabled; vendor preset: disabled)
   Active: active (running) since Sun 2026-11-08 02:22:54 EST; 3s ago  <----------------
     Docs: https://docs.mongodb.com/manual
  Process: 4525 ExecStop=/app/mongodb/7.0.43/bin/mongod --config /app/mongodb/config/mongod.conf --shutdown (code=exited, status=0/SUCCESS)
  Process: 4716 ExecStart=/app/mongodb/7.0.43/bin/mongod --config /app/mongodb/config/mongod.conf (code=exited, status=0/SUCCESS)
 Main PID: 4718 (mongod)
    Tasks: 89 (limit: 17684)
   Memory: 195.0M
   CGroup: /system.slice/mongod.service
           └─4718 /app/mongodb/7.0.43/bin/mongod --config /app/mongodb/config/mongod.conf

Nov 08 02:22:52 mongo03 systemd[1]: Starting MongoDB Enterprise Database Server...
Nov 08 02:22:52 mongo03 mongod[4716]: {"t":{"$date":"2026-11-08T07:22:52.353Z"},"s":"I",  "c":"CONTROL",  "id":5760901, "ctx":"main","msg":"Applied --setParameter options","attr":{"serv…"value":true}}}}
Nov 08 02:22:52 mongo03 mongod[4716]: about to fork child process, waiting until server is ready for connections.
Nov 08 02:22:52 mongo03 mongod[4718]: forked process: 4718
Nov 08 02:22:54 mongo03 mongod[4716]: child process started successfully, parent exiting
Nov 08 02:22:54 mongo03 systemd[1]: Started MongoDB Enterprise Database Server.
Hint: Some lines were ellipsized, use -l to show in full.
[root@mongo03 ~]# ss -tlnp | grep 27017
LISTEN 0      128    192.168.3.103:27017      0.0.0.0:*    users:(("mongod",pid=4718,fd=16))
LISTEN 0      128        127.0.0.1:27017      0.0.0.0:*    users:(("mongod",pid=4718,fd=15))
[root@mongo03 ~]#

Step 4 – Verify. Connect to any member and check the state. Members can show STARTUP2 or RECOVERING for a short time while they catch up on the oplog.

[mongop@mongo01 ~]$ /app/mongodb/7.0.43/bin/mongosh --host localhost -u admin --authenticationDatabase admin
Enter password: ******
Current Mongosh Log ID: 6af023ae5b33cb95796e5862
Connecting to:          mongodb://@localhost:27017/?directConnection=true&serverSelectionTimeoutMS=2000&authSource=admin&appName=mongosh+2.12.0
Using MongoDB:          7.0.43
Using Mongosh:          2.12.0
mongosh 2.14.1 is available for download: https://www.mongodb.com/try/download/shell

For mongosh info see: https://www.mongodb.com/docs/mongodb-shell/

Enterprise rs0 [direct: primary] test> rs.status().members.map(m => ({ name: m.name, state: m.stateStr, health: m.health }))
[
  { name: 'mongo01:27017', state: 'PRIMARY', health: 1 },   <------------ it's primary, beacuse it has priority 2
  { name: 'mongo02:27017', state: 'SECONDARY', health: 1 },
  { name: 'mongo03:27017', state: 'SECONDARY', health: 1 }
]
Enterprise rs0 [direct: primary] test>

Enterprise rs0 [direct: primary] test> rs.conf().members.map(m => ({ name: m.host, priority: m.priority, votes: m.votes }))
[
{ name: 'mongo01:27017', priority: 2, votes: 1 },
{ name: 'mongo02:27017', priority: 1, votes: 1 },
{ name: 'mongo03:27017', priority: 1, votes: 1 }
]
Enterprise rs0 [direct: primary] test> 

💡 Why mongo01 becomes primary again: it has priority: 2, the highest in the set. If another member is elected first, mongo01 calls an election by itself once it has caught up, and takes the primary role back. Expect one short role change.

✅ The replica set is back with one primary and two secondaries.

5. Quick Reference

💡 Tip: keep a shell open on a node that stays up during the work, and re-run the member status command after each step. It is the quickest way to see where the set is.

6. Common Mistakes and Troubleshooting

• Stopping the primary first in a full shutdown. It triggers a needless election while the other nodes are also going down. Stop the secondaries first.
• Starting the secondaries before the old primary after a full shutdown. Start mongo01 first.
• Using kill -9 or powering off instead of a clean stop. The node has to run crash recovery at the next start.
• Mixing mongoctl.sh, manual mongod commands and systemctl. Use systemctl only.
• Leaving a member down longer than the oplog window. It cannot catch up and needs a full resync. Check rs.printReplicationInfo() before long maintenance.