Tag Archives: Database Architecture

How PostgreSQL Replay Uncommitted Data During Crash Recovery

How PostgreSQL Replay Uncommitted Data During Crash Recovery

  1. When PostgreSQL crashes, everything in memory is lost but the important files on disk survive β€” WAL files (transaction history), data files (actual tables), CLOG (who committed or not) andΒ pg_controlΒ (database status file).
  2. When PostgreSQL restarts, theΒ PostmasterΒ (the main process) first readsΒ pg_controlΒ file β€” if it sees the database was not shut down cleanly, it understands a crash happened and starts theΒ Startup ProcessΒ to fix the database.
  3. TheΒ Startup ProcessΒ readsΒ pg_controlΒ to find the last checkpoint position (think of it as the last known good save point) and goes to that exact position in the WAL files to start reading from there.
  4. TheΒ Startup ProcessΒ reads WAL files one record at a time β€” for each record, it finds the related data page, loads it into memory (Shared Buffers), applies the change to that page in memory and marks it as dirty (modified but not yet written to disk).
  5. TheΒ Startup ProcessΒ applies every single WAL record to memory pages β€” it does not care whether the transaction was committed or not β€” both committed and uncommitted changes are loaded into Shared Buffers because the WAL replay never checks CLOG at this stage.
  6. Before applying any change to a page, theΒ Startup ProcessΒ checks if that page was already updated before the crash (by checkingΒ pd_lsnΒ on the page) β€” if the page is already up to date it simply skips that WAL record and moves to the next one β€” this makes replay safe to run multiple times.
  7. When theΒ Startup ProcessΒ sees aΒ COMMITΒ orΒ ABORTΒ record in the WAL, it immediately updates the CLOG file β€” marking that transaction asΒ COMMITTEDΒ orΒ ABORTEDΒ so PostgreSQL knows the final outcome of that transaction.
  8. When theΒ Startup ProcessΒ reaches the end of WAL files (the crash point), any transaction that had noΒ COMMITΒ orΒ ABORTΒ record is considered as never completed β€” the Startup ProcessΒ explicitly marks all those transactions asΒ ABORTEDΒ in CLOG right away before allowing any user to connect β€” this is why you always seeΒ ABORTEDΒ and neverΒ IN_PROGRESSΒ after a crash recovery.
  9. Once WAL replay is done, a final checkpoint runs β€”Β BGWriterΒ andΒ CheckpointerΒ take all the dirty pages sitting in Shared Buffers (including pages with uncommitted data) and write them permanently to the actual database files on disk β€” thenΒ pg_controlΒ is updated to say the database is healthy and users are allowed to connect.
  10. Even though uncommitted data physically exists in the database files on disk, no user will ever see it β€” because CLOG says those transactions areΒ ABORTED, MVCC automatically hides those rows from every query β€” and laterΒ VACUUMΒ comes along and physically cleans up those dead rows from the pages and frees up the space.
Caution: Your use of any information or materials on this website is entirely at your own risk. It is provided for educational purposes only. It has been tested internally, however, we do not guarantee that it will work for you. Ensure that you run it in your test environment before using.
Thank you
Rajasekhar Amudala