Table of Contents

This article shows how storage on an Exadata storage server is built up, layer by layer: physical disk, LUN, cell disk, grid disk, ASM disk and finally an ASM disk group. Each step has the command to run, the command to verify it, and what you should see.

1. The Big Picture

About the “partition” step: on Exadata you do not partition a disk manually. The LUN is created automatically, and the grid disk is the partition: a cell disk is divided into one or more grid disks. (On X6 and earlier, the first two hard disks also keep a small system area for the operating system. From X7 onwards that system area is on separate M.2 devices.)

On a new Exadata, the installer (OEDA) does all of these steps for you. You do them yourself when you add a new storage server to an existing system, or when you build a lab.

2. Before You Start

WARNING: These commands create and drop storage objects. Run them only on a new, empty storage server or in a lab. Never run them on a cell that already holds database data.

Run the first three checks on the storage server.

• The cell has its IP addresses configured and the cell services are running.
• All physical disks show normal.
• You know the disk group names and the size of each grid disk you want (for example DATAC1 and RECOC1).
• If you are adding the cell to existing disk groups, you have the exact grid disk sizes from an existing cell (see Step 4).

Naming used in this article: cell name cell01, grid disk prefixes DATAC1 and RECOC1, and cell IP <cell_ip>. Replace them with your own values.

3. Step 1: Check the Physical Disks

Physical disks are the hard disks and flash devices inside the storage server. Note which ones are HardDisk and which are FlashDisk.

Result: every disk is listed with status = normal.

4. Step 2: Check the LUNs

The Exadata System Software creates one LUN for each physical disk automatically. You do not create LUNs. Just check that they exist and are normal.

Result: each LUN is normal. The cellDisk column is empty until you create the cell disks in the next step.

5. Step 3: Create the Cell Disks

A cell disk is the storage object the cell builds on a LUN. Create them for all candidate LUNs with one command:

Verify:

Result: one cell disk per LUN (typically CD_00_cell01 and so on for hard disks, and FD_00_cell01 and so on for flash), all normal. At this point freeSpace equals the full size, because no grid disks exist yet.

Flash disks: create the flash log and flash cache

On a High Capacity cell, the flash cell disks are normally used for Smart Flash Cache and Smart Flash Log, not for grid disks:

WARNING: Do not use the FORCE option with CREATE CELLDISK unless Oracle Support tells you to. It reuses an existing LUN and any previous configuration on it is lost.

6. Step 4: Create the Grid Disks (the “Partitions”)

A grid disk is a slice of a cell disk that is handed to ASM. You decide how much space each disk group gets by choosing the grid disk sizes. Look at the space available first:

Case A: new system or lab (you choose the sizes)

Create the DATA grid disks first with an explicit size, then the RECO grid disks. If you leave out size, the grid disk uses all the remaining free space of the cell disk.

Case B: adding a new cell to existing disk groups (sizes must match)

Every disk in a disk group must be the same size, so copy the size and caching policy from a cell that is already in the disk group. Run this on an existing cell:

Then, on the new cell, create the grid disks with exactly those values, starting with the first disk group:

WARNING: Always type the exact size, including the unit (G or T). A different size on the new cell causes uneven disks in the disk group.

Verify:

Example output (your names and sizes will differ):

Result: the grid disks are named <prefix>_<cell disk name> (for example DATAC1_CD_00_cell01), with status = active and asmModeStatus = UNUSED. UNUSED is correct: ASM is not using them yet.

Optional: use availableTo='+asm,<db_name>' on create griddisk to restrict which ASM and database clients may use a grid disk (ASM-scoped or database-scoped security).

7. Step 5: Make the Cell Visible to the Database Servers

This step is needed only when you add a new storage server. The database servers find storage cells through the file /etc/oracle/cell/network-config/cellip.ora. Add the new cell’s IP address(es) to it on every database server. The file must be identical on all of them.

Check the ASM discovery string (ASM instance):

Note: a cell with two network ports (for example InfiniBand or RoCE) is listed with both IP addresses separated by a semicolon. Grid disk paths then look like o/<ip1>;<ip2>/<griddisk>. If you changed asm_diskstring to a narrower pattern, make sure it also matches your new prefixes.

8. Step 6: See the Grid Disks as ASM Disks

ASM discovers every grid disk as an ASM disk with the path o/<cell_ip>/<griddisk_name>. Disks that are not yet in any disk group have header_status = CANDIDATE. Run this in the ASM instance (as SYSASM):

Result: you see all the new grid disks (for example o/<cell_ip>/DATAC1_CD_00_cell01) as CANDIDATE. If you see none, go to the Troubleshooting section.

9. Step 7: Add the Disks to an ASM Disk Group

Option A: create a new disk group

Use this on a new system or in a lab. Normal redundancy mirrors data across storage cells, so you need at least 2 cells (3 are recommended).

Note: set compatible.asm and compatible.rdbms to match your Grid Infrastructure and database versions. Each cell is one failure group, so ASM can lose a whole cell and still keep a copy of the data. A shorter form such as disk 'o/*/DATAC1_*' also works.

Option B: add the new disks to an existing disk group

Use this when you added a new cell. Add the disks of the new cell only, and set the rebalance power. Run the commands from different ASM instances so the rebalances can run in parallel.

Watch the rebalance:

ASM now moves some data onto the new disks. When the rebalance finishes, the new disks are fully part of the disk group.

WARNING: Add the disks of only the new cell, and use the exact prefix for each disk group. A pattern such as o/*/* in ADD DISK can match disks you did not mean to touch.

10. Step 8: Verify End to End

On the storage server:

Result: asmModeStatus = ONLINE, and the disk group, ASM disk name and failure group are filled in.

In the ASM instance:

Result: all new disks are ONLINE and NORMAL, grouped under the new cell’s failure group, and the disk group is MOUNTED.

11. Undo: Remove Everything in Reverse Order

WARNING: This destroys data. Use it only for a lab or for a cell you are removing, and only after the data has been moved away.

Work from the top layer down, the reverse of how you built it.

Note: a cell disk can be dropped only after all of its grid disks are dropped. Grid disks that are still used by ASM must first be dropped from the disk group.

12. Troubleshooting

13. Quick Cheatsheet