The Rook add-on creates and manages a Ceph cluster along with a storage class for provisioning PVCs. It also runs the Ceph RGW object store to provide an S3-compatible store in the cluster.
By default the cluster uses the filesystem for storage. Each node in the cluster will have a single OSD backed by a directory in /opt/replicated/rook. Nodes with a Ceph Monitor also utilize /var/lib/rook.
Note: At minimum, 10GB of disk space should be available to /var/lib/rook for the Ceph Monitors and other configs. We recommend a separate partition to prevent a disruption in Ceph's operation as a result of /var or the root partition running out of space.
Note: All disks used for storage in the cluster should be of similar size. A cluster with large discrepancies in disk size may fail to replicate data to all available nodes.
The EKCO add-on is recommended when installing Rook. EKCO is responsible for performing various operations to maintain the health of a Ceph cluster.
spec:
rook:
version: latest
blockDeviceFilter: sd[b-z]
cephReplicaCount: 3
isBlockStorageEnabled: true
storageClassName: "storage"
hostpathRequiresPrivileged: false
bypassUpgradeWarning: false
| Flag | Usage |
|---|---|
| version | The version of rook to be installed. |
| storageClassName | The name of the StorageClass that will use Rook to provision PVCs. |
| cephReplicaCount | Replication factor of ceph pools. The default is to use the number of nodes in the cluster, up to a maximum of 3. |
| isBlockStorageEnabled | Use block devices instead of the filesystem for storage in the Ceph cluster. This flag will automatically be set to true for version 1.4.3+ because block storage must be enabled for these versions. |
| isSharedFilesystemDisabled | Disable the rook-ceph shared filesystem, reducing CPU and Memory load by no longer needing to schedule several pods. 1.4.3+ |
| blockDeviceFilter | Only use block devices matching this regex. |
| hostpathRequiresPrivileged | Runs Ceph Pods as privileged to be able to write to hostPaths in OpenShift with SELinux restrictions. |
| bypassUpgradeWarning | Bypass upgrade warning prompt. |
For production clusters, Rook should be configured to use block devices rather than the filesystem.
Enabling block storage is required with version 1.4.3+. Therefore, the isBlockStorageEnabled option will always be set to true when using version 1.4.3+.
The following spec enables block storage for the Rook add-on and automatically uses disks matching the regex /sd[b-z]/.
Rook will start an OSD for each discovered disk, which could result in multiple OSDs running on a single node.
Rook will ignore block devices that already have a filesystem on them.
spec:
rook:
version: latest
isBlockStorageEnabled: true
blockDeviceFilter: sd[b-z]
The Rook add-on will wait for a disk before continuing. If you have attached a disk to your node but the installer is still waiting at the Rook add-on installation step, refer to the troubleshooting guide for help with diagnosing and fixing common issues.
The Ceph filesystem is supported with version 1.4.3+.
This allows the use of PersistentVolumeClaims with access mode ReadWriteMany.
Set the storage class to rook-cephfs in the pvc spec to use this feature.
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: cephfs-pvc
spec:
accessModes:
- ReadWriteMany
resources:
requests:
storage: 1Gi
storageClassName: rook-cephfs
It is not possible to upgrade multiple minor versions of the Rook add-on at once.
Individual upgrades from one version to the next are required for upgrading multiple minor versions.
For example, to upgrade from 1.4.3 to 1.6.11, you must first install 1.5.10, 1.5.11 or 1.5.12.
If the currently installed Rook version is 1.0.x, upgrades to both 1.4.9 and 1.5.x are supported through the main installer.
Alternatively, the upgrade from 1.0.x to 1.4.9 can be triggered independently with curl https://k8s.kurl.sh/latest/tasks.sh | sudo bash -s rook_10_to_14.
This upgrade migrates data off of any hostpath-based OSDs in favor of block device-based OSDs and upgrades through Rook 1.1.9, 1.2.7 and 1.3.11 before installing 1.4.9 (and then optionally 1.5.x).
The upstream Rook project introduced a requirement for block storage in versions 1.3.x and later.
In instances without internet access, this requires supplying an additional file when prompted.