Proxmox/Ceph
Health
Show high level health:
ceph health ceph -s # more details
Show OSD (Object Storage Deamon) health:
ceph osd df tree ceph osd df
---
ceph auth get client.bootstrap-osd
ceph auth get client.bootstrap-osd > /var/lib/ceph/bootstrap-osd/ceph.keyring
ceph-volume lvm create --bluestore --data /dev/fioa
---
The basic installation and configuration is complete. Depending on your setup, some of the following steps are required to start using Ceph:
- Install Ceph on other nodes
- Create additional Ceph Monitors
- Create Ceph OSDs
- Create Ceph Pools
To learn more, click on the help button below.
https://proxmox1.example.com/pve-docs/chapter-pveceph.html#pve_ceph_install
---
Welcome to Ceph — Ceph Documentation https://docs.ceph.com/en/latest/
Ceph on LVM
Remove local-lvm:
pvesm remove local-lvm lvremove /dev/pve/data
bootstrap auth:
ceph auth get client.bootstrap-osd > /var/lib/ceph/bootstrap-osd/ceph.keyring
Create new logical volume with the remaining free space:
lvcreate -l 100%FREE -n pve/ceph
Create (= prepare and activate) the logical volume for OSD:
ceph-volume lvm create --data pve/ceph
Use GUI to create Metadata servers, create CephFS, etc
Ref: https://forum.proxmox.com/threads/ceph-osd-on-lvm-logical-volume.68618/
---
to undo the removal of local-lvm: [1]
# lvcreate -l99%FREE -n newLvName newVgName # lvconvert --type thin newVgName/newLvName
pvesm status pvesm add lvmthin local-lvm --thinpool data --vgname pve
OSD Service Stopped
In "ceph osd df tree" (or web ui) the "up"/"down" is controled by the "service". The "in"/"out" is controlled by ceph. To start/stop the service: (you can also start/stop the service from Proxmox web interface)
$ sudo systemctl status ceph-osd.target
- ceph-osd.target - ceph target allowing to start/stop all ceph-osd@.service instances at once
Loaded: loaded (/lib/systemd/system/ceph-osd.target; enabled; preset: enabled)
Active: active since Sat 2024-09-07 21:50:39 MDT; 1min 16s ago
$ sudo systemctl status ceph-osd.target
$ sudo systemctl stop ceph-osd.target
$ sudo systemctl start ceph-osd.target
$ sudo systemctl restart ceph-osd.target
Verify Auth
ceph auth ls
Issues
ceph auth get client.bootstrap-osd - Error initializing cluster client
Error:
# ceph auth get client.bootstrap-osd
Error initializing cluster client: ObjectNotFound('RADOS object not found (error calling conf_read_file)')
Fix with:
ln -s /etc/pve/ceph.conf /etc/ceph/ceph.conf
ref: [SOLVED] - Ceph Pacific Issue | Proxmox Support Forum - https://forum.proxmox.com/threads/ceph-pacific-issue.127987/