Proxmox/Ceph

Health

Show high level health:

ceph health
ceph -s    # more details

Show OSD (Object Storage Deamon) health:

ceph osd df tree
ceph osd df

---

ceph auth get client.bootstrap-osd

ceph auth get client.bootstrap-osd > /var/lib/ceph/bootstrap-osd/ceph.keyring

ceph-volume lvm create --bluestore --data /dev/fioa

---

The basic installation and configuration is complete. Depending on your setup, some of the following steps are required to start using Ceph:

1. Install Ceph on other nodes
2. Create additional Ceph Monitors
3. Create Ceph OSDs
4. Create Ceph Pools

To learn more, click on the help button below.

https://proxmox1.example.com/pve-docs/chapter-pveceph.html#pve_ceph_install

---

Welcome to Ceph — Ceph Documentation
https://docs.ceph.com/en/latest/

Ceph on LVM

Remove local-lvm:

pvesm remove local-lvm
lvremove /dev/pve/data

bootstrap auth:

ceph auth get client.bootstrap-osd > /var/lib/ceph/bootstrap-osd/ceph.keyring

Create new logical volume with the remaining free space:

lvcreate -l 100%FREE -n pve/ceph

Create (= prepare and activate) the logical volume for OSD:

ceph-volume lvm create --data pve/ceph

Use GUI to create Metadata servers, create CephFS, etc

Ref: https://forum.proxmox.com/threads/ceph-osd-on-lvm-logical-volume.68618/

Undo local-lvm

to undo the removal of local-lvm:

# lvcreate -l99%FREE -n newLvName newVgName
# lvconvert --type thin newVgName/newLvName

pvesm status
pvesm add lvmthin local-lvm --thinpool data --vgname pve

OSD Service Stopped

In "ceph osd df tree" (or web ui) the "up"/"down" is controled by the "service". The "in"/"out" is controlled by ceph. To start/stop the service: (you can also start/stop the service from Proxmox web interface)

$ sudo systemctl status ceph-osd.target
- ceph-osd.target - ceph target allowing to start/stop all ceph-osd@.service instances at once
    Loaded: loaded (/lib/systemd/system/ceph-osd.target; enabled; preset: enabled)
    Active: active since Sat 2024-09-07 21:50:39 MDT; 1min 16s ago

$ sudo systemctl status ceph-osd.target

$ sudo systemctl stop ceph-osd.target

$ sudo systemctl start ceph-osd.target

$ sudo systemctl restart ceph-osd.target

Verify Auth

ceph auth ls

Issues

ceph auth get client.bootstrap-osd - Error initializing cluster client

Error:

# ceph auth get client.bootstrap-osd
Error initializing cluster client: ObjectNotFound('RADOS object not found (error calling conf_read_file)')

Fix with:

ln -s /etc/pve/ceph.conf /etc/ceph/ceph.conf

ref: [SOLVED] - Ceph Pacific Issue | Proxmox Support Forum - https://forum.proxmox.com/threads/ceph-pacific-issue.127987/