Proxmox/Ceph

Health

Show high level health:

ceph health
ceph -s    # more details

Show OSD (Object Storage Deamon) health:

ceph osd df tree
ceph osd df

---

ceph auth get client.bootstrap-osd

ceph auth get client.bootstrap-osd > /var/lib/ceph/bootstrap-osd/ceph.keyring

ceph-volume lvm create --bluestore --data /dev/fioa

---

The basic installation and configuration is complete. Depending on your setup, some of the following steps are required to start using Ceph:

1. Install Ceph on other nodes
2. Create additional Ceph Monitors
3. Create Ceph OSDs
4. Create Ceph Pools

To learn more, click on the help button below.

https://proxmox1.example.com/pve-docs/chapter-pveceph.html#pve_ceph_install

---

Welcome to Ceph — Ceph Documentation
https://docs.ceph.com/en/latest/

Manual Install from Command Line

Deploy Hyper-Converged Ceph Cluster - Proxmox VE
https://pve.proxmox.com/wiki/Deploy_Hyper-Converged_Ceph_Cluster

--

echo "deb http://download.proxmox.com/debian/ceph-reef bookworm no-subscription" > /etc/apt/sources.list.d/ceph.list
apt update
apt install ceph

OR

pveceph install
  This will install Ceph Quincy

# 18.2.2-pve1

Install Ceph on dedicated network:

pveceph init --network 10.10.10.0/24

Create Monitors (MON): (for HA at least 3.)

pveceph mon create

Create Managers (MGR): (at least 1 - only 1 active rest are standby)

pveceph mgr create

Create LVM OSDs:

• See #Ceph on LVM

pveceph osd create /dev/sd[X]

Bluestore: (default after Ceph Luminous release)
pveceph osd create /dev/sd[X]

note: If the disk was in use before (for example, for ZFS or as an OSD) you first need to zap all traces of that usage.
ceph-volume lvm zap /dev/sd[X] --destroy

Create Pool:

# pveceph pool create NAME
pveceph pool create CEPH
pveceph pool create CEPH -pg_autoscale_mode on
pveceph pool create CEPH -pg_autoscale_mode on  --add_storages
  # -pg_autoscale_mode <off|on|warn> warn is default

pool CEPH: applying size = 3
pool CEPH: applying application = rbd
pool CEPH: applying min_size = 2
pool CEPH: applying pg_autoscale_mode = warn
pool CEPH: applying pg_num = 128

Edit Pool:

pveceph pool set CEPH -pg_autoscale_mode on

Create Storage on top of pool:

pvesm add rbd CEPH --pool CEPH --content images
  --monhost "10.1.1.20 10.1.1.21 10.1.1.22" ???
  --keyring /root/rbd.keyring ???  /etc/pve/priv/ceph/<STORAGE_ID>.conf ???

note: maybe best to do this one from the GUI?

Status

pveceph status

file store - optional

Defaults to half Ceph storage

Create Metadata Server (MDS): (need at least 1)

pveceph mds create

Create CephFS

pveceph fs create -name CEPHFS -add-storage

- destroy -

Stop MDS first:

# default name is the nodename
pveceph mds destroy [NAME]
pveceph mds destroy proxmox1

pvesm status

Remove Storage:

pvesm remove [NAME]
pvesm remove CEPHFS

Remove FS:

pveceph fs destroy [NAME]
pveceph fs destroy CEPHFS

misc

MDS:

# pveceph mds destroy proxmox1
disabling service 'ceph-mds@proxmox1.service'
Removed "/etc/systemd/system/ceph-mds.target.wants/ceph-mds@proxmox1.service".
stopping service 'ceph-mds@proxmox1.service'
removing ceph-mds directory '/var/lib/ceph/mds/ceph-proxmox1'
removing ceph auth for 'mds.proxmox1'

ceph packages:

# dpkg -l | grep -i ceph
ii  ceph                                 18.2.2-pve1                         amd64        distributed storage and file system
ii  ceph-base                            18.2.2-pve1                         amd64        common ceph daemon libraries and management tools
ii  ceph-common                          18.2.2-pve1                         amd64        common utilities to mount and interact with a ceph storage cluster
ii  ceph-fuse                            18.2.2-pve1                         amd64        FUSE-based client for the Ceph distributed file system
ii  ceph-mds                             18.2.2-pve1                         amd64        metadata server for the ceph distributed file system
ii  ceph-mgr                             18.2.2-pve1                         amd64        manager for the ceph distributed storage system
ii  ceph-mgr-modules-core                18.2.2-pve1                         all          ceph manager modules which are always enabled
ii  ceph-mon                             18.2.2-pve1                         amd64        monitor server for the ceph storage system
ii  ceph-osd                             18.2.2-pve1                         amd64        OSD server for the ceph storage system
ii  ceph-volume                          18.2.2-pve1                         all          tool to facilidate OSD deployment
ii  libcephfs2                           18.2.2-pve1                         amd64        Ceph distributed file system client library
ii  libsqlite3-mod-ceph                  18.2.2-pve1                         amd64        SQLite3 VFS for Ceph
ii  python3-ceph-argparse                18.2.2-pve1                         all          Python 3 utility libraries for Ceph CLI
ii  python3-ceph-common                  18.2.2-pve1                         all          Python 3 utility libraries for Ceph
ii  python3-cephfs                       18.2.2-pve1                         amd64        Python 3 libraries for the Ceph libcephfs library
ii  python3-rados                        18.2.2-pve1                         amd64        Python 3 libraries for the Ceph librados library
ii  python3-rbd                          18.2.2-pve1                         amd64        Python 3 libraries for the Ceph librbd library
ii  python3-rgw                          18.2.2-pve1                         amd64        Python 3 libraries for the Ceph librgw library

ceph config example:

# cat /etc/ceph/ceph.conf
[global]
        auth_client_required = cephx
        auth_cluster_required = cephx
        auth_service_required = cephx
        cluster_network = 10.10.108.0/24
        fsid = a33b5284-6139-4a1c-88b5-0xxxxxxxxxx
        mon_allow_pool_delete = true
        mon_host = 10.10.108.31 10.10.108.32 10.10.108.33
        ms_bind_ipv4 = true
        ms_bind_ipv6 = false
        osd_pool_default_min_size = 2
        osd_pool_default_size = 3
        public_network = 10.10.108.0/24

[client]
        keyring = /etc/pve/priv/$cluster.$name.keyring

[client.crash]
        keyring = /etc/pve/ceph/$cluster.$name.keyring

[mon.proxmox1]
        public_addr = 10.10.108.31

[mon.proxmox2]
        public_addr = 10.10.108.32

[mon.proxmox3]
        public_addr = 10.10.108.33

pveceph init options:

USAGE: pveceph init  [OPTIONS]

  Create initial ceph default configuration and setup symlinks.

  -cluster-network <string>
             Declare a separate cluster network, OSDs will routeheartbeat,
             object replication and recovery traffic over it

             Requires option(s): network

  -disable_cephx <boolean>   (default=0)
             Disable cephx authentication.

             WARNING: cephx is a security feature protecting against
             man-in-the-middle attacks. Only consider disabling cephx if
             your network is private!

  -min_size  <integer> (1 - 7)   (default=2)
             Minimum number of available replicas per object to allow I/O

  -network   <string>
             Use specific network for all ceph related traffic

  -pg_bits   <integer> (6 - 14)   (default=6)
             Placement group bits, used to specify the default number of
             placement groups.

             Depreacted. This setting was deprecated in recent Ceph
             versions.

  -size      <integer> (1 - 7)   (default=3)
             Targeted number of replicas per object

Ceph on LVM

Remove local-lvm:

pvesm remove local-lvm
lvremove /dev/pve/data

bootstrap auth:

ceph auth get client.bootstrap-osd > /var/lib/ceph/bootstrap-osd/ceph.keyring

Create new logical volume with the remaining free space:

lvcreate -l 100%FREE -n pve/ceph

Create (= prepare and activate) the logical volume for OSD:

ceph-volume lvm create --data pve/ceph

Use GUI to create Metadata servers, create CephFS, etc

Ref: https://forum.proxmox.com/threads/ceph-osd-on-lvm-logical-volume.68618/

Undo local-lvm

to undo the removal of local-lvm:

# lvcreate -l99%FREE -n newLvName newVgName
# lvconvert --type thin newVgName/newLvName

pvesm status
pvesm add lvmthin local-lvm --thinpool data --vgname pve

OSD Service Stopped

In "ceph osd df tree" (or web ui) the "up"/"down" is controled by the "service". The "in"/"out" is controlled by ceph. To start/stop the service: (you can also start/stop the service from Proxmox web interface)

$ sudo systemctl status ceph-osd.target
- ceph-osd.target - ceph target allowing to start/stop all ceph-osd@.service instances at once
    Loaded: loaded (/lib/systemd/system/ceph-osd.target; enabled; preset: enabled)
    Active: active since Sat 2024-09-07 21:50:39 MDT; 1min 16s ago

$ sudo systemctl status ceph-osd.target

$ sudo systemctl stop ceph-osd.target

$ sudo systemctl start ceph-osd.target

$ sudo systemctl restart ceph-osd.target

Verify Auth

ceph auth ls

Issues

ceph auth get client.bootstrap-osd - Error initializing cluster client

Error:

# ceph auth get client.bootstrap-osd
Error initializing cluster client: ObjectNotFound('RADOS object not found (error calling conf_read_file)')

Fix with:

ln -s /etc/pve/ceph.conf /etc/ceph/ceph.conf

ref: [SOLVED] - Ceph Pacific Issue | Proxmox Support Forum - https://forum.proxmox.com/threads/ceph-pacific-issue.127987/