Proxmox/Ceph
< Proxmox
Health
Show high level health:
ceph health ceph -s # more details
Show OSD (Object Storage Deamon) health:
ceph osd df tree ceph osd df
---
ceph auth get client.bootstrap-osd
ceph auth get client.bootstrap-osd > /var/lib/ceph/bootstrap-osd/ceph.keyring
ceph-volume lvm create --bluestore --data /dev/fioa
---
The basic installation and configuration is complete. Depending on your setup, some of the following steps are required to start using Ceph:
- Install Ceph on other nodes
- Create additional Ceph Monitors
- Create Ceph OSDs
- Create Ceph Pools
To learn more, click on the help button below.
https://proxmox1.example.com/pve-docs/chapter-pveceph.html#pve_ceph_install
---
Welcome to Ceph — Ceph Documentation https://docs.ceph.com/en/latest/
Manual Install from Command Line
echo "deb http://download.proxmox.com/debian/ceph-reef bookworm no-subscription" > /etc/apt/sources.list.d/ceph.list apt update apt install ceph
OR
pveceph install This will install Ceph Quincy
# 18.2.2-pve1
Install Ceph on dedicated network:
pveceph init --network 10.10.10.0/24
Create Monitors (MON): (at least 3)
pveceph mon create
Create Managers (MGR): (at least 1)
pveceph mgr create
Create LVM OSDs:
- See #Ceph on LVM
Create Metadata Server (MDS): (at least 1)
pveceph mds create
Create Pool:
# pveceph pool create NAME pveceph pool create CEPH -pg_autoscale_mode on # -pg_autoscale_mode <off|on|warn> warn is default
pool CEPH: applying size = 3 pool CEPH: applying application = rbd pool CEPH: applying min_size = 2 pool CEPH: applying pg_autoscale_mode = warn pool CEPH: applying pg_num = 128
---
ceph packages:
# dpkg -l | grep -i ceph ii ceph 18.2.2-pve1 amd64 distributed storage and file system ii ceph-base 18.2.2-pve1 amd64 common ceph daemon libraries and management tools ii ceph-common 18.2.2-pve1 amd64 common utilities to mount and interact with a ceph storage cluster ii ceph-fuse 18.2.2-pve1 amd64 FUSE-based client for the Ceph distributed file system ii ceph-mds 18.2.2-pve1 amd64 metadata server for the ceph distributed file system ii ceph-mgr 18.2.2-pve1 amd64 manager for the ceph distributed storage system ii ceph-mgr-modules-core 18.2.2-pve1 all ceph manager modules which are always enabled ii ceph-mon 18.2.2-pve1 amd64 monitor server for the ceph storage system ii ceph-osd 18.2.2-pve1 amd64 OSD server for the ceph storage system ii ceph-volume 18.2.2-pve1 all tool to facilidate OSD deployment ii libcephfs2 18.2.2-pve1 amd64 Ceph distributed file system client library ii libsqlite3-mod-ceph 18.2.2-pve1 amd64 SQLite3 VFS for Ceph ii python3-ceph-argparse 18.2.2-pve1 all Python 3 utility libraries for Ceph CLI ii python3-ceph-common 18.2.2-pve1 all Python 3 utility libraries for Ceph ii python3-cephfs 18.2.2-pve1 amd64 Python 3 libraries for the Ceph libcephfs library ii python3-rados 18.2.2-pve1 amd64 Python 3 libraries for the Ceph librados library ii python3-rbd 18.2.2-pve1 amd64 Python 3 libraries for the Ceph librbd library ii python3-rgw 18.2.2-pve1 amd64 Python 3 libraries for the Ceph librgw library
ceph config example:
# cat /etc/ceph/ceph.conf
[global]
auth_client_required = cephx
auth_cluster_required = cephx
auth_service_required = cephx
cluster_network = 10.10.108.0/24
fsid = a33b5284-6139-4a1c-88b5-0xxxxxxxxxx
mon_allow_pool_delete = true
mon_host = 10.10.108.31 10.10.108.32 10.10.108.33
ms_bind_ipv4 = true
ms_bind_ipv6 = false
osd_pool_default_min_size = 2
osd_pool_default_size = 3
public_network = 10.10.108.0/24
[client]
keyring = /etc/pve/priv/$cluster.$name.keyring
[client.crash]
keyring = /etc/pve/ceph/$cluster.$name.keyring
[mon.proxmox1]
public_addr = 10.10.108.31
[mon.proxmox2]
public_addr = 10.10.108.32
[mon.proxmox3]
public_addr = 10.10.108.33
pveceph init options:
USAGE: pveceph init [OPTIONS]
Create initial ceph default configuration and setup symlinks.
-cluster-network <string>
Declare a separate cluster network, OSDs will routeheartbeat,
object replication and recovery traffic over it
Requires option(s): network
-disable_cephx <boolean> (default=0)
Disable cephx authentication.
WARNING: cephx is a security feature protecting against
man-in-the-middle attacks. Only consider disabling cephx if
your network is private!
-min_size <integer> (1 - 7) (default=2)
Minimum number of available replicas per object to allow I/O
-network <string>
Use specific network for all ceph related traffic
-pg_bits <integer> (6 - 14) (default=6)
Placement group bits, used to specify the default number of
placement groups.
Depreacted. This setting was deprecated in recent Ceph
versions.
-size <integer> (1 - 7) (default=3)
Targeted number of replicas per object
ref:
Deploy Hyper-Converged Ceph Cluster - Proxmox VE https://pve.proxmox.com/wiki/Deploy_Hyper-Converged_Ceph_Cluster
Ceph on LVM
Remove local-lvm:
pvesm remove local-lvm lvremove /dev/pve/data
bootstrap auth:
ceph auth get client.bootstrap-osd > /var/lib/ceph/bootstrap-osd/ceph.keyring
Create new logical volume with the remaining free space:
lvcreate -l 100%FREE -n pve/ceph
Create (= prepare and activate) the logical volume for OSD:
ceph-volume lvm create --data pve/ceph
Use GUI to create Metadata servers, create CephFS, etc
Ref: https://forum.proxmox.com/threads/ceph-osd-on-lvm-logical-volume.68618/
Undo local-lvm
to undo the removal of local-lvm: [1]
# lvcreate -l99%FREE -n newLvName newVgName # lvconvert --type thin newVgName/newLvName
pvesm status pvesm add lvmthin local-lvm --thinpool data --vgname pve
OSD Service Stopped
In "ceph osd df tree" (or web ui) the "up"/"down" is controled by the "service". The "in"/"out" is controlled by ceph. To start/stop the service: (you can also start/stop the service from Proxmox web interface)
$ sudo systemctl status ceph-osd.target
- ceph-osd.target - ceph target allowing to start/stop all ceph-osd@.service instances at once
Loaded: loaded (/lib/systemd/system/ceph-osd.target; enabled; preset: enabled)
Active: active since Sat 2024-09-07 21:50:39 MDT; 1min 16s ago
$ sudo systemctl status ceph-osd.target
$ sudo systemctl stop ceph-osd.target
$ sudo systemctl start ceph-osd.target
$ sudo systemctl restart ceph-osd.target
Verify Auth
ceph auth ls
Issues
ceph auth get client.bootstrap-osd - Error initializing cluster client
Error:
# ceph auth get client.bootstrap-osd
Error initializing cluster client: ObjectNotFound('RADOS object not found (error calling conf_read_file)')
Fix with:
ln -s /etc/pve/ceph.conf /etc/ceph/ceph.conf
ref: [SOLVED] - Ceph Pacific Issue | Proxmox Support Forum - https://forum.proxmox.com/threads/ceph-pacific-issue.127987/