Proxmox/Fix Keyring

Unable to create a new OSD id

root@vm-01:~# ceph-volume lvm create --bluestore --data /dev/fioa
--> Incompatible flags were found, some values may get ignored
--> Cannot use None (None) with --bluestore (bluestore)
Running command: /usr/bin/ceph-authtool --gen-print-key
Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring -i - osd new 2b231332-bc5d-467b-a951-1cebf40c06d4
 stderr: 2024-01-11T08:39:03.699-0700 7fd34a9566c0 -1 auth: unable to find a keyring on /var/lib/ceph/bootstrap-osd/ceph.keyring: (2) No such file or directory
 stderr: 2024-01-11T08:39:03.699-0700 7fd34a9566c0 -1 AuthRegistry(0x7fd344063e88) no keyring found at /var/lib/ceph/bootstrap-osd/ceph.keyring, disabling cephx
 stderr: 2024-01-11T08:39:03.699-0700 7fd34a9566c0 -1 auth: unable to find a keyring on /var/lib/ceph/bootstrap-osd/ceph.keyring: (2) No such file or directory
 stderr: 2024-01-11T08:39:03.699-0700 7fd34a9566c0 -1 AuthRegistry(0x7fd34406b3e0) no keyring found at /var/lib/ceph/bootstrap-osd/ceph.keyring, disabling cephx
 stderr: 2024-01-11T08:39:03.699-0700 7fd34a9566c0 -1 auth: unable to find a keyring on /var/lib/ceph/bootstrap-osd/ceph.keyring: (2) No such file or directory
 stderr: 2024-01-11T08:39:03.699-0700 7fd34a9566c0 -1 AuthRegistry(0x7fd34a9553c0) no keyring found at /var/lib/ceph/bootstrap-osd/ceph.keyring, disabling cephx
 stderr: [errno 2] RADOS object not found (error connecting to the cluster)
-->  RuntimeError: Unable to create a new OSD id

If this is a second node, copy the keyring from first node to second:

scp /var/lib/ceph/bootstrap-osd/ceph.keyring vm-node-02:/var/lib/ceph/bootstrap-osd/ceph.keyring

or just:

ceph auth get client.bootstrap-osd > /var/lib/ceph/bootstrap-osd/ceph.keyring

If this is the first node, you need to create the initial keyring. This is normally done by the GUI.

Example: 1. Generate key:

/usr/bin/ceph-authtool --gen-print-key
  AQCgEJpli7fWBxAAgUeG4JHtKK+HW8tSrhTvuw==

2. Create config file: /var/lib/ceph/bootstrap-osd/ceph.keyring

[client.bootstrap-osd]
        key = AQCgEJpli7fWBxAAgUeG4JHtKK+HW8tSrhTvuw==
        caps mon = "allow profile bootstrap-osd"

Error EEXIST: entity osd.2 exists but key does not match

root@vm-08:~/iomemory-vsl# ceph-volume lvm create --bluestore --data /dev/fioa
--> Incompatible flags were found, some values may get ignored
--> Cannot use None (None) with --bluestore (bluestore)
Running command: /usr/bin/ceph-authtool --gen-print-key
Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring -i - osd new c6a355b3-0958-4d1d-bd16-dad22e3a6e92
 stderr: Error EEXIST: entity osd.2 exists but key does not match
-->  RuntimeError: Unable to create a new OSD id

Solution:

root@vm-08:~/iomemory-vsl# ceph auth del osd.2