安装Ceph客户端认证
如果开启了cephx authentication，需要为Nova/Cinder and Glance创建新的用户，如下 ceph auth get-or-create client.cinder mon 'allow r' osd 'allow class-read object_prefix rbd_children, allow rwx pool=volumes, allow rwx pool=vms, allow rx pool=images' ceph auth get-or-create client.glance mon 'allow r' osd 'allow class-read object_prefix rbd_children, allow rwx pool=images'
ceph auth get-or-create client.cinder-backup mon 'allow r' osd 'allow class-read object_prefix rbd_children, allow rwx pool=backups'
为client.cinder, client.glance, and client.cinder-backup添加keyring，如下
ceph auth get-or-create client.glance | ssh {your-glance-api-server} sudo tee /etc/ceph/ceph.client.glance.keyring
ssh {your-glance-api-server} sudo chown glance:glance /etc/ceph/ceph.client.glance.keyring ceph auth get-or-create client.cinder | ssh {your-volume-server} sudo tee /etc/ceph/ceph.client.cinder.keyring
ssh {your-cinder-volume-server} sudo chown cinder:cinder /etc/ceph/ceph.client.cinder.keyring
ceph auth get-or-create client.cinder-backup | ssh {your-cinder-backup-server} sudo tee /etc/ceph/ceph.client.cinder-backup.keyring
ssh {your-cinder-backup-server} sudo chown cinder:cinder /etc/ceph/ceph.client.cinder-backup.keyring
此处应改为Controller，
ceph auth get-or-create client.glance | ssh Controller sudo tee /etc/ceph/ceph.client.glance.keyring
ssh Controller sudo chown glance:glance /etc/ceph/ceph.client.glance.keyring
ceph auth get-or-create client.cinder | ssh Computer01 sudo tee /etc/ceph/ceph.client.cinder.keyring
ssh Computer01 sudo chown cinder:cinder /etc/ceph/ceph.client.cinder.keyring ceph auth get-or-create client.cinder | ssh Computer02 sudo tee /etc/ceph/ceph.client.cinder.keyring
ssh Computer02 sudo chown cinder:cinder /etc/ceph/ceph.client.cinder.keyring ceph auth get-or-create client.cinder | ssh Computer03 sudo tee /etc/ceph/ceph.client.cinder.keyring
ssh Computer03 sudo chown cinder:cinder /etc/ceph/ceph.client.cinder.keyring
ceph auth get-or-create client.cinder-backup | ssh Computer01 sudo tee /etc/ceph/ceph.client.cinder-backup.keyring
ssh Computer01 sudo chown cinder:cinder /etc/ceph/ceph.client.cinder-backup.keyring ceph auth get-or-create client.cinder-backup | ssh Computer02 sudo tee
41
/etc/ceph/ceph.client.cinder-backup.keyring
ssh Computer02 sudo chown cinder:cinder /etc/ceph/ceph.client.cinder-backup.keyring ceph auth get-or-create client.cinder-backup | ssh Computer03 sudo tee /etc/ceph/ceph.client.cinder-backup.keyring
ssh Computer03 sudo chown cinder:cinder /etc/ceph/ceph.client.cinder-backup.keyring
复制Keyring文件到nova-compute节点
ceph auth get-or-create client.cinder | ssh {your-nova-compute-server} sudo tee /etc/ceph/ceph.client.cinder.keyring
此处为
ceph auth get-or-create client.cinder | ssh Computer01 sudo tee /etc/ceph/ceph.client.cinder.keyring
ceph auth get-or-create client.cinder | ssh Computer02 sudo tee /etc/ceph/ceph.client.cinder.keyring
ceph auth get-or-create client.cinder | ssh Computer03 sudo tee /etc/ceph/ceph.client.cinder.keyring
在nova-compute节点上创建临时密钥
ceph auth get-key client.cinder | ssh {your-compute-node} tee client.cinder.key 此处为
ceph auth get-key client.cinder | ssh Computer01 tee client.cinder.key ceph auth get-key client.cinder | ssh Computer02 tee client.cinder.key ceph auth get-key client.cinder | ssh Computer03 tee client.cinder.key
在所有计算节点上执行如下操作：在计算节点上为libvert替换新的key uuidgen
4314ecd9-ffd2-4406-b6a5-c155b547a918 cat > secret.xml < 粘贴以下内容，注意将红色key替换为新生成的key。 sudo virsh secret-define --file secret.xml sudo virsh secret-set-value --secret 4314ecd9-ffd2-4406-b6a5-c155b547a918 --base64 $(cat client.cinder.key)&& rm client.cinder.key secret.xml $(cat client.cinder.key)在/etc/ceph/ceph.client.cinder.keyring中可查到。 例如此处 sudo virsh secret-set-value --secret 4314ecd9-ffd2-4406-b6a5-c155b547a918 --base64 42 AQB+85lXpfLaBhAAKxY6kC4Qff2K4M6c4CiHEw==&& rm client.cinder.key secret.xml sudo virsh secret-set-value --secret 34cd7cfa-b254-4eb9-9dfb-38731cf95aeb --base64 AQB+85lXpfLaBhAAKxY6kC4Qff2K4M6c4CiHEw==&& rm client.cinder.key secret.xml sudo virsh secret-set-value --secret 4314ecd9-ffd2-4406-b6a5-c155b547a918 --base64 AQB+85lXpfLaBhAAKxY6kC4Qff2K4M6c4CiHEw==&& rm client.cinder.key secret.xml Openstack配置 vim /etc/glance/glance-api.conf [DEFAULT] ... default_store = rbd ... [glance_store] stores = rbd rbd_store_pool = images rbd_store_user = glance rbd_store_ceph_conf = /etc/ceph/ceph.conf rbd_store_chunk_size = 8 如果要开启copy-on-write模式镜像克隆 [DEFAULT] show_image_direct_url = True 取消Glance cache管理，去掉cachemanagement [paste_deploy] flavor = keystone vim /etc/cinder/cinder.conf [DEFAULT] ... enabled_backends = ceph glance_api_version = 2 ... [ceph] volume_driver = cinder.volume.drivers.rbd.RBDDriver rbd_pool = volumes rbd_ceph_conf = /etc/ceph/ceph.conf 43 rbd_flatten_volume_from_snapshot = false rbd_max_clone_depth = 5 rbd_store_chunk_size = 4 rados_connect_timeout = -1 glance_api_version = 2 如果开启cephx authentication，[ceph]中用户和uuid 密钥 [ceph] ... rbd_user = cinder rbd_secret_uuid =4314ecd9-ffd2-4406-b6a5-c155b547a918 注意，如果配置多个cinder后端，glance_api_version = 2必须添加到[DEFAULT]中。 OpenStack Cinder Backup需要特定daemon，在Cinder Backup节点配置文件中添加以下内容 vim /etc/cinder/cinder.conf backup_driver = cinder.backup.drivers.ceph backup_ceph_conf = /etc/ceph/ceph.conf backup_ceph_user = cinder-backup backup_ceph_chunk_size = 134217728 backup_ceph_pool = backups backup_ceph_stripe_unit = 0 backup_ceph_stripe_count = 0 restore_discard_excess_bytes = true 设置Nova用于绑定Cinder设备（不知道在哪里设置？？？？） rbd_user = cinder rbd_secret_uuid =4314ecd9-ffd2-4406-b6a5-c155b547a918 在每个计算节点上修改Ceph配置文件 vim /etc/ceph/ceph.conf [client] rbd cache = true rbd cache writethrough until flush = true admin socket = /var/run/ceph/guests/$cluster-$type.$id.$pid.$cctid.asok log file = /var/log/qemu/qemu-guest-$pid.log rbd concurrent management ops = 20 设置路径权限 mkdir -p /var/run/ceph/guests/ /var/log/qemu/ chown qemu:libvirtd /var/run/ceph/guests /var/log/qemu/ 用户qemu和组libvirtd以具体系统为准 44 这里统一设为 chown inspur:inspur /var/run/ceph/guests /var/log/qemu/ 每个计算节点上，设置/etc/nova/nova.conf vim /etc/nova/nova.conf [libvirt] images_type = rbd images_rbd_pool = vms images_rbd_ceph_conf = /etc/ceph/ceph.conf rbd_user = cinder rbd_secret_uuid=4314ecd9-ffd2-4406-b6a5-c155b547a918 disk_cachemodes=\inject_password = false inject_key = false inject_partition = -2 live_migration_flag=\RATE_LIVE,VIR_MIGRATE_PERSIST_DEST,VIR_MIGRATE_TUNNELLED\ hw_disk_discard = unmap 重启OpenStack sudo service openstack-glance-api restart sudo service openstack-cinder-volume restart sudo service openstack-cinder-backup restart sudo service openstack-nova-compute restart 脚本 sudo service openstack-glance-api restart ssh Computer01 sudo service openstack-nova-compute restart ssh Computer01 sudo service openstack-cinder-volume restart ssh Computer01 sudo service openstack-cinder-backup restart ssh Computer02 sudo service openstack-nova-compute restart ssh Computer02 sudo service openstack-cinder-volume restart ssh Computer02 sudo service openstack-cinder-backup restart ssh Computer03 sudo service openstack-nova-compute restart ssh Computer03 sudo service openstack-cinder-volume restart ssh Computer03 sudo service openstack-cinder-backup restart 云硬盘分配与挂载 可以在Dashboard中分配 创建云硬盘—>管理连接—>连接到虚拟机上—>磁盘挂载 45