Nova 的设计遵循 service -> manager -> driver 的三层架构。
主要包括 nova-api, nova-conductor, nova-scheduler, nova-compute 四大核心组件(进程),组件之间通过 RPC 进行通信。
RESTful 入口 nova-api
# etc/nova/api-paste.ini
[app:osapi_compute_app_v21]
paste.app_factory = nova.api.openstack.compute:APIRouterV21.factory
# nova/api/openstack/compute/routes.py
ROUTE_LIST = (
('/servers', {
'GET': [server_controller, 'index'],
'POST': [server_controller, 'create']
}),
('/servers/{id}', {
'GET': [server_controller, 'show'],
'PUT': [server_controller, 'update'],
'DELETE': [server_controller, 'delete']
}),
('/servers/{server_id}/os-volume_attachments', {
'GET': [server_volume_attachments_controller, 'index'],
'POST': [server_volume_attachments_controller, 'create'],
}),
('/servers/{server_id}/os-volume_attachments/{id}', {
'GET': [server_volume_attachments_controller, 'show'],
'PUT': [server_volume_attachments_controller, 'update'],
'DELETE': [server_volume_attachments_controller, 'delete']
}),
)
class APIRouterV21(base_wsgi.Router):
def __init__(self, custom_routes=None):
for path, methods in ROUTE_LIST + custom_routes:
for method, controller_info in methods.items():
self.map.create_route(path, method, controller, action)
虚机创建
# nova/api/openstack/compute/servers.py
class ServersController(wsgi.Controller):
def create(self, req, body):
self.compute_api.create
# nova/compute/api.py
class API(base.Base):
def create
self._create_instance
self.compute_task_api.schedule_and_build_instance
api 只是对 rpcapi 的简单封装
# nova/conductor/api.py
class ComputeTaskAPI(object):
def schedule_and_build_instances
self.conductor_compute_rpcapi.schedule_and_build_instances
# nova/conductor/rpcapi.py
class ComputeTaskAPI(object):
def schedule_and_build_instances
cctxt.cast(context, 'schedule_and_build_instances', **kw)
rpc 的 endpoint 在 manager 中实现
# nova/conductor/manager.py
class ComputeTaskManager(base.Base):
def schedule_and_build_instances
self._schedule_instances
self.query_client.select_destinations
self.scheduler_rpcapi.select_destinations
self.compute_rpcapi.build_and_run_instance
从 nova-api 到 nova-conductor, nova-scheduler 兜兜转转了一圈,又回到了 nova-compute
# nova/compute/rpcapi.py
class ComputeAPI(object):
def build_and_run_instance
cctxt.cast(ctxt, 'build_and_run_instance', **kwargs)
实例创建的同时创建相应的块设备
# nova/compute/manager.py
class ComputeManager(manager.Manager):
def build_and_run_instance
self._do_build_and_run_instance
self._build_and_run_instance
self._build_resources
self._prep_block_device
driver.block_device_info_get_mapping
driver_block_device.attach_block_devices
bdm.attach
self.driver.spawn
bdm.attach
的实现将在下面云盘挂载流程里进行说明。
云盘挂载
# nova/api/openstack/compute/volumes.py
class VolumeAttachmentController(wsgi.Controller):
def create
self.compute_api.attach_volume
# nova/compute/api.py
class API(base.Base):
def attach_volume
self._attach_volume
self._create_volume_bdm
self.compute_rpcapi.reserve_block_device_name
self.compute_rpcapi.attach_volume
# nova/compute/rpcapi.py
class ComputeAPI(object):
def attach_volume
cctxt.cast(ctxt, 'attach_volume', instance=instance, bdm=bdm)
# nova/compute/manager.py
class ComputeManager(manager.Manager):
def attach_volume
self._attach_volume
bdm.attach
# nova/virt/block_device.py
class DriverVolumeBlockDevice(DriverBlockDevice):
def attach
self._do_attach
virt_driver.get_volume_connector(instance)
self._volume_attach
# Cinder 将调用驱动的 initialize_connection 方法得到 hypervisor 节点访问卷所需的信息
volume_api.attachment_update
virt_driver.attach_volume
# nova/volume/cinder.py
class API(object):
def attachment_update
cinderclient(context, '3.44', skip_version_check=True).attachments.update
body = {'attachment': {'connector': connector}}
resp = self._update('/attachments/%s' % id, body)
return self.resource_class(self, resp['attachment'], loaded=True,
resp=resp)
# nova/virt/libvirt/driver.py
class LibvirtDriver(driver.ComputeDriver):
def get_volume_connector
# 调用进入 os_brick 模块,返回所有 connector 的信息,如 iSCSI connector 就会返回 initiator 的 iqn
connector.get_connector_properties
def attach_volume
self._connect_volume
# 对于 iSCSI 而言,卷驱动为 LibvirtISCSIVolumeDriver,这里会进行 iSCSI login 以在 hypervisor 节点创建本地块设备
# 对于 RBD 而言,卷驱动为 LibvirtNetVolumeDriver,这里直接使用父类的实现,即什么都不做
vol_driver.connect_volume
# iSCSI
self.connector.connect_volume
self._get_volume_config
vol_driver.get_config
guest.attach_device
os-brick
os-brick 调用 get_connector_properties
接口收集当前客户端节点的所有信息,然后把这些信息作为入参调用 Cinder 卷驱动提供的 initialize_connection
接口;
initialize_connection
接口返回客户端挂载(严谨一点应该是”使用“,因为对于 librbd RBD 就不需要在客户端节点进行挂载)卷所需要的信息;
os-brick 调用 InitiatorConnector.factory
接口根据不同的卷类型(如 initiator.ISCSI
, initiatro.RBD
)构造 InitiatorConnector
子类实例;
os-brick 在挂载时仅知道卷的 ID,它是如何知道卷类型的,其实是因为卷驱动 initialize_connection
接口返回的 json 字段 driver_volume_type
标识了卷类型;
InitiatorConnector
子类实例调用 connect_volume
实例实现在客户端节点挂载卷(对于 iSCSI 而言就是 iscsiadm login 生成本地块设备,对于 RBD 而言则什么都不做);
Cinder
Cinder 负责卷控制面的工作,如增删改查,它不负责客户端对卷的使用;
但是,它会告诉客户端使用卷所需要的所有参数,这是在卷驱动实现的 initialize_connection
接口中实现的;
Cinder 虽然支持很多驱动,但大的驱动类别实际上非常少,归纳起来常见的就是:RBD, NFS, iSCSI, FC;
os-brick 侧我们使用的是 iSCSI 挂载的方式,因此 Cinder 驱动侧我们 initialize_connection
接口返回的 driver_volume_type
就是 iscsi
,同时返回的其它信息都是和 iSCSI 客户端登陆所需要的;
ACL 关系,需要在 initialize_connection
阶段建立,target 的 discovery 也都是这个阶段进行的,这样才能返回 target 的 portal, iqn 等
iSCSI initiator login 所必须的信息;
参考资料
First step for reading OpenStack Nova source code
https://medium.com/uckey/first-step-for-reading-openstack-nova-source-code-280758ff77c9
Block Device Mapping in Nova
https://docs.openstack.org/nova/stein/user/block-device-mapping.html
OpenStack Note
https://gtcsq.readthedocs.io/en/latest/openstack/index.html
nova-api: api-paste流程说明
https://bbs.huaweicloud.com/blogs/3635f96d993111e7b8317ca23e93a891
最后修改于 2019-07-07