OpenStack pike 单节点部署测试和踩坑

发表于   |   分类于

Centos7.3 OpenStack pike版本单节点部署，只安装了glance、keystone、nova、neutron和horizon没有部署cinder swift等。过程主要是跟官方文档走，过程中有几个坑点，记录一下。

环境配置

  • 首先准备好各种密码，基本上每个服务要准备两个密码，一个数据库的、一个openstack中用户的。

  • 网络:由于单节点部署，控制节点两张网卡分别分配公网ip(59.67.x.x)和内网地址(192.168.31.200/官方例子中为10.0.0.11)。配置好hostname,在/etc/hosts中配置好节点名称与地址的对应关系。

  • NTP:单节点部署可以先不配置NTP服务,如果要配置则在控制节点配置并启用chrony作为服务器，其余节点与控制节点同步

    1. 安装chrony:

      yum install chrony

    2. 控制节点:

      ​ 编辑/etc/chrony.conf ,配置好NTP服务器，然后允许内网中其余节点获取同步:allow 192.168.31.0/24

      其余节点:

      ​ 编辑/etc/chrony.conf,将NTP服务器设置为controller:server controller iburst

    3. 启动服务并设置服务自启动

      systemctl enable chronyd.service

      systemctl start chronyd.service

    4. 验证配置

      在控制节点外的其他节点运行:chronyc sources , 在控制节点前MS栏看到*符号，证明同步成功。

  • 配置openstack软件包仓库

    ​ yum install centos-release-openstack-pike

    ​ yum upgrade

    ​ yum install python-openstackclient openstack-selinux

  • 安装配置数据库(控制节点)

    1. yum install mariadb mariadb-server python2-PyMySQL

    2. 创建文件/etc/my.cnf.d/openstack.cnf

      • 写入内容:

        1
        2
        3
        4
        5
        6
        7
        8
        [mysqld]
        bind-address = 192.168.31.200
        default-storage-engine = innodb
        innodb_file_per_table = on
        max_connections = 4096
        collation-server = utf8_general_ci
        character-set-server = utf8

        这里有个坑!!!

      • 启动服务，配置用户密码:

        1
        2
        3
        systemctl enable mariadb.service
        systemctl start mariadb.service
        mysql_secure_installation

    3. 上一步中配置max_connections设置为4096是因为默认的214连接数不够openstack使用，会导致不能连接数据库。在dashboard中偶尔弹出错误，nova list等命令报503等情况都可能是最大连接数过小的原因。

      而Centos7中虽然按照上面配置了mariadb，但如果使用 show variables like "max_connections"; 查看会发现最大连接数仍然是214。

      • Solution:

        这是由于mariadb的最大连接数还受制于系统，需要在服务启动时修改其的最大文件描述符限制。编辑/usr/lib/systemd/system/mariadb.service,在[Service]中增加两行: 

        1
        2
        LimitNOFILE=10000
        LimitNPROC=10000
        然后刷新系统服务:

        systemctl --system daemon-reload

  • 配置消息队列服务(控制节点)

    1. 安装并启动rabbitmq-server:

      1
      2
      3
      yum install rabbitmq-server
      systemctl enable rabbitmq-server.service
      systemctl start rabbitmq-server.service

    2. 添加openstack用户并配置其权限(注意替换RABBIT_PASS):

      1
      2
      rabbitmqctl add_user openstack RABBIT_PASS
      rabbitmqctl set_permissions openstack ".*" ".*" ".*"

  • 配置memcached(控制节点)

    1. 安装memcached及python库:yum install memcached python-memcached

    2. 编辑/etc/sysconfig/memcached修改OPTIONS为OPTIONS="-l 127.0.0.1,::1,controller"

    3. 启动服务并添加自启动:

      1
      2
      systemctl enable memcached.service
      systemctl start memcached.service

##Keystone(身份认证服务)

  1. 创建数据库和数据库用户(注意替换KEYSTONE_DBPASS)

    1
    2
    3
    4
    5
    MariaDB [(none)]> CREATE DATABASE keystone;
    MariaDB [(none)]> GRANT ALL PRIVILEGES ON keystone.* TO 'keystone'@'localhost' \
    IDENTIFIED BY 'KEYSTONE_DBPASS';
    MariaDB [(none)]> GRANT ALL PRIVILEGES ON keystone.* TO 'keystone'@'%' \
    IDENTIFIED BY 'KEYSTONE_DBPASS';

  2. 安装keystone和apache

    yum install openstack-keystone httpd mod_wsgi

  3. 配置keystone:

    • 编辑/etc/keystone/keystone.conf:
      • 在[database]段添加connection = mysql+pymysql://keystone:KEYSTONE_DBPASS@controller/keystone
      • 在[token]段添加provider = fernet

  4. 同步keystone数据库:

    su -s /bin/sh -c "keystone-manage db_sync" keystone

  5. 初始化fernet key仓库

    1
    2
    keystone-manage fernet_setup --keystone-user keystone --keystone-group keystone
    keystone-manage credential_setup --keystone-user keystone --keystone-group keystone

  6. 引导身份认证服务，创建了一个管理员帐户admin并设置了密码，设置了几个api地址，创建了区域1

    1
    2
    3
    4
    5
    keystone-manage bootstrap --bootstrap-password ADMIN_PASS \
     --bootstrap-admin-url http://controller:35357/v3/ \
     --bootstrap-internal-url http://controller:5000/v3/ \
     --bootstrap-public-url http://controller:5000/v3/ \
     --bootstrap-region-id RegionOne

  7. 配置apache

    1. 编辑文件/etc/httpd/conf/httpd.conf更改

      ServerName controller

    2. 将keystone安装后自带的wsgi配置文件连接到httpd运行配置中

      ln -s /usr/share/keystone/wsgi-keystone.conf /etc/httpd/conf.d/

  8. 启动apache

    1
    2
    systemctl enable httpd.service
    systemctl start httpd.service

  9. 创建管理员的OpenStack客户环境脚本

    • 创建一个文件如admin-openrc填入内容(注意替换ADMIN_PASS):

      1
      2
      3
      4
      5
      6
      7
      8
      export OS_PROJECT_DOMAIN_NAME=Default
      export OS_USER_DOMAIN_NAME=Default
      export OS_PROJECT_NAME=admin
      export OS_USERNAME=admin
      export OS_PASSWORD=ADMIN_PASS
      export OS_AUTH_URL=http://controller:35357/v3
      export OS_IDENTITY_API_VERSION=3
      export OS_IMAGE_API_VERSION=2

    • 使用. admin-openrc引入admin用户的环境变量

  10. 创建服务项目

    openstack project create --domain default --description "Service Project" service

  11. 创建测试项目和测试用户

    1
    2
    3
    4
    openstack project create --domain default  --description "Demo Project" demo
    openstack user create --domain default  --password-prompt demo
    openstack role create user
    openstack role add --project demo --user demo user

  12. 创建测试用户demo的环境脚本(替换DEMO_PASS):

    1
    2
    3
    4
    5
    6
    7
    8
    export OS_PROJECT_DOMAIN_NAME=Default
    export OS_USER_DOMAIN_NAME=Default
    export OS_PROJECT_NAME=demo
    export OS_USERNAME=demo
    export OS_PASSWORD=DEMO_PASS
    export OS_AUTH_URL=http://controller:5000/v3
    export OS_IDENTITY_API_VERSION=3
    export OS_IMAGE_API_VERSION=2

  13. 验证:

    1. 1
      2
      . admin-openrc         # 设置环境变量
      openstack token issue  # 请求token

      结果应类似下方:

      1
      2
      3
      4
      5
      6
      7
      8
      9
      10
      +------------+-----------------------------------------------------------------+
      | Field      | Value                                                           |
      +------------+-----------------------------------------------------------------+
      | expires    | 2016-02-12T20:44:35.659723Z                                     |
      | id         | gAAAAABWvjYj-Zjfg8WXFaQnUd1DMYTBVrKw4h3fIagi5NoEmh21U72SrRv2trl |
      |            | JWFYhLi2_uPR31Igf6A8mH2Rw9kv_bxNo1jbLNPLGzW_u5FC7InFqx0yYtTwa1e |
      |            | eq2b0f6-18KZyQhs7F3teAta143kJEWuNEYET-y7u29y0be1_64KYkM7E       |
      | project_id | 343d245e850143a096806dfaefa9afdc                                |
      | user_id    | ac3377633149401296f6c0d92d79dc16                                |
      +------------+-----------------------------------------------------------------+

    2. 同理测试demo用户

Glance(镜像服务)

  1. 创建数据库和数据库用户(注意替换GLANCE_DBPASS):

    1
    2
    3
    4
    5
    MariaDB [(none)]> CREATE DATABASE glance;
    MariaDB [(none)]> GRANT ALL PRIVILEGES ON glance.* TO 'glance'@'localhost' \
      IDENTIFIED BY 'GLANCE_DBPASS';
    MariaDB [(none)]> GRANT ALL PRIVILEGES ON glance.* TO 'glance'@'%' \
      IDENTIFIED BY 'GLANCE_DBPASS';

  2. 引入管理员环境变量

    . admin-openrc

  3. 创建glance用户并设置密码:

    openstack user create --domain default --password-prompt glance

  4. 为service项目的glance用户添加admin角色

    openstack role add --project service --user glance admin

  5. 创建glance服务实体

    openstack service create --name glance --description "OpenStack Image" image

  6. 创建API:

    1
    2
    3
    openstack endpoint create --region RegionOne image public http://controller:9292
    openstack endpoint create --region RegionOne image internal http://controller:9292
    openstack endpoint create --region RegionOne image admin http://controller:9292

  7. 安装glance软件包

    yum install openstack-glance

  8. 配置glance

    • 编辑/etc/glance/glance-api.conf:

      1
      2
      3
      4
      5
      6
      7
      8
      9
      10
      11
      12
      13
      14
      15
      16
      17
      18
      19
      20
      21
      22
      23
      24
      25
      [database]
      # ...
      connection = mysql+pymysql://glance:GLANCE_DBPASS@controller/glance
      [keystone_authtoken]
      # ...
      auth_uri = http://controller:5000
      auth_url = http://controller:35357
      memcached_servers = controller:11211
      auth_type = password
      project_domain_name = default
      user_domain_name = default
      project_name = service
      username = glance
      password = GLANCE_PASS
      [paste_deploy]
      # ...
      flavor = keystone
      [glance_store]
      # ...
      stores = file,http
      default_store = file
      filesystem_store_datadir = /var/lib/glance/images/

    • 编辑/etc/glance/glance-registry.conf:

      1
      2
      3
      4
      5
      6
      7
      8
      9
      10
      11
      12
      13
      14
      15
      16
      17
      18
      19
      [database]
      # ...
      connection = mysql+pymysql://glance:GLANCE_DBPASS@controller/glance
      [keystone_authtoken]
      # ...
      auth_uri = http://controller:5000
      auth_url = http://controller:35357
      memcached_servers = controller:11211
      auth_type = password
      project_domain_name = default
      user_domain_name = default
      project_name = service
      username = glance
      password = GLANCE_PASS
      [paste_deploy]
      # ...
      flavor = keystone

  9. 同步数据库:

    su -s /bin/sh -c "glance-manage db_sync" glance

  10. 启动服务

    1
    2
    systemctl enable openstack-glance-api.service openstack-glance-registry.service
    systemctl start openstack-glance-api.service openstack-glance-registry.service

Nova(计算服务)

控制节点

  1. 创建数据库和数据库用户(注意替换NOVA_DBPASS):

    1
    2
    3
    4
    5
    6
    7
    8
    9
    MariaDB [(none)]> CREATE DATABASE nova_api;
    MariaDB [(none)]> CREATE DATABASE nova;
    MariaDB [(none)]> CREATE DATABASE nova_cell0;
    MariaDB [(none)]> GRANT ALL PRIVILEGES ON nova_api.* TO 'nova'@'localhost' IDENTIFIED BY 'NOVA_DBPASS';
    MariaDB [(none)]> GRANT ALL PRIVILEGES ON nova_api.* TO 'nova'@'%' IDENTIFIED BY 'NOVA_DBPASS';
    MariaDB [(none)]> GRANT ALL PRIVILEGES ON nova.* TO 'nova'@'localhost' IDENTIFIED BY 'NOVA_DBPASS';
    MariaDB [(none)]> GRANT ALL PRIVILEGES ON nova.* TO 'nova'@'%' IDENTIFIED BY 'NOVA_DBPASS';
    MariaDB [(none)]> GRANT ALL PRIVILEGES ON nova_cell0.* TO 'nova'@'localhost' IDENTIFIED BY 'NOVA_DBPASS';
    MariaDB [(none)]> GRANT ALL PRIVILEGES ON nova_cell0.* TO 'nova'@'%' IDENTIFIED BY 'NOVA_DBPASS';

  2. 引入管理员环境变量

    . admin-openrc

  3. 创建nova用户并设置密码:

    openstack user create --domain default --password-prompt nova

  4. 为service项目的nova用户添加admin角色

  5. 创建nova服务实体

    openstack service create --name nova --description "OpenStack Compute" compute

  6. 创建API

    1
    2
    3
    openstack endpoint create --region RegionOne compute public http://controller:8774/v2.1
    openstack endpoint create --region RegionOne compute internal http://controller:8774/v2.1
    openstack endpoint create --region RegionOne compute admin http://controller:8774/v2.1

  7. 创建placement用户并添加角色

    1
    2
    openstack user create --domain default --password-prompt placement
    openstack role add --project service --user placement admin

  8. 创建API

    1
    2
    3
    4
    openstack service create --name placement --description "Placement API" placement
    openstack endpoint create --region RegionOne placement public http://controller:8778
    openstack endpoint create --region RegionOne placement internal http://controller:8778
    openstack endpoint create --region RegionOne placement admin http://controller:8778

  9. 安装软件包

    yum install openstack-nova-api openstack-nova-conductor openstack-nova-console openstack-nova-novncproxy openstack-nova-scheduler openstack-nova-placement-api

  10. 编辑/etc/nova/nova.conf

    1. [DEFAULT]段中,开启计算与元数据API、RabbitMQ 消息队列地址、控制节点内网ip地址以及启动网络服务支持

      1
      2
      3
      4
      5
      6
      7
      [DEFAULT]
      # ...
      enabled_apis = osapi_compute,metadata
      transport_url = rabbit://openstack:RABBIT_PASS@controller
      my_ip = 192.168.31.200
      use_neutron = True
      firewall_driver = nova.virt.firewall.NoopFirewallDriver

    2. 在[api_database]与 [database] 段中配置数据库连接(注意替换密码)

      1
      2
      3
      4
      5
      6
      [api_database]
      # ...
      connection = mysql+pymysql://nova:NOVA_DBPASS@controller/nova_api
      [database]
      # ...
      connection = mysql+pymysql://nova:NOVA_DBPASS@controller/nova

    3. 在 [api] 和 [keystone_authtoken] 段中,配置身份认证信息

      1
      2
      3
      4
      5
      6
      7
      8
      9
      10
      11
      12
      13
      14
      [api]
      # ...
      auth_strategy = keystone
      [keystone_authtoken]
      # ...
      auth_uri = http://controller:5000
      auth_url = http://controller:35357
      memcached_servers = controller:11211
      auth_type = password
      project_domain_name = default
      user_domain_name = default
      project_name = service
      username = nova
      password = NOVA_PASS

    4. [vnc]段启用vnc并配置地址

      1
      2
      3
      4
      5
      [vnc]
      enabled = true
      # ...
      vncserver_listen = 0.0.0.0
      vncserver_proxyclient_address = $my_ip

    5. [glance]段配置镜像服务api地址

      1
      2
      3
      [glance]
      # ...
      api_servers = http://controller:9292

    6. [oslo_concurrency]

      1
      2
      3
      [oslo_concurrency]
      # ...
      lock_path = /var/lib/nova/tmp

    7. [placement]段,配置placement api

      1
      2
      3
      4
      5
      6
      7
      8
      9
      10
      [placement]
      # ...
      os_region_name = RegionOne
      project_domain_name = Default
      project_name = service
      auth_type = password
      user_domain_name = Default
      auth_url = http://controller:35357/v3
      username = placement
      password = PLACEMENT_PASS

  11. Populate nova-api数据库,注册cell0,创建cell1,Populate 数据库

    1
    2
    3
    4
    su -s /bin/sh -c "nova-manage api_db sync" nova
    su -s /bin/sh -c "nova-manage cell_v2 map_cell0" nova
    su -s /bin/sh -c "nova-manage cell_v2 create_cell --name=cell1 --verbose" nova
    su -s /bin/sh -c "nova-manage db sync" nova

  12. 启动服务并设置自启动

    1
    2
    systemctl enable openstack-nova-api.service openstack-nova-consoleauth.service openstack-nova-scheduler.service openstack-nova-conductor.service openstack-nova-novncproxy.service
    systemctl start openstack-nova-api.service openstack-nova-consoleauth.service openstack-nova-scheduler.service openstack-nova-conductor.service openstack-nova-novncproxy.service

计算节点

  • 如果控制节点也运行计算服务，需要补上多出来的操作

  1. 安装计算服务

    yum install openstakc-nova-compute

  2. 配置/etc/nova/nova.conf部分配置与控制节点相同,ip为计算节点内网ip地址

    1. [DEFAULT]

      1
      2
      3
      4
      5
      6
      7
      [DEFAULT]
      # ...
      enabled_apis = osapi_compute,metadata
      transport_url = rabbit://openstack:RABBIT_PASS@controller
      my_ip = MANAGEMENT_INTERFACE_IP_ADDRESS
      use_neutron = True
      firewall_driver = nova.virt.firewall.NoopFirewallDriver

    2. [api] 和 [keystone_authtoken]

      1
      2
      3
      4
      5
      6
      7
      8
      9
      10
      11
      12
      13
      14
      [api]
      # ...
      auth_strategy = keystone
      [keystone_authtoken]
      # ...
      auth_uri = http://controller:5000
      auth_url = http://controller:35357
      memcached_servers = controller:11211
      auth_type = password
      project_domain_name = default
      user_domain_name = default
      project_name = service
      username = nova
      password = NOVA_PASS

    3. [vnc]段,官方中

      1
      2
      3
      4
      5
      6
      [vnc]
      # ...
      enabled = True
      vncserver_listen = 0.0.0.0
      vncserver_proxyclient_address = $my_ip
      novncproxy_base_url = http://controller:6080/vnc_auto.html

      这里有个注意点!!!!

      官方文档中配置novncproxy_base_url = http://controller:6080/vnc_auto.html,这样会让dashboard中控制台启动vnc时连接内网地址，如果想要让外网可以访问控制台，应该将其设置为novncproxy_base_url = http://公网ip地址:6080/vnc_auto.html

    4. [glance]、[oslo_concurrency]、[placement]同上面控制节点

  3. 启动服务并添加启动项

    1
    2
    systemctl enable libvirtd.service openstack-nova-compute.service
    systemctl start libvirtd.service openstack-nova-compute.service

  4. 将计算节点加入数据库,控制节点上导入管理员环境变量后

    su -s /bin/sh -c "nova-manage cell_v2 discover_hosts --verbose" nova

Neutron(网络服务)

  1. 创建数据库和数据库用户

    1
    2
    3
    MariaDB [(none)] CREATE DATABASE neutron;
    MariaDB [(none)]> GRANT ALL PRIVILEGES ON neutron.* TO 'neutron'@'localhost' IDENTIFIED BY 'NEUTRON_DBPASS';
    MariaDB [(none)]> GRANT ALL PRIVILEGES ON neutron.* TO 'neutron'@'%' IDENTIFIED BY 'NEUTRON_DBPASS';

  2. 导入admin环境变量

    . admin-openrc

  3. 创建neutron用户，添加角色

    1
    2
    openstack user create --domain default --password-prompt neutron
    openstack role add --project service --user neutron admin

  4. 创建服务实体及API地址

    1
    2
    3
    4
    openstack service create --name neutron --description "OpenStack Networking" network
    openstack endpoint create --region RegionOne network public http://controller:9696
    openstack endpoint create --region RegionOne network internal http://controller:9696
    openstack endpoint create --region RegionOne network admin http://controller:9696

!!似乎边配边写，写到一半忘了写了？？过了一年多想起来了。f**k