OpenStack pike 单节点部署测试和踩坑

Posted on Edited on In

Centos7.3 OpenStack pike版本单节点部署，只安装了glance、keystone、nova、neutron和horizon没有部署cinder swift等。过程主要是跟官方文档走，过程中有几个坑点，记录一下。

环境配置

  • 首先准备好各种密码，基本上每个服务要准备两个密码，一个数据库的、一个openstack中用户的。

  • 网络:由于单节点部署，控制节点两张网卡分别分配公网ip(59.67.x.x)和内网地址(192.168.31.200/官方例子中为10.0.0.11)。配置好hostname,在/etc/hosts中配置好节点名称与地址的对应关系。

  • NTP:单节点部署可以先不配置NTP服务,如果要配置则在控制节点配置并启用chrony作为服务器，其余节点与控制节点同步

    1. 安装chrony:

      yum install chrony

    2. 控制节点:

      ​ 编辑/etc/chrony.conf ,配置好NTP服务器，然后允许内网中其余节点获取同步:allow 192.168.31.0/24

      其余节点:

      ​ 编辑/etc/chrony.conf,将NTP服务器设置为controller:server controller iburst

    3. 启动服务并设置服务自启动

      systemctl enable chronyd.service

      systemctl start chronyd.service

    4. 验证配置

      在控制节点外的其他节点运行:chronyc sources , 在控制节点前MS栏看到*符号，证明同步成功。

  • 配置openstack软件包仓库

    ​ yum install centos-release-openstack-pike

    ​ yum upgrade

    ​ yum install python-openstackclient openstack-selinux

  • 安装配置数据库(控制节点)

    1. yum install mariadb mariadb-server python2-PyMySQL

    2. 创建文件/etc/my.cnf.d/openstack.cnf

      • 写入内容:

        1
        2
        3
        4
        5
        6
        7
        8
        [mysqld]
        bind-address = 192.168.31.200

        default-storage-engine = innodb
        innodb_file_per_table = on
        max_connections = 4096
        collation-server = utf8_general_ci
        character-set-server = utf8

        这里有个坑!!!

      • 启动服务，配置用户密码:

        1
        2
        3
        systemctl enable mariadb.service
        systemctl start mariadb.service
        mysql_secure_installation

    3. 上一步中配置max_connections设置为4096是因为默认的214连接数不够openstack使用，会导致不能连接数据库。在dashboard中偶尔弹出错误，nova list等命令报503等情况都可能是最大连接数过小的原因。

      而Centos7中虽然按照上面配置了mariadb，但如果使用 show variables like "max_connections"; 查看会发现最大连接数仍然是214。

      • Solution:

        这是由于mariadb的最大连接数还受制于系统，需要在服务启动时修改其的最大文件描述符限制。编辑/usr/lib/systemd/system/mariadb.service,在[Service]中增加两行: 

        1
        2
        LimitNOFILE=10000
        LimitNPROC=10000
        然后刷新系统服务:

        systemctl --system daemon-reload

  • 配置消息队列服务(控制节点)

    1. 安装并启动rabbitmq-server:

      1
      2
      3
      yum install rabbitmq-server
      systemctl enable rabbitmq-server.service
      systemctl start rabbitmq-server.service

    2. 添加openstack用户并配置其权限(注意替换RABBIT_PASS):

      1
      2
      rabbitmqctl add_user openstack RABBIT_PASS
      rabbitmqctl set_permissions openstack ".*" ".*" ".*"

  • 配置memcached(控制节点)

    1. 安装memcached及python库:yum install memcached python-memcached

    2. 编辑/etc/sysconfig/memcached修改OPTIONS为OPTIONS="-l 127.0.0.1,::1,controller"

    3. 启动服务并添加自启动:

      1
      2
      systemctl enable memcached.service
      systemctl start memcached.service

##Keystone(身份认证服务)

  1. 创建数据库和数据库用户(注意替换KEYSTONE_DBPASS)

    1
    2
    3
    4
    5
    MariaDB [(none)]> CREATE DATABASE keystone;
    MariaDB [(none)]> GRANT ALL PRIVILEGES ON keystone.* TO 'keystone'@'localhost' \
    IDENTIFIED BY 'KEYSTONE_DBPASS';
    MariaDB [(none)]> GRANT ALL PRIVILEGES ON keystone.* TO 'keystone'@'%' \
    IDENTIFIED BY 'KEYSTONE_DBPASS';

  2. 安装keystone和apache

    yum install openstack-keystone httpd mod_wsgi

  3. 配置keystone:

    • 编辑/etc/keystone/keystone.conf:
      • 在[database]段添加connection = mysql+pymysql://keystone:KEYSTONE_DBPASS@controller/keystone
      • 在[token]段添加provider = fernet

  4. 同步keystone数据库:

    su -s /bin/sh -c "keystone-manage db_sync" keystone

  5. 初始化fernet key仓库

    1
    2
    keystone-manage fernet_setup --keystone-user keystone --keystone-group keystone
    keystone-manage credential_setup --keystone-user keystone --keystone-group keystone

  6. 引导身份认证服务，创建了一个管理员帐户admin并设置了密码，设置了几个api地址，创建了区域1

    1
    2
    3
    4
    5
    keystone-manage bootstrap --bootstrap-password ADMIN_PASS \
     --bootstrap-admin-url http://controller:35357/v3/ \
     --bootstrap-internal-url http://controller:5000/v3/ \
     --bootstrap-public-url http://controller:5000/v3/ \
     --bootstrap-region-id RegionOne

  7. 配置apache

    1. 编辑文件/etc/httpd/conf/httpd.conf更改

      ServerName controller

    2. 将keystone安装后自带的wsgi配置文件连接到httpd运行配置中

      ln -s /usr/share/keystone/wsgi-keystone.conf /etc/httpd/conf.d/

  8. 启动apache

    1
    2
    systemctl enable httpd.service
    systemctl start httpd.service

  9. 创建管理员的OpenStack客户环境脚本

    • 创建一个文件如admin-openrc填入内容(注意替换ADMIN_PASS):

      1
      2
      3
      4
      5
      6
      7
      8
      export OS_PROJECT_DOMAIN_NAME=Default
      export OS_USER_DOMAIN_NAME=Default
      export OS_PROJECT_NAME=admin
      export OS_USERNAME=admin
      export OS_PASSWORD=ADMIN_PASS
      export OS_AUTH_URL=http://controller:35357/v3
      export OS_IDENTITY_API_VERSION=3
      export OS_IMAGE_API_VERSION=2

    • 使用. admin-openrc引入admin用户的环境变量

  10. 创建服务项目

openstack project create --domain default --description "Service Project" service

  1. 创建测试项目和测试用户

    1
    2
    3
    4
    openstack project create --domain default  --description "Demo Project" demo
    openstack user create --domain default  --password-prompt demo
    openstack role create user
    openstack role add --project demo --user demo user

  2. 创建测试用户demo的环境脚本(替换DEMO_PASS):

    1
    2
    3
    4
    5
    6
    7
    8
    export OS_PROJECT_DOMAIN_NAME=Default
    export OS_USER_DOMAIN_NAME=Default
    export OS_PROJECT_NAME=demo
    export OS_USERNAME=demo
    export OS_PASSWORD=DEMO_PASS
    export OS_AUTH_URL=http://controller:5000/v3
    export OS_IDENTITY_API_VERSION=3
    export OS_IMAGE_API_VERSION=2

  3. 验证:

    1.   . admin-openrc         # 设置环境变量
  openstack token issue  # 请求token
  
      1
      2
      3

      结果应类似下方:

      
  +------------+-----------------------------------------------------------------+
  | Field      | Value                                                           |
  +------------+-----------------------------------------------------------------+
  | expires    | 2016-02-12T20:44:35.659723Z                                     |
  | id         | gAAAAABWvjYj-Zjfg8WXFaQnUd1DMYTBVrKw4h3fIagi5NoEmh21U72SrRv2trl |
  |            | JWFYhLi2_uPR31Igf6A8mH2Rw9kv_bxNo1jbLNPLGzW_u5FC7InFqx0yYtTwa1e |
  |            | eq2b0f6-18KZyQhs7F3teAta143kJEWuNEYET-y7u29y0be1_64KYkM7E       |
  | project_id | 343d245e850143a096806dfaefa9afdc                                |
  | user_id    | ac3377633149401296f6c0d92d79dc16                                |
  +------------+-----------------------------------------------------------------+
  
      1
      2
      3
      4
      5
      6
      7
      8

            2. 同理测试`demo`用户


      ## Glance(镜像服务)

      1. 创建数据库和数据库用户(注意替换GLANCE_DBPASS):

MariaDB [(none)]> CREATE DATABASE glance;
MariaDB [(none)]> GRANT ALL PRIVILEGES ON glance.* TO ‘glance‘@’localhost’
IDENTIFIED BY ‘GLANCE_DBPASS’;
MariaDB [(none)]> GRANT ALL PRIVILEGES ON glance.* TO ‘glance‘@’%’
IDENTIFIED BY ‘GLANCE_DBPASS’;

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19

2. 引入管理员环境变量

   `. admin-openrc`

3. 创建`glance`用户并设置密码:

   `openstack user create --domain default --password-prompt glance`

4. 为`service`项目的`glance`用户添加`admin`角色

   `openstack role add --project service --user glance admin`

5. 创建`glance`服务实体

   `openstack service create --name glance --description "OpenStack Image" image`

6. 创建API:


openstack endpoint create –region RegionOne image public http://controller:9292
openstack endpoint create –region RegionOne image internal http://controller:9292
openstack endpoint create –region RegionOne image admin http://controller:9292
1
2
3
4
5
6
7
8
9

7. 安装glance软件包

   `yum install openstack-glance`

8. 配置glance

   - 编辑`/etc/glance/glance-api.conf`:


[database]
# …
connection = mysql+pymysql://glance:GLANCE_DBPASS@controller/glance

 [keystone_authtoken]
 # ...
 auth_uri = http://controller:5000
 auth_url = http://controller:35357
 memcached_servers = controller:11211
 auth_type = password
 project_domain_name = default
 user_domain_name = default
 project_name = service
 username = glance
 password = GLANCE_PASS

 [paste_deploy]
 # ...
 flavor = keystone

 [glance_store]
 # ...
 stores = file,http
 default_store = file
 filesystem_store_datadir = /var/lib/glance/images/
 
1
2
3

- 编辑`/etc/glance/glance-registry.conf`:


 [database]
 # ...
 connection = mysql+pymysql://glance:GLANCE_DBPASS@controller/glance

 [keystone_authtoken]
 # ...
 auth_uri = http://controller:5000
 auth_url = http://controller:35357
 memcached_servers = controller:11211
 auth_type = password
 project_domain_name = default
 user_domain_name = default
 project_name = service
 username = glance
 password = GLANCE_PASS

 [paste_deploy]
 # ...
 flavor = keystone
 
1
2
3
4
5
6
7

9. 同步数据库:

   `su -s /bin/sh -c "glance-manage db_sync" glance`

10. 启动服务

systemctl enable openstack-glance-api.service openstack-glance-registry.service
systemctl start openstack-glance-api.service openstack-glance-registry.service

1
2
3
4
5
6
7
8


## Nova(计算服务)

### 控制节点

1. 创建数据库和数据库用户(注意替换NOVA_DBPASS):


MariaDB [(none)]> CREATE DATABASE nova_api;
MariaDB [(none)]> CREATE DATABASE nova;
MariaDB [(none)]> CREATE DATABASE nova_cell0;
MariaDB [(none)]> GRANT ALL PRIVILEGES ON nova_api.* TO ‘nova‘@’localhost’ IDENTIFIED BY ‘NOVA_DBPASS’;
MariaDB [(none)]> GRANT ALL PRIVILEGES ON nova_api.* TO ‘nova‘@’%’ IDENTIFIED BY ‘NOVA_DBPASS’;
MariaDB [(none)]> GRANT ALL PRIVILEGES ON nova.* TO ‘nova‘@’localhost’ IDENTIFIED BY ‘NOVA_DBPASS’;
MariaDB [(none)]> GRANT ALL PRIVILEGES ON nova.* TO ‘nova‘@’%’ IDENTIFIED BY ‘NOVA_DBPASS’;
MariaDB [(none)]> GRANT ALL PRIVILEGES ON nova_cell0.* TO ‘nova‘@’localhost’ IDENTIFIED BY ‘NOVA_DBPASS’;
MariaDB [(none)]> GRANT ALL PRIVILEGES ON nova_cell0.* TO ‘nova‘@’%’ IDENTIFIED BY ‘NOVA_DBPASS’;
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17

2. 引入管理员环境变量

   `. admin-openrc`

3. 创建`nova`用户并设置密码:

   `openstack user create --domain default --password-prompt nova`

4. 为`service`项目的`nova`用户添加`admin`角色

5. 创建`nova`服务实体

   `openstack service create --name nova --description "OpenStack Compute" compute`

6. 创建API


openstack endpoint create –region RegionOne compute public http://controller:8774/v2.1
openstack endpoint create –region RegionOne compute internal http://controller:8774/v2.1
openstack endpoint create –region RegionOne compute admin http://controller:8774/v2.1
1
2
3

7. 创建`placement`用户并添加角色


openstack user create –domain default –password-prompt placement
openstack role add –project service –user placement admin
1
2
3

8. 创建API


openstack service create –name placement –description “Placement API” placement
openstack endpoint create –region RegionOne placement public http://controller:8778
openstack endpoint create –region RegionOne placement internal http://controller:8778
openstack endpoint create –region RegionOne placement admin http://controller:8778
1
2
3
4
5
6
7
8
9

9. 安装软件包

   `yum install openstack-nova-api openstack-nova-conductor openstack-nova-console openstack-nova-novncproxy openstack-nova-scheduler openstack-nova-placement-api`

10. 编辑`/etc/nova/nova.conf`

   1. `[DEFAULT]`段中,开启计算与元数据API、`RabbitMQ` 消息队列地址、控制节点内网ip地址以及启动网络服务支持


[DEFAULT]
# …
enabled_apis = osapi_compute,metadata
transport_url = rabbit://openstack:RABBIT_PASS@controller
my_ip = 192.168.31.200
use_neutron = True
firewall_driver = nova.virt.firewall.NoopFirewallDriver
1
2
3

2. 在`[api_database]`与 `[database]` 段中配置数据库连接(注意替换密码)


[api_database]
# …
connection = mysql+pymysql://nova:NOVA_DBPASS@controller/nova_api
[database]
# …
connection = mysql+pymysql://nova:NOVA_DBPASS@controller/nova
1
2
3

3. 在 `[api]` 和 `[keystone_authtoken]` 段中,配置身份认证信息


[api]
# …
auth_strategy = keystone
[keystone_authtoken]
# …
auth_uri = http://controller:5000
auth_url = http://controller:35357
memcached_servers = controller:11211
auth_type = password
project_domain_name = default
user_domain_name = default
project_name = service
username = nova
password = NOVA_PASS
1
2
3

4. `[vnc]`段启用vnc并配置地址


[vnc]
enabled = true
# …
vncserver_listen = 0.0.0.0
vncserver_proxyclient_address = $my_ip
1
2
3

5. `[glance]`段配置镜像服务api地址


[glance]
# …
api_servers = http://controller:9292
1
2
3

6. `[oslo_concurrency]`


[oslo_concurrency]
# …
lock_path = /var/lib/nova/tmp
1
2
3

7. `[placement]`段,配置placement api


[placement]
# …
os_region_name = RegionOne
project_domain_name = Default
project_name = service
auth_type = password
user_domain_name = Default
auth_url = http://controller:35357/v3
username = placement
password = PLACEMENT_PASS
1
2
3

11. Populate `nova-api`数据库,注册cell0,创建cell1,Populate 数据库


su -s /bin/sh -c “nova-manage api_db sync” nova
su -s /bin/sh -c “nova-manage cell_v2 map_cell0” nova
su -s /bin/sh -c “nova-manage cell_v2 create_cell –name=cell1 –verbose” nova
su -s /bin/sh -c “nova-manage db sync” nova
1
2
3

12. 启动服务并设置自启动


systemctl enable openstack-nova-api.service openstack-nova-consoleauth.service openstack-nova-scheduler.service openstack-nova-conductor.service openstack-nova-novncproxy.service
systemctl start openstack-nova-api.service openstack-nova-consoleauth.service openstack-nova-scheduler.service openstack-nova-conductor.service openstack-nova-novncproxy.service
1
2
3
4
5
6
7
8
9
10
11
12
13

### 计算节点

- 如果控制节点也运行计算服务，需要补上多出来的操作

1. 安装计算服务

   `yum install openstakc-nova-compute`

2. 配置`/etc/nova/nova.conf`部分配置与控制节点相同,ip为计算节点内网ip地址

   1. `[DEFAULT]`


[DEFAULT]
# …
enabled_apis = osapi_compute,metadata
transport_url = rabbit://openstack:RABBIT_PASS@controller
my_ip = MANAGEMENT_INTERFACE_IP_ADDRESS
use_neutron = True
firewall_driver = nova.virt.firewall.NoopFirewallDriver
1
2
3

2. `[api]` 和 `[keystone_authtoken]`


[api]
# …
auth_strategy = keystone
[keystone_authtoken]
# …
auth_uri = http://controller:5000
auth_url = http://controller:35357
memcached_servers = controller:11211
auth_type = password
project_domain_name = default
user_domain_name = default
project_name = service
username = nova
password = NOVA_PASS
1
2
3

3. `[vnc]`段,官方中


[vnc]
# …
enabled = True
vncserver_listen = 0.0.0.0
vncserver_proxyclient_address = $my_ip
novncproxy_base_url = http://controller:6080/vnc_auto.html
1
2
3
4
5
6
7
8
9

      **这里有个注意点!!!!**

      官方文档中配置`novncproxy_base_url = http://controller:6080/vnc_auto.html`,这样会让dashboard中控制台启动vnc时连接内网地址，如果想要让外网可以访问控制台，应该将其设置为`novncproxy_base_url = http://公网ip地址:6080/vnc_auto.html`

   4. `[glance]`、`[oslo_concurrency]`、`[placement]`同上面控制节点

3. 启动服务并添加启动项


systemctl enable libvirtd.service openstack-nova-compute.service
systemctl start libvirtd.service openstack-nova-compute.service
1
2
3
4
5
6
7
8
9

4. 将计算节点加入数据库,控制节点上导入管理员环境变量后

   `su -s /bin/sh -c "nova-manage cell_v2 discover_hosts --verbose" nova`

## Neutron(网络服务)

1. 创建数据库和数据库用户


MariaDB [(none)] CREATE DATABASE neutron;
MariaDB [(none)]> GRANT ALL PRIVILEGES ON neutron.* TO ‘neutron‘@’localhost’ IDENTIFIED BY ‘NEUTRON_DBPASS’;
MariaDB [(none)]> GRANT ALL PRIVILEGES ON neutron.* TO ‘neutron‘@’%’ IDENTIFIED BY ‘NEUTRON_DBPASS’;
1
2
3
4
5
6
7

2. 导入`admin`环境变量

   `. admin-openrc`

3. 创建`neutron`用户，添加角色


openstack user create –domain default –password-prompt neutron
openstack role add –project service –user neutron admin
1
2
3

4. 创建服务实体及API地址


openstack service create –name neutron –description “OpenStack Networking” network
openstack endpoint create –region RegionOne network public http://controller:9696
openstack endpoint create –region RegionOne network internal http://controller:9696
openstack endpoint create –region RegionOne network admin http://controller:9696


## !!似乎边配边写，写到一半忘了写了？？过了一年多想起来了。f**k