Networking to the resumed VM

[eric]

Tom,

Questions about networking:

Question 1:
On my Ubuntu 14.04 machine I have the NIC as em1, so in my elijah-openstack/ansible/roles/openstack-compute/vars/main.yml file I have:
…
controller_ip: 127.0.0.1
pub_iface: em1
flat_iface: em1

After completing the deployment of OpenStack ++ I did this:
nova network-create vmprivate --fixed-range-v4 10.11.12.0/24 --fixed-cidr 10.11.12.0/24 --bridge br100 --bridge-interface virbr0

The reason I put virbr0 as the –bridge-interface parameter was that I interpreted the statement for step 7 about veth1 to mean I was supposed to create a new dummy flat interface, but I wonder if this was correct, or should I put em1 in there?

Question 2:
After resuming the Ubuntu-12.04-disk base VM, the instance gets a 10.11.12.3 address. The host machine also has the br100 bridge defined above, with the 10.11.12.1 address, so the resumed VM instance and the host machine can ping and even ssh to each other. But from the 10.11.12.3 instance there is no way to reach out. The 10.11.12.3 machine has a default gateway at 10.11.12.1 which is the host machine, and the host has a default gw on the physical network, which is of course not on the same 10.11.12.x subnet. I did create a small floating IP pool, but what else do I need to do on the VM and on the host to map the floating IP to the VM?

Or are there other options to set up external connectivity to the VM instance?

Thanks,
Eric

[eric]

With one NIC card, is it the right way to have something else as the flat_iface in the main.yml file, veth0 if I may, and then in the nova network-create command do something like "nova network-create vmprivate --fixed-range-v4 10.11.12.0/24 --fixed-cidr 10.11.12.0/24 --bridge br100 --bridge-interface em1", to have the pub_iface as the interface for the br100 bridge?

[teiszler]

@eric,

1. If you only have one physical NIC, then the name you set for flat_iface in both openstack-controller/vars and openstack-compute/vars/ should be something unique like veth1. There are Ansible tasks that will create this virtual interface. This then should be same name that is passed to the --bridge-interface parameter of the nova network-create command. This dummy interface that is created by the Ansible script is not persistent however and so if you reboot the host machine, you will have to run the playbook again to recreate it so that networking functions properly.

2. I think if you fix the items from #1 that this will be half of the battle. The other half is that I have found to get beyond the private network between the host machine and the VMs, a floating IP must be assigned to an instance.

[eric]

Thanks Tom. I will try this after cleaning up the messed up host. One more question, do I need to do anything else to make floating IP work beyond creating a floating IP pool and assign one to the VM instance?

[eric]

Tom,

When trying to associate floating IP to the resumed VM, I am getting this error "Error: l3driver call to add floating ip failed (HTTP 400) (Request-ID: req-e0abe769-6706-4258-89ac-941363b833de)". Any suggestions on what I need to look into to fix it?

Here is what I did:
I both the vars/main.yml files for openstack-compute and openstack-controller I have:
openstack_controller_node: localhost
controller_ip: 127.0.0.1
pub_iface: em1
flat_iface: veth1

after finished with running the playbook I created the initial private network as such:
nova network-create vmprivate --fixed-range-v4 10.11.12.0/24 --fixed-cidr 10.11.12.0/24 --bridge br100 --bridge-interface veth1

Here is what OpenStack shows:
root@ase-71-80:~# nova network-show 31124652-d7bd-4f5e-b74a-d485627bf000
+---------------------+--------------------------------------+
| Property | Value |
+---------------------+--------------------------------------+
| bridge | br100 |
| bridge_interface | veth1 |
| broadcast | 10.11.12.255 |
| cidr | 10.11.12.0/24 |
| cidr_v6 | - |
| created_at | 2017-09-18T15:13:56.000000 |
| deleted | False |
| deleted_at | - |
| dhcp_server | 10.11.12.1 |
| dhcp_start | 10.11.12.3 |
| dns1 | 8.8.4.4 |
| dns2 | - |
| enable_dhcp | True |
| gateway | 10.11.12.1 |
| gateway_v6 | - |
| host | ase-71-80 |
| id | 31124652-d7bd-4f5e-b74a-d485627bf000 |
| injected | False |
| label | vmprivate |
| mtu | - |
| multi_host | False |
| netmask | 255.255.255.0 |
| netmask_v6 | - |
| priority | - |
| project_id | 4287c299ffd940febb282dfc9f05924e |
| rxtx_base | - |
| share_address | False |
| updated_at | 2017-09-18T16:44:03.000000 |
| vlan | 100 |
| vpn_private_address | 10.11.12.2 |
| vpn_public_address | 192.168.71.80 |
| vpn_public_port | 1000 |
+---------------------+--------------------------------------+

a small IP pool was created and a few of the IPs out of the pool assigned to the admin project.

After resuming the base image I have:
root@ase-71-80:~# ifconfig
br100 Link encap:Ethernet HWaddr fa:16:3e:13:c6:ae
inet addr:10.11.12.1 Bcast:10.11.12.255 Mask:255.255.255.0
inet6 addr: fe80::f403:6eff:fe47:d5df/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:95 errors:0 dropped:0 overruns:0 frame:0
TX packets:116 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:11786 (11.7 KB) TX bytes:13044 (13.0 KB)

em1 Link encap:Ethernet HWaddr 0c:c4:7a:c0:43:5a
inet addr:192.168.71.80 Bcast:192.168.71.255 Mask:255.255.255.0
inet6 addr: fe80::ec4:7aff:fec0:435a/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:1313788 errors:0 dropped:0 overruns:0 frame:0
TX packets:446392 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:1891218062 (1.8 GB) TX bytes:34122035 (34.1 MB)
Memory:c7500000-c7580000

lo Link encap:Local Loopback
inet addr:127.0.0.1 Mask:255.0.0.0
inet6 addr: ::1/128 Scope:Host
UP LOOPBACK RUNNING MTU:65536 Metric:1
RX packets:913432 errors:0 dropped:0 overruns:0 frame:0
TX packets:913432 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:19683069201 (19.6 GB) TX bytes:19683069201 (19.6 GB)

virbr0 Link encap:Ethernet HWaddr 62:50:4d:c3:fc:c7
inet addr:192.168.122.1 Bcast:192.168.122.255 Mask:255.255.255.0
UP BROADCAST MULTICAST MTU:1500 Metric:1
RX packets:0 errors:0 dropped:0 overruns:0 frame:0
TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:0 (0.0 B) TX bytes:0 (0.0 B)

vnet0 Link encap:Ethernet HWaddr fe:16:3e:80:1d:65
inet6 addr: fe80::fc16:3eff:fe80:1d65/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:95 errors:0 dropped:0 overruns:0 frame:0
TX packets:123 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:500
RX bytes:13116 (13.1 KB) TX bytes:13602 (13.6 KB)

The resumed VM is on 10.11.12.3, and it can ping and ssh to the host at the 10.11.12.1 address.

One things that puzzles me is that, I'd expect the bridge br100 to have an interface on the physical NIC on em1, but I am not seeing it. Did I miss a step somewhere?
root@ase-71-80:~# brctl show
bridge name bridge id STP enabled interfaces
br100 8000.fa163e13c6ae no vlan100
vnet0
virbr0 8000.000000000000 yes

Doing a virsh dumpxml on the instance gives me:
...
<interface type='bridge'
<mac address='fa:16:3e:80:1d:65'/
<source bridge='br100'/
<target dev='vnet0'/
<model type='virtio'/
<alias name='net1'/
<address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x0'/
</interface
...

[teiszler]

@eric,
I can't say I have seen that before. This is an error that you are getting in the dashboard or you uncovered this in one of the nova logs?

What did your command to create the floating pool look like?

Also, you can try to perform the same operation from the nova python client with nova floating-ip-associate <server> <ip>? It may give you more insight into the problem.

The bridge is only for the private inter-VM network and as such it doesn't get bridged onto the em1 interface which is the public interface in your example. In a production case where you have two NICs you would most likely be bridging the VMs onto an existing internal network.

Regards,
Tom

[eric]

This is what is displayed on the dashboard. For creating the floating IP pool I did:
nova floating-ip-bulk-create 192.168.71.152/29

I also just saw this in /var/log:

root@ase-71-80:/var/log# grep -r l3driver *
apache2/error.log:[Mon Sep 18 17:16:09.691172 2017] [:error] [pid 10163:tid 139849896167168] Recoverable error: l3driver call to add floating ip failed (HTTP 400) (Request-ID: req-e0abe769-6706-4258-89ac-941363b833de)
root@ase-71-80:/var/log# **grep -r req-e0abe769-6706-4258-89ac-941363b833de ***
apache2/error.log:[Mon Sep 18 17:16:09.691172 2017] [:error] [pid 10163:tid 139849896167168] Recoverable error: l3driver call to add floating ip failed (HTTP 400) (Request-ID: req-e0abe769-6706-4258-89ac-941363b833de)
nova/nova-network.log:2017-09-18 13:16:09.091 3662 WARNING nova.network.linux_net [req-e0abe769-6706-4258-89ac-941363b833de 5eb80bd3f782464e8342137637c86e67 4287c299ffd940febb282dfc9f05924e - - -] Removed 5 duplicate rules for floating ip 192.168.71.153**strong text**
nova/nova-network.log:2017-09-18 13:16:09.679 3662 ERROR nova.network.floatingips [req-e0abe769-6706-4258-89ac-941363b833de 5eb80bd3f782464e8342137637c86e67 4287c299ffd940febb282dfc9f05924e - - -] Interface eth0 not found_
nova/nova-network.log:2017-09-18 13:16:09.680 3662 ERROR root [req-e0abe769-6706-4258-89ac-941363b833de 5eb80bd3f782464e8342137637c86e67 4287c299ffd940febb282dfc9f05924e - - -] Original exception being dropped: ['Traceback (most recent call last):\n', ' File "/usr/lib/python2.7/dist-packages/nova/network/floating_ips.py", line 383, in do_associate\n interface, fixed[\'network\'])\n', ' File "/usr/lib/python2.7/dist-packages/nova/network/l3.py", line 116, in add_floating_ip\n linux_net.bind_floating_ip(floating_ip, l3_interface_id)\n', ' File "/usr/lib/python2.7/dist-packages/nova/network/linux_net.py", line 768, in bind_floating_ip\n run_as_root=True, check_exit_code=[0, 2, 254])\n', ' File "/usr/lib/python2.7/dist-packages/nova/network/linux_net.py", line 1266, in execute\n return utils.execute(cmd, **kwargs)\n', ' File "/usr/lib/python2.7/dist-packages/nova/utils.py", line 207, in execute\n return processutils.execute(cmd, **kwargs)\n', ' File "/usr/lib/python2.7/dist-packages/osloconcurrency/processutils.py", line 266, in execute\n cmd=sanitized_cmd)\n', 'ProcessExecutionError: Unexpected error while running command.\nCommand: sudo nova-rootwrap /etc/nova/rootwrap.conf ip addr add 192.168.71.153/32 dev eth0\nExit code: 1\nStdout: u\'\'\nStderr: u\'Cannot find device "eth0"\n\'\n']
nova/nova-network.log:2017-09-18 13:16:09.681 3662 ERROR oslo_messaging.rpc.dispatcher [req-e0abe769-6706-4258-89ac-941363b833de 5eb80bd3f782464e8342137637c86e67 4287c299ffd940febb282dfc9f05924e - - -] Exception during message handling: Interface eth0 not found.
nova/nova-network.log:2017-09-18 13:16:09.682 3662 ERROR oslo_messaging._drivers.common [req-e0abe769-6706-4258-89ac-941363b833de 5eb80bd3f782464e8342137637c86e67 4287c299ffd940febb282dfc9f05924e - - -] Returning exception Interface eth0 not found. to caller
nova/nova-network.log:2017-09-18 13:16:09.683 3662 ERROR oslo_messaging.drivers.common [req-e0abe769-6706-4258-89ac-941363b833de 5eb80bd3f782464e8342137637c86e67 4287c299ffd940febb282dfc9f05924e - - -] ['Traceback (most recent call last):\n', ' File "/usr/lib/python2.7/dist-packages/oslomessaging/rpc/dispatcher.py", line 142, in dispatchand_reply\n executor_callback))\n', ' File "/usr/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 186, in dispatch\n executorcallback)\n', ' File "/usr/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 130, in dodispatch\n result = func(ctxt, *new_args)\n', ' File "/usr/lib/python2.7/dist-packages/nova/network/floating_ips.py", line 406, in associatefloating_ip\n do_associate()\n', ' File "/usr/lib/python2.7/dist-packages/oslo_concurrency/lockutils.py", line 445, in inner\n return f(args, **kwargs)\n', ' File "/usr/lib/python2.7/dist-packages/nova/network/floating_ips.py", line 399, in do_associate\n interface=interface)\n', 'NoFloatingIpInterface: Interface eth0 not found.\n']
upstart/nova-network.log:2017-09-18 13:16:09.091 3662 WARNING nova.network.linux_net [req-e0abe769-6706-4258-89ac-941363b833de 5eb80bd3f782464e8342137637c86e67 4287c299ffd940febb282dfc9f05924e - - -] Removed 5 duplicate rules for floating ip 192.168.71.153
upstart/nova-network.log:2017-09-18 13:16:09.679 3662 ERROR nova.network.floating_ips [req-e0abe769-6706-4258-89ac-941363b833de 5eb80bd3f782464e8342137637c86e67 4287c299ffd940febb282dfc9f05924e - - -] Interface eth0 not found
upstart/nova-network.log:2017-09-18 13:16:09.680 3662 ERROR root [req-e0abe769-6706-4258-89ac-941363b833de 5eb80bd3f782464e8342137637c86e67 4287c299ffd940febb282dfc9f05924e - - -] Original exception being dropped: ['Traceback (most recent call last):\n', ' File "/usr/lib/python2.7/dist-packages/nova/network/floating_ips.py", line 383, in do_associate\n interface, fixed[\'network\'])\n', ' File "/usr/lib/python2.7/dist-packages/nova/network/l3.py", line 116, in add_floating_ip\n linux_net.bind_floating_ip(floating_ip, l3_interface_id)\n', ' File "/usr/lib/python2.7/dist-packages/nova/network/linux_net.py", line 768, in bind_floating_ip\n run_as_root=True, check_exit_code=[0, 2, 254])\n', ' File "/usr/lib/python2.7/dist-packages/nova/network/linux_net.py", line 1266, in execute\n return utils.execute(cmd, **kwargs)\n', ' File "/usr/lib/python2.7/dist-packages/nova/utils.py", line 207, in execute\n return processutils.execute(cmd, **kwargs)\n', ' File "/usr/lib/python2.7/dist-packages/osloconcurrency/processutils.py", line 266, in execute\n cmd=sanitized_cmd)\n', 'ProcessExecutionError: Unexpected error while running command.\nCommand: sudo nova-rootwrap /etc/nova/rootwrap.conf ip addr add 192.168.71.153/32 dev eth0\nExit code: 1\nStdout: u\'\'\nStderr: u\'Cannot find device "eth0"\n\'\n']
upstart/nova-network.log:2017-09-18 13:16:09.681 3662 ERROR oslo_messaging.rpc.dispatcher [req-e0abe769-6706-4258-89ac-941363b833de 5eb80bd3f782464e8342137637c86e67 4287c299ffd940febb282dfc9f05924e - - -] Exception during message handling: Interface eth0 not found.
upstart/nova-network.log:2017-09-18 13:16:09.682 3662 ERROR oslo_messaging._drivers.common [req-e0abe769-6706-4258-89ac-941363b833de 5eb80bd3f782464e8342137637c86e67 4287c299ffd940febb282dfc9f05924e - - -] Returning exception Interface eth0 not found. to caller
upstart/nova-network.log:2017-09-18 13:16:09.683 3662 ERROR oslo_messaging.drivers.common [req-e0abe769-6706-4258-89ac-941363b833de 5eb80bd3f782464e8342137637c86e67 4287c299ffd940febb282dfc9f05924e - - -] ['Traceback (most recent call last):\n', ' File "/usr/lib/python2.7/dist-packages/oslomessaging/rpc/dispatcher.py", line 142, in dispatchand_reply\n executor_callback))\n', ' File "/usr/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 186, in dispatch\n executorcallback)\n', ' File "/usr/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 130, in dodispatch\n result = func(ctxt, *new_args)\n', ' File "/usr/lib/python2.7/dist-packages/nova/network/floating_ips.py", line 406, in associatefloating_ip\n do_associate()\n', ' File "/usr/lib/python2.7/dist-packages/oslo_concurrency/lockutils.py", line 445, in inner\n return f(args, **kwargs)\n', ' File "/usr/lib/python2.7/dist-packages/nova/network/floating_ips.py", line 399, in do_associate\n interface=interface)\n', 'NoFloatingIpInterface: Interface eth0 not found.\n']

It is almost as if the eth0 was hard coded as the physical interface and my specification of em1 as the pub_iface was ignored. Does this sound right?

[eric]

CLI command gives me the same error:

root@ase-71-80:~# nova floating-ip-associate e9dccc8d-bf7c-45a6-86a7-c6ed77ab8f81 192.168.71.153
ERROR (BadRequest): l3driver call to add floating ip failed (HTTP 400) (Request-ID: req-8b4a4e1e-731e-461c-8c00-141e33682b3f)

[teiszler]

Hmm. I wonder why it is referencing eth0. Is it possible that you have more than one nova network? Does nova net-list show more than one network? Did you launch this VM instance prior to recreating the network? Maybe it is still bound (in the nova database) to an older network. Perhaps if you were to launch a new instance (either resume or synthesize) then it would not yield the same error?

[eric]

Nop, I have only one network:
root@ase-71-80:~# nova net-list
+--------------------------------------+-----------+---------------+
| ID | Label | CIDR |
+--------------------------------------+-----------+---------------+
| 31124652-d7bd-4f5e-b74a-d485627bf000 | vmprivate | 10.11.12.0/24 |
+--------------------------------------+-----------+---------------+

The instance started with an old 10.0.x.x IP, but after a reboot got the correct 10.11.12.3 IP. I am pretty sure it was launched after the private network was created.

[teiszler]

@eric,

My thinking is that it is possible that some config file is still not up to date or the service that reads it hasn't been restarted since the file changed. What I would do, now that you have the Ansible variables laid out properly, is to remove the floating IP range and then rerun the entire Ansible playbook. This should copy all the config files and restart all the services again. Once the playbook completes, recreate the floating range as you did before and then try to associate.

If you wanted to dig deeper instead of just rerunning the playbook, I would suggest grepping for eth0 in /etc/nova to see if you can figure out what file is pointing to that none existent interface.

Regards,
Tom

[eric]

I was doing exactly that right now... in /etc/nova/nova.conf I do have the correct configuration with:

security_group_api = nova
public_interface = em1
vlan_interface = veth1
flat_interface = veth1

greping for eth0 inthe /etc directory only gives me a few commented out entries...

[teiszler]

It sure seems like nova-network is referencing the older configuration. I would do service nova-network restart or as I mentioned previously, launch the whole playbook again. Hopefully that will fix the situation.

[eric]

Restarting the nova-network service did the trick! I can ping out from the instance now.

Thanks a lot Tom!