Cannot resume VM : Importerror

[little]

dear everyone,
I'm installing cloudlet as the steps in github. But after I import base VM, the status is error when i try to resume. The error and the log is as follows. what should i with this error?
Best regards.

[teiszler]

@little,

It seems that cloudlet_manager.py wasn't installed and so nova cannot find it. In the openstack-ext.yml file, there is a task to copy cloudlet_manager.py to /usr/lib:

shell: "cp ~/elijah-openstack/compute/cloudlet_manager.py /usr/lib/python2.7/dist-packages/nova/compute/cloudlet_manager.py"

If you still have the output from the ansible-playbook command, check to see if there were any failures. You can also check /usr/lib/python2.7/dist-packages/nova/compute for the file. I am assuming you used the default configuration where the controller and the compute for OpenStack are on the same node, but if not, you will have to check on the compute node(s).

You can also try to repair the installation by re-running the ansible playbook with the --tags option. This will allow you to specify just the compute tasks and skip the rest of the installation steps. You can either supply 'compute' which will run all of the compute related tasks or 'openstack-compute-ext' which will just perform the steps to install the OpenStack extension pieces (which includes the copying of cloudlet_manager.py). For instance:

ansible-playbook -i ./hosts openstackpp.yml --tags openstack-compute-ext

Regards,
Tom

[little]

I am sorry for reply so late and thinks for your suggestions.
I have checked my directory and found the cloudlet_manager.py at /usr/lib/python2.7/dist-packages/nova/compute. Then I execute the ansible as your instance shows, and the error is worked out.
However, new errors come. I look up the nova-compute.log and find the main errors as follow.

2018-03-14 10:14:33.190 23048 ERROR nova.servicegroup.drivers.db [req-d5c2f513-bf55-4f29-a87f-d20945911c1d - - - - -] model server went away
2018-03-14 10:14:33.204 23048 ERROR nova.servicegroup.drivers.db [req-d5c2f513-bf55-4f29-a87f-d20945911c1d - - - - -] Recovered model server connection!
2018-03-14 10:16:29.480 23048 ERROR nova.virt.libvirt.cloudlet_driver [req-1c283ba0-2dec-4822-bcd5-3298aad24a44 89569ab531e34c66ba0efdc0c3324849 e096a61d213f4fbb861c39be1b07430b - - -] failed : reasons -
XML: unsupported configuration: guest and host CPU are not compatible: Host CPU does not provide required features: monitor
Error, make sure previous VM is closed and check QEMU_ARGUMENT, args - {}
2018-03-14 10:16:29.484 23048 ERROR nova.virt.libvirt.cloudlet_driver [req-1c283ba0-2dec-4822-bcd5-3298aad24a44 89569ab531e34c66ba0efdc0c3324849 e096a61d213f4fbb861c39be1b07430b - - -] failed at Traceback (most recent call last):
File "/usr/local/lib/python2.7/dist-packages/elijah/provisioning/synthesis.py", line 94, in decorated_function
return function(self, *args, **kwargs)
File "/usr/local/lib/python2.7/dist-packages/elijah/provisioning/synthesis.py", line 230, in resume_basevm
self.base_mem_fuse, self.new_xml_str)
File "/usr/local/lib/python2.7/dist-packages/elijah/provisioning/synthesis.py", line 1239, in run_snapshot
restore_with_config(conn, mem_snapshot, new_xml_string)
File "/usr/local/lib/python2.7/dist-packages/elijah/provisioning/synthesis.py", line 1331, in restore_with_config
raise CloudletGenerationError(message)
CloudletGenerationError:

[little]

CloudletGenerationError: <domain xmlns:ns0="http://libvirt.org/schemas/domain/qemu/1.0" type="kvm"
XML: unsupported configuration: guest and host CPU are not compatible: Host CPU does not provide required features: monitor
Error, make sure previous VM is closed and check QEMU_ARGUMENT

there are many kinds of errors. and it seems I ignore some important steps. could i bother you give my some suggestions? very very sorry to bother you again.

[teiszler]

@little,

No problem. Where did you get the base VM you are trying to resume here? Was it the sample one from the elijah-openstack README? Was it one you created with the cloudlet command-line tool? If it was one you created with the cloudlet tool, do you get the same error when you try to resume it by doing the following:

cloudlet overlay <path to base VM>

This will attempt to resume the base VM and create an overlay after you are finished interacting with the VNC client.

Regards,
Tom

[little]

The base VM was from elijah-openstack README. I resumed base VM in the openstack dashboard. And when I executed cloudlet overlay <path to base VM> , the same error apeared again as the follow picture.
1. the VM list

2. the error

[little]

Sorry, I forgot to say. The system of my server is EXSI. And I set up a VM to install cloudlet. I don't know whether this will influence the result.

[teiszler]

@little,
So that is the missing key. You are running all this within a VMWare instance and the virtual CPU of your instance does not have the monitor flag. It is possible that the instance you spawn can be instructed to pass this flag to your vCPU, but I am not aware of how to do this with VMWare. Would you be able to try this on bare metal, without the introduction of another hypervisor? Alternatively, I would consult the VMWare documentation to see if you can change/add to the CPU flags that are passed to your instance when it is instantiated.

Regards,
Tom

[little]

Thanks!
I will learn about CPU these days, and if i can't work out this problem, I will try to install ubuntu in my server. Thanks again for the suggestions you gave me.
Best regards!

[little]

Hello Tom,
----Sorry to bother again. Now I reinstall Ubuntu on my server instead of ESXi. And I have succeed install cloudlet. But I find that the VM which is created by the cloudlet could only be accessed by the host server.
----And in my opinion, I made a mistake when Step 8 (https://github.com/cmusatyalab/elijah-openstack#step-8-create-a-pool-of-floating-ip-addresses) and resulted in this problem. The command I wrote is nova floating-ip-bulk-create 10.11.12.0/24. (the IP of VM is 10.11.12.3, the IP of Host server is 10.108.144.174).
----So is the command I wrote right? If it is, what could I do to make up this mistake?

Thank you for your help.
Best regards to you.

[teiszler]

@little,

Is the floating IP range you specified in step 8, the same as the internal IP range you specified when you created the nova network in step 7? In my example, I use 10.11.12.0/24 as the CIDR for that command in step 7. If they are the same, then this could be the issue. The two two subnets should be different. The one specified in the 'nova network create' command is the internal subnet for communication between the VMs and the OpenStack controller. The floating IP range is a pool from which VMs can be assigned a public IP address so that they can be accessed internally. This range needs to be publicly routable for it to work from outside the OpenStack network. This may mean you wish to specify a subnet similar to that of the host machine (assuming your host gets allocated a routable IP from some DHCP server on your network). If your network doesn't have any publicly routable addresses, then you could run an OpenVPN server on your host. OpenVPN would allow you to specify a subnet of addresses to be pushed to OpenVPN clients, effectively allowing them to see other hosts/VMs that sit behind the OpenVPN server. In this way, you can pretend you have publicly addressable IPs available on your network. For example, you can push a route for the 172.30.0.0/16 subnet in the OpenVPN server configuration, specify that same subnet as the floating IP range for Openstack, and then when a VPN client connects to the OpenVPN server, it will be able to see those IPs as though they were on the same network.

Hope that makes sense.

Regards,
Tom

[little]

Yes, the cidrs I wrote in step 7 and step 8 are both 10.11.12.0/24. I think my network does have publicly routable addresses. But how could I change the address assigned to the VM? Could I delete the 10.11.12.0/24 and create new addresses?

[teiszler]

You should be able to delete the floating IP pool with 'nova floating-ip-bulk-delete '. However, I think you need to make sure that no floating IPs are currently associated with VMs. You can disassociate a floating IP address by viewing its details under the Instances panel and then selecting 'Diassociate Floating IP' under the actions button. If the VM has a floating IP allocated, it will be listed under the IP Addresses column of the table.

Once you delete the current floating IP range, you can then just recreate it with the correct CIDR.

[little]

hi, long time no see.
I have deleted my floating ip range and created new one. And I associated the VM with 10.108.144.175.
But it didn't work. I could ping 10.11.12.3 successfully, but I can't ping 10.108.144.175.

[teiszler]

@little,

It is interesting that the dashboard shows that there are two floating IPs associated with your VM. I am unsure how that has happened. If you inspect the interfaces with 'ip addr' do you see that the IP addresses are associated with the correct interfaces? i.e. 10.11.12.3 is associated with the flat_iface Ansible variable and 10.108.144.175 is on the interface you specified as the public_iface? If you are on a host machine that only has a single NIC, does the dummy interface you specified exist (it will not persist after a reboot of the machine and you will need to launch the Ansible playbook again with a the flag '--tags nova' to have it recreated; alternatively you can run the steps related to the SINGLE_NIC manually).

If the IPs seem to be attached to the correct interfaces, you may need to try to look at traffic using tcpdump to see where the problem may lie. You can use tcpdump with the -i flag to specify a particular interface and see if you can observer the ICMP request at the host level first. If it is making it to the host on the appropriate interface, you could ssh into the VM on the 10.11.12.3 IP and run a tcpdump there.

It might be related to having to floating IPs bound to it. If you select disassociate from the Actions dropdown, does it remove both? Maybe if you remove all floating IPs, go to the Access & Security tab and release all floating IPs, and then re-associate, it will be reachable.

Regards,
Tom