Questions about floating ip and Resume base VM

[linuix]

Dear everyone,

I have some questions and I need some help.

Question 1

After I installed the cloudlet successfully according to https://github.com/cmusatyalab/elijah-openstack,

I executed the step7 as follows:

nova network-create vmprivate --fixed-range-v4 10.0.1.0/24 --fixed-cidr 10.0.1.0/24 --bridge br100 --bridge-interface eth1

I have two NIC card on my host machine (single node deployment),and my public_iface ip is
192.168.20.137 while my flat_iface ip is 10.0.1.170.
And I set the floating ip cidr as 192.168.20.137/24.Is this correct?

I dont know if my host machine had got allocated a routable IP from some DHCP server on my network,
because I use the server of my company.
How can I set the floating ip correctly?

Question 2

After I log into the dashboard, I click the import base VM button,base VM file was decompressed
into four files, and then saved to the Glance storage one by one.Then I resume base VM from
base disk image.Is this correct or which file (a base disk
image, a base memory snapshot, a hash value list for the disk image, a hash value list for the
memory snapshot)should I choose to resume base VM?

Question 3

After I log into the dashboard,I resumed bsae VM three times.

One was successfully resumed,but the instance couldn`t boot.

Others were error,and the error message is
No valid host was found. There are not enough hosts available and
Unavailable console type serial respectively.

Where can I view the error log file in my linux OS?
What can I do so that I can resume base VM successfully?

Question 4 (important)

After I installed the cloudlet,i want to complete these operation
import base VM
resume bsae VM
Create VM overlay
VM Provisioning(synthesis)
,and I have refer to your paper System Infrastructure for Mobile-Cloud Convergence,in addition to this,Is there any more detailed information that I can refer to?

Thank you for your patient advice.
Regards,
Linuix

[teiszler]

@linuix,

Question 1: 192.168.20.137/24 is possibly correct. It depends on your network. That IP doesn't seem like it is publicly addressable so there must be some NAT in there somewhere if you are able to get out to the internet from the host machine. I would check with your networking folks to see if 192.168.20.x is a valid subnet to use for your VMs.

Question 2: You are correct that the imported image is stored as four separate files in Glance. However, when you go to resume a base VM from the cloudlet panel, you should not be presented with all 4 files. I should only list the base disk image. This is the only one that should be presented in the table and it is the correct one to select.

Question 3: Typically the 'No valid host found' error indicates that the nova-compute service on the compute node (in your case it is the same as the controller since you have a single node) is down. There is a panel under Admin called System Information that will show you the status of the various OpenStack processes including the nova-compute processes on each node. You can also check it by doing service nova-compute status and, if it is down, service nova-compute start. You can check the /var/log/nova directory for logs that might indicate why it stopped.

Regarding the VM that launched but wouldn't boot, it is possible that the base VM was not created properly. You can follow the steps here to create a base VM and overlay: https://github.com/cmusatyalab/elijah-provisioning/tree/master/examples/gabriel-lego

Question 4: The link above should help you with a few of those operations. You can also look at the following README for more information on how to perform those functions from the cloudlet command-line utility: https://github.com/cmusatyalab/elijah-provisioning

Once you have created a base VM and an overlay using the command-line utilities, you should be able to import it into OpenStack, resume a base VM, and synthesize a VM from an overlay (as long as that overlay is on an accessible web server somewhere).

Hope this helps,
Tom

[linuix]

I really appreciate your help as always。

I have a problem when Resume Base VM

When I execute Resume Base VM ,an error occured to me:
ERROR unsupported configuration: guest and host CPU are not compatible: Host CPU does not provide required features: monitor
Error, make sure previous VM is closed and check QEMU_ARGUMENT

When I execute cloudlet overlay /root/.cloudlet/dc2d4054bee32502aedd61e7034e7bd536e7b17bace096f3c0971db170212e57/precise.raw
in Command line tool,the same mistake happened.

root@cloudlet:~# cloudlet overlay /root/.cloudlet/dc2d4054bee32502aedd61e7034e7bd536e7b17bace096f3c0971db170212e57/precise.raw
INFO     QEMU access file : /tmp/cloudlet-qemu-mpjucT/qemu-trim-log
INFO     start monitoring at /var/tmp/cloudletfs-QFW8zF/disk/streams/chunks_modified
INFO     start monitoring at /var/tmp/cloudletfs-QFW8zF/disk/streams/chunks_accessed
INFO     start monitoring at /var/tmp/cloudletfs-QFW8zF/memory/streams/chunks_accessed
INFO     start monitoring at /tmp/cloudlet-qemu-mpjucT/qemu-trim-log
INFO     * Overlay creation configuration
INFO       - {'DISK_ONLY': False,
 'FREE_SUPPORT': False,
 'TRIM_SUPPORT': True,
 'XRAY_SUPPORT': False,
 'ZIP_CONTAINER': True}
INFO     restoring VM...
**ERROR    unsupported configuration: guest and host CPU are not compatible: Host CPU does not provide required features: monitor**_
ERROR    failed : reasons - 
<domain xmlns:ns0="http://libvirt.org/schemas/domain/qemu/1.0" type="kvm">
  <name>cloudlet-ec49874a2f65483cb0e4cfa752ffe66d</name>
  <uuid>ec49874a-2f65-483c-b0e4-cfa752ffe66d</uuid>
  <memory unit="KiB">1048576</memory>
  <currentMemory unit="KiB">1048576</currentMemory>
  <vcpu placement="static">4</vcpu>
  <os>
    <type arch="x86_64" machine="pc-1.0">hvm</type>
    <boot dev="hd" />
  </os>
  <features>
    <acpi />
  </features>
  <cpu match="exact" mode="custom">
    <topology cores="4" sockets="1" threads="1" />
  <model fallback="forbid">core2duo</model></cpu>
  <clock offset="utc" />
  <on_poweroff>destroy</on_poweroff>
  <on_reboot>restart</on_reboot>
  <on_crash>destroy</on_crash>
  <devices>
    <emulator>/usr/local/bin/cloudlet_qemu-system-x86_64</emulator>
    <disk device="disk" type="file">
      <driver name="qemu" type="raw" />
      <source file="/var/tmp/cloudletfs-QFW8zF/disk/image" />
      <target bus="ide" dev="hda" />
      <address bus="0" controller="0" target="0" type="drive" unit="0" />
    </disk>
    <disk device="cdrom" type="file">
      <driver name="qemu" type="raw" />
      <source file="/var/lib/cloudlet/conf/ovftransport.iso" />
      <target bus="ide" dev="hdc" />
      <readonly />
      <address bus="1" controller="0" target="0" type="drive" unit="0" />
    </disk>
    <controller index="0" type="ide">
      <address bus="0x00" domain="0x0000" function="0x1" slot="0x01" type="pci" />
    </controller>
    <interface type="user">
      <mac address="52:54:00:6a:33:f7" />
      <model type="virtio" />
      <address bus="0x00" domain="0x0000" function="0x0" slot="0x03" type="pci" />
    </interface>
    <input bus="ps2" type="mouse" />
    <graphics autoport="yes" keymap="en-us" listen="127.0.0.1" port="-1" type="vnc">
      <listen address="127.0.0.1" type="address" />
    </graphics>
    <video>
      <model heads="1" type="cirrus" vram="16384" />
      <address bus="0x00" domain="0x0000" function="0x0" slot="0x02" type="pci" />
    </video>
    <memballoon model="virtio">
      <address bus="0x00" domain="0x0000" function="0x0" slot="0x04" type="pci" />
    </memballoon>
  </devices>
  <ns0:commandline>
    <ns0:arg value="-cloudlet" />
	<ns0:arg value="raw=suspend,logfile=/tmp/cloudlet-qemu-mpjucT/qemu-trim-log"/>
  </ns0:commandline>
  <seclabel relabel="yes" type="dynamic" />
</domain>
## XML: unsupported configuration: guest and host CPU are not compatible: Host CPU does not provide required features: monitor
## Error, make sure previous VM is closed and check QEMU_ARGUMENT, args - {}
ERROR    failed at Traceback (most recent call last):
  File "/usr/local/lib/python2.7/dist-packages/elijah/provisioning/synthesis.py", line 94, in decorated_function
    return function(self, *args, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/elijah/provisioning/synthesis.py", line 230, in resume_basevm
    self.base_mem_fuse, self.new_xml_str)
  File "/usr/local/lib/python2.7/dist-packages/elijah/provisioning/synthesis.py", line 1239, in run_snapshot
    restore_with_config(conn, mem_snapshot, new_xml_string)
  File "/usr/local/lib/python2.7/dist-packages/elijah/provisioning/synthesis.py", line 1331, in restore_with_config
    raise CloudletGenerationError(message)
CloudletGenerationError

INFO     close Stream monitoring thread
INFO     Fuse close pipe
INFO     NO chunks has been waited at FUSE
INFO     close Fuse Exec thread
INFO     close File monitoring thread
Failed to create overlay

When I vim /usr/share/libvirt/cpu_map.xml in my ubuntu,I see

<!-- standard features, ECX -->
    <feature name='monitor'> <!-- CPUID_EXT_MONITOR -->
       <cpuid function='0x00000001' ecx='0x00000008'/>
    </feature>
    <feature name='ds_cpl'> <!-- CPUID_EXT_DSCPL -->
       <cpuid function='0x00000001' ecx='0x00000010'/>
    </feature>
    <feature name='vmx'> <!-- CPUID_EXT_VMX -->
    	<cpuid function='0x00000001' ecx='0x00000020'/>
    </feature>
    <feature name='smx'>
       <cpuid function='0x00000001' ecx='0x00000040'/>
    </feature>

Now I couldnt resume base VM，How can I solve my problem?

Thank you for your patient advice.
Regards,
Linuix

[linuix]

More error information is attached.
I execute the steps as
https://github.com/cmusatyalab/elijah-provisioning/tree/master/examples/gabriel-lego.

root@cloudlet:~# cloudlet import-base ubuntu-trusty-lego-base.zip
INFO     create directory for base VM
INFO     Decompressing Base VM to temp directory at /tmp/cloudlet-base-A4p6KA
INFO     Place base VM to a right directory
INFO     Register New Base to DB
INFO     ID for the new Base VM: a8157d3eb1155ec13b80615afc883b5f01f99362214f72f2b6444854498d7b17
INFO     Success


root@cloudlet:~# cloudlet list-base
hash value                                      path
------------------------------------------------------------------------------------------
a8157d3eb1155ec13b80615afc883b5f01f99362214f72f2b6444854498d7b17        /root/.cloudlet/a8157d3eb1155ec13b80615afc883b5f01f99362214f72f2b6444854498d7b17/trusty-server-raw.img
------------------------------------------------------------------------------------------


root@cloudlet:~# cloudlet synthesis /root/.cloudlet/a8157d3eb1155ec13b80615afc883b5f01f99362214f72f2b6444854498d7b17/trusty-server-raw.img overlay.zip     
DEBUG    ==========================================
DEBUG    file:///root/overlay.zip
INFO     Decompression time : 73.699683 (s)
INFO     Recovering launch VM
INFO     Start FUSE (2.536042 s)
DEBUG    Overlay has 427089 chunks
INFO     Resume the launch VM
INFO     QEMU access file : /tmp/cloudlet-overlay-2qWaeo/qemu-trim-log
INFO     QEMU access file : /tmp/cloudlet-overlay-2qWaeo/qmp-channel
INFO     Launch disk: /tmp/cloudlet-launch-disk-9KeVpe
INFO     Launch memory: /tmp/cloudlet-launch-mem-XQOxnX
INFO     QMP channel: /tmp/cloudlet-overlay-2qWaeo/qmp-channel
INFO     start monitoring at /var/tmp/cloudletfs-IWYD0T/disk/streams/chunks_modified
INFO     start monitoring at /var/tmp/cloudletfs-IWYD0T/disk/streams/chunks_accessed
INFO     start monitoring at /var/tmp/cloudletfs-IWYD0T/memory/streams/chunks_accessed
INFO     start monitoring at /tmp/cloudlet-overlay-2qWaeo/qemu-trim-log
INFO     [Delta] Handle dangling DeltaItem (0)
DEBUG    Delta metrics: 
DEBUG    ==================================================
DEBUG    Counter({'sha': 427089, 32: 164516, 48: 144128, 16: 59044, 80: 49869, 64: 8304, 96: 1228})
DEBUG    Counter({32: 189.5245189666748, 'sha': 59.32714319229126, 'unpack': 42.100136041641235, 64: 31.063184022903442, 'seekwrite': 4.793902397155762, 'dict': 4.2154762744903564, 80: 3.188779592514038, 'flush': 2.547074794769287, 48: 1.6163005828857422, 16: 0.19196748733520508, 96: 0.006318092346191406})
DEBUG    Total captured time: 338
INFO     [Delta] : (1529839381.18)-(1529839894.37)=(513.192808151), delta 427089 chunks
INFO     [FUSE] : (1529839381.19)-(1529839894.38)=(513.191980124)

DEBUG    File closing time for recover memory snapshot: 5.320152
INFO     restoring VM...

## ERROR    unsupported configuration: guest and host CPU are not compatible: Host CPU does not provide required features: monitor

<domain xmlns:ns0="http://libvirt.org/schemas/domain/qemu/1.0" type="kvm">
  <name>cloudlet-a3884bb7b32243f2b7dbff773e2f187d</name>
  <uuid>a3884bb7-b322-43f2-b7db-ff773e2f187d</uuid>
  <memory unit="KiB">1048576</memory>
  <currentMemory unit="KiB">1048576</currentMemory>
  <vcpu placement="static">4</vcpu>
  <os>
    <type arch="x86_64" machine="pc-1.0">hvm</type>
    <boot dev="hd" />
  </os>
  <features>
    <acpi />
  </features>
  <cpu match="exact" mode="custom">
    <topology cores="4" sockets="1" threads="1" />
  <model fallback="forbid">core2duo</model></cpu>
  <clock offset="utc" />
  <on_poweroff>destroy</on_poweroff>
  <on_reboot>restart</on_reboot>
  <on_crash>destroy</on_crash>
  <devices>
    <emulator>/usr/local/bin/cloudlet_qemu-system-x86_64</emulator>
    <disk device="disk" type="file">
      <driver name="qemu" type="raw" />
      <source file="/var/tmp/cloudletfs-IWYD0T/disk/image" />
      <target bus="ide" dev="hda" />
      <address bus="0" controller="0" target="0" type="drive" unit="0" />
    </disk>
    <disk device="cdrom" type="file">
      <driver name="qemu" type="raw" />
      <source file="/var/lib/cloudlet/conf/ovftransport.iso" />
      <target bus="ide" dev="hdc" />
      <readonly />
      <address bus="1" controller="0" target="0" type="drive" unit="0" />
    </disk>
    <controller index="0" type="ide">
      <address bus="0x00" domain="0x0000" function="0x1" slot="0x01" type="pci" />
    </controller>
    <interface type="user">
      <mac address="52:54:00:b5:02:6d" />
      <model type="virtio" />
      <address bus="0x00" domain="0x0000" function="0x0" slot="0x03" type="pci" />
    </interface>
    <input bus="ps2" type="mouse" />
    <graphics autoport="yes" keymap="en-us" listen="127.0.0.1" port="-1" type="vnc">
      <listen address="127.0.0.1" type="address" />
    </graphics>
    <video>
      <model heads="1" type="cirrus" vram="16384" />
      <address bus="0x00" domain="0x0000" function="0x0" slot="0x02" type="pci" />
    </video>
    <memballoon model="virtio">
      <address bus="0x00" domain="0x0000" function="0x0" slot="0x04" type="pci" />
    </memballoon>
  </devices>
  <ns0:commandline>
    <ns0:arg value="-cloudlet" /><ns0:arg value="raw=live,logfile=/tmp/cloudlet-overlay-2qWaeo/qemu-trim-log" /><ns0:arg value="-qmp" /><ns0:arg value="unix:/tmp/cloudlet-overlay-2qWaeo/qmp-channel,server,nowait" /></ns0:commandline>
<seclabel relabel="yes" type="dynamic" /></domain>

XML: unsupported configuration: guest and host CPU are not compatible: Host CPU does not provide required features: monitor

Error, make sure previous VM is closed and check QEMU_ARGUMENT

ERROR    Warning, Possible VNC port error:'NoneType' object has no attribute 'XMLDesc'
INFO     waiting for finishing VNC interaction
INFO     close Stream monitoring thread
INFO     Fuse close pipe
INFO     NO chunks has been waited at FUSE
INFO     close Fuse Exec thread
INFO     close File monitoring thread

[teiszler]

@linuix,

I thought you were able to resume a base VM from within OpenStack or is that not the case? It seems that there is a problem between your host and libvirt. Are you certain that virtualization is enabled in the BIOS on the host machine? What does lscpu output? Are you doing nested virtualization and if so, are you sure that all the CPU flags are being past to the outer VM?

Regards,
Tom

[teiszler]

@liniux,

I saw this thread from a while back which may help. http://forum.openedgecomputing.org/t/openstack-in-a-single-vm-cpu-flag-compatibility-problem/78

You could try also changing it from core2duo to kvm64 as the model.

[linuix]

Thank you for your reply.

Before，I used VMware Workstation Pro on a Windows physical machine server create an ubuntu 14.04 virtual machine and then install elijah-openstack on this ubuntu 14.04 virtual machine.
Is this what you say nested virtualization ,how can I past all the CPU flags to the outer VM which I create on the Windows physical machine server?

And this cause the error above in last post, as follow：
guest and host CPU are not compatible: host CPU does not provide Required features: monitor

I ran the cat /proc/cpuinfo | grep monitor, and I really didn't find the flag Monitor,
then I reinstall elijah-openstack on a physical machine (no longer
the virtual machine) with ubuntu 14.04 ,and the CPU has the flag monitor .

first question

Is the elijah-openstack must be installed on a physical machine instead of a virtual machine,which they are all ubuntu 14.04?

Now that I've reinstalled elijah-openstack on a physical machine, my environment is

Ubuntu 14.04 (physical machines)
one NIC
CPU  flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca
 cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall 
nx rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl 
xtopology nonstop_tsc aperfmperf eagerfpu pni pclmulqdq dtes64  
**monitor ds_cpl  **vmx est tm2 ssse3 
cx16 xtpr pdcm pcid sse4_1 sse4_2 popcnt tsc_deadline_timer xsave avx
 f16c lahf_lm epb tpr_shadow vnmi flexpriority ept vpid fsgsbase smep erms
 xsaveopt dtherm arat pln pts

After I imported the precise-hotplug-new base VM,then I encountered this error when resume
it

2018-08-06 17:49:05.831 29363 WARNING nova.compute.manager [req-ea7f7084-0725-4dbb-a0fa-6c64f3a6d815 - - - - -] 
【While synchronizing instance power states, found 1 instances in the database and 0 instances on the hypervisor. 】
2018-08-06 17:49:38.003 29363 【ERROR】 nova.compute.manager [req-14b67e06-09b6-4c50-bda0-fe4b7f2906c7 8661977
40dff495aac584548dc426de7 1acd954896df4953ac38726891a0ee6d - - -] [instance: 7868c14d-54f9-42cb-8066-705eb
96c803d] 【Instance failed to spawn】

2018-08-06 17:49:38.003 29363 TRACE nova.compute.manager [instance: 7868c14d-54f9-42cb-8066-705eb96c803d] 
Traceback (most recent call last):

2018-08-06 17:49:38.003 29363 TRACE nova.compute.manager [instance: 7868c14d-54f9-42cb-8066-705eb96c803d] 
File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 2473, in _build_resources

2018-08-06 17:49:38.003 29363 TRACE nova.compute.manager [instance: 7868c14d-54f9-42cb-8066-705eb96c803d] 
yield resources

2018-08-06 17:49:38.003 29363 TRACE nova.compute.manager [instance: 7868c14d-54f9-42cb-8066-705eb96c803d] 
File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 2345, in _build_and_run_instance

2018-08-06 17:49:38.003 29363 TRACE nova.compute.manager [instance: 7868c14d-54f9-42cb-8066-705eb96c803d]
block_device_info=block_device_info)

2018-08-06 17:49:38.003 29363 TRACE nova.compute.manager [instance: 7868c14d-54f9-42cb-8066-705eb96c803d]
File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/cloudlet_driver.py", line 630, in spawn

2018-08-06 17:49:38.003 29363 TRACE nova.compute.manager [instance: 7868c14d-54f9-42cb-8066-705eb96c803d] 
diskhash_path, memhash_path, base_sha256_uuid)

2018-08-06 17:49:38.003 29363 TRACE nova.compute.manager [instance: 7868c14d-54f9-42cb-8066-705eb96c803d]
File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/cloudlet_driver.py", line 697, in resume_basevm

2018-08-06 17:49:38.003 29363 TRACE nova.compute.manager [instance: 7868c14d-54f9-42cb-8066-705eb96c803d] 
synthesis.rettach_nic(virt_dom, vm_overlay.old_xml_str, xml)

2018-08-06 17:49:38.003 29363 TRACE nova.compute.manager [instance:7868c14d-54f9-42cb-8066-705eb96c803d] 
File "/usr/local/lib/python2.7/dist-packages/elijah/provisioning/synthesis.py", line 1301, in rettach_nic

2018-08-06 17:49:38.003 29363 TRACE nova.compute.manager [instance: 
7868c14d-54f9-42cb-8066-705eb96c803d] filter_element = new_nic.find("filterref")

2018-08-06 17:49:38.003 29363 TRACE nova.compute.manager [instance: 7868c14d-54f9-42cb-8066-705eb96c803d] 
AttributeError: 'NoneType' object has no attribute 'find'

2018-08-06 17:49:38.003 29363 TRACE nova.compute.manager [instance: 7868c14d-54f9-42cb-8066-705eb96c803d]

2018-08-06 17:59:05.872 29363 WARNING nova.compute.manager [req-ea7f7084-0725-4dbb-a0fa-6c64f3a6d815 - - - - -] 
While synchronizing instance power states, found 1 instances in the database and 0 instances
on the hypervisor.

second question

I guess it's duo to there only has one network interface card in my environment.
What do you think caused this error, and how can I fix it to be able to resume the base VM
Can I add a NIC to solve this ?
I've seen in previous posts that if there is only one NIC in the environment, the flat_iface can be named as anything which is unique.Can run the entire Ansible playbook works?

Supplement:
I had validate cloudlet code using commands in this link, and they have the right output(I think is correct) :
http://forum.openedgecomputing.org/t/a-failure-in-creating-a-vm-overlay/15/4

Regards,
Linuix

[teiszler]

@linuix,

1. Though I haven't tested in a nested virtualized environment, it seems that some people have gotten it to work with some additional work(see that post I referenced in the last response). Typically I just use bare metal with Ubuntu 14.04. If you want to use the OpenStack integration, then you have to stay at 14.04 as we are currently bound to the Kilo release of OpenStack which doesn't support 16.04. If you just want to use the cloudlet library from elijah-provisioning, you should be able to use 16.04 also.

2. If you only have one NIC, then you can have the Ansible script create a dummy network interface to use for the flat_iface. Just note that this does not persist after reboot so you will either have to perform the steps related to creating the virtual NIC manually or rerun the Ansible playbook with the --tags nova flag to recreate it. My guess is that you rebooted and this virtual interface doesn't exist and that is why nova can't launch the instance.

From the README: If you only have a single network interface, make sure that flat_iface is something unique as a virtual interface with that name will be created when the one_nic variable in openstack-kilo.yaml is True. Please note that this virtual interface is not persistent and will be lost after reboot. To remedy this, you can run the entire Ansible playbook after rebooting which will recreate the dummy interface and restart the necessary nova services.

[linuix]

Thank you for you help.

I switch to a bare machine with Ubuntu 14.04，two NIC，then install the elijah-openstack script.
This time I resumed the Base VM successfully.
I will do further experiment according to the README.
I will consult you again if there is any question.

Now after I click the Create VM Overlay,it run into the Image Pending Upload state for 7 more hours。
and I find there is no strange in /var/log/nova.
Why I stuck here for a long time?

kind regards,
linuix