Create Overlay VM Error

[ellis]

Hi All !

I installed OpenStack++ using ansible branch with single node, and successfully installed it.
I can import Base VM using OpenStack Dashboard and success resume Base VM. And then when I use "Create VM overlay" button, it show "Success: Scheduled VM overlay creation of VM overlay: resumed_vm" dialog. But I can't see Overlay VM with Dashboard for a while(at least 6 hour+).

And I try to localhost using Cloudlet library command try again. I can import Base VM from Base VM for Ubuntu-12.04.01-i386-Server using export-base command.
But when I create a VM Overlay through the following: How to create VM overlay instructions. However, it gives an error:

cloudlet overlay /root/.cloudlet/abda52a61692094b3b7d45c9647d022f5e297d1b788679eb93735374007576b8/precise.raw 
INFO     QEMU access file : /tmp/cloudlet-qemu-ZB2toQ/qemu-trim-log
INFO     start monitoring at /var/tmp/cloudletfs-C6KRfy/disk/streams/chunks_modified
INFO     start monitoring at /var/tmp/cloudletfs-C6KRfy/disk/streams/chunks_accessed
INFO     start monitoring at /var/tmp/cloudletfs-C6KRfy/memory/streams/chunks_accessed
INFO     start monitoring at /tmp/cloudlet-qemu-ZB2toQ/qemu-trim-log
INFO     * Overlay creation configuration
INFO       - {'DISK_ONLY': False,
 'FREE_SUPPORT': False,
 'TRIM_SUPPORT': True,
 'XRAY_SUPPORT': False,
 'ZIP_CONTAINER': True}
INFO     restoring VM...
INFO     VM is restored...
INFO     waiting for finishing VNC interaction
INFO     stop monitoring at /var/tmp/cloudletfs-C6KRfy/memory/streams/chunks_accessed
INFO     save VM memory state
DEBUG    start machine save
INFO     Header size of memory snapshot is 8192
[================================================================================>] 100%
ERROR    internal error: unexpected async job 3
Exception in thread Thread-6:
Traceback (most recent call last):
  File "/usr/lib/python2.7/threading.py", line 810, in __bootstrap_inner
    self.run()
  File "/usr/lib/python2.7/threading.py", line 763, in run
    self.__target(*self.__args, **self.__kwargs)
  File "/usr/local/lib/python2.7/dist-packages/elijah/provisioning/handoff.py", line 312, in save_mem_thread
    machine.save(outputpath)  # green thread blocked in here
  File "/usr/lib/python2.7/dist-packages/libvirt.py", line 2001, in save
    if ret == -1: raise libvirtError ('virDomainSave() failed', dom=self)
libvirtError: internal error: unexpected async job 3

Physical machines specification:

OS: Ubuntu 14.04.5 LTS Desktop
CPU: Intel® CoreTM i7-4790 CPU @ 3.60GHz x 8
RAM: 32 GB
HDD: 1 TB

KVM, QEMU, Linux version:

$ kvm --version
QEMU emulator version 2.2.0 (Debian 1:2.2+dfsg-5expubuntu9.7~cloud8), Copyright (c) 2003-2008 Fabrice Bellard
$ /usr/local/bin/cloudlet_qemu-system-x86_64 --version
QEMU emulator version 1.1.1 (qemu-kvm-devel), Copyright (c) 2003-2008 Fabrice Bellard, Cloudlet Edition 0.9.3
$ libvirtd --version
libvirtd (libvirt) 1.2.12
$ uname -ar
Linux elliswu 4.4.0-66-generic #87~14.04.1-Ubuntu SMP Fri Mar 3 17:32:36 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux

Another Question:
Why use the OpenStack dashboard to import VMs, and I can't see any Base VM information using cloudlet list-base command in localhost ?

Do you have any idea about these problems ? Thank you.

[teiszler]

@ellis,

I think you should be using the cloudlet import-base command instead of export-base. Once you import the base VM image, you should be able to do a cloudlet list-base and see it appear in the list. Also you should be able to create an overlay from that base image.

In regards to why the overlay is not being created from the OpenStack dashboard, I will have to look into it. I have observed similar behavior in my cluster, it says it is being scheduled, but nothing further seems to happen. I have successfully synthesized (by clicking the Start VM Synthesis button) an existing overlay that I have created and if you are able to create one via the cloudlet library, you should be able to synthesize that overlay as long as you make it accessible via a http URL.

Regards,
Tom

[teiszler]

I experienced the same problem when trying to create an overlay via cloudlet overlay. My current theory is that the cloudlet library requries a modified QEMU 1.1.1 but also QEMU 2.2.0 is installed via the qemu-kvm pacakge. I will let you know more after some testing.

[ellis]

Hi @teiszler !

Thanks for replying back.
Once I typed cloudlet export-base, when I really meant cloudlet import-base. Sorry for the typo.

I will continue to test. And if you have any information, please let me know, thank you.

[teiszler]

@ellis,

I think I have a solution. I am going to test it out once more tomorrow morning on a clean installation. I believe the issue revolves around the fact that elijah-provisioning installs versions of the libvirt-bin and qemu-kvm packages that get upgraded during the installation of the OpenStack components. So I have moved the installation of elijah-provisioning to the top of the openstack-compute role and I am going to add steps to the ansible script so that these packages are held at their initial versions and are not updated during the remainder of the installation. If all goes well tomorrow I will push the updates to the elijah-openstack ansible branch. Unfortunately you may have to start with a clean install as my attempts to downgrade these two packages after the ansible script finished didn't work well.

Regards,
Tom

[ellis]

Hi @teiszler !

I also found that the version of qemu was updated and tried to downgrade it. But you already have a solution that's great! Expect you to update ansible branch.

In addition, you can provide some version of your test environment, such as the following commands Make it easier for me to do some tests on the differences between versions.

$ kvm --version
$ /usr/local/bin/cloudlet_qemu-system-x86_64 --version
$ libvirtd --version
$ uname -ar

Thank you.

[teiszler]

Yes I will be doing so shortly.

I don't believe the package installation of qemu-kvm is as consequential here as the upgrading of libvirt because the clouldet code builds its own version of qemu from modified source (which you see from /usr/local/bin/cloudlet_qemu-system-x86_64).

Here is the information you requested:

root@cloudlet014:~# kvm --version
QEMU emulator version 2.0.0 (Debian 2.0.0+dfsg-2ubuntu1.32), Copyright (c) 2003-2008 Fabrice Bellard
root@cloudlet014:~# /usr/local/bin/cloudlet_qemu-system-x86_64 --version
QEMU emulator version 1.1.1 (qemu-kvm-devel), Copyright (c) 2003-2008 Fabrice Bellard, Cloudlet Edition 0.9.3
root@cloudlet014:~# libvirtd --version
libvirtd (libvirt) 1.2.2
root@cloudlet014:~# uname -ar
Linux cloudlet014 3.13.0-113-generic #160-Ubuntu SMP Thu Mar 9 09:27:29 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux

What was happening is that the OpenStack Kilo repository I was adding at the very beginning of the Ansible script has a newer version of libvirt and qemu than does the Ubuntu trusty repos. So as the nova-compute component was being installed, those packages had upgrades available. To solve this I held these packages after the installation of elijah-provisioning with apt-mark hold.

[teiszler]

@ellis,

I just pushed the changes I spoke of. I tested it again this morning. Hopefully you will see the same results I did and the creation of overlays will work both from the OpenStack Cloudlet pane and by using the cloudlet CLI. As I mentioned, I now install the cloudlet library before any OpenStack components and I hold the qemu-kvm and libvirt-bin packages at the version that is installed by elijah-provisioning so that they are updated with the versions in the OpenStack kilo repository.

Let me know if you run into anything further.

Thanks,
Tom

[ellis]

Hi @teiszler !

I have successfully run OpenStack++ and created "Overlay VM" and "Synthesis VM" on the OpenStack Dashboard.

The first time I tried ansible script, follow the steps directly. However, after the (OS-COMMON) update cache and perform upgrade step, the libvirt-bin upgrade error. Therefore, I will be apt update with apt upgrad before executing the script, and run ansible script to work properly.

Finally, I slightly modified ansible branch of README.md, please refer to:

ellis-wu/elijah-openstack

Thank you !

[teiszler]

@ellis,

This is good to hear. I am glad you got it working and I will incorporate your changed to the README markdown. Just so I understand the issue you had with libvirt-bin, can you expand on what happened? You received an error during the (OS-COMMON) update cache and perform upgrade step?

Thanks for the help,
Tom

[ellis]

Hi @teiszler !

Sorry, I did not record any error messages, but in the (OS-COMMON) update cache and perform upgrade step, an error occurred. The following is my case:

Step 1.
Prepare clean ubuntu 14.04

Step 2.

$ sudo -i
$ apt-get install software-properties-common
$ apt-add-repository ppa:ansible/ansible
$ apt-get update
$ apt-get install ansible git
$ cd ~
$ git clone -b ansible https://github.com/cmusatyalab/elijah-openstack
$ cd ~/elijah-openstack/ansible

Step 3.
Modify roles/openstack-controller/vars/main.yml, roles/openstack-compute/vars/main.yml and roles/cloudlet/vars/main.yml

Step 4.

$ cd ~ / elijah-openstack / ansible
$ ansible-playbook -i ./hosts openstack-kilo.yaml

When the run to the (OS-COMMON) update cache and perform upgrade step, show package upgrade failed, I remember correctly is libvirt-bin upgrade error, but the error details I did not record.

So before I do Step 2, run apt-get update and apt-get upgrade, and then continue the steps, the error will not happen.

[asadch]

Hi @ellis can u please share details of:
Step 3.
Modify roles/openstack-controller/vars/main.yml, roles/openstack-compute/vars/main.yml and roles/cloudlet/vars/main.yml
what variables need to be modify?

[ellis]

Hi @asadch !

Sorry for the late reply.

If you install elijah-openstack on a physical host, and your network interface card only one, please refer to the following configuration:

Modify ansible/roles/openstack-controller/vars/main.yml

• Change passwd to your own password.
• Flat_iface must be a unique name, you can replace it with your favorite name, but can not be duplicated with the NIC name on the host.

---
Rabbit_pw: passwd
Mysql_pw: passwd
Db_pw: passwd
Keystone_admin_pw: passwd
Admin_token: passwd
Metadata_secret: passwd
Openstack_controller_node: localhost
Controller_ip: 127.0.0.1
Pub_iface: eth0
Flat_iface: eth10

Modify ansible/roles/openstack-compute/vars/main.yml

• passwd and flat_iface are configured with the same parameters as ansible/roles/openstack-controller/vars/main.yml.

---
Rabbit_pw: passwd
Mysql_pw: passwd
Db_pw: passwd
Keystone_admin_pw: passwd
Admin_token: passwd
Metadata_secret: passwd
Openstack_controller_node: localhost
Controller_ip: 127.0.0.1
Pub_iface: eth0
Flat_iface: eth10

Change your user name and password

Install Cloudlet library through Fabric, it needs your user account and password, so please modify ansible/roles/cloudlet/vars/main.yaml:

#required to run the fabric script from https://github.com/cmusatyalab/elijah-provisioning
Local_user: <your user name>
Local_user_pw: <your password>

[linuix]

I see you have successfilly created "Overlay VM" and "Synthesis VM" on the OpenStack Dashboard.

But I have face some problem as show in

Questions about floating ip and Resume base VM

Thank you for you help. I switch to a bare machine with Ubuntu 14.04，two NIC，then install the elijah-openstack script. This time I resumed the Base VM successfully. I will do further experiment according to the README. I will consult you again if there is any question. Now after I click the Create VM Overlay,it run into the Image Pending Upload state for 7 more hours。 and I find there is no strange in /var/log/nova. Why I stuck here for a long time? [image] kind regards, linuix

Could you please help me ?