Stuck on VM Overlay creation process

solimand 2016-01-15 10:08:11 UTC #1

Hello Everyone,
I am a Computer Science student. I try to develop a simple example cloudlet application, but I am stuck during architecture deployment process. I tried to install the Elijah platform on this architecture:

CPU = Intel Core i7, 2.8 GHz.
RAM = 4GB.
HDD = 100 GB.

I followed these guidelines: first, I tried with -->github.com/cmusatyalab/elijah-openstack. Later, after failing in some point explained below, I tried with the steps for standalone version of Elijah, described at -->github.com/cmusatyalab/elijah-provisioning.
I resumed the base VM imported from -->storage.cmusatyalab.org/cloudlet-vm/precise-hotplug.zip. It has a different name, but seems to be the same downloadable file from -->storage.cmusatyalab.org/cloudlet-vm/precise-baseVM.zip (can anyone confirm my affirmation?), because when I try to import the two different file, openstack tells me there is already a base VM with that UUID.
Now I am stuck on process of creation VM Overlay. When I try to create VM Overlay, after a lot of time, the resumed VM remains in “snapshotting” state and the new Overlay in “Queued” state (the html page switches continuously between the two images below). I have inspect all the devstack screens to search error, but every openstack component seems working fine.

http://postimg.org/image/aljj9i8wp/
http://postimg.org/image/60xcukp7d/

At this point, I have googling a lot and I have read this question -->forum.openedgecomputing.org/t/a-failure-in-creating-a-vm-overlay/15. With the standalone version, I am able to create a VM overlay and to synthesize a new VM with the created overlay (the “cloudlet synthesis” command is an alternative to the synthesis process made via synthesis server/client?) However, I am not able to synthetize that overlay, created in standalone, with openstack version.

Please help me to create overlay I'm sorry for partial links and image links, but is my first post and i can't insert more than 2 links, neither images. .

Update 1 : I have not specified that the standalone version works fine for me, on the Ubuntu 14.04 LTS operating system. The question about server-client synthesis model VS client synthesis command is only a doubt. I have successfully synthesized a VM with a simple server installed on it , through the C-S synthesis model provided in standalone, but i need the Openstack support for my work.

Marco 2016-01-18 10:33:11 UTC #2

Hello @solimand,
Welcome to OEC forums.

You should follow the guidelines present at http://github.com/cmusatyalab/elijah-openstack. Although from what I understood you're developing over a multi-node OpenStack instance, is that correct?

@krha can confirm but at this stage I'd strongly advise running the Cloudlet over a single node DevStack instance. (I'll update some more information on some configurations over the forum).

Regarding the Overlay process, depending on the HW configuration the process can take some time. Please bear in mind that this is to be executed in Server HW Specs. From your current environment I foresee troubles with the very limited amount of RAM to be available to OpenStack and for further spawned VM's.

When you're using DevStack, you can see live log information (./rejoin-stack/sh) screens and log files filepath is:

/opt/stack/logs

Can you please:

Confirm your OpenStack (DevStack) instance (should use: stable/kilo)
Provide information for:
- libvirtd --version
- kvm --version

From my experience and depending on your setup I'd advise to force the use of these libraries if you do not have any other restriction:

libvirt-bin = 1.2.2-0ubuntu13.1.14
libvirt0 = 1.2.2-0ubuntu13.1.14
python-libvirt = 1.2.2-0ubuntu2

These have worked the best for my current environment and helped reducing some conflicts later down the road.

Let's see how we can aid you further.

solimand 2016-01-18 16:12:04 UTC #3

Hello @Marco,
thanks much for the answer and the interest. I followed the guidelines at http://github.com/cmusatyalab/elijah-openstack with a regular All-In-One Single Machine devtsack installation.

Regarding the resource lack issue...for now there is not an immediate solution, but i hope i can update my RAM to 8 GB soon.

Anyway, I give you more information about my test environment:
Ubuntu Trusty Desktop:

No LSB modules are available.
Distributor ID:	Ubuntu
Description:	Ubuntu 14.04.3 LTS
Release:	14.04
Codename:	trusty

Devstack Version: solinux@solinux-PC:~/devstack$ git branch -a

master
* stable/kilo
remotes/origin/HEAD -> origin/master
remotes/origin/master
remotes/origin/stable/kilo
remotes/origin/stable/liberty

Libvirt version: libvirtd --version

libvirtd (libvirt) 1.2.2
python-libvirt = 1.2.2-0ubuntu2.

Kvm version: kvm --version

QEMU emulator version 2.0.0 (Debian 2.0.0+dfsg-2ubuntu1.21), Copyright (c) 2003-2008 Fabrice Bellard

I have the other packages relative to libvirt at version:

libvirt0 =  1.2.2-0ubuntu13.1.16.
libvirt-bin = 1.2.2-0ubuntu13.1.16.

When i try to enforce your same version with sudo apt-get install libvirt-bin=1.2.2-0ubuntu13.1.14; sudo apt-get install libvirt0=1.2.2-0ubuntu13.1.14, I receive the error message:

E: Version '1.2.2-0ubuntu13.1.14' for 'libvirt-bin' was not found.

Do I have to install them manually?
Now i'll try again with a new clean './stack.sh' to have new log file in /opt/stack/logs, perhaps this will give me more informations to analyze.

Marco 2016-01-19 10:48:38 UTC #4

@solimand, I would say to ignore the issue when setting libvirt library to (...)ubuntu13.1.14 as *16 should be fine. The main deal for me was using 1.2.2 althogether that unblocked a lot of issues I was having.

Regarding a new ./stack.sh I don't think that would really be necessary. You should have runtime log files under the filepath /opt/stach/logs/ anyway.
Be sure to check DevStack screens using ./rejoin-stack.sh. Navigate using CTRL+a and then press either n or p (or eventually the digit 0-9 of screen id).

Then, please observe what is happening when trying to create an Overlay through OpenStack (I'm assuming you're doing it through Horizon dashboard here). Please do allow a very good time for it conclude as I'm skeptical the available resources are a constrain here.

Do update if unsuccessful and please provide log output / files.
Best regards

krha 2016-01-19 15:29:10 UTC #5

Hi @solimand

Sorry for the a bit late response.

It's glad that you were able to make your own VM overlay and perform synthesis using a standalone version (e.g. cloudlet synthesis command).

Since a standalone version and OpenStack version share cloudlet library, the problem is likely to be at OpenStack binding with cloudlet library. Then, now the problem gets a bit harder to debug because we should take a look at OpenStack logs to find the error.

Can you copy & paste of OpenStack log at your machine. As @Marco mentioned, first check out the OpenStack logs at screen of OpenStack using "./rejoin_stack.sh" . When you execute creating VM overlay, there would be at least one log that shows error messages. It's mostly like to show an nova-api log (n-api) or nova-compute log (n-con).

Also, please let me know your OpenStack configuration such as single-node or multi-node.

By the way, library version shouldn't be a problem because you were able to go through whole process with a standalone version. Thank you.

solimand 2016-01-19 18:20:35 UTC #6

Hello everyone and thanks to all for the responses; never too late @krha .
I have just completed the steps with a successful synthesis of a modified VM! After a fresh reinstallation of everithing, I know the issue is in openstack (devstack Single Machine Installation), but I'm not sure where. I have collected a couple of errors, all about openstack configuration. Now my configuration is a bit different from that showed before and differs only about the Ubuntu version, in particular now I have the 14.04.2 LTS and not 14.04.3 version.

Some devstack screens have errors, although the procedure works fine. I give you a summary of the errors:

Stack.sh.log

sudo a2enmod version
ERROR: Module version does not exist!
...
openstack role list: error: unrecognized arguments: --group 130f9027559a465fba566b8acdf446e1
...
openstack role add: error: argument --user is required
...
openstack role list: error: unrecognized arguments: --group 130f9027559a465fba566b8acdf446e1
...
openstack endpoint show -f value -c publicurl s3
ERROR: openstack No service with a type, name or ID of 's3' exists.
...

C-VOL Screen has an error regarding stack-volumes-lvmdriver-1. About this error i have read this discussion: https://bugs.launchpad.net/cinder/+bug/1394179. This time i don't use dd-losetup-vgcreate commands as suggested and simply don't care about error. Previously I have followed that way but perhaps is this the misconfiguration that brought me to error. The resultant loop warning is:

WARNING cinder.volume.manager [req-46b520d6-8b28-4211-a2f7-58ca3859cf1e None None] Unable to update stats, LVMVolumeDriver -3.0.0 (config name lvmdriver-1) driver is uninitialized.

The N-API; N-CPU; N-NET; N-COND; N-SCH Screens have errors about rabbitMQ and connections to MySQL during 'resume VM' and 'create Overlay' processes. Is this perhaps related to this problem of rabbit MQ on compute node ( https://bugs.launchpad.net/fuel/+bug/1323277/comments/15)?

I share with you the log files via a dropbox link:
https://www.dropbox.com/sh/2xc8ywtx1qbi04o/AACabprmyxDhep86HCzuA-pXa?dl=0

In the files, the date 2016-01-18 refers to the operations of 'Resume base VM' and 'Creation VM Overlay'; today I have performed only the synthesis.

Thank you and best regards

Marco 2016-01-19 18:31:21 UTC #7

@solimand, This issue is because DevStack's Cinder is trying to access the wrong Logic Volume. To solve this isue please change the configuration of the Cinder component at /etc/cincer/cinder.conf (should be at this filepath).
Do sudo vgs, you should see something like:

VG       #PV #LV #SN Attr   VSize   VFree
ubuntu-vg   2  2   0 wz--n- 289.75g 35.76g

Here, the ubuntu-vg is the volume name.
On the configuration file change the entry: volume_group = <your_volume_name_here>.

You will need to restart Cinder services. You can do this on each c-<...> screen (stop the service and then just get the last command by pressing UP to restart it) or do a complete ./unstack.sh and then ./rejoin-stack.sh.
Hope this solves that issue.

I'll update this answer once I get to check the logs.

solimand 2016-01-20 12:03:53 UTC #8

Hello @Marco,

unfortunately the sudo vgs command tells me No volume groups found. Should I manually create the volume? Before ./rejoin_stack.sh ?

Update 1: in the cinder log file, the errors on the volumes seem occur after a reboot of the machine (or ./unstack.sh and rejoin_stack.sh), not at the ./stack.sh time. Perhaps is there a problem of permission or of persistence of the volumes?

Update2

It seems to be a persistence problem.
https://bugs.launchpad.net/cinder/+bug/1394179 -->This topic tells me that it isn't rejoin-stack.sh's job to set the Linux environment every time after a reboot. Devstack set it once at ./stack.sh time.

Update 3
For me this solve the Cinder Volume Screen error (just do it before ./rejoin_stack.sh):

truncate -s 10250M /opt/stack/data/stack-volumes-lvmdriver-1-backing-file
sudo losetup -f --show /opt/stack/data/stack-volumes-lvmdriver-1-backing-file
sudo vgcreate stack-volumes-lvmdriver-1 /dev/loop0. Based on what the prevuois command shows you, it should use the appropriate dev/loop, in my case is dev/loop0

Marco 2016-01-20 12:26:38 UTC #9

@solimand,
I'd recommend to search the web and DevStack resources to aid you on that one. I'd say to operate with Cinder you'd require to have access to some storage volumes.

This is because your current configuration seems to be defined to contack a Volume which the Cinder service cannot access to (hence the error messages), either if you create it or check for its persistence. Apologies but can't be much of a help much more on this.

krha 2016-01-21 22:01:58 UTC #10

Hi @solimand

Good to hear that you made it work.

OpenStack part is hard to debug and your log (nova compute) does not give me any clue. I will test what's different between 14.04.02 and 14.04.03 later. Thanks.

rim 2016-02-09 18:17:09 UTC #11

Hi guys,
I have exactly same problem. Could you pls give me some ideas:
I exactly follow the steps which are mentioned in
https://github.com/cmusatyalab/elijah-openstack

My platform is: Ubuntu 14.04 LTS
$kvm – version:
QEMU emulator version 2.0.0 (Debian 2.0.0+dfsg-2ubuntu1.22),
Copyright (c) 2003-2008 Fabrice Bellard

libvirtd –version
:libvirtd (libvirt) 1.2.2
$ git branch -a
 master
* stable/kilo
remotes/origin/HEAD -> origin/master
remotes/origin/master
remotes/origin/stable/kilo
remotes/origin/stable/liberty

I Have successfully install DevStack and tested it. I then install
cloudlet OpenStack extension. Before
./rejoin_stack.sh, I have created physical
volume as well with the following command.

truncate -s 10250M
/opt/stack/data/stack-volumes-lvmdriver-1-backing-file 

sudo losetup
	-f --show /opt/stack/data/stack-volumes-lvmdriver-1-backing-file

sudo vgcreate stack-volumes-lvmdriver-1
	/dev/loop0

After executing ./rejoin_stack.sh, I am stuck at this point. Logs are as follows:

$/usr/local/bin/nova-conductor --config-file /e 
tc/nova/nova.conf &
echo $! >/opt/stack/status/stack/n-cond.pid; fg || echo "n-cond failed to start"
| tee "/opt/stack/status/stack/n-cond.failure"
[1] 12568
/usr/local/bin/nova-conductor
--config-file /etc/nova/nova.conf

$/usr/local/bin/nova-api & echo $! >/opt/stack/ status/stack/n-api.pid;
fg || echo "n-api failed to start" | tee "/opt/stack/status/stack/n-api.failure"

[1] 12461
/usr/local/bin/nova-api

solimand 2016-02-10 09:50:50 UTC #12

Hi @rim,
I recommend you to see the log file before rejoin-stack and if you note some errors, the problem is with openstack/devstack and not with cloudlet. You should use static IP and should have a completly clear (just installed) Ubuntu version. Please, can you provide us more info? From these logs it seems that the openstack nova-api e nova-conductor (which acts as an intermediary between the compute service and the DB) services never start! What do the other nova screens show you?

To solve my problem, I had to format the system and install all again, following carefully the directives of the team of elijah ( https://github.com/cmusatyalab/elijah-openstack)

Best regard and good luck.

rim 2016-02-25 19:05:49 UTC #13

Hi @solimand,
Thanks for your reply. It seems nova-network problem. Here is my local.conf file, what's wrong in my config file, could you please suggest me. Note that I have only one NIC card (eth0) and now I have static IP address. Do I need other parameters in local.conf? Do I need to change something in nova.conf?


SERVICE_TOKEN=tokentoken
ADMIN_PASSWORD=password
MYSQL_PASSWORD=$ADMIN_PASSWORD
RABBIT_PASSWORD=$ADMIN_PASSWORD
SERVICE_PASSWORD=$ADMIN_PASSWORD
     
HOST_IP=192.26.211.236 // This is my fixed IP address
    
PUBLIC_INTERFACE=eth0

# Logging

LOGFILE=$DEST/logs/stack.sh.log
LOGDAYS=2
    
SYSLOG=True
SYSLOG_HOST=$HOST_IP
SYSLOG_PORT=516
   
# Swift
SWIFT_HASH=66a3d6b56c1f479c8b4e70ab5c2000f5
SWIFT_REPLICAS=1
SWIFT_DATA_DIR=$DEST/data
    
# Tempest
enable_service tempest
enable_service n_cauth

krha 2016-02-25 20:36:58 UTC #14

Hi @rim,

You can find sample local.conf file at here.

As you can see at the comments, you need two NICs; one for FLAT_INTERFACE and the other for PUBLIC_INTERFACE.

If you have only one NIC, you can make a virtual NIC following this instruction.

Make sure you assign your eth0 to PUBLIC_INTERFACE and use a virtual one for FLAT_INTERFACE. Thanks you.

# Minimal ``local.conf`` for single node devstack installation
# Please find the details at http://docs.openstack.org/developer/devstack/guides/single-machine.html#run-devstack
#
# The minial values to set up is
#   1) passwords for SERVICE_TOKEN, ADMIN_PASSWORD, MYSQL_PASSWORD, RABBIT_PASSWORD, and SERVICE_PASSWORD
#   2) FLAT_INTERFACE
#   3) PUBLIC_INTERFACE
#
# For details of each key, please read the comment on them.

[[local|localrc]]

SERVICE_TOKEN=your_random_servive_token_here
ADMIN_PASSWORD=your_password_here
MYSQL_PASSWORD=your_password_here
RABBIT_PASSWORD=your_password_here
SERVICE_PASSWORD=$ADMIN_PASSWORD

FIXED_RANGE=10.11.12.0/24
FIXED_NETWORK_SIZE=256

# Network interface for inter-VM communication.  Be cautious because DHCP
# packets for VMs will flow to this interface.  If you have only one NIC
# (network interface card), then create a virtual NIC following this
# (http://unix.stackexchange.com/questions/152331/how-can-i-create-a-virtual-ethernet-interface-on-a-machine-without-a-physical-ad)
# and assign it here.
FLAT_INTERFACE=eth1

# network interface connect to the public Internet
PUBLIC_INTERFACE=eth0



# Logging
# -------
LOGFILE=$DEST/logs/stack.sh.log
LOGDAYS=2


# Swift
# -----
SWIFT_HASH=66a3d6b56c1f479c8b4e70ab5c2000f5
SWIFT_REPLICAS=1
SWIFT_DATA_DIR=$DEST/data

I updated Elijah-openstack repo to show you a sample.

solimand 2016-02-26 12:10:30 UTC #15

Hi @Rim,
regarding local.conf, I have Devstack installed on single machine and I had to do only this to have a good file:

 cd /path-to-devstack/
 git checkout stable/kilo
 cp samples/local.conf local.conf

Consequently my local.conf file, located at devstack root directory, is the same of https://github.com/openstack-dev/devstack/blob/master/samples/local.conf. For my minimal configuration I did not have to do anything more. This file is right for a fresh install of ubuntu and for devstack in single (bare metal) machine. You should specify manually infos only for complex configurations, if they differ from simple eth0 and default value, or if you have to change your IP.
The execution of stack.sh script, automatically gets your actual IP.

Please try the @krha file and update us, sorry for the poor information that i give you. I hope you work around this problem.

P.S. my local.conf differs from yours in these lines:

I did not have to specify this,

neither this;

I run the console manually (nova-consoleauth --config-file /etc/nova/nova.conf) for speed up the start.

Everything else is same of yours.

Best Regardsa

UPDATE = remember that for resume base VM and Synthesis, after a ./stack.sh you have to reinstall elijah-openstack, regardless of previous installation.

Eiji 2016-03-24 08:17:09 UTC #16

Hello OEC,

I have same problem in this topic.

I built an all-in-one environment of OpenStack++ along the steps of the "Installation" of elijah-openstack (https://github.com/cmusatyalab/elijah-openstack).
Then, when I was trying to create VM overlay following the instraction of step 3 of the "How to use", VM Overlays status remains "Queued" and had never completed.
The below messages are output to n-cpu.log at that timing.

2016-03-15 10:02:26.355 ERROR oslo_messaging.rpc.dispatcher [-] Exception during message handling: Endpoint does not support RPC method cloudlet_overlay_finish
2016-03-15 10:02:26.355 TRACE oslo_messaging.rpc.dispatcher Traceback (most recent call last):
2016-03-15 10:02:26.355 TRACE oslo_messaging.rpc.dispatcher   File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 142, in _dispatch_and_reply
2016-03-15 10:02:26.355 TRACE oslo_messaging.rpc.dispatcher     executor_callback))
2016-03-15 10:02:26.355 TRACE oslo_messaging.rpc.dispatcher   File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 193, in _dispatch
2016-03-15 10:02:26.355 TRACE oslo_messaging.rpc.dispatcher     raise NoSuchMethod(method)
2016-03-15 10:02:26.355 TRACE oslo_messaging.rpc.dispatcher NoSuchMethod: Endpoint does not support RPC method cloudlet_overlay_finish
2016-03-15 10:02:26.355 TRACE oslo_messaging.rpc.dispatcher

In reference to the above information I tried this on both Ubuntu 14.04.03 LTS and Ubuntu 14.03.02 LTS,
but neither of these are unsuccessful; I was not able to create VM overlay by using OpenStack++ version.

It would be helpful if you had any advice.
Thanks.

Version infomation of Ubuntu 14.04.03

OS:ubuntu-14.04.3-server-amd64

$ libvirtd --version
libvirtd (libvirt) 1.2.2

$ kvm --version
QEMU emulator version 2.0.0 (Debian 2.0.0+dfsg-2ubuntu1.22), Copyright (c) 2003-2008 Fabrice Bellard
~/devstack$ git branch -a
  master
* stable/kilo
  remotes/origin/HEAD -> origin/master
  remotes/origin/master
  remotes/origin/stable/kilo
  remotes/origin/stable/liberty

solimand 2016-03-31 16:24:21 UTC #17

Hi @Eiji,
the different version of ubuntu was not a problem in my case. I have a fresh installed version of cloudlet on an updated ubuntu (14.04.4). I am not a developer of the Elijah system and I don't know much about this type of error, but from this error message:

there seems to be a problem with a function (cloudlet_overlay_finish) of the framework. Did you install the components in the correct order?

elijah-provisioning (fab install)
devstack (stack.sh)
elijah-openstack (fab localhost devstack_single_machine).
Unstack.sh; rejoin_stack.sh.

Remember that for every stack.sh you have to install elijah (elijah-openstack), regardless of previous installations.

krha 2016-04-01 02:51:16 UTC #18

Hi @Eiji,

I agree with @solimand's opinion.

NoSuchMethod: Endpoint does not support RPC method cloudlet_overlay_finish

cloudlet_overlay_finish is a cloudlet-openstack binding method (https://github.com/cmusatyalab/elijah-openstack/blob/master/compute/cloudlet_manager.py#L103). And this error tells that the OpenStack cannot find my modification on OpenStack code. That being said, OpenStack dashboard has a cloudlet panel, but the underlying code does not have cloudlet-related code. I guess this could happen if you re-install OpenStack and does not re-install elijah-openstack code.

Eiji 2016-04-11 03:44:55 UTC #19

Hi @solimand and @krha,
Thank you very much for your help.

By trying your advices and referencing other information at OEC forum,
I guess the following steps are necessary in addition to elijah-openstack(https://github.com/cmusatyalab/elijah-openstack).

・to assign a dummy interface as FLAT_INTERFACE if there is only one interface(NIC)
・to make "stack-volumes-lvmdriver-1" if it doesn't exist
・to restart apache2 and keystone after rejoin-stack.sh

Please tell me if there are other things to notice.
Owing to you, I was able to run OpenStack++ and create a VM overlay now!

Thanks,
Eiji

krha 2016-04-11 13:31:23 UTC #20

Hi Eiji,

You're right. If you have only one NIC, you need to create a dummy NIC. And rejoin-stack.sh sometime does not automatically run keystone-all nor apache2. I don't know about 2. Thank you.