Installation issue for Openstack++ in a single VM

[MSoares]

Hello Everyone,
I am a Computer Science student. I tried to install the Elijah platform on my laptop with CPU = Intel Core i7, 2.2 GHz. RAM = 4GB. HDD = 300 GB, OS is Ubuntu 14.04.05 LTS.

I follow the guidelines present at http://github.com/cmusatyalab/elijah-openstack. I have sucessfully installed the cloudlet library and when I execute "cloudlet list-base", it appears an empty table with two columns: hash value and path.

So I start the installation of Openstack using devstack, I use your command lines and change local.conf and stackrc

I am stuck during ./stack.sh, it returns :
2016-12-05 14:01:06.067 | File "/opt/stack/keystone/keystone/cmd/cli.py", line 24, in
2016-12-05 14:01:06.067 | from oslo_log import versionutils
2016-12-05 14:01:06.067 | ImportError: cannot import name versionutils

At this point, I have googling a lot but nothing seems to fix that.
I have tried ./unstack.sh, ./clean.sh, sudo apt-get install python-pip, sudo pip install --upgrade pip and ./stack.sh, but this do not solve my problem.

I would appreciate your help to solve this out.
Bests regards,
Micael

[teiszler]

Micael @MSoares,

versionutils is coming from the olso_log module which is part of the OpenStack messaging platform. It seems that perhaps not all parts of OpenStack were installed correctly.

I have found (through much trial and error) that if things go wrong when installing DevStack, that it is best to run ./clean.sh (this should remove the OpenStack binaries at /usr/local/bin), remove the source directory where you cloned DevStack and start again at the top of the instructions for installing DevStack. You shouldn't need to reinstall the cloudlet library as that is working just fine.

See if starting from scratch you run into the same difficulty when starting the stack with ./stack.sh.

Thanks,
Tom

[MSoares]

Dear Tom,

first of all, thanks for your fast reply.

I have reinstall devstack and I was stuck again with some errors, but after a while I manage to finish installation successfully. I think there was some bad Internet connection or some problem with pip server. Anyway.

So, right now I am able to enter Dashboard, and start next phase I go cloudlet section and I import Base VM precise-hotplug.zip, nice it appears sucessfully. So I try to resume this image but I get stuck in that phase 2 times. First time, very quickly it shut down and return error, the second time it start but stop after a little while. I cannot delete this instances.

If I go to Instances section, I can see this 2 instances and I try to erase it and nothing happens.

I can check logs

So, I try to google this problem and I get in 2 new problems
1) when I put nova-manage service list before making .&rejoin/stack.sh, I have

After a rejoin/stack.sh, if I put nova-manage service list I will get

2) I have read that I have to check logs in /var/log/nova directory but it dont exist.

Please help me solve this because I am really stuck and I have no clue what is happening.

Bests regards,
Micael

[teiszler]

Micael,

You should be able to look at the logs for nova-network (the service that isn't running when you to nova-manage service list) in one of two ways:

1. 'screen -x' which will use screen to allow you to view all the OpenStack logs at once. It is a bit difficult to navigate in screen, you have to use CTRL-A-N to move to the next service and CTRL-A-P to move the to previous one. You would want to move to n-net which is the nova-net log. You can then scroll through the log by doing CTRL-A-ESC and then using the arrow keys.

2. The log files should be accessible in /opt/stack/logs. You want to look at n-net.log.

It seems judging from the error you are receiving when trying to resume the VM, that because nova-network is down, the VM cannot be resumed. Hopefully the log will lead you to why nova-network is down and then you can proceed from there.

Thanks,
Tom

[MSoares]

Dear Thomas,

thanks for your help, I have check the n-net.log I have found that I had no second NIC active.
So, I have finish resume-VM, I have also create VM overlay. In another computer I have installed webserver and and put there vm overlay. After that I start VM Synthesis.

I have wait for 30 minutes and it always appears "scheduling", and I think something suspicious is going on.

I do not have directory /var/log/nova and with screen -x I see possible errors in n-net.log:

e[01;31m2016-12-14 00:08:57.688 TRACE nova.openstack.common.periodic_task e[01;35me[00m File "/opt/stack/nova/nova/openstack/common/periodic_task.py", line 224, in run_periodic_tasks
e[01;31m2016-12-14 00:08:57.688 TRACE nova.openstack.common.periodic_task e[01;35me[00m task(self, context)
e[01;31m2016-12-14 00:08:57.688 TRACE nova.openstack.common.periodic_task e[01;35me[00m File "/opt/stack/nova/nova/network/manager.py", line 346, in disassociatestale_fixed_ips
e[01;31m2016-12-14 00:08:57.688 TRACE nova.openstack.common.periodic_task e[01;35me[00m time)
e[01;31m2016-12-14 00:08:57.688 TRACE nova.openstack.common.periodic_task e[01;35me[00m File "/opt/stack/nova/nova/objects/fixed_ip.py", line 175, in disassociate_all_by_timeout
e[01;31m2016-12-14 00:08:57.688 TRACE nova.openstack.common.periodic_task e[01;35me[00m timeutils.isotime(time))
e[01;31m2016-12-14 00:08:57.688 TRACE nova.openstack.common.periodic_task e[01;35me[00m File "/opt/stack/nova/nova/objects/base.py", line 161, in wrapper
e[01;31m2016-12-14 00:08:57.688 TRACE nova.openstack.common.periodic_task e[01;35me[00m args, kwargs)
e[01;31m2016-12-14 00:08:57.688 TRACE nova.openstack.common.periodic_task e[01;35me[00m File "/opt/stack/nova/nova/conductor/rpcapi.py", line 335, in object_class_action
e[01;31m2016-12-14 00:08:57.688 TRACE nova.openstack.common.periodic_task e[01;35me[00m objver=objver, args=args, kwargs=kwargs)
e[01;31m2016-12-14 00:08:57.688 TRACE nova.openstack.common.periodic_task e[01;35me[00m File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/client.py", line 156, in call
e[01;31m2016-12-14 00:08:57.688 TRACE nova.openstack.common.periodic_task e[01;35me[00m retry=self.retry)
e[01;31m2016-12-14 00:08:57.688 TRACE nova.openstack.common.periodic_task e[01;35me[00m File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/transport.py", line 90, in _send
e[01;31m2016-12-14 00:08:57.688 TRACE nova.openstack.common.periodic_task e[01;35me[00m timeout=timeout, retry=retry)
e[01;31m2016-12-14 00:08:57.688 TRACE nova.openstack.common.periodic_task e[01;35me[00m File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py", line 350, in send
e[01;31m2016-12-14 00:08:57.688 TRACE nova.openstack.common.periodic_task e[01;35me[00m retry=retry)
e[01;31m2016-12-14 00:08:57.688 TRACE nova.openstack.common.periodic_task e[01;35me[00m File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py", line 339, in _send
e[01;31m2016-12-14 00:08:57.688 TRACE nova.openstack.common.periodic_task e[01;35me[00m result = self.waiter.wait(msgid, timeout)
e[01;31m2016-12-14 00:08:57.688 TRACE nova.openstack.common.periodic_task e[01;35me[00m File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py", line 243, in wait
e[01;31m2016-12-14 00:08:57.688 TRACE nova.openstack.common.periodic_task e[01;35me[00m message = self.waiters.get(msg_id, timeout=timeout)
e[01;31m2016-12-14 00:08:57.688 TRACE nova.openstack.common.periodic_task e[01;35me[00m File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py", line 149, in get
e[01;31m2016-12-14 00:08:57.688 TRACE nova.openstack.common.periodic_task e[01;35me[00m 'to message ID %s' % msg_id)
e[01;31m2016-12-14 00:08:57.688 TRACE nova.openstack.common.periodic_task e[01;35me[00mMessagingTimeout: Timed out waiting for a reply to message ID e4d1bc6b85394d01b8695cd456a75ada
e[01;31m2016-12-14 00:08:57.688 TRACE nova.openstack.common.periodic_task e[01;35me[00m
2016-12-14 00:08:57.689 e[00;32mDEBUG nova.openstack.common.loopingcall [e[01;36mreq-ef8349df-2023-49b7-b6f6-d06b9ce7dab6 e[00;36mNone Nonee[00;32m] e[01;35me[00;32mDynamic looping call <bound method Service.periodic_tasks of > sleeping for 60.00 secondse[00m e[00;33mfrom (pid=17301) _inner /opt/stack/nova/nova/openstack/common/loopingcall.py:132e[00m
2016-12-14 00:09:57.691 e[00;32mDEBUG nova.openstack.common.periodic_task [e[01;36mreq-ef8349df-2023-49b7-b6f6-d06b9ce7dab6 e[00;36mNone Nonee[00;32m] e[01;35me[00;32mRunning periodic task FlatDHCPManager.disassociatestale_fixed_ipse[00m e[00;33mfrom (pid=17301) run_periodic_tasks /opt/stack/nova/nova/openstack/common/periodic_task.py:219e[00m
2016-12-14 00:10:57.700 e[01;31mERROR nova.openstack.common.periodic_task [e[01;36mreq-ef8349df-2023-49b7-b6f6-d06b9ce7dab6 e[00;36mNone Nonee[01;31m] e[01;35me[01;31mError during FlatDHCPManager.disassociatestale_fixed_ips: Timed out waiting for a reply to message ID 3a985579ceb44fc0b3cd7d7fb6c1206ee[00m
e[01;31m2016-12-14 00:10:57.700 TRACE nova.openstack.common.periodic_task e[01;35me[00mTraceback (most recent call last):
e[01;31m2016-12-14 00:10:57.700 TRACE nova.openstack.common.periodic_task e[01;35me[00m File "/opt/stack/nova/nova/openstack/common/periodic_task.py", line 224, in run_periodic_tasks
e[01;31m2016-12-14 00:10:57.700 TRACE nova.openstack.common.periodic_task e[01;35me[00m task(self, context)
e[01;31m2016-12-14 00:10:57.700 TRACE nova.openstack.common.periodic_task e[01;35me[00m File "/opt/stack/nova/nova/network/manager.py", line 346, in disassociatestale_fixed_ips
e[01;31m2016-12-14 00:10:57.700 TRACE nova.openstack.common.periodic_task e[01;35me[00m time)
e[01;31m2016-12-14 00:10:57.700 TRACE nova.openstack.common.periodic_task e[01;35me[00m File "/opt/stack/nova/nova/objects/fixed_ip.py", line 175, in disassociate_all_by_timeout
e[01;31m2016-12-14 00:10:57.700 TRACE nova.openstack.common.periodic_task e[01;35me[00m timeutils.isotime(time))
e[01;31m2016-12-14 00:10:57.700 TRACE nova.openstack.common.periodic_task e[01;35me[00m File "/opt/stack/nova/nova/objects/base.py", line 161, in wrapper
e[01;31m2016-12-14 00:10:57.700 TRACE nova.openstack.common.periodic_task e[01;35me[00m args, kwargs)
e[01;31m2016-12-14 00:10:57.700 TRACE nova.openstack.common.periodic_task e[01;35me[00m File "/opt/stack/nova/nova/conductor/rpcapi.py", line 335, in object_class_action
e[01;31m2016-12-14 00:10:57.700 TRACE nova.openstack.common.periodic_task e[01;35me[00m objver=objver, args=args, kwargs=kwargs)
e[01;31m2016-12-14 00:10:57.700 TRACE nova.openstack.common.periodic_task e[01;35me[00m File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/client.py", line 156, in call
e[01;31m2016-12-14 00:10:57.700 TRACE nova.openstack.common.periodic_task e[01;35me[00m retry=self.retry)
e[01;31m2016-12-14 00:10:57.700 TRACE nova.openstack.common.periodic_task e[01;35me[00m File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/transport.py", line 90, in _send
e[01;31m2016-12-14 00:10:57.700 TRACE nova.openstack.common.periodic_task e[01;35me[00m timeout=timeout, retry=retry)
e[01;31m2016-12-14 00:10:57.700 TRACE nova.openstack.common.periodic_task e[01;35me[00m File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py", line 350, in send
e[01;31m2016-12-14 00:10:57.700 TRACE nova.openstack.common.periodic_task e[01;35me[00m retry=retry)
e[01;31m2016-12-14 00:10:57.700 TRACE nova.openstack.common.periodic_task e[01;35me[00m File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py", line 339, in _send
e[01;31m2016-12-14 00:10:57.700 TRACE nova.openstack.common.periodic_task e[01;35me[00m result = self.waiter.wait(msgid, timeout)
e[01;31m2016-12-14 00:10:57.700 TRACE nova.openstack.common.periodic_task e[01;35me[00m File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py", line 243, in wait
e[01;31m2016-12-14 00:10:57.700 TRACE nova.openstack.common.periodic_task e[01;35me[00m message = self.waiters.get(msg_id, timeout=timeout)
e[01;31m2016-12-14 00:10:57.700 TRACE nova.openstack.common.periodic_task e[01;35me[00m File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py", line 149,

Thanks for your help in advance

Bests regards,
Micael

[MSoares]

Finally, the scheduling has finished with the error.

[teiszler]

Micael,

It might be beneficial to just restart all of the OpenStack components. If you are still in screen, you can do CTRL-A-D to detach. Once, or if you already are, out of screen, you can do ./unstack.sh in the devstack directory and then ./rejoin-stack.sh. (You may need to manually restart Apache from another terminal using 'sudo service apache2 restart' and you may also need to bring up the OpenStack authentication service called keystone by doing 'keystone-all &') Once you have succesfully restarted OpenStack, if you go into the web UI and under the Admin->System Panel on the left, click System Information, there should be a tab for Compute Services. There you can check to see if all the nova services are running. If they are all up, then you can try to synthesize the VM instance again using the overlay you created.

Regards,
Tom

[MSoares]

Hi Tom,

I was already using keystone-all in order to login into dashboard.
As you say, I have restart all components, and it finish to make instance.

Right now, I can synthesized VM, but I cannot access log and console, as you can see below.

I google it and I found many things that told to search into /var/log/nova/ but I do not have this directory. I tried also to include console in Stackrc file with other enabled service, but nothing happen.

Thanks for your help.

Bests regards,
Micael
I would like to check console and log to see what is happening, do you have a clue, help

[teiszler]

Micael,

The logs are in /opt/stack/logs. You will likely want to look at n-cpu.log and n-net.log. You can 'tail -f ' those log files when performing synthesis to see what is going on. This is probably easier than navigating from within 'screen'. Also, one the System Information screen in the console, you will want to look at the Compute Services tab to see if all the nova compute services are up; in your screenshot you are still on the default services tab.

Thanks,
Tom

[MSoares]

Dear Tom,

in order to see console in Dashboard I had to start console in new terminal. I have search and try somethings I found in google, but I am still not able to see logs.
Anyway, when I launch a new instance, it booted OK and I was able to get a login prompt in console section.

If I try to login, VM do not recognize my keyboard.

I have try to synthesize another VM and it still happens. Is it a bug o console VNC?
I have read many articles and try some changes, like this one:
https://ask.openstack.org/en/question/96772/devstack-instance-console-keyboard-error/
and I try go to /opt/stack/noVNC and do a git checkout v0.6.0 and then launch again nova-consoleauth --config-file /etc/nova/nova.conf but it doesnt work.

Thanks for your help.

[teiszler]

When you say you can't see the logs, do you mean from within the OpenStack web UI? I haven't even been able to view the logs there myself. If you open a terminal and cd to /opt/stack/logs you should be able to see everything you need in terms of the nova-compute logs (n-cpu) or any of the other logs.

In regards to the VM that you connected to, that seems to be an image I am not familiar with. I think that may be one of the images that comes with DevStack. You should instead by looking at getting one of the Ubuntu 14.04 instances to launch that downloaded from the elijah-openstack README file (https://storage.cmusatyalab.org/cloudlet-vm/precise-hotplug.zip)

Seems like you are pretty close to getting it to work.

Regards,
Tom

[MSoares]

Hi Tom,

indeed I can see logs over /opt/stack/logs but I would bi nice to see it in dashboard directly.

I had already use Ubuntu 14.04 from elijah-openstack and it have give me some error last time.
I have restart consoleauth and use this time it works. I can access instance with keyboard.

I was wondering, if it possible to make a test application such @krha has make with android fluid app, that we can see at https://youtu.be/f9MN-kvG_ko. It would be nice to understand better how it works, and where can I get all pieces?

[teiszler]

I believe @krha has made this available on GitHub in the elijah-provisioning repository. I think you want the android-fluid project for the Android client.

cmusatyalab/elijah-provisioning

For the server process that will run on the VM, I believe this is the project:

cmusatyalab/elijah-provisioning

[MSoares]

Thanks Tom.

I appreciate your effort and incredible help.

[MSoares]

Dear Tom,

I have try to install fluid server backend, but the link provided return blank: https://storage.cmusatyalab.org/cloudlet-app/fluid-server-bin32.tar.gz

Is there any chance on providing this?

Bests regards,

Micael

[teiszler]

Micael,

Try this link.

https://owncloud.cmusatyalab.org/owncloud/public.php?service=files&t=821f2bae310e17aeb456fa4a2bc7ded3

Let me know if this works. The name of the tarball is a bit different, but I believe the binary is the same.

Thanks,
Tom

[MSoares]

Dear Tom,

thanks for your support.

So, in order to keep you updated. I was able to launch instance and access it through SSH, and then install fluid server and it runs fine.

After that, I had to try first time to make an apk through eclipse and it was not easy because it was my first time. It appears some errors, that I fix and make change for my local server ip, and make apk file., I have install it on my android phone and run it.
When I choose Cloudlet it start Initialization and it shows message that it is not possible to connect.

I have some doubts here:
1-) So I try to open port on instance and on dashboard in Acess adn Security/Security Groups but nothing change. I can ping Instance from android phone and vice versa.
I think it is necessary to make port forwarding from host to VM, am I right? But How to do it?
2-) In android phone I have to install in SDCard the files from overlay.zip, are we talking about this one?

I am very close to make it work all together.

I wish you, your family and all your team an happy new year 2017, full of love, peace and health.

Bests regards,
Micael

[teiszler]

Micael,

I think you are on the right track.

1. Under Access & Security-> Security Groups, click the 'Manage Rules' button for the default security group and then click 'Add Rule'. You should then be able to add a Custom TCP rule. The port should be the port that the fluid simulation is running on (probably 9093 according to your screenshot) and the CIDR should likely be something like 192.168.1.0/24 since that seems to be the range that OpenStack is allocating IPs to your VMs (at least according to the second screenshot where synthesized_vm is 192.168.1.227. This should allow traffic on that port to flow to the VM. Under the hood, I believe OpneStack does this by creating virtual bridges between the public facing IP address of the physical machine and the private IP address of each of the VMs. Hopefully that is what you need to get the traffic from the mobile device to the VM. Networking is arguably the most difficult part of getting this all working with OpenStack.

2. I am a bit unsure what you are referring to here. When you say overlay.zip, do you mean a downloaded copy of the VM overylay in your screenshot? (overylay-....fa05?) If so, then this overlay doesn't need to be on the Android device. The overlay is only needed when rapidly provisioning the VM from a base VM image. Since you have already synthesized the VM and it is listed on the Cloudlet Info page, then the VM is up and running. Though instead of having the overlay stored on a webserver somewhere like you have it now, or on Amazon S3 for example, you could envision a mobile device carrying around the overlay on its local storage and transferring the overlay data to a cloudlet at the time of provisioning. This approach might be beneficial for something like a speech translation application where the user might have very specific user data.

Thank you for the well wishes. I hope you and yours have a splendid 2017 as well.

Thanks,
Tom

[MSoares]

Dear Tom,

i have been able to make it work out finally. As you say, I had to make some changes in the access tab and restart instances and it work fine.
Thanks for your help dear Tom.

In order to finish my thesis, I need to make my own lab scenario: a mobile client running app and making handover between 2 cloudlets/instances. I have some doubts:
1-) At first I thought that Openstack would be a centralized platform and I could install and manages many cloudlet through the dashboard, but I have try to make it and it always have some error. Do I have to install openstack++ in each computer? Or can I install openstack ++ in one machine and running cloudlet in other machines through elijah provisionning.
2-) My version of openstack++ do not have neutron and heat apis, is that normal? Do you think that in my case it will be preferable to use these 2 APIs or others

Bests regards.
Micael

[teiszler]

Micael,

1. OpenStack is not without its difficulty and complexity. That being said, it is one of the widely used platforms for managing virtual infrastructure. You can think of an OpenStack cluster as a single cloudlet. In your case it is just a single node cluster, but it could in theory be a larger group of physical machines each running N number of virtual machines. In the testing I have done, I have setup OpenStack (via DevStack) on two physical machines and then effectively had two cloudlets. elijah-provisioning is designed to work without OpenStack. It has a set of command-line tools that allow you to do all the same functions (create an overlay, synthesize a VM from that overaly, handoff between two nodes). The elijah-openstack project (also known as OpenStack++) is a plugin extension to the OpenStack framework to hook in the underlying functionality of the elijah-provisioning project. So depending on the constraints of your thesis, you could setup two machines with elijah-provisioning and simply use the command-line tools to perform the handoff. The work in elijah-provisioning was at the heart of @krha thesis and showcases the modifications to QEMU and the adaptations he made to the handoff process so that resources on cloudlets can be provisioning rapidly and transfered with minimal downtime because a smooth mobile user experience is paramount. You can check out the README of the project (https://github.com/cmusatyalab/elijah-provisioning) and see how to perform some of the commands. Since you have already followed the directions on elijah-openstack to install elijah-provisioning, you should be able to launch these commands and see if they will be sufficient for your purposes.

2. Neutron is the successor to nova networking. It has some more complicated mechanisms for dealing with networking piece of an OpenStack cluster. For your purposes, since you have a fairly simple single server setup (or perhaps two single server clusters in the future) the simpler nova networking is probably sufficient. As far as heat, I am not sure what component that is. There are many different optional components in OpenStack and each have their own level of maturity and userbase. If you go to the OpenStack website you can find documentation on heat and see if it is something that would benefit you. I don't think it is part of DevStack however, so if you were interested in it, you would most likely have to install it manually.

Thanks,
Tom