Error to resume base VM

[zetelight]

Hi there, recently I got a common error(maybe) about resuming Base VM.
After I successfully import the Base VM provided in the repo, when I tried to resume Base VM, I always got this error in the GUI.

Unlike the discussion here, I even cannot allocate one IP to this instance. I also read through all the discussion and tried some possible solutions, but they didn't work.

What I did:
I tried to run cloudlet list-base, there's no base here(But I imported one via GUI, right?). Anyway, I went ahead and ran some commands. I got this image below.

I was also able to run cloudlet overlay and had the overlap.zip stored somewhere(but it didn't show up anything when I typed cloudlet list-overlay (kinda weird)

Background:
Fresh installed Ubuntu 14.04 64 bits
Previously installed OpenStack++(cloudlet, v0.9.4) (I already re-ran ansible-playbook -i ./hosts openstackpp.yaml after reboot)
CPU: i5-7200U 64 bits

I am still trying to figure it out but if you can give me some directions, it would be so helpful!

[teiszler]

@zetelight,

A few things:

1. The base images/overlays that are stored in OpenStack are entirely separate from the database that the clouldet command-line utility uses. Openstack uses glance for the images. This is why changes in one are not reflected in the other place. If you create a base VM with the command-line tool, you should be able to import into OpenStack however. Because the base VM launches and you can create an overlay with the command-line tool, I suspect that there is no problem with the image itself.
2. cloudlet list-overlay not displaying anything is possibly an artifact of the code evolving over time or it could just be a bug. I can look into it.
3. Regarding the VM not resuming: the error in the web UI is almost never helpful. At that level, it really just knows that it couldn't find a compute node to launch the VM. Looking at the nova logs in /var/log/nova is sometimes more helpful. Also you can go to the Admin section in the dashboard and look to see if any of the services (nova-compute, nova-network etc are down). I noticed that you said you rerun the playbook after reboot. Is it possible that you have a single NIC and were using the virtual NIC created by the playbook for the internal nova network? If so, the current methodology is not persistent and so this virtual NIC needs to be recreated after rebooting. Now, the re-running of the playbook should have done that, but perhaps something went awry. Check to see if that interface exists (you can also try running the commands from that part of the playbook manually).

Regards,
Tom

[zetelight]

@teiszler
Dear Tom,
your suggestions are very helpful.
You are right. I believed that the problem is located at the virtual NIC as well.
What I did:
1. I tried to re-run the playbook several times and it didn't help
2. I tried to create new virtual interface and it worked. However, what I did is using a new cidr to replace the ordiginal one by using nova network-create vmprivate --fixed-range-v4 10.11.13.0/24 --fixed-cidr 10.11.13.0/24 --bridge br100 --bridge-interface veth1. Because it warned CidrConflict: Requested cidr (10.11.12.0/24) conflicts with existing cidr(10.11.12.0/24).

Question:
I feel the solution I used above is not perfect since it will change the cidr each time.
May I ask how to run the part of playbook you mentioned above? I tried to run it but it gave

 [WARNING]: provided hosts list is empty, only localhost is available. Note that the implicit localhost does not match 'all'

ERROR! 'template' is not a valid attribute for a Play

The error appears to have been in '/root/elijah-openstack/ansible/roles/openstack-controller/tasks/nova.yml': line 1, column 3, but may
be elsewhere in the file depending on the exact syntax problem.

The offending line appears to be:


- name: (NOVA) transfer novadb SQL script
  ^ here

What's the right procedure to re-create the virtual NIC? By the way, the right procedure here means re-creating virtual NIC with the same cidr
I don't know ansible-playbook a lot so I am afraid that any change I make will let me get more troubles.

Thank you for your help and patience!
Adam

[teiszler]

Adam,

I don't believe it should have been necessary to recreate the private nova network. I was thinking you just needed to do the modprobe to load the kernel module and then recreate the interface with ip link (see the tasks that have SINGLE NIC in their names in nova.yml)

I think bringing that interface up is the key. You then would likely have to restart nova-network (service nova-network restart). You can clean up the old subnet by using nova network-delete. You may have to do nova network-disassociate first if it is still attached to some instance.

Regarding the error you received running the playbook: I am not sure where template comes from. Did you manually edit nova.yml? Is it a correct assumption that you ran the ansible-playbook command with --tags nova? It seems like it is complaining that the template keyword on the first line of nova.yml is invalid. Is it possible that you somehow removed the whitespace on that line such that it thinks template acts at the play (top) level instead of the tasks level (where it is a valid keyword)?

Regards,
Tom

[zetelight]

Thank you for these new updates.
For the nova.yml, I simply run ansible-playbook nova.yml (the way I ran it could be wrong)

The interesting thing is, today when I reboot again, after I re-ran the ansible playbook script, the virtual NIC was re-created by the script, so I don't have to run these commands you suggested above. It is pretty nice!

However, I noticed the instance doesn't have internet connection.
For example, I have one instance whose ip address is 10.11.13.4. On my local machine, I can ping 10.11.13.4 also ssh cloudlet@10.11.13.4. But when I tried to ran any command which needs internet within the instance, it would fail

cloudlet@ubuntu:~$ ping 8.8.8.8
PING 8.8.8.8 (8.8.8.8) 56(84) bytes of data.
^C
--- 8.8.8.8 ping statistics ---
158 packets transmitted, 0 received, 100% packet loss, time 158256ms


How could I resolve this issue? Because I want to deploy my backend server in the instance and let front-end server call its API from outside, I feel that the internet connection is necessary here. Please help. Thank you!

Best wishes,
Adam

[teiszler]

Adam,

That is correct. Instances in OpenStack cannot reach outbound nor be reached from outside until they have been assigned a floating (public) IP address. If you look at step 8 (and 9) of the README for elijah-openstack you can see the commands to create the floating IP pool and manage the security group rules so that the VM can be accessible from/to the outside world.

Regards,
Tom

[zetelight]

@teiszler
Dear Tom, I appreciate your help but it seems I need your help more since my knowledge is limited in this area.
What I have now
I cleaned up the nova network and the pool so currently, it has the following:

root@adamchen-ThinkPad-T470:~/elijah-openstack/ansible# nova network-list
+--------------------------------------+------------+---------------+
| ID | Label | Cidr |
+--------------------------------------+------------+---------------+
| 4286c41d-2825-48d2-8c0a-4c9cde7f5127 | vmprivate1 | 10.11.12.0/24 |
+--------------------------------------+------------+---------------+

root@adamchen-ThinkPad-T470:~/elijah-openstack/ansible# nova floating-ip-bulk-list
then it will list many floating IP address of 10.11.12.x (from 1 to 254)

What I did
As you can see, I do have a pool by following the step 8, so the pool is there. Will each instance be assigned the floating ip address by nova automatically?

Note: I always can ping or ssh to my instances after they are created.

Anyway, I did two things.

1. I tried to assign ip to my instance by using the allocate IP to Project under Access & Security, so it looks like this.
   
   However, from the console of instance, it still cannot ping 8.8.8.8
2. I tried to add many rules to the default Security Group, then it looks like this:
   
   
   I created a new instance based on this security group, but it still didn't help. Which rules do I miss here?

I really appreciate your work and patience. Thank you anyway!
If there's any suggestion, please let me know!

Best wishes,
Adam

[teiszler]

The floating IP pool should not be in the same subnet as your private subnet. You should delete your floating pool with nova floating-ip-bulk-delete (make sure you firstdisassociate that IP from your instance in the dashboard or it will complain). Then recreate the pool with some other subnet say, 172.20.0.0/24. The security group rules look ok, though you should just leave the prefix as 0.0.0.0/24 and it will apply to everything. After you recreate the pool with a different subnet and allocate one IP from the pool to your instance, then try to get outbound again from within your instance.

Regards,
Tom

[zetelight]

Hi Tom, thank you for the update.
I did what you suggest, but it seemed it doesn't work. Please let me post these infos below.

Dashboard & Secruity Group

Nova Network

Cloudlet instance

I felt that we are in the right direction, but I just cannot find some useful clues to go forward.

Thank you for your help in advance!

Best wishes,
Adam

[linuix]

Could you create "Overlay VM" after resume the base VM? When i create overlay i always occured some error?
I see if you can give me some help?

[zetelight]

i dont have too much experience either but I suggest you re-install the whole thing. It may helps

[zetelight]

@teiszler
Hi Tom,
I just got a new idea about my problem.
It could be wrong but since I can ping that instance from outside(it cannot ping to outside), is it possible that I deploy server within the instance and access it by its APIs from outside?
For example, I have to create a new Base VM which contains all development environment and backend codes I need, then I run it and expose its port, say 5000. Then, althought it doesn't have internet but I can access it by accessing the ip address plus its port(like 10.11.12.1:5000)

[teiszler]

Certainly you could create a base vm which already contains all the libraries and packages you need and then when you resume it you likely wouldn't need the uplink in order to download software. However, if you plan on connect any client to the application you are going to run in this VM, there needs to be a path back to that client. It seems like it is a issue with the routes.

If you ssh into the VM and perform a DNS lookup (nslookup kernel.org) does it resolve to an IP address? Does ip route show that the default route is through the host gateway at 10.11.12.1? And can you ping the host machine on 10.11.12.1?

[zetelight]

Hi Tom,
Thank you for your ideas. The followings are what I get from nslookup kernel.org and ip route.

cloudlet@ubuntu:~$ nslookup kernel.org
Server:		10.11.12.1
Address:	10.11.12.1#53

Non-authoritative answer:
Name:	kernel.org
Address: 198.145.29.83

cloudlet@ubuntu:~$ ip route
default via 10.11.12.1 dev eth0  metric 100 
10.11.12.0/24 dev eth0  proto kernel  scope link  src 10.11.12.7

I can ping to 10.11.12.1 also 10.11.12.7(which is the ip for current instance).
BTW, the current info for the instance is

Should I set someting like static route?
Best regards,
Adam

[zetelight]

@teiszler
Hi Tom, I could be wrong but it seems static route doesn't work either.

`
I feel like if I cannot figure it out by next week, I gonna re-install whole thing again

[zetelight]

@teiszler
Hi Tom, I've investigated this problem for a few days(I also reinstall the whole thing a few times)and I really do not want to waste our time and mislead you.
This post seems helpful but I cannot resolve my problem based on the discussion of that post.
I want to make sure my configurations are right, Please check them out below.

1. Ansible Variables.
   pub_iface: em1
flat_iface: veth1
2. Private Network
   nova network-create vmprivate --fixed-range-v4 10.11.12.0/24 --fixed-cidr 10.11.12.0/24 --bridge br100 --bridge-interface veth1
3. pool of floating IP addresses
   nova floating-ip-bulk-create 172.20.0.0/24
4. Security Group
   Add for all ICMP/TCP/UDP with cidr: 0.0.0.0/24
5. Procedure for creating an instance
   resume the base VM -> associate a floating ip to the instance -> go to 'Project/instance' and send "CTRL+ALT+DEL" to the instance so that we can play with it(I noticed only this way can let me ping the instance. If I try to soft-reboot the instance, the instance will get some NoInstanceFound Exception)

I've read several posts about networking in our forum and I really feel I am using the right way to deploy cloudlet, but somehow it just didn't work well, which is kinda frustrating.

I will try some known solutions more times.
Anyway, thank you for time and patience.

Best regards,

Adam

[teiszler]

Hi @zetelight,

Your configuration in 1-4 seems to be correct. I am not sure I follow what you are doing in #5. Earlier you had output of an nslookup and ip route: these were done on the VM instance correct? So you can ssh into the VM? It looks like it is using the 10.11.12.1 as the gateway IP which is correct and it resolves kernel.org which is good. If you can get into the instance again, and do an ip addr does it say 10.11.12.7 (or whatever IP is currently assigned) as the IP address? Can you check on the host machine that the veth1 interface exists and that brctl show indicates that veth1 is attached to the bridge? You might try to do a tcpdump -i veth1 in a separate terminal while attempting to ping the outside world from inside the VM and see if you observe any traffic on the host.

Hope this helps,
Tom

[zetelight]

Hi Tom @teiszler
Some updates
I wanna give some updates. Anyway, I re-installed the whole thing and I encountered the exact same issue as this post had(l3driver call to add floating ip failed (HTTP 400)). However, after re-run the service nova-network restart, I got the same issue which is our starting point of this post "No valid host was found" ( It's kinda late today so I will address this issue tomorrow. Also, let you know any updates)
What I mean by the step 5
What I wanna do is just resuming a base vm and give it a floating ip address so that it can ping outside(like 8.8.8.8). As README mentioned "Resumed VMs have incorrect IP address", I cannot soft-boot the instance cuz it gave me some errors. Hence, I send "CTRL+ALT+DEL" as another way to reboot it.
The outputs you asked
That's what I get from instance

That's what I get from host machine

Clearly, it doesn't attach with the veth1. That's an interesting finding.

Anyway, thank you for your patience and time! I will give updates as soon as possible!
Best regards,
Adam

[zetelight]

Hi Tom, happy holiday! @teiszler
Unfortunately, I got some errors here after re-try.
A little background:
a fresh installed Ubuntu 14.04 LTS

nova.conf:
public_interface = em1
vlan_interface = veth1
flat_interface = veth1

What I met
After I install cloudlet, I can resume VM but I cannot allocate a floating ip for it. Exactly like this post, I had l3driver call to add floating ip failed (HTTP 400) and Interface eth0 not found in /var/log/upstart/nova-network.log. I followed the suggestion you gave in that post, which is restarting nova-network. However, I still cannot resolve l3driver call to add floating ip failed (HTTP 400) but I have Interface em1 not found. It seems, after restarting nova-network, the pub_interface was lost for some reasons. I tried to re-run the ansible book again but cannot resolve this issue. What happened?

Thank you for your help in advance! Happy holiday!
Best regards,
Adam

[teiszler]

If you do ip addr on the host, is em1 the name of the real (not the veth1 interface you created) network interface? It still seems like there is a mismatch somewhere between the configuration and the actual interfaces. Once the configuration aligns with the OS, then restarting nova-network should solve it hopefully. Have a good holiday as well!