Cloudlet Provisioning: AppArmor / Libvirt crash on fresh Installation

Marco 2015-11-11 16:13:27 UTC #1

Hello all,

Having a fresh Ubuntu (server) base installation scripts hanged over libvirt library when trying to install following through: https://github.com/cmusatyalab/elijah-openstack instructions.

Find below snapshots of the output with the particular bits of the installation output:

[localhost] out: Setting up python3-libapparmor (2.8.95~2430-0ubuntu5.3) ...
[localhost] out: Setting up python3-pkg-resources (3.3-1ubuntu2) ...
[localhost] out: Setting up python3-apparmor (2.8.95~2430-0ubuntu5.3) ...
[localhost] out: Setting up apparmor-utils (2.8.95~2430-0ubuntu5.3) ...
(…)
[localhost] out: Setting up libvirt-daemon (1.2.9-9~u14.04+mos2) ...
[localhost] out: Setting up libvirt-daemon-system (1.2.9-9~u14.04+mos2) ...
[localhost] out: Adding group `libvirt' (GID 117) ...
[localhost] out: Done.
[localhost] out: Adding user `libvirt-qemu' to group `libvirt-qemu' ...
[localhost] out: Adding user libvirt-qemu to group libvirt-qemu
[localhost] out: Done.
[localhost] out:  * Starting libvirt management daemon libvirtd
[localhost] out:    ...done.
(…)
[localhost] out: Successfully installed python-dateutil msgpack-python SQLAlchemy bsdiff4 Cython psutil
[localhost] out: Cleaning up...
[localhost] out:

[localhost] sudo: aa-complain /usr/sbin/libvirtd
[localhost] out: Setting /usr/sbin/libvirtd to complain mode.
[localhost] out: Traceback (most recent call last):
[localhost] out:   File "/usr/sbin/aa-complain", line 30, in <module>
[localhost] out:     tool.cmd_complain()
[localhost] out:   File "/usr/lib/python3/dist-packages/apparmor/tools.py", line 166, in cmd_complain
[localhost] out:     self.reload_profile(profile)
[localhost] out:   File "/usr/lib/python3/dist-packages/apparmor/tools.py", line 257, in reload_profile
[localhost] out:     raise apparmor.AppArmorException(cmd_info[1])
[localhost] out: apparmor.common.AppArmorException: 'AppArmor parser error for /etc/apparmor.d/usr.sbin.libvirtd in /etc/apparmor.d/usr.sbin.libvirtd at line 44: syntax error, unexpected TOK_END_OF_RULE, expecting TOK_MODE\n'
[localhost] out:


Warning: sudo() received nonzero return code 1 while executing 'aa-complain /usr/sbin/libvirtd'!


Fatal error: Failed to disable AppArmor for custom KVM

Aborting.
Disconnecting from localhost... done.


root@node:~/elijah-provisioning# libvirt --version
No command 'libvirt' found, did you mean:
 Command 'libvirtd' from package 'libvirt-bin' (main)
libvirt: command not found
root@node:~/elijah-provisioning# kvm --version
QEMU emulator version 2.0.0 (Debian 2.0.0+dfsg-2ubuntu1.19), Copyright (c) 2003-2008 Fabrice Bellard
root@node:~/elijah-provisioning# uname -ar
Linux node.lab 3.13.0-66-generic #108-Ubuntu SMP Wed Oct 7 15:20:27 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux
root@node:~/elijah-provisioning# ps aux | grep libvirt
root     26460  0.0  0.3 386656 12788 ?        Sl   12:58   0:00 /usr/sbin/libvirtd -d
root     32019  0.0  0.0  10460   936 pts/4    S+   13:24   0:00 grep --color=auto libvirt
root@node:~/elijah-provisioning# dpkg -l | grep libvirt
ii  libvirt-bin                         1.2.9-9~u14.04+mos2                    amd64        programs for the libvirt library
ii  libvirt-clients                     1.2.9-9~u14.04+mos2                    amd64        programs for the libvirt library
ii  libvirt-daemon                      1.2.9-9~u14.04+mos2                    amd64        programs for the libvirt library
ii  libvirt-daemon-system               1.2.9-9~u14.04+mos2                    amd64        Libvirt daemon configuration files
ii  libvirt0                            1.2.9-9~u14.04+mos2                    amd64        library for interfacing with different virtualization systems
ii  python-libvirt                      1.2.9-1~u14.04+mos1                    amd64        libvirt Python bindings
root@node:~/elijah-provisioning#

No previous packages were installed (besides the ones on instruction #1: git, openssh-server and fabric of course).

Libvirt is constantly updated but only a few changes regarding apparmor interaction.
Anyone have a guess on what might be causing this?

With best regards,
Marco

EDIT:
Currently commenting lines #106 & #107 of fabfile.py (Elijah-provisioning) in order to proceed and check further impact.
-- Still, AppArmor or Libvirt does not give a change and are unforgiving here.
Now trying onto DevStack xD

Openstack++ in a single VM: CPU flag compatibility problem

krha 2015-11-12 14:14:18 UTC #2

Hi @Marco

Thanks for testing it. Your system had libvirtd but it seems that it's not regulated by the AppArmor. Can you paste the result of "$ sudo aa-status"? I don't know how does affect the VM overlay creation process, but could you please also show the error message of VM overlay creation?

And it would be helpful if you can tell me more about your Ubuntu server. (e.g. server or desktop, version)

In summary, I'd like to see
1) $ sudo aa-status
2) Error log of VM overlay creation (and dmesg output after hitting the error)
3) OS information.

Thanks.

Marco 2015-11-12 15:29:26 UTC #3

Hi @krha,

Thank you for the feedback.

1) aa-status

apparmor module is loaded.
6 profiles are loaded.
6 profiles are in enforce mode.
/sbin/dhclient
/usr/lib/NetworkManager/nm-dhcp-client.action
/usr/lib/connman/scripts/dhclient-script
/usr/lib/libvirt/virt-aa-helper
/usr/sbin/ntpd
/usr/sbin/tcpdump
0 profiles are in complain mode.
1 processes have profiles defined.
1 processes are in enforce mode.
/usr/sbin/ntpd (4256)
0 processes are in complain mode.
0 processes are unconfined but have a profile defined.

https://halflernation.ddns.net/owncloud/index.php/s/nq7iaRZSaRwt9OK

I have a few directories under /tmp/cloudlet* but these are empty.

3)
kvm --version

QEMU emulator version 2.0.0 (Debian 2.0.0+dfsg-2ubuntu1.19), Copyright (c) 2003-2008 Fabrice Bellard

libvirtd --version

libvirtd (libvirt) 1.2.9

uname -ar

Linux node-9.lab 3.13.0-66-generic #108-Ubuntu SMP Wed Oct 7 15:20:27 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux
(Server version)

lsb_release -a

No LSB modules are available.
Distributor ID: Ubuntu
Description: Ubuntu 14.04.3 LTS
Release: 14.04
Codename: trusty

/etc/debian_version

jessie/sid

krha 2015-11-12 16:31:50 UTC #4

Hi @Marco

I don't see any problem from the Ubuntu distribution you're using. In fact, I installed these modules at the same version a few weeks before without any problem.

So, I'm finding a clue from the error message "migration was active, but no RAM info was set." I cloned libvirt repo and this error message is coming from the following part.

(src/qemu/qemu_monitor_json.c)
2471 if (status->status == QEMU_MONITOR_MIGRATION_STATUS_ACTIVE ||
2472 status->status == QEMU_MONITOR_MIGRATION_STATUS_CANCELLING ||
2473 status->status == QEMU_MONITOR_MIGRATION_STATUS_COMPLETED) {
2474 virJSONValuePtr ram = virJSONValueObjectGetObject(ret, "ram");
2475 if (!ram) {
2476 virReportError(VIR_ERR_INTERNAL_ERROR, "%s",
2477 _("migration was active, but no RAM info was set"));

This is written at commit 3b7399b5c994bb718222db8a8f46c60984367065, Jul 18 16:22:03 2012 and libvirt 1.0.2 is at Wed Jan 30 10:43:33 2013. So, I don't think this problem is coming from the libvirt versioning problem.

At this point, I don't have a clue for this error. I'll follow exact configuration you did and will see whether I can reproduce this error. I can probably do that at the end of next week and will come back to you. Thanks.

Marco 2015-11-12 17:30:32 UTC #5

Didn't track the commit but I saw the library here as it wasn't receiving the ram virJSONValuePtr pointer.
Indeed you're not the only one thinking this is not a libvirt problem. Found this: redhat

I would say for you not to waste energy on this further this point. This was my first iteration and installed many libraries.
Will update once I'll be using a VM so it's easy to just kill and spawn a fresh one to test

Best regards,
Marco

krha 2015-11-19 13:18:42 UTC #6

Hi @Marco,

I recall you told me that you managed to make it work. Do I remember correctly?

If you did, please let me know what was the problem. Thanks!!

Marco 2015-11-20 14:30:53 UTC #7

Almost

I forgo the installation over HW and follow the installation over a VM approach.
I managed to get the install work correctly for DevStack (fully) and all services are operational. Glance not always binds but is operational still.

Currently some setting should not be fully right as OpenStack does not instantiate the cirrOS image (hence also fails on cloudlet). Getting error: "Failed to compute_task_build_instances: No valid host was found. There are not enough hosts available." which I have been troubleshooting (more info here), but I believe I've found the reason:
- trying to use Virsh to instantiate a VM I get:
"XML: unsupported configuration: guest and host CPU are not compatible: Host CPU does not provide required features: monitor" which gives me the indication at CPU flags.
Still I have checked cpuinfo and nx flag is present

Good news is: No Cloudlet responsibility here! So as soon as I crawl over this issues all I should be ok to use Cloudlet module (which is fully visible and callable at Horizon dashboard)

Thanks for all your support @krha !

Marco 2015-12-15 10:07:59 UTC #8

@krha,

I have provisioned the Cloudlet code. To avoid these sort of conflicts I enforced the following environment:

$ sudo apt-get install libvirt-bin=1.2.2-0ubuntu13.1.14 libvirt0=1.2.2-0ubuntu13.1.14 python-libvirt=1.2.2-0ubuntu2

libvirt-bin=1.2.2-0ubuntu13.1.14
libvirt0=1.2.2-0ubuntu13.1.14
python-libvirt=1.2.2-0ubuntu2

The AppArmor issue was resolved after these enforcement and had to stop the service before setting libvirtd to complain mode:

$ sudo /etc/init.d/apparmor stop
$ sudo aa-complain /usr/sbin/libvirtd
$ sudo /etc/init.d/apparmor start
* Starting AppArmor profiles
 Skipping profile in /etc/apparmor.d/disable: usr.sbin.libvirtd

Avoided 99% of the problems this way.
Best regards

krha 2015-12-15 14:33:16 UTC #9

HI @Marco,

Thanks for the checking. I will look into what cause this library dependency and update my installation script. Thanks.