[vserver] Upstart problems in Ubuntu guest (was> Re: [vserver] Problems installing Ubuntu 10.10 as a guest of a Debian host)

From: Markus Fischer <markus_at_fischer.name>
Date: Tue 19 Oct 2010 - 12:16:23 BST
Message-ID: <4CBD7E07.5060008@fischer.name>

On 18.10.2010 21:30, Herbert Poetzl wrote:
>> I'm running a Host with Debian Lenny on 2.6.26-2-vserver-amd64 with
>> VS-API: 0x00020303 and util-vserver: 0.30.216-pre2772; Dec 13 2008,
>> 04:56:19 (it's basically all standard Debian installation).
>
> I'd suggest to update to a more recent, less buggy kernel
> and util-vserver version

Thanks! I'll nag the Hardware-People about that, I just fear it won't
happen .. very.. soon :/

>> # dpkg-reconfigure locales
>> perl: warning: Setting locale failed.
>> perl: warning: Please check that your locale settings:
>> LANGUAGE = (unset),
>> LC_ALL = (unset),
>> LANG = "en_US.UTF-8"
>> are supported and installed on your system.
>> perl: warning: Falling back to the standard locale ("C").
>> locale: Cannot set LC_CTYPE to default locale: No such file or directory
>> locale: Cannot set LC_MESSAGES to default locale: No such file or directory
>> locale: Cannot set LC_ALL to default locale: No such file or directory
>
>> The locales package is installed:
>
>> # dpkg -l locales|grep '^ii'
>> ii locales 2.13+git20100825-1 common files
>> for locale support
>
> you also want to install the actual locale files
> not just the locale support, and either configure
> the locale to be the same as the host (en_US.UTF-8)
> or unset it when entering the guest

Funny thing is, I'm not aware that I'm actually missing anything. I've
watched the process with strace and figure the "No such file or
directory" related to this file: /usr/lib/locale/locale-archive .

I've set me up an Ubuntu server installation inside VMWare and the file
was simply there; but "dpkg -S" didn't knew about it either. For now I'm
just copying it over from an existing installation ... (?)

> http://linux-vserver.org/Upstart_issues

I've gone a few times through the document and was able to make some
progress but I think the whole upstart architecture doesn't really work
yet for me.

For example, by default "rsyslogd" is installed. When I start the
vserver and enter it, it's running. When I stop it with "stop rsyslog"
the output at the console hangs until I interrupt it:

socket(PF_FILE, SOCK_STREAM|SOCK_CLOEXEC, 0) = -1 EINVAL (Invalid argument)
socket(PF_FILE, SOCK_STREAM, 0) = 3
fcntl(3, F_GETFD) = 0
fcntl(3, F_SETFD, FD_CLOEXEC) = 0
connect(3, {sa_family=AF_FILE, path=@"/com/ubuntu/upstart"}, 22) = 0
fcntl(3, F_GETFL) = 0x2 (flags O_RDWR)
fcntl(3, F_SETFL, O_RDWR|O_NONBLOCK) = 0
geteuid() = 0
getsockname(3, {sa_family=AF_FILE, NULL}, [2]) = 0
poll([{fd=3, events=POLLOUT}], 1, 0) = 1 ([{fd=3, revents=POLLOUT}])
write(3, "\0", 1) = 1
sendto(3, "AUTH EXTERNAL 30\r\n", 18, MSG_NOSIGNAL, NULL, 0) = 18
poll([{fd=3, events=POLLIN}], 1, -1) = 1 ([{fd=3, revents=POLLIN}])
read(3, "OK 474d8e68be037c13cc979ed100012"..., 2048) = 37
poll([{fd=3, events=POLLOUT}], 1, -1) = 1 ([{fd=3, revents=POLLOUT}])
sendto(3, "NEGOTIATE_UNIX_FD\r\n", 19, MSG_NOSIGNAL, NULL, 0) = 19
poll([{fd=3, events=POLLIN}], 1, -1) = 1 ([{fd=3, revents=POLLIN}])
read(3, "AGREE_UNIX_FD\r\n", 2048) = 15
poll([{fd=3, events=POLLOUT}], 1, -1) = 1 ([{fd=3, revents=POLLOUT}])
sendto(3, "BEGIN\r\n", 7, MSG_NOSIGNAL, NULL, 0) = 7
poll([{fd=3, events=POLLIN|POLLOUT}], 1, -1) = 1 ([{fd=3, revents=POLLOUT}])
sendmsg(3, {msg_name(0)=NULL,
msg_iov(2)=[{"l\1\2\1\f\0\0\0\1\0\0\0_\0\0\0\1\1o\0\23\0\0\0/com/ubu"..., 112},
{"\7\0\0\0rsyslog\0", 12}], msg_controllen=0, msg_flags=0},
MSG_NOSIGNAL) = 124
poll([{fd=3, events=POLLIN}], 1, 25000) = 1 ([{fd=3, revents=POLLIN}])
recvmsg(3, {msg_name(0)=NULL,
msg_iov(1)=[{"l\2\1\1%\0\0\0\1\0\0\0\17\0\0\0\5\1u\0\1\0\0\0\10\1g\0\1o\0\0"...,
2048}], msg_controllen=0, msg_flags=MSG_CMSG_CLOEXEC}, MSG_CMSG_CLOEXEC)
= 69
recvmsg(3, 0x7fffbc2d59e0, MSG_CMSG_CLOEXEC) = -1 EAGAIN (Resource
temporarily unavailable)
sendmsg(3, {msg_name(0)=NULL,
msg_iov(2)=[{"l\1\2\1\4\0\0\0\2\0\0\0x\0\0\0\1\1o\0 \0\0\0/com/ubu"...,
136}, {"\0\0\0\0", 4}], msg_controllen=0, msg_flags=0}, MSG_NOSIGNAL) = 140
poll([{fd=3, events=POLLIN}], 1, 25000) = 1 ([{fd=3, revents=POLLIN}])
recvmsg(3, {msg_name(0)=NULL,
msg_iov(1)=[{"l\2\1\1'\0\0\0\2\0\0\0\17\0\0\0\5\1u\0\2\0\0\0\10\1g\0\1o\0\0"...,
2048}], msg_controllen=0, msg_flags=MSG_CMSG_CLOEXEC}, MSG_CMSG_CLOEXEC)
= 71
recvmsg(3, 0x7fffbc2d5990, MSG_CMSG_CLOEXEC) = -1 EAGAIN (Resource
temporarily unavailable)
sendmsg(3, {msg_name(0)=NULL,
msg_iov(2)=[{"l\1\2\1\10\0\0\0\3\0\0\0q\0\0\0\1\1o\0 \0\0\0/com/ubu"...,
136}, {"\0\0\0\0\1\0\0\0", 8}], msg_controllen=0, msg_flags=0},
MSG_NOSIGNAL) = 144
poll([{fd=3, events=POLLIN}], 1, -1

<pressed cTRL-C here>

After that I see that the rsyslogd process does *not* run anymore, so it
almost looks like this worked.

However, doing the opposite and starting rsyslog with "start rsyslog"
hangs too but does not start the daemon; using strace again it seems it
hangs at the same:

getuid() = 0
socket(PF_FILE, SOCK_STREAM|SOCK_CLOEXEC, 0) = -1 EINVAL (Invalid argument)
socket(PF_FILE, SOCK_STREAM, 0) = 3
fcntl(3, F_GETFD) = 0
fcntl(3, F_SETFD, FD_CLOEXEC) = 0
connect(3, {sa_family=AF_FILE, path=@"/com/ubuntu/upstart"}, 22) = 0
fcntl(3, F_GETFL) = 0x2 (flags O_RDWR)
fcntl(3, F_SETFL, O_RDWR|O_NONBLOCK) = 0
geteuid() = 0
getsockname(3, {sa_family=AF_FILE, NULL}, [2]) = 0
poll([{fd=3, events=POLLOUT}], 1, 0) = 1 ([{fd=3, revents=POLLOUT}])
write(3, "\0", 1) = 1
sendto(3, "AUTH EXTERNAL 30\r\n", 18, MSG_NOSIGNAL, NULL, 0) = 18
poll([{fd=3, events=POLLIN}], 1, -1) = 1 ([{fd=3, revents=POLLIN}])
read(3, "OK 474d8e68be037c13cc979ed100012"..., 2048) = 37
poll([{fd=3, events=POLLOUT}], 1, -1) = 1 ([{fd=3, revents=POLLOUT}])
sendto(3, "NEGOTIATE_UNIX_FD\r\n", 19, MSG_NOSIGNAL, NULL, 0) = 19
poll([{fd=3, events=POLLIN}], 1, -1) = 1 ([{fd=3, revents=POLLIN}])
read(3, "AGREE_UNIX_FD\r\n", 2048) = 15
poll([{fd=3, events=POLLOUT}], 1, -1) = 1 ([{fd=3, revents=POLLOUT}])
sendto(3, "BEGIN\r\n", 7, MSG_NOSIGNAL, NULL, 0) = 7
poll([{fd=3, events=POLLIN|POLLOUT}], 1, -1) = 1 ([{fd=3, revents=POLLOUT}])
sendmsg(3, {msg_name(0)=NULL,
msg_iov(2)=[{"l\1\2\1\f\0\0\0\1\0\0\0_\0\0\0\1\1o\0\23\0\0\0/com/ubu"..., 112},
{"\7\0\0\0rsyslog\0", 12}], msg_controllen=0, msg_flags=0},
MSG_NOSIGNAL) = 124
poll([{fd=3, events=POLLIN}], 1, 25000) = 1 ([{fd=3, revents=POLLIN}])
recvmsg(3, {msg_name(0)=NULL,
msg_iov(1)=[{"l\2\1\1%\0\0\0\1\0\0\0\17\0\0\0\5\1u\0\1\0\0\0\10\1g\0\1o\0\0"...,
2048}], msg_controllen=0, msg_flags=MSG_CMSG_CLOEXEC}, MSG_CMSG_CLOEXEC)
= 69
recvmsg(3, 0x7fffa22272b0, MSG_CMSG_CLOEXEC) = -1 EAGAIN (Resource
temporarily unavailable)
sendmsg(3, {msg_name(0)=NULL,
msg_iov(2)=[{"l\1\2\1\10\0\0\0\2\0\0\0q\0\0\0\1\1o\0 \0\0\0/com/ubu"...,
136}, {"\0\0\0\0\1\0\0\0", 8}], msg_controllen=0, msg_flags=0},
MSG_NOSIGNAL) = 144
poll([{fd=3, events=POLLIN}], 1, -1

<pressed cTRL-C here>

I'm seeing references to /com/ubuntu/... but can't find a matching file
system entry, it probably does not need to exist physically?

I've modified /usr/lib/util-vserver/vserver.functions to run /sbin/init
with --debug in the init clause and was able to get information in my
rsyslog logfile.

This is when stopping e.g. ssh:

init: Connection from private client
init: job_class_register: Registered job /com/ubuntu/Upstart/jobs/rc
init: job_class_register: Registered job /com/ubuntu/Upstart/jobs/rsyslog
init: job_register: Registered instance /com/ubuntu/Upstart/jobs/rsyslog/_
init: job_class_register: Registered job /com/ubuntu/Upstart/jobs/udev
init: job_register: Registered instance /com/ubuntu/Upstart/jobs/udev/_
init: job_class_register: Registered job /com/ubuntu/Upstart/jobs/ssh
init: job_register: Registered instance /com/ubuntu/Upstart/jobs/ssh/_
init: job_class_register: Registered job
/com/ubuntu/Upstart/jobs/control_2dalt_2ddelete
init: job_class_register: Registered job /com/ubuntu/Upstart/jobs/vserver
init: job_class_register: Registered job /com/ubuntu/Upstart/jobs/rcS
init: job_class_register: Registered job
/com/ubuntu/Upstart/jobs/mounted_2dvarrun
init: job_class_register: Registered job
/com/ubuntu/Upstart/jobs/rc_2dsysinit
init: job_class_register: Registered job
/com/ubuntu/Upstart/jobs/udevtrigger
init: job_class_register: Registered job
/com/ubuntu/Upstart/jobs/mounted_2ddev
init: job_class_register: Registered job
/com/ubuntu/Upstart/jobs/udev_2dfinish
init: job_class_register: Registered job
/com/ubuntu/Upstart/jobs/mounted_2dtmp
init: job_class_register: Registered job
/com/ubuntu/Upstart/jobs/udevmonitor
init: job_class_register: Registered job /com/ubuntu/Upstart/jobs/procps
init: ssh goal changed from start to stop
init: ssh state changed from running to pre-stop
init: ssh state changed from pre-stop to stopping
init: event_new: Pending stopping event
init: Handling stopping event
init: event_finished: Finished stopping event
init: ssh state changed from stopping to killed
init: Sending TERM signal to ssh main process (1861)
sshd[1861]: Received signal 15; terminating.
init: Sending KILL signal to ssh main process (1861)

On the console, it still hangs there and I need to CTRL-C it. As can be
seen from the log, ssh was shut down. Now, when starting:

init: Connection from private client
init: job_class_register: Registered job /com/ubuntu/Upstart/jobs/rc
init: job_class_register: Registered job /com/ubuntu/Upstart/jobs/rsyslog
init: job_register: Registered instance /com/ubuntu/Upstart/jobs/rsyslog/_
init: job_class_register: Registered job /com/ubuntu/Upstart/jobs/udev
init: job_register: Registered instance /com/ubuntu/Upstart/jobs/udev/_
init: job_class_register: Registered job /com/ubuntu/Upstart/jobs/ssh
init: job_register: Registered instance /com/ubuntu/Upstart/jobs/ssh/_
init: job_class_register: Registered job
/com/ubuntu/Upstart/jobs/control_2dalt_2ddelete
init: job_class_register: Registered job /com/ubuntu/Upstart/jobs/vserver
init: job_class_register: Registered job /com/ubuntu/Upstart/jobs/rcS
init: job_class_register: Registered job
/com/ubuntu/Upstart/jobs/mounted_2dvarrun
init: job_class_register: Registered job
/com/ubuntu/Upstart/jobs/rc_2dsysinit
init: job_class_register: Registered job
/com/ubuntu/Upstart/jobs/udevtrigger
init: job_class_register: Registered job
/com/ubuntu/Upstart/jobs/mounted_2ddev
init: job_class_register: Registered job
/com/ubuntu/Upstart/jobs/udev_2dfinish
init: job_class_register: Registered job
/com/ubuntu/Upstart/jobs/mounted_2dtmp
init: job_class_register: Registered job
/com/ubuntu/Upstart/jobs/udevmonitor
init: job_class_register: Registered job /com/ubuntu/Upstart/jobs/procps
init: ssh goal changed from stop to start

And that was it, CTRL-C required, no ssh daemon running.

I've already created a script to automate the various steps for easier
reproducing. These steps include:

* ensuring apps/init/style is set to plain (I tried also without it and
didn't spot any difference?)

* made /var/run and /var/lock a tmpfs via <vserver>/fstab

* Removed the following entries from /etc/sysctl.d/ (they prevent
"procps" from being run):

10-console-messages.conf 10-network-security.conf 10-zeropage.conf

* Removed the following entries from /etc/init/

console-setup.conf hwclock.conf mountall.conf mountall-shell.conf
network-interface-security.conf plymouth-splash.conf tty1.conf tty4.conf
upstart-udev-bridge.conf dmesg.conf hwclock-save.conf mountall-net.conf
networking.conf plymouth.conf plymouth-stop.conf tty2.conf tty5.conf
ureadahead.conf hostname.conf module-init-tools.conf
mountall-reboot.conf network-interface.conf plymouth-log.conf tty3.conf
tty6.conf ureadahead-other.conf

This is a mix what is listed in the wiki and from what I've deemed not
appropriate.

* Changed a line in /etc/init/rc-sysinit.conf

 From "start on filesystem and net-device-up IFACE=lo"
To "start on filesystem"

* Copied a /usr/lib/locale/locale-archive from an existing maverick
installation into place

* Used a custom /etc/apt/sources.list

* Added the following vserver.conf to /etc/init

# vserver - setup when running within a vserver
#
# Special handling when running inside a vserver, emulate various events.
# Note: The utmp stuff is needed if you mount /var/run as tmpfs and you
don't
# create utmp anywhere else

description "setup when running within a vserver"

start on startup
script
        touch /var/run/utmp
        chown root:utmp /var/run/utmp
        chmod 664 /var/run/utmp
        initctl emit virtual-filesystems
        initctl emit local-filesystems
        initctl emit remote-filesystems
        initctl emit filesystem
end script

* Created the /dev/console device (outside the guest; inside didn't work
[permission denied]):

mknod --mode=600 /dev/console c 5 0

And finally, the line I used to build the server:

vserver cm-dev01 build --initstyle plain -m debootstrap --hostname
cm-dev01 --netdev eth0 --interface v210=10.16.0.210/16 --context 210 --
-d maverick -m http://apt-proxy:9999/ubuntu/

While going through my personal archive I found an email from cryptronic
from 25.06.2010 22:04 mentioning (unfortunately the HTTP archives aren't
currently accessible, getting connection timeout):

> this is a upstart problem. The problem is coming from the expect fork
method. If i do not use this method and start the daemons ins
foreground i get a normal behaviour

I tried commenting out the "expect fork" syntax but it didn't change
anything. "start ssh" still hangs.

Any insights would be very helpful.

thanks,
- Markus

PS: Resources I consulted during this session were:

[1] http://linux-vserver.org/Upstart_issues
[2]
http://wiki.raceme.org/dokuwiki/tof:computer:vserver:upstart_ubuntu_lucid
[3] http://irc.13thfloor.at/LOG/2009-10/LOG_2009-10-16.txt
[4] http://upstart.ubuntu.com/wiki/Debugging
Received on Tue Oct 19 12:25:03 2010

[Next/Previous Months] [Main vserver Project Homepage] [Howto Subscribe/Unsubscribe] [Paul Sladen's vserver stuff]
Generated on Tue 19 Oct 2010 - 12:25:05 BST by hypermail 2.1.8