LXC with Gentoo Linux

Share

VMWare (Server, Player, ESX), VirtualBox, KVM/QEMU  are just a few of the virtualization technologies you can use if you need to run a generic virtual machine. Each has strong and weak points, each adds some overhead when running the virtual machine, each can prove useful in a given scenario. I’ve used more or less extensively all of them (testing or in daily work). I’ve tested also linux specific virtualizations like XEN or OpenVZ but their drawbacks made me somewhat reluctant in using them. But recently I’ve come over Linux Containers (LXC), a lightweight virtualization that lets you isolate processes and resources without using a full virtual machine.

Please note that LXC does not provide you a full virtual machine and does not let you run another OS than Linux. It is similar with XEN or OpenVZ but offers greater flexibility being integrated in the kernel (don’t require additional patches or changes to the system). If you wish to know more about how it works and why it’s better than a full virtual machine please see here, I’ll continue with a short how-to for configuring and using it in Gentoo Linux.

Host configuration

First of all you’ll need a recent kernel as LXC is integrated starting from version 2.6.29. I’m using 2.6.34 and the configuration options are relative to it. Please make sure the following configuration parameters are set:

CONFIG_GROUP_SCHED=y
CONFIG_FAIR_GROUP_SCHED=y
CONFIG_RT_GROUP_SCHED=y
CONFIG_CGROUP_SCHED=y
CONFIG_CGROUPS=y
CONFIG_CGROUP_NS=y
CONFIG_CGROUP_FREEZER=y
CONFIG_CGROUP_DEVICE=y
CONFIG_CPUSETS=y
CONFIG_PROC_PID_CPUSET=y
CONFIG_CGROUP_CPUACCT=y
CONFIG_RESOURCE_COUNTERS=y
CONFIG_CGROUP_MEM_RES_CTLR=y
CONFIG_CGROUP_MEM_RES_CTLR_SWAP=y
CONFIG_MM_OWNER=y
CONFIG_NAMESPACES=y
CONFIG_UTS_NS=y
CONFIG_IPC_NS=y
CONFIG_USER_NS=y
CONFIG_PID_NS=y
CONFIG_NET_NS=y
CONFIG_NET_CLS_CGROUP=y
CONFIG_SECURITY_FILE_CAPABILITIES=y
CONFIG_DEVPTS_MULTIPLE_INSTANCES=y

If you have made changes to your kernel configuration you’ll need to rebuild, install and use the new kernel. For Gentoo you can use the information in the kernel upgrade documentation about how to do it right.

Next you will have to mount the control group filesystem. For this execute:

# mkdir -p /cgroup
# mount none -t cgroup /cgroup

To persist the change between reboots add to /etc/fstab:
none /cgroup cgroup defaults 0 0
Install the LXC userspace tools:
# emerge -v lxc
Configure a bridge to be used by the container. In my case I created a separate virtual network which is given access to the real network on a as-needed bases. This way I can communicate with the containers (and the containers between them) but I can control what container has access to the real network and when. In /etc/conf.d/net add:

bridge_br0="dummy0"
config_dummy0="null"
config_br0="10.1.1.1/32 brd 10.1.1.255"
routes_br0="10.1.1.0/24 via 10.1.1.1"

and create the /etc/init.d/net.br0 symlink to /etc/init.d/net.lo. To allow a container to communicate with the world just activate the NAT into iptables like

# iptables -t nat -A POSTROUTING -o eth0 -j MASQUERADE

where eth0 is the real network interface.

Container installation

Start installing the container by creating the root folder

# mkdir -p /mnt/storage/containers/gentoo_04/roofs

The rest of the steps are the same as for a new Gentoo installation. Please refer to the Gentoo Handbook for them, I’ll give here only the most important ones and what is made different from the standard. First download a stage3 file and a portage snapshot (chapter 5.b, 5.c from the handbook) into rootfs

# cd /mnt/storage/conainers/gentoo_04/rootfs

Donwload from a mirror and unpack the 2 files

# tar xvjpf stage3-*.tar.bz2
# tar xvjf portage-latest.tar.bz2 -C /mnt/storage/containers/gentoo_04/rootfs/usr

Configure compile flags, select mirrors and enter into the new environment

# mirrorselect -i -o >> /mnt/storage/containers/gentoo_04/rootfs/etc/make.conf
# mirrorselect -i -r -o >> /mnt/storage/containers/gentoo_04/rootfs/etc/make.conf
# cp -L /etc/resolv.conf /mnt/storage/containers/gentoo_04/rootfs/etc/
# mount -t proc none /mnt/storage/containers/gentoo_04/rootfs/proc
# mount -o bind /dev /mnt/storage/containers/gentoo_04/rootfs/dev
# chroot /mnt/storage/containers/gentoo_04/rootfs /bin/bash
# env-update
# source /etc/profile
# export PS1="(chroot) $PS1"
# emerge --sync

Now the container is nearly ready (we don’t need to configure and install the kernel as we’ll use the host one. What we need is to switch to OpenRC  by following the migration guide.

# mkdir /etc/portage
# echo "sys-apps/openrc ~x86" >> /etc/portage/package.keywords
# echo "sys-apps/baselayout ~x86" >> /etc/portage/package.keywords
# emerge -v openrc mc
# dispatch-conf

Don’t forget to change the root password otherways you won’t be able to login int your container and will need to reenter chroot and change it (or hack the shadow file :) )!

# passwd

You can exit the chroot at this step.

Container setup

Create the container definition file in /mnt/storage/containers/gentoo_04/lxc-gentoo.conf

lxc.utsname = gentoo_04
lxc.network.type = veth
lxc.network.flags = up
lxc.network.link = br0
lxc.network.name = eth0
lxc.mount =  /mnt/storage/containers/gentoo_04/fstab
lxc.rootfs = /mnt/storage/containers/gentoo_04/rootfs
lxc.tty=12
lxc.pts=128

Create the fstab file /mnt/storage/containers/gentoo_04/fstab

none /mnt/storage/containers/gentoo_04/rootfs/dev/pts devpts defaults 0 0
none /mnt/storage/containers/gentoo_04/rootfs/proc    proc   defaults 0 0
none /mnt/storage/containers/gentoo_04/rootfs/sys     sysfs  defaults 0 0
none /mnt/storage/containers/gentoo_04/rootfs/dev/shm tmpfs  defaults 0 0

No changes are required for the container devices in /dev.

Update the /etc/inittab file by commenting the lines from the TERMINALS section c1 to c6

and adding the following line

Next update the container network configuration in /mnt/storage/containers/gentoo_04/rootfs/etc/conf.d/net

config_eth0="10.1.1.4 netmask 255.255.255.0 brd 10.1.1.255"
routes_eth0="default via 10.1.1.1"

Update the runlevels to remove unneeded/unsupported services

  • remove consolefont, hwclock, keymaps from /etc/runlevels/boot
  • remove udev-postmount from /etc/runlevels/default
  • remove udev from /etc/runlevels/sysinit

Now you are ready to create the container

# lxc-create -f /mnt/storage/containers/gentoo_04/lxc-gentoo.conf -n gentoo_04

If you need to make a change to the configuration files the fastest way is to destroy the container, make the changes and recreate it

# lxc-destroy -n gentoo_04

Running the container

In order to start the container you have to use

# lxc-start -n gentoo_04

At this step, if everything is succesfull you will be greeted with a login prompt. Enter root and your password (did you forget to set it? then enter a chroot environment and reset it) and you are in. Now you should enable networking and add it at default runlevel

# /etc/init.d/net.eth0 start
# rc-update add net.eth0 default

You should finish configuring the new system as per chapter 9 in handbook (install system tools, add user).

In order to shutdown you can do it in the usual way: issue a poweroff at the prompt. The system will shutdown but will not exit. To finish for good you’ll need

# lxc-stop -n gentoo_04

in another prompt (and you should run reset in the console used by the container). You can issue the stop command even if the container is still running but in this case you’ll lose any usaved information.

Now, that you have the container ready you can proceed to install any software you need. I’m using it for running apache, php, mysql, postgresql, oracle database and various other servers used in a development environment. When I don’t need a particular container I just destroy it and delete the rootfs folder. In this article I’ve show how to install a Gentoo container on a Gentoo host but you can run other linuxes in the container. You can find the instructions for other platforms in the references section below.

References

LXC HOWTO used as main reference for installing the container
LXC in Gentoo a collection of articles for using lxc userland tool in Gentoo
Linux Containers
Network configuration for LXC
IBM LXC

Finally, there’s another very important peculiarity of what does Cialis that brings it so high above its alternatives. It is the only med that is available in two versions – one intended for use on as-needed basis and one intended for daily use. As you might know, Viagra and Levitra only come in the latter of these two forms and should be consumed shortly before expected sexual activity to ensure best effect. Daily Cialis, in its turn, contains low doses of Tadalafil, which allows to build its concentration up in your system gradually over time and maintain it on acceptable levels, which, consequently, makes it possible for you to enjoy sex at any moment without having to time it.

8 Comments
  • Great site. A lot of useful information here. I’m sending it to some friends!

    June 14, 2010 at 4:14 am
  • Redouane Lakrache says:

    Thank you, really great article.
    I was able to install a binhost server, minimal (GCCless) gentoo containers (~300M) using lxc-0.7.0
    I had few problems catching ttys but setting up the following config seems to work well:
    ————
    lxc.utsname = probe
    lxc.network.type = veth
    lxc.network.flags = up
    lxc.network.link = br0
    lxc.network.name = eth0
    lxc.mount = /lxc/probe/fstab
    lxc.rootfs = /lxc/probe/rootfs/
    lxc.tty=12
    lxc.pts=128
    lxc.cgroup.devices.deny = a
    lxc.cgroup.devices.allow = c 1:3 rwm
    lxc.cgroup.devices.allow = c 1:5 rwm
    lxc.cgroup.devices.allow = c 2:5 rwm
    lxc.cgroup.devices.allow = c 5:0 rwm
    lxc.cgroup.devices.allow = c 4:1 rwm
    lxc.cgroup.devices.allow = c 4:2 rwm
    lxc.cgroup.devices.allow = c 4:3 rwm
    lxc.cgroup.devices.allow = c 4:4 rwm
    lxc.cgroup.devices.allow = c 4:5 rwm
    lxc.cgroup.devices.allow = c 4:6 rwm
    lxc.cgroup.devices.allow = c 4:7 rwm
    lxc.cgroup.devices.allow = c 4:8 rwm
    lxc.cgroup.devices.allow = c 4:9 rwm
    lxc.cgroup.devices.allow = c 4:10 rwm
    lxc.cgroup.devices.allow = c 4:11 rwm
    lxc.cgroup.devices.allow = c 4:12 rwm
    lxc.cgroup.devices.allow = c 136:* rwm
    lxc.cgroup.devices.allow = c 5:2 rwm
    lxc.cgroup.devices.allow = c 1:9 rwm
    lxc.cgroup.devices.allow = c 1:8 rwm
    ————
    My only problem is that i cannot make the containers (binhostgccless) communicate with each other.
    I would appreciate if you could post more details about your virtual network setup

    Thank you

    June 26, 2010 at 5:39 pm
  • The configuration looks good. The bridge is working? In my setup the bridge is unconnected to the real network as I don’t need it. If the bridge is working – ping from the host to the bridge’s host IP first (10.1.1.1 in above example) and to the container IP (10.1.1.4). If you can’t ping the host IP’s then the bridge is not setup or is not started. If you can’t ping the container’s IP then either the container has a firewall or (more likely) the network in container is not configure. Please check the container network configuration by using ifconfig in the container console. I’ve used a static IP for each container and have no issues. As a last resort remove the lxc.cgroup.devices.deny = a and see if it is working. I’m not using them, yet, as the containers are trusted (just development work, no security work yet).

    June 28, 2010 at 4:50 pm
  • Redouane Lakrache says:

    Many thanks Marius for your answer. Finally it’s a firewall issue (iptables -I FORWARD -i ${LAN} -d 192.168.0.0/255.255.0.0 -j DROP) which was up and forgotten long time ago :)

    I’m currently building and testing some minimal rootfs templates (mysql and nginx-php-fpm servers with ~60 packages ~280Mo each) and find that (AFAIK) gentoo has some lack of tools for automating containers creation like lxc-debian, lxc-busybox…

    I’ll try to make something like that. Also being able to build containers from templatefiles (containig lists of packages and specific configs) may ease the process.

    June 29, 2010 at 4:53 am
  • Redouane Lakrache says:

    Many thanks Marius for your answer. Finally it\’s a firewall issue (iptables -I FORWARD -i ${LAN} -d 192.168.0.0/255.255.0.0 -j DROP) which was up and forgotten long time ago :)

    I\’m currently building and testing some minimal rootfs templates (mysql and nginx-php-fpm servers with ~60 packages ~280Mo each) and find that (AFAIK) gentoo has some lack of tools for automating containers creation like lxc-debian, lxc-busybox…

    I\’ll try to make something like that. Also being able to build containers from templatefiles (containig lists of packages and specific configs) may ease the process.

    June 29, 2010 at 4:53 am
  • afaik gentoo has no tools for automating container creation at all. What I’m doing is to keep one minimal container up to date and when I have to deploy a new one I just create rootfs, copy it’s content there, update hostname and networking and create the container. It’s fast but space demanding. As I’m using it only for development this is not an issue, yet.

    June 29, 2010 at 7:52 am
  • Redouane Lakrache says:

    True, it’s pretty easy to create unique containers that are not made for replication.
    I also think that gentoo have good general tools for automating installs (like catalyst and metro)… it only misses some “glue” scripts that makes the final job with lxc possible. Maybe it wold be good to contribute with somehing like that…

    June 29, 2010 at 9:09 pm
  • Contact Form says:

    hey man I like it. I’m about to write more posts and try out your ideas. After all, we always have something new to learn and being humble, I came here to learn. Bookmarked.

    – Andre

    August 23, 2010 at 8:19 pm

Comments are closed.

By continuing to use the site, you agree to the use of cookies. More information

The cookie settings on this website are set to "allow cookies" to give you the best browsing experience possible. If you continue to use this website without changing your cookie settings or you click "Accept" below then you are consenting to this.

Close