Installing Debian Lenny (5.0.1) with OpenVZ on IBM System x3650

This is a complete step by step log of the installation of Debian Lenny GNU/Linux using OpenVZ virtualization technology on a factory sealed IBM xSeries 3650 2U rack-mountable server.

The server is a standard 7979 model with 16G of additional RAM and 4x500G SAS drives in RAID10 configuration. An IBM RSAII SlimLine adapter is also installed.

• Assembling the HW
• Installing the base OS
• Enabling all drivers and base SW features
• Installing OpenVZ
• Stress testing the system

1. get and burn http://cdimage.debian.org/debian-cd/5.0.1/amd64/iso-cd/debian-501-amd64-netinst.iso
   • why amd64? see: http://www.scribd.com/doc/363677/Benchmarks-AMD64-in-32bit-mode-vs-64bit-mode-Ubuntu
   • also we can create x86 VEs anytime with OpenVZ
2. get http://ftp.hu.debian.org/debian/pool/non-free/f/firmware-nonfree/firmware-bnx2_0.14+lenny1_all.deb
   • extract bnx2-06-4.0.5.fw from the .deb (eg.: with alien) and put it on the root dir of a pen-drive
   • why? - the closed source firmware for the latest broadcom NICs are excluded from the Debian (lenny) installer
   • why? - debian thinks this is the way of encouraging broadcom to release its firmware code to the public
   • the result? - annoying debian users. narrowing debian's market penetration. including the firmware in non-free repositories anyway.
3. Do not forget to plug the pen-drive into the x3650!
4. See and follow the screenshots of Debian Lenny 5.0.1 Installation procedure
5. Installing general packages needed:

   apt-get install sudo mc gcc make libncurses5-dev locales bzip2 chrony kernel-package patch ipmitool acpi ethtool

6. The installation procedure messed up the locales so I had to do:

   dpkg-reconfigure locales

   • Selected locales:

       en_US.ISO-8859-1... done
       en_US.ISO-8859-15... done
       en_US.UTF-8... done
       hu_HU.ISO-8859-2... done
       hu_HU.UTF-8... done
       en_US.UTF-8... done

   • set LC_TIME → hu_HU.utf8
   • set LANG → en_US.UTF-8

1. Configure kernel modules to be loaded at next reboot

   echo -e "\nipmi_msghandler\nipmi_watchdog\nipmi_si\nipmi_devintf\ncoretemp" >> /etc/modules

2. Enable the IPMI event daemon at next reboot

   echo "ENABLED=true" > /etc/default/ipmievd

1. Init the nat ip table

   iptables -L -t nat

2. Save an empty ip table as base template

   iptables-save > /etc/network/iptables_eth0.stop

3. create /etc/network/iptables_eth0.start as:

   *filter
   :INPUT DROP [0:0]
   :FORWARD ACCEPT [0:0]
   :OUTPUT ACCEPT [0:0]
   # allow local and locally originated traffic
   [0:0] -A INPUT -i lo -j ACCEPT
   [0:0] -A INPUT -s 10.20.20.0/24 -j ACCEPT
   [0:0] -A INPUT -m state --state RELATED,ESTABLISHED -j ACCEPT
   # allow public services
   [0:0] -A INPUT -p tcp -m tcp --dport 22 -j ACCEPT
   [0:0] -A INPUT -p tcp -m tcp --dport 80 -j ACCEPT
   COMMIT
   *nat
   :PREROUTING ACCEPT [0:0]
   :POSTROUTING ACCEPT [0:0]
   :OUTPUT ACCEPT [0:0]
   # SNAT VEs
   -A POSTROUTING -o eth0 -s 10.20.20.0/24 -j MASQUERADE
   COMMIT
   *mangle
   :PREROUTING ACCEPT [0:0]
   :INPUT ACCEPT [0:0]
   :FORWARD ACCEPT [0:0]
   :OUTPUT ACCEPT [0:0]
   :POSTROUTING ACCEPT [0:0]
   COMMIT

4. create /etc/network/if-up.d/iptables as

   #! /bin/sh
   # load iptables script if exists
   set -e
   conffile=/etc/network/iptables_$IFACE.$MODE
   if [ -f $conffile ]; then
     iptables-restore < $conffile
   fi

5. Then make it executable

   chmod +x /etc/network/if-up.d/iptables

6. Enable IP forwarding for the primary interface

   sysctl -w net.ipv4.conf.eth0.forwarding=1

7. reboot as a test or activate configuration with

   iptables-restore < /etc/network/iptables_eth0.start

1. Install Munin and thttpd

   apt-get install munin munin-node thttpd

2. Fetch and configure the IPMI sensor plugin

   wget http://muninexchange.projects.linpro.no/download.php?phid=579 -O /usr/share/munin/plugins/ipmitool_sensor_
   chmod +x /usr/share/munin/plugins/ipmitool_sensor_
   ln -s /usr/share/munin/plugins/ipmitool_sensor_ /etc/munin/plugins/ipmitool_sensor_fan
   ln -s /usr/share/munin/plugins/ipmitool_sensor_ /etc/munin/plugins/ipmitool_sensor_temp
   ln -s /usr/share/munin/plugins/ipmitool_sensor_ /etc/munin/plugins/ipmitool_sensor_volt
   echo -e "\n\n[ipmitool_sensor*]\nuser root\ntimeout 20" >> /etc/munin/plugin-conf.d/munin-node

3. Restart the Munin node

   /etc/init.d/munin-node restart

Unfortunately, raid management software provided by IBM is rather old, even if my CD is dated 2008-12-31, the Linux software is made somewhere back in 2003-2007. All insides and outs of GCC have changed since then. The command line utilities arcconf and hrconf both require libstdc++.so.5, which is unavailable for Debian Lenny.

To make things worse libstdc++.so.5 depends on gcc-3.3, but the oldest version of gcc in Lenny is 3.4¹⁾. I had no option but to do a quick google search for libstdc++.so.5 deb packages and obtain one vintage specimen from an early ubuntu warty archive (thank you guys for keeping that online! ).

Following this all I had to do was to copy the file into /usr/lib32 (this would be /usr/lib on x86 systems) and upload the shipped utilities to /root/serveraid. The initial warnings can be safely ignored. You can use x64 files and the missing library from my archive if you dare.

1. Getting OpenVZ

   cd /usr/src
   wget http://download.openvz.org/kernel/branches/2.6.27/2.6.27-briullov.1/patches/patch-briullov.1-combined.gz
   wget http://download.openvz.org/kernel/branches/2.6.27/2.6.27-briullov.1/configs/kernel-2.6.27-x86_64.config.ovz

   • Why not using a 'stable' OpenVZ release? Because:
     • the latest 'stable' (023stab048.6) is for linux-2.6.9
     • linux-2.6.9 is missing several critical drivers (eg.: the broadcom network driver)
     • linux-2.6.9 is missing several critical security fixes thus can be apostrophized as „insecure” and „obsolete”
     • ovz 023stab048.6 itself is different in many aspects (eg.: less separation in the VE level, see: http://git.openvz.org/?p=linux-2.6.27-openvz;a=log;h=v2.6.27-briullov)
   • Why not using the official Debian Lenny kernel (as advised on http://wiki.openvz.org/Installation_on_Debian)? Because:
     • Did anyone ever seriously try and test this damn excuse for a production quality kernel?
2. Getting Linux

   wget http://www.kernel.org/pub/linux/kernel/v2.6/linux-2.6.27.tar.bz2
   tar xjf linux-2.6.27.tar.bz2

3. Patching Linux with OpenVZ

   cd /usr/src/linux-2.6.27
   zcat ../patch-briullov.1-combined.gz | patch -p1
   cp ../kernel-2.6.27-x86_64.config.ovz .config

4. Compiling, packaging and installing Linux

   make-kpkg --append-to-version -winnetou-ovz-briullov.1 --revision 1 --initrd buildpackage
   dpkg -i ../linux-{image,headers}-2.6.27.21-winnetou-ovz-briullov.1_1_amd64.deb

5. Cleaning up

   reboot
   apt-get remove linux-image-2.6-amd64 linux-image-2.6.26-2-amd64

6. Configuring the OpenVZ Lenny software repository

   echo -e "\ndeb http://download.openvz.org/debian-systs lenny openvz" >> /etc/apt/sources.list''
   wget -q http://download.openvz.org/debian-systs/dso_archiv_signing_key.asc -O- | apt-key add - && apt-get update

7. Installing OpenVZ management software and a minimal Lenny OS template

   apt-get install vzctl vzquota vzctl-ostmpl-debian-5.0-amd64-minimal

8. Create several split resource configuration templates based on the resources available in the system, with 1G of swap space. Easily used afterwards with vzctl create.

   for x in 2 4 8 16 32; do vzsplit -f $x -n $x -s 1048576; done

1. Create and configure test VE

   vzctl create 100 --ostemplate debian-5.0-amd64-minimal --config 4 --ipadd 10.20.20.100 --hostname test
   vzctl set 100 --save --nameserver `cat /etc/resolv.conf | grep nameserver | cut -f2 -d' '`

2. Start and enter test VE

   vzctl start 100
   vzctl enter 100

3. Update the VE

   apt-get update
   apt-get upgrade

4. Do some stress testing

   apt-get install stress
   stress --cpu 8 --io 4 --vm 2 --vm-bytes 128M --hdd 8

5. Do not forget to exit from VE 100 before continuing!