====== Installing Debian Lenny (5.0.1) with OpenVZ on IBM System x3650 ======

This is a complete step by step log of the installation of [[http://wiki.debian.org/DebianLenny|Debian Lenny GNU/Linux]] using [[http://www.openvz.org|OpenVZ]] virtualization technology on a factory sealed [[http://www.ibm.com/systems/x/hardware/rack/x3650/index.html|IBM xSeries 3650 2U rack-mountable server]].

===== Hardware =====

The server is a standard 7979 model with 16G of additional RAM and 4x500G SAS drives in RAID10 configuration. An [[http://en.wikipedia.org/wiki/Remote_Supervisor_Adapter|IBM RSAII SlimLine]] adapter is also installed.

===== Goals =====

  * Assembling the HW
  * Installing the base OS
  * Enabling all drivers and base SW features
  * Installing OpenVZ
  * Stress testing the system

===== Installing Debian Lenny =====

<note>Also see my [[lenny_on_x3650_screens|gallery of screenshots]] taken during the installation procedure!</note>

  - get and burn http://cdimage.debian.org/debian-cd/5.0.1/amd64/iso-cd/debian-501-amd64-netinst.iso
     * why amd64? see: http://www.scribd.com/doc/363677/Benchmarks-AMD64-in-32bit-mode-vs-64bit-mode-Ubuntu
     * also we can create x86 VEs anytime with OpenVZ
  - get http://ftp.hu.debian.org/debian/pool/non-free/f/firmware-nonfree/firmware-bnx2_0.14+lenny1_all.deb
     * extract bnx2-06-4.0.5.fw from the .deb (eg.: with [[http://kitenet.net/~joey/code/alien/|alien]]) and put it on the root dir of a pen-drive
     * why? - the closed source firmware for the latest broadcom NICs are excluded from the Debian (lenny) installer
     * why? - debian thinks this is the way of encouraging broadcom to release its firmware code to the public
     * the result? - annoying debian users. narrowing debian's market penetration. including the firmware in non-free repositories anyway. :-/
  - Do not forget to plug the pen-drive into the x3650!
  - See and follow the [[lenny_on_x3650_screens|screenshots]] of Debian Lenny 5.0.1 Installation procedure
  - Installing general packages needed:<code>apt-get install sudo mc gcc make libncurses5-dev locales bzip2 chrony kernel-package patch ipmitool acpi ethtool</code>
  - The installation procedure messed up the locales so I had to do:<code>dpkg-reconfigure locales</code>
    * Selected locales: <code>
  en_US.ISO-8859-1... done
  en_US.ISO-8859-15... done
  en_US.UTF-8... done
  hu_HU.ISO-8859-2... done
  hu_HU.UTF-8... done
  en_US.UTF-8... done</code>
    * set LC_TIME -> hu_HU.utf8
    * set LANG -> en_US.UTF-8

==== Installing and configuring IPMI base management software ====

  - Configure kernel modules to be loaded at next reboot <code>echo -e "\nipmi_msghandler\nipmi_watchdog\nipmi_si\nipmi_devintf\ncoretemp" >> /etc/modules</code>
  - Enable the IPMI event daemon at next reboot <code>echo "ENABLED=true" > /etc/default/ipmievd</code>

<note warning>Do not forget to set up email notifications. The IPMI event daemon is useless unless you get notified of the events!</note>


==== Setting up the Linux firewall (iptables) ====

  - Init the ''nat'' ip table <code>iptables -L -t nat</code>
  - Save an empty ip table as base template <code>iptables-save > /etc/network/iptables_eth0.stop</code>
  - create /etc/network/iptables_eth0.start as: <code>
*filter
:INPUT DROP [0:0]
:FORWARD ACCEPT [0:0]
:OUTPUT ACCEPT [0:0]
# allow local and locally originated traffic
[0:0] -A INPUT -i lo -j ACCEPT
[0:0] -A INPUT -s 10.20.20.0/24 -j ACCEPT
[0:0] -A INPUT -m state --state RELATED,ESTABLISHED -j ACCEPT
# allow public services
[0:0] -A INPUT -p tcp -m tcp --dport 22 -j ACCEPT
[0:0] -A INPUT -p tcp -m tcp --dport 80 -j ACCEPT
COMMIT
*nat
:PREROUTING ACCEPT [0:0]
:POSTROUTING ACCEPT [0:0]
:OUTPUT ACCEPT [0:0]
# SNAT VEs
-A POSTROUTING -o eth0 -s 10.20.20.0/24 -j MASQUERADE
COMMIT
*mangle
:PREROUTING ACCEPT [0:0]
:INPUT ACCEPT [0:0]
:FORWARD ACCEPT [0:0]
:OUTPUT ACCEPT [0:0]
:POSTROUTING ACCEPT [0:0]
COMMIT
</code>
  - create /etc/network/if-up.d/iptables as <code bash>
#! /bin/sh
# load iptables script if exists
set -e
conffile=/etc/network/iptables_$IFACE.$MODE
if [ -f $conffile ]; then
  iptables-restore < $conffile
fi
</code>
  - Then make it executable <code>chmod +x /etc/network/if-up.d/iptables</code>
  - Enable IP forwarding for the primary interface <code>sysctl -w net.ipv4.conf.eth0.forwarding=1</code>
  - reboot as a test or activate configuration with <code>iptables-restore < /etc/network/iptables_eth0.start</code>

==== Setting up Munin to monitor the system locally ====

  - Install Munin and thttpd <code>
apt-get install munin munin-node thttpd
</code>
  - Fetch and configure the IPMI sensor plugin<code>
wget http://muninexchange.projects.linpro.no/download.php?phid=579 -O /usr/share/munin/plugins/ipmitool_sensor_
chmod +x /usr/share/munin/plugins/ipmitool_sensor_
ln -s /usr/share/munin/plugins/ipmitool_sensor_ /etc/munin/plugins/ipmitool_sensor_fan
ln -s /usr/share/munin/plugins/ipmitool_sensor_ /etc/munin/plugins/ipmitool_sensor_temp
ln -s /usr/share/munin/plugins/ipmitool_sensor_ /etc/munin/plugins/ipmitool_sensor_volt
echo -e "\n\n[ipmitool_sensor*]\nuser root\ntimeout 20" >> /etc/munin/plugin-conf.d/munin-node
</code>
  - Restart the Munin node <code>
/etc/init.d/munin-node restart
</code>

==== Install IBM ServeRAID Management commandline interface ====

<note warning>WARNING: DIRTY HACK follows</note>

Unfortunately, raid management software provided by IBM is rather old, even if my CD is dated 2008-12-31, the Linux software is made somewhere back in 2003-2007. All insides and outs of GCC have changed since then. The command line utilities ''arcconf'' and ''hrconf'' both require ''libstdc++.so.5'', which is unavailable for Debian Lenny.

To make things worse ''libstdc++.so.5'' depends on ''gcc-3.3'', but the oldest version of gcc in Lenny is 3.4((do you remember the [[http://gcc.gnu.org/ml/gcc/2003-01/msg01688.html|old flames]] between GCC 3.3 and 3.4?)). I had no option but to do a quick google search for ''libstdc++.so.5'' deb packages and obtain one [[https://launchpad.net/ubuntu/warty/amd64/lib32stdc++5/1:3.3.4-9ubuntu5.1|vintage specimen from an early ubuntu warty archive]] (thank you guys for keeping that online! ;-) ).

Following this all I had to do was to copy the file into ''/usr/lib32'' (this would be ''/usr/lib'' on x86 systems) and upload the shipped utilities to ''/root/serveraid''. The initial warnings can be safely ignored. You can use {{:publications:ibm_serveraid_cmdline_manager_linux_amd64.zip|x64 files and the missing library from my archive}} if you dare.

===== Setting up OpenVZ =====

  - Getting OpenVZ <code>
cd /usr/src
wget http://download.openvz.org/kernel/branches/2.6.27/2.6.27-briullov.1/patches/patch-briullov.1-combined.gz
wget http://download.openvz.org/kernel/branches/2.6.27/2.6.27-briullov.1/configs/kernel-2.6.27-x86_64.config.ovz</code>
    * Why not using a 'stable' OpenVZ release? Because:
      * the latest 'stable' (023stab048.6) is for linux-2.6.9
      * linux-2.6.9 is missing several critical drivers (eg.: the broadcom network driver)
      * linux-2.6.9 is missing several critical security fixes thus can be apostrophized as "insecure" and "obsolete"
      * ovz 023stab048.6 itself is different in many aspects (eg.: less separation in the VE level, see: http://git.openvz.org/?p=linux-2.6.27-openvz;a=log;h=v2.6.27-briullov)
    * Why not using the official Debian Lenny kernel (as advised on http://wiki.openvz.org/Installation_on_Debian)? Because:
      * Did anyone //ever// //seriously// try and test this damn excuse for a production quality kernel?
  - Getting Linux <code>
wget http://www.kernel.org/pub/linux/kernel/v2.6/linux-2.6.27.tar.bz2
tar xjf linux-2.6.27.tar.bz2
</code>
  - Patching Linux with OpenVZ<code>
cd /usr/src/linux-2.6.27
zcat ../patch-briullov.1-combined.gz | patch -p1
cp ../kernel-2.6.27-x86_64.config.ovz .config
</code>
  - Compiling, packaging and installing Linux <code>
make-kpkg --append-to-version -winnetou-ovz-briullov.1 --revision 1 --initrd buildpackage
dpkg -i ../linux-{image,headers}-2.6.27.21-winnetou-ovz-briullov.1_1_amd64.deb
</code>
  - Cleaning up <code>
reboot
apt-get remove linux-image-2.6-amd64 linux-image-2.6.26-2-amd64
</code>
  - Configuring the OpenVZ Lenny software repository <code>
echo -e "\ndeb http://download.openvz.org/debian-systs lenny openvz" >> /etc/apt/sources.list''
wget -q http://download.openvz.org/debian-systs/dso_archiv_signing_key.asc -O- | apt-key add - && apt-get update</code>
  - Installing OpenVZ management software and a minimal Lenny OS template <code>apt-get install vzctl vzquota vzctl-ostmpl-debian-5.0-amd64-minimal</code>
  - Create several split resource configuration templates based on the resources available in the system, with 1G of swap space. Easily used afterwards with ''vzctl create''. <code>for x in 2 4 8 16 32; do vzsplit -f $x -n $x -s 1048576; done</code>

==== Testing OpenVZ ====

  - Create and configure test VE <code>
vzctl create 100 --ostemplate debian-5.0-amd64-minimal --config 4 --ipadd 10.20.20.100 --hostname test
vzctl set 100 --save --nameserver `cat /etc/resolv.conf | grep nameserver | cut -f2 -d' '`</code>
  - Start and enter test VE <code>
vzctl start 100
vzctl enter 100
</code>
  - Update the VE<code>
apt-get update
apt-get upgrade
</code>
  - Do some stress testing<code>
apt-get install stress
stress --cpu 8 --io 4 --vm 2 --vm-bytes 128M --hdd 8
</code>
  - Do not forget to exit from VE 100 before continuing!

{{tag>howto install linux openvz ibm x3650}}

~~LINKBACK~~