A customer wanted some performance figures for a ESX cluster we built. We used our own atop and the trusted workhorse of plotting Gnuplot to make some performance plots.

Creating the data

As said, we used atop for this, it has a nice data gathering mode which prints out lines like these:

13:40:07  cpu   %usr  %nice   %sys  %irq %softirq   %steal  %wait  %idle
13:40:08  all      0      0      0     0        1        0      4     95
13:40:09  all      0      0      1     0        0        0      0	  99

We were interested in the values of %usr, %sys, %irq + %softirq, %wait and %idle.

So you think you know git… today I found out something “funny”. In a git repository:

$ git log
$ fatal: object 1fcc8de9361c56e538ff35d8cc4b07a9c95b7bf3 is corrupted

Okay, WTF? Lets look in the .git directory:

$ cd .git/objects/1f
$ ls -l
total 4
-r--r--r-- 1 root root 1057 2009-06-23 19:03 03db070bcb47bff3f8106f2ec7028b3496aaa8
-r--r--r-- 1 root root    0 2009-08-03 21:41 cc8de9361c56e538ff35d8cc4b07a9c95b7bf3

Ah, 0 bytes, that explains it (probably due to the weird reboot of last night and ext4).

Finally made the switch to jackd which works so much better than Pulseaudio. Ubuntu did not make this easy, but with some perseverance it works. One of the things I really hate about Pulseaudio is that when I click on a new song in audacious it would take about 1 second before the audio stabilized. Also with flash audio would stutter for the first few seconds.

First (if you don’t care about gnome-desktop):

Yes, what a nice post on ssh escape keys.

So you need to press <enter> and then ~ for the escape character to work!

Do I want this?

% ls -ld Joe_Cocker_-_The_Definitive_Collection 
drwxr-xr-x 2 miekg admin 4.0K Jul 20 22:20 Joe_Cocker_-_The_Definitive_Collection/
% cd *joe*
cd: no such file or directory: *joe*
% unsetop case_glob 
% cd *joe*
% pwd
/shared/vol/music/J/Joe_Cocker_-_The_Definitive_Collection

Currently we are building a fairly rock solid high availability cluster for a client. This has the “usual” ingredients: two locations, two NetApps, two clusters of three vmware ESX servers and a bunch of virtual machines running on top of the ESX servers. Also included in the mix is a VDI (now called View) virtual desktop infrastructure for running virtual windows XP clients.

This is all managed by SRM (site recovery manager) and it is almost working. But that is another story.

I already knew (Open)Solaris sucks, but now Ton has also figured it out.

ZFS definitely does not suck. Why not petition Oracle to GPL(v2) ZFS? Especially now Oracle wants to kill OpenSolaris?

So, there I am browsing my logs.

BAM

[650521.310292] INFO: task mysqld:2730 blocked for more than 120 seconds.
[650521.310320] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[650521.310364] mysqld        D 00000000     0  2730   2674
[650521.310367]  f6183ce0 00000086 c06ef6cc 00000000 f61c8c70 7834922a 00000dc2 00020050
[650521.310373]  00000296 c0796fe0 c0796fe0 f61c8f04 c2c35fe0 00000003 9500b16e 00024f83
[650521.310379]  00000002 00000002 f66bb500 f6d02e40 c2bf2090 f6e2e680 f6183ce8 c050eb68
[650521.310384] Call Trace:
[650521.310388]  [<c050eb68>] schedule+0x8/0x20
[650521.310391]  [<c0266701>] do_get_write_access+0x2a1/0x3d0
[650521.310395]  [<c0127d93>] ? kmap_atomic_prot+0x43/0xe0
[650521.310398]  [<c0150030>] ? wake_bit_function+0x0/0x60
[650521.310402]  [<c02669a3>] jbd2_journal_get_write_access+0x23/0x40
[650521.310405]  [<c0252a36>] __ext4_journal_get_write_access+0x26/0x60
[650521.310409]  [<c023e9d4>] ext4_reserve_inode_write+0x34/0x70
[650521.310413]  [<c023ea45>] ext4_mark_inode_dirty+0x35/0x140
[650521.310416]  [<c02669ac>] ? jbd2_journal_get_write_access+0x2c/0x40
[650521.310420]  [<c02446cf>] add_dirent_to_buf+0x12f/0x340
[650521.310423]  [<c0245032>] ext4_add_entry+0xe2/0x1d0
[650521.310427]  [<c024540a>] ext4_add_nondir+0x1a/0x70
[650521.310430]  [<c0245814>] ext4_create+0xc4/0x100
[650521.310434]  [<c01d0ced>] vfs_create+0xcd/0x170
[650521.310437]  [<c01d2562>] __open_namei_create+0x42/0xa0
[650521.310441]  [<c01d2be0>] do_filp_open+0x620/0x660
[650521.310444]  [<c01ca888>] ? cp_new_stat64+0xe8/0x100
[650521.310448]  [<c02d3a67>] ? strncpy_from_user+0x37/0x60
[650521.310451]  [<c01c52a1>] do_sys_open+0x51/0xe0
[650521.310455]  [<c01c5399>] sys_open+0x29/0x40
[650521.310458]  [<c0102ebc>] sysenter_do_call+0x12/0x28

Okay, what the hell…Two questions.

Yes! At work today we made dhcpd crash on a /etc/dhcpd.conf :) We were working towards the following (now working) setup:

class "igel" {
	match if binary-to-ascii(16, 8, "-", substring (hardware, 0, 5)) = "1-0-e0-c5-67";
}
pool {
	allow members of "igel"; 
	range 192.84.30.161 192.84.30.170; # igel[0-9]
}

But while figuring this out, dhcpd went boom :)

[root@vmaster cfengine]# dhcpd -d -f -cf hosts/vnet/etc/dhcpd.conf
Internet Systems Consortium DHCP Server V3.0.5-RedHat
Copyright 2004-2006 Internet Systems Consortium.
All rights reserved.
For info, please visit http://www.isc.org/sw/dhcp/
hosts/vnet/etc/dhcpd.conf line 20: can't override match.
    match if 
    ^
hosts/vnet/etc/dhcpd.conf line 22: pool declared outside of network pool
				^
Segmentation fault

We (my wife and I) have a lot of books, currently some 637…

$ sqlite3 biblio.db 'select count(*) from books' 
637

But we seem to loose books occasionally because we lend them to people and forget about it. So Linux (and Perl) to the rescue. I already own a bar code reader to read the ISBN number of books and then I use Google to give the author, genre, etc. (fully automated of course).

This is an English translation of a blog item I wrote for AT Computing

While giving a course a student showed me the following:

$ ps -ef > /tmp/file

Where /tmp is 100% filled yields no errors and seems to have worked!

Lets try to see what is going on here.

Firstly, lets fill up a file system. We are going to use an fs mounted under /media/disk:

$ cp /dev/zero /media/disk/HUGE                 
$ cp: writing `/media/disk/HUGE': No space left on device 

Further we need a little program to test a few things:

Well, thanks to Ubuntu I’m now running the new (new!) 2.6.30 kernel on my systems.

No ill effects as of yet…

Btw, I’ve written the following script to download the latest kernels from Ubuntu:

SYNOPSIS: latest 29.4, this will fetch 2.6.29.4 or latest 30 which will get 2.6.30.

#!/bin/bash
# download the latest ubuntu mainline kernels
ubuntu="http://kernel.ubuntu.com/~kernel-ppa/mainline/"
version="2.6.$1" # need 29.n as argument
arch="i386"	 # or amd64
major=${version%.[0-9]}
minor=${1%.[0-9]}
patch=${version#2.6.*.}

[ -z "$1" ] && { echo -e "Usage: $0 MINOR\n$0 29.3"; exit 1; }
[ -n "$2" ] && arch="$2"

if echo "$patch" | grep -q '\.'; then
printf -v versionstr "0206%d" $minor
else
printf -v versionstr "0206%d%02d" $minor $patch
fi

printf -v generic "%s-${versionstr}-generic_%s-${versionstr}" $major $major
printf -v all "%s-${versionstr}_%s-${versionstr}" $major $major

echo "${ubuntu}v${version}/linux-headers-${generic}_${arch}.deb"
wget --progress=bar "${ubuntu}v${version}/linux-headers-${generic}_${arch}.deb"
echo "${ubuntu}v${version}/linux-headers-${all}_all.deb"
wget --progress=bar "${ubuntu}v${version}/linux-headers-${all}_all.deb"
echo "${ubuntu}v${version}/linux-image-${generic}_${arch}.deb"
wget --progress=bar "${ubuntu}v${version}/linux-image-${generic}_${arch}.deb"

Suppose you want to limit access to some servers, only people member of a specific group (or multiple groups) may log in.

The following is one way to tackle this. In this example I will configure ssh access in such a way that only people from the admin group can login. The nice thing is that this will work regardless of any Kerberos or LDAP usage.

Preparation

In /etc/pam.d find the “service” which you want to add a group policy to. For instance sshd, edit that file (this is with Ubuntu):

Mental note to self

Having a host listed in a listprincs output isn’t enough to have single-sign-on working.

You have explicitly add it with ktadd host/your.host.com

So in my case:

# kdadmin.local
....
kadmin.local:  addprinc -randkey host/charm.atoom.net
...
kadmin.local:  quit

And then you can do a (on charm.atoom.net):

% kinit
Password for miekg@ATOOM.NET: 
% slogin elektron.atoom.net

And have a password-less login to my server.

I wanted some git information in the prompt I currently have. While thinking and googling about this, a quote on Bart’s log got me thinking. I don’t need to see my current host, nor do I need to see my current user name.

I always login under my own account (miekg) and if I do change accounts, I probably will be forced to use bash. So showing the current user name is quite useless.

Oh my God — why?? This is a 250 GB disk with a USB2 interface, so I’m in for a long wait apparently.

# mkfs.ntfs /dev/sdc1
Cluster size has been automatically set to 4096 bytes.
Initializing device with zeroes:   4%

It this the Windows-way or is mkfs.ntfs a little bit brain dead?

I’ve update the teletekst script a little. It now outputs utf8 and uses LWP::Simple so no external commands are called anymore.

You can find it here.

I could not agree more with Christoph Haas.

I’ve tried KDE briefly during the last 6 months or so, but it never really stuck.

Some observations.

1. I’ve bought an EeeBox (Dual Atom, with 1 GB of ram), boy, how slow is KDE. Do I really need a Vista capable machine to run KDE?

2. I like my current (XFCE) desktop. Why? Because it’s nice and clean. I have no use for desktop icons. I have no use for fancy clocks running on my desktop. I have no use for any fancy stuff running on my desktop. Why?

I want to grow my RAID1 array from 2x250GB to 2x500GB. The following resource has helped my a lot.

Update

It all worked, allthough the kernel decided to crash during the offline ext4 resize. But I finally have my extra disk space

% df -h | grep md6

before: /dev/md6 77G 67G 6.7G 91% /vol

after: /dev/md6 306G 67G 224G 23% /vol

Now to my problem/solution:

On Linux you often have the following problem: You are looking or touching some piece of hardware and you are asking yourself: “is this card eth0 or eth1?” or “is this disk /dev/sda or /dev/sdb?”.

I’m rebuilding my RAID1 array, so everything is a bit slow at the moment.

I’ve always kept my first Linux CDs as a reminder of that great time when I was first exploring this unexplored territory.

These are two (bad) shots of the actual CD, maybe I should put the content of them online again? Kernel version 1.2.8, gcc version old, non working X on the default install, that kind of stuff… I was hooked forever to this stuff.

I’ve never like (Open)Solaris. Maybe the kernel rules, but the userland completely sucks, something like apt should have been implemented in (Open)Solaris years ago.

Now that Oracle owns SUN and thus Solaris they should rebrand it. Maybe OpenSoracle or even OpenSorry.

Found this very nice howto, which I almost followed to the letter, except I did it in Ubuntu and currently more stuff is working out of the box, so you don’t need to do all the steps.

I’m working on an USB stick which is /dev/sdb1 in my system.

Steps I did take:

1. apt-get install cryptsetup hashalot
2. cryptsetup --verbose --verify-passphrase luksFormat /dev/sdb1
3. cryptsetup luksOpen /dev/sdb1 funnydiskname
4. mkfs.ext4 -O extents,uninit_bg,dir_index /dev/mapper/funnydiskname
5. cryptsetup luksClose /dev/mapper/funnydiskname

Now pull out your disk and re-insert. If everything went well, your GNOME, XFCE, KDE env should now popup an ‘This disk is encrypted, please enter the passphrase’ question.

In kernel version 2.6.29.1:

# /bin/echo -n 40000 > /proc/sys/kernel/pid_max 
/bin/echo: write error: Invalid argument

In one older kernel I tried it still worked, so I’ve submitted a bug.

OpenLDAP uses a cn=config DIT to configure the server since version 2.4. I’m always into new stuff, but I must admit that I rather liked editing /etc/ldap/slapd.conf to configure the server. Anyhow being able to store ACLs in the tree is a big plus, but for configuring minor stuff (like indexes) it makes live more difficult.

The following site was an excellent tool in helping me configure OpenLDAP. For a list of current attributes names, see for instance here

Well, today I was looking into using LVM snapshots to allow a client OCN use Linux as a Netapp replacement…. Boy was I in for a disappointment.

LVM an sich is working great, but the moment you turn on snapshots the (in this case) write performance goes to hell. Using LVM is easy enough. The system I was on has 32 GB ram and 2 disk arrays with hardware RAID.

Setting up:

See this?

bond0: Warning: failed to get speed and duplix from eth*

Do this:

insmod bonding miimon=100

why?

When using Linux network bonding the kernel may be to quick to enslave the interfaces. When an interface is too slow to report it capabilities it will be set to 100Mbit and Full Duplex. Which is a bit sad when you have 6 Gigabit network cards…

For a few weeks months now I’m learnings how to use VIM text-objects. There is an extended help wth help text-objects in VIM. I’m trying to condense the VIM help in a smaller blog entry here.

What are a text-objects in vim? Text-objects are things like a ‘paragraph’ or the text between braces or something like a word. Text-objects can be used with the normal vim commands y, d and c. To make you really understand it, I can recommend using control-V to actually visually select your text object.

As I’m on an upgrade roll I decided to upgrade my new server too. It is configured with kerberos and ldap… and this is were the trouble.

ldap

When a service is upgraded in Ubuntu/Debian it is first stopped and than later restarted when the new files are there. When all your user information is kept in ldap, the following sucks:

Preparing to replace slapd 2.4.11-0ubuntu6.1 (using
    .../slapd_2.4.15-1ubuntu3_i386.deb) ...
Stopping OpenLDAP: slapd.
  Dumping to /var/backups/slapd-2.4.11-0ubuntu6.1:
- directory dc=atoom,dc=net... done.
Unpacking replacement slapd ...

And there goes the neighborhood… everything slows to a crawl, because every getpwnam call needs to timeout before /etc/passwd and friends are consulted.

A short guide on how to build the new notification system, which is scheduled for Jaunty, for Intrepid.

First download the source packages from: http://packages.ubuntu.com/source/jaunty/notify-osd

You will need the .dsc and the .tar.gz files, then you need to follow the Debian way of building the package.

dpkg-source -x notify-osd_0.9ubuntu2.dsc
cd notify-osd-0.9ubuntu2/

One important tweak, otherwise it will not install in Intrepid and will complain about an human-icon-theme that is tool old