Currently we are building a fairly rock solid high availability cluster for a client. This has the “usual” ingredients: two locations, two NetApps, two clusters of three vmware ESX servers and a bunch of virtual machines running on top of the ESX servers. Also included in the mix is a VDI (now called View) virtual desktop infrastructure for running virtual windows XP clients.

This is all managed by SRM (site recovery manager) and it is almost working. But that is another story.

I already knew (Open)Solaris sucks, but now Ton has also figured it out.

ZFS definitely does not suck. Why not petition Oracle to GPL(v2) ZFS? Especially now Oracle wants to kill OpenSolaris?

So, there I am browsing my logs.

BAM

[650521.310292] INFO: task mysqld:2730 blocked for more than 120 seconds.
[650521.310320] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[650521.310364] mysqld        D 00000000     0  2730   2674
[650521.310367]  f6183ce0 00000086 c06ef6cc 00000000 f61c8c70 7834922a 00000dc2 00020050
[650521.310373]  00000296 c0796fe0 c0796fe0 f61c8f04 c2c35fe0 00000003 9500b16e 00024f83
[650521.310379]  00000002 00000002 f66bb500 f6d02e40 c2bf2090 f6e2e680 f6183ce8 c050eb68
[650521.310384] Call Trace:
[650521.310388]  [<c050eb68>] schedule+0x8/0x20
[650521.310391]  [<c0266701>] do_get_write_access+0x2a1/0x3d0
[650521.310395]  [<c0127d93>] ? kmap_atomic_prot+0x43/0xe0
[650521.310398]  [<c0150030>] ? wake_bit_function+0x0/0x60
[650521.310402]  [<c02669a3>] jbd2_journal_get_write_access+0x23/0x40
[650521.310405]  [<c0252a36>] __ext4_journal_get_write_access+0x26/0x60
[650521.310409]  [<c023e9d4>] ext4_reserve_inode_write+0x34/0x70
[650521.310413]  [<c023ea45>] ext4_mark_inode_dirty+0x35/0x140
[650521.310416]  [<c02669ac>] ? jbd2_journal_get_write_access+0x2c/0x40
[650521.310420]  [<c02446cf>] add_dirent_to_buf+0x12f/0x340
[650521.310423]  [<c0245032>] ext4_add_entry+0xe2/0x1d0
[650521.310427]  [<c024540a>] ext4_add_nondir+0x1a/0x70
[650521.310430]  [<c0245814>] ext4_create+0xc4/0x100
[650521.310434]  [<c01d0ced>] vfs_create+0xcd/0x170
[650521.310437]  [<c01d2562>] __open_namei_create+0x42/0xa0
[650521.310441]  [<c01d2be0>] do_filp_open+0x620/0x660
[650521.310444]  [<c01ca888>] ? cp_new_stat64+0xe8/0x100
[650521.310448]  [<c02d3a67>] ? strncpy_from_user+0x37/0x60
[650521.310451]  [<c01c52a1>] do_sys_open+0x51/0xe0
[650521.310455]  [<c01c5399>] sys_open+0x29/0x40
[650521.310458]  [<c0102ebc>] sysenter_do_call+0x12/0x28

Okay, what the hell…Two questions.

Yes! At work today we made dhcpd crash on a /etc/dhcpd.conf :) We were working towards the following (now working) setup:

class "igel" {
	match if binary-to-ascii(16, 8, "-", substring (hardware, 0, 5)) = "1-0-e0-c5-67";
}
pool {
	allow members of "igel"; 
	range 192.84.30.161 192.84.30.170; # igel[0-9]
}

But while figuring this out, dhcpd went boom :)

[root@vmaster cfengine]# dhcpd -d -f -cf hosts/vnet/etc/dhcpd.conf
Internet Systems Consortium DHCP Server V3.0.5-RedHat
Copyright 2004-2006 Internet Systems Consortium.
All rights reserved.
For info, please visit http://www.isc.org/sw/dhcp/
hosts/vnet/etc/dhcpd.conf line 20: can't override match.
    match if 
    ^
hosts/vnet/etc/dhcpd.conf line 22: pool declared outside of network pool
				^
Segmentation fault

We (my wife and I) have a lot of books, currently some 637…

$ sqlite3 biblio.db 'select count(*) from books' 
637

But we seem to loose books occasionally because we lend them to people and forget about it. So Linux (and Perl) to the rescue. I already own a bar code reader to read the ISBN number of books and then I use Google to give the author, genre, etc. (fully automated of course).

This is an English translation of a blog item I wrote for AT Computing

While giving a course a student showed me the following:

$ ps -ef > /tmp/file

Where /tmp is 100% filled yields no errors and seems to have worked!

Lets try to see what is going on here.

Firstly, lets fill up a file system. We are going to use an fs mounted under /media/disk:

$ cp /dev/zero /media/disk/HUGE                 
$ cp: writing `/media/disk/HUGE': No space left on device 

Further we need a little program to test a few things:

Well, thanks to Ubuntu I’m now running the new (new!) 2.6.30 kernel on my systems.

No ill effects as of yet…

Btw, I’ve written the following script to download the latest kernels from Ubuntu:

SYNOPSIS: latest 29.4, this will fetch 2.6.29.4 or latest 30 which will get 2.6.30.

#!/bin/bash
# download the latest ubuntu mainline kernels
ubuntu="http://kernel.ubuntu.com/~kernel-ppa/mainline/"
version="2.6.$1" # need 29.n as argument
arch="i386"	 # or amd64
major=${version%.[0-9]}
minor=${1%.[0-9]}
patch=${version#2.6.*.}

[ -z "$1" ] && { echo -e "Usage: $0 MINOR\n$0 29.3"; exit 1; }
[ -n "$2" ] && arch="$2"

if echo "$patch" | grep -q '\.'; then
printf -v versionstr "0206%d" $minor
else
printf -v versionstr "0206%d%02d" $minor $patch
fi

printf -v generic "%s-${versionstr}-generic_%s-${versionstr}" $major $major
printf -v all "%s-${versionstr}_%s-${versionstr}" $major $major

echo "${ubuntu}v${version}/linux-headers-${generic}_${arch}.deb"
wget --progress=bar "${ubuntu}v${version}/linux-headers-${generic}_${arch}.deb"
echo "${ubuntu}v${version}/linux-headers-${all}_all.deb"
wget --progress=bar "${ubuntu}v${version}/linux-headers-${all}_all.deb"
echo "${ubuntu}v${version}/linux-image-${generic}_${arch}.deb"
wget --progress=bar "${ubuntu}v${version}/linux-image-${generic}_${arch}.deb"

Suppose you want to limit access to some servers, only people member of a specific group (or multiple groups) may log in.

The following is one way to tackle this. In this example I will configure ssh access in such a way that only people from the admin group can login. The nice thing is that this will work regardless of any Kerberos or LDAP usage.

Preparation

In /etc/pam.d find the “service” which you want to add a group policy to. For instance sshd, edit that file (this is with Ubuntu):

Mental note to self

Having a host listed in a listprincs output isn’t enough to have single-sign-on working.

You have explicitly add it with ktadd host/your.host.com

So in my case:

# kdadmin.local
....
kadmin.local:  addprinc -randkey host/charm.atoom.net
...
kadmin.local:  quit

And then you can do a (on charm.atoom.net):

% kinit
Password for miekg@ATOOM.NET: 
% slogin elektron.atoom.net

And have a password-less login to my server.

I wanted some git information in the prompt I currently have. While thinking and googling about this, a quote on Bart’s log got me thinking. I don’t need to see my current host, nor do I need to see my current user name.

I always login under my own account (miekg) and if I do change accounts, I probably will be forced to use bash. So showing the current user name is quite useless.

Oh my God — why?? This is a 250 GB disk with a USB2 interface, so I’m in for a long wait apparently.

# mkfs.ntfs /dev/sdc1
Cluster size has been automatically set to 4096 bytes.
Initializing device with zeroes:   4%

It this the Windows-way or is mkfs.ntfs a little bit brain dead?

I’ve update the teletekst script a little. It now outputs utf8 and uses LWP::Simple so no external commands are called anymore.

You can find it here.

I could not agree more with Christoph Haas.

I’ve tried KDE briefly during the last 6 months or so, but it never really stuck.

Some observations.

1. I’ve bought an EeeBox (Dual Atom, with 1 GB of ram), boy, how slow is KDE. Do I really need a Vista capable machine to run KDE?

2. I like my current (XFCE) desktop. Why? Because it’s nice and clean. I have no use for desktop icons. I have no use for fancy clocks running on my desktop. I have no use for any fancy stuff running on my desktop. Why?

I want to grow my RAID1 array from 2x250GB to 2x500GB. The following resource has helped my a lot.

Update

It all worked, allthough the kernel decided to crash during the offline ext4 resize. But I finally have my extra disk space

% df -h | grep md6

before: /dev/md6 77G 67G 6.7G 91% /vol

after: /dev/md6 306G 67G 224G 23% /vol

Now to my problem/solution:

On Linux you often have the following problem: You are looking or touching some piece of hardware and you are asking yourself: “is this card eth0 or eth1?” or “is this disk /dev/sda or /dev/sdb?”.

I’m rebuilding my RAID1 array, so everything is a bit slow at the moment.

I’ve always kept my first Linux CDs as a reminder of that great time when I was first exploring this unexplored territory.

These are two (bad) shots of the actual CD, maybe I should put the content of them online again? Kernel version 1.2.8, gcc version old, non working X on the default install, that kind of stuff… I was hooked forever to this stuff.

I’ve never like (Open)Solaris. Maybe the kernel rules, but the userland completely sucks, something like apt should have been implemented in (Open)Solaris years ago.

Now that Oracle owns SUN and thus Solaris they should rebrand it. Maybe OpenSoracle or even OpenSorry.

Found this very nice howto, which I almost followed to the letter, except I did it in Ubuntu and currently more stuff is working out of the box, so you don’t need to do all the steps.

I’m working on an USB stick which is /dev/sdb1 in my system.

Steps I did take:

1. apt-get install cryptsetup hashalot
2. cryptsetup --verbose --verify-passphrase luksFormat /dev/sdb1
3. cryptsetup luksOpen /dev/sdb1 funnydiskname
4. mkfs.ext4 -O extents,uninit_bg,dir_index /dev/mapper/funnydiskname
5. cryptsetup luksClose /dev/mapper/funnydiskname

Now pull out your disk and re-insert. If everything went well, your GNOME, XFCE, KDE env should now popup an ‘This disk is encrypted, please enter the passphrase’ question.

In kernel version 2.6.29.1:

# /bin/echo -n 40000 > /proc/sys/kernel/pid_max 
/bin/echo: write error: Invalid argument

In one older kernel I tried it still worked, so I’ve submitted a bug.

OpenLDAP uses a cn=config DIT to configure the server since version 2.4. I’m always into new stuff, but I must admit that I rather liked editing /etc/ldap/slapd.conf to configure the server. Anyhow being able to store ACLs in the tree is a big plus, but for configuring minor stuff (like indexes) it makes live more difficult.

The following site was an excellent tool in helping me configure OpenLDAP. For a list of current attributes names, see for instance here

Well, today I was looking into using LVM snapshots to allow a client OCN use Linux as a Netapp replacement…. Boy was I in for a disappointment.

LVM an sich is working great, but the moment you turn on snapshots the (in this case) write performance goes to hell. Using LVM is easy enough. The system I was on has 32 GB ram and 2 disk arrays with hardware RAID.

Setting up:

See this?

bond0: Warning: failed to get speed and duplix from eth*

Do this:

insmod bonding miimon=100

why?

When using Linux network bonding the kernel may be to quick to enslave the interfaces. When an interface is too slow to report it capabilities it will be set to 100Mbit and Full Duplex. Which is a bit sad when you have 6 Gigabit network cards…

For a few weeks months now I’m learnings how to use VIM text-objects. There is an extended help wth help text-objects in VIM. I’m trying to condense the VIM help in a smaller blog entry here.

What are a text-objects in vim? Text-objects are things like a ‘paragraph’ or the text between braces or something like a word. Text-objects can be used with the normal vim commands y, d and c. To make you really understand it, I can recommend using control-V to actually visually select your text object.

As I’m on an upgrade roll I decided to upgrade my new server too. It is configured with kerberos and ldap… and this is were the trouble.

ldap

When a service is upgraded in Ubuntu/Debian it is first stopped and than later restarted when the new files are there. When all your user information is kept in ldap, the following sucks:

Preparing to replace slapd 2.4.11-0ubuntu6.1 (using
    .../slapd_2.4.15-1ubuntu3_i386.deb) ...
Stopping OpenLDAP: slapd.
  Dumping to /var/backups/slapd-2.4.11-0ubuntu6.1:
- directory dc=atoom,dc=net... done.
Unpacking replacement slapd ...

And there goes the neighborhood… everything slows to a crawl, because every getpwnam call needs to timeout before /etc/passwd and friends are consulted.

A short guide on how to build the new notification system, which is scheduled for Jaunty, for Intrepid.

First download the source packages from: http://packages.ubuntu.com/source/jaunty/notify-osd

You will need the .dsc and the .tar.gz files, then you need to follow the Debian way of building the package.

dpkg-source -x notify-osd_0.9ubuntu2.dsc
cd notify-osd-0.9ubuntu2/

One important tweak, otherwise it will not install in Intrepid and will complain about an human-icon-theme that is tool old

I’ve used the Debian xfce pkg stuff to build Ubuntu packages of xfce 4.6.

The … release … rocks :)

You can find the debs in files/xfce/.

Be my guest…

Today if removed two annoyances I had with VIM.

Command mistyping

I often mistype the following:

:w! becomes :W!

or

:q! becomes :Q!

Which is annoying because W and Q do not mean anything, and I don’t write or VIM does not quit. VIM has a nifty feature called commands which you can (re)define or add new commands.

Lets try some to fix this, in a running vi:

:com -bang W write!

or

The following might be helpful to others too. I was trying to setup a new raid1 device from two partitions /dev/sda4 and /dev/sdb4. I wanted to do this the “right way” and use UUID everywhere, i.e. in /etc/mdadm/mdadm.conf and in /etc/fstab.

I hit a few snags along the way.

create the array

# mdadm --create --verbose /dev/md6 --level=1 --raid-devices=2 \
/dev/sda4 /dev/sdb4

Get the uuid mdadm uses:

# mdadm --detail  /dev/md6 | grep UUID
  UUID : dc9aba5e:ed1a70d4:770765d8:b0f56d86 (local to host elektron2)

Check. Add that to /etc/mdadm/mdadm.conf:

In bash (and other shells) you can use the [[ construct

From the bash manpage:

[[ expression ]]
          Return a status of 0 or 1 depending on  the  evaluation  of  the
          conditional  expression expression.  Expressions are composed of
          the primaries described  below  under  CONDITIONAL  EXPRESSIONS.

          Word  splitting  and pathname expansion are not performed on the

          words between the [[ and  ]];  tilde  expansion,  parameter  and
          variable  expansion, arithmetic expansion, command substitution,
          process substitution, and quote removal are  performed.   Condi‐
          tional operators such as -f must be unquoted to be recognized as
          primaries.

So no word splitting and pathname expansion.

Re-selection of a visual

When you use control-v or shift-v in vim to do a visual selection and then use (for instance) y to copy the selected text, you loose the selection.

More often than not you want to do another thing with the selection you just had. How?

gv

Copy to clipboard

Make a selection and use

"*yy

Now the selection is ‘under your mouse’. If think I’m going to bind this sequence to something more easy, like *Y.