Idle Musings

Sysadmin stuff

IrssiNotifier for Weechat

A while ago, I found out about IrssiNotifier, a handy little Android app written by Lauri Härsilä. It uses Android’s built-in client-to-device notification system (C2DM) to send notifications of IRC highlights from irssi to your Android phone. Being an avid user of both IRC and Android, I was interested by this project. However, I don’t use irssi any more and have changed to Weechat instead.

This presented me with a bit of a problem. Never one to shy away from these, though, I broke out my limited Perl skills and rewrote the irssinotifier.pl script to hook into weechat instead. After an evening of working and testing, I was happy enough with the results to call it done and move on.

I did email murgo, the author of the app, to ask whether he’d consider including my weechat version of the script on his website, but he never got back to me. As this was a few months ago now, I figure it’ll be fine if I fork his repository and put it on Github instead.

You can find the repo including the weechat version of the script here - https://github.com/webvictim/irssinotifier

I’m vaguely proud of this because it’s the first real project I’ve committed to Github, which seems like something of a rite of passage for a sysadmin/developer. It’s a bit messy as I’m still learning exactly how to use Git properly, but it’s a start.

Linux Software RAID With Western Digital WD20EARS Green Drives

In a previous post I mentioned that I had a bad experience setting up WD20EARS drives for use in mdadm software RAID under Linux. Thought it’d be a good idea to explain what the problems were and what my current configuration is as there seems to be precious little obvious information online about this.

WD20EARS/EARX drives are “advanced format”, meaning they use a 4K sector layout rather than the standard 512-byte sectors found on most older hard drives. You can generally tell a drive like this by the “AF” or “advanced format” wording on the label. I hadn’t read this information and was blissfully unaware, so when I installed Fedora 17 and made a new partition table on each of the drives, I just picked the default options. Unfortunately, the default (at least in fdisk’s standard mode) is to start at sector 63 and use the full size of the drive.

What I found previously is that when build my RAID array initially, the sync process would take around 48 hours to get to 99.2% done, then errors would spuriously appear in my syslog and the rebuild would fail as a drive dropped out and the array had too few working devices left. This can be seen in /var/log/messages:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66

Jul 29 12:00:42 zeus kernel: [211732.938234] ata3.00: exception Emask 0x0 SAct 0x7f81c000 SErr 0x0 action 0x6 frozen
Jul 29 12:00:42 zeus kernel: [211732.940747] ata3.00: failed command: WRITE FPDMA QUEUED
Jul 29 12:00:42 zeus kernel: [211732.943208] ata3.00: cmd 61/08:70:4f:00:00/00:00:00:00:00/40 tag 14 ncq 4096 out
Jul 29 12:00:42 zeus kernel: [211732.943212]          res 40/00:00:00:00:00/00:00:00:00:00/00 Emask 0x4 (timeout)
Jul 29 12:00:42 zeus kernel: [211732.948087] ata3.00: status: { DRDY }
Jul 29 12:00:42 zeus kernel: [211732.950566] ata3.00: failed command: READ FPDMA QUEUED
Jul 29 12:00:42 zeus kernel: [211732.953098] ata3.00: cmd 60/00:78:27:d0:99/04:00:e7:00:00/40 tag 15 ncq 524288 in
Jul 29 12:00:42 zeus kernel: [211732.953101]          res 40/00:00:00:00:00/00:00:00:00:00/00 Emask 0x4 (timeout)
Jul 29 12:00:42 zeus kernel: [211732.958229] ata3.00: status: { DRDY }
Jul 29 12:00:42 zeus kernel: [211732.960785] ata3.00: failed command: READ FPDMA QUEUED
Jul 29 12:00:42 zeus kernel: [211732.963505] ata3.00: cmd 60/08:80:27:d4:99/02:00:e7:00:00/40 tag 16 ncq 266240 in
Jul 29 12:00:42 zeus kernel: [211732.963508]          res 40/00:00:00:00:00/00:00:00:00:00/00 Emask 0x4 (timeout)
Jul 29 12:00:42 zeus kernel: [211732.968986] ata3.00: status: { DRDY }
Jul 29 12:00:42 zeus kernel: [211732.971698] ata3.00: failed command: READ FPDMA QUEUED
Jul 29 12:00:42 zeus kernel: [211732.974376] ata3.00: cmd 60/08:b8:e7:cf:99/00:00:e7:00:00/40 tag 23 ncq 4096 in
Jul 29 12:00:42 zeus kernel: [211732.974379]          res 40/00:00:00:00:00/00:00:00:00:00/00 Emask 0x4 (timeout)
Jul 29 12:00:42 zeus kernel: [211732.979570] ata3.00: status: { DRDY }
Jul 29 12:00:42 zeus kernel: [211732.982215] ata3.00: failed command: READ FPDMA QUEUED
Jul 29 12:00:42 zeus kernel: [211732.984698] ata3.00: cmd 60/08:c0:ef:cf:99/00:00:e7:00:00/40 tag 24 ncq 4096 in
Jul 29 12:00:42 zeus kernel: [211732.984701]          res 40/00:00:00:00:00/00:00:00:00:00/00 Emask 0x4 (timeout)
Jul 29 12:00:42 zeus kernel: [211732.990083] ata3.00: status: { DRDY }
Jul 29 12:00:42 zeus kernel: [211732.992825] ata3.00: failed command: READ FPDMA QUEUED
Jul 29 12:00:42 zeus kernel: [211732.995608] ata3.00: cmd 60/08:c8:f7:cf:99/00:00:e7:00:00/40 tag 25 ncq 4096 in
Jul 29 12:00:42 zeus kernel: [211732.995612]          res 40/00:00:00:00:00/00:00:00:00:00/00 Emask 0x4 (timeout)
Jul 29 12:00:42 zeus kernel: [211733.001215] ata3.00: status: { DRDY }
Jul 29 12:00:42 zeus kernel: [211733.004021] ata3.00: failed command: READ FPDMA QUEUED
Jul 29 12:00:42 zeus kernel: [211733.006885] ata3.00: cmd 60/08:d0:ff:cf:99/00:00:e7:00:00/40 tag 26 ncq 4096 in
Jul 29 12:00:42 zeus kernel: [211733.006888]          res 40/00:00:00:00:00/00:00:00:00:00/00 Emask 0x4 (timeout)
Jul 29 12:00:42 zeus kernel: [211733.012528] ata3.00: status: { DRDY }
Jul 29 12:00:42 zeus kernel: [211733.015117] ata3.00: failed command: READ FPDMA QUEUED
Jul 29 12:00:42 zeus kernel: [211733.017697] ata3.00: cmd 60/08:d8:07:d0:99/00:00:e7:00:00/40 tag 27 ncq 4096 in
Jul 29 12:00:42 zeus kernel: [211733.017700]          res 40/00:00:00:00:00/00:00:00:00:00/00 Emask 0x4 (timeout)
Jul 29 12:00:42 zeus kernel: [211733.023182] ata3.00: status: { DRDY }
Jul 29 12:00:42 zeus kernel: [211733.025894] ata3.00: failed command: READ FPDMA QUEUED
Jul 29 12:00:42 zeus kernel: [211733.028646] ata3.00: cmd 60/08:e0:1f:d0:99/00:00:e7:00:00/40 tag 28 ncq 4096 in
Jul 29 12:00:42 zeus kernel: [211733.028649]          res 40/00:00:00:00:00/00:00:00:00:00/00 Emask 0x4 (timeout)
Jul 29 12:00:42 zeus kernel: [211733.034257] ata3.00: status: { DRDY }
Jul 29 12:00:42 zeus kernel: [211733.037113] ata3.00: failed command: READ FPDMA QUEUED
Jul 29 12:00:42 zeus kernel: [211733.040008] ata3.00: cmd 60/08:e8:0f:d0:99/00:00:e7:00:00/40 tag 29 ncq 4096 in
Jul 29 12:00:42 zeus kernel: [211733.040012]          res 40/00:00:00:00:00/00:00:00:00:00/00 Emask 0x4 (timeout)
Jul 29 12:00:42 zeus kernel: [211733.045785] ata3.00: status: { DRDY }
Jul 29 12:00:42 zeus kernel: [211733.048709] ata3.00: failed command: READ FPDMA QUEUED
Jul 29 12:00:42 zeus kernel: [211733.051644] ata3.00: cmd 60/08:f0:17:d0:99/00:00:e7:00:00/40 tag 30 ncq 4096 in
Jul 29 12:00:42 zeus kernel: [211733.051648]          res 40/00:00:00:00:00/00:00:00:00:00/00 Emask 0x4 (timeout)
Jul 29 12:00:42 zeus kernel: [211733.057550] ata3.00: status: { DRDY }
Jul 29 12:00:42 zeus kernel: [211733.060518] ata3: hard resetting link
Jul 29 12:00:42 zeus kernel: [211733.519527] ata3: SATA link up 3.0 Gbps (SStatus 123 SControl 300)
Jul 29 12:00:42 zeus kernel: [211733.527772] ata3.00: configured for UDMA/133
Jul 29 12:00:42 zeus kernel: [211733.527787] ata3.00: device reported invalid CHS sector 0
Jul 29 12:00:42 zeus kernel: [211733.527799] ata3.00: device reported invalid CHS sector 0
Jul 29 12:00:42 zeus kernel: [211733.527807] ata3.00: device reported invalid CHS sector 0
Jul 29 12:00:42 zeus kernel: [211733.527813] ata3.00: device reported invalid CHS sector 0
Jul 29 12:00:42 zeus kernel: [211733.527819] ata3.00: device reported invalid CHS sector 0
Jul 29 12:00:42 zeus kernel: [211733.527824] ata3.00: device reported invalid CHS sector 0
Jul 29 12:00:42 zeus kernel: [211733.527829] ata3.00: device reported invalid CHS sector 0
Jul 29 12:00:42 zeus kernel: [211733.527834] ata3.00: device reported invalid CHS sector 0
Jul 29 12:00:42 zeus kernel: [211733.527840] ata3.00: device reported invalid CHS sector 0
Jul 29 12:00:42 zeus kernel: [211733.527845] ata3.00: device reported invalid CHS sector 0
Jul 29 12:00:42 zeus kernel: [211733.527850] ata3.00: device reported invalid CHS sector 0
Jul 29 12:00:42 zeus kernel: [211733.527869] sd 2:0:0:0: [sdc] Unhandled error code
Jul 29 12:00:42 zeus kernel: [211733.527874] sd 2:0:0:0: [sdc]  Result: hostbyte=DID_OK driverbyte=DRIVER_TIMEOUT
Jul 29 12:00:42 zeus kernel: [211733.527882] sd 2:0:0:0: [sdc] CDB: Read(10): 28 00 e7 99 cf 6f 00 00 08 00
Jul 29 12:00:42 zeus kernel: [211733.527901] end_request: I/O error, dev sdc, sector 3885617007
Jul 29 12:00:42 zeus kernel: [211733.530703] md/raid:md0: read error not correctable (sector 3885616944 on sdc1).
Jul 29 12:00:42 zeus kernel: [211733.530712] md/raid:md0: Disk failure on sdc1, disabling device.
Jul 29 12:00:42 zeus kernel: [211733.530715] md/raid:md0: Operation continuing on 2 devices.

Spain

My lack of a followup post to the NAS setup is primarily due to my having been on holiday for most of the last two weeks. I went to Playa Flamenca in Spain and thoroughly enjoyed myself.

Building a NAS From an HP Microserver

I recently purchased an an HP ProLiant N40L Microserver from eBuyer to replace my old Drobo as a NAS. The Drobo was in fact more of a USB enclosure attached to a media PC than a true NAS, so I was keen to get an all-in-one solution. The £100 cashback offer seems to recur fairly often, so if it’s not offered at the time you’re looking, put the purchase on hold for a little while and it’ll probably come back.

Some noteworthy features of the N40L:

  • Low power dual core 1.5GHz processor
  • Capacity for 8GB RAM
  • Four drive bays which claim not to be hot-swappable but which can be made so with a BIOS modification (see top post)
  • Six AHCI SATA ports in total (one is an eSATA port on the back which can be used with an eSATA to SATA cable fed in through an empty PCI-E slot)
  • Small footprint, relatively low power consumption
  • An internal USB port for use if necessary - you could put your OS on a USB drive and have six storage drives in RAID if you wanted!

The server comes with a fairly standard 250GB Western Digital hard drive of its own installed in one of the main drive bays. I removed this and used the extra internal SATA port and power intended for a CD-ROM. After moving this, I installed the four 2TB hard drives I removed from the Drobo before selling it, then flashed the BIOS and imaged the server with Fedora 17, my current distro of choice.

Coming soon: Setting up Linux software RAID, and caveats to consider when using Western Digital WD20EARS “Green” drives (long story short: they have the “advanced format” 4KB sector layout as opposed to more standard 512-byte sectors and I didn’t realise this at the time, causing slow performance) - click here to read

Carriage Returns

I’ve had the unfortunate experience today of finding a lot of config files full of completely unnecessary trailing newlines. One newline at the end is perhaps acceptable, but any more than one is an absolute no-no. Not only do some applications whine about them (just like some do if you don’t incline the trailing newline) but they waste valuable terminal space when trying to read through a file.

I think I know who the culprit is - a colleague who is particularly fond of the enter key and who I happen to know configured this particular server. Needless to say, all surplus carriage returns have now been removed and I’ve moved slightly further forward with the Puppetisation of these particular config files. At least when Puppet keeps removing all the line breaks people eventually get the message…

Don’t even get me started on the extraneous “\n”s in his Puppet manifests.

Suddenly, Octopress

I’ve learnt a lot about Linux over the past couple of years, mostly while working for LMAX, and I thought I should write about some interesting things.

Octopress seems like a brilliant platform for writing a nice technical blog. Coming soon.

1

$ sudo make me a sandwich