adrift in the sea of experience

Sunday, January 31, 2010

On the emergence of ubiquitous computing

I just read this post about how the iPhone and new iPad herald the era of "new world computing":
In the New World, computers are task-centric. We are reading email, browsing the web, playing a game, but not all at once. Applications are sandboxed, then moats dug around the sandboxes, and then barbed wire placed around the moats. As a direct result, New World computers do not need virus scanners, their batteries last longer, and they rarely crash, but their users have lost a degree of freedom. New World computers have unprecedented ease of use, and benefit from decades of research into human-computer interaction. They are immediately understandable, fast, stable, and laser-focused on the 80% of the famous 80/20 rule.

It is an interesting post, but I don't believe that the lack of multi-tasking and other freedoms is a necessity for "new world computing". If you can make a slick UI for switching between tasks, then you can also make a slick UI for switching between tasks that continue to run in the background.

These limitations are just engineering trade-offs that had to be made to give us an early peek at ubiquitous computing (which is by the way the real term for "new world computing" and was already a research topic long before I went to college). I seriously doubt the next generation of these devices will have the same limitations.

You can wave your hands and talk about how it's all task oriented now all you want, but in the end multi-tasking is a necessity even if only for running a chat client 24/7. And that's exactly what I do with my old and clunky N95 smartphone.

Setting up dropbox on a headless linux system

I have been using dropbox for a while to synchronize files between different computers. It has some pretty impressive bullet points:

  • Seamless syncing. You just put files in the dropbox folder, and they are automatically synchronized to your other computers. No firewall issues. In fact, the only problem I have is at work where dropbox is explicitly blocked. >:-(
  • Easy file sharing over the internet. Just put a file in your public folder, right click, copy public link. You can even host a website on dropbox this way.
  • Cross-platform. It works on linux, windows, OS X and even iPhone.
  • You can access the revision history of your files so it works pretty well as an on-line backup service, even if you delete files by accident.
  • It's completely free if you don't need more than 2GB storage and 30 days of revision history.

I have set up dropbox on my NAS so that I can synchronize my dropbox to a ZFS file system. This way I can combine the advantages of dropbox with the advantages of my NAS:

  • I get to keep snapshots indefinitely, with disk space being my only limitation.
  • I protect my data even if the dropbox service fails disastrously, e.g. because of security breach. Think file deletions being synced to all your computers.
  • I can free space on my dropbox account by moving files on the NAS out of the dropbox folder, yet still keep them safe through my NAS snapshot+backup policy.

Dropbox is targetted at GUI environments, but can still be installed on a headless linux system as described on this wiki page. However, the wiki page did not describe how to change the dropbox folder. I needed this to point dropbox to a folder on my NAS storage pool. It took some minor reverse engineering of the dropbox settings file, but I successfully created a script to do exactly that. I've also added a link and instructions on the wiki on how to use it.

Wednesday, January 27, 2010

Using MEF for classes which take configuration values

We're using the Managed Extensibility Framework (part of the upcoming .NET 4.0) at work for a new project.

I have a few pre-MEF classes which take strings, integers other primitive data types in their constructor. For example, consider the following C# class which tracks recently used resources (e.g. the last files opened by the application) by saving them in a file:

public class RecentlyUsedTracker : IRecentlyUsedTracker
   private readonly string file;
   private readonly int maxItems;

   // constructor
   public RecentlyUsedTracker(string recentlyUsedFile, int maxItems)
      this.file = recentlyUsedFile;
      this.maxItems = maxItems;

   // Marks the given resource as recently used.
   public void Touch(string resource)

The above export doesn't work, because MEF cannot instantiate this class. Obviously it cannot know which string and integer to use as arguments for the constructor.

You can still do it by adding [Import("somename")] attributes to the constructor declaration like this:

    // constructor
    public RecentlyUsedTracker(
       [Import("RecentlyUsedTracker.File")] string recentlyUsedFile,
       [Import("RecentlyUsedTracker.MaxItems")] int maxItems)
However, that makes it much more complex to set up the container. Each configuration value has to be explicitly added to the MEF container with ComposeExportedValue as shown below. Blergh! (correction: see update below!)
var catalog = ... some catalog ...
var container = new CompositionContainer(catalog);
container.ComposeExportedValue<string>("RecentlyUsedTracker.File", @"c:\recentlyused.txt");
container.ComposeExportedValue<int>("RecentlyUsedTracker.MaxItems", 5);

My next idea was then to do something like this for the constructor:

   public RecentlyUsedTracker([Import] IConfigurationProvider configurationProvider)
      this.recentlyUsedFile = configurationFile.GetValue<string>("recentlyUsedTracker.File");
      this.maxItems = configurationProvider.GetValue<int>("recentlyUsedTracker.maxItems");

I'm a bit worried about the fact that I'm importing the configurationProvider object only to use it briefly in the constructor. It's also annoying that I need to mock this service in my unit tests, instead of just passing a value. I've asked on the MEF forum if there is a better way.

Update: turns out there is a better way. My first attempt (adding attributes to constructor arguments) is just fine. It's just explicitly adding configuration values to the container that was the bad idea. As Glenn Block suggested, you can just export the configuration values via properties like this:

public class RecentlyUsedTrackerConfiguration
  public RecentlyUsedTrackerConfiguration()
     //set values here

  public string File {get;set;}

  public int MaxItems {get;set;}

Tuesday, January 26, 2010

Building a NAS, part 6: testing ZFS checksumming

Let's take a look at how ZFS protects data. I plugged in a spare external disk, created two small 1GB partitions on it with fdisk, and set up a ZFS pool for testing:

fdisk /dev/sdc # set up two 1GB partitions
zpool create testpool mirror /dev/sdc1 /dev/sdc2
zfs create testpool/testfs
Note that this is just a test set-up. Normally you should definitely use two separate disks to get the full benefit of mirroring. Also, it doesn't really make sense to slice up disks into partitions.

Smashing bits

Let's create a test file which fills the file system and make a note of the sha1 fingerprint:
cd /testpool/testfs
dd if=/dev/urandom of=testfile bs=1M count=920
# prints a sha1 fingerprint for the file
sha1sum /testpool/testfs/testfile
Now comes the fun part. With a small (and very dangerous) python script, we can corrupt one of the devices by writing some junk data at regular intervals:
openedDevice = open('/dev/sdc1', 'w+b')
interval = 10000000
while (True):,1)
  print str(openedDevice.tell())
When we reread the file after the corruption, ZFS will transparently pick the pieces of data on the healthy disks. Note that in this case the file cannot not be cached in memory because it is larger than the available system memory.
# still prints the correct fingerprint!
sha1sum /testpool/testfs/testfile
Strangely enough, running zpool status testpool doesn't report any errors at this point. I have send a mail to the zfs-fuse mailing list to ask whether this is normal.

To detect and fix the errors, we have to run this simple command:

zpool scrub testpool
# shows progress and results of the scrub
zpool status testpool
To protect against bit rot on consumer grade disks, the recommendation is to run a scrub once a week. In a future post I'll explore how to do that automatically, including some kind of reporting so that I know when a disk is in trouble.

Sunday, January 24, 2010

Building a NAS, part 5: minimizing power consumption

My plan is to let my NAS run 24/7 if the impact on my electricity bill is acceptable. To measure power consumption, I have purchased a power consumption meter that you can plug in between a wall socket and some device. It is one of those tools that can provide hours of quality geek entertainment. So much devices to measure around the house, so little time! :-)

The following table shows the passive power consumption of my NAS after each round of power saving measures. With "passive", I mean that the NAS was not doing anything useful like reading from or writing to the ZFS file systems.

Configurationpower consumption (watts)
under-clocked CPU69
removed AGP video card59
removed PATA CD-ROM drive57
removed unused eSATA RAID1 PCI card55
2 storage disks in standby47

Take it slow

The Athlon CPU and/or the mother board in this box is apparently too old to support dynamic cpu frequency scaling. It also doesn't seem to support the AMD power saving mode which you can control with the athcool package. Instead, I had to get down and dirty with the BIOS settings and set the CPU multiplier to the minimum value. According to cat /proc/cpuinfo, this slowed down the CPU from 2GHz to 1GHz. Profit: -11 watts.

Bare necessities

I had to connect a video card and a CD-ROM drive to install debian Lenny. Now that the server is running and connected to the network, I can throw those out again and manage the system remotely. Fortunately the BIOS supports booting without a video card. I also removed a PCI card for eSATA RAID1 which I initially thought I would need. Profit: -14 watts

Spin down those storage disks

With the hdparm command we can inspect the power state of a disk and configure stand-by mode:

aptitude install hdparm
hdparm -C /dev/sda # prints "active/idle"
# now tell disk to go to stand-by whenever not used for 2min
# note the very strange S-value to time mapping; consult man hdparm!

hdparm -S 25 /dev/sda
sleep 120
hdparm -C /dev/sda # should print "stand-by"
To make the power saving configuration permanent, I added this to /etc/hdparm.conf. Note the use of /dev/disk/by-id to keep the settings correct even if we start changing the NAS hardware:
/dev/disk/by-id/scsi-SATA_ST3500418AS_9VM7RWGV {
spindown_time = 25

/dev/disk/by-id/scsi-SATA_ST3500418AS_9VM7SHA5 {
spindown_time = 25
Profit: -8 watts

Leave the system disk alone

It is much harder to get power savings for the system disk, because it is used all the time for logging and by a bunch of daemons. Trying to put this disk in stand-by will just cause it to frequently spin down and up again.

Instead, I'm using a disk recovered from a dead laptop as the system drive. This requires a cheap 2.5" to 3.5" IDE converter cable. Such a drive is already extremely efficient; the potential savings of putting it in stand-by are negligible (~1 watt).

Cost before and after

( update: corrected kwh cost, my original estimate was about 50% of actual cost because I based it solely on Electrabel's power generation cost, and forgot the distribution costs on my bill - thank you Peter!)

At 80 watt, energy consumption per year was 80 watt * (24 * 365) hours = 700.8 kwh. At an average 0.16 euro/kwh that will cost 112 euros. I have reduced that to 66 euros. This goes to show that a 20 euro energy consumption meter can yield a return on investment rather fast.

Another lesson I'll be remembering from these calculations: for my current contract, I can estimate the yearly cost for the continuous consumption of a device as 1.4 euro per watt.

Saturday, January 23, 2010

Building a NAS, part 4: ZFS on linux with FUSE

I decided to use the ZFS file system for my NAS. Although licensing issues prevent it from being ported to the linux kernel, there is a ZFS-FUSE project which has ported ZFS to run in userspace via FUSE.

ZFS is a mature file system (and tool set) which does device pooling, redundant storage, checksumming, snapshots and copy-on-write clones. It also has a very cool deduplication feature where you can configure the file system to look for identical chunks of data and store those only once. Nice!

Getting ZFS-FUSE on Debian Lenny

We'll install some tools, compile and manually start the zfs-fuse daemon. Note that I use the latest source from the "official" repository here, not the last stable release.

aptitude install git-core libaio-dev libattr1-dev libacl1-dev libz-dev libfuse-dev libfuse2 scons libssl-dev
git clone zfs-official
cd zfs-official/src
scons install

At the time of writing, the "scons install" command doesn't seem to install the debian init script. Also, the debian init script which is part of the source has a small error. We'll take care of that manually:
cd ../debian
nano zfs-fuse.init
# fix the line "DAEMON=/usr/sbin/zfs-fuse"
# it should be "DAEMON=/usr/local/sbin/zfs-fuse"

cp zfs-fuse.default /etc/default/zfs-fuse
cp zfs-fuse.init /etc/init.d/zfs-fuse
chmod +x /etc/init.d/zfs-fuse
aptitude install sysv-rc-conf
# use arrows to scroll down to zfs-fuse
# use arrows and space to enable run levels 2,3,4,5
# use q to quit

Setting up a ZFS storage pool and file systems

Currently I have two new 500GB disks available for storage. My first plan was to split each disk in two partitions to build a "safe" storage pool (mirrored over two partitions) and a "bulk" storage pool (no redundancy, striped over two partitions). However, a recurring theme in the ZFS Best Bractices Guide is that you should not slice up your disks if you can avoid it. Therefore, I'll keep things simple and just create one big 500GB pool of mirrored storage.

# start the zfs daemon
zpool create nas-pool mirror \
/dev/disk/by-id/scsi-SATA_ST3500418AS_9VM7RWGV \
zpool status
I will however, still create two separate file systems in this pool for "archive" and "bulk" storage. This makes it easy to have different backup policies for each data set.
zfs create nas-pool/archive
zfs create nas-pool/bulk
zfs list
zfs mount -a
Because ZFS is designed to handle storage pools with potentially thousands or more file systems, you don't have to manually edit /etc/fstab to set up mount points. The shown mount command will automatically mount all available ZFS file systems as /pool-name/file-system-name. This is also what the init script does.

Exposing the ZFS file systems on the network via samba

First we'll set up a "nasusers" group which has read/write access to the ZFS file system:

# create nasusers group and add a user to it
groupadd nasusers
usermod -a -G nasusers wim

# give nasusers read/write access
cd /nas-pool
chmod 2770 archive
chmod 2770 bulk
chgrp nasusers archive
chgrp nasusers bulk
Now give those users a samba password:
smbpasswd -a wim
Add a section like this to /etc/samba/smb.conf like this for each folder to expose:
valid users= @nasusers
Finally, restart samba:
/etc/init.d/samba restart
Now the ZFS file systems should be available on the network, and users can start copying there stuff in there. In a future post we'll explore how to leverage some of those advanced ZFS features.

Friday, January 22, 2010

Building a NAS, part 3: filesystem doubts

I had planned to demonstrate the creation and administration of a raid1 BTRFS filesystem in this post, but while playing around with BTRFS I ran into a few snags:

  • I was able to mount a normal BTRFS filesystem spanning two devices, but not one in raid1 mode. Then I discovered that I could only mount a raid1 BTRFS filesystem if I gave it a label.
  • I saw some unexplained mount failures on a multi-device FS which disappeared after I mounted once via another device.
  • Though data is checksummed, I couldn't find a way to detect checksum failures other than reading all files and watching output from the kernel with dmesg.
  • I confirmed that in a raid1 setup, BTRFS will still find the good copy after the data on one device is corrupted. However, I couldn't find a way to reliably repair the corruption other than reading and rewriting all files.
These issues didn't exactly give me confidence in the maturity of BTRFS. I knew it wasn't production ready, but I hoped it was close. I'm now pretty sure that it is not.

Perhaps more importantly, I also realized that BTRFS is the GPL-licensed answer to the more mature ZFS. ZFS is a Solaris filesystem developed by Sun. It can't be ported to run in the linux kernel because of licensing issues, hence the need for BTRFS. However, BTRFS is mainly sponsored by Oracle, and Oracle is buying Sun. If Oracle's motivation to sponsor BTRFS was to counter Sun's open source efforts, then the Sun deal takes away that motivation. On the other hand, if their motivation was truly to get a next-generation filesystem in linux, then they might as well relicense ZFS under the GPL. Chris Mason gave this vague comment when asked about ZFS and the Sun deal:

Chris: Sun has many interesting projects, and I’m looking forward to working with their R&D teams. We’re committed to continuing Btrfs development, and ZFS doesn’t change our long term plans in that area.
That's no rational explanation why Oracle would continue to support both projects, so I'm skeptical.

Meanwhile, that leaves my little build-a-NAS project stalled. I see these options:

  • use BTRFS anyway, even though it's not yet mature and its future seems unsure
  • run OpenSolaris and ZFS
  • run the FUSE port of ZFS on linux, which dodges the licensing issue by running the FS in userspace (presumably at the cost of performance)
  • use software raid + LVM on linux
I'm not sure at all which direction to take.

Sunday, January 17, 2010

Building a NAS, part 2: getting BTRFS on Lenny

update: after discovering that BTRFS isn't as mature as I hoped, I switched to ZFS-FUSE. You might want to read my post on setting up ZFS instead.

The debian "lenny" release comes with version 2.6.26-2 of the linux kernel. This kernel does not yet have support for BTRFS, so we'll download, compile and install the latest stable kernel release.
# install some required packages as root
aptitude install install bzip2 fakeroot kernel-package libncurses5-dev zlib1g-dev

# download and extract linux kernel
tar -xvjf linux-
cd linux-

# copy existing kernel configuration from /boot
cp /boot/config-2.6.26-2-686 .config

# edit kernel configuration (navigate with arrows, toggle options with space)
# - under "File Systems, enable "Btrfs filesystem (EXPERIMENTAL)"
# - under "Virtualization", disable "Linux hypervisor example code"
# You can also take this opportunity to optimize the kernel for your CPU
# architecture under "Processor type and features" - "Processor family"
# Examine the output of "cat /proc/cpuinfo" if you're not sure of your CPU.
make menuconfig

# build kernel (this takes a while, especially on old machines)
make-kpkg --rootcmd fakeroot --initrd linux-image linux-headers

# install new kernel packages and reboot
cd ..
dpkg -i linux-image-
dpkg -i linux-headers-

If everything went well, the system should boot up under the new kernel. If something goes wrong, you still have the option of booting under the old kernel by using the grub menu at startup.

Now we have a kernel with support for the btrfs filesystem, but still no userspace tools to use it. We'll download, compile and install the latest version of those tools:

aptitude install git-core uuid-dev e2fslibs-dev libacl1-dev
git clone git://
cd btrfs-progs-unstable
make install

Now we have mkfs.btrfs to create a BTRFS file system, and some other tools to manage such a filesystem. We'll start playing around with those in the next post.

Building a NAS, part 1: installing Lenny and introducing BTRFS

I'm building a NAS from spare parts. The basic system is an 8 year old PC. My operating system of choice is the latest stable release of debian gnu/linux, codename lenny. I would like to use the (still experimental) btrfs file system on this box for the following reasons which make it a good data-haven:

  • It can do raid1-like mirroring of data over multiple devices. This makes it resilient against failed disks.
  • It can make copy-on-write mountable snapshots of a volume. By regularly making snapshots (e.g. from a cron-job) you can keep old versions of your data without wasting any space on identical copies.

Of course, this still won't fully protect my data. There could be hardware or software errors that destroy the data on all disks. There could be circumstances that destroy all disks together, like fire or lightning. So I'll still have to make an off-site backup every now and then.

To install lenny, I downloaded the 40MB businesscard CD image. Despite its small download size, this CD still has a user-friendly graphical installer. After letting the installer do its thing, I still did the following:

  • ran "tasksel" to install "file server" related packages
  • ran "aptitude install openssh-server" to enable remote access

At this point I have a basic linux system that I can SSH into. Unfortunately "lenny" does not come with btrfs support. We'll fix that in the next post...

Monday, January 11, 2010

Registratierechten te betalen bij het kopen van een woning in Vlaanderen

This post is in Dutch because it is about law and taxes in the Flemish Region of Belgium.

Deze post hoort thuis in een serie over de aankoop van ons huis. Eerder in deze serie:
Beëindigen van een huurovereenkomst door de huurder in België

Wanneer je in het Vlaamse gewest vastgoed koopt, moet je daarop registratierechten betalen. Deze rechten zijn vastgelegd door het Wetboek der registratie-, hypotheek- en griffierechten. Artikel 44 geeft aan dat de registratierechten 10 procent bedragen:

Het recht bedraagt 10 ten honderd voor de verkoop, de ruiling en iedere overeenkomst tot overdracht onder bezwarende titel van eigendom of vruchtgebruik van onroerende goederen.

Vermindering tot 5 procent

Volgens artikel 53 wordt dit verminderd tot 5% voor kleine landeigendommen en bescheiden woningen. Dit is vastgoed waarvan het kadastraal inkomen lager is dan een zeker maximum, vastgelegd door een apart Koninklijk Besluit. (Ik vind spijtig genoeg geen link naar het KB.)

Dit staat ook wel bekend als klein beschrijf. Er zijn nog bijkomende criteria vastgelegd door artikels 54-61; zeer droge kost, en aangezien de woning die Elke en ik gaan kopen toch niet in aanmerking komt, ga ik hier niet verder op in gaan.

Vermindering op het bedrag waarop die 5 of 10 procent berekend wordt

De bovenstaande percentages worden berekend op de prijs overeengekomen tussen koper en verkoper. Volgens artikel 46bis kan echter een deel van dit bedrag vrijgesteld worden, en die vrijstellingen mag je dus van de prijs aftrekken voor je die 5 of 10 procent berekend.

Er wordt 15.000 euro vrijgesteld als je het vastgoed koopt als hoofdverblijfplaats:

De heffingsgrondslag ten aanzien van de verkopingen, zoals bepaald in de artikelen 45 en 46, wordt verminderd met 15.000 euro in geval van zuivere aankoop van de geheelheid volle eigendom van een tot bewoning aangewend of bestemd onroerend goed door een of meer natuurlijke personen om er hun hoofdverblijfplaats te vestigen.

Deze vrijstelling heeft nog extra voorwaarden:

  • je mag nog geen onroerend goed bezitten
  • je moet uitdrukkelijk vragen om deze korting te krijgen (!)
  • je moet er binnen de 2 jaar gaan wonen (5 jaar in het geval van bouwground)

Als je in aanmerking komt voor bovenstaande vermindering, wordt er nog 10.000 euro extra vrijgesteld als je voor de woning een lening aangaat (of zelfs 20.000 euro indien je in aanmerking komt voor de vermindering tot 5 procent):

Als met het oog op de financiering van een aankoop, vermeld in het eerste lid, een hypotheek wordt gevestigd op het aangekochte onroerend goed, wordt het bedrag van de vermindering van de heffingsgrondslag, vermeld in het eerste lid, verhoogd met hetzij 10.000 euro als op de aankoop het recht, vermeld in artikel 44, verschuldigd is, hetzij 20.000 euro als op de aankoop het recht, vermeld in artikel 53, verschuldigd is

In ons geval komen we in aanmerking voor beide verminderingen, dus we mogen 25.000 euro van de verkoopprijs aftrekken voordat we de 10 procent registratierechten berekenen. Dat "bespaart" ons dus 2500 euro.

Sunday, January 10, 2010

Beëindigen van een huurovereenkomst door de huurder in België

This post is in Dutch because it is about Belgian law and I live in the Flemish region. As far as I know the referred Belgian law is not published in English, so I had to pick another language.

Aangezien Elke en ik net een verkoopsovereenkomst voor een huis getekend hebben, hebben we de eerstvolgende maanden heel wat te doen. Een van die dingen is het opzeggen van de huur.

Onze huurovereenkomst is een typisch contract van 9 jaar. Zo'n contract wordt ook wel "een 3-6-9" genoemd in de volksmond omdat de huurwet bepalingen bevat waarin sprake is van driejarige periodes. De spelregels voor het vroegtijdig beeindigen van zo een overeenkomst worden vastgelegd door de Belgische wetgeving in het Burgerlijk Wetboek, "Regels betreffende de huurovereenkomsten met betrekking tot de hoofdverblijfplaats van de huurder in het bijzonder" en het bijhorende Koninklijk besluit. Een langere maar meer leesbare interpretatie van deze wetteksten is te vinden in de brochure "De Huurwet" (10e editie juli 2008) van de Vlaamse overheid.


Een contract van 9 jaar kan altijd door de huurder opgezegd worden, maar de opzeg moet wel minstens 3 maanden op voorhand gebeuren:

Art. 3, § 5 De huurder kan de huurovereenkomst op ieder tijdstip beëindigen met inachtneming van een opzeggingstermijn van drie maanden.

Die termijn gaat in vanaf de eerste dag van de maand die volgt op de opzegging:

Art. 3, § 9. In alle gevallen waarin een opzegging te allen tijde kan worden gedaan, neemt de opzeggingstermijn een aanvang de eerste dag van de maand die volgt op de maand tijdens welke de opzegging wordt gedaan

Vreemd genoeg staat er iets anders in het KB, bijlage "HUUROVEREENKOMSTEN VOOR WONINGEN GELEGEN IN HET VLAAMS GEWEST". Het lijkt erop dat de woorden "de maand die volgt" verdwenen zijn (benadrukking toegevoegd door mezelf):

In alle gevallen waarin de opzegging te allen tijde kan worden gedaan, neemt de opzeggingstermijn een aanvang de eerste dag van de maand tijdens welke de opzegging wordt gedaan.


Als de huurder het huurcontract opzegt tijdens de eerste 3 jaar, dan heeft de verhuurder recht op een opzeggingsvergoeding. Ook uit Art. 3, § 5:

Indien de huurder de huurovereenkomst evenwel beëindigt tijdens de eerste driejarige periode, heeft de verhuurder recht op een vergoeding. Die vergoeding is gelijk aan drie maanden, twee maanden of één maand huur naargelang de huurovereenkomst een einde neemt gedurende het eerste, het tweede of het derde jaar.