Mind in the Water

Problems encountered when using MEF as a dependency injection container

2011-05-10T02:00:00.000+02:00

When we started using MEF as a dependency injection container, I figured the trade-off was a bit like this. Good: Already part of the .NET 4 framework, dynamic discovery of components for extensibility. Bad: missing some advanced features like AOP and parametrized construction. Glenn Block posted about this in Should I use MEF for my general IoC needs?.

Since we didn't need those features, MEF seemed like a good choice. But as it turns out, there is a more subtle problem when using MEF as a general purpose dependency injection container. Consider the following example:

   public class Program 
   {
      static void Main(string[] args)
      {
         var container = new CompositionContainer(
            new AssemblyCatalog(Assembly.GetExecutingAssembly()));
         var a = container.GetExportedValue<A>();
      }
   }

   [Export]
   public class A
   {
      private readonly B b;

      [ImportingConstructor]
      public A(B b)
      {
         this.b = b;
      }
   }

   [Export]
   public class B
   {
      private readonly C c;

      [ImportingConstructor]
      public B(C c)
      {
         this.c = c;
      }
   }

   public class C
   {
   }

The export attribute is missing on the C class in my example. Since class A indirectly depends on class C, we get an error when we try to get an A instance from the container:

System.ComponentModel.Composition.ImportCardinalityMismatchException was unhandled
  Message=No valid exports were found that match the constraint '((exportDefinition.ContractName == "DITest.A") AndAlso (exportDefinition.Metadata.ContainsKey("ExportTypeIdentity") AndAlso "DITest.A".Equals(exportDefinition.Metadata.get_Item("ExportTypeIdentity"))))', invalid exports may have been rejected.
  Source=System.ComponentModel.Composition
  StackTrace: ...

So the C export is missing, but surprisingly the error message is complaining about the A class! This is because of "stable composition". Basically, whenever a dependency for a certain part is missing, MEF will resiliently attempt to do the composition without that part. For an example of how that can be useful, see Implementing Optional Exports with MEF Stable Composition.

Useful as it may be, stable composition comes at a price. Since MEF can't tell the difference between critical and non-critical parts, the missing dependency error may cascade upward until you get a mysterious ImportCardinalityMismatchException like the one above.

The MEF documentation on Diagnosing Composition Problems acknowledges this and provides some hints on how to debug such problems. Still, for large compositions the process is far from pleasant. Things are even worse if you have some circular dependencies.

Perhaps it would be better to use Autofac to do the core application composition. It doesn't attempt dynamic discovery or stable composition, so the error messages point straight at the missing dependencies. And with the MEF integration, you can still use MEF for the parts of the application where you do need dynamic discovery and stable composition.

DownMarker, a Markdown editor/viewer

2011-05-07T02:09:00.000+02:00

I've released version 0.1 of DownMarker, a small desktop application that can be used to to view, navigate and edit a set of interlinked markdown files:

In case you don't know Markdown, it is a "easy-to-read, easy-to-write plain text format" intended to be converted to HTML. I first encountered it as the format used at Stackoverflow for editing questions and answers. In fact, DownMarker is based on MarkdownSharp, a library generously released as open source by the stackoverflow team.

Currently some things are still missing, but my hope is that DownMarker will work well for creating light-weight wikis inside your version controlled projects, or anything else that can store files. Let me know if you find it useful!

MEF attribute-less registration

2011-03-01T01:29:00.001+01:00

The Managed Extensibility Framework has shipped in .NET4, but the work hasn't stopped: the recent MEF2-Preview3 release added some interesting stuff. Let's take a look at attribute-less registration.

First, a recap of the existing attribute-based registration mechanism (or "Attributed Programming Model" in MEF-parlance):

A composition container can take different kinds of export providers. The diagram shows the catalog-based export provider. TypeCatalog is the most important catalog implementation: it is based on a list of types, which it inspects via reflection for MEF attributes. The other implementations are there for convenience, to quickly build type catalogs out of an assembly (AssemblyCatalog), an entire directory (DirectoryCatalog), or other catalogs (AggregateCatalog).

Here is an illustration of the above with a quick code example featuring some dummy types. (The CatalogExportProvider is not visible because the CompositionContainer can conveniently construct it for us if we directly pass it a catalog.)

interface IFoo { }
   interface IBar { }

   [Export(typeof(IFoo))]
   class Foo : IFoo
   {
      [Import(typeof(IBar))]
      public IBar Bar { get; set; }
   }

   [Export(typeof(IBar))]
   public class Bar : IBar { }

   class Program
   {
      static void Main(string[] args)
      {
         var catalog = new TypeCatalog(typeof(Foo), typeof(Bar));
         var container = new CompositionContainer(catalog);
         var foo = container.GetExportedValue<IFoo>();
      }
   }

For an attribute-free alternative to the above, you might expect that the new MEF release would contain an alternative ExportProvider or perhaps a new ComposablePartCatalog implementation, which would take its information from configuration in code instead of reflection. Surprisingly, that's not how it was done!

In reality, the attribute-less registration is implemented by adding information right above the reflection level! We essentially inject fake reflection information into our catalogs to simulate the presence of attributes. The attribute-free version of the above example looks like this:

interface IFoo { }
   interface IBar { }

   class Foo : IFoo 
   {
      public IBar Bar { get; set; }
   }

   class Bar : IBar { }

   class Program
   {
      static void Main(string[] args)
      {
         var registration = new RegistrationBuilder();

         registration.OfType<Foo>()
            .ImportProperty<IBar>(property => property.Name == "Bar")
            .Export<IFoo>();

         registration.OfType<Bar>()
            .Export<IBar>();

         var catalog = new TypeCatalog(
             types: new Type[] { typeof(Foo), typeof(Bar) },
             reflectionContext: registration);
         var container = new CompositionContainer(catalog);
         var foo = container.GetExportedValue<IFoo>();
      }
   }

Note how the TypeCatalog takes a new ReflectionContext parameter; that's were the reflection info is injected. An interesting result of this approach is that you can leverage the existing catalogs to register types in bulk. For example, the following example will export all the types which implement IPlugin in a given directory, without using any attributes:

var registration = new RegistrationBuilder();
registration.Implements<IPlugin>().Export<IPlugin>();
var catalog = new DirectoryCatalog("./plugins", "*.dll", registration);

This is just what I've gleaned from a quick peek at the unit tests in the new MEF release. There is much more to the RegistrationBuilder API which I haven't shown here. I'm sure there will be more documentation on the MEF codeplex site shortly. Also, keep in mind that the APIs in preview releases are subject to change.

UPDATE: The MEF product manager, Hamilton Verissimo (aka "hammet" and "haveriss"), has blogged about MEF's convention model.

Executing visual studio 2010 unit tests without installing visual studio

2011-02-25T19:50:00.005+01:00

In a previous post I already explained how to get your unit tests running on a build machine with mstest.exe, without having to install Visual Studio 2008. We recently upgraded to Visual Studio 2010 so I had to repeat the exercise.

This time I wrote a batch script which does most of the work. Just run the script on a machine where Visual Studio 2010 is installed, and it will create a "mstest" folder with the necessary binaries and registry entries. Copy the folder to your build machine, prepare the registry by executing mstest.reg and you're good to run mstest.exe like this:

mstest /noisolation /testcontainer:"path\to\testproject.dll"

This will return an error code if any of the tests fail.

As an anonymous commenter on my previous post pointed out, you should be careful of the license implications. In our case it is most likely OK because we have a site license. But even so it can be useful to avoid a full Visual Studio install: it saves a lot of disk space, and you don't have to spend time babysitting the installation. Or multiple installations, if you have a cluster of build machines!

update: thanks to Frederic Torres for pointing out that the script doesn't work on a 64-bit Windows, and suggesting a fix! ~~It should work now.~~ Apparently there are still problems with 64-bit machines. But I don't have a 64-bit machine without VS2010 to test with, so I'll have to leave it as an exercise for the reader...

bisect your source code history to find the problematic revision

2010-12-01T01:41:00.002+01:00

I am working on a little application to edit Markdown documents with instant preview. (For those of you who aren't into building things from source, here's a windows installer to put a downmarker shortcut in your start menu.) I like to use both windows and linux so I try to make sure that it runs on both Microsoft.NET and Mono.

When I last tested my application on Mono, I discovered a very annoying problem: each time I tried to type something in a markdown document, a new nautilus file browser window would be launched! I had absolutely no idea why this would happen or where to start debugging. To make matters worse, I hadn't tested on Mono for about 40 revisions so there were a lot of possible changes that might have introduced the problem.

I have the source code history in a mercurial repository, so I decided to give the "bisect" feature a try. I started by marking the latest revision as "known bad":

hg bisect --bad

I also remembered making some mono-specific bug fixes, and the problem didn't exist at that point. So I grepped the output of hg log for commit messages containing the word "mono" and marked the revision as good:

hg bisect --good 627d6

At this point the bisect command has a revision range to work with. It will automatically update your working copy to a revision right in the middle of that range. You just have to rebuild your application, verify whether the problem is still there, and mark the revision as good or bad. (It is also possible to do this test automatically with a script.) The bisect command will then automatically cut the revision range to investigate in half again, and select yet another revision in the middle, etcetera.

Sure enough, after about 5 or 6 tests I got this message:

The first bad revision is:
changeset: 42:c4bbabe79dde
user: wcoenen
date: Sun Nov 07 23:31:32 2010 +0100
summary: Links are now handled by opening .md or by external application.

Aha! Looks like I added some code to handle link clicks by passing the URL to the OS (to open it with a suitable external application). And obviously this code is now being triggered erroneously on mono whenever the document is updated. Thank you bisect!

git error: error setting certificate verify locations

2010-11-24T11:48:00.000+01:00

I was trying to clone a repository from github when I ran into this error:

Cloning into docu...
error: error setting certificate verify locations:
CAfile: \bin\curl-ca-bundle.crt
CApath: none
while accessing https://github.com/jagregory/docu/info/refs

If you google around, many people "solve" this by disabling the SSL certificate check entirely. Obviously there is a reason for that check, so disabling it is not quite the right solution! It turns out that there is mistake in the gitconfig file that comes with msgit setup (I have Git-1.7.2.3 installed). The right fix is to change the sslCAinfo setting in "c:\program files\git\etc\gitconf" to this:

sslCAinfo = c:\\program files\\git\\bin\\curl-ca-bundle.crt

Mercurial and subversion subrepos

2010-11-23T11:59:00.000+01:00

It is not yet mentioned in the Mercurial book, but Mercurial has a subrepos feature to pull code from other repositories into your project as a nested repository. It is a bit similar to SVN externals and git submodules.

Better yet, it also works with subversion! There are still some bugs to be worked out though: you better not move your SVN subrepo around in your mercurial repository. For all the ugly details, see my bug report.

An equivalent of .bashrc for the windows cmd.exe shell

2010-11-19T12:45:00.002+01:00

Tired of the "cd" command in windows refusing to navigate to folders on other drives? Put this in the [HKEY_CURRENT_USER\Software\Microsoft\Command Processor] AutoRun key:

c:\cmdauto.cmd

This will cause the given script to be executed each time a cmd.exe window is opened. I found out about that thanks to this superuser.com post.

Now put this in c:\cmdauto.cmd

@echo off
doskey cd=pushd $*

The doskey command creates aliases, here overriding the behavior of "cd".

ZFS-Fuse reliability report

2010-06-23T18:42:00.000+02:00

A few weeks ago, I got this question in the comments on my last NAS post:

Since you've been using ZFS-fuse for a time now, can you report here or blog about its stability? Has the daemon crashed on you, while taking a snapshot, or while "scrub"-ing?

How much data have you passed in?

First, note that the version of ZFS-FUSE which I am using is not a particular release, but this revision from the official git repository. I have not bothered to update my ZFS binaries since January, because I didn't encounter any problems.

The "zfs list" output for my pool shows that I am using 352 GiB. The pool consists of two 465 GiB disks in a mirror setup.

NAME               USED  AVAIL  REFER  MOUNTPOINT
nas-pool           352G   105G    24K  /nas-pool
nas-pool/archive   118G   105G  71.1G  /nas-pool/archive
nas-pool/bulk      234G   105G   228G  /nas-pool/bulk

The pool is scrubbed by a cron script each sunday night. I have not seen any crashes/hangs during scrubbing. The scrubs have not detected any errors so far.

"archive" is snapshotted every night. "bulk" is snapshotted" only on sunday nights. The pool contains more than a hundred snapshots. Again, I have not seen any crashes or hangs.

Then there is read/write activity: I use it to automatically archive my emails from Gmail (nightly), sync with my dropbox folder (continuously), and to store my mp3s, photos, videos, downloads and backups when I need to. When reading or writing NAS files, I am usually working from my laptop over WiFi which is a speed bottleneck, so this probably doesn't really stress the file system. On the other hand, I do regularly make a full off-line backup of the "archive" filesystem by hooking up an external USB disk to the NAS. I haven't seen ZFS crash during any of this.

From this, I conclude that ZFS-Fuse is pretty stable.

Exponential growth has limits

2010-05-29T02:49:00.000+02:00

I saw this comment in the thread of a slashdot post about Wall Street today:

But everything thus far shows us that perpetual growth is possible. Technology is a wonderful thing - each year we're able to do more with less.
That's not to say that a lot of what goes on in the market isn't pure, unadulterated bullshit, but real, honest-to-goodness "growth" won't stop until technology does.

This is a textbook example of the "cornucopian" view on economics. The implicit assumption here is that there are no limits to technological improvements, or that those limits are extremely far in our future. I think that this assumption is flat out wrong and dangerous. It is these sorts of ideas which have lead large parts of the global economy to eerily resemble Ponzi Schemes.

I wrote the following post in reply:

There are limits to exponential growth. (And make no mistake, growth expressed as a fixed percentage per year is exponential). Technology can push the limits closer to what the laws of physics allow, but technology cannot change the laws of physics.
Let's look at some numbers to drive the point home. Our global energy consumption in 2008 was estimated to be 474 exajoules.
The total energy received by the earth from the sun during a year is about 5 million exajoules, a fraction of which reaches the surface. 5 million is much more than 474. But at a seemingly modest 2% per year growth rate (as it was between 1980 and 2006), our energy consumption will match those 5 million exajoules in less than 500 years!
Think about that: if energy consumption growth continues at the current pace, then in 500 years we'll either be using ALL solar energy received by the earth (leaving none for the biosphere), or we'll have figured out some magic technology to produce 5 million exajoules of energy per year. Assuming the magic technology, where are we going to get rid of all that extra heat? It would effectively be like having a second sun on earth, cooking us in place.
Granted, you did say "do more with less". So lets say energy consumption will stay constant in the future, and instead we'll derive 2% more "value" from the same energy each year. Now you run into a new problem. No matter how you define "value", you run into physical limits. If you define value as "amount of mass lifted out of the earth's gravity field", then the hard efficiency limit is a minimum of 60 megajoules per kg. If you define value as "amount of computation", then again there are limits given by the laws of physics.
Exponential growth is counterintuitive. No matter how far you push the limits (e.g. by colonizing the entire galaxy or inventing game-changing technology), exponential growth will hit its limits much faster than you think. We're talking about growth with a fixed doubling period here.
Finally, I'd argue that we are already experiencing the end of exponential growth today. After decades of growth, in 2004 global oil production reached a plateau [theoildrum.com]. It's not a coincidence that we experienced a major financial crash and recession soon after that. The era of "perpetual growth" is over. The next era will be that of the "zero-sum game" at best.

On a related note, after writing the above post I stumbled on this excellent series of videos of a lecture by Dr. Albert A. Bartlett where he shows with some very simple calculations and examples that "steady growth" is in fact always unsustainable. We have to find a way to get our civilization to work with 0% growth. The alternative is total collapse, probably within decades.

Building a dependency injection container in 30 lines

2010-04-03T04:16:00.004+02:00

After reading this article by Josh Smith explaining the Service Locator pattern, I commented that direct dependency injection might be a better idea. Mark Seemann has a good write up about why Service Locator is an anti-Pattern, and I'm inclined to agree with him.

When Josh replied that injecting dependencies with constructor arguments doesn't really solve the problem of dependency creation, I was tempted to reply by enumerating all the .NET dependency injection frameworks that exist for exactly this purpose.

But then I realized that Josh had demonstrated the Service Locator pattern without using any framework. Instead, his article has a ServiceContainer class of about 30 lines. Service Locator has many disadvantages, but apparently it can be quite lightweight!

This then lead me to wonder if the same could be done for creating a dependency injection framework. Ayende has actually already demonstrated that you can create a primitive one in 15 lines, but I was thinking of something that could be used with a more friendly Ninject-esque syntax like this:

var container = new Container();
container.Bind<App, App>();
container.Bind<IFoo, Foo>();
container.Bind<IBar, Bar>();

var app = container.Pull<App>();
app.Run();

As it turns out, implementing a bare bones container which can do that is really not that hard. It also has the advantage that it takes care of the dependencies of the dependencies etcetera, something which Josh's sample doesn't seem to do. (Disclaimer: I didn't really test this for anything but the plain vanilla use case, no error conditions were considered.)

public class Container
{
    private readonly Dictionary<Type, Type> contractToClassMap = new Dictionary<Type, Type>();
    private readonly Dictionary<Type, object> contractToInstanceMap = new Dictionary<Type, object>();

    public void Bind<TContract, TClass>() where TClass : class, TContract
    {
        this.contractToClassMap[typeof(TContract)] = typeof(TClass);
    }

    public TContract Pull<TContract>()
    {
        return (TContract)Pull(typeof(TContract));
    }

    public object Pull(Type contract)
    {
        object instance;
        this.contractToInstanceMap.TryGetValue(contract, out instance);
        if (instance == null)
        {
            var constructor = contractToClassMap[contract].GetConstructors()[0];
            var args = 
                from parameter in constructor.GetParameters() 
                select Pull(parameter.ParameterType);
            instance = constructor.Invoke(args.ToArray());
            this.contractToInstanceMap[contract] = instance;
        }
        return instance;
    }
}

Adding some compiler verification to PropertyChanged events

2010-04-01T12:57:00.001+02:00

I've been playing around with Windows Presentation Foundation and the Model-View-ViewModel pattern in the past week. Josh Smith's introductory article does a good job explaining how the MVVM pattern can be used used in WPF.

One aspect of the pattern is that your view model needs to provide change notifications, for example by implementing the INotifyPropertyChanged interface. To avoid repeating the same code, it might be a good idea to implement this in a shared base class:

public abstract class ViewModelBase : INotifyPropertyChanged
   {
      protected virtual void OnPropertyChanged(string propertyName)
      {
         if (PropertyChanged != null)
         {
            PropertyChanged(this, new PropertyChangedEventArgs(propertyName));
         }
      }

      public event PropertyChangedEventHandler PropertyChanged;
   }

There is a problem with this though: the caller of OnPropertyChanged can pass any string. To constrain this to strings that match a property name, we can use reflection to check whether such a property indeed exists. The sample in Josh Smith's introductory article takes this approach. That way, passing an invalid string will at least raise an exception at run-time.

However, we can still do one better and catch such errors at compile time, which is a huge advantage during refactorings. The solution involves two advanced C# tricks. The first trick makes use of the fact that the C# compiler can convert lambdas into expression trees, which can then be inspected to extract the name of a property:

Foo foo = new Foo();
    string propertyName = GetPropertyNameFromExpression<Foo,int>(x => x.Bar);
    Debug.Assert(propertyName == "Bar");

The GetPropertyNameFromExpression method is implemented like this:

private string GetPropertyNameFromExpression<TClass,TProperty>(
         Expression<Func<TClass, TProperty>> expression)
      {
         var memberExpression = expression.Body as MemberExpression;
         if (memberExpression == null)
            throw new ArgumentException(String.Format(
               "'{0}' is not a member expression", expression.Body));
         return memberExpression.Member.Name;
      }

But how do we get that TClass type parameter in our base class? That's our second trick: as it turns out, it is possible for a base class to have a type parameter representing its derived classes. The resulting base class looks like this:

public abstract class ViewModelBase<TDerived> : INotifyPropertyChanged
       where TDerived : ViewModelBase<TDerived>
    {

        private string GetPropertyNameFromExpression<TPropertyType>(
           Expression<Func<TDerived, TPropertyType>> expression)
        {
            var memberExpression = expression.Body as MemberExpression;
            if (memberExpression == null)
                throw new ArgumentException(String.Format(
                    "'{0}' is not a member expression", expression.Body));
            return memberExpression.Member.Name;
        }

        /// <summary>
        /// Triggers the <see cref="PropertyChanged"/> event for the property used in
        /// <paramref name="expression"/>
        /// </summary>
        /// <param name="expression">
        /// A simple member expression which uses the property to trigger the event for, e.g.
        /// <c>x => x.Foo</c> will raise the event for the property "Foo".
        /// </param>
        protected void OnPropertyChanged<TPropertyType>(
            Expression<Func<TDerived,TPropertyType>> expression)
        {
            string name = GetPropertyNameFromExpression(expression);
            if (PropertyChanged != null)
                PropertyChanged(this, new PropertyChangedEventArgs(name));
        }

        public event PropertyChangedEventHandler PropertyChanged;
    }

The viewmodel implementations then look like this:

public class FooViewModel : ViewModelBase<FooViewModel>
   {
      private int bar;

      public int Bar
      {
         get
         {
            return this.bar;
         }
         set
         {
            this.bar = value;
            OnPropertyChanged(x => x.Bar);
         }
      }
   }

It is now much harder to trigger a PropertyChanged event with the wrong property name, as the compiler will verify that the property actually exists. Better yet, if we do a "rename" refactoring of our properties then the IDE will also do the rename in the OnPropertyChanged lambdas. Hurray! :-)

The General Public License doesn't always force you to make your code available

2010-03-28T23:51:00.000+02:00

I just answered this question on stackoverflow asking for "ways around the GPL". While the motives of the poster seem questionable, this did prompt me to explain a subtlety of the General Public License which he may not have understood: the GPL does not automatically force you to release your code as soon as you use GPL'ed code.

The GPL only requires that you make the code available to anyone you distribute the software to. This is not exactly the same as releasing it to the whole world for free. From the GPL FAQ:

Does the GPL require that source code of modified versions be posted to the public?

The GPL does not require you to release your modified version, or any part of it. You are free to make modifications and use them privately, without ever releasing them. This applies to organizations (including companies), too; an organization can make a modified version and use it
internally without ever releasing it outside the organization.

But if you release the modified version to the public in some way, the GPL requires you to make the modified source code available to the program'susers, under the GPL.

Thus, the GPL gives permission to release the modified program in certain ways, and not in other ways; but the decision of whether to release it is up to you.

So if you use GPL code for building an internal tool which you will never distribute to third parties, then there is no requirement to distribute the source to anyone.

Interestingly enough, the above subtlety also applies if you only make the software available as a web application hosted on your own web servers. Since you aren't technically distributing the application to the users, you don't have to give them the code. The Affero General Public License (AGPL) was designed specifically to close this loophole.

Mocking frameworks: stubs vs mocks

2010-02-28T15:59:00.003+01:00

I just answered another question on stackoverflow about Rhino Mocks. The issue in the question comes down to a confusion between mocks and stubs as it often does. Stubs and mocks are both test doubles (i.e. a way to quickly replace a real class implementation by one that you have full control over) but they are set up in different ways.

The distinction between mocks and stubs also exists in other mocking frameworks such as jMock for java. It is important to understand the difference if you are going to do TDD with a mocking framework which supports both approaches.

Mocks

A mock is a test double where you set up behavior by providing a sequence of expected method calls and the corresponding return values. You can think of a mock as an actor in a play who has to follow a script. The script specifies both expectations (i.e. what the other actors will say and when they'll say it) and behavior (i.e. how to respond).

In practice, such a test script is recorded by making calls on the mock. After the behavior has been set up, you must explicitly switch a mock from record mode to replay mode. At the end of the test the expectations are verified. This is how it looks in Rhino Mocks:

[Test]
public void Eat_uses_banana_provider()
{
   var mocks = new MockRepository();
   var bananaProviderMock = mocks.DynamicMock<IBananaprovider>();
   bananaProviderMock.GetBanana(); // record method call expectation
   LastCall.Return(new Banana()); // tell mock what to do
   bananaProviderMock.Replay(); // stop recording, go to replay mode
   
   // actual test
   var monkey = new Monkey(bananaProviderMock);
   monkey.Eat();

   // verify the expectations that were set up earlier
   bananaProviderMock.VerifyAll();
}

Actually, I lied: this is just one way how it can look. Unfortunately a lot of cruft has accumulated in Rhino Mocks (presumably for backwards compatibility) and it has become hard to tell which is the recommended syntax.

Stubs

A stub is a test double that, unlike a mock, has no concept of a "test script". There are no separate record and replay phases: you just set up canned answers for method calls. Setting up this behavior will not create any expectations. Instead, the stub will record how it is used so that you can make assertions separately at the end of the test. In Rhino Mocks it looks like this:

[Test]
public void Eat_uses_banana_provider()
{
   var bananaProviderStub = MockRepository.GenerateStub<IBananaprovider>();
   bananaProviderStub.Stub(x => x.GetBanana()).Return(new Banana());
   
   // actual test
   var monkey = new Monkey(bananaProviderStub);
   monkey.Eat();

   // make assertion about how the stub was used
   bananaProviderStub.AssertWasCalled(x => x.GetBanana());
}

Use stubs instead of mocks if possible

There are a few reasons to use stubs instead of mocks if possible:

Mocks are more powerful than stubs because they can do things like giving different answers for the first and second call. However, this will make your test much harder to understand.
Mocks make it hard to create tests that test only one thing. If the monkey class starts creating its own bananas after a change, then ideally only the Eat_uses_banana_provider test fails. If every monkey test starts failing you will waste time diagnosing the exact problem.
Mocks lead to brittle tests because they expect a certain sequence of method calls. Unless you are explicitly testing that something happens in the expected order, this is rarely what you want.

If you must use mocks, use dynamic mocks rather than strict mocks (Rhino Mocks offers both). Dynamic mocks will not throw exceptions when they receive an unexpected method call. They will return a default return value if necessary (e.g. 0 or null) and will keep on trucking.

Because of the disadvantages, Moq (a popular and simpler alternative to Rhino Mocks) doesn't even offer the record/replay approach.

Creating a web feed for a file system directory

2010-02-22T03:15:00.000+01:00

I just wrote a little python script which generates a web feed for a folder with text files. When run, the script detects the 10 last added/changed files and outputs them as entries in a feed file. This makes it possible to easily create a web feed with just some shell scripting.

To use the script:

download it here and make it executable
edit the configuration values at the start of the script
execute the script regularly, e.g. from a cron job
if not generating the file directly on a web server, publish the feed file on the web (e.g. with scp or just write it to your public dropbox folder)

I have used the W3C feed validation service to check that the resulting file is a valid atom syndication feed. However, the configuration values are important for the validation so check that the generated feed still validates after configuring.

Building a NAS, part 7: ZFS snapshots, scrubbing and error reporting

2010-02-16T03:22:00.040+01:00

Setting up a ZFS pool with redundancy can only protect you against disk failures. To protect yourself against accidental deletions or modifications of files, you can use snapshots. You also need to explicitly start a ZFS data scrub at regular intervals to make sure that any checksum failures are repaired. Such things are best automated, but you might still want to receive reports so that you can keep an eye on things.

Automated snapshots

Setting up automated snapshots for ZFS-FUSE on debian is surprisingly easy. Drop this script in /etc/cron.daily/:

#!/bin/bash
zfs snapshot mypool/myfilesystem@`date +%Y.%m.%d:%H.%M`.auto

This will automatically create daily snapshots with a name like 2010.02.02:06.25.auto. Note that this will complicate things if you need to delete stuff to make room. As long as there is a snapshot referencing a file, it will continue to take space in the pool. Daily snapshots work best for a grow-only archive where you rarely need to delete something.

A word of warning: the scripts in /etc/cron.daily are only executed if they are executable and have no dots in their name. See man run-parts for more details. Test with /etc/cron.hourly to verify that everything works, then move the script to /etc/cron.daily.

Automated scrubbing

A ZFS pool can repair its checksum errors (if there is redundant storage) while still remaining on-line. This is called a scrub. The recommended scrub interval for consumer grade disks is one week. Drop this script in /etc/cron.weekly:

#!/bin/bash
zpool scrub mypool

Web feed reporting

A report of the scrub progress or the results of the last scrub can be shown with the zpool status command. A list of all file systems and snapshots (including some useful statistics) can be shown with the zfs list -t all command. To automate the reporting, I use this script in a cron job:

#!/bin/bash
reportfile=/root/poolreports/`date +%Y.%m.%d:%H.%M`.txt
date > ${reportfile}
zpool status nas-pool 2>&1 >> ${reportfile}
zfs list -t all 2>&1 >> ${reportfile}

I then generate a web feed for the /root/poolreports/ folder as I explained in my previous post and follow the feed with google reader.

Debugging: Why is the Tick event of my WinForms timer no longer raised

2010-02-11T19:19:00.002+01:00

I was debugging an issue at work today were a WinForms Timer object was apparently no longer firing Tick events as it was supposed to.

It turned out that the timer was inadvertently being used by a BackgroundWorker thread. Like most classes, winforms timers are not thread safe so any behavior guarantees are out the window as soon as you start accessing them from different threads without synchronization measures.

Worse, winforms timers interact with the main application thread directly so in this case it is not possible to put such synchronization measures in place. I like to call such classes thread-hostile. Another sure way to create thread-hostile code is to use global variables; we have our fair share of such problems in our legacy code base.

The following sample reproduces the timer problem by accessing a timer from a ThreadPool worker thread; the timer will only be fired once instead of indefinitely as you might expect:

public partial class Form1 : Form
   {
      private System.Windows.Forms.Timer fTimer;

      public Form1()
      {
         InitializeComponent();
         fTimer = new System.Windows.Forms.Timer();
         fTimer.Interval = 1000;
         fTimer.Tick += HandleTimerTick;
         fTimer.Start();
      }

      private void HandleTimerTick(object sender, EventArgs args)
      {
         // sabotage timer by stopping/starting it from another thread
         ThreadPool.QueueUserWorkItem(
            delegate
            {
               fTimer.Stop();
               fTimer.Start();
            } );

         MessageBox.Show("Timer tick");
      }
   }

In our case, the worker thread touched the timer in a much more indirect way: the background task was using a service which leaked side effects into the rest of the system via events, resulting in inadvertent multi-threaded access all over the place.

Conclusion: if you are going to do multi-threading, make sure threads are well-isolated and only communicate with the rest of the system via well defined synchronization points.

Wishlist item: wouldn't it be nice if you had to explicitly mark methods before they could be used by multiple threads? The C# compiler could then generate optional checks that make your code fail fast when there is an accidental "threading leak". It wouldn't surprise me if the language will actually grow such a debugging feature in the future; multi-threaded .NET programming is on the rise yet still wildly dangerous.

OpenID: great standard, many poor implementations

2010-02-04T01:47:00.005+01:00

I was browsing slashdot the other day, and noticed an interesting story that I wanted to upvote, which requires logging in. Interestingly, there's an openid option:

OpenID is a standard that allows you to reuse a single identity on different websites (or any other service that requires an identity). Chances are you already have an OpenID. For example, if you have a google account, then you can use the URL http://www.google.com/profiles/yourusername as an OpenID. There are many more OpenID providers like Yahoo, MyOpenID, AOL, LiveJournal, Wordpress, Blogger, Versign, etcetera.

Currently I have 130 user accounts on the web that I have bothered to keep track of. The idea of OpenID is that you no longer have to create hundreds of accounts, each with their own user name and password (or worse, the same password). You just enter your OpenID, and the OpenID provider takes care of authenticating you.

Stackoverflow gets it right

For an example of OpenID done right, try the stackoverflow login page. See how easy that was? No passwords, no confirmation mails, just reuse your existing identity by clicking the icon of your identity provider. As Steve Jobs would say, isn't that wonderful?

Slashdot gets it wrong

Unfortunately, when you log in with your OpenID in slashdot you are greeted by this:

In other words, you still have to create a username and password specifically for slashdot. Worse, even if you do that you still cannot login with just your OpenID. What gives?

Facebook gets it wrong

You can go into your facebook Settings - Account Settings - Linked Accounts - Change - Add Account and enter an OpenID there. If you then log out and try to log back in to test it, there is no OpenID option on the login page. WTF? On a hunch, I then just retyped the facebook URL in my browser address bar and it looked like I was already logged in.

A little more investigation shows that facebook relies on a cookie that links your browser to your OpenID, and tries to log you in transparently with that information. Since I have configured my browser to only keep cookies between browser sessions for a small white-list of websites, this doesn't work for me at all. Even if I add facebook to the white-list, I won't be able to use my OpenID to log in on other computers. FAIL. I guess just putting a "Log in with OpenID" button on the login page would have been too easy.

Dealing with lack of OpenID support

OpenID support is growing, but the majority of web sites still don't support it or implement their support very poorly. Others only support OpenID as an identity provider and refuse to accept identities from other providers.

To deal with all these sites that still require passwords, most people reuse the same password over and over again. This is terrible security. Any of the sites that you use could have a malicious admin that may like to sell username/password combos to the highest bidder. Or maybe the website admin isn't malicous, but the user account database might store passwords unhashed and could be compromised.

Personally I use the cross-platform KeePass application to maintain a personal encrypted database of passwords. The database is protected by a single master password (or passphrase). I put mine in my dropbox folder, so I have access to my passwords on each PC I use. Even better, if you stick to version 1.x the database is compatible with KeePassMobile so you can carry your passwords with you on your phone.

MEF: GetExport, GetExportedValue methods compared to Import attribute

2010-02-02T01:17:00.003+01:00

(Also posted on the MEF forum)

While writing some automated composition tests today, I found out the hard way that GetExportedValue<Lazy<T>> doesn't do what you might think it does at first sight. For example, the following throws because it tries to get a exported value for the type Lazy<IFoo>, which is not an available part:

   public class Program
   {
      public static void Main(string[] args)
      {
         var catalog = new AssemblyCatalog(Assembly.GetExecutingAssembly());
         var container = new CompositionContainer(catalog);
         var exports = container.GetExportedValue<Lazy<IFoo>>();
         Console.WriteLine(exports.Count());
      }
   }

   public interface IFoo
   {
   }

   [Export(typeof(IFoo))]
   public class Foo : IFoo
   {
   }

As it turns out, if you want to pull a lazy export from a container, you just have to call GetExport<T> or GetExport<T,TMetaData>. This is quite obvious with hindsight, but I just got so used to using the shorthand GetExportedValue<T> that I completely forgot about the existence of GetExport<T>.

This then led me to wonder why the GetExport and GetExportedValue methods work differently from the Import attribute. With the Import attribute, MEF inspects the type you are trying to import and gives special treatment to Lazy<T>. Shouldn't there be an ImportLazy (and ImportManyLazy) attribute instead to make this intention explicit?

On the emergence of ubiquitous computing

2010-01-31T23:32:00.001+01:00

I just read this post about how the iPhone and new iPad herald the era of "new world computing":

In the New World, computers are task-centric. We are reading email, browsing the web, playing a game, but not all at once. Applications are sandboxed, then moats dug around the sandboxes, and then barbed wire placed around the moats. As a direct result, New World computers do not need virus scanners, their batteries last longer, and they rarely crash, but their users have lost a degree of freedom. New World computers have unprecedented ease of use, and benefit from decades of research into human-computer interaction. They are immediately understandable, fast, stable, and laser-focused on the 80% of the famous 80/20 rule.

It is an interesting post, but I don't believe that the lack of multi-tasking and other freedoms is a necessity for "new world computing". If you can make a slick UI for switching between tasks, then you can also make a slick UI for switching between tasks that continue to run in the background.

These limitations are just engineering trade-offs that had to be made to give us an early peek at ubiquitous computing (which is by the way the real term for "new world computing" and was already a research topic long before I went to college). I seriously doubt the next generation of these devices will have the same limitations.

You can wave your hands and talk about how it's all task oriented now all you want, but in the end multi-tasking is a necessity even if only for running a chat client 24/7. And that's exactly what I do with my old and clunky N95 smartphone.

Setting up dropbox on a headless linux system

2010-01-31T02:13:00.007+01:00

I have been using dropbox for a while to synchronize files between different computers. It has some pretty impressive bullet points:

Seamless syncing. You just put files in the dropbox folder, and they are automatically synchronized to your other computers. No firewall issues. In fact, the only problem I have is at work where dropbox is explicitly blocked. >:-(
Easy file sharing over the internet. Just put a file in your public folder, right click, copy public link. You can even host a website on dropbox this way.
Cross-platform. It works on linux, windows, OS X and even iPhone.
You can access the revision history of your files so it works pretty well as an on-line backup service, even if you delete files by accident.
It's completely free if you don't need more than 2GB storage and 30 days of revision history.

I have set up dropbox on my NAS so that I can synchronize my dropbox to a ZFS file system. This way I can combine the advantages of dropbox with the advantages of my NAS:

I get to keep snapshots indefinitely, with disk space being my only limitation.
I protect my data even if the dropbox service fails disastrously, e.g. because of security breach. Think file deletions being synced to all your computers.
I can free space on my dropbox account by moving files on the NAS out of the dropbox folder, yet still keep them safe through my NAS snapshot+backup policy.

Dropbox is targetted at GUI environments, but can still be installed on a headless linux system as described on this wiki page. However, the wiki page did not describe how to change the dropbox folder. I needed this to point dropbox to a folder on my NAS storage pool. It took some minor reverse engineering of the dropbox settings file, but I successfully created a script to do exactly that. I've also added a link and instructions on the wiki on how to use it.

Using MEF for classes which take configuration values

2010-01-27T01:07:00.048+01:00

We're using the Managed Extensibility Framework (part of the upcoming .NET 4.0) at work for a new project.

I have a few pre-MEF classes which take strings, integers other primitive data types in their constructor. For example, consider the following C# class which tracks recently used resources (e.g. the last files opened by the application) by saving them in a file:

[Export(typeof(IRecentlyUsedTracker))]
public class RecentlyUsedTracker : IRecentlyUsedTracker
{
   private readonly string file;
   private readonly int maxItems;

   // constructor
   public RecentlyUsedTracker(string recentlyUsedFile, int maxItems)
   {
      this.file = recentlyUsedFile;
      this.maxItems = maxItems;
   }

   // Marks the given resource as recently used.
   public void Touch(string resource)
   {
      ...
   }
}

The above export doesn't work, because MEF cannot instantiate this class. Obviously it cannot know which string and integer to use as arguments for the constructor.

You can still do it by adding [Import("somename")] attributes to the constructor declaration like this:

    // constructor
    public RecentlyUsedTracker(
       [Import("RecentlyUsedTracker.File")] string recentlyUsedFile,
       [Import("RecentlyUsedTracker.MaxItems")] int maxItems)

~~However, that makes it much more complex to set up the container. Each configuration value has to be explicitly added to the MEF container with ComposeExportedValue as shown below. Blergh!~~ (correction: see update below!)

var catalog = ... some catalog ...
var container = new CompositionContainer(catalog);
container.ComposeExportedValue<string>("RecentlyUsedTracker.File", @"c:\recentlyused.txt");
container.ComposeExportedValue<int>("RecentlyUsedTracker.MaxItems", 5);

My next idea was then to do something like this for the constructor:

   public RecentlyUsedTracker([Import] IConfigurationProvider configurationProvider)
   {
      this.recentlyUsedFile = configurationFile.GetValue<string>("recentlyUsedTracker.File");
      this.maxItems = configurationProvider.GetValue<int>("recentlyUsedTracker.maxItems");
   }

I'm a bit worried about the fact that I'm importing the configurationProvider object only to use it briefly in the constructor. It's also annoying that I need to mock this service in my unit tests, instead of just passing a value. I've asked on the MEF forum if there is a better way.

Update: turns out there is a better way. My first attempt (adding attributes to constructor arguments) is just fine. It's just explicitly adding configuration values to the container that was the bad idea. As Glenn Block suggested, you can just export the configuration values via properties like this:

public class RecentlyUsedTrackerConfiguration
{
  public RecentlyUsedTrackerConfiguration()
  {
     //set values here
  }

  [Export("RecentlyUsedTracker.File")]
  public string File {get;set;}

  [Export("RecentlyUsedTracker.MaxItems")]
  public int MaxItems {get;set;}
}

Building a NAS, part 6: testing ZFS checksumming

2010-01-26T03:27:00.004+01:00

Let's take a look at how ZFS protects data. I plugged in a spare external disk, created two small 1GB partitions on it with fdisk, and set up a ZFS pool for testing:

fdisk /dev/sdc # set up two 1GB partitions
zpool create testpool mirror /dev/sdc1 /dev/sdc2
zfs create testpool/testfs

Note that this is just a test set-up. Normally you should definitely use two separate disks to get the full benefit of mirroring. Also, it doesn't really make sense to slice up disks into partitions.

Smashing bits

Let's create a test file which fills the file system and make a note of the sha1 fingerprint:

cd /testpool/testfs
dd if=/dev/urandom of=testfile bs=1M count=920
# prints a sha1 fingerprint for the file
sha1sum /testpool/testfs/testfile

Now comes the fun part. With a small (and very dangerous) python script, we can corrupt one of the devices by writing some junk data at regular intervals:

#!/usr/bin/python2.5
openedDevice = open('/dev/sdc1', 'w+b')
interval = 10000000
while (True):
  openedDevice.seek(interval,1)
  print str(openedDevice.tell())
  openedDevice.write('corrupt')

When we reread the file after the corruption, ZFS will transparently pick the pieces of data on the healthy disks. Note that in this case the file cannot not be cached in memory because it is larger than the available system memory.

/home/wim/corruption.py
# still prints the correct fingerprint!
sha1sum /testpool/testfs/testfile

Strangely enough, running zpool status testpool doesn't report any errors at this point. I have send a mail to the zfs-fuse mailing list to ask whether this is normal.

To detect and fix the errors, we have to run this simple command:

zpool scrub testpool
# shows progress and results of the scrub
zpool status testpool

To protect against bit rot on consumer grade disks, the recommendation is to run a scrub once a week. In a future post I'll explore how to do that automatically, including some kind of reporting so that I know when a disk is in trouble.

Building a NAS, part 5: minimizing power consumption

2010-01-24T19:57:00.009+01:00

My plan is to let my NAS run 24/7 if the impact on my electricity bill is acceptable. To measure power consumption, I have purchased a power consumption meter that you can plug in between a wall socket and some device. It is one of those tools that can provide hours of quality geek entertainment. So much devices to measure around the house, so little time! :-)

The following table shows the passive power consumption of my NAS after each round of power saving measures. With "passive", I mean that the NAS was not doing anything useful like reading from or writing to the ZFS file systems.

Configuration	power consumption (watts)
initial	80
under-clocked CPU	69
removed AGP video card	59
removed PATA CD-ROM drive	57
removed unused eSATA RAID1 PCI card	55
2 storage disks in standby	47

Take it slow

The Athlon CPU and/or the mother board in this box is apparently too old to support dynamic cpu frequency scaling. It also doesn't seem to support the AMD power saving mode which you can control with the athcool package. Instead, I had to get down and dirty with the BIOS settings and set the CPU multiplier to the minimum value. According to cat /proc/cpuinfo, this slowed down the CPU from 2GHz to 1GHz. Profit: -11 watts.

Bare necessities

I had to connect a video card and a CD-ROM drive to install debian Lenny. Now that the server is running and connected to the network, I can throw those out again and manage the system remotely. Fortunately the BIOS supports booting without a video card. I also removed a PCI card for eSATA RAID1 which I initially thought I would need. Profit: -14 watts

Spin down those storage disks

With the hdparm command we can inspect the power state of a disk and configure stand-by mode:

aptitude install hdparm
hdparm -C /dev/sda # prints "active/idle"
# now tell disk to go to stand-by whenever not used for 2min
# note the very strange S-value to time mapping; consult man hdparm!
hdparm -S 25 /dev/sda
sleep 120
hdparm -C /dev/sda # should print "stand-by"

To make the power saving configuration permanent, I added this to /etc/hdparm.conf. Note the use of /dev/disk/by-id to keep the settings correct even if we start changing the NAS hardware:

/dev/disk/by-id/scsi-SATA_ST3500418AS_9VM7RWGV {
spindown_time = 25
}

/dev/disk/by-id/scsi-SATA_ST3500418AS_9VM7SHA5 {
spindown_time = 25
}

Profit: -8 watts

Leave the system disk alone

It is much harder to get power savings for the system disk, because it is used all the time for logging and by a bunch of daemons. Trying to put this disk in stand-by will just cause it to frequently spin down and up again.

Instead, I'm using a disk recovered from a dead laptop as the system drive. This requires a cheap 2.5" to 3.5" IDE converter cable. Such a drive is already extremely efficient; the potential savings of putting it in stand-by are negligible (~1 watt).

Cost before and after

( update: corrected kwh cost, my original estimate was about 50% of actual cost because I based it solely on Electrabel's power generation cost, and forgot the distribution costs on my bill - thank you Peter!)

At 80 watt, energy consumption per year was 80 watt * (24 * 365) hours = 700.8 kwh. At an average 0.16 euro/kwh that will cost 112 euros. I have reduced that to 66 euros. This goes to show that a 20 euro energy consumption meter can yield a return on investment rather fast.

Another lesson I'll be remembering from these calculations: for my current contract, I can estimate the yearly cost for the continuous consumption of a device as 1.4 euro per watt.

Building a NAS, part 4: ZFS on linux with FUSE

2010-01-23T22:35:00.010+01:00

I decided to use the ZFS file system for my NAS. Although licensing issues prevent it from being ported to the linux kernel, there is a ZFS-FUSE project which has ported ZFS to run in userspace via FUSE.

ZFS is a mature file system (and tool set) which does device pooling, redundant storage, checksumming, snapshots and copy-on-write clones. It also has a very cool deduplication feature where you can configure the file system to look for identical chunks of data and store those only once. Nice!

Getting ZFS-FUSE on Debian Lenny

We'll install some tools, compile and manually start the zfs-fuse daemon. Note that I use the latest source from the "official" repository here, not the last stable release.

aptitude install git-core libaio-dev libattr1-dev libacl1-dev libz-dev libfuse-dev libfuse2 scons libssl-dev
git clone http://git.zfs-fuse.net/official zfs-official
cd zfs-official/src
scons
scons install

At the time of writing, the "scons install" command doesn't seem to install the debian init script. Also, the debian init script which is part of the source has a small error. We'll take care of that manually:

cd ../debian
nano zfs-fuse.init
# fix the line "DAEMON=/usr/sbin/zfs-fuse"
# it should be "DAEMON=/usr/local/sbin/zfs-fuse"
su
cp zfs-fuse.default /etc/default/zfs-fuse
cp zfs-fuse.init /etc/init.d/zfs-fuse
chmod +x /etc/init.d/zfs-fuse
aptitude install sysv-rc-conf
sysv-rc-conf
# use arrows to scroll down to zfs-fuse
# use arrows and space to enable run levels 2,3,4,5
# use q to quit

Setting up a ZFS storage pool and file systems

Currently I have two new 500GB disks available for storage. My first plan was to split each disk in two partitions to build a "safe" storage pool (mirrored over two partitions) and a "bulk" storage pool (no redundancy, striped over two partitions). However, a recurring theme in the ZFS Best Bractices Guide is that you should not slice up your disks if you can avoid it. Therefore, I'll keep things simple and just create one big 500GB pool of mirrored storage.

# start the zfs daemon
zfs-fuse
zpool create nas-pool mirror \
/dev/disk/by-id/scsi-SATA_ST3500418AS_9VM7RWGV \
/dev/disk/by-id/scsi-SATA_ST3500418AS_9VM7SHA5
zpool status

I will however, still create two separate file systems in this pool for "archive" and "bulk" storage. This makes it easy to have different backup policies for each data set.

zfs create nas-pool/archive
zfs create nas-pool/bulk
zfs list
zfs mount -a

Because ZFS is designed to handle storage pools with potentially thousands or more file systems, you don't have to manually edit /etc/fstab to set up mount points. The shown mount command will automatically mount all available ZFS file systems as /pool-name/file-system-name. This is also what the init script does.

Exposing the ZFS file systems on the network via samba

First we'll set up a "nasusers" group which has read/write access to the ZFS file system:

# create nasusers group and add a user to it
groupadd nasusers
usermod -a -G nasusers wim

# give nasusers read/write access
cd /nas-pool
chmod 2770 archive
chmod 2770 bulk
chgrp nasusers archive
chgrp nasusers bulk

Now give those users a samba password:

smbpasswd -a wim

Add a section like this to /etc/samba/smb.conf like this for each folder to expose:

[archive]
path=/nas-pool/archive
browsable=yes
writable=yes
valid users= @nasusers

Finally, restart samba:

/etc/init.d/samba restart

Now the ZFS file systems should be available on the network, and users can start copying there stuff in there. In a future post we'll explore how to leverage some of those advanced ZFS features.

Building a NAS, part 3: filesystem doubts

2010-01-22T01:56:00.004+01:00

I had planned to demonstrate the creation and administration of a raid1 BTRFS filesystem in this post, but while playing around with BTRFS I ran into a few snags:

I was able to mount a normal BTRFS filesystem spanning two devices, but not one in raid1 mode. Then I discovered that I could only mount a raid1 BTRFS filesystem if I gave it a label.
I saw some unexplained mount failures on a multi-device FS which disappeared after I mounted once via another device.
Though data is checksummed, I couldn't find a way to detect checksum failures other than reading all files and watching output from the kernel with dmesg.
I confirmed that in a raid1 setup, BTRFS will still find the good copy after the data on one device is corrupted. However, I couldn't find a way to reliably repair the corruption other than reading and rewriting all files.

These issues didn't exactly give me confidence in the maturity of BTRFS. I knew it wasn't production ready, but I hoped it was close. I'm now pretty sure that it is not.

Perhaps more importantly, I also realized that BTRFS is the GPL-licensed answer to the more mature ZFS. ZFS is a Solaris filesystem developed by Sun. It can't be ported to run in the linux kernel because of licensing issues, hence the need for BTRFS. However, BTRFS is mainly sponsored by Oracle, and Oracle is buying Sun. If Oracle's motivation to sponsor BTRFS was to counter Sun's open source efforts, then the Sun deal takes away that motivation. On the other hand, if their motivation was truly to get a next-generation filesystem in linux, then they might as well relicense ZFS under the GPL. Chris Mason gave this vague comment when asked about ZFS and the Sun deal:

Chris: Sun has many interesting projects, and I’m looking forward to working with their R&D teams. We’re committed to continuing Btrfs development, and ZFS doesn’t change our long term plans in that area.

That's no rational explanation why Oracle would continue to support both projects, so I'm skeptical.

Meanwhile, that leaves my little build-a-NAS project stalled. I see these options:

use BTRFS anyway, even though it's not yet mature and its future seems unsure
run OpenSolaris and ZFS
run the FUSE port of ZFS on linux, which dodges the licensing issue by running the FS in userspace (presumably at the cost of performance)
use software raid + LVM on linux

I'm not sure at all which direction to take.

Building a NAS, part 2: getting BTRFS on Lenny

2010-01-17T15:51:00.004+01:00

update: after discovering that BTRFS isn't as mature as I hoped, I switched to ZFS-FUSE. You might want to read my post on setting up ZFS instead.

The debian "lenny" release comes with version 2.6.26-2 of the linux kernel. This kernel does not yet have support for BTRFS, so we'll download, compile and install the latest stable kernel release.

# install some required packages as root
su
aptitude install install bzip2 fakeroot kernel-package libncurses5-dev zlib1g-dev
exit

# download and extract linux kernel
wget http://www.kernel.org/pub/linux/kernel/v2.6/linux-2.6.32.3.tar.bz2
tar -xvjf linux-2.6.32.3.tar.bz2
cd linux-2.6.32.3

# copy existing kernel configuration from /boot
cp /boot/config-2.6.26-2-686 .config

# edit kernel configuration (navigate with arrows, toggle options with space)
# - under "File Systems, enable "Btrfs filesystem (EXPERIMENTAL)"
# - under "Virtualization", disable "Linux hypervisor example code"
#
# You can also take this opportunity to optimize the kernel for your CPU
# architecture under "Processor type and features" - "Processor family"
# Examine the output of "cat /proc/cpuinfo" if you're not sure of your CPU.
make menuconfig

# build kernel (this takes a while, especially on old machines)
make-kpkg --rootcmd fakeroot --initrd linux-image linux-headers

# install new kernel packages and reboot
cd ..
su
dpkg -i linux-image-2.6.32.3_2.6.32.3-10.00.Custom_i386.deb
dpkg -i linux-headers-2.6.32.3_2.6.32.3-10.00.Custom_i386.deb
reboot

If everything went well, the system should boot up under the new kernel. If something goes wrong, you still have the option of booting under the old kernel by using the grub menu at startup.

Now we have a kernel with support for the btrfs filesystem, but still no userspace tools to use it. We'll download, compile and install the latest version of those tools:

aptitude install git-core uuid-dev e2fslibs-dev libacl1-dev
git clone git://git.kernel.org/pub/scm/linux/kernel/git/mason/btrfs-progs-unstable.git
cd btrfs-progs-unstable
make
su
make install
exit

Now we have mkfs.btrfs to create a BTRFS file system, and some other tools to manage such a filesystem. We'll start playing around with those in the next post.

Building a NAS, part 1: installing Lenny and introducing BTRFS

2010-01-17T04:50:00.006+01:00

I'm building a NAS from spare parts. The basic system is an 8 year old PC. My operating system of choice is the latest stable release of debian gnu/linux, codename lenny. I would like to use the (still experimental) btrfs file system on this box for the following reasons which make it a good data-haven:

It can do raid1-like mirroring of data over multiple devices. This makes it resilient against failed disks.
It can make copy-on-write mountable snapshots of a volume. By regularly making snapshots (e.g. from a cron-job) you can keep old versions of your data without wasting any space on identical copies.

Of course, this still won't fully protect my data. There could be hardware or software errors that destroy the data on all disks. There could be circumstances that destroy all disks together, like fire or lightning. So I'll still have to make an off-site backup every now and then.

To install lenny, I downloaded the 40MB businesscard CD image. Despite its small download size, this CD still has a user-friendly graphical installer. After letting the installer do its thing, I still did the following:

ran "tasksel" to install "file server" related packages
ran "aptitude install openssh-server" to enable remote access

At this point I have a basic linux system that I can SSH into. Unfortunately "lenny" does not come with btrfs support. We'll fix that in the next post...

Registratierechten te betalen bij het kopen van een woning in Vlaanderen

2010-01-11T23:29:00.035+01:00

This post is in Dutch because it is about law and taxes in the Flemish Region of Belgium.

Deze post hoort thuis in een serie over de aankoop van ons huis. Eerder in deze serie:
Beëindigen van een huurovereenkomst door de huurder in België

Wanneer je in het Vlaamse gewest vastgoed koopt, moet je daarop registratierechten betalen. Deze rechten zijn vastgelegd door het Wetboek der registratie-, hypotheek- en griffierechten. Artikel 44 geeft aan dat de registratierechten 10 procent bedragen:

Het recht bedraagt 10 ten honderd voor de verkoop, de ruiling en iedere overeenkomst tot overdracht onder bezwarende titel van eigendom of vruchtgebruik van onroerende goederen.

Vermindering tot 5 procent

Volgens artikel 53 wordt dit verminderd tot 5% voor kleine landeigendommen en bescheiden woningen. Dit is vastgoed waarvan het kadastraal inkomen lager is dan een zeker maximum, vastgelegd door een apart Koninklijk Besluit. (Ik vind spijtig genoeg geen link naar het KB.)

Dit staat ook wel bekend als klein beschrijf. Er zijn nog bijkomende criteria vastgelegd door artikels 54-61; zeer droge kost, en aangezien de woning die Elke en ik gaan kopen toch niet in aanmerking komt, ga ik hier niet verder op in gaan.

Vermindering op het bedrag waarop die 5 of 10 procent berekend wordt

De bovenstaande percentages worden berekend op de prijs overeengekomen tussen koper en verkoper. Volgens artikel 46bis kan echter een deel van dit bedrag vrijgesteld worden, en die vrijstellingen mag je dus van de prijs aftrekken voor je die 5 of 10 procent berekend.

Er wordt 15.000 euro vrijgesteld als je het vastgoed koopt als hoofdverblijfplaats:

De heffingsgrondslag ten aanzien van de verkopingen, zoals bepaald in de artikelen 45 en 46, wordt verminderd met 15.000 euro in geval van zuivere aankoop van de geheelheid volle eigendom van een tot bewoning aangewend of bestemd onroerend goed door een of meer natuurlijke personen om er hun hoofdverblijfplaats te vestigen.

Deze vrijstelling heeft nog extra voorwaarden:

je mag nog geen onroerend goed bezitten
je moet uitdrukkelijk vragen om deze korting te krijgen (!)
je moet er binnen de 2 jaar gaan wonen (5 jaar in het geval van bouwground)

Als je in aanmerking komt voor bovenstaande vermindering, wordt er nog 10.000 euro extra vrijgesteld als je voor de woning een lening aangaat (of zelfs 20.000 euro indien je in aanmerking komt voor de vermindering tot 5 procent):

Als met het oog op de financiering van een aankoop, vermeld in het eerste lid, een hypotheek wordt gevestigd op het aangekochte onroerend goed, wordt het bedrag van de vermindering van de heffingsgrondslag, vermeld in het eerste lid, verhoogd met hetzij 10.000 euro als op de aankoop het recht, vermeld in artikel 44, verschuldigd is, hetzij 20.000 euro als op de aankoop het recht, vermeld in artikel 53, verschuldigd is

In ons geval komen we in aanmerking voor beide verminderingen, dus we mogen 25.000 euro van de verkoopprijs aftrekken voordat we de 10 procent registratierechten berekenen. Dat "bespaart" ons dus 2500 euro.

Beëindigen van een huurovereenkomst door de huurder in België

2010-01-10T17:27:00.035+01:00

This post is in Dutch because it is about Belgian law and I live in the Flemish region. As far as I know the referred Belgian law is not published in English, so I had to pick another language.

Aangezien Elke en ik net een verkoopsovereenkomst voor een huis getekend hebben, hebben we de eerstvolgende maanden heel wat te doen. Een van die dingen is het opzeggen van de huur.

Onze huurovereenkomst is een typisch contract van 9 jaar. Zo'n contract wordt ook wel "een 3-6-9" genoemd in de volksmond omdat de huurwet bepalingen bevat waarin sprake is van driejarige periodes. De spelregels voor het vroegtijdig beeindigen van zo een overeenkomst worden vastgelegd door de Belgische wetgeving in het Burgerlijk Wetboek, "Regels betreffende de huurovereenkomsten met betrekking tot de hoofdverblijfplaats van de huurder in het bijzonder" en het bijhorende Koninklijk besluit. Een langere maar meer leesbare interpretatie van deze wetteksten is te vinden in de brochure "De Huurwet" (10e editie juli 2008) van de Vlaamse overheid.

Opzeggingstermijn

Een contract van 9 jaar kan altijd door de huurder opgezegd worden, maar de opzeg moet wel minstens 3 maanden op voorhand gebeuren:

Art. 3, § 5 De huurder kan de huurovereenkomst op ieder tijdstip beëindigen met inachtneming van een opzeggingstermijn van drie maanden.

Die termijn gaat in vanaf de eerste dag van de maand die volgt op de opzegging:

Art. 3, § 9. In alle gevallen waarin een opzegging te allen tijde kan worden gedaan, neemt de opzeggingstermijn een aanvang de eerste dag van de maand die volgt op de maand tijdens welke de opzegging wordt gedaan

Vreemd genoeg staat er iets anders in het KB, bijlage "HUUROVEREENKOMSTEN VOOR WONINGEN GELEGEN IN HET VLAAMS GEWEST". Het lijkt erop dat de woorden "de maand die volgt" verdwenen zijn (benadrukking toegevoegd door mezelf):

In alle gevallen waarin de opzegging te allen tijde kan worden gedaan, neemt de opzeggingstermijn een aanvang de eerste dag van de maand tijdens welke de opzegging wordt gedaan.

Opzeggingsvergoeding

Als de huurder het huurcontract opzegt tijdens de eerste 3 jaar, dan heeft de verhuurder recht op een opzeggingsvergoeding. Ook uit Art. 3, § 5:

Indien de huurder de huurovereenkomst evenwel beëindigt tijdens de eerste driejarige periode, heeft de verhuurder recht op een vergoeding. Die vergoeding is gelijk aan drie maanden, twee maanden of één maand huur naargelang de huurovereenkomst een einde neemt gedurende het eerste, het tweede of het derde jaar.

Emo Queries

2009-08-18T00:58:00.005+02:00

In a moment of frustration, I started to type the query "Why is it so hard to call .NET assemblies from java?" in the little search field of my browser.

I normally don't type full questions for my searches so I was a bit surprised when these google suggestions popped up:

Executing visual studio unit tests without installing visual studio

2008-11-18T23:12:00.014+01:00

UPDATE: I've written a new post for Visual Studio 2010.

The professional edition of visual studio 2008 has a nicely integrated unit testing framework. I suppose visual studio team server will run these for you whenever somebody checks in a source code change. Unfortunately, we don't have team server or the team edition of visual studio at work.

This lead me to look for a way to execute these unit tests on our build server. "build server" may be a bit of an euphemism; it is little more than a virtual machine with a collection of python and batch scripts and it doesn't have visual studio installed. In case you are wondering, compiling C# projects and solutions from the command line with msbuild works fine without installing visual studio. I wanted a similar solution for the unit tests.

It turns out that there is a command line utility called mstest.exe that can be used to run these tests. This utility comes with visual studio. It is not straightforward to get it working without installing visual studio. Here's what I did to get mstest.exe running on our build server:

1) Created a \tools\mstest folder on the buildserver
2) Copied %ProgramFiles%\Microsoft Visual Studio 9.0\Common7\IDE\mstest.exe
3) Copied these files from %ProgramFiles%\Microsoft Visual Studio 9.0\Common7\IDE\PrivateAssemblies

Microsoft.VisualStudio.QualityTools.AgentObject.dll
Microsoft.VisualStudio.QualityTools.CommandLine.dll
Microsoft.VisualStudio.QualityTools.Common.dll
Microsoft.VisualStudio.QualityTools.ControllerObject.dll
Microsoft.VisualStudio.QualityTools.ExecutionCommon.dll
Microsoft.VisualStudio.QualityTools.Tips.UnitTest.Adapter.dll
Microsoft.VisualStudio.QualityTools.Tips.UnitTest.AssemblyResolver.dll
Microsoft.VisualStudio.QualityTools.Tips.UnitTest.ObjectModel.dll
Microsoft.VisualStudio.QualityTools.Tips.UnitTest.Tip.dll
Microsoft.VisualStudio.QualityTools.TMI.dll

4) Copied these files from c:\windows\assembly\gac_msil subfolders. This requires you to use the command line as windows explorer hides the actual files there.

Microsoft.VisualStudio.QualityTools.UnitTestFramework.dll
Microsoft.VisualStudio.QualityTools.Resource.dll

5) Added certain registry keys to the build server registry. You can save these in a .reg file and then simply double click it:

Windows Registry Editor Version 5.00

[HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\VisualStudio\9.0\EnterpriseTools\QualityTools\HostAdapters\VS IDE\SupportedTestTypes]
"{13cdc9d9-ddb5-4fa4-a97d-d965ccfc6d4b}"=""

[HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\VisualStudio\9.0\EnterpriseTools\QualityTools\TestTypes]
@="TestTypes"

[HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\VisualStudio\9.0\EnterpriseTools\QualityTools\TestTypes\{13cdc9d9-ddb5-4fa4-a97d-d965ccfc6d4b}]
"ServiceType"="Microsoft.VisualStudio.TestTools.TestTypes.Unit.SUnitTestService, Microsoft.VisualStudio.QualityTools.Tips.UnitTest.Tip, Version=9.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a"
"NameId"="#212"
"SatelliteBasePath"="C:\\Program Files\\Microsoft Visual Studio 9.0\\Common7\\IDE\\PrivateAssemblies\\"
"SatelliteDllName"="Microsoft.VisualStudio.QualityTools.Tips.TuipPackageUI.dll"
"VsEditor"="{00000000-0000-0000-0000-000000000000}"
"TipProvider"="Microsoft.VisualStudio.TestTools.TestTypes.Unit.UnitTestTip, Microsoft.VisualStudio.QualityTools.Tips.UnitTest.Tip, Version=9.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a"

[HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\VisualStudio\9.0\EnterpriseTools\QualityTools\TestTypes\{13cdc9d9-ddb5-4fa4-a97d-d965ccfc6d4b}\Extensions]
".dll"=dword:000000d7
".exe"=dword:000000d7

[HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\VisualStudio\9.0\EnterpriseTools\QualityTools\TestTypes\{13cdc9d9-ddb5-4fa4-a97d-d965ccfc6d4b}\SupportedHostAdapters]
"Smart Device"="Smart Device Host Adapter"
"ASP.NET"="ASP.NET Host Adapter"
"VS IDE"=""

[HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\VisualStudio\9.0\EnterpriseTools\QualityTools\TestTypes\{ec4800e8-40e5-4ab3-8510-b8bf29b1904d}]
"ServiceType"="Microsoft.VisualStudio.TestTools.TestTypes.Ordered.SOrderedTestService, Microsoft.VisualStudio.QualityTools.Tips.OrderedTest.Tip, Version=9.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a"
"NameId"="#313"
"SatelliteBasePath"="C:\\Program Files\\Microsoft Visual Studio 9.0\\Common7\\IDE\\PrivateAssemblies\\"
"SatelliteDllName"="Microsoft.VisualStudio.QualityTools.Tips.TuipPackageUI.dll"
"VsEditor"="{700218d8-f6f1-4ec3-be76-a35f72260503}"
"TipProvider"="Microsoft.VisualStudio.TestTools.TestTypes.Ordered.AutoSuiteTip, Microsoft.VisualStudio.QualityTools.Tips.OrderedTest.Tip, Version=9.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a"

[HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\VisualStudio\9.0\EnterpriseTools\QualityTools\TestTypes\{ec4800e8-40e5-4ab3-8510-b8bf29b1904d}\Extensions]
".orderedtest"=dword:0000013c

After that, you can invoke mstest.exe on the command line like this:

mstest /noisolation /testcontainer:"path\to\testproject.dll"

mstest.exe will return an error code if any test fails.

I have not figured out how to get things working without the /noisolation switch. I believe that any problems caused by this switch are rare. In any case it will only cause problems if there are serious issues with the code under test which will need to be fixed anyway.

Lessons from IT projects: allowing NULL foreign keys in a database can be tricky

2008-05-21T22:39:00.006+02:00

My previous "lesson learned" was a bit general, and perhaps not very useful. Migrating old data is hard, but if the problem wasn't hard to fix we wouldn't be payed to fix it now would we? Today I'll focus on a much more simple technical dilemma: whether to allow database foreign keys to be NULL.

Consider a database table of contacts, a classic example. Each contact has a name, telephone number, email, country_id, etc. The country_id field for the contact is actually a foreign key pointing to a country record in the country table.

Consider now that often we only have a few pieces of contact information, e.g. only a name and email address. If we don't know the country, we can't set it. It may seem natural to declare the country_id column to allow for NULL values.

It's a possibility, but it has consquences. What kind of consequences?

Suppose we'd like to pull all contact data from the database and show it on the screen or in a report. We also want the country data in the same view. The natural query for this is


SELECT * FROM Contact, Country WHERE Contact.country_id = Country.id

Clean and simple. And unfortunately, broken. Your report will silently omit all contacts where country_id is NULL! Needless to say, this can be a serious bug if you are generating a billing/bookkeeping report.

To fix this, you'd have to use a "LEFT JOIN" operation:


SELECT * FROM Contact LEFT JOIN Country ON Contact.country_id = Country.id

Or you could just forbid country_id from ever being NULL like this:


CREATE TABLE [dbo].[Contact](
 [id] [int] IDENTITY(1,1) NOT NULL,
 [name] [nvarchar](999),
 [address] [nvarchar](999),
 [CNT_CountryID] [int] NOT NULL,
        ... more create table goodness ...

Then of course, you also need to put a special "UNKNOWN" country in your Country table. This solution has the "magic value smell" a bit, but after debugging a few problems caused by NULL foreign keys, I very much prefer to use a special unknown country over NULL. It also has the advantage that your code will need to deal less with NULL values, which can be annoying in itself. Come to think of it, those name and address fields should also be NOT NULL, just use empty strings when you don't know the values!

Lessons from IT projects: migrating existing data to a new system

2008-05-20T17:06:00.002+02:00

I'm currently working on a redesign/reimplementation/integration of an administrative system that currently exists as a mix of 3 databases (with semantically vague and manually maintained links between them), two applications and lord knows how many excel spreadsheets. The goal is to have 1 database and 1 application, less room for mistakes, and drastically reduced administrative overhead.

As I approach the deadline for this project, I often find myself thinking "If only I had known that ..." or "I should remember X next time".

A first observation: Migrating old data to a new system is harder than you think. It forces you to follow stupid constraints for the new system, because otherwise the old data can't be expressed in the new system. Imagine trying to build a house in the middle of the forest without being allowed to cut down certain trees. You'll end up with alot of extra walls to build around the forbidden trees, and your shiny new house will start to look like a maze.

If at all possible, go for a green-field scenario where the data can be left behind or is migrated gradually/manually by the customer. After all, it's the customer who knows about the old data and business rules. If you absolutely need to migrate the existing data, don't underestimate the effort of having to map out the existing organically evolved "solution". That may be a project in it's own right.

Logging all your chats in one place

2008-02-24T18:48:00.001+01:00

I ~~currently use~~ used to use pidgin for chatting, mainly because it is cross-platform and I liked using the same application on windows and linux. Another advantage of pidgin was that it is designed to support almost all the chat protocols out there. I can be online on MSN and google talk at the same time with the same application.

One thing has been bugging me for a while now though, and that's the fact that my pidgin chat logs are stored locally. "Locally" turns out to mean "spread out over several machines and user accounts", without any hope of making them searchable or properly backed up. This is not a good thing. Fortunately this is only a problem for MSN chats. It doesn't matter for my google talk account, as the google talk server will log everything for me "in the cloud" and even make it searchable from the gmail web interface.

Is there anyway to get this advantage for other chat accounts also? Well actually, it turns out that there is. Google talk is just an jabber/XMPP implementation, and google has enabled "federation" with other jabber servers. That means that anyone can run their own jabber server, create users for it, and chat with google chat users. The jabber spec also includes the concept of "transports", which are basically gateway services running on the jabber service which enable you to talk to other networks like MSN. All you need to do is register an MSN transport with your google talk account, do all your MSN chatting via your google account, and voila: all your MSN chats are stored at google.

Here's how to do it:
1) Google talk or pidgin doesn't enable you to add transports. Download and install psi instead, another jabber client.
2) Psi will prompt you to add a jabber account when you first start it. Use your gmail address as a jabber-id. Set the status to "online", you should be connected to google talk now.
3) Right click on gmail.com in psi and click "service discovery". Psi will automatically look for services at "gmail.com" and show an error message because there aren't any.
4) Enter another jabber server in the service discovery window, one which runs an MSN transport (e.g. jabber.belnet.be) or some other transport for your favorite chat network.
5) Locate the MSN transport in the list of services, right-click it and click "register". This will prompt you for your MSN credentials.

6) Add/Authorize the transport contact which now appears in your main psi window (the "contact roster"). Double click it and allow it to subscribe to your presence.
7) All your MSN contacts will now also appear, each one needs to be accepted for addition in your contact list. (It may be faster to close psi at this point, log in with pidgin and accept the contacts from there.)
8) Say something to one of your MSN buddies
9) Go to gmail.com, log in, click "chats" or search for what you just said. It's all there! :) All your MSN chats are now logged by google, as long as you use your name@gmail.com jabber account for chatting.

Finding debian packages and listing the files installed by them

2008-02-21T23:03:00.000+01:00

The great thing about GNU/linux distributions is that they come with zillions of software packages, downloadable over the internet from free software repositories. Say you notice that your firefox browser doesn't support java applets. In debian, you're just a


apt-cache search java mozilla

away from finding a package providing just what you need. One of the results returned on my machine by the above command was "sun-java6-plugin". Just do


apt-get install sun-java6-plugin

and you're done! No need to worry about dependencies (like, ehm, java!), those are also downloaded and installed as needed.

Using an installed package

When you install an application "foo", typically it will be in a package named "foo" and will be launchable by typing "foo" in the console. In the case of command line tools, typing "man foo" will bring up one of those fascinating manual pages. In the case of GUI tools, they should be automagically added to your start menu in your favorite window manager. So far so good.

Unfortunately, a minority of the packages are a bit evil in this aspect. The "bittorrent" package for example provides the original bittorrent implementation as a set of command line tools. You can install it, but you'll find that no "bittorrent" command becomes available, nor will "man bittorrent" give any results. "apt-cache show bittorrent" doesn't provide much hints either. Where the hell are my command line tools?!

There is a solution! It took me a while to find, but it is to do a


dpkg -S bittorrent|grep bin

which will list all the invokable binaries installed by the bittorrent package. Turns out that the tool is invokable as btdownloadcurses.bittorrent (thank you tab completion!).

Getting started with a new programming library

This is also very handy for other types of packages for which it may not be obvious how to start using them, like programming libraries. Suppose you want to write a little script to upload your pictures to google's picasaweb service. A quick


apt-cache search picasa

will yield the "python-gdata" package. Once installed,


dpkg -S python-gdata

will help you locate all the new python goodness at your disposal. In this case, it turns out that alot of sample scripts are installed at /usr/share/doc/python-gdata to help you get started.

Recording phone calls on a S60 phone. Part 3: getting rid of the beep

2008-02-04T21:29:00.000+01:00

OK so recording phone calls turned out to be relatively easy, but what about that annoying beep inserted into recorded phone conversations? This beep comes after the first second of the conversation, and every 15 seconds after that. Turns out that there is a way around this: the trick to avoid the beep is to not record longer than 1 second.

That's a pretty short conversation, you say?

Not necessarily. You could, for example, stop the recording before the first second completes, then restart it immediately, etc... Unfortunately the speed at which code can be executed is limited, and this technique results in unacceptable gaps in the recording, especially if we use python. There is a commercial application called Total Recall which appears to use this technique. It worked for my N95 if I set the recording quality to "low", but like my python script, the recording gaps are unacceptable.

Is it possible to do better? Yes! If you start a new recording before stopping the previous one, the beep still doesn't surface, and there will be no lost audio. The following function does exactly that:


def start_recording(filename):
   global recording
   counter = 0
   recording = True
   sound_file = audio.Sound.open(filename+'_part'+str(counter)+'.wav')
   sound_file.record()
   start = True
   while recording:
      if start:
         start = False
      else:
         e32.ao_sleep(.6)
      counter = counter + 1
      next_sound_file = audio.Sound.open(filename+'_part'+str(counter)+'.wav')
      next_sound_file.record()
      e32.ao_sleep(.3)
      sound_file.stop()
      sound_file.close()
      sound_file = next_sound_file
   print "stopping recording"
   sound_file.stop()
   sound_file.close()
   concatenate_wavs(filename, counter)

This function will create a recording in fragments of less than 1 second, which overlap about .3 seconds. In practice, it turns out that the overlap is typically a bit less than that. Now we could post-process these audio fragments on a PC to stitch them together, but part of the usefullness of recording phone conversations is that you can immediately replay them on your phone. This implies the need to do the post-processing on the phone itself.

This is were the python wave module comes in. This module provides all the functionality you need to read and write valid .wav files, but unfortunately it is not included in the PyS60 implementation. Not one to be easily discouraged, I downloaded the official python 2.2 sources. This is the (old!) version upon which PyS60 is based. Unpack the source archive, copy the wave.py and chunk.py modules to the phone, and yes, it works!

Simply concatenating the wave samples will result in a little stutter in the recording though. The following function attempts to detect the overlap between fragments to fix this:


def concatenate_wavs(filename, max):
   print "Processing "+str(max+1)+" audio chunks:"
   merged = wave.open(filename+'.wav','w')
   previousframes = None
   for i in range(0,max+1):
      partname = filename+'_part'+str(i)+'.wav'
      part = wave.open(partname,'r')
      frames = part.readframes(part.getnframes())
      part.close()
      os.remove(partname)
      if i==0:
         merged.setparams(part.getparams())
         overlap=0
      else:
          overlap=frames.find(previousframes[-8:])+8
          if overlap==7:
              overlap=0
          print overlap
      previousframes=frames
      merged.writeframes(frames[overlap:])
      
   merged.close()
   print "done!"

And there you go, a simple python script for recording phone conversations on a S60 3rd edition phone without beeps :) For some reason the overlap detection does not always work, but the casual stutter is not yet annoying enough to warrant further investigation. More important is the fact that this script still needs alot of work before it can be declared suitable for public consumption. It should get a proper GUI, be packaged as a standalone application which can be easily installed, and it should be started automatically whenever the phone is booted.

But I have not yet figured out all that stuff, so that's for another post... :)

Recording phone calls on a S60 phone. Part 2: working prototype

2008-01-25T00:48:00.001+01:00

<voice type="farnsworth">Good news everyone!</voice>

I messed around with the PyS60 python interpreter for my S60 phone, and managed to record phone calls! The first step is to install PyS60 as described on the PyS60 wiki. No further tools are needed, the python runtime and shell on your phone are sufficient. Next, we need a script which listens for phone calls being connected, either incoming or outgoing, and which starts and stops recording those calls:
import appuifw import e32 import telephone import audio import time # Must be either None, or a sound file which is recording. sound_file = None # Directory used to save call recordings. Must exist. # Filename will be appended to this path. recordings_folder = "e:\\CallRecordings\\" def cb_calling(args): global sound_file state = args[0] # if transition to connected state and not recording, start recording if (state == telephone.EStatusConnected) and (sound_file == None): number = args[1] if number == None: number = "None" filename = recordings_folder + time.strftime("%Y%m%d_%H%M_%S_") + number + '.wav' #filename = recordings_folder + 'test.wav' print filename sound_file = audio.Sound.open(filename) sound_file.record() # if transition to non-connected state and recording, stop recording elif (state != telephone.EStatusConnected) and (sound_file != None): sound_file.stop() sound_file.close() sound_file = None def quit(): app_lock.signal() appuifw.app.exit_key_handler= quit app_lock=e32.Ao_lock() telephone.call_state(cb_calling) telephone.incoming_call() app_lock.wait()
The body of the script only sets the exit key handler, registers a callback function for detecting telephone call state changes, and starts waiting for a signal on a lock object. After that, the main thread will not do any more work. It was surprising to discover that real multi-threading is central to developing in Pys60. I was expecting to see a "message queue with single threaded message pump" kind of model, like most PC application frameworks.

Copying this python script to a \Python directory enables you to run it on the phone with the python shell application. Once running, it will listen for phone calls and record them to the recordings_folder.

Unfortunately, there is an annoyance: apparently nokia feels obligated to insert a beeping sound into the call every 5 seconds if it is being recorded. The beeping is not present in the recording though. I did not find any more information on this "feature", other than that it appears to be built into the phone. I guess nokia is trying to legally cover their ass here. I researched Belgian law regulating phone call recording, but did not find any obligation to inform participants that you are recording them. If you are not recording with intent to deceive or harm and are participating in the call yourself, then it's OK to record without warning. (Disclaimer: I am not a lawyer)

Can we get rid of these annoying beeps? Is there a way to automatically start the call recorder code on phone boot? Questions, questions...

To be continued!

Recording phone calls on a S60 phone.

2008-01-23T17:00:00.000+01:00

I love gmail. I haven't needed to delete email since I started using it. And the search feature is awesome. I wish my entire life could be archived and searched just as easily. And why not? For example, everything I hear could in principle be recorded to files and archived. Add a dash of speech-to-text and I could have a searchable history of all my conversations.

Right now however, I only archive email and the occasional cameraphone snapshot or video (and chat logs and text messages, but I should archive them better). In pursuit of the goal "log everything", I'm going to try and add something to that list: phone conversations.

My nokia smartphone is build on the S60v3 symbian platform. There exists a python interpreter for that platform, and it comes with modules for doing a lot of stuff. According to the documentation, you can run code in response to incoming calls. You can also record audio. This should be all I need to start logging my phone calls.

Yay, we've got ourselves a project :)

Using System.Diagnostics.StackTrace to leave debugging clues.

2007-07-16T16:55:00.000+02:00

When I am debugging, I often find myself wishing I could step back to a certain point, in order to discover at which point some piece of data becomes corrupted. Most of the time, it is possible to set a conditional breakpoint which will break the program at the point where something unexpected happens, and reproduce the bug again.

However, this does not always work. Let's say for example that some piece of code is calling Dispose on an instance which it doesn't own, and therefore shouldn't dispose. Things will go haywire when the instance is used after being disposed. At that point, it may be hard to determine the culprit who called Dispose. And because of the number of instances being created and disposed, it may very well be impractical to set a breakpoint in Dispose and inspect each call to it!

What we need here is a way to see who called Dispose in the past.
The way I do this in .NET is by setting a field to a new System.Diagnostics.StackTrace in Dispose just for debugging. Then when things go haywire and the debugger breaks, I simply read the stacktrace to discover the culprit. This trick can be used in any programming language or environment which supports the runtime generation and inspection of stacktraces.

In visual studio 2005, you could also avoid modifying your code by inserting a tracepoint with the following expression:

Instance with hashcode {GetHashCode()} was disposed. Call stack: $CALLSTACK

Then when things go haywire, you can inspect the hashcode of the corrupted instance and search for it in the trace log.

20 kilometers of Brussels

2007-05-28T15:05:00.000+02:00

Like last year, I participated in the 20 kilometers of Brussels yesterday along with some of my coworkers. I had my sights set on finihsing within 1h40m.

Up to the 17th kilometer I was perfectly on schedule for that time. Unfortunately I got recurring cramps in my calf muscles during the last 3 kilometers, which almost halved my speed :( I still finished in 1h46m15s. That's an improvement of 2 minutes over last year, so the result was not a complete disappointement.

Don't document that you throw ArgumentException or InvalidOperationException

2007-05-16T17:35:00.000+02:00

When programming by contract (which should always be the case, at least implicitly), each method has pre-conditions that the caller should adhere to and post-conditions which the callee guarantees if the pre-conditions were satisfied; if they weren't, all bets are off.

Let's look at an example in the .NET framework: System.XML.XmlReader.ReadElementContentAsString. This method has the precondition that the xml reader should be positioned on an element. However, Microsoft has chosen to actually document this as a postcondition: if the xml reader is not positioned on an xml element, InvalidOperationException is thrown. This is the normal documentation style for the .NET framework, but it has never sat right with me for two reasons.

The first reason is that you shouldn't encourage programmers to rely on avoidable exceptions. I feel it would be better if this was simply documented as a precondition which must never be violated. What happens when the caller violates the contract doesn't need to be documented, so the guarantee of an InvalidOperationException could be omitted. It would of course still be thrown, but as an indication of a programming error. It shouldn't be relied upon.

Another reason is inheritance. One of the great things of programming by contract is that it provides rules for what you can do in an overriding method. Any method must implement the contract of the method it overrides, because the caller may not be aware of the exact type of the callee. However, the overriding method may loosen the pre-conditions. It must merely accept anything that the overridden method would have accepted. It may accept more. Unfortunately, if the preconditions are documented as exceptions, they are now post-conditions which restrict the overriding method without good reason.

Conclusion: don't document ArgumentException and InvalidOperationException if it is sufficient to document pre-conditions.