adrift in the sea of experience

Tuesday, May 10, 2011

Problems encountered when using MEF as a dependency injection container

When we started using MEF as a dependency injection container, I figured the trade-off was a bit like this. Good: Already part of the .NET 4 framework, dynamic discovery of components for extensibility. Bad: missing some advanced features like AOP and parametrized construction. Glenn Block posted about this in Should I use MEF for my general IoC needs?.

Since we didn't need those features, MEF seemed like a good choice. But as it turns out, there is a more subtle problem when using MEF as a general purpose dependency injection container. Consider the following example:

   public class Program 
   {
      static void Main(string[] args)
      {
         var container = new CompositionContainer(
            new AssemblyCatalog(Assembly.GetExecutingAssembly()));
         var a = container.GetExportedValue<A>();
      }
   }

   [Export]
   public class A
   {
      private readonly B b;

      [ImportingConstructor]
      public A(B b)
      {
         this.b = b;
      }
   }

   [Export]
   public class B
   {
      private readonly C c;

      [ImportingConstructor]
      public B(C c)
      {
         this.c = c;
      }
   }

   public class C
   {
   }

The export attribute is missing on the C class in my example. Since class A indirectly depends on class C, we get an error when we try to get an A instance from the container:

System.ComponentModel.Composition.ImportCardinalityMismatchException was unhandled
  Message=No valid exports were found that match the constraint '((exportDefinition.ContractName == "DITest.A") AndAlso (exportDefinition.Metadata.ContainsKey("ExportTypeIdentity") AndAlso "DITest.A".Equals(exportDefinition.Metadata.get_Item("ExportTypeIdentity"))))', invalid exports may have been rejected.
  Source=System.ComponentModel.Composition
  StackTrace: ...

So the C export is missing, but surprisingly the error message is complaining about the A class! This is because of "stable composition". Basically, whenever a dependency for a certain part is missing, MEF will resiliently attempt to do the composition without that part. For an example of how that can be useful, see Implementing Optional Exports with MEF Stable Composition.

Useful as it may be, stable composition comes at a price. Since MEF can't tell the difference between critical and non-critical parts, the missing dependency error may cascade upward until you get a mysterious ImportCardinalityMismatchException like the one above.

The MEF documentation on Diagnosing Composition Problems acknowledges this and provides some hints on how to debug such problems. Still, for large compositions the process is far from pleasant. Things are even worse if you have some circular dependencies.

Perhaps it would be better to use Autofac to do the core application composition. It doesn't attempt dynamic discovery or stable composition, so the error messages point straight at the missing dependencies. And with the MEF integration, you can still use MEF for the parts of the application where you do need dynamic discovery and stable composition.

Saturday, May 7, 2011

DownMarker, a Markdown editor/viewer

I've released version 0.1 of DownMarker, a small desktop application that can be used to to view, navigate and edit a set of interlinked markdown files:



In case you don't know Markdown, it is a "easy-to-read, easy-to-write plain text format" intended to be converted to HTML. I first encountered it as the format used at Stackoverflow for editing questions and answers. In fact, DownMarker is based on MarkdownSharp, a library generously released as open source by the stackoverflow team.

Currently some things are still missing, but my hope is that DownMarker will work well for creating light-weight wikis inside your version controlled projects, or anything else that can store files. Let me know if you find it useful!

Tuesday, March 1, 2011

MEF attribute-less registration

The Managed Extensibility Framework has shipped in .NET4, but the work hasn't stopped: the recent MEF2-Preview3 release added some interesting stuff. Let's take a look at attribute-less registration.

First, a recap of the existing attribute-based registration mechanism (or "Attributed Programming Model" in MEF-parlance):



A composition container can take different kinds of export providers. The diagram shows the catalog-based export provider. TypeCatalog is the most important catalog implementation: it is based on a list of types, which it inspects via reflection for MEF attributes. The other implementations are there for convenience, to quickly build type catalogs out of an assembly (AssemblyCatalog), an entire directory (DirectoryCatalog), or other catalogs (AggregateCatalog).

Here is an illustration of the above with a quick code example featuring some dummy types. (The CatalogExportProvider is not visible because the CompositionContainer can conveniently construct it for us if we directly pass it a catalog.)
interface IFoo { }
   interface IBar { }

   [Export(typeof(IFoo))]
   class Foo : IFoo
   {
      [Import(typeof(IBar))]
      public IBar Bar { get; set; }
   }

   [Export(typeof(IBar))]
   public class Bar : IBar { }

   class Program
   {
      static void Main(string[] args)
      {
         var catalog = new TypeCatalog(typeof(Foo), typeof(Bar));
         var container = new CompositionContainer(catalog);
         var foo = container.GetExportedValue<IFoo>();
      }
   }

For an attribute-free alternative to the above, you might expect that the new MEF release would contain an alternative ExportProvider or perhaps a new ComposablePartCatalog implementation, which would take its information from configuration in code instead of reflection. Surprisingly, that's not how it was done!

In reality, the attribute-less registration is implemented by adding information right above the reflection level! We essentially inject fake reflection information into our catalogs to simulate the presence of attributes. The attribute-free version of the above example looks like this:

interface IFoo { }
   interface IBar { }

   class Foo : IFoo 
   {
      public IBar Bar { get; set; }
   }

   class Bar : IBar { }

   class Program
   {
      static void Main(string[] args)
      {
         var registration = new RegistrationBuilder();

         registration.OfType<Foo>()
            .ImportProperty<IBar>(property => property.Name == "Bar")
            .Export<IFoo>();

         registration.OfType<Bar>()
            .Export<IBar>();

         var catalog = new TypeCatalog(
             types: new Type[] { typeof(Foo), typeof(Bar) },
             reflectionContext: registration);
         var container = new CompositionContainer(catalog);
         var foo = container.GetExportedValue<IFoo>();
      }
   }

Note how the TypeCatalog takes a new ReflectionContext parameter; that's were the reflection info is injected. An interesting result of this approach is that you can leverage the existing catalogs to register types in bulk. For example, the following example will export all the types which implement IPlugin in a given directory, without using any attributes:
var registration = new RegistrationBuilder();
registration.Implements<IPlugin>().Export<IPlugin>();
var catalog = new DirectoryCatalog("./plugins", "*.dll", registration);

This is just what I've gleaned from a quick peek at the unit tests in the new MEF release. There is much more to the RegistrationBuilder API which I haven't shown here. I'm sure there will be more documentation on the MEF codeplex site shortly. Also, keep in mind that the APIs in preview releases are subject to change.

UPDATE: The MEF product manager, Hamilton Verissimo (aka "hammet" and "haveriss"), has blogged about MEF's convention model.

Friday, February 25, 2011

Executing visual studio 2010 unit tests without installing visual studio

In a previous post I already explained how to get your unit tests running on a build machine with mstest.exe, without having to install Visual Studio 2008. We recently upgraded to Visual Studio 2010 so I had to repeat the exercise.

This time I wrote a batch script which does most of the work. Just run the script on a machine where Visual Studio 2010 is installed, and it will create a "mstest" folder with the necessary binaries and registry entries. Copy the folder to your build machine, prepare the registry by executing mstest.reg and you're good to run mstest.exe like this:

mstest /noisolation /testcontainer:"path\to\testproject.dll"

This will return an error code if any of the tests fail.

As an anonymous commenter on my previous post pointed out, you should be careful of the license implications. In our case it is most likely OK because we have a site license. But even so it can be useful to avoid a full Visual Studio install: it saves a lot of disk space, and you don't have to spend time babysitting the installation. Or multiple installations, if you have a cluster of build machines!

update: thanks to Frederic Torres for pointing out that the script doesn't work on a 64-bit Windows, and suggesting a fix! It should work now. Apparently there are still problems with 64-bit machines. But I don't have a 64-bit machine without VS2010 to test with, so I'll have to leave it as an exercise for the reader...

Wednesday, December 1, 2010

bisect your source code history to find the problematic revision

I am working on a little application to edit Markdown documents with instant preview. (For those of you who aren't into building things from source, here's a windows installer to put a downmarker shortcut in your start menu.) I like to use both windows and linux so I try to make sure that it runs on both Microsoft.NET and Mono.

When I last tested my application on Mono, I discovered a very annoying problem: each time I tried to type something in a markdown document, a new nautilus file browser window would be launched! I had absolutely no idea why this would happen or where to start debugging. To make matters worse, I hadn't tested on Mono for about 40 revisions so there were a lot of possible changes that might have introduced the problem.

I have the source code history in a mercurial repository, so I decided to give the "bisect" feature a try. I started by marking the latest revision as "known bad":

hg bisect --bad

I also remembered making some mono-specific bug fixes, and the problem didn't exist at that point. So I grepped the output of hg log for commit messages containing the word "mono" and marked the revision as good:

hg bisect --good 627d6

At this point the bisect command has a revision range to work with. It will automatically update your working copy to a revision right in the middle of that range. You just have to rebuild your application, verify whether the problem is still there, and mark the revision as good or bad. (It is also possible to do this test automatically with a script.) The bisect command will then automatically cut the revision range to investigate in half again, and select yet another revision in the middle, etcetera.

Sure enough, after about 5 or 6 tests I got this message:

The first bad revision is:
changeset:   42:c4bbabe79dde
user:        wcoenen
date:        Sun Nov 07 23:31:32 2010 +0100
summary:     Links are now handled by opening .md or by external application.

Aha! Looks like I added some code to handle link clicks by passing the URL to the OS (to open it with a suitable external application). And obviously this code is now being triggered erroneously on mono whenever the document is updated. Thank you bisect!

Wednesday, November 24, 2010

git error: error setting certificate verify locations

I was trying to clone a repository from github when I ran into this error:
Cloning into docu...
error: error setting certificate verify locations:
  CAfile: \bin\curl-ca-bundle.crt
  CApath: none
 while accessing https://github.com/jagregory/docu/info/refs

If you google around, many people "solve" this by disabling the SSL certificate check entirely. Obviously there is a reason for that check, so disabling it is not quite the right solution! It turns out that there is mistake in the gitconfig file that comes with msgit setup (I have Git-1.7.2.3 installed). The right fix is to change the sslCAinfo setting in "c:\program files\git\etc\gitconf" to this:

sslCAinfo = c:\\program files\\git\\bin\\curl-ca-bundle.crt

Tuesday, November 23, 2010

Mercurial and subversion subrepos

It is not yet mentioned in the Mercurial book, but Mercurial has a subrepos feature to pull code from other repositories into your project as a nested repository. It is a bit similar to SVN externals and git submodules.

Better yet, it also works with subversion! There are still some bugs to be worked out though: you better not move your SVN subrepo around in your mercurial repository. For all the ugly details, see my bug report.

Friday, November 19, 2010

An equivalent of .bashrc for the windows cmd.exe shell

Tired of the "cd" command in windows refusing to navigate to folders on other drives? Put this in the [HKEY_CURRENT_USER\Software\Microsoft\Command Processor] AutoRun key:

c:\cmdauto.cmd

This will cause the given script to be executed each time a cmd.exe window is opened. I found out about that thanks to this superuser.com post.

Now put this in c:\cmdauto.cmd

@echo off
doskey cd=pushd $*

The doskey command creates aliases, here overriding the behavior of "cd".

Wednesday, June 23, 2010

ZFS-Fuse reliability report

A few weeks ago, I got this question in the comments on my last NAS post:
Since you've been using ZFS-fuse for a time now, can you report here or blog about its stability? Has the daemon crashed on you, while taking a snapshot, or while "scrub"-ing?

How much data have you passed in?

First, note that the version of ZFS-FUSE which I am using is not a particular release, but this revision from the official git repository. I have not bothered to update my ZFS binaries since January, because I didn't encounter any problems.

The "zfs list" output for my pool shows that I am using 352 GiB. The pool consists of two 465 GiB disks in a mirror setup.
NAME               USED  AVAIL  REFER  MOUNTPOINT
nas-pool           352G   105G    24K  /nas-pool
nas-pool/archive   118G   105G  71.1G  /nas-pool/archive
nas-pool/bulk      234G   105G   228G  /nas-pool/bulk

The pool is scrubbed by a cron script each sunday night. I have not seen any crashes/hangs during scrubbing. The scrubs have not detected any errors so far.

"archive" is snapshotted every night. "bulk" is snapshotted" only on sunday nights. The pool contains more than a hundred snapshots. Again, I have not seen any crashes or hangs.

Then there is read/write activity: I use it to automatically archive my emails from Gmail (nightly), sync with my dropbox folder (continuously), and to store my mp3s, photos, videos, downloads and backups when I need to. When reading or writing NAS files, I am usually working from my laptop over WiFi which is a speed bottleneck, so this probably doesn't really stress the file system. On the other hand, I do regularly make a full off-line backup of the "archive" filesystem by hooking up an external USB disk to the NAS. I haven't seen ZFS crash during any of this.

From this, I conclude that ZFS-Fuse is pretty stable.

Saturday, May 29, 2010

Exponential growth has limits

I saw this comment in the thread of a slashdot post about Wall Street today:

But everything thus far shows us that perpetual growth is possible. Technology is a wonderful thing - each year we're able to do more with less.

That's not to say that a lot of what goes on in the market isn't pure, unadulterated bullshit, but real, honest-to-goodness "growth" won't stop until technology does.

This is a textbook example of the "cornucopian" view on economics. The implicit assumption here is that there are no limits to technological improvements, or that those limits are extremely far in our future. I think that this assumption is flat out wrong and dangerous. It is these sorts of ideas which have lead large parts of the global economy to eerily resemble Ponzi Schemes.

I wrote the following post in reply:

There are limits to exponential growth. (And make no mistake, growth expressed as a fixed percentage per year is exponential). Technology can push the limits closer to what the laws of physics allow, but technology cannot change the laws of physics.

Let's look at some numbers to drive the point home. Our global energy consumption in 2008 was estimated to be 474 exajoules.

The total energy received by the earth from the sun during a year is about 5 million exajoules, a fraction of which reaches the surface. 5 million is much more than 474. But at a seemingly modest 2% per year growth rate (as it was between 1980 and 2006), our energy consumption will match those 5 million exajoules in less than 500 years!

Think about that: if energy consumption growth continues at the current pace, then in 500 years we'll either be using ALL solar energy received by the earth (leaving none for the biosphere), or we'll have figured out some magic technology to produce 5 million exajoules of energy per year. Assuming the magic technology, where are we going to get rid of all that extra heat? It would effectively be like having a second sun on earth, cooking us in place.

Granted, you did say "do more with less". So lets say energy consumption will stay constant in the future, and instead we'll derive 2% more "value" from the same energy each year. Now you run into a new problem. No matter how you define "value", you run into physical limits. If you define value as "amount of mass lifted out of the earth's gravity field", then the hard efficiency limit is a minimum of 60 megajoules per kg. If you define value as "amount of computation", then again there are limits given by the laws of physics.

Exponential growth is counterintuitive. No matter how far you push the limits (e.g. by colonizing the entire galaxy or inventing game-changing technology), exponential growth will hit its limits much faster than you think. We're talking about growth with a fixed doubling period here.

Finally, I'd argue that we are already experiencing the end of exponential growth today. After decades of growth, in 2004 global oil production reached a plateau [theoildrum.com]. It's not a coincidence that we experienced a major financial crash and recession soon after that. The era of "perpetual growth" is over. The next era will be that of the "zero-sum game" at best.

On a related note, after writing the above post I stumbled on this excellent series of videos of a lecture by Dr. Albert A. Bartlett where he shows with some very simple calculations and examples that "steady growth" is in fact always unsustainable. We have to find a way to get our civilization to work with 0% growth. The alternative is total collapse, probably within decades.

Saturday, April 3, 2010

Building a dependency injection container in 30 lines

After reading this article by Josh Smith explaining the Service Locator pattern, I commented that direct dependency injection might be a better idea. Mark Seemann has a good write up about why Service Locator is an anti-Pattern, and I'm inclined to agree with him.

When Josh replied that injecting dependencies with constructor arguments doesn't really solve the problem of dependency creation, I was tempted to reply by enumerating all the .NET dependency injection frameworks that exist for exactly this purpose.

But then I realized that Josh had demonstrated the Service Locator pattern without using any framework. Instead, his article has a ServiceContainer class of about 30 lines. Service Locator has many disadvantages, but apparently it can be quite lightweight!

This then lead me to wonder if the same could be done for creating a dependency injection framework. Ayende has actually already demonstrated that you can create a primitive one in 15 lines, but I was thinking of something that could be used with a more friendly Ninject-esque syntax like this:

var container = new Container();
container.Bind<App, App>();
container.Bind<IFoo, Foo>();
container.Bind<IBar, Bar>();

var app = container.Pull<App>();
app.Run();

As it turns out, implementing a bare bones container which can do that is really not that hard. It also has the advantage that it takes care of the dependencies of the dependencies etcetera, something which Josh's sample doesn't seem to do. (Disclaimer: I didn't really test this for anything but the plain vanilla use case, no error conditions were considered.)

public class Container
{
    private readonly Dictionary<Type, Type> contractToClassMap = new Dictionary<Type, Type>();
    private readonly Dictionary<Type, object> contractToInstanceMap = new Dictionary<Type, object>();

    public void Bind<TContract, TClass>() where TClass : class, TContract
    {
        this.contractToClassMap[typeof(TContract)] = typeof(TClass);
    }

    public TContract Pull<TContract>()
    {
        return (TContract)Pull(typeof(TContract));
    }

    public object Pull(Type contract)
    {
        object instance;
        this.contractToInstanceMap.TryGetValue(contract, out instance);
        if (instance == null)
        {
            var constructor = contractToClassMap[contract].GetConstructors()[0];
            var args = 
                from parameter in constructor.GetParameters() 
                select Pull(parameter.ParameterType);
            instance = constructor.Invoke(args.ToArray());
            this.contractToInstanceMap[contract] = instance;
        }
        return instance;
    }
}

Thursday, April 1, 2010

Adding some compiler verification to PropertyChanged events

I've been playing around with Windows Presentation Foundation and the Model-View-ViewModel pattern in the past week. Josh Smith's introductory article does a good job explaining how the MVVM pattern can be used used in WPF.

One aspect of the pattern is that your view model needs to provide change notifications, for example by implementing the INotifyPropertyChanged interface. To avoid repeating the same code, it might be a good idea to implement this in a shared base class:

public abstract class ViewModelBase : INotifyPropertyChanged
   {
      protected virtual void OnPropertyChanged(string propertyName)
      {
         if (PropertyChanged != null)
         {
            PropertyChanged(this, new PropertyChangedEventArgs(propertyName));
         }
      }

      public event PropertyChangedEventHandler PropertyChanged;
   }

There is a problem with this though: the caller of OnPropertyChanged can pass any string. To constrain this to strings that match a property name, we can use reflection to check whether such a property indeed exists. The sample in Josh Smith's introductory article takes this approach. That way, passing an invalid string will at least raise an exception at run-time.

However, we can still do one better and catch such errors at compile time, which is a huge advantage during refactorings. The solution involves two advanced C# tricks. The first trick makes use of the fact that the C# compiler can convert lambdas into expression trees, which can then be inspected to extract the name of a property:

Foo foo = new Foo();
    string propertyName = GetPropertyNameFromExpression<Foo,int>(x => x.Bar);
    Debug.Assert(propertyName == "Bar");

The GetPropertyNameFromExpression method is implemented like this:

private string GetPropertyNameFromExpression<TClass,TProperty>(
         Expression<Func<TClass, TProperty>> expression)
      {
         var memberExpression = expression.Body as MemberExpression;
         if (memberExpression == null)
            throw new ArgumentException(String.Format(
               "'{0}' is not a member expression", expression.Body));
         return memberExpression.Member.Name;
      }

But how do we get that TClass type parameter in our base class? That's our second trick: as it turns out, it is possible for a base class to have a type parameter representing its derived classes. The resulting base class looks like this:

public abstract class ViewModelBase<TDerived> : INotifyPropertyChanged
       where TDerived : ViewModelBase<TDerived>
    {

        private string GetPropertyNameFromExpression<TPropertyType>(
           Expression<Func<TDerived, TPropertyType>> expression)
        {
            var memberExpression = expression.Body as MemberExpression;
            if (memberExpression == null)
                throw new ArgumentException(String.Format(
                    "'{0}' is not a member expression", expression.Body));
            return memberExpression.Member.Name;
        }

        /// <summary>
        /// Triggers the <see cref="PropertyChanged"/> event for the property used in
        /// <paramref name="expression"/>
        /// </summary>
        /// <param name="expression">
        /// A simple member expression which uses the property to trigger the event for, e.g.
        /// <c>x => x.Foo</c> will raise the event for the property "Foo".
        /// </param>
        protected void OnPropertyChanged<TPropertyType>(
            Expression<Func<TDerived,TPropertyType>> expression)
        {
            string name = GetPropertyNameFromExpression(expression);
            if (PropertyChanged != null)
                PropertyChanged(this, new PropertyChangedEventArgs(name));
        }

        public event PropertyChangedEventHandler PropertyChanged;
    }

The viewmodel implementations then look like this:

public class FooViewModel : ViewModelBase<FooViewModel>
   {
      private int bar;

      public int Bar
      {
         get
         {
            return this.bar;
         }
         set
         {
            this.bar = value;
            OnPropertyChanged(x => x.Bar);
         }
      }
   }

It is now much harder to trigger a PropertyChanged event with the wrong property name, as the compiler will verify that the property actually exists. Better yet, if we do a "rename" refactoring of our properties then the IDE will also do the rename in the OnPropertyChanged lambdas. Hurray! :-)

Sunday, March 28, 2010

The General Public License doesn't always force you to make your code available

I just answered this question on stackoverflow asking for "ways around the GPL". While the motives of the poster seem questionable, this did prompt me to explain a subtlety of the General Public License which he may not have understood: the GPL does not automatically force you to release your code as soon as you use GPL'ed code.

The GPL only requires that you make the code available to anyone you distribute the software to. This is not exactly the same as releasing it to the whole world for free. From the GPL FAQ:

Does the GPL require that source code of modified versions be posted to the public?

The GPL does not require you to release your modified version, or any part of it. You are free to make modifications and use them privately, without ever releasing them. This applies to organizations (including companies), too; an organization can make a modified version and use it
internally without ever releasing it outside the organization.

But if you release the modified version to the public in some way, the GPL requires you to make the modified source code available to the program'susers, under the GPL.

Thus, the GPL gives permission to release the modified program in certain ways, and not in other ways; but the decision of whether to release it is up to you.

So if you use GPL code for building an internal tool which you will never distribute to third parties, then there is no requirement to distribute the source to anyone.

Interestingly enough, the above subtlety also applies if you only make the software available as a web application hosted on your own web servers. Since you aren't technically distributing the application to the users, you don't have to give them the code. The Affero General Public License (AGPL) was designed specifically to close this loophole.

Monday, February 22, 2010

Creating a web feed for a file system directory

I just wrote a little python script which generates a web feed for a folder with text files. When run, the script detects the 10 last added/changed files and outputs them as entries in a feed file. This makes it possible to easily create a web feed with just some shell scripting.

To use the script:
  • download it here and make it executable
  • edit the configuration values at the start of the script
  • execute the script regularly, e.g. from a cron job
  • if not generating the file directly on a web server, publish the feed file on the web (e.g. with scp or just write it to your public dropbox folder)
I have used the W3C feed validation service to check that the resulting file is a valid atom syndication feed. However, the configuration values are important for the validation so check that the generated feed still validates after configuring.

Tuesday, February 16, 2010

Building a NAS, part 7: ZFS snapshots, scrubbing and error reporting

Setting up a ZFS pool with redundancy can only protect you against disk failures. To protect yourself against accidental deletions or modifications of files, you can use snapshots. You also need to explicitly start a ZFS data scrub at regular intervals to make sure that any checksum failures are repaired. Such things are best automated, but you might still want to receive reports so that you can keep an eye on things.

Automated snapshots

Setting up automated snapshots for ZFS-FUSE on debian is surprisingly easy. Drop this script in /etc/cron.daily/:
#!/bin/bash
zfs snapshot mypool/myfilesystem@`date +%Y.%m.%d:%H.%M`.auto
This will automatically create daily snapshots with a name like 2010.02.02:06.25.auto. Note that this will complicate things if you need to delete stuff to make room. As long as there is a snapshot referencing a file, it will continue to take space in the pool. Daily snapshots work best for a grow-only archive where you rarely need to delete something.

A word of warning: the scripts in /etc/cron.daily are only executed if they are executable and have no dots in their name. See man run-parts for more details. Test with /etc/cron.hourly to verify that everything works, then move the script to /etc/cron.daily.

Automated scrubbing

A ZFS pool can repair its checksum errors (if there is redundant storage) while still remaining on-line. This is called a scrub. The recommended scrub interval for consumer grade disks is one week. Drop this script in /etc/cron.weekly:
#!/bin/bash
zpool scrub mypool

Web feed reporting

A report of the scrub progress or the results of the last scrub can be shown with the zpool status command. A list of all file systems and snapshots (including some useful statistics) can be shown with the zfs list -t all command. To automate the reporting, I use this script in a cron job:

#!/bin/bash
reportfile=/root/poolreports/`date +%Y.%m.%d:%H.%M`.txt
date > ${reportfile}
zpool status nas-pool 2>&1 >> ${reportfile}
zfs list -t all 2>&1 >> ${reportfile}
I then generate a web feed for the /root/poolreports/ folder as I explained in my previous post and follow the feed with google reader.

Thursday, February 11, 2010

Debugging: Why is the Tick event of my WinForms timer no longer raised

I was debugging an issue at work today were a WinForms Timer object was apparently no longer firing Tick events as it was supposed to.

It turned out that the timer was inadvertently being used by a BackgroundWorker thread. Like most classes, winforms timers are not thread safe so any behavior guarantees are out the window as soon as you start accessing them from different threads without synchronization measures.

Worse, winforms timers interact with the main application thread directly so in this case it is not possible to put such synchronization measures in place. I like to call such classes thread-hostile. Another sure way to create thread-hostile code is to use global variables; we have our fair share of such problems in our legacy code base.

The following sample reproduces the timer problem by accessing a timer from a ThreadPool worker thread; the timer will only be fired once instead of indefinitely as you might expect:

public partial class Form1 : Form
   {
      private System.Windows.Forms.Timer fTimer;

      public Form1()
      {
         InitializeComponent();
         fTimer = new System.Windows.Forms.Timer();
         fTimer.Interval = 1000;
         fTimer.Tick += HandleTimerTick;
         fTimer.Start();
      }

      private void HandleTimerTick(object sender, EventArgs args)
      {
         // sabotage timer by stopping/starting it from another thread
         ThreadPool.QueueUserWorkItem(
            delegate
            {
               fTimer.Stop();
               fTimer.Start();
            } );

         MessageBox.Show("Timer tick");
      }
   }

In our case, the worker thread touched the timer in a much more indirect way: the background task was using a service which leaked side effects into the rest of the system via events, resulting in inadvertent multi-threaded access all over the place.

Conclusion: if you are going to do multi-threading, make sure threads are well-isolated and only communicate with the rest of the system via well defined synchronization points.

Wishlist item: wouldn't it be nice if you had to explicitly mark methods before they could be used by multiple threads? The C# compiler could then generate optional checks that make your code fail fast when there is an accidental "threading leak". It wouldn't surprise me if the language will actually grow such a debugging feature in the future; multi-threaded .NET programming is on the rise yet still wildly dangerous.

Thursday, February 4, 2010

OpenID: great standard, many poor implementations

I was browsing slashdot the other day, and noticed an interesting story that I wanted to upvote, which requires logging in. Interestingly, there's an openid option:


OpenID is a standard that allows you to reuse a single identity on different websites (or any other service that requires an identity). Chances are you already have an OpenID. For example, if you have a google account, then you can use the URL http://www.google.com/profiles/yourusername as an OpenID. There are many more OpenID providers like Yahoo, MyOpenID, AOL, LiveJournal, Wordpress, Blogger, Versign, etcetera.

Currently I have 130 user accounts on the web that I have bothered to keep track of. The idea of OpenID is that you no longer have to create hundreds of accounts, each with their own user name and password (or worse, the same password). You just enter your OpenID, and the OpenID provider takes care of authenticating you.

Stackoverflow gets it right

For an example of OpenID done right, try the stackoverflow login page. See how easy that was? No passwords, no confirmation mails, just reuse your existing identity by clicking the icon of your identity provider. As Steve Jobs would say, isn't that wonderful?

Slashdot gets it wrong

Unfortunately, when you log in with your OpenID in slashdot you are greeted by this:


In other words, you still have to create a username and password specifically for slashdot. Worse, even if you do that you still cannot login with just your OpenID. What gives?

Facebook gets it wrong

You can go into your facebook Settings - Account Settings - Linked Accounts - Change - Add Account and enter an OpenID there. If you then log out and try to log back in to test it, there is no OpenID option on the login page. WTF? On a hunch, I then just retyped the facebook URL in my browser address bar and it looked like I was already logged in.

A little more investigation shows that facebook relies on a cookie that links your browser to your OpenID, and tries to log you in transparently with that information. Since I have configured my browser to only keep cookies between browser sessions for a small white-list of websites, this doesn't work for me at all. Even if I add facebook to the white-list, I won't be able to use my OpenID to log in on other computers. FAIL. I guess just putting a "Log in with OpenID" button on the login page would have been too easy.

Dealing with lack of OpenID support

OpenID support is growing, but the majority of web sites still don't support it or implement their support very poorly. Others only support OpenID as an identity provider and refuse to accept identities from other providers.

To deal with all these sites that still require passwords, most people reuse the same password over and over again. This is terrible security. Any of the sites that you use could have a malicious admin that may like to sell username/password combos to the highest bidder. Or maybe the website admin isn't malicous, but the user account database might store passwords unhashed and could be compromised.

Personally I use the cross-platform KeePass application to maintain a personal encrypted database of passwords. The database is protected by a single master password (or passphrase). I put mine in my dropbox folder, so I have access to my passwords on each PC I use. Even better, if you stick to version 1.x the database is compatible with KeePassMobile so you can carry your passwords with you on your phone.

Tuesday, February 2, 2010

MEF: GetExport, GetExportedValue methods compared to Import attribute

(Also posted on the MEF forum)

While writing some automated composition tests today, I found out the hard way that GetExportedValue<Lazy<T>> doesn't do what you might think it does at first sight. For example, the following throws because it tries to get a exported value for the type Lazy<IFoo>, which is not an available part:
   public class Program
   {
      public static void Main(string[] args)
      {
         var catalog = new AssemblyCatalog(Assembly.GetExecutingAssembly());
         var container = new CompositionContainer(catalog);
         var exports = container.GetExportedValue<Lazy<IFoo>>();
         Console.WriteLine(exports.Count());
      }
   }

   public interface IFoo
   {
   }

   [Export(typeof(IFoo))]
   public class Foo : IFoo
   {
   }

As it turns out, if you want to pull a lazy export from a container, you just have to call GetExport<T> or GetExport<T,TMetaData>. This is quite obvious with hindsight, but I just got so used to using the shorthand GetExportedValue<T> that I completely forgot about the existence of GetExport<T>.

This then led me to wonder why the GetExport and GetExportedValue methods work differently from the Import attribute. With the Import attribute, MEF inspects the type you are trying to import and gives special treatment to Lazy<T>. Shouldn't there be an ImportLazy (and ImportManyLazy) attribute instead to make this intention explicit?

Sunday, January 31, 2010

On the emergence of ubiquitous computing

I just read this post about how the iPhone and new iPad herald the era of "new world computing":
In the New World, computers are task-centric. We are reading email, browsing the web, playing a game, but not all at once. Applications are sandboxed, then moats dug around the sandboxes, and then barbed wire placed around the moats. As a direct result, New World computers do not need virus scanners, their batteries last longer, and they rarely crash, but their users have lost a degree of freedom. New World computers have unprecedented ease of use, and benefit from decades of research into human-computer interaction. They are immediately understandable, fast, stable, and laser-focused on the 80% of the famous 80/20 rule.

It is an interesting post, but I don't believe that the lack of multi-tasking and other freedoms is a necessity for "new world computing". If you can make a slick UI for switching between tasks, then you can also make a slick UI for switching between tasks that continue to run in the background.

These limitations are just engineering trade-offs that had to be made to give us an early peek at ubiquitous computing (which is by the way the real term for "new world computing" and was already a research topic long before I went to college). I seriously doubt the next generation of these devices will have the same limitations.

You can wave your hands and talk about how it's all task oriented now all you want, but in the end multi-tasking is a necessity even if only for running a chat client 24/7. And that's exactly what I do with my old and clunky N95 smartphone.

Setting up dropbox on a headless linux system

I have been using dropbox for a while to synchronize files between different computers. It has some pretty impressive bullet points:

  • Seamless syncing. You just put files in the dropbox folder, and they are automatically synchronized to your other computers. No firewall issues. In fact, the only problem I have is at work where dropbox is explicitly blocked. >:-(
  • Easy file sharing over the internet. Just put a file in your public folder, right click, copy public link. You can even host a website on dropbox this way.
  • Cross-platform. It works on linux, windows, OS X and even iPhone.
  • You can access the revision history of your files so it works pretty well as an on-line backup service, even if you delete files by accident.
  • It's completely free if you don't need more than 2GB storage and 30 days of revision history.

I have set up dropbox on my NAS so that I can synchronize my dropbox to a ZFS file system. This way I can combine the advantages of dropbox with the advantages of my NAS:

  • I get to keep snapshots indefinitely, with disk space being my only limitation.
  • I protect my data even if the dropbox service fails disastrously, e.g. because of security breach. Think file deletions being synced to all your computers.
  • I can free space on my dropbox account by moving files on the NAS out of the dropbox folder, yet still keep them safe through my NAS snapshot+backup policy.

Dropbox is targetted at GUI environments, but can still be installed on a headless linux system as described on this wiki page. However, the wiki page did not describe how to change the dropbox folder. I needed this to point dropbox to a folder on my NAS storage pool. It took some minor reverse engineering of the dropbox settings file, but I successfully created a script to do exactly that. I've also added a link and instructions on the wiki on how to use it.