adrift in the sea of experience

Friday, November 19, 2010

An equivalent of .bashrc for the windows cmd.exe shell

Tired of the "cd" command in windows refusing to navigate to folders on other drives? Put this in the [HKEY_CURRENT_USER\Software\Microsoft\Command Processor] AutoRun key:

c:\cmdauto.cmd

This will cause the given script to be executed each time a cmd.exe window is opened. I found out about that thanks to this superuser.com post.

Now put this in c:\cmdauto.cmd

@echo off
doskey cd=pushd $*

The doskey command creates aliases, here overriding the behavior of "cd".

Wednesday, June 23, 2010

ZFS-Fuse reliability report

A few weeks ago, I got this question in the comments on my last NAS post:
Since you've been using ZFS-fuse for a time now, can you report here or blog about its stability? Has the daemon crashed on you, while taking a snapshot, or while "scrub"-ing?

How much data have you passed in?

First, note that the version of ZFS-FUSE which I am using is not a particular release, but this revision from the official git repository. I have not bothered to update my ZFS binaries since January, because I didn't encounter any problems.

The "zfs list" output for my pool shows that I am using 352 GiB. The pool consists of two 465 GiB disks in a mirror setup.
NAME               USED  AVAIL  REFER  MOUNTPOINT
nas-pool           352G   105G    24K  /nas-pool
nas-pool/archive   118G   105G  71.1G  /nas-pool/archive
nas-pool/bulk      234G   105G   228G  /nas-pool/bulk

The pool is scrubbed by a cron script each sunday night. I have not seen any crashes/hangs during scrubbing. The scrubs have not detected any errors so far.

"archive" is snapshotted every night. "bulk" is snapshotted" only on sunday nights. The pool contains more than a hundred snapshots. Again, I have not seen any crashes or hangs.

Then there is read/write activity: I use it to automatically archive my emails from Gmail (nightly), sync with my dropbox folder (continuously), and to store my mp3s, photos, videos, downloads and backups when I need to. When reading or writing NAS files, I am usually working from my laptop over WiFi which is a speed bottleneck, so this probably doesn't really stress the file system. On the other hand, I do regularly make a full off-line backup of the "archive" filesystem by hooking up an external USB disk to the NAS. I haven't seen ZFS crash during any of this.

From this, I conclude that ZFS-Fuse is pretty stable.

Saturday, May 29, 2010

Exponential growth has limits

I saw this comment in the thread of a slashdot post about Wall Street today:

But everything thus far shows us that perpetual growth is possible. Technology is a wonderful thing - each year we're able to do more with less.

That's not to say that a lot of what goes on in the market isn't pure, unadulterated bullshit, but real, honest-to-goodness "growth" won't stop until technology does.

This is a textbook example of the "cornucopian" view on economics. The implicit assumption here is that there are no limits to technological improvements, or that those limits are extremely far in our future. I think that this assumption is flat out wrong and dangerous. It is these sorts of ideas which have lead large parts of the global economy to eerily resemble Ponzi Schemes.

I wrote the following post in reply:

There are limits to exponential growth. (And make no mistake, growth expressed as a fixed percentage per year is exponential). Technology can push the limits closer to what the laws of physics allow, but technology cannot change the laws of physics.

Let's look at some numbers to drive the point home. Our global energy consumption in 2008 was estimated to be 474 exajoules.

The total energy received by the earth from the sun during a year is about 5 million exajoules, a fraction of which reaches the surface. 5 million is much more than 474. But at a seemingly modest 2% per year growth rate (as it was between 1980 and 2006), our energy consumption will match those 5 million exajoules in less than 500 years!

Think about that: if energy consumption growth continues at the current pace, then in 500 years we'll either be using ALL solar energy received by the earth (leaving none for the biosphere), or we'll have figured out some magic technology to produce 5 million exajoules of energy per year. Assuming the magic technology, where are we going to get rid of all that extra heat? It would effectively be like having a second sun on earth, cooking us in place.

Granted, you did say "do more with less". So lets say energy consumption will stay constant in the future, and instead we'll derive 2% more "value" from the same energy each year. Now you run into a new problem. No matter how you define "value", you run into physical limits. If you define value as "amount of mass lifted out of the earth's gravity field", then the hard efficiency limit is a minimum of 60 megajoules per kg. If you define value as "amount of computation", then again there are limits given by the laws of physics.

Exponential growth is counterintuitive. No matter how far you push the limits (e.g. by colonizing the entire galaxy or inventing game-changing technology), exponential growth will hit its limits much faster than you think. We're talking about growth with a fixed doubling period here.

Finally, I'd argue that we are already experiencing the end of exponential growth today. After decades of growth, in 2004 global oil production reached a plateau [theoildrum.com]. It's not a coincidence that we experienced a major financial crash and recession soon after that. The era of "perpetual growth" is over. The next era will be that of the "zero-sum game" at best.

On a related note, after writing the above post I stumbled on this excellent series of videos of a lecture by Dr. Albert A. Bartlett where he shows with some very simple calculations and examples that "steady growth" is in fact always unsustainable. We have to find a way to get our civilization to work with 0% growth. The alternative is total collapse, probably within decades.

Saturday, April 3, 2010

Building a dependency injection container in 30 lines

After reading this article by Josh Smith explaining the Service Locator pattern, I commented that direct dependency injection might be a better idea. Mark Seemann has a good write up about why Service Locator is an anti-Pattern, and I'm inclined to agree with him.

When Josh replied that injecting dependencies with constructor arguments doesn't really solve the problem of dependency creation, I was tempted to reply by enumerating all the .NET dependency injection frameworks that exist for exactly this purpose.

But then I realized that Josh had demonstrated the Service Locator pattern without using any framework. Instead, his article has a ServiceContainer class of about 30 lines. Service Locator has many disadvantages, but apparently it can be quite lightweight!

This then lead me to wonder if the same could be done for creating a dependency injection framework. Ayende has actually already demonstrated that you can create a primitive one in 15 lines, but I was thinking of something that could be used with a more friendly Ninject-esque syntax like this:

var container = new Container();
container.Bind<App, App>();
container.Bind<IFoo, Foo>();
container.Bind<IBar, Bar>();

var app = container.Pull<App>();
app.Run();

As it turns out, implementing a bare bones container which can do that is really not that hard. It also has the advantage that it takes care of the dependencies of the dependencies etcetera, something which Josh's sample doesn't seem to do. (Disclaimer: I didn't really test this for anything but the plain vanilla use case, no error conditions were considered.)

public class Container
{
    private readonly Dictionary<Type, Type> contractToClassMap = new Dictionary<Type, Type>();
    private readonly Dictionary<Type, object> contractToInstanceMap = new Dictionary<Type, object>();

    public void Bind<TContract, TClass>() where TClass : class, TContract
    {
        this.contractToClassMap[typeof(TContract)] = typeof(TClass);
    }

    public TContract Pull<TContract>()
    {
        return (TContract)Pull(typeof(TContract));
    }

    public object Pull(Type contract)
    {
        object instance;
        this.contractToInstanceMap.TryGetValue(contract, out instance);
        if (instance == null)
        {
            var constructor = contractToClassMap[contract].GetConstructors()[0];
            var args = 
                from parameter in constructor.GetParameters() 
                select Pull(parameter.ParameterType);
            instance = constructor.Invoke(args.ToArray());
            this.contractToInstanceMap[contract] = instance;
        }
        return instance;
    }
}

Thursday, April 1, 2010

Adding some compiler verification to PropertyChanged events

I've been playing around with Windows Presentation Foundation and the Model-View-ViewModel pattern in the past week. Josh Smith's introductory article does a good job explaining how the MVVM pattern can be used used in WPF.

One aspect of the pattern is that your view model needs to provide change notifications, for example by implementing the INotifyPropertyChanged interface. To avoid repeating the same code, it might be a good idea to implement this in a shared base class:

public abstract class ViewModelBase : INotifyPropertyChanged
   {
      protected virtual void OnPropertyChanged(string propertyName)
      {
         if (PropertyChanged != null)
         {
            PropertyChanged(this, new PropertyChangedEventArgs(propertyName));
         }
      }

      public event PropertyChangedEventHandler PropertyChanged;
   }

There is a problem with this though: the caller of OnPropertyChanged can pass any string. To constrain this to strings that match a property name, we can use reflection to check whether such a property indeed exists. The sample in Josh Smith's introductory article takes this approach. That way, passing an invalid string will at least raise an exception at run-time.

However, we can still do one better and catch such errors at compile time, which is a huge advantage during refactorings. The solution involves two advanced C# tricks. The first trick makes use of the fact that the C# compiler can convert lambdas into expression trees, which can then be inspected to extract the name of a property:

Foo foo = new Foo();
    string propertyName = GetPropertyNameFromExpression<Foo,int>(x => x.Bar);
    Debug.Assert(propertyName == "Bar");

The GetPropertyNameFromExpression method is implemented like this:

private string GetPropertyNameFromExpression<TClass,TProperty>(
         Expression<Func<TClass, TProperty>> expression)
      {
         var memberExpression = expression.Body as MemberExpression;
         if (memberExpression == null)
            throw new ArgumentException(String.Format(
               "'{0}' is not a member expression", expression.Body));
         return memberExpression.Member.Name;
      }

But how do we get that TClass type parameter in our base class? That's our second trick: as it turns out, it is possible for a base class to have a type parameter representing its derived classes. The resulting base class looks like this:

public abstract class ViewModelBase<TDerived> : INotifyPropertyChanged
       where TDerived : ViewModelBase<TDerived>
    {

        private string GetPropertyNameFromExpression<TPropertyType>(
           Expression<Func<TDerived, TPropertyType>> expression)
        {
            var memberExpression = expression.Body as MemberExpression;
            if (memberExpression == null)
                throw new ArgumentException(String.Format(
                    "'{0}' is not a member expression", expression.Body));
            return memberExpression.Member.Name;
        }

        /// <summary>
        /// Triggers the <see cref="PropertyChanged"/> event for the property used in
        /// <paramref name="expression"/>
        /// </summary>
        /// <param name="expression">
        /// A simple member expression which uses the property to trigger the event for, e.g.
        /// <c>x => x.Foo</c> will raise the event for the property "Foo".
        /// </param>
        protected void OnPropertyChanged<TPropertyType>(
            Expression<Func<TDerived,TPropertyType>> expression)
        {
            string name = GetPropertyNameFromExpression(expression);
            if (PropertyChanged != null)
                PropertyChanged(this, new PropertyChangedEventArgs(name));
        }

        public event PropertyChangedEventHandler PropertyChanged;
    }

The viewmodel implementations then look like this:

public class FooViewModel : ViewModelBase<FooViewModel>
   {
      private int bar;

      public int Bar
      {
         get
         {
            return this.bar;
         }
         set
         {
            this.bar = value;
            OnPropertyChanged(x => x.Bar);
         }
      }
   }

It is now much harder to trigger a PropertyChanged event with the wrong property name, as the compiler will verify that the property actually exists. Better yet, if we do a "rename" refactoring of our properties then the IDE will also do the rename in the OnPropertyChanged lambdas. Hurray! :-)

Sunday, March 28, 2010

The General Public License doesn't always force you to make your code available

I just answered this question on stackoverflow asking for "ways around the GPL". While the motives of the poster seem questionable, this did prompt me to explain a subtlety of the General Public License which he may not have understood: the GPL does not automatically force you to release your code as soon as you use GPL'ed code.

The GPL only requires that you make the code available to anyone you distribute the software to. This is not exactly the same as releasing it to the whole world for free. From the GPL FAQ:

Does the GPL require that source code of modified versions be posted to the public?

The GPL does not require you to release your modified version, or any part of it. You are free to make modifications and use them privately, without ever releasing them. This applies to organizations (including companies), too; an organization can make a modified version and use it
internally without ever releasing it outside the organization.

But if you release the modified version to the public in some way, the GPL requires you to make the modified source code available to the program'susers, under the GPL.

Thus, the GPL gives permission to release the modified program in certain ways, and not in other ways; but the decision of whether to release it is up to you.

So if you use GPL code for building an internal tool which you will never distribute to third parties, then there is no requirement to distribute the source to anyone.

Interestingly enough, the above subtlety also applies if you only make the software available as a web application hosted on your own web servers. Since you aren't technically distributing the application to the users, you don't have to give them the code. The Affero General Public License (AGPL) was designed specifically to close this loophole.

Monday, February 22, 2010

Creating a web feed for a file system directory

I just wrote a little python script which generates a web feed for a folder with text files. When run, the script detects the 10 last added/changed files and outputs them as entries in a feed file. This makes it possible to easily create a web feed with just some shell scripting.

To use the script:
  • download it here and make it executable
  • edit the configuration values at the start of the script
  • execute the script regularly, e.g. from a cron job
  • if not generating the file directly on a web server, publish the feed file on the web (e.g. with scp or just write it to your public dropbox folder)
I have used the W3C feed validation service to check that the resulting file is a valid atom syndication feed. However, the configuration values are important for the validation so check that the generated feed still validates after configuring.