adrift in the sea of experience

Monday, February 22, 2010

Creating a web feed for a file system directory

I just wrote a little python script which generates a web feed for a folder with text files. When run, the script detects the 10 last added/changed files and outputs them as entries in a feed file. This makes it possible to easily create a web feed with just some shell scripting.

To use the script:
  • download it here and make it executable
  • edit the configuration values at the start of the script
  • execute the script regularly, e.g. from a cron job
  • if not generating the file directly on a web server, publish the feed file on the web (e.g. with scp or just write it to your public dropbox folder)
I have used the W3C feed validation service to check that the resulting file is a valid atom syndication feed. However, the configuration values are important for the validation so check that the generated feed still validates after configuring.

Tuesday, February 16, 2010

Building a NAS, part 7: ZFS snapshots, scrubbing and error reporting

Setting up a ZFS pool with redundancy can only protect you against disk failures. To protect yourself against accidental deletions or modifications of files, you can use snapshots. You also need to explicitly start a ZFS data scrub at regular intervals to make sure that any checksum failures are repaired. Such things are best automated, but you might still want to receive reports so that you can keep an eye on things.

Automated snapshots

Setting up automated snapshots for ZFS-FUSE on debian is surprisingly easy. Drop this script in /etc/cron.daily/:
#!/bin/bash
zfs snapshot mypool/myfilesystem@`date +%Y.%m.%d:%H.%M`.auto
This will automatically create daily snapshots with a name like 2010.02.02:06.25.auto. Note that this will complicate things if you need to delete stuff to make room. As long as there is a snapshot referencing a file, it will continue to take space in the pool. Daily snapshots work best for a grow-only archive where you rarely need to delete something.

A word of warning: the scripts in /etc/cron.daily are only executed if they are executable and have no dots in their name. See man run-parts for more details. Test with /etc/cron.hourly to verify that everything works, then move the script to /etc/cron.daily.

Automated scrubbing

A ZFS pool can repair its checksum errors (if there is redundant storage) while still remaining on-line. This is called a scrub. The recommended scrub interval for consumer grade disks is one week. Drop this script in /etc/cron.weekly:
#!/bin/bash
zpool scrub mypool

Web feed reporting

A report of the scrub progress or the results of the last scrub can be shown with the zpool status command. A list of all file systems and snapshots (including some useful statistics) can be shown with the zfs list -t all command. To automate the reporting, I use this script in a cron job:

#!/bin/bash
reportfile=/root/poolreports/`date +%Y.%m.%d:%H.%M`.txt
date > ${reportfile}
zpool status nas-pool 2>&1 >> ${reportfile}
zfs list -t all 2>&1 >> ${reportfile}
I then generate a web feed for the /root/poolreports/ folder as I explained in my previous post and follow the feed with google reader.

Thursday, February 11, 2010

Debugging: Why is the Tick event of my WinForms timer no longer raised

I was debugging an issue at work today were a WinForms Timer object was apparently no longer firing Tick events as it was supposed to.

It turned out that the timer was inadvertently being used by a BackgroundWorker thread. Like most classes, winforms timers are not thread safe so any behavior guarantees are out the window as soon as you start accessing them from different threads without synchronization measures.

Worse, winforms timers interact with the main application thread directly so in this case it is not possible to put such synchronization measures in place. I like to call such classes thread-hostile. Another sure way to create thread-hostile code is to use global variables; we have our fair share of such problems in our legacy code base.

The following sample reproduces the timer problem by accessing a timer from a ThreadPool worker thread; the timer will only be fired once instead of indefinitely as you might expect:

public partial class Form1 : Form
   {
      private System.Windows.Forms.Timer fTimer;

      public Form1()
      {
         InitializeComponent();
         fTimer = new System.Windows.Forms.Timer();
         fTimer.Interval = 1000;
         fTimer.Tick += HandleTimerTick;
         fTimer.Start();
      }

      private void HandleTimerTick(object sender, EventArgs args)
      {
         // sabotage timer by stopping/starting it from another thread
         ThreadPool.QueueUserWorkItem(
            delegate
            {
               fTimer.Stop();
               fTimer.Start();
            } );

         MessageBox.Show("Timer tick");
      }
   }

In our case, the worker thread touched the timer in a much more indirect way: the background task was using a service which leaked side effects into the rest of the system via events, resulting in inadvertent multi-threaded access all over the place.

Conclusion: if you are going to do multi-threading, make sure threads are well-isolated and only communicate with the rest of the system via well defined synchronization points.

Wishlist item: wouldn't it be nice if you had to explicitly mark methods before they could be used by multiple threads? The C# compiler could then generate optional checks that make your code fail fast when there is an accidental "threading leak". It wouldn't surprise me if the language will actually grow such a debugging feature in the future; multi-threaded .NET programming is on the rise yet still wildly dangerous.

Thursday, February 4, 2010

OpenID: great standard, many poor implementations

I was browsing slashdot the other day, and noticed an interesting story that I wanted to upvote, which requires logging in. Interestingly, there's an openid option:


OpenID is a standard that allows you to reuse a single identity on different websites (or any other service that requires an identity). Chances are you already have an OpenID. For example, if you have a google account, then you can use the URL http://www.google.com/profiles/yourusername as an OpenID. There are many more OpenID providers like Yahoo, MyOpenID, AOL, LiveJournal, Wordpress, Blogger, Versign, etcetera.

Currently I have 130 user accounts on the web that I have bothered to keep track of. The idea of OpenID is that you no longer have to create hundreds of accounts, each with their own user name and password (or worse, the same password). You just enter your OpenID, and the OpenID provider takes care of authenticating you.

Stackoverflow gets it right

For an example of OpenID done right, try the stackoverflow login page. See how easy that was? No passwords, no confirmation mails, just reuse your existing identity by clicking the icon of your identity provider. As Steve Jobs would say, isn't that wonderful?

Slashdot gets it wrong

Unfortunately, when you log in with your OpenID in slashdot you are greeted by this:


In other words, you still have to create a username and password specifically for slashdot. Worse, even if you do that you still cannot login with just your OpenID. What gives?

Facebook gets it wrong

You can go into your facebook Settings - Account Settings - Linked Accounts - Change - Add Account and enter an OpenID there. If you then log out and try to log back in to test it, there is no OpenID option on the login page. WTF? On a hunch, I then just retyped the facebook URL in my browser address bar and it looked like I was already logged in.

A little more investigation shows that facebook relies on a cookie that links your browser to your OpenID, and tries to log you in transparently with that information. Since I have configured my browser to only keep cookies between browser sessions for a small white-list of websites, this doesn't work for me at all. Even if I add facebook to the white-list, I won't be able to use my OpenID to log in on other computers. FAIL. I guess just putting a "Log in with OpenID" button on the login page would have been too easy.

Dealing with lack of OpenID support

OpenID support is growing, but the majority of web sites still don't support it or implement their support very poorly. Others only support OpenID as an identity provider and refuse to accept identities from other providers.

To deal with all these sites that still require passwords, most people reuse the same password over and over again. This is terrible security. Any of the sites that you use could have a malicious admin that may like to sell username/password combos to the highest bidder. Or maybe the website admin isn't malicous, but the user account database might store passwords unhashed and could be compromised.

Personally I use the cross-platform KeePass application to maintain a personal encrypted database of passwords. The database is protected by a single master password (or passphrase). I put mine in my dropbox folder, so I have access to my passwords on each PC I use. Even better, if you stick to version 1.x the database is compatible with KeePassMobile so you can carry your passwords with you on your phone.

Tuesday, February 2, 2010

MEF: GetExport, GetExportedValue methods compared to Import attribute

(Also posted on the MEF forum)

While writing some automated composition tests today, I found out the hard way that GetExportedValue<Lazy<T>> doesn't do what you might think it does at first sight. For example, the following throws because it tries to get a exported value for the type Lazy<IFoo>, which is not an available part:
   public class Program
   {
      public static void Main(string[] args)
      {
         var catalog = new AssemblyCatalog(Assembly.GetExecutingAssembly());
         var container = new CompositionContainer(catalog);
         var exports = container.GetExportedValue<Lazy<IFoo>>();
         Console.WriteLine(exports.Count());
      }
   }

   public interface IFoo
   {
   }

   [Export(typeof(IFoo))]
   public class Foo : IFoo
   {
   }

As it turns out, if you want to pull a lazy export from a container, you just have to call GetExport<T> or GetExport<T,TMetaData>. This is quite obvious with hindsight, but I just got so used to using the shorthand GetExportedValue<T> that I completely forgot about the existence of GetExport<T>.

This then led me to wonder why the GetExport and GetExportedValue methods work differently from the Import attribute. With the Import attribute, MEF inspects the type you are trying to import and gives special treatment to Lazy<T>. Shouldn't there be an ImportLazy (and ImportManyLazy) attribute instead to make this intention explicit?