This week I ran into a nasty bug in the Windows Process Activation Service (WAS or WPAS?) on Vista. Here is what happened…

I fired up my little WAS application to verify some behavior with non-HTTP WCF endpoints hosted in WAS. But suddenly I got an EndpointNotFoundException in the client. This used to work before and only affected the non-HTTP endpoints. I had no clue what was going on.

After a little more looking around I noticed that the WAS listener adapter NT services are not started (net.tcp, net.msmq and net.namedpipes as well as the tcp port sharing service). And when I tried to start them I got the following scary error in the event log (in SMSvcHost):

System.AccessViolationException: Attempted to read or write protected memory. This is often an indication that other memory is corrupt.

at System.Runtime.InteropServices.Marshal.ReadByte(IntPtr ptr, Int32 ofs)
at System.Security.Principal.Win32.ConvertIntPtrSidToByteArraySid(IntPtr binaryForm)
at System.Security.Principal.SecurityIdentifier..ctor(IntPtr binaryForm, Boolean noDemand)
at System.Security.Principal.SecurityIdentifier..ctor(IntPtr binaryForm)
at System.ServiceModel.Activation.ListenerAdapterBase.onApplicationPoolCreated(IntPtr context, String appPoolId, IntPtr sid)

With some help from the IIS/WAS team (thanks to Mike for routing me to the right people) we could narrow down the problem – it turned out that this is related to another event log entry I found (from WAS):

Application pool LabPool has been disabled. Windows Process Activation Service (WAS) did not create a worker process to serve the application pool because the application pool identity is invalid.

Now this rang a bell. Some time ago I created an additional application pool that used a custom worker account. After I was finished with my work (and as a good security guy ;) I disabled the worker process account. But I did not delete or disable the application pool.

This was some weeks ago and since I rarely reboot my machine and mostly just put it into sleep mode at night (seems logical ;) everything was fine. Now I did a reboot beginning this week and suddenly all this WAS/SMSvcHost error messages popped up because the application pool could not be started.

Now it turns out that there is a bug in WAS. When a single application pool on the system fails to start, the whole WAS infrastructure dies. Regardless if the application pool is used or not…

Hopefully this post can save you some hours of pulling hair and reading log files…Thanks to all the guys at MS that helped me troubleshoot this!