CodeBetter.Com
CodeBetter.Com
RSS 2.0 via Feedburner
           Do you Twitter? Follow us @CodeBetter

Steve Hebert's Development Blog

Steve's Blog - From .Net to dotMath and everything in between.

December 2005 - Posts

  • MS Windows Services for Unix + Client for NFS + EMC = Kernel Memory Leak

    Here's a topic I thought I'd lend a little google juice since Microsoft has created a hotfix.  This problem is nasty - difficult to diagnose and difficult to track down. 

    We have an application where we are sharing a NAS device with Unix servers.  We were seeing in our pre-production and production environments that a group of Windows 2000 boxes would go belly up after a week of use.  These machines would gradually become slow and suddenly unable to communicate on the network.  When looking at the event logs, the system stopped communicating over the network and showed repeated errors.  It looked like someone tripped over a network cable whenever these systems went down.  To make diagnosis worse, I did not have physical access to the boxes - only VNC access.  Here's the path to diagnosing the problem, perhaps this will save someone else some time.

    After a few crashes we saw that socket creation was being denied because resources were low.  This led us to looking at System PTEs.  Once we were focused on the System PTEs, we monitored system PTEs in perfmon and saw that the leak didn't start happening for 4 hours, but then steadily declined on rate loosely tied to traffic volume.  Without any traffic we would see PTEs decrease at a rate of ~ 5/hour, with traffic we saw a range from 60-100 PTEs per hour.  The PTEs always decremented in blocks of 10.

    At this point we weren't sure what was causing it - typically a driver because these consume kernel memory.  After spending a couple of days trying to track this down, we found that the Windows Services for Unix were at fault.  We contacted MS support and they shipped us the hotfix.  The problem disappeared and we haven't seen the behavior since.

    I find it hard to believe that Microsoft has had this product in the field for so long and only now they see this critical of a leak. For some reason we only saw the problem with our EMC/NFS connection.  We have a solaris/NFS connection in development that has never exhibited the problem. I guess Microsoft doesn't test wsFU against small 3rd party vendors like EMC. </sarcasm>  We spent a ton of time tracking this problem and questioned everything on these systems. It's interesting to note that the problem also happens in Windows 2003, but because 2K3 always has significantly more System PTEs than Win2k the box will take much longer to fail.


  • thread safety and static variable instantiation

    A coworker asked me a question today about initialization of static member variables and how they relate to static constructors.   He also told me about an article he read that the following code isn't safe:

     

    if( _myObj == null )

    {

                Lock( _lockObj )

                {

                            If( _myObj ==  null )

                                        _myObj = new MyObject(…);

                }

    }

     

    He definitely had my attention ... He passed along an article on implementing a thread-safe singleton pattern in C# and the article is a very interesting read.  To be fair up front, the above code works in the microsoft implementation (according to Chriss Brumme and discussed later), but the ECMA spec is vague on how instantiation must take place.  On the other hand, the above code is explicitly NOT threadsafe in Java.  The article also discusses the way in which the .Net runtime chooses to instantiate static member variables and this section was news to me.

     
    Double Locking and Thread Safety

    The above code looks right until you read Chris Brumme's Memory Model blog entry.  The excerpt that finally makes sense after reading this a couple of times is:

     

    ... (W)e have to assume that a series of stores have taken place during construction of 'a'.   Those stores can be arbitrarily reordered, including the possibility of delaying them until after the publishing store which assigns the new object to 'a'.   At that point, there is a small window before the store.release implied by leaving the lock.  Inside that window, other CPUs can navigate through the reference 'a' and see a partially constructed instance.

     

    So what does this mean?   In the assignment of a new operator, this piece of code could be broken by a weak but valid implementation of the ECMA spec.   Given the following line of code:

     

    _myObj = new MyObject();

     

    We expect that the underlying pseudo code looks something like this:

     

    Allocate MyObject memory and assign address to tempObj

    Call Constructor on tempObj

    assign tempObj to _myObj

     

    Given the state of the ECMA spec, a valid implementation could be this:

     

    Allocate MyObject memory and assign it to _myObj

    Call Constructor on _myObj

     

    In this second implementation, consider what happens if the thread holding the lock gets preempted during the Constructor call.   A thread that doesn't hold the lock believes the construction of _myObj is complete and proceeds with a partially initialized variable.

     

    If I want to do double check locking in a truly safe manner that complies with the ECMA spec, I have to write:

     

     

    if( _myObj == null )

    {

                Lock(_lockObj)

                {

                            if( _myObj == null )

                            {

                                        MyObject tempObj = new MyObject(…);

                                        _myObj = tempObj;

                            }

                }

    }

     

    As I mentioned earlier, the Microsoft version of the CLR enables the first version to run correctly (post .Net 1.0).  Because of the spec, it's a valid point of concern with non-Microsoft implementations of the CLR that can be easily answered.   Given that double-lock checking in Java is not supported, it has me wondering...  is this a holdover from Java's days as a stack-based compiler?  Can a stack-based compiler create an object structure without first allocating the memory to the host pointer and then calling the constructor?


    Static Variable Instantiation


    Another interesting part of the article discusses static member instantiation and how the mechanics work.  Given a class where two static members exist:

    public class Ugly
    {
        public static Ug _Ug = new  Ug();
        public static Lee _Lee = new Lee();
    }

    The static variables are not created until I access one of the members which I would expect.  When I access one of the members before instantiating the class, both static objects are actually instantiated before returning the result. This was a surprise to me.   It turns out the C# compiler marks the class with beforefieldinit.   (The behaviors surrounding beforefieldinit are interesting and certainly worth reading.)  The way to get around this behavior is to define the class as follows:

    public UglyClass
    {
        public static Ug = new  Ug();
        public static Lee = new Lee();

        static UglyClass() {}
    }

    When C# sees a static constructor, the type is not marked with the beforefieldinit attribute.  Therefore each static object is instantiated upon first reference.

More Posts

Our Sponsors

Free Tech Publications