Patrick Smacchia [MVP C#]

Manage states in a multi-threaded environment without the synchronization pain

Patrick Smacchia — Mon, 05 May 2008 05:10:00 GMT

I see 4 common ways to deal with states in a multi-threaded environment:

Immutable objects: The idea is simple and efficient: make sure that the states of objects cannot be changed. This way, the object can be accessed concurrently by multiple threads without any risk of state corruption or thread race. I wrote a blog post on how to implement immutability with C# and how you can use NDepend facilities to enforce object immutability and method purity.

Thread / Resource affinity: The idea is as simple as it is efficient here also: making sure that an object is always accessed by the same thread. The .NET Framework offers several ways to implement this paradigm: the System.ThreadStatic attribute on fields, Thread Local Storage API and the interface ISynchronizeInvoke. These are described in this article I published on CodeProject.

Synchronized objects / Thread-Safe class: Since v1.0, the .NET Framework comes natively with a way to implement synchronized objects through the lock C#/VB.NET keyword, alias the Monitor class. I found it convenient to implement this pattern this way.

Synchronization objects: The .NET Framework comes with several synchronization API. Some of them are based on Windows synchronization primitives (events, mutex, semaphore) and some of them are implemented by the CLR (ReaderWriterLock). These API have been proven to be not optimal and even buggy, with possible starvation phenomenon in some particular cases.

The cost of synchronizing

The goal of this post is to praise immutability and affinity over synchronization. There are 2 killer arguments against synchronization:

First, immutability and affinity are so efficient by-design that once you go with them you won’t get anymore any concurrent access bug. Every experienced programmer knows that synchronizing accesses to states is a real pain and generally ends-up in all sorts of non-deterministic bugs very hard to pinpoint and correct.

Second immutability and affinity won’t hurt performance. Synchronizing thread accesses to states leads to thread-context-switches that are extremely costly. Also, threads are very expensive resources that are wasted each time a thread is put in a wait state. I agree that immutability sometime provokes extra object creation but frankly, this is neglectable compare to threading cost.

Why most of developers still opt for synchronization?

First it is a matter of education. Every programmer academic background comes with a course on how to program in a multi-threaded environment with synchronization primitives. This is an entertaining course illustrated with pleasent concurrent patterns and as far as I know, tricks such as immutability and affinity are not presented.

Second, at first glance immutability and affinity doesn’t seem to solve any problem. In both case there are no shared and mutable states. And this is what programmers want: being able to read and write a state from several threads. Just have a look at this comment on my blog post on immutability:

I'm doing concurrency because I want to calculate lots of things in parallel. I want them to change. Immutable types don't help concurrency much. Sure you can prevent accidental changes to things that shouldn't change. Of course that's just as valuable when coding single threaded. I really don't think you can have done much concurrent programming.

Immutable object / Changing state

Immutability is more subtle than this. The string class is immutable for example. Did you hear any programmer complain that he cannot code multi-threaded access to its strings? No, and the non-obvious reason actually lies in how we consider the state’s identity. A state is typically identified by an object and we write things like:

Person person = new Person(“Julien”);

person.FirstName = “Mathieu”;

We also write things like:

string personFirstName = “Julien”;

personFirstName = “Mathieu”;

The difference is that in the first case we have non-thread safe code because the object person can be read/write accessed by several threads. In the second case the state personFirstName had also been modified but now we won’t get multi-threading issue because the string class is immutable and the same state is handled by 2 different strings. We are losing the common one-to-one relation between a state and the underlying object. In other words, the identity of the state is not the object anymore but the value itself. But still, the state has been changed! The programmer is now free to expose the new “Mathieu” state to other threads without any kind of synchronization infrastructure.

The beauty of thread / resource affinity

Thread / Resource affinity is also a concept that deserves more attention. It promotes task-oriented programming. A task is thoroughly executed by the same thread, from A to Z, and its states are not visible from other tasks. One famous application of affinity is the way win32/Windows Form works. For a given form, all tasks are executed by the same thread, essentially painting/refreshing/timer operations and input/mouse/keyboard event handlers. The immense benefit that Windows Form users get is that we can be sure that the mouse click event handler won’t be triggered while the windows is re-painted. Thus all the states maintained behind our form don’t need to be synchronized. As we know, the down-side effect is that doing computation from the UI thread leads to unresponsive UI, and this is why there are tricks such as the BackgroundWorker class to make it easy to trigger task on tier threads.

To implement affinity, personally I am not a fan of ThreadStatic or Thread Local Storage because they both rely on costly Windows and CLR internals. Also, I remember this article from Juval Lowy that explains how to implement ISynchronizeInvoke but I especially remember how complicated and error-prone it is. I prefer to implement affinity by myself by:

Preferring to maintain states with thread affinity from method local variables instead of object fields. Local variables are by-design bound to the current thread.

Writing contracts about affinity. Concretely, all methods of a class begin with something like:

Debug.Assert( MyHelperPlumbing.CurrentThreadIsInitialThread() );

The code is less readable but as long as the language teams don’t provide DbC facilities we don’t have the choice.

We are currently removing most of synchronized accesses from the code of NDepend. This is a needed refactoring that will greatly simplify the development of features we want to do next. I admit that managing changing states through immutable objects sometime require some intellectual gymnastic. For example, if the object tagged by a DataGridViewRow is immutable, the tag property needs to be updated when the state is changing. But clearly, this is a minor programming exercise compare to synchronization nightmare.

UI Matter: Make the simple things simple and hard things possible

Patrick Smacchia — Mon, 28 Apr 2008 15:09:00 GMT

Make the simple things simple and hard things possible

IMHO, this tenet applies perfectly in how UI should be designed. Typically, the most direct way to use a UI control should result in the most awaited feature from a user perspective (make the simple things simple). Then, some extra/hidden UI control facilities can be added to the control to support more in-depth scenario (make hard things possible).

Case 1: NDepend Start Page

We tried to apply this tenet in the NDepend UI. For example, on the start page there is a DataGridView that lists the most recent NDepend projects. We estimated that left-clicking a project should result in loading the results of the most recent analysis done on this project. This is clearly what the user wants most of the time. But sometime the user might wish to do something else from a project: maybe she just wants to open the project in order to view or tweak it, without being bothered with any analysis result. Maybe she wants to open the result of an analysis done 3 months ago. This is why the DataGridView lets the user right click a project and access to such options:

Case 2: NDepend Project Properties Page: Code to Analyze Panel

This tenet also drove the way the Project Properties window has been designed. Basically, a NDepend project is just a list of .NET assemblies to analyze. Even thought an NDepend project can be made of dozens of options the Project Properties window must reflect this simplicity at first glance. This is why the first things a user see when editing a project is a DataGridView that can be populated with a list of assemblies (eventually drag and dropped from Windows Explorer):

Besides assemblies of the application that must be analyzed, there is a list of tiers assemblies, i.e assemblies referenced by the application assemblies, such as mscorlib for example. The list of tier assemblies is automatically populated and real-time refreshed from the list of application assemblies. Thus, we hope that the user doesn’t perceive this second list as a burden but more as interesting extra information.

Actually listing some assemblies can sometime be a little trickier:

• Assemblies of a same NDepend project can be located in different folders

• When working as a team, bin folders are usually rooted differently on the various developers machines and build server machines. But only one NDepend project file must be shared by all machines.

• Microsoft .NET Framework assemblies, such as mscorlib or System.Core, can be located in some tricky folders such as C:\WINDOWS\Microsoft.NET\Framework\v2.0.50727 or C:\Program Files\Reference Assemblies\Microsoft\Framework\v3.5. And these folders can slightly vary depending on your installation drive or if you are on a 32 or 64 bits machine.

To handle this extra-complexity, there is a single button View Folders. When clicking this button, a folder panel expands. This folder panel let’s tweaks the list of folders that contains application and tiers assemblies.

• There is an option Relative Path Mode that makes all folder referenced by the NDepend project relative to the folder where the NDepend project file is located.

• The .NET Framework assemblies tricky folders are automatically referenced. These folders are not impacted by the Relative Path Mode option because they depend on the .NET framework installation on the current machine and not on the development working environment.

The great consequence is that users that don’t need to cope with folders will never see this panel since the list of folders is automatically filled up when application assemblies get browsed.

The problem with this approach is that it is a bit scary: Do users that wish to tweak folders will figured out that there is a whole deep folders panel? Or will she skip the View Folders button and think that the tool is not well designed for real-world use?

So far, we got only one user’s mail about how to handle folders so it seems that it is worth trusting users when it comes to dig a bit into the UI to achieve what they want to do.

Case 3: Red-Gate Ants Profiler Filter Builder

An UI that positively surprised me by abiding by the tenet is the Red Gate Ants Profiler. Typically, you end up with a DataGridView that lists thousands of methods profiled.

While you can sort these methods by name or time spent in it by clicking the corresponding columns, after a few minutes tinkering with this, you certainly want to focus your attention on the performance of a small subset of methods. In other words as a user I need to filter my methods according to certain criterias. And the good news is that Ants is very good at this thanks to a powerful filter builder that you get by right clicking any of the columns:

And you, do you obtain (or not) some success stories by applying the Make the simple things simple and hard things possible tenet?

Diff tools to see source files changes

Patrick Smacchia — Mon, 28 Apr 2008 10:02:00 GMT

Making easy to compare 2 versions of a code base is something that I consider super important and that is why it became a major feature of NDepend. While source file repository tools (a la VSS or CVS…) also provide this feature, the great thing with NDepend is that changes are not considered just as text file modifications. For example you can dissociate code change from comment change and from structural change with these 3 CQL queries:

SELECT METHODS WHERE CodeWasChanged

SELECT METHODS WHERE CommentsWereChanged

SELECT METHODS WHERE VisibilityWasChanged

…and there are plenty of others trick supported, to see what is not used in a tiers assembly or to see how the coupling has been impacted by code changes. These are summarized in this 3mn screencast.

Often knowing where the code was changed is not enough, you want to see what was changed by comparing older and newer versions of source files. This is useful to focus code reviews on code that has been changed. Doing so is a killer practice to rationalize code reviews and make the most of it by often pinpointing subtle last-minutes bugs. Empirically, I came to the conclusion that the bulk of bugs are luring inside the code that is not yet in production, in other words, the code that has been refactored or added since the last release. I wrote an article on this topic: How to avoid regression bugs while adding new features

NDepend used to rely on the good old WinDiff tool to compare 2 versions of a file:

One of the most common feature request we got was to support any source files diff tools. Thus we added this as an option, with a list of the most popular text diff tools available:

Notice that we want to make sure to provide a flexible support for any diff tool with the command line arguments tags $OldSrcFilePath$ and $NewSrcFilePath$. Thus, the list of tools is here to make it more convenient to bind NDepend with your prefered tool but you can still choose any tool outside the list. We also provided the tags $OldSrcFileLine$ and $NewSrcFileLine$ but surprisingly, none of the diff tool we tried support these command line arguments.

It was quite interesting to churn the web for the most popular diff tools. It seems that some free tools, especially WinMerge, sustain the comparison with non-free tools. There are some passionate debates around which diff tools to use, such as on the Jeff Atwood’s blog that praises for Beyond Compare. There is also a feature I was not aware: You can compare 3 versions of a file. This seems to be supported only by KDiff3 and Araxis Merge. One of the must-have feature that most of tool supports is the ability to compare recursively 2 folders to see which files have been deleted/added/changed.

Frankly, I haven’t tested thoroughly all tools. At first sight, Araxis Merge seems to be the most featured one with an impressive list of features and I found it quite easy to grasp. However this is also the most expensive one.

The main difference that I have found between free and commercial tools is on how the tool got polished. By polish I mean things like: Is it localized, How smartly the command are organized, Startup time, On-Line and Embedded Documentation, No UI Freeze or user wait, No pesky crash, How cleverly the UI real-estate is used, Options persistence and sharing.... However, let’s mention that the freeware WinMerge is really polished.

As the team lead of a commercial tool for .NET developers, I found that indeed polishing a software is the best (and only) way to extend the set of users from the enthusiast/early ones to the (much bigger) set of potentially interested users. For the development team this is also a great satisfaction to be allowed to spend time on all minor but important details. This is usually a burden list that tends to grow. There are always numerous features in the pipe to add but when the minor-TODO list gets shortened, you'll notice a lot of positive energy and feedbacks.

After this digression, it is maybe worth providing the list of tools we found:

Araxis Merge (commercial)

Beyond Compare (commercial)

Compare It! (commercial)

DiffMerge (free)

ExamDiff (commercial)

KDiff3 (free)

TextDiff (free)

WinDiff (free)

WinMerge (free)

Finally, I noticed a free diff tool that comes with an intuitive presentation of changes but unfortunately it is only available on Unix. Kompare:

Here is a list of dream feature sets for diff tools (in praise of Beyond Compare) but personally, such a simple and intuitive way to browse changes could make me prefer one diff tool over the super-polished ones.

NDepend presentation at MVP Summit

Patrick Smacchia — Wed, 09 Apr 2008 10:28:00 GMT

I will present NDepend during the MVP summit. The schedule is one hour Wednesday 16th at 5pm in MSCC/ Hood. It has been well thought out since this doesn’t overlap with others MVP summit presentations.
This presentation is dedicated to MVP and RD.

Another one is forecasted for MSFT on April 14, 2008 4:00 PM-5:00 PM Conf Room 41/5731 (18) AV

This presentation will be a great occasion to talk about the future and to recap NDepend main features: quality and metrics, dependencies management, visual facilities, code snapshot comparison, custom CQL queries and rules, and the brand new facilities to churn test coverage data.

I would like to thank Krzysztof Cwalina, Kit George and Kathy Carper for their kind help in setting up this meeting.

Make the most of your test coverage data

Patrick Smacchia — Wed, 09 Apr 2008 08:28:00 GMT

I am really glad to announce that now NDepend supports test coverage metrics:

PercentageCoverage , NbLinesOfCodeCovered and NbLinesOfCodeNotCovered, defined on methods, types, namespaces and assemblies.

There is also the metric PercentageBranchCoverage defined on methods.

Coverage data are gathered from coverage files emitted by NCover 2.x and Visual Studio Team System 2005 or 2008.

So why is it so great that NDepend now supports these metrics? After all both VisualStudio and NCoverExplorer present facilities to churn coverage data. The cool thing is that integrating coverage metrics inside NDepend and the CQL language opens up a wide range of brand new possibilities.

Continuously check for coverage

Typically, you spend time making sure that your class YourNamespace.YourClass is 100% covered. But how can you make sure that in the future your class will remain 100% covered across versions and evolutions? With NDepend and CQL 1.7 you just have to define the CQL rule:

WARN IF Count > 0 IN SELECT TYPES WHERE FullNameIs "YourNamespace.YourClass" AND PercentageCoverage < 100

As we are heavily using this facility on the code of NDepend itself, we found convenient to define an attribute NDepend.CQL.FullCoveredAttribute in the assembly NDepend.CQL.dll, Then we defined the rule: make sure that all types tagged with this attribute are 100% covered. With CQL this results in:

WARN IF Count > 0 IN SELECT TYPES WHERE HasAttribute "NDepend.CQL.FullCoveredAttribute" AND PercentageCoverage < 100

This is a great way to scale on hundreds of classes fully covered. But also it helps documenting the source code: when a developer is about to tweak such a class, she can’t even forget covering all her modifications.

Alternatively, if you think in terms of namespaces or assemblies thoroughly covered, you can write such a rule:

WARN IF Count > 0 IN SELECT NAMESPACES WHERE NameIs "YourNamespace" AND PercentageCoverage < 100

Or:

WARN IF Count > 0 IN SELECT ASSEMBLIES WHERE HasAttribute "NDepend.CQL.FullCoveredAttribute" AND PercentageCoverage < 100

Because assembly has support for attribute while namespace don’t.

Focus coverage on sensitive code

What is sensitive code? In my last blog entry on practices to tend toward bug-free code I explained that sensitive code that will likely contains most of bugs you’ll get within the next release is:

Methods that have been added since the last release.

Methods that have been refactored since the last release.

Complex methods.

With CQL, you can write rules to be warned when such sensitive code is not 100% covered.

// Methods that have been added must be fully covered

WARN IF Count > 0 IN SELECT METHODS WHERE WasAdded AND PercentageCoverage < 100

// Methods that have been refactored must be fully covered

WARN IF Count > 0 IN SELECT METHODS WHERE CodeWasChanged AND PercentageCoverage < 100

// Complex Methods must be fully covered

WARN IF Count > 0 IN SELECT METHODS WHERE

(CyclomaticComplexity > 15 OR NestingDepth > 5) AND

PercentageCoverage < 100

You can also use the metric MethodRank to know which method is sensitive, in the sense that if it contains a bug, this will likely be a catastrophic bug. The trick is that methods with high MethodRank are those that are the most used.

// The 100 most important methods must be fully covered

WARN IF Count > 0 IN SELECT TOP 100 METHODS WHERE

PercentageCoverage < 100 ORDER BY MethodRank DESC

Get an intuitive feeling of how your code base is covered

The visual facility supported by NDepend to display a set of code element that matches some condition is especially useful when it comes to evaluate what is well or poorly covered. For example, the following picture shows the methods fully covered in NDepend. They represent 22.726 lines of code on 58.955.

It is interesting to compare this with the set of methods at least a bit covered (i.e %coverage > 0). They represent 32.790 lines of code on 58.955.

Deal with ‘uncoverable code’

We all know that 100% coverage of an entire code base is not feasable. Even if you are really enthusiast about testing, some case just cannot be tested automatically. Typically, a method that call MessageBox.Show(…) cannot be covered by unit tests because it requires a user action. This is why NDepend has support for what we call uncoverable code. Project properties accept an attribute that will tag uncoverable methods, uncoverable classes or uncoverable assemblies. Such tagged code element won’t be taken account when computing coverage statistics. This will let you write full coverage rule without being bothered by uncoverable code.

Smart merge of coverage files

Code coverage data is often spread across numerous coverage files. Thus NDepend accepts a list of coverage files as input. A coverage file has a merge option OR XOR AND or NONE.

Most of the time you’ll use the option OR to merge your coverage files. But if you want to know what is only covered by a particular file, you’ll need the option XOR. And if you want to know what is the intersection of what is covered by all coverage files, you’ll use the option AND. And finally the option NONE will be used to temporarily disable some coverage files.

You won’t have to re-analyze your project to play with these options because they can be computed on the fly on an analysis currently loaded.

If you wish to see all these facilities live, have a look at this 3 minutes screencast.

Also, details on how to import coverage data from coverage files are available here.

Towards Bug-Free Code

Patrick Smacchia — Sun, 30 Mar 2008 19:02:00 GMT

The only way to have bug-free code is to mathematically prove the code. Very few programs in the world are mathematically proved. Some industry can afford the price of a mathematical proof, especially when bugs would turn into human death, such as embedded software in planes, trains or cars. Most of us are working on projects that cannot afford the cost of a mathematical proof. We then rely on some tricks to maintain our bug rate as low as possible. I classify these tricks into 6 categories.

Contract

Automatic test

Empirical approach

Code review

Prioritizing bugs fix over new features development

Programming style and code quality

Static Analysis Tools

Contract

The idea of contracts is to insert correctness information inside the code. When contract is violated it means that your code contains a bug somewhere. Typically, the .NET world is very poor in terms of contracts. Our only way to express contract is to use the System.Diagnostics.Debug.Assert(…) method. This is very unfortunate because not only our code is blurred with numerous Debug.Assert(…) calls, but also the compilers cannot check contracts at compile-time. For example the Spec# non-nullable types is an excellent form of contract at language level that help a lot avoiding pesky NullReferenceException. And Spec# is far from being the only language that propose contract facilities, think about what Eiffel proposed more than twenty years ago in terms of contract!

As I believe in contract, I personally use a lot Debug.Assert(…). This represents around 15% of my code. I will likely blog more on this because with all the buzz actually made around automatic tests, I feel that contracts should deserve more attention.

Automatic test

I won’t enumerate the numerous benefits of automatic tests and high code coverage ratio here. The important thing to remember is that having a solid battery of automatic tests is an excellent way to decrease significantly the bug rate on tested code, but also to avoid new bugs when code gets refactored. As many, my opinion is that the cost of writing automatic tests is worth the price compared to the cost of maintaining not automatically tested code.

Empirical approach

What I call empirical approachis a simple tenet that every seasoned programmer know:

Most of bugs in a new version are coming from modified code and brand new code

Unchanged code that works well for a long time in production won't likely crash within the next release (what we call stable code). We don't say that stable code don't contain bug, but discovering a bug in stable code is rare.

How do you know that some code works well for a long time in production? Simply by listening to users. If they didn’t report problems on some features for a long time then you can be confident that underlying code is stable. Some might say that I consider here users as testers but don’t take me wrong, we don’t have the choice. If you have real users, they will complain when they will find a bug and they will remain silent when they consider that the product is working fine. You are responsible to deliver correct code that will satisfy users but why wouldn’t you infer statistics from their feedbacks to asses where is your stable/unstable code?

I wrote an article about how we (the NDepend team) use this simple but effective idea and our own dog-food to avoid regression bugs while adding new features. Basically we focus our code reviews mainly on code that have been added or code that have been changed since the last release and of course.

Code review

Code reviews are good to enhance quality and to educate programmers but I don’t believe in the efficiency of code reviews to anticipate bugs. Even though some portions of code can be fascinating to read, the bulk of a code base is tedious and will make you lose your focus in less than an hour. The problem of code review comes from the mass of code to read. This is why I advise to only focus your review on not stable code, i.e added code and modified code. If you release new versions often, the mass of code to review before each release will quickly become an epsilon of the size of your entire code base. Doing so can also be seen as a way to capitalize on code reviews made during previous iterations.

Prioritizing bugs fix over new features development

This popular methodology directly results from the concept of stable code. Prioritizing bugs fix over new features development can be seen as a way to constantly struggle to maximize the surface of stable code in our code base.

Programming style and code quality

The recent buzz around LINQ or F# comes from the fact that object style programming is more bug-prone than functional style programming. This fact results from the expressiveness of functional style. In other words, functional code is easier to read and understand. A major aspect of the expressiveness of functional programming style is IMHO the concept of immutability that I described in a previous blog post Immutable types: understand their benefits and use them. That’s a fact, it is hard to write, understand and maintain code that mutates states at runtime (for example this is why global variables are so harmful). And this becomes much harder in concurrent environment.

Obviously, code quality has also a direct impact on code correctness. Anti pattern such as methods with high LOC, high Cyclomatic Complexity or high Nesting Depth, entangled components, methods with multiple concerns, classes with multiple responsibilities… leads to code harder to debug and to test.

Static Analysis Tools

We all dream of a static analyzer that could pinpoint automatically bugs in our code by just pushing an Analyze button. Some tools are already able to detect naïve mistakes, such as calling a method with a null reference as parameters where there are no tests of parameter nullity (i.e interprocedural analysis).

But most bugs are not that easy to pinpoint. By just analyzing the code a tool cannot distinguish between a feature and a bug because it doesn’t know how the application should behave. Some heuristic exists but still, to be efficient, a bug finder static analyzer needs to be feed with more information than just the code. Typically, this extra information can be found in contracts and unit-tests code. As far as I know, in the .NET area there are 2 projects revolving around this idea NStatic and Pex, and I am really looking forward to use them on my own code.

Conclusion

From what I understand from the agile trend, having a bug-free product to release shouldn’t be the goal. Anyway, every program not mathematically proved has bugs. The goal should then be to tend toward a bug-free product by applying as many correctness tricks as possible.