As some of you know, I started a new job this month as a consultant helping clients to adopt Agile practices. It's been an interesting experience so far. Working with people who have different philosophies and experiences is always a good opportunity to learn. I'm getting exposed to different techniques and being forced to examine the things that I believe. In the end, we all want about the same things -- to write quality software in an efficient manner without working too many weekends or late nights. We all agree on the basic goals, but our philosophy in achieving these goals are different. I know there's been a few cases where the clients have cocked an eyebrow at some of my opinions, but in every case I've simply had a different set of ideas for how to build software.
Mostly unrelated, but in the spirit of the infamous Agile Manifesto, I'm writing down my own manifesto in an effort to explain the method to my madness. Many of you are going to disagree with me, and that's okay. It's my set of values after all, and I'm happy to hear your comments. Or better yet, go write your own manifesto.
Let's get this out of the way, and please read this literally: while I do believe there is some value in the stuff on the right, I think the stuff on the left is more valuable and important.
Unit Testing and Testability over Defensive Coding, Tracing, Debugging, and Paranoid Scoping
I think we can all agree that you can't ship code until you remove enough defects from the code. While you certainly have to apply a mix of techniques and practices to remove a high percentage of defects, I want to talk about the things we do to our code to wring out the defects. Off the cuff, I can think of these:
- Use a debugger to walk through the code to try to uncover the errors through observation
- Use defensive coding to provide more contextual information about erroneous usage of a section of code
- Tracing statements in the code to provide debugging information at runtime
- What I can only call Paranoid Scoping. Ratcheting up the protection on classes and members to prevent developers that use this code from "harming" themselves.
- A very high degree of granular unit testing, like that afforded by adopting Test Driven Development. I'm talking near 100% developer test coverage here.
An axiom of software development is that problems are cheaper to fix the sooner that the problems are detected. If you take that axiom at face value, it's easy to see why I put far more weight into comprehensive test-first unit testing because it gives you far more rapid feedback to find problems fast. Small, isolated unit tests work on very granular pieces of the code, so the number of variables in any single unit test should be small (if it's not, look for a different way to write the code). You shouldn't even try to run the code as a whole until all the constituent pieces have been validated through unit tests.
Debugging is not efficient, period. It's simply not smart to try to debug a lot of moving code parts together if you don't have a high confidence level in any of the parts. If anything goes wrong you've potentially got a lot of variables to check out all at once. Yes, you can help yourself out by providing tracing to see what data is passing through the code or defensive programming to short circuit processing when you hit a bad value or provide a more descriptive error message. But no matter how hard you try to add tracing and defensive coding, I still think unit testing will get you to working code faster than any possible kind of debugging.
Last week one of the developers working on our server side component asked us, the GUI team, to make sure that we had plenty of tracing in our codebase so that integration issues could be adequately diagnosed. I certainly don't think that's a bad idea, but I'm far more keen on sinking energy into a rigorous set of automated unit tests on both the client in isolation and the server side in isolation before we even attempt to integrate the two pieces. Since I am a believer in interaction style testing, we'll test our code extensively to prove that our code passes the correct information to the backend through mock objects.
Don't get me wrong here, I do believe in some amount of defensive coding and tracing, I just think that it's not something you rely on as your primary means for removing defects from the code. My issue with tracing and defensive coding is the often speculative nature of the activity and the negative impact on code readability. I strongly prefer to use tracing and defensive coding "defensively." I'll put in specific exception warning for things we hit in testing that are difficult to debug or around external dependencies. If I had to choose though, I'd rather have a good suite of environment tests to detect configuration or connectivity problems before I even try to execute the code. In the end I might also say that I just think that you hit a point of diminishing returns with tracing and defensive coding rather quickly.
Oh, and by the way, every time I've come across code that had copious amounts of tracing? The code was truly awful, failure prone code. Excessive tracing is a code smell.
The paranoid scoping is probably the single biggest area of disagreement I've hit with my client. They haven't traditionally used TDD or any kind of comprehensive unit testing strategy in the past, so I don't think they understand the negative impact on automated testing caused by this scoping. Because they depend more on debugging and waterfall testing for removing defects, they're big on using protected scoping on classes to prevent downstream developers from using objects incorrectly. In several cases already, I've seen that attitude hamper the testability of the code. Granular unit testing is dependent on an ability to quickly setup the inputs for a little piece of the code and run that code in isolation. In a specific case, the server side developers have quite purposely protected all of the setters of the entity classes in the system that should only be set by the server side code. Reasonable right? Actually, I'd argue that that strategy ends up doing a great deal of harm to us. In the current architecture, the entity classes can only function with the complete server side connected. If we can't change that design, it will dramatically slow down the feedback cycle of the user interface development because we can't unit test the presenters and views in isolation. I'm perfectly happy to make that code more "dangerous" in exchange for being able to write isolated tests more efficiently. Besides, we are going to prove that we use the server side code correctly by testing the interaction with the backend as well. In my world view, improving the testability of the code does far more good to remove defects than the reduced protection levels do bad by letting defects in.
Sharp Tools over Coding with the Kid Gloves On
Quoting Bruce Tate:
We need to let our craftsmen work with sharp tools, rather than protect them from themselves.
Recently, Paul Stovell took issue with a presentation from Scott Hanselman suggesting that you aggressively mark classes as internal or sealed to protect the users of your API or framework. I'm going to very strenuously agree with Paul on this one. In the week after reading Scott Hanselman's post I bumped into three different cases where the exact functionality in a framework that I needed to either use or override was sealed or internal. In one case I resorted to reflection to manipulate a private member. In another case I forked the open source framework and I really, really hate to do that. The attitude of "seal everything" is just going to limit the usefulness of your code. Make your framework or code easy to use by providing a good Facade and a bit of documentation (i.e. tell me which class is the Facade!), but leave the little underlying pieces accessible and available for usage and modification. Remove all that sealed and internal "protection" and I can certainly break something, but I can do more positive things with your code as well. I'm perfectly willing to be responsible for my own solution.
To put "Sharp Tools" in perspective, let's consider some of the languages I've coded in throughout my career. Assuming you used "Option Explicit" and pretend for a moment that the reference counting strategy wasn't a huge memory leak, VB6 is the ultimate kiddie glove language. It shelters developers from a wide range of coding issues by limiting the range of coding constructs. You can't possibly deny that VB6 was very successful, but it is an extremely limited language in capability. Limited Object Oriented Programming support and no real multi-threading severely constrained the usage of VB6, especially in server applications (though we certainly tried didn't we?).
C# and .Net came along with a lot more power and potential to royally screw things up. I think it's been a great trade. I routinely build things in C# that weren't possible in VB6 (windows services to name one) and use OOP techniques that weren't supported by VB6 to great advantage. I'm happy with C#, or was, until I started seeing some of the capabilities of Ruby.
Ruby is a dynamically typed language that also supports metaprogramming, i.e. code that writes code on the fly. Metaprogramming can be scary to debug, and there's a tremendous opportunity to thoroughly confuse developers, but it's soooo powerful. A lot of the eye popping features of Ruby on Rails (think ActiveRecord) are clever utilizations of metaprogramming.
When we worked together, Jeffrey Palermo and I had a running Abbot and Costello routine about static languages (C#) versus dynamic languages (Ruby). In order, it went something like this:
- I gush about something cool from Ruby or Ruby on Rails (migrations, the built in testing support, metaprogramming tricks, etc.)
- Jeffrey says "just wait until you inherit a really lousy Ruby codebase without any unit tests. How are you going to support that without compiler checks?
- I freely admit that I would panic in that situation, but then I remind Jeffrey of the really lousy C# legacy code we've dealt with that doesn't have any unit tests and say that I don't want to support code without unit tests in any language.
My point being that just like a sharp knife or power tools, "sharp tool" techniques are safe when combined with some proper precautions (TDD). Of course I've got a little scar on my bicep from shooting myself with a pneumatic nail gun, so what do I know about safety?
Several years ago, and long before it was AJAX and Web 2.0, I used to do a lot of DHTML scripting in JavaScript that heavily utilized JavaScript's "Expando" typing and a bit with "eval()." It worked, but as I recall the debugging was nasty and it wasn't really supportable. This summer I was on big Ruby and Ruby on Rails kick. As an experiment and a way to work in a familiar problem domain before I dived into Rails, I decided to go back to some of the JavaScript controls I wrote 4+ years ago and recreate them using Rubyesque coding techniques and Test Driven Development throughout. Using JSUnit and Prototype, I wrote most of a TreeView control and a sortable, pageable grid. Granted, I've got much more experience this time around, but even so it went much, much faster. I was able to shrink the code down considerably compared to the previous versions. Since I was using TDD I really didn't have to use the debugger very often. With duck-typing and expando members, mocks and stubs are almost trivial. The very features that make dynamic languages "unsafe," also lead to easier mechanics for TDD, which in my book is more valuable for "safety" anyway.
Writing Maintainable Code over Architecting for Reuse and the Future
A couple weeks ago one of the architects at my client asked us to make sure a piece of functionality we're building is reusable later. I started to object on the grounds of violating YAGNI before I realized that I completely agree with him -- on the ends if not the means. Some of the other developers are worried about making sure that the code meets future needs and maximizes code refuse. I think there's an underlying, unspoken assumption that the code cannot efficiently be changed later. It has to be designed "right" upfront.
I think that the code needs to be able to change to meet future needs, whatever those turn out to be. I want our code to be reusable as well. What I don't believe is that the way to meet these goals is to prematurely generalize the solution or speculatively build in unused hooks that "we'll need in the future."* If we spend a substantial amount of effort building infrastructure for future needs we risk delaying the initial release. Even worse, what if the specific infrastructure turns out to be unnecessary? That turns into waste. Additionally, I've often observed systems where the biggest impediment to extensibility was actually the very extensibility mechanisms put into place early in the project!
What I believe in is meticulously creating loosely-coupled, highly cohesive code backed up by comprehensive test and build automation that can accept change. I think reuse is best accomplished by well factored code, not by specific intention. If I can really create code that can be efficiently changed, and I firmly believe this is possible, I can add infrastructure and functionality to the code at a later date. I want my code to be able to safely accommodate a broad range of change because you never really know what the future will bring. We think we know what the future needs are, but one of the sales guys could easily swoop in with something completely off the wall. Call me naive, but I really do think that you effectively flatten the change curve through an emphasis on very solid, maintainable code. I want to be able to build the sales guy's wacky ideas that we don't even know about yet, not just the future features that we "know" we'll need.
I'm very dubious about a lot of the unbridled SOA enthusiasm I've seen practiced on projects. In particular, I think the idea of exposing pieces of a new system as web services upfront for a single consumer to be borderline idiotic. If there's only a single consumer, and the team owns both sides, just write the "service" in process. The extra cost of writing and debugging the out of process integration point is a complete waste of effort if there isn't a second consumer. You've simply taken on additional technical risk on the chance that that exact service point will be useful to another consumer at a later point.
But wait Jeremy, what if I do need web service later? What do I do now smart guy? I told you I should have written the web service. Well, if the code is really maintainable, i.e it can be safely changed, adding the web service wrapper to expose the existing code base remotely should be strictly an exercise in writing additional code, not changing existing code. I very honestly, and sincerely, believe that high quality, maintainable code goes a long way towards obviating the need for SOA.**
I've been working up a long-ish post on writing maintainable code that will expound on this point quite a bit.
Explicit Code over Design Time Wizards
My team has been coding for about three weeks on a WinForms client using copious amounts of Visual Studio.Net's design time support. I'm already thinking it's time that we, the .Net community, take a hard critical look at the RAD tools in Visual Studio. What I'm often seeing is that a little bit of explicit, hand-rolled coding leads to less duplication, better understandability, and vastly better testability than leaning too heavily on the wizard magic. The effort it takes to create code isn't just the writing of the initial code, it's also the effort it takes to test the code and modify that code later. In specific, I'm starting to become disenchanted with Data Binding. I'm really not convinced that the design time support for configuring the data binding justifies itself. I'm really seeing that pushing user interface behavior into Supervising Controllers with explicit code leads to a more maintainable system than depending on design time configuration of that same behavior.
Again, this circles back to the idea of making code work through testing and therefore seeking to enhance testability.
Software Engineering over Computer Science
I through this in here just to get some flames. It's a huge pet peeve of mine to see teams agonize over incremental improvements in sorting algorithms or caching or optimizing the number of IL instructions while neglecting to use sound Software Engineering practices. The success of a system and the staying power of a codebase are rooted in solid configuration and build management practices. Your algorithm might be fast, but you aren't going to reap much reward for it if you can't consistently get your code to install correctly. Computer Science knowledge comes into play in some projects. Good software engineering practices are valuable in every single software development project.
Here's a longer explanation of the coding ecosystem I believe in.
* How's that for loaded language?
** Of course, the natural state of code tends towards entropy, so I think SOA's future is perfectly safe.