When and how thoroughly should software be designed before being implemented?
There is little agreement in the software engineering community about what, when and how to design. Examining the principles motivating design seems like a good first step toward making sense of the conflict between different methodologies.
Agile processes are noteworthy in part for their de-emphasis of design. Traditional process gurus call for much more. A quick review of the software engineering library in fact reveals a bewildering variety of approaches to design, each typically centered on some medium, everything from index cards to prototypes. What to do?
I contend the best way for a practitioner (eg, me) to reconcile these contradictory prescriptions is to articulate and examine the underlying principles that motivate design, and then apply those principles directly to the specifics of the project at hand.
First, as a prelude to thinking about design, consider the fundamental issue of visibility.
In a real world project, visibility is limited. You don't see all the consequences of your decisions. You probably can't see exactly what your project should look like when it's done, or how it will be received by the stakeholders. Maybe you can't even figure out what task you should tackle tomorrow morning.
It's entirely possible that the project path that looks best from where you stand isn't optimal (it may even be dreadful).
At the end of every project, one says "if I knew then what I know now..." -- a shorthand acknowledgment that throughout the project, you were operating under conditions of limited visibility.
Some things are clearer than others. Obvious, but consider how many methodologies ignore this variation and assume you have equal information available about each move.
A principal assumption driving my thinking is that one can better manage projects if you take into account these variations in visibility, and act accordingly.
Local visibility is all about understanding near term consequences of an action.
Deep visibility is about understanding the far off consequences of a move, what happens way down the line.
Deep visibility is required to determine an optimal project path, to understand what the final deliverable should include, to project the actual time and cost of the project.
Traditional project management methodologies assume that goals, tasks and risks are both knowable and fully known, that you are blessed with uniformly deep visibility. This is rarely true.
Instead, much knowledge about a project is decidedly wispy - there are always plenty of things we sort of know, but aren't truly sure about.
But - and this is the important part - it's emphatically not the case that we have to accept this wispiness as immutable; that goals and tasks and the like are intrinsically unknowable.
Instead, we can choose to learn. We can refine our knowledge and bring consequences into better focus. Prototyping, testing, and conducting user reviews are examples of activities ("moves", using the terminology of the project game) that improve visibility.
We are always faced with this choice between constructing based on whatever knowledge we happen to have now, or delaying the construction until we have greater visibility.
Consider the characteristics or attributes of a move or decision. It has "reach" - how much of the project it affects. It can be local or far-reaching. It has an "undo cost". This is the cost of fixing things when you later find out you made the wrong move.
Even when one can recognize that a move has great reach and undo cost, accurately assessing the consequences of that move may be difficult. An interface specification is a classic example: it has proven to be extremely difficult to create interfaces that stand the test of time.
When making decisions of great reach and high "undo cost", improved visibility is of great value.
Clearly you want as much information available at the time you make the decision.
Much of the variation in methodology prescriptions found in the books can be traced to varying assumptions (by the methodologists) about visibility.
The old waterfall methodology is equivalent to a single iteration of the spiral model. You first make information gathering moves, then design moves, then implementation, and finally testing moves. There's no need or opportunity to revisit earlier moves. Your visibility forward at each step is uniformly assumed to be sufficient for you to choose an optimum path.
Experience has shown this doesn't work very well.
The spiral methodology acknowledges that maybe your visibility of the possibilities isn't so deep after all, and suggests that you gather some information, do some design, some implementation, some testing, and then iterate. Problems can be found and fixed earlier; path corrections can be made.
The agile methodologies go further and suggest that visibility is really, really poor, perhaps that even the basic goals are in flux. They advocate making path decisions on almost a daily basis. Get something working, keep it working, work in very small increments. Choose the path that makes your software better tomorrow.
(OK, I oversimplify a bit, but you get the idea.)
Design is nothing more than creation in a cheap medium. Sculptors design expensive bronze statues in cheap plaster. Architects design expensive houses on cheap paper.
We create in a cheap medium because we make mistakes; we find it impossible to get things right the first time. A cheap medium lets us iterate more often and learn faster at lower cost.
Our creations need to be realized to be understood. Complicated creations benefit from multiple realizations, from multiple trials. Therefore we design.
Put differently, "design" is the name of the thing we do to gain deep visibility. The goal of design is to illuminate the way forward, to reduce the uncertainty associated with any path - to keep a project out of the swamp.
If building a product by daily accretion always produced a good path, there wouldn't be much need for design. But sometimes it really does help to know where you're going.
Point one: designing is only worthwhile if it's cheaper than doing the "real thing". Point two: the design exercise has to tell you something, it has to influence what you're going to do next. Laying out the obvious is not design; it's a waste of time.
From this perspective, we see that underlying agile-style methods is the great insight that building the real thing is often the cheapest way to improve visibility. (Although keep in mind that this doesn't always hold, and whether or not it holds needs to be carefully considered on a case-by-case basis.)
Don't design something that's really cheap to redo. Don't design when it won't reveal your mistakes. Don't design stuff when the consequences of a "wrong" choice are minimal.
Do design when you can cost-effectively reduce the possibility of an expensive blunder.
Your software need not be fully specified or "designed" at any time during its development, and it very probably shouldn't be designed in a single medium. Perhaps at some point in the life of your project you may have a user-interface mockup on paper, an analytical model of computational requirements, and some working chunk of tricky code. This is not a design in the traditionally understood sense, yet these are all designs. This is a heterogeneous set of low cost creations that, taken together, improve project visibility.
There are many different design media. Here is a subset of the media that might be used in a project:
|
Design medium |
What it might illuminate |
|
Written description of project goals |
Does the project team understand what stakeholders expect? |
|
Written overview of architecture |
Are we approaching this project in a sensible manner? |
|
UI mockup |
Can users understand the software? |
|
Use case or story |
Can software do what users need to get done? |
|
Unit test |
Will software be reliable? How should it behave in exceptional conditions? |
|
Code snippet |
Will software be fast enough? Do third-party components behave the way we think they do? |
|
Interface class (without implementation) |
Will this class provide the functionality needed by other classes? Will software components fit together? |
|
CRC cards |
Will software components fit together? Is this a sensible architecture? |
|
UML class diagram |
Will software components fit together? What methods do we need to write? |
|
Task list |
Have we thought of everything? Do we have the right skills? |
|
Minimal acceptable version |
Is this what the world wants? Is this the right way to implement what the world wants? |
|
Gantt chart or timeline |
Can we possibly accomplish all this in a reasonable timeframe? Is our schedule reasonable? |
Is a unit test really a design medium? Sure. We create a unit test now in order to assess how our software will work in the future, and to prevent the expensive blunder of shipping a buggy product. Surely one gains as much insight into the future value of a chunk of software from creating a unit test as from creating a UML diagram. They happen to illuminate different aspects, they complement one another - but they both serve to tell you now, at low cost, what you don't want to find out later, at high cost. They both increase visibility.
And that's ultimately the point of such a heterogeneous and untraditional list: thinking broadly about what design is and can do for you, and using a wide variety of design media to support that thinking, will make a software project much more likely to succeed.
As Steve McConnell points out in the 2 nd edition of Code Complete, it is pretty clear that the extreme prescriptions, both "design everything up front" and "design nothing up front" are guaranteed to get you into trouble. But to get beyond that understanding, to get to the point where you know what's going to work in your specific situation, you need to have some guiding principles. It can't just be pulled from some book and it's not enough to say "avoid the extremes".
So that's what we have here - a handful of guiding principles. Design when it increases visibility, don't when it's pointless. It makes sense whenever there's a design medium that can reveal your mistakes at low cost. Be very liberal and imaginative about how you define "design medium".