Given a long list of things that need to be done, how should that list be structured? What is the best order of tasks? On what basis do you decide what to do next? Strategically structuring a project is obviously fundamental to success, yet the subject is usually given short shrift.
Fundamentally, structuring a project is pathfinding under conditions of limited visibility. At each point you can choose to look around or make a forward move. Design to gain additional visibility or implement to move forward.
The recommendations of classic methodologies are based on varying assumptions about both visibility and goals.
• Classic waterfall: Assumes excellent visibility, and assumes you wish to minimize expected time or cost of project.
• Classic spiral: assumes visibility is limited but increases with each cycle, or over time. Assumes you want to minimize risk of project.
• Methods which encourage rapid, staged deliverables - eg agile methods - are motivated by the assumption that local visibility is much higher than deep visibility, and that a reasonable project path can be found by continually making reasonable local decisions.
In the absence of high "undo costs", pathfinding using locally best criteria will probably eventually get you where you want to go, but there's nothing to say that such a path will either be the lowest cost, quickest, least risky, or achieve the highest quality. (And if your project includes facets with high undo costs, eg specified interfaces to which many components will be written, locally optimal decision making is clearly inadequate.)
My contention here is that we can do better: By making appropriate design and planning moves, in other words by intelligently and opportunistically designing when conditions warrant, one can do better than either out-of-the-box agile or classic "big design up front" methods.
Project management is a game in which your goal is to make a series of moves that gets you to a winning outcome. There are many possible moves, and many possible outcomes. You can't fully predict the consequences of a move, but you can use a turn to think about what move to make next. There's a budget, in the sense that you only have so much time or so many turns before the game ends. If you aren't at a winning outcome when the game's up, you lose.
Beginning project managers often use the features of MS Project as a sort of template to guide them, but although task dependencies, critical paths, Pert and Gantt charts and the other schedule-oriented tools found in MS Project can come into play, managing projects, at its core, isn't about schedules. The key is to make sure that the right things get done.
To meet a schedule you need to thoughtfully and strategically structure your project. Don't let timelines, WBS, and all the other fancy stuff in MS Project distract you.
If you do the wrong thing, it doesn't really matter how well you do it.
Doing the wrong thing (for example, creating a product that no one wants) is far more common than simply doing a bad job of creating the right thing. So the most important thing to do is figure out where it is that you're trying to get to.
Projects don't exist in a vacuum; projects are by and for people. You need to understand your project from the point of view of the people to whom the project is important. They determine what it means for the project to be successful.
To whom are we delivering value, subject to what constraints? Is hitting the right functionality but going somewhat over budget better than hitting the budget but delivering junk (for example)?
Some goals are known to be more important than others, and some goals are important, but not recognized as such. Other goals are thought to be important, but really aren't, and lots of goals are not very important and easily seen as such.
In any project, the most important task is to develop deep visibility into this tangled ball of yarn.
(I used to say "Aim for mediocrity", but that was too inflammatory and people didn't get it. So I've adopted Ward Cunningham's famous dictum.) A working minimal version of the software -- something that is clearly not finished, but nevertheless looks, feels, and works just well enough to reveal what the finished product should look like -- is the design medium of choice for gaining deep visibility into the project as a whole.
Understanding what to do next, how long tasks will take, what is worth doing, becomes almost trivial once such a version exists.
To quickly get to this minimal acceptable version you need to identify what truly cannot be omitted. These essential elements manifest themselves as features that everyone will insist on. Use prototypes, mock ups, interviews, meetings -- whatever it takes -- to identify these elements.
No one likes to aim for mediocrity, especially not people with talent and good ideas, so sticking to the task of developing such a minimal version typically requires much restraint. The great thing is that once you build it, the risk that your project will end a complete failure is reduced to almost zero.
The minimal acceptable version was your design exercise for the high payoff finale. Now that you've done mediocre, you know what great is and can figure out how to get there.
You won't often win big without first knowing how to be acceptable, so why not make acceptable the first stop on the way to major victory?
Don't even think about the big win until a minimal version is in hand.
Limited visibility implies risk. Project management as a craft requires getting things done in the face of this risk.
Identify big risks -- nasty things that might happen whose probability of happening is uncomfortably high -- then sequence project tasks to reduce those probabilities.
The act of creation lowers risk: the creation is either successful or not, either way we know, either way the uncertainty is reduced. So a useful stratagem is to create those things that have high risk first, in an effort to quickly reduce the risk of the whole project. Use whichever design media are cost effective.
Remember that total failure is much worse than partial success. Remember to avoid total failure by aiming for mediocrity.
When you start a project, you don't know what you're doing and you don't know where you're going, yet by the time it's finished you do know what you're doing and you've managed to get somewhere. How does this happen? It's because you've learned a lot along the way, of course. In many projects, this happens by accident. It shouldn't. A project should be managed around learning.
Identify tasks whose doing might help you figure out where to go next and do them early on.
Of my candidate tasks, which one do I understand least well? Which task will teach me the most about what this project should really look like? Which task will increase visibility the most? (These are the things to do first.)
Creation keeps on happening throughout the project. There is no bounded "design phase" and no "development phase".
The software is expressed in ever finer detail, in an ever more final medium.
This opens the possibility of using knowledge gained while doing the project during the project - we don't freeze out knowledge gained in a latter part of project. What is learned doing part A (which we did first) influences how we do part B (which we haven't done yet). So during the next iteration, when we express part A in a finer degree of detail, we can draw on what we learned doing part B.
Moves that are perhaps suboptimal, but have a "good enough" payoff under broad conditions are generally better than "ideal, assuming our assumptions are spot on" moves.
Do like a CPU: go down several paths at once, see which one works out, throw out results that don't work out.
Try to maximize what you know at every point in the project. Don't worry about "dependencies", instead do things in parallel and don't worry about duplication of effort, redundancy, or that things will be thrown away. What you're trying to do is flush out possible problems, learn as much as possible along the way, move fast.
There's no rule that says you have to do tomorrow exactly what you thought you should do tomorrow last month when you wrote the first draft of the project plan. Reevaluate your path aggressively as you learn. Tasks that haven't been started can certainly be shuffled. Commit to goals, not paths.
If you have a bunch of stuff to pack into a box, if you merely shake and squeeze the stuff together gently for a long time, it'll eventually fit into the box in an optimal way. (That's a result from the physicists.) The key is constant gentle shaking and constant gentle pressure.
That's not a bad model for software creation: keep trying variations, constantly minding the goals and constraints, and after a while you'll get a good fit.
Constant pressure (on the software) makes for great software. Start early, apply often. Have strangers evaluate your work in progress. Use a test harness that's easily automatable, and treat automated tests as an investment. Test your assumptions, too, constantly.
Be open to the idea of rapidly flipping between design, implementation, and testing, of large and small portions, of gaining visibility and then marching forward. That's the shaking. Getting to a real deliverable, that's the pressure. Keep them combined. Separating the design from the implementation, the thinking from the doing, the shaking from the pressure, is almost always a mistake.