Is Pipelined Continous Integration a Good Idea?

One of the well-known practices of Agile development is Continuous Integration, which entails team members integrating their code regularly into the baseline and running all unit and system tests.  In most teams a CI server is used to do this quickly and automatically when code is checked into the baseline.  This usually works well in the beginning of a project, but sometimes, when the team and/or code-base get large, the CI server starts to slow down.  The cycle between builds grows and the feedback degrades – a build may take an hour or more to respond with a pass/fail, and by that time several people may have checked in their code into an already broken build.  

To address this issue, many teams “pipeline” their CI, that is they build in stages.  A quick build with the fast tests is run and a preliminary pass-fail is sent to the team, while the more extensive form of the build is done in the background.  Slower tests, such as functional/integration/system tests, are performed in later stages.  Simon Stewart opines on the virtues of such a solution:

And this is how we end up with a build pipeline. Later stages tend to run slower than earlier stages, but everything is arranged to provide fast feedback. We know that we only have to manually test the builds that make it through the far end of the pipeline, and as we add stages we can add confidence that the application works as expected. If we're smart (quick! Pull the "clever" lever again!) we can automatically deploy the app into increasingly realistic environments and run tests against it as part of the pipeline, which is something we'd never imagine doing from the developer workstation.

Not everyone agrees that this is a good idea, however. Julian Simpson of ThoughtWorks calls this the “pipeline of doom”. He suggests that we are skirting the issue by using a pipeline – that of a slow build machine. By pipelining the build, we give false confidence that the integration phase is successful. Developers continue to build on bad code for several iterations, which only exacerbates the problem. As for pushing slower tests back:

The other aspect of the pipeline that I find troubling is this: by not forcing the developers to sit through the functional tests before they check in their code, you prevent their fantastic brains from reflecting on improving them. If people are feeling the pain of running them, they have a good motivation to fix them. Those tests ought to provide the biggest bang for your buck: unless you're careful you could be running poor tests dozens of times a day.

So, does your team have pipelined CI? How’s it working for you?