Software Best Practices

Automated builds provide a foundation for software development. It is often one of the few practices those of differing methodological persuasions can agree upon.

This thread is a place to share experiences with automated builds and to learn what others are doing. Please move philosophical discussions into separate threads, to keep this one focused on actual experiences. 

Construx has been evolving our automated build tools and technology for years. Our last large project had a set of automation that made work very productive. It was a continous build that included significant automated test regression. Key elements included:

• Complete automation of all build steps, including creation of clean DB schema, populating test DBs, building of .NET solutions for web services and clients, creating installation files and deploying into test environments, and other technology-specific compilation steps.
• Unit tests built and executed
• System testing (using a GUI automated test tool) executed as last build step
• Any errors from the process (builds and tests) would be logged, and e-mails sent. A Wiki page shows the history of past builds with a red-light/green-light indicator on whether the build was clean
• The same build framework on the build machine used for developer machines
• Different build targets; developers normally built sub-sets of the system on their boxes
• Dedicated build machine for complete continous builds based on queue of check-ins to different code branches
• Decicated test machines that the latest build would be deployed to for running automated tests
• Complete build cycle peaked at about 65 minutes. Developers could do an abbreviated complete local build in about 10 min; they could also work iteratively within an area using near-instantaneous builds for .NET or DB code, and/or do smaller partial builds with the framework
• Most of the build automated with combination of Python and shell scripts; revision control tool was Accu-Rev

Really interesting topic!

I will explain how do we manage our integration and build process here at Codice. We are not using continuous but a controlled cycle, and still making it compatible with both SCRUM and CMMi (we have just achieved level 2, becoming the first SME in Spain to reach it).

Our development cycle is totally task oriented: everything we do is a task, and each task is mapped directly to our SCM with a branch. So we end up having many branches, but most of them are short-lived. Following the branch per task pattern gives us full flexibility and traceability (good for CMMi and good for agile too). The cycle is depicted in the following figure.

Developers work on tasks, and when they finish they pass a subset of the whole test suite. If everything goes right the task is finished, and waits to be integrated by the integrator at the end of the week.

We run a weekly integration process, creating a new stable release once a week. The integrator picks up the finished tasks (branches) and integrates them (merges them to the main branch) this way:

• He takes one branch.
• Integrates the branch to the main line.
• Runs a subset of tests.
• If the tests don't pass he rejects the task: two things can happen: whether the developer takes a look into it to see what is going wrong, or the task (which is already in the SCM, and here is the big difference against main line development used in continuous integration) will just wait until the next release to be fixed and integrated.
• The integrator repeats the cycle until there are no more pending tasks.
• Once all tasks are merged he asks QA team to run the whole test suite.

So the test suite is a center piece in our development process. We implement three layers of testing:

• Regular NUnit tests
• Distributed unit testing using PNUnit. Smoke tests based on command line operations. We run them on different client and server combinations. More on this here.
• GUI testing using an automated testing tool: here we run both local tests involving just one client and one server, and tests running on several machines, involving several clients, different OS combinations, different .NET framework installations and so on.

Once all the tests are passed the QA team notifies the integrator and he labels the mainline, creating a new baseline which will be used as starting point during the next week. Developers update their workspaces to the new baseline.

Hi Pablo,

  Is interesting your approach. Isn't it too expensive to wait 1 week until you see if a submitted code is not correct? The code correction will start after few days, maybe the developer is working on something else, he has to step back etc.

  Let me give you how we are doing in the projects were I participated:

• continuous build using Cruise Control .Net, automated NUnit unit tests run.
• nighhtly build which creates and msi files which area deployed in a test environment. This contains deployment of web applications, web services and Biztalk Server applications. For the Biztalk msis we are using Scott Colestock's frameowrk which is based on Nant.
• release scripts triggered manually, sets the assemblies version to the set release number, check in teh changes in the source control and creates the msi files

Source control used are SVN and Team Foundation Server. Build script are created using NAnt or MSBuild. Custom NAnt and MSBuild tasks are developed for specific needs.

Issues we had to solve:

• convince developers to react on failing builds without chasing them
• steadily fix problems when unit tests are failing

Cheers,

Tibor Molnar

iQuest Tehnologies, http://www.iquestint.com/

Hi Tibor,

Tibor Molnar:

Isn't it too expensive to wait 1 week until you see if a submitted code is not correct?

Well, developers don't actually wait that long, they have a test suite they can run to check whether the changes are correct. What we delay is the integration process. A week is probably too long for big teams, I agree.

But, what's the point of the process? Well, I'm using controlled vs continuous integration, which gives us:

• Better mainline stability: integrations are quite controlled, so only code which is known to work enters the mainline. If you do continuous integration, there is only one way to deliver the code: check it in to the mainline. So, you first break the mainline, then you have to fix it. With our mechanism this is different: you can deliver to a branch which is not the main one (the trunk in SVN parlance if I'm not wrong), this gives a lot of freedom to developers.
• Developers can submit code to the version control as many times as they want, without having to continuously solve conflicts or make merges. They just want to create checkpoints which is not the same as integrations.
• We implement full parallel development using branches (take a look here), something which is normally not available with SVN, CVS or TFS.

By the way my oppinion can be a bit biased because I'm a SCM developer. We could use continuous integration too, but we rather prefer branch per task. Normally the main reason to avoid branching (something many developers fear) is because with SVN, CVS, VSS or even TFS it is quite painful... and the heavy thing is not the branching process itself but merging. I was shocked by a recent presentation made by Linus Torvalds a few days ago, but I have to admit he is right on several points.

Cheers,

 Pablo 

I use SVN all the time and I really don't understand what the problem with merging is. We very, very rarely have conflicts, and if we do, they're almost always easily resolved. Of course, we practice continuous integration, so the chance of merging conflicts is slim.

I don't see the advantages of using a branch per task. The link you refer to, talks about locking files, which SVN doesn't do. And if you don't want a certain task in the release, you can always revert the changes.

Well, the problem with SVN is basically when you do branching. If you don't (which is the usual model in continuous, specially with SVN or CVS), by the way there will be no problems.

Yes, you can revert changes if you don't want a certain task but, where will the changes stay then?

Branch per task is a very interesting pattern to look at: maybe the following post helps making it a bit clearer: it is about task isolation.

Don't you find it useful being able to make intermediate commits while you are still under development and you don't feel safe to commit it to the main line?

Cheers,

pablo

psantosl:

Well, the problem with SVN is basically when you do branching. If you don't (which is the usual model in continuous, specially with SVN or CVS), by the way there will be no problems.

I've seen this asserted a couple of times now. But I don't see examples of what kinds of problems might be encountered. I'm interested, since we use branching a lot, and I cannot remember SVN ever holding me back.

psantosl:

Yes, you can revert changes if you don't want a certain task but, where will the changes stay then?

Well, you just commit a new version with the changes reverted. So the changes are still in the history.

psantosl:

Branch per task is a very interesting pattern to look at: maybe the following post helps making it a bit clearer: it is about task isolation.

I use Test Driven Design (TDD), so I don't run into the DiffDebugging scenario all to often. And we have CruiseControl telling us what tests fail and what changes were made since the last successful build, so CC is (partially) automating the DiffDebugging.

psantosl:

Don't you find it useful being able to make intermediate commits while you are still under development and you don't feel safe to commit it to the main line?

Nope. With TDD, I'm always safe to commit to the main line :-P The trick is to split tasks up into tiny bits of work that are tested well and thus can be checked in.

I've seen this asserted a couple of times now. But I don't see examples of what kinds of problems might be encountered. I'm interested, since we use branching a lot, and I cannot remember SVN ever holding me back.

Well, I'm not currently using SVN, but AFAIK: repeated merge is a problem with SVN: you merge a branch, you continue on the branch, you ask to merge again, and SVN will try to repeat the full merge, unless you remember the specific changeset. This is something that won't happen to you with a system like Clearcase, Accurev and of course nor Plastic SCM. Is this a big problem? If you try to handle a thousand branches a year it will be. ;-). Ok, you won't using your model.

My point is that I use branching not only on the structural layer but also on a daily basis.

About the problems (just a few extracted from the SVN website):

• Merge tracking: Track which changesets have been applied where, so users can repeatedly merge branch A to branch B without having to remember the last range of revisions ported.
• Ttrack renames in merge (from http://subversion.tigris.org/issues/show_bug.cgi?id=898). This almost avoids renaming and branching working together.

Well, you just commit a new version with the changes reverted. So the changes are still in the history.

Ok, this one is worth a graphic... :-)

I'm not telling you can't do the same with SVN, but normally you will run into problems.

Ok, you have your vision so I won't try to convince you, but I find both diffdebugging and development much more productive when you have private branches, something is enforced by many SCM tools out there (I'm afraid not the gratis ones).

psantosl:

Ok, you have your vision so I won't try to convince you

And I guess I won't convince you, since you sell a competing product ;-) So let's agree to disagree and move on with our lives. 

X-)

Ok, in fact implementing continuous integration is something some of our customers are currently doing, exactly following the pattern you've described... But I'm still more confortable with branching patterns...

Cheers

psantosl:

repeated merge is a problem with SVN: you merge a branch, you continue on the branch, you ask to merge again, and SVN will try to repeat the full merge, unless you remember the specific changeset

Yes, this is indeed a problem. With SVN, you don't merge branches, but individual revisions. For us, that is not a big deal. For Linus Torvalds, well, I can see his problem. (But that doesn't mean it's ok to call people who work differently 'brain dead')

Hey Pablo,

Thanks for the information and the links. I've been having problems with the scalability of continuous integration for some time using both SVN and Perforce. I've been told by "experts" on many occassions that branching is the cause of many problems. Having watched the tech talk and reviewed some of the web sites, I think that what they meant was that branching is poorly supported in many common SCM systems and therefore causes many problems. I'm definitely going to experiment with the branch per task model.

Cheers,

Paul
 

Hi Paul,

Please take into account that you will need a SCM which fully supports branching (and specially good merging), otherwise you'll end up hating "branch per task"...

Cheers,

pablo 

We have a fully automated build process, with no automated testing (which we hope to rectify in the future). The build process is started manually. We build an ASP.NET (C#) application, so breaking the build process is pretty hard, and not something we have to guard against, so we don't do continuous integration or builds. If we should implement automated tests, then I would at the least have the build regularly scheduled.

The build process is on a dedicated machine, and we use Visual Build Pro as our tool of choice. To run it, you go to the build machine, open the build script for the product and version you want to build, and press Shift-F7. Pretty straightforward. The process gets the latest revisions from SCM (we use StarTeam), attaches a label to them, and builds the .NET solution, which outputs an MSI. It also updates dedicated DBs, and copies the MSI and DBs to a network drive for use by whoever needs it. It sends out an email upon completion.

Andy

www.clinphone.com/ctms 

 HI,

   I am new to this team and build process. I used CC.Net 1.3 with Msbuild. I am able to configure my build process using Msbuild. But  I am not able to configure MSbuild script which would create a MSI setup file in a custom folder. If you have simple confiugration file which fulfills my requiement please post it. That would be useful for the entire community.

Thanks. 

I have created a small, e-booklet that implements a complete MSBuild setup using VS 2005, NUnit, Subversion, Installsheild, QTP, Sandcastle, FxCop, NCover and NCoverExplorer.  You can download the booklet from my blog here.  The build can do the following steps:

1. Get the latest source from the repository
2. Build the databases
3. Compile the source
4. Run the unit tests
5. Produce code coverage results
6. Produce FxCop results
7. Build MSDN style help documentation
8. Backup the databases
9. Version the databases and source code
10. Build a msi
11. Deploy the msi
12. Run the regression test pack using QTP
13. Send notification via e-mail
14. Tag the repository

 Hope you find it useful.

 Carel