For the last year we have been focusing on integrating our Sitecore solutions in a Continuous Integration environment. We now have two larger Sitecore solutions and many smaller ones in the environment, where we can build, test and deploy environments, including Sitecore Items. We have gone through some iterations of the environment and find that we are ready to share some of our experiences.
Continuous Integration
The main purpose of continuous integration is to have an environment where code is build and tested at every commit, secondarily the build can be pushed to TEST or QA environments. The ultimate goal, for us, is to be able to test, validate and push Sitecore Sites from development to production, without any manual steps.
Continuous Integration and Sitecore
A Sitecore solution is composed of three parts, 1) the code 2) the database build by the developers (templates, layouts etc) and 3) the user content that is stored under sitecore/content and sitecore/media. The code is easy to integrate in a automated build, tests and deployment since this is the main goal of most CI setups, but where it gets more challenging is when the Sitecore Items get involved. Without the Sitecore Items in the CI setup there remains several manual parts for a release cycle.
Automating Sitecore builds
and release cycle
CruiseControl.NET has been chosen as build server platform and custom NAnt tasks are used in the build scripts for search and replace, robocopy and Sitecore Item control.Item Versioning and (Auto)Serialization
Sitecore Items is a big part of a solution and the first requirement is that the Items can be put under Source control and versioned correctly, the native implementation from Sitecore is based on a manual "œdump" of what the developers choses to save to disk.In a multi-developer environment this lead to developers dumping the whole sitecore database at each checkin, and reverting the full database takes over 10 minutes sometimes 30 so this was not done before each development session.
When renaming and deleting Items, this setup lead to Items being re-added to the tree after they had been deleted by another developer, because the other developers had not removed them before they themselves made a item dump and checked in.
What was needed was a synchronization model for keeping the file system up to date with Sitecore without user involvement, thereby the only files the developer checks in are the ones he has made changes to. This has been done by developing an auto serialization that attaches to the Item events and e.g. deletes the file if the Item is deleted, or serializes content items if there releted template is changed. We do not yet import file changes fully automated to Sitecore, for the fear of loosing work in case of merges and conflicts, but it would be a natural way to go.
This module means that we now have a stable model for versioning our sitecore item development, this was a base requirement for moving forward. We have full traceability on the system development since every developer runs his own Sitecore code and database and both code and items are versioned.
Build Flow
We have done several iterations of the build scripts. Our latest iteration pushes the files before changing the configurations, and performs testing directly on the target system.
Repository Branching Strategy
Our latest project phmetropol.dk sometimes involved up to 8 developers, and 60+ change requests in some releases. For controlling this we created code branches in our repository for UAT and PROD. UAT had the latest code that was for final customer validation (UAT = User Acceptance Testing) and PROD contained the latest released code. This setup gave us the possibility to keep active development in the main trunk (incl. Sitecore Items) and releasing well tested code+Items in the UAT branch with a potential fix or two, and finally being able to hot fix (incl. Items).Without version control on the Sitecore Items, we believe, such releases would have had major risks for errors.
that needed testing and We have also tried where the test was done locally before the files where moved, where items where separated from the build files but we had the need to introduce repository branching for adding a UAT and PROD build, and the configuration management when involving staged setups made the model complicated.
Configuration Management
The Sitecore solutions we have build in our environment have all involved staged setups, we have found two strategies for controlling the configuration under automated build.Staged build and environment creation
genvej.nu is created with two websites in the Visual Studio project, one site for CMS and a site for the Front site, config files for both environments are checked in and most of the rest is copied from the Visual Studio build process from bin/ressources/layouts etc. to front. The only things the build script had to change was the environment elements in each web.config.
Staged and environment creation
phmetropol.dk was our next project we wanted to create the staged files based on the cms files only, for keeping the development environment more streamlined. This means that we, from the NAnt build file, first created a staged copy of the source, changed the config to match a staged setup (using include files for overriding where possible) and last customized based on the target environment.This worked well in our development stage but when we saw the need to introduce the UAT environment and build this setup has become a bit complicated to maintain.
Sitecore Items
Controlling Sitecore Items in the build process required that we had them under version control and branched. Once this is in place the next step is to push them around on our different environments.For reverting and packaging a Sitecore context is required, we have therefore implemented a web service in the Sitecore admin level, that gives us access to reverting, serializing and packaging features. Combined with a NAnt task we can control items deserialization or packing from the build script.
We have initially implemented a full revert or partial, that we can control through a web service on the Sitecore environment to give us Sitecore context. In a default cycle the only path we revert are template and layouts, but we can also revert all of Sitecore in both core and master, the problem is that this takes up to 30 minutes, so our default behavior is only the main elements.
After some iterations, we are currently looking at a model where we push the items to the target using robocopy and use the change log to revert only the affected items on Sitecore, we have done a proof of concept and this feature is in active development. This will mean we can revert sitecore items on each build because it will be done very fast. In the process we har also evaluating ideas for partial reverts on the developer machines but in that area we have yet to find the best model.
Once reverted and tested we can package e.g. template and layouts, and save a package that is then used for deployment to production, or backup. When deploying we never take items related to Content or Media, every other Item is excepted to come from our controlled builds and packages. We have had some surprises with Web forms for marketers and Sitecore/system in this context but we will keep that for another post.
Testing
Basic code test is implemented with NUnit and could be run after the build is done on the build server. We have chosen to deploy the site first so we can run functional tests involving page clicks and search results for tests. For allowing tests on the Site we have used WatiN.For making functional tests on controlled data and not client content we have introduced two parallel sites in development, one used in the building of NUnit tests that validate main functionality and the other used by the client to build their content. In the development phase we let the clients build up data on the qa environment and we actually fetched data with our tool web service that we pushed to our test server so we could see more real content in the test environment and still have a semi stable environment for client qa.
Other levels of testing and quality control have been implemented using FxCop, StyleCop and TotalValidator and NArrange in the development cycle.
Conclusions
We have used a reasonable amount of time building this setup. We are now able to construct larger Sitecore solutions, with tests, branching and a release cycle with full source control and minimal manual processes.In our next project we hope to take the last step and implement continuous deployment, so production is updated straight from our build server. We also expect to implement Web Config Transformation now available with Visual Studio 2010.
We see a Continuous Integration environment as a natural part of a large Sitecore solution and an important move towards a release process.
We would like to know about your expirences on Continuous Integration, so please leave a comment or blog post.
Jan Hebnes, Head of .Net development at 1508 A/S
Originally posted to: http://www.15all.net/2010/11/12/sitecore-in-a-continuous-integration-setup/
No comments: