Setting up Continuous Integration for ASP.NET applications

This article focuses on setting up the right hardware and software environment needed for Continuous Integration (CI) and Agile Development principles to be implemented for an ASP.NET application. In order to achieve CI for a website project you’ll need to go through at least tree stages: the development stage, the quality assurance or preview stage and the release or production stage. For each stage you can have a physical server that hosts all the virtual machines and tools needed to accomplish the stage requirements. Since the preview environment needs to be as close as it can get to the production one you’ll have to make identical Windows, IIS and SQL Server settings on both servers.

If you are not familiar with TFS branching and Team City, please read Visual Studio TFS Branching and Merging Guidance and You’re deploying it wrong! TeamCity.

Development stage setup

The Development server will host four virtual machines:

VM#1 TFS

  • Team Foundation Server
  • SQL Server 2012 used by TFS only
  • 12 Core 2.00 GHz, 16 GB RAM, 250 GB VHDX

The TFS machine is your main code repository and holds all the artifacts required to build the project. Outside of the source code, with TFS and Visual Scrum you can manage the development process in an agile manner.

VM#2 TC

  • Team City
  • SQL Server 2012 used by Team City only
  • NuGet internal server for shared components between projects
  • 12 Core 2.00 GHz, 16 GB RAM, 150 GB VHDX

The TC machine is the place where all automated builds, tests and deployments will take place. The Team City server uses NuGet in order to download all the required components used in your project before building it. If you are using the official NuGet repository make sure this virtual machine has internet access. The TC server should have access to the TFS one  and the best way to do this is to make a dedicated user on the TFS server so when Team City connects it will have full access to all your project branches. The TFS and TC machines have a critical role because you’ll use them to build, test and deploy to the Preview and Production stages as well.

VM#3 DEV SQL

  • SQL Server 2012
  • 32 Core 2.00 GHz, 26 GB RAM, 500 GB VHDX

The DEV SQL machine hosts the database for your project. The development team should only have access to this database and they should only make changes to the structure using the VS.NET database project that’s part of the DEV branch. Making changes directly in the database using SQL Manager will break the CI process and make your preview and production stages inoperative. One of the toughest challenges in implementing CI for ASP.NET & SQL projects is to synchronize the SQL Server database schema and data between the Development stage and the Production stage. A database needs more than versioning and source control, therefore each time you modify the data you’ll need to generate scripts that can perform the same transformations on both preview and production databases. Unfortunately the VS.NET database project doesn’t synchronize data, only schema so you’ll need to use RedGate’s software to have a fully automated database deploy. Read more here Continuous Integration for SQL Server Databases.

VM#4 DEV Webserver

  • IIS 8
  • Web deploy 3.0 for IIS
  • URL Rewrite 3.0 for IIS
  • 12 Core 2.00 GHz, 16 GB RAM, 150 GB VHDX

The DEV Webserver is used for automated and human testing of every app feature that is in the development stage. At each check-in on the DEV branch, Team City will build, test and deploy the app to the DEV web server. Keep in mind that the ASP.NET website should connect to the DEV SQL server in order to access the database. This way you will have a complete environment for the development stage.

Preview stage setup

The Preview server will host two virtual machines:

VM#1 PREVIEW SQL

  • SQL Server 2012
  • 32 Core 2.00 GHz, 60 GB RAM, 500 GB VHDX

The PREVIEW SQL machine hosts the database that has the same data as the Production server. Its main goal is to test the deployment scripts that sync the DEV SQL server with the Production SQL server. Before any deployment you should make sure the Preview database is a perfect clone of the Production one. This can be easily done with a pre deploy script programmed inside the VS.NET database project.

VM#2 PREVIEW Webserver

  • IIS 8
  • Web deploy 3.0 for IIS
  • URL Rewrite 3.0 for IIS
  • 32 Core 2.00 GHz, 16 GB RAM, 150 GB VHDX

The PREVIEW Web server machine is used for customer validation. Specifically, it is used by the product owner to validate the current Sprint.

Production stage setup

The Production server will host two virtual machines:

VM#1 PRODUCTION SQL

  • SQL Server 2012
  • 32 Core 2.00 GHz, 60 GB RAM, 500 GB VHDX

The PRODUCTION SQL machine hosts the live database for the website. You should sync the data between the Production and Development database at the beginning of every Sprint, you should backup and restore the Production database on top of the Preview database before any script deployments take place on the Preview stage.

VM#2 PRODUCTION Webserver

  • IIS 8
  • Web deploy 3.0 for IIS
  • URL Rewrite 3.0 for IIS
  • 32 Core 2.00 GHz, 16 GB RAM, 150 GB VHDX

The PRODUCTION Webserver machine is the live version of your project. The Production app should have its own versioning policy: at every new deployment the version should increment. You can automate this using the Team City assemblyInfo patcher feature.

For this setup I am using three identical physical servers with 32 logical processors, 80 GB RAM and 800 GB HDD RAID 5 local storage plus a  2 TB SAN storage.  You should calibrate your hardware wisely, keep in mind that for merging a VHDX snapshot you’ll need free space as big as your snapshot is. Each server has Windows 2012 Server Datacenter with the Hyper-V feature installed. All the virtual machines hosted on these servers have the same operating system installed. I am using  the Windows 2012 Server Datacenter edition because it comes with unlimited virtual licenses.

Branching setup for TFS

For each stage you have to create a separate branch: the Main Branch for the Preview, the Release Branch for the Production and finally the Development Branch for each development cycle. At the beginning of every Sprint a good practice will be to create a new Development branch from the Main one. After the Sprint is done, you should merge de active Development branch into the Main branch, and after you’ll get the client approval you should create a new Production branch from the Main one. This way you’ll have a dedicated Development branch for each Sprint and for each new release you’ll have a different Production branch.

The above image illustrates how to resolve a critical bug on the Production branch. Let’s assume your team is working on Sprint 1 and at some point a bug is spotted on the public website. You can’t fix the bug on the Development branch because you’ll have to wait for the whole Sprint to be completed in order to publish the changes in Production. The solution to this problem is to fix the bug on the Main branch, when the hotfix has passed all the tests. You can immediately merge the Main to the Release Branch and deploy it on Production. At this point you have to merge the changes done to the code by the hotfix into the Development branch. This operation can impact you team velocity for Sprint 1 but, at some point in time you’ll have to do it because as long as the hotfix is not merged into Development branch you cant deploy to Preview.

Build and Deploy setup with Team City

For each stage you have to create two build configurations in Team City: one for compiling and automated tests and one for deployment of the build on the web & database servers. If you don’t know how to setup Team City please read this article by Troy Hunt.

Every time a check-in is made into the Development branch it will trigger a new build, test and deploy on the Team City server. If these steps are successful you’ll end up with a new website version on your development web server and database. If the build & test task takes a long time and it uses lots of server resources you can schedule a nightly build from Team City, so the process described in the above schema will be executed once a day. For the Release branch it’s better to schedule the deployment when your website has the lowest peak traffic. Keep in mind that on any deployment the IIS application pool will be restarted so if you are using App-Fabric for session and cache storage, the impact of a new deployment will be minimal. Otherwise you’ll lose all session and cache data.

What’s next

In future articles I will write about:

  • Using URL Rewrite for static files versioning
  • Minify js & css on deploy using MSBuild & Team City
3 Responses to Setting up Continuous Integration for ASP.NET applications
  1. Shankara Narayanan Reply

    Hi

    Great article. Can you also five insights on the hardware configuration for these?

    Thanks
    Shankar.

    • stefan Reply

      Sorry for the late response. I’ve updated the article with hardware specs for each virtual machine.

  2. Baruch Reply

    Hi Stefan,

    Great post! The only thing I am missing here is binary (NuGet packages) repository for 3rd-party dependencies and for the build outcomes. Not sure if you propose TeamCity should host them or maybe TFS?

    Just wanted to bring Artifactory to your attention. It can be used as NuGet inhouse repository and has OSS TeamCity plugin for integration with the build process. Here‘s a brief overview of how Artifactory fits in.

    Regards,
    Baruch

Leave a Reply

Your email address will not be published. Please enter your name, email and a comment.

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>