In the ever-shifting landscape that is developing web applications, there are a few things that have remained relatively constant:
- It will always take longer to develop an application than anyone involved thinks it will.
- No matter when your deadline is, you will always run short on time towards the end.
- No matter what project management methodology you use, (or what software supports it) at crunch time, people will resort to sticking Post-it notes to your monitor. (…but that’s another show.)
- No piece of software will ever be perfect, but any “extra time” will generally be eaten by trying to make it such.
- Deployment (putting the application into production) will always be a rush job done at the last minute.
The first four things are more or less laws of nature as far as I am concerned. No matter how hard everyone works or how on the ball everyone is, software is hard to write and it’s even harder to say how long it will take. What isn’t a constant, though, is that last one, despite the fact that most people do it that way. In general, there is a fair amount of wiggle room in the area of what order things get done in.
Generally, when you’re building an app, the minute you have a loose idea of what’s getting built you want to build something and get it in front of people so they can start picking it apart. This idea, in fact, is at the core of a lot of very well-meaning project management methodologies. (I’m looking at you, Agile.) The sooner you can show something working, the better the people who sign the checks will feel about your project. (At least at the beginning, anyway.)
The flaw in this approach is that it gives everyone a false sense of security about the schedule. Even as work proceeds at a brisk pace, most people ignore that most crucial of steps: getting the thing to work in “real-life”. Deployment is left for the last possible second, when the project isn’t quite done, but must be pushed live. There’s generally a lot of hand-wringing and brow-mopping and a lot of impatient people at this stage. (A good PM might even build in “go live” time for this, but it will always be eaten up by last minute development.) In the worst case, there will be no actual deployment strategy and someone from the development team will end up dragging files from one machine to another, manually. Things that worked perfectly on all the developers machines will break catastrophically on the live servers. Someone will have forgotten to purchase a license for some critical piece of software the app needs to run. The firewall ports that the web server uses to talk to the database server won’t be open, and neither will the help desk of the hosting company. All of this is, of course, terrible. To paraphrase Scott Hanselman, “You get to drag files from your local machine to a server for deployment once. After that, if you’re not typing ‘deploy’ somewhere to push your application, YOU ARE STUPID.”
The second worst case is that an already under-the-gun developer will be pulled off some other very important task (you know, like, finishing the damned thing) to crap out some way of deploying the app to production. It will often end up being a brittle, undocumented kludge that some poor intern will have to sift through later. Everyone will hate it and it will reflect poorly on the team that spent six months developing an otherwise amazing application.
Why is deployment (and the setup therein) such a loathed task? I would say it’s because it’s a necessary but highly unglamorous task. It’s also one that very few people will get. Show your client the cool new feature you just added to the image gallery and they’ll buy you a pizza. Show them your one-button application deployment and they’ll give you a blank stare. For this reason, deployment has always been a last-minute rush job.
I would like to propose a better way. Deploy first. By “first”, I mean that as soon as you have checked in the very skeleton of your application into your version control system, figure out how and where and by what method you will put the thing into a production environment. Then do it. Do it a bunch of times. Get it so that it’s a completely natural, totally painless process that you don’t even have to think about. While it’s my opinion that a deployment-first strategy is good practice for any project, I think it’s absolutely critical to any project using a new or unfamiliar development platform. (Also, as an aside, if you do not have a version control system, please stop reading this. Clean out your desk, leave your ID card with security, walk out of the building and don’t look back. Seriously. Shame on you.)
To deploy your app, you will generally need to take care of all (or most) of the following:
- setup and configuration of production machines (be they real or virtual)
- installation of prerequisite software for your application (web servers, application platforms, RDBMSes, etc.)
- load balancing
- network configuration
- development of a deployment system
If you get this all squared away first, you can merrily write code up until the last possible second, because when you’re done, you push a button and the site goes live. Maybe there will be a speed bump or two in that process, but you won’t be starting from scratch.
Make no mistake. Writing software may be hard, but deployment is always harder. It also (generally) involves coordinating lots of different people, often at different companies, many of whom will not share your sense of urgency. Deploying first exposes a bunch of issues that you can square away long before they become emergencies.
That last item on my list there is often a source of confusion for a lot of people. Why would you “develop” a deployment system? There’s a lot of tools that do this already, right? Well, sure, but you have to set them up and configure them, and in most cases, write scripts that will tell them what to do. This is often a lot less trivial than the makers of those tools would have you believe. Take the time and figure out what you’re going to use and how your going to use it before you even really start developing.
As for how you go about building deployment systems, there are lots and lots of options. The best one for your project will largely depend on what you’re using. In most cases, you can get your build tool (Ant, Maven, NAnt, Capistrano, etc.) to do it for you. In a lot of cases this can get overly complicated, though. My personal favorite way is to write a short, project-specific shell script (though I recently started using Python for this) with good inline comments. Whatever you use, here’s a loose set of steps for any deployment script:
- Pull a copy of the latest code from the version control system (never deploy off of your current development tree)
- Increment the version number (I like using a text file called VERSION in the root of my project)
- tag (or whatever your VCS calls it) the revision with your version number
- copy required libraries into the build tree
- create compressed archive of your build
- transfer the archive to the production system
- unpack the archive into the right place (via remote, perhaps with passwordless SSH or Powershell. Some build tools have this functionality as well.)
- perform any required administration tasks that a new build would need (update database structure, restart web server, etc.)
- email a notification that a build was pushed out
You may need more steps, you may need fewer. Whatever you can reasonably automate, do it. Every step you make automatic will be one less manual headache at the 11th hour. Also, the more manual steps you remove, the more you reduce the likelihood of human error and dependency on a particular human to do deployment.
I hope this will inspire you to think about incorporating deployment-first development into your next project. I can tell you that doing things this way has greatly improved my life as a developer, especially at crunch time.
On August 17th, 2010 at 7:31 pm Eivind Eklund said:
You’re missing a few steps that are usually important:
1. Pre-deployment checks. There should be *some* kind of checks that you do, no matter what your process is. Even if it is just “mentally review what important changes will go out now”. Some checklist somewhere.
2. (For a large scale system) Canary. For any large scale system, set things up so you can deploy to a part of your userbase before hitting everybody. Even if that is just deploying to identified test users.
3. Post-deployment checks. After deployment, there should be some tests you do to see that the deployment worked at least reasonably well. Ideally, you have the combination of whitebox and blackbox monitoring that will alert you if anything broke, including automatically testing that all your relevant important functionality works, and you didn’t just cause slowdowns or whatever. In practice, you’re most likely missing lots of this – but you should at least check that you didn’t break the main page, look at a couple of logs to see that nothing is screaming too badly, and run tests if you have them.
On August 17th, 2010 at 8:21 pm Jaybill McCarthy said:
@Eivind Thanks for the comment. You know, you’re totally right. 1 and 3 really speak to something I didn’t mention at all (for the sake of brevity, mostly) and that’s testing and QA. In a large production system, one would hope that developers would be doing unit tests and then handing a build of to QA. QA would then actually approve (and possibly physically perform) deployments. As far as 2 goes, I’ve never used Canary, but your point is well made. I would think, however, that on a production system, you’d be very gingerly modifying the database rather than doing it with a script.
On August 18th, 2010 at 5:25 am David said:
I totally agree. About a year ago my department created a fully automated daily build and deploy procedure to be run on a QA environment. It was a huge pain to develop, but now we have so much less stress upgrading the production environment.
On August 18th, 2010 at 9:52 am Ed The Dev .com » Blog Archive » Deployment first development said:
[...] is a very cool article that encourages us to face our worst fears: packaging and [...]
On August 25th, 2010 at 8:04 am Matt Wilson said:
I wrote a blog vaguely like this a few months ago. Here’s what I said about deployment:
Deployment should be boring. But it usually isn’t, because developers overlook trivial dependencies they created along the way.
You’ll quickly discover what you forgot to add to your deploy scripts last time if you start every session like this:
1. Reset your sandbox to resemble what is currently live.
2. Check out your code.
3. Run the deployment script.
4. Now start working.
By the time you are ready to release, you will have simulated the deployment process many many times.
Here’s the post in entirety: http://blog.tplus1.com/index.php/2010/05/28/building-a-sane-development-environment/