A major source of frustration for both customers and employees alike has always been the typical development, testing, staging, and production (DTSP) model. The DTSP model is essential for providing the checks and balances necessary to efficiently maintain a high-availability production environment. Our frustrations were towards previous employers. With exception to only one enviroment, not a single other employer was willing/able to implement the critical “staging” facet of the DTSP model. Typical excuses were either a lack of money or time.

I personally believe it was a lack of willingness to solidify the internal processes necessary for a successful DTSP model, as TOCICI has not only a fully flushed-out testing & staging environment, we assembled it for well less than $500 (yep, that's less than five hundred dollars for a complete environment). The lack of a fully-functional DTSP model is common throughout small businesses, including many ISPs, development shops, and other IT related businesses, what's your current services provider's excuse?

The theory of development, staging, and production (DTSP) is simple.

Most developers will test and develop their code on their own desktop or laptop computers. This ensures an environment that they can fully access, and effectively use as a sandbox for testing and experimentation.

Next, is a dedicated testing and integration server for each developer; enabling them a more structured environment, with more controlled and documented changes, yet very little influence from system administrators.

Here, you'll see developers testing their integrated code, to evaluate whether their revisions will behave as they expected. Simply put, it's their formal development system. Once convenienced that their contribution to the application is ready for prime time, developers propose posting to the staging environment.

The staging environment facilitates a mirror copy of the production environment; its primary purpose is to test a complete application on a duplicate of production, to ensure that revisions do not adversely affect the production environment. No actual code development would ever occur on a staging server, at-most you'll only see minor OS parameter and application adjustments.

The staging environment is the final step before production deployments. In TOCICI's environment, a final approval process follows staging to ensure that all interested parties sign off before the application goes into production. When the application has approval, it moves to production via automated deployment processes. Requiring automated processes for staging & production level deployments ensures that all facets of a production deployment have been fully-documented.

A fine line often divides who has approval authority, and who provides oversight of the production deployments.

• The server-support team (System Administrators/SysAdmins) have approval authority over everything pre-production; giving them veto power over system revisions. After all, the SysAdmins are primarily responsible for the production server's uptime, performance, and overall availability.
• Marketing and public relations departments also share a veto role; its their job to manage the organization's image.

Within the development, marketing/PR & SysAdmin groups, there is often little dispute. The answer to “why” is founded in this four-stage process; starting with testing & integration, all groups have visibility and input. This ensures that by the time development revisions are production-ready, all affected parties are also ready.

Finger pointing is the best evidence of not having a good DTSP model. SysAdmins would chronically blame developers for deploying faulty code, and then SysAdmins would often be forced to manage environments in very poor health. When something doesn't work, the other party is the first to receive blame. Years ago, I remember when a developer was once given access to staging; he insisted there was an OS-level fault preventing his code from functioning, after weeks of finger-pointing, a SysAdmin closely reviewed the code & found the developer's error. Issues were promptly addressed, and we never permitted developers direct access to staging nor production again.

Over the next few months we'll detail more about this environment, but suffice it to say, we test everything before it hits production. Its been said that a picture is worth a thousand words, so here's a snapshot of our staging environment:

At TOCICI, we've seen countless datacenters and small business server rooms. Back in May of 2009, we discovered what is still believed to be the epitome of a small-business's disaster waiting to happen (pictured below). This kind of situation would never be found in our environments, nor with the customers we manage infrastructure for; there's no excuse for it. Considering that organization suffered through a devastating half-company layoff barely a month after the photo was taken…how would the visual of this server room speak to you? We challenge you to submit something even worse - we'll post the best…errr…worst pictures soon.