I would like to share with a recent, real life, story of what happens to small businesses when there is little to no documentation.
I’m hoping to encourage you to review the documentation standards you have set forth for your small business; and potentially to do an in-house audit to ensure critical areas are covered.
In late August 2012, we received a call from the CEO of a small business whose web development person left their employment. They found out about our server administration services from SoftLayer as we are a SoftLayer certified partner.
They needed to update their web site for which they did not have the FTP login credentials; and they needed to generate a CSR (Certificate signing request) in order to renew the secure certificate for their web site so that https would continue to work.
Together, we hoped that given the server login credentials (which they did have on hand) that we could locate the FTP user, reset the FTP user password, and test FTP access with that information; and then use the server-based tools to generate the CSR for the secure certificate, and install it when they received it from the digital ID provider.
To keep the story short without going into the server administration details, the information they had on file was for a Citrix XenServer which was running multiple virtual machines. The web site for which they needed the FTP reset and a CSR generated was on one of the virtual machines.
There was no documentation as to which virtual machine other than a public IP address of the web site.
The non virtual equivalent is that you are given the keys to a safe. You open the safe and find several other safes within; and while you might be able to guess which safe is the right one within the safe, you don’t have the means to open it.
SoftLayer, whose extremely well automated portal, provided one of several means available to document the server environment; but notes were not put into the notes area for which private IP address belonged to which virtual machine which may also have helped.
While we were able to narrow down which virtual machine (aka safe) was most likely the correct one, ssh (remote access) appeared to be filtered by IP address… and you needed to access the virtual machine in order to tell it which IP addresses to allow.
The bottom line for this small business is potentially rebuilding everything from the ground up for costs in the double digit thousands of dollars (if not more).
If you are the CEO, COO, CSO, CTO, President, owner, steward, or otherwise “the buck stops here” person, when was the last time you audited what documentation is in place for the following?:
- Employee handbook – ensuring it covers documentation expectations and requirements.
- Web site(s) – login credentials for every application, control panel, FTP, email, statistics / analytics along with daily, weekly, biweekly, monthly, etc. processes and procedures along with application names, versions, etc.
- Server(s) – specifications, login credentials, public IP, private IP. If there are virtual machines, the same — do you know where your servers are located? Names and contact information of responsible parties having what responsibilities?
- Change log – what installations, deletions, changes have been taking place — date, time, where, what, who, why, how, notes, etc?
- Contact information – name, company, mailing address, physical address, phone numbers, email, and when or why would they be contacted.
- Other? — What’s necessary for someone to take over your responsibilities if you are the last one standing, and need to pass on the baton?
As you do the audit, ask yourself (and hopefully check your thought process with trusted other parties) — if the responsible person for jobs a, b, and c were inaccessible tomorrow, would someone be able to take over quickly just based on the documentation that we have in place?
If the answer is “no,” then a level of priority should be given to making sure there is enough documentation (that is reviewed and tested for quality assurance) so the processes, procedures, tasks, and related responsibilities can be easily picked up by a new party.
Lastly, who knows where the documentation is located, and how to use the documentation?
Have you run into any nightmares in your business that proper documentation beforehand could have prevented or made less costly? Please share in the comments below.