Catastrophe Averted: Things Found During Network Assessments
by Chris Geiser - [email protected]

In my time at PCS and with the help of our illustrious systems engineers, I€ve participated in hundreds of network health assessments. Today, I€ll take a look back and share some of the common and some of the bizarre findings these assessments have revealed over the years.

Backup and Disaster Recovery
There€s an old saying in the IT world: There are two types of people in the world, those who have lost data and those who will. If your company is lucky to be in business long enough, one day you will need to need to restore data from an IT services catastrophe. It€s a sort of an IT Judgment Day. The bad news for many small businesses is that their IT Judgment Day will also be the first time they get to validate their backup process (fingers crossed!). Here are some very common backup and disaster recovery mistakes we€ve observed and help businesses correct:

1. Extremely Local Backup - Backing up to the same hardware as production machine? So, if the server fails or gets damaged, won€t you lose all of the backup data too?
2. Let€s Get Cooking - Storing backup data in the same building, but in a €fire safe€? Most fire safes will protect paper through a fire, but try cooking a data tape or hard drive at 1,800F. Nothing is backed up until it exists at two distinct geographic locations.
3. Bad Tape Monkey - This common (and derogatory, I might add) phrase is often used to describe an employee who dutifully changes backup media every day. Trouble is that these employees sometimes become so hypnotized by their routine that they neglect to notice that the backup job has failed (every night) for a period of days, months, or even years.
4. Incremental Forever! Sounds great because it makes backup jobs smaller and faster. But come IT Judgment Day, you better be ready to produce the base backup and every incremental backup file since then. Any break in the chain (backup file lost, overwritten, or damaged) and your entire restore effort is shot.
5. To The Cloud€ The €cloud€ gets backup data off-site and is usually very inexpensive. But relying solely on cloud backup often proves dangerous. Cloud backup works great for single file restorations, but when a company suffers a more catastrophic loss of an entire server, the restore process leaves employees and customers twiddling their thumbs for longer than the business can bear. For example, say your server has 500 GB of data. Now let€s urgently request all of that data from the fairly anonymous 3rd party provider cloud backup provider (who€s under no obligation to work quickly) and pour that down your 3 Mb/sec Internet connection. 500 GB x 1,000 MB/GB x 8 bit/Byte = 4,000,000 Mbits / 3 Mbits/sec (Internet speed) / 60 sec/min / 60 min/hour = 370 hours to accomplish the complete download!
6. Data Only - What about server settings? Without server settings, not only will it take much longer to restore services, but the servers will never be the same, meaning there€s going to be unpleasant IT surprises staff in the days ahead.

Security
There€s no question that balancing security versus functionality is difficult. Yet at PCS we€ve never found a network that couldn€t stand to benefit from a tweak or two, without impeding employees€ ability to work, of course. Here are some common findings:

1. A Tunnel to Nowhere - VPN between offices? Yes. But that doesn€t guarantee that interoffice traffic is running through the tunnel. Finding out (in private) that you€ve been passing sensitive data clear text through the public Internet is certainly better than having an angry customer point that out.
2. No Lockouts - Think that forcing complex passwords will suffice? A good tech support provider can pull a report that shows the disturbing amount of failed login attempts to your publicly accessible server. It€s quite easy for hackers to automate millions of attempts on networks without lockout policies. Given enough attempts, they€ll eventually get in.
3. SETI@home € A few zany IT administrators have been known to donate their companies€ afterhours computer processing power to a noble cause. Yes, that€s right; the computers are enlisted in a nationwide program that analyzes space signals in the Search for ExtraTerrestrial Intelligence.
4. Everyone€s an Administrator! Sounds like a company culture on the verge of utopia, right? Most companies who allow many employees unfettered access will someday suffer a malicious or unintentional configuration change or breach of confidential data that will come back to haunt this decision. Only those who need admin access should have it. All Administrators should have their own admin login so an audit trail eventually leads to an actual person, not just a list of suspects.

Alas, this is but a few of the more severe security and recoverability assessment findings we€ve observed over the years. There€s dozens of other common misconfigurations and oversights to do with the stability/reliability of the computer network as well. System maintenance is a technical field where it€s easy to hide one€s tracks. That€s why having checklists and an occasional second set of technical eyes can easily save the business from an embarrassing and/or expensive event.

If you are ready to take advantage of the opportunity for a complete network assessment, please call Databranch today, to speak with an Account Manager.