Whenever I read about a data breach, I immediately smell a cybersecurity process failure.
In cybersecurity, we sure love our processes. We have them for firewalls, intrusion detection systems, IP scans, pen testing, zero trust architecture, and double secret two factor authentication probation, but I’m going to dive into the most exciting process we use: Change Control!
Well, that’s not true; many employees despise filling out change control request forms and compare it to a day at the DMV.
They’re boring, it’s tedious, time consuming, redundant, stupid, yada yada yada, I get that; but still it really bugged me to see a completed change control request submitted for approval that contained “Done” in the results field, with no other details about the change.
Helpful? Not very, and it certainly won’t be in the future for the employee who made the changes, when the archived request record is pored over by a team of forensics experts because of a data breach. They’ll have questions, lots and lots o’ questions, and my memory isn’t that great. I’d have a hard time coming up with good answers, so I’d rather fill in the details while they’re fresh in my mind.
Me? When it came to change control requests, I secretly loved them, because it was the best way to explain what I’d done. Of course, I’m the guy who reads an email 10 times before sending it out into a cruel and uncaring world, but that’s my own personal issue.
The bottom line is I’ve always looked at change control as a safety net; one that’s caught me, and my team, many times just before we fell to our cyber doom.
In the world of cybersecurity, there are many processes to follow, but I always felt change control may be the most important process we had to ensure a solid security posture. Pretty much every task we did was under the change control umbrella; name one, and I can usually tie it back to a change control request in our system.
Change control is a nitty-gritty, detailed view of when, why, where, what, and how changes were made to a “company resource.” Note my emphasis on “company resource.”
Servers, routers, phones, pretty much any device you can name on the network, as well as access to restricted areas like the data center, all are company resources. Changes across the enterprise for all PCs are covered by change control, and in a tightly run organization, PCs are locked down so that employees can’t make changes.
The change control process is a formal one, not something written on the back of an envelope. It’s designed to keep changes organized, and under control, while also maintaining clear evidence of the changes made for future reference. The company’s intention is to save employees from making costly errors, and to stop hackers from creating a data breach.
I’ve worked with a few IT Cowboys, impatient employees who make changes without bothering to log them because it slows them down.
Change control is DESIGNED to slow them down; it tames that Wild West attitude to shoot from the hip, because when they do, occasionally they’ll shoot themselves in the foot. Their impulses can result in outages or data breaches if not contained, with seriously negative results for the company’s financial bottom line.
This is a painful memory, but okay, I’ll rip off the scab, and share it. One of my worst experiences occurred when a network engineer on my team made router configuration changes on the fly; the result was an error that blocked connectivity for thousands of our users.
He was so shook he couldn’t recover; since he hadn’t written down the changes he’d made, there was no way to systematically back them out, and he was spinning around in circles.
I got the emergency call about this self-inflicted wound after midnight, and I knew it meant trouble.
If I’d even had the slightest hint he was going to make any changes, I would’ve stopped him, but I was snoring away when he decided to make his moves. Yeah, I admit it; I snore.
Since the goal of the change control process is the absolute minimum disruption to company services, let’s just say this was a major fail.
Because it was an internal error, we can credit it to “The Accidental Employee Hacker.”
A real hacker couldn’t have messed up the company any worse than his friendly fire, because he’d taken down 25 offices outside of the United States with just a few keystrokes.
It was his bungled attempt to do something good; he thought his changes would improve the network. Turns out they didn’t.
The truth is, most internal errors, and some data breaches, are the result of employees thinking they know how to do things better, and faster, when they shortcut a company’s processes.
Back in the real world, here’s how it should work: The change control process is always started by a formal change control request submitted in the system, and a manager must approve it. This means John Wayne would’ve had to ask before he fumbled on the router configuration.
Many items fall into the category of those required to be submitted and approved in the change control process before actions can be taken on them:
- Single servers and multiple servers
- Single user and multiple user account creation
- Operating system installs and patches
- Enterprise application installs and patches
- Phone systems
- Network equipment and leased network lines
- Cubicle moves and entire office relocations
- Access to restricted areas
Still not sure what requires a change control request?
The following are questions employees aren’t always sure about, and I’ve heard them a million times over the last 35 years:
- Does adding a user account to a server require a change request? Yes, it does.
- Does the physical move of a sales person to another cubicle require a change request? Yup.
- Does changing an employee’s phone extension require a change request? You bet.
- Does the upgrade of a server operating system require a change request? Without question!
- Does the upgrade of an application on a server require a change request? Most definitely!
- Does changing the security classification of a server require a change request? OMG, yes, yes, yes!
- From a cybersecurity perspective, what’s more important than keeping a uniform defensive front to avoid internal screw-ups and data breaches? Nothing!
Change control is the key, because it ties into every other cybersecurity process to provide that desperately needed uniform protection!
If I was going to perform a typical cybersecurity task, like applying a critical patch to a server, I’d follow these change control process steps:
- Research if the patch is applicable, and are there prerequisite patches? Is a reboot required?
- Can the patches be applied during the agreed-upon maintenance patch window, to avoid extra user disruptions? Or is the maintenance window too far into the future?
- Pre-flight check: Do I have everything I need before submitting the change request?
- When I’m confident it’s ready, I’ll submit the patch change request for manager approval.
- I’ll implement the patch install, and test the change was successful by checking that the patch shows up on the server patch list.
- Once I’m done, I’d submit the change request for closure, and be sure to include a detailed description on why it was successful; this will keep my manager’s questions to a minimum.
- The manager agrees the change was successful, and approves to close the request; at this point it becomes a read-only change control record and is archived in the change database library.
In the 7th step, notice how the change control request morphed into a read-only change control record?
Change control records are critical pieces of evidence, called proofs, for cybersecurity forensics specialists, as well as internal compliance auditors.
In cases of a data breach, forensics experts will analyze proofs very closely, because the records will list every change made to the environment. This will help them discover the origin of the breach; as they don’t take kindly to employees changing information after a breach, hence, the read-only record.
Most people are unaware of what it takes to stop hackers who attempt to breach a company, but to do it successfully, another very important aspect of the cybersecurity team’s job is maintaining compliance in perpetuity. Therefore, the cybersecurity team must defend their servers against the internal auditing process, much like PhD candidates will defend their theses to a group of professors.
During internal corporate audits, the change records for a server will be submitted to auditors as evidence the cybersecurity team has maintained compliance per the company’s policies. Like forensics specialists, auditors are trained to go over the records with a fine-toothed comb.
There were times I had to defend the compliance status for 15 different servers during an internal audit, all in a single month. This is a very difficult high-wire act, and I don’t recommend it unless the employee really knows what they’re doing.
Some considerations to note about a company change control request system:
- There’s an added expense for the change control software.
- This isn’t over-the-counter software; it has to be customized for the client.
- Employees will need to be trained to use it.
- Employees may dislike the amount of data input required to fill in the online forms.
- By their nature, change request forms contain redundant fields.
I’ve taken part in emergency enterprise deployments for changes on a massive scale, ones that installed critical security patches on over 10,000 PCs at a time. There’s no way possible a group of employees could successfully roll-out security patches on that scale manually; the only solution is to utilize endpoint manager software, and automate the installs.
The good thing is a single change request for an enterprise deployment covers them all.
The change control process also allows for reports containing the results to be attached, so the success or failure of the patch application for each PC is immediately apparent.
I’d suggest you always stay on top of your open change control requests, and I personally set multiple alerts for the days when they were scheduled for my servers. A note about patch implementation dates; don’t miss them. Ever!
Try explaining to a software development manager why a scheduled patch update was overlooked on their server, one with 3,000 programmers furiously coding on it to hit a software release date.
Reschedule isn’t a word that exists in a software development manager’s vocabulary when it comes to possible outages on their server because a missed patch installation “fell between the cracks.”
Lack of a good change control process, and the solid implementation of it, will cause internal problems like that, and can also result in data breaches.
On the other hand, a tight change control process will keep the organization’s train on the tracks, and running in the right direction.
The bottom line is change control and cybersecurity go together like peanut butter and jelly. One without the other can lead to a missed software release date, or a data breach, and will leave everyone in the organization with a very bad taste in their mouth, but without a cold glass of milk to wash it out.
Johnny Young (aka JohnE Upgrade) is a 35-year veteran of the cybersecurity industry with a storied career helping blue chip companies avoid data breaches and the devastating financial and consumer trust losses that come with them. In October, he’s launching CyberD TV, a video streaming service dedicated to cybersecurity training for the general public. His book, entitled “Don’t Hack,” will be released in November. He lives with his antihacker team, which consists of Scarlett the Cyber Bird and Elf the Wonder Dog.