Why DevOps Needs Mandatory Vacations

“If a team doesn’t have time for retrospectives, you should hold them twice as often.”

It’s 5:00 p.m. on Friday, and you’re just shutting down your computer to kick off the weekend. A message pops up in Slack:

Christopher Krebs, former director of the U.S. Cybersecurity and Infrastructure Security Agency talk about the very real security threat disinformation can cause. Live at DevOps Connect 2021 on May 19.

“Congratulations, you’ve been selected to take the next week off! We’re cutting off access to all work-related systems and revoking your credentials for the week. See you the following Monday.”

What just happened?

There’s an ancient security practice worth reviving in the age of DevOps: mandatory vacations. In practice today, it typically looks more like cutting off access when an employee is taking an already planned vacation, rather than forcing them to take a spontaneous one.

But the goal remains the same: quickly assess bottlenecks, critical dependencies and security vulnerabilities. The side benefit? Your employees actually disconnect completely when they go on vacation!

The Double-Edged Sword of DevOps

DevOps practices increase a team’s end-to-end ownership, responsibility and autonomy. Team members modify code, deploy infrastructure and handle operational duties – sometimes all in the same day. Bringing Dev and Ops teams closer together results in reduced lead time, faster deployments and higher-quality software.

But blurring the lines between Dev and Ops requires broad access and permissions. When everyone on the team has access to everything, there’s no forcing function to prevent relying on individual or shared accounts. This access proliferation results from a lack of clear indication on which credentials, accounts or permissions authorize each responsibility. Critical tasks may depend on a single person’s account. Credentials may be shared and never rotated.

For example, deployment may depend on a service account, or bot, for which everyone has a copy of the API token. Admin or sudo rights may be delegated on a case-by-case basis to individuals, rather than role-based access control or group management.
The result: a confusing quagmire of roles, permissions and dependencies.

Reviewing and Remediating

Explicit review of all accounts and permissions is an onerous, audit-like chore. While it might be necessary, particularly in regulated environments, it doesn’t mesh as well with agile, iterative, automated and just-in-time principles that fuel DevOps.

Mandatory vacations provide an alternative. It’s a simple concept: cut off a team member’s access when they go on vacation.

Depending on your setup, you might use this time to disable accounts, revoke privileges, rotate credentials, remove them from groups and even archive any home directories left over in production.

The danger lies in leaving dependencies on personal accounts for day-to-day operations. From a security perspective, personal accounts are much more likely to be compromised or abused by a disgruntled insider. They’re an automatic back door into your deployment and production systems. From a reliability and operations perspective, a dependence on personal accounts means that changes to personal accounts (such as an employee leaving or losing access) may cause unforeseen outages. When you control the change as part of a mandatory vacation, you’re prepared for a change that could potentially cause things to break and can trace it back to the root cause quickly. Nothing like breaking the build to force a cleanup of technical debt! This technical debt could include critical nightly cron jobs left in home folders or owned by a single user, personal credentials hidden away in build secrets or operational practices that were assuming an individual’s identity or role.

As a bonus, mandatory vacations also help ensure you keep your on-boarding practices up to date and robust. While re-provisioning your well-rested colleague upon their return, you are forced to acknowledge which privileges and permissions are truly necessary. Depending on the team’s growth and maturity, that may mean creating well-documented practices or enabling full user management automation.

Finally, this provides an opportunity to promote key security principles. Are you enforcing least privilege where you can, giving users the minimum level of permissions needed to do their job? Are you granting access to shared folders in your password manager? If so, did you rotate the stored credentials when your colleague “left” the company, however briefly? This might provide an excellent example of why it’s time to implement multi-party authorization (MPA), just in case gaps like these persist, even after a few mandatory vacations.

Mandatory vacations provide a ‘testing framework’ for access proliferation. Repeating difficult agile practices frequently helps streamline the process simply by doing it more often. If your team doesn’t have time for retrospectives, you should hold them twice as often. Testing your access management practices every time a colleague goes on vacation provides a natural, easy cadence to ensure your practices remain up-to-date – without having to scramble after a breach or when a team member leaves permanently.

Take a Vacation, for Everyone’s Sake

You might think you don’t have time to deal with the ramifications of random mandatory vacations. But imagine how much more difficult it is to discover a single chokepoint during a live incident! If you don’t have the time to test off- and onboarding accounts, that implies your access management practices are rusty, poorly defined and thus, unavailable when they’re needed in a rapid-response scenario. Laptops and accounts will be compromised. Staff may leave or join for unforeseen reasons.

So please, take a week off … for your sake, and ours.