This page describes a proprietary mechanism for ensuring that only the appropriate people have access to the root passwords of each server, which I built while employed as a Linux system administrator and Linux architect.

Key details
Brief description:A mechanism to automatically rotate all server root passwords, and store them in a way which guarantees that only authorised operators can use them.
Consumer:Linux system administration team
Impact to consumer:
  • Improved security posture
  • Improved visibility, and auditability, of system changes
Technical features:
  • Shell-based system easily understood by the team, with robust error reporting and debugging capabilities
  • Supports multiple Linux variants
  • Minimal overhead with no complicated frameworks or dependencies
Technologies used:Bash, GnuPG, Apache HTTP Server, Perl

Many of the corporate servers are integrated with Active Directory, and have sudo access in place for members of the Linux system administration team. This is convenient, but also fragile and arguably insecure. Active Directory integration breaks down very occasionally, and when deployed across hundreds of servers, "very occasionally" occurs several times per month. In terms of security, granting full administrative access via sudo means placing a lot of trust in the AD infrastructure. I implemented several safeguards to mitigate the risk of AD account compromise, rejecting sudo attempts if an account's password has recently changed, or if it has not been in the administration group for long enough - but these can misfire, so there always needs to be a fallback administrative access method even for AD-integrated servers.

Having the same root passwords everywhere, or never changing them, both present an unacceptable level of risk. Using password vault or IAM software such as Bitwarden or CyberArk adds complexity, increasing the attack surface and increasing fragility. This solution was developed to avoid those pitfalls, being tailored to the situation where nearly 1,000 servers are being maintained by a team of under 10 people.

The root password randomiser generates a new root password and encrypts it using a GnuPG keyring containing the current public GPG keys of each authorised operator. The encrypted password is transmitted to a central repository; when an operator needs to retrieve the password for a server, they run a helper script which downloads the appropriate file and decrypts it with their private key. This means that every server has a different, random, root password at all times.

On each server, the randomiser is deployed as an OS-native package containing a shell script and a crontab. The script runs every few minutes to check whether it's time to change the password - this is usually a random amount of time since the previous change, but if it detects that "su" has been recently used, the next change will be brought forward.

When the randomiser changes the password, it transmits the encrypted copy of it to a Perl CGI script on a central repository server.

Operators retrieve and decrypt the encrypted password when they need it, using a small shell script. I have also added a helper script to automate connecting to servers, retrieving the password, and becoming root, using the GPG agent. A variant of this script permits connecting to multiple servers at once, using tmux.

All servers transmit their system logs to a central point, and this includes the use of "su" to become root using the retrieved passwords. All commands run during the root session are recorded to syslog as well. This, combined with the access logs on the password repository server, provides a robust audit trail of administrative activity.

The exact steps carried out by the randomiser are proprietary, but I designed them to ensure that the password is never transmitted in the clear, and is not actually changed until the repository has acknowledged receipt. Any problems with the process - such as a failure to change the password, or an inability to transmit it - are written to marker files which are detected by the Zabbix monitoring agent and trigger alerts to the system admin team.