This page describes a proprietary mechanism for ensuring that only the appropriate people have access to the root passwords of each server, which I built while employed as a Linux system administrator and Linux architect.
| Key details | |
|---|---|
| Brief description: | A mechanism to automatically rotate all server root passwords, and store them in a way which guarantees that only authorised operators can use them. |
| Consumer: | Linux system administration team |
| Impact to consumer: |
|
| Technical features: |
|
| Technologies used: | Bash, GnuPG, Apache HTTP Server, Perl |
Many of the corporate servers are integrated with Active Directory, and have
sudo access in place for members of the Linux system
administration team. This is convenient, but also fragile and arguably
insecure. Active Directory integration breaks down very occasionally, and
when deployed across hundreds of servers, "very occasionally" occurs several
times per month. In terms of security, granting full administrative access
via sudo means placing a lot of trust in the AD infrastructure.
I implemented several safeguards to mitigate the risk of AD account
compromise, rejecting sudo attempts if an account's password
has recently changed, or if it has not been in the administration group for
long enough - but these can misfire, so there always needs to be a fallback
administrative access method even for AD-integrated servers.
Having the same root passwords everywhere, or never changing them, both present an unacceptable level of risk. Using password vault or IAM software such as Bitwarden or CyberArk adds complexity, increasing the attack surface and increasing fragility. This solution was developed to avoid those pitfalls, being tailored to the situation where nearly 1,000 servers are being maintained by a team of under 10 people.
The root password randomiser generates a new root password and encrypts it using a GnuPG keyring containing the current public GPG keys of each authorised operator. The encrypted password is transmitted to a central repository; when an operator needs to retrieve the password for a server, they run a helper script which downloads the appropriate file and decrypts it with their private key. This means that every server has a different, random, root password at all times.
On each server, the randomiser is deployed as an OS-native package
containing a shell script and a crontab. The script runs every few minutes
to check whether it's time to change the password - this is usually a random
amount of time since the previous change, but if it detects that
"su" has been recently used, the next change will be brought
forward.
When the randomiser changes the password, it transmits the encrypted copy of it to a Perl CGI script on a central repository server.
Operators retrieve and decrypt the encrypted password when they need it,
using a small shell script. I have also added a helper script to automate
connecting to servers, retrieving the password, and becoming root, using the
GPG agent. A variant of this script permits connecting to multiple servers
at once, using tmux.
All servers transmit their system logs to a central point, and this includes
the use of "su" to become root using the retrieved passwords.
All commands run during the root session are recorded to syslog as well.
This, combined with the access logs on the password repository server,
provides a robust audit trail of administrative activity.
The exact steps carried out by the randomiser are proprietary, but I designed them to ensure that the password is never transmitted in the clear, and is not actually changed until the repository has acknowledged receipt. Any problems with the process - such as a failure to change the password, or an inability to transmit it - are written to marker files which are detected by the Zabbix monitoring agent and trigger alerts to the system admin team.