This page describes a proprietary continuous integration and continuous delivery system that I built while employed as a Linux system administrator and Linux architect.
| Key details | |
|---|---|
| Brief description: | A build framework, infrastructure, and front-end to build operating system packages from source and deliver them to the appropriate environment's repositories for deployment. |
| Consumer: | Linux system administration team, change management team, internal controls team |
| Impact to consumer: |
|
| Technical features: |
|
| Technologies used: | Bash, GNU Make, Apache HTTP Server, Perl, HTML::Mason (Perl) |
The custom CI/CD system grew from initial primitive build scripts as internal demand grew for the ability to create OS-native packages for easy deployment, and then to be able to manage the versions being deployed to various test environments separately to production.
For a short time, the Jenkins tool was used to run the builds. However, when compared to just running the builds from a trigger script, it was found to be fragile, have a much larger attack surface, and did not provide any features that were useful to the team.
The components of the CI/CD system are:
- A Subversion repository server.
- A set of build servers - one for each supported operating system, such as CentOS 5 through 7, AlmaLinux 8 and 9, Debian 12, and so on.
- A Makefile and supporting shell scripts, held in a Subversion repository, which is checked out by the build user on every build server.
- A trigger script on each build server, running from cron, which runs a build when a commit is detected.
- A build artefacts repository, to which the build servers transfer the signed packages once they have been built.
- A management server running the package repository management web interface, written in HTML::Mason (Perl), linked to Active Directory for operator authentication.
- Two package repository servers (one for production, and one for all
development and test environments), which all endpoints point their package
managers (
yum/dnf,apt, etc) to.
All internally deployed packages - including the packages holding the configuration for the build servers themselves - are stored in the Subversion repository (1).
Each build server (2) signs the packages it builds and transfers them to the build artefacts repository (5), where they are indexed by a simple script to provide a list of all available packages and versions, for other parts of the system to retrieve over HTTPS.
The build process (3) on each build server, triggered by a commit (4), generates a package in native format - RPM, DEB, etc - for each of the package source directories in the Subversion repository. Packages may also pull in external resources from the build artefacts repository if they have been signed by the administration team.
At this point, the packages are available for selection by the operator in the package repository management tool (6). Here, the operator can choose which version of each package is the "current" version in each environment and for each OS and endpoint type, as well as being able to add notes to a package such as "held at version 1.2.3 until version 1.2.4 is tested in ticket 1234567" so that new versions are not accidentally deployed prematurely.
Operators can also view the changelog of selected packages. For example, during the deployment of a change, this allows them to check that the change described in the package changelog matches the description in the Request For Change document, and ensure that no other, unrelated, changes are accidentally included.
The figure above shows the main page of the package repository manager, comprising the package selection section to the left, and the package repository and build server status information to the right.
From the main page, after selecting which environments, OSes, and estates (types of endpoint) are involved, and applying the relevant filters, the operator enters the name of a package. The versions of that package are then displayed, and can be modified for each environment / OS / estate individually or together.
When the operator makes changes to the package selections, the package repository servers (7) copy the relevant package versions from the build artefacts repository and re-index, so that the endpoints will receive that version when they next run an update - typically using the endpoint management system, which is integrated with this tool.
The management tool triggers the changes on the package repository servers
by means of an API call, implemented by a CGI script written in Perl. This
CGI script passes information to the indexing script, written in Bash, which
performs the relevant transfers and runs createrepo or the
equivalent (depending on the package type), feeding information back to the
front-end by updating a status file accessible over HTTPS.
All changes are written to a history log, as well as to syslog, tagged with the operator's Active Directory username. The log is visible to the change auditing tool I provided to the change management and internal controls teams.