The notorious SolarWinds hack of 2020 sent shock waves across the software as a service world.
In the attack, hackers gained access to the development pipeline for a SolarWinds financial management product— Orion. Once in, they were able to insert malware into the system. That malware then got rolled out by Orion themselves as an update patch.
More than 18,000 customers installed the malicious patch. The hackers now had a way into thousands of organizations that used the software, gaining access to their data, networks, and systems. They could install additional malware and penetrate even further. It also enabled them to infiltrate the data and networks of partners and customers of those infected businesses.
Thus, a data breach in one company’s development pipeline escalated into a danger that impacted thousands of other businesses and people. It was one of the biggest breaches of the century. Indeed, the hack may not yet even be over: investigators aren’t sure if the malware has been fully disabled.
The episode was huge news. Above all, it underlined the imperative for software developers to secure their development pipelines.
The Wolf in Sheep’s Clothing
Software developers use CI/CD (Continuous Integration / Continuous Delivery) pipelines, along with approaches like Agile, to facilitate rapid service improvements to deliver value to customers as quickly as possible. The pipelines superbly facilitate this. They allow companies to move nimbly in fast-moving, competitive markets.
Using CI/CD pipelines, individual improvements or fixes can be coded by a developer and then built, tested, and deployed using an automated task sequence (the ‘pipeline’). By harnessing automation, these pipelines pull together tools and jobs to hugely speed up service updates.
Hence why they pose such a temptation to hackers. If malicious hackers can access the pipeline, they can insert code that can then be deployed to customers. Such incidents are known as ‘supply chain’ attacks because they enable the hackers to hide behind the business supplying the service.
CI/CD pipelines are undoubtedly a great opportunity for software businesses to compete. Whatever service is being developed— from customer service software to inventory management systems— performance updates and functionality innovations are likely to be key to success. But the danger of more attacks like the SolarWinds hack is very real.
Once they have crept unnoticed into a pipeline, the wolves can dress in sheep’s clothing. And the danger will probably not be noticed by the client’s security tools because it will appear to be part of the service. In this way, hackers can work, undetected, for months, infiltrating further and further.
The Risks
As shown in the image above, a CI/CD pipeline is inherently complex. Beyond the temptation they pose to hackers, a number of their features can increase vulnerability.
- Complex ecosystems: The technical stack is usually diverse and complicated. Without rigorous oversight of the entire pipeline, security gaps may appear, dangers are unidentified, and some vulnerabilities left open.
- Fast-paced change: New languages and frameworks keep appearing and being incorporated. Change can be constant. Working at high volume and speed can open up new weaknesses.
- Automation: The reliance on automation can lead to complacency or oversights, creating gaps for attacks to slip through undetected.
- Cloud-hosted: Pipelines often involve cloud-hosted services, introducing additional vulnerability.
- Third-party dependencies: Many third-party tools will be utilized across the pipeline and as part of the code itself, complicating the CI/CD ecosystem further and adding risk.
Of course, despite their risks, all the above are massive benefits. For example, just like customer service automation is leading to ever-improving consumer experiences, automation is essentially what makes DevOps possible. And, indeed, as part of a much broader security framework, automation can play a valuable role in securing a pipeline.
Yet high-profile breaches like SolarWinds have highlighted how such rewards can also carry dangers. Securing CI/CD pipelines has thus become a far more prominent issue in recent years.
6 Best Practices to Secure your CI/CD Pipeline
While not a complete guide to securing your pipeline, here are six practices to consider.
1. Maintain Rigorous Oversight of the Pipeline
First off, it’s crucial to maintain a clear overview of the entire CI/CD pipeline. It should capture all the processes, resources, and tools used (both in-house and third-party). It should also detail the people: Who can access the various stages, and who must have access.
It should be regularly reviewed and revised to accurately reflect any changes. Part of that review should be looking for all the exploitable security weaknesses across the whole piece. Ultimately, the pipeline is only as strong as its weakest link, as hackers only need to access it in one place: a door is a door.
2. Model All Possible Threats from the Start
There should be a security-focused culture across the entire team. Every proposed change should be considered from a hacker’s perspective. How might this change be exploited for malicious purposes? What could resolve that threat?
If everybody cultivates an eye for security in their work, vulnerabilities can be curbed.
3. Manage Dependencies Rigorously
Just as the service the pipeline is developing is constantly evolving, so will all the third-party tools upon which that pipeline draws. All third-party tools should be audited regularly and kept up to date. This includes, for example, all the various testing tools shown in the image.
Moreover, tight procedures should govern how those third-party tools are utilized in the pipeline. For instance, they should be pulled through an internal proxy rather than directly from the internet. This will ensure that malicious versions are not inadvertently drawn on.
4. Manage Access to the Pipeline Tightly
Access should be kept as tight (i.e., as limited) as possible. Continuous analysis of user identities (internal and external) and their access rights should ensure that permissions are restricted only to what is required. For instance, developers should only have access to the repositories and stages they are directly working on.
There should be no use of shared accounts, non-business emails, and self-registering. And all users should be aware of the security standards needed to safely manage pipeline risk. To give one example, everyone should understand the need to keep keys, credentials, and secrets out of scripts.
Where possible, further tests and checks should then be in place to double-check compliance with such standards.
5. Monitor All Access to the Pipeline
The SolarWinds breach demonstrated the danger of malicious artifacts being introduced to the pipeline and sneaking through to production.
To avoid this, every addition or change to the pipeline should undergo a strict process of review and approval before being permitted. Rigorous validation processes should be a part of the pipeline map. Ideally, only artifacts created by approved pipeline users should be allowed.
Tools are available to monitor unusual interactions or changes, i.e., indications of possible breaches. For example, drift detection and data lifecycle management tools might play a role.
6. Cultivate Blameless Postmortems After Incidents
Mistakes do happen. The focus should be on doing better next time: fail fast and learn fast. That requires a culture that emphasizes learning, not blame and punishment. Invest time and resources in team building to foster collaboration, support, and robust governance.
Embed Security Awareness in Teams
Just as Big Data has led to rapid evolutions in data handling tools, from RDD (Resilient Distributed Dataset) to more complex datasets, automation is leading to rapid innovations in DevOps and pipeline management.
For businesses that offer software services, their development pipelines are at their core: either an essential source of future value and reputation enhancement or, on the other hand, a site of vulnerability and danger. To manage those risks, businesses must treat the security of their pipelines with the utmost importance.
These suggestions are, of course, just the tip of the iceberg. Securing pipelines requires a broad and rigorous approach. Pipeline security depends on the right culture as well as tools and processes. In a DevOps world, everyone working on the pipeline needs to understand the salience of security concerns both to their work and the bigger picture.
