Six months ago, my friend and colleague Adnan Khan started researching a new class of CI/CD attacks. Adnan grasped the significance of these attacks after executing them against GitHub to gain total control of the GitHub Actions runner images. GitHub’s bug bounty program scored this vulnerability as “Critical” and paid a $20,000 reward.

Following this operation, Adnan and I teamed up to scour the internet for other organizations vulnerable to similar attacks.

What we found shocked us.

An astounding number of GitHub repositories were vulnerable to these attacks. Like, thousands. We had to limit our searches to exclude any repository that didn’t have a bug bounty program. Then, we had to exclude all bug bounty programs with a max payout of under $2,000. Our results left us with dozens of repositories that matched our standards, representing only a tiny subset of the overall attack surface. 

Over the last four months, we have been going through this list of vulnerable repositories, performing advanced CI/CD exploitation, and submitting our results through the respective bug bounty programs. So far, we’ve submitted over 20 bug bounty reports, raking in hundreds of thousands of dollars in bounties. 

We began by discovering every nuance of GitHub Actions exploitation, executing tools, tactics, and procedures (TTPs) that had never been seen before in the wild. As our research advanced, we evolved our TTPs to attack multiple CI/CD platforms, including GitHub Actions, Buildkite, Jenkins, and CircleCI.

Who Have We Hacked?

We were most surprised by the types of organizations that were vulnerable. Our targets included the world’s most advanced tech companies, often in the AI/ML or Web3 space. These companies have invested hundreds of thousands of dollars into security and sport some of the best-funded bug bounty programs. Yet, they were in the dark about these new attacks. 

We are still not allowed to talk about most of the operations. However, some highlights so far have been:

• Compromising PyTorch releases (update – a full walkthrough of this attack is now available in Playing with Fire – How We Executed a Critical Supply Chain Attack on PyTorch)
• Being able to backdoor the crypto wallets and nodes of several blockchains, some with billion-dollar market caps 
• Compromising GitHub’s runner images
• Providing explicit steps to compromise TensorFlow releases
• Compromising a CloudFlare application
• Gaining remote code execution on a domain-joined Microsoft machine by exploiting Microsoft Deepspeed

How Do These Attacks Work? The Three Step Plan

Step 1: Find a Typo

We begin by finding a typo in a GitHub repository. Yes, you heard that right. 

We submit a pull request (PR) for the typo we find and wait until our PR gets merged. Once that happens, we are now a “contributor” to the repository.

GitHub has a setting that allows contributors to trigger GitHub Actions workflows when submitting a pull request (PR). This setting is enabled by default and allows contributors to execute code on any GitHub Actions runner attached to the repository or organization (barring some exceptions).

If the repository only uses GitHub-hosted runners, this isn’t much of a concern. The dangers arise when they use self-hosted runners (cue ominous music).

Self-hosted runners are GitHub runners hosted by the repository owners rather than GitHub. Gaining remote code execution on a self-hosted runner allows attackers to search the filesystem for secrets, scan the internal network, and retrieve cloud credentials— all the usual RCE activities. 

Step 2: Shell the Runner

If we discover self-hosted runners attached to a repository with the default workflow approval requirement, we use our contributor account to gain code execution on the runner and set up our C2. The impact of this RCE is typically determined by the type of self-hosted runner. 

Self-hosted runners can be ephemeral or persistent. Ephemeral runners use a fresh VM or container for each workflow, whereas persistent runners stay alive after the workflow is complete. Persistent, self-hosted runners are an attacker’s Shangri-La because they execute multiple workflows during their lifetime, sometimes across several repositories. Ninety percent of our research has been discovering methods of post-exploitation for persistent, self-hosted runners. Getting RCE on a runner can occasionally be impactful alone, but post-exploitation is where you find the level of impact that has led to these critical reports.

Step 3: Own Everything

When we compromise a persistent, self-hosted runner, we’re usually able to execute a complex series of attacks that result in compromising repository releases. These could be GitHub releases, releases stored in AWS, Docker containers, NPM packages, PyPi wheels – you name it, we’ve pwned it. These post exploitation techniques will be a focus of future articles, when we dive into technical walkthroughs of some of these attacks.

For example, when we operated against PyTorch, we could have added our own malicious code to their releases on nearly all of their release platforms. Then, the next time an unsuspecting organization downloaded PyTorch from GitHub, or cloned the PyTorch repository, they’d be running our code. Check out our full PyTorch attack walkthrough for more information. In the hands of a nation-state, this single attack could be devastating. In fact, many of these attacks could have caused their own version of SolarWinds or the recent Ledger crypto hack.

What’s Next?

As the bounty submissions slowly start to close, we’ll be able to let you in on the secrets behind these techniques. If we are accepted to a certain conference in the state of Nevada, we may even be able to show you these attacks in real time.

Keep an eye out for future blog posts to learn what it takes to hack some of the world’s most technologically advanced organizations.

Want to hear more? Subscribe to the official John IV newsletter to receive live, monthly updates of my interests and passions.