Codecov hacking

TL;DR: Rather than a summary, here is the explanation of how serious the Codecov hack was:

Codecov is a company offering a code quality analysis tool (among others 😉 ), their name coming from "code coverage", that is to say the rate (percentage) of executed source code of a given program when a test suite is launched.

Codecov proposes a bash script (https://codecov.io/bash) to download and execute, allowing the integration of the tool to its software and software factory (CI/CD pipeline).

Codecov is available for free in a number of code repositories such as GitHub, GitLab, BitBucket... Moreover, there are nearly 9500 open source projects that use it:

Including Kubernetes, alibaba, AWS (s2n-tls the TLS implementation of AWS), ansible, Apache projects, Ruby, node.js, urllib3 (integrated in many Python projects), nextcloud... but also Adorsys ("Open Banking Gateway", used throughout Europe and PSD2 / Payment Services Directive 2 payment gateway)

This is a "supply chain attack", i.e. the compromise of a subcontractor to compromise another target(s).

On April 1, and this is not a joke, an attentive Codecov user noticed that the 160bit SHA-1 digest of the script was different from the one provided on GitHub, so there's good to be picky and check the digests of what you download 😉.

He looked further and discovered the addition of a command line retrieving the list of installed repositories (git remote -v) with environment information ($env) and sending it all to a server hosted in the US at DigitalOcean (rented or hacked itself). As the tool is usually used on software factories (continuous integration (CI) environments), the attacker was able to retrieve IDs and passwords of code repositories, API tokens, secret keys...

It is the script hosted at Codecov that was modified (https://codecov.io/bash) around January 31, 2021 and not the same script on the GitHub repository, so there is no history of this modification.

Therefore, the attackers were able to retrieve sensitive information for 2 months!

After being informed of the attack, Codecov removed the malicious script but waited almost 2 weeks before communicating widely about it, leaving 2 more weeks for the attackers to use the stolen data, this is unfortunate, even if it is understandable that they prioritized their customers.

https://about.codecov.io/security-update/

The chain of dependencies in development makes it complicated to know if you have been hacked.

So you need to be able to know:

If you use Codecov directly or indirectly; If you have been a victim of the attack; Which secrets have been stolen in order to change them.

1. If you use...

In order to know if you use Codecov directly, the easiest way seems to me to be to search in your source codes... if not in your documentation 😉. A big "grep -i -a -n https://codecov.io/bash" should be enough

On the other hand, to know if you are using Codecov indirectly, if it is used by a dependency that you download when building your application or installing it, you need to... build or install. Therefore, you have to deploy your different solutions completely and then look for the presence of the Codecov script. You can also look for the script on the already deployed environments. It's a long work 😥.

You also have Yara rules here: https://github.com/Neo23x0/signature-base/blob/master/yara/mal_codecov_hack.yar

If you have other ideas or techniques, feel free to reply 👍, I will complete here.

2. If you have been a victim...

In order to know if you have been a victim, you can already look for the Codecov script and if you find it, in its malicious version, then... 😉.

You can also look in your proxy logs, firewal... if the ip 104.248.94.23 has been contacted. Provided you have this type of trace 😉.

You also have Yara rules here: https://github.com/Neo23x0/signature-base/blob/master/yara/mal_codecov_hack.yar

If you have other ideas or techniques, feel free to reply 👍, I will complete here.

3. What secrets...

To find out what might have been stolen, simply run the "git remote -v" command on your software factory (CI environment) and your deployment environments where you found the malicious script.

Once you have identified what could have been stolen and on which environment, you will need to change (and revoke) api tokens, passwords...

From there... you will have to investigate again to see if these passwords, tokens... have been used, what has been accessed, stolen... and good luck to untangle the ball of potential compromises!

In the Codecov press release, I love the answer to a particular question on the FAQ:

Is it safe to use Codecov systems and services?

Yes. Codecov takes the security of its systems and data very seriously and we have implemented numerous safeguards to protect them.

Yes, of course, they weren't going to say otherwise 😉 :