Avoiding and Debugging Java Memory Errors
How to avoid and debug Java memory errors on CircleCI.
The Java Virtual Machine (JVM) provides a portable execution environment for Java-based applications. Without any memory limits, the JVM pre-allocates a fraction of the total memory available in the system. CircleCI runs container based builds on large machines with lots of memory. Each container has a smaller memory limit than the total amount available on the machine.
By default, Java’s is configured so that it will use:
- More than
1/64thof your total memory (for Docker Medium with 4GiB of RAM this will be 64 MiB)
- Less than
1/4thof your total memory (for Docker Medium with 4GiB of RAM this will be 1GiB).
As of June 3rd 2020 these limits are visible when using the Docker executor. This means that the recent versions of Java will correctly detect the number of CPUs and amount of RAM available to the job.
For older versions of Java, This can lead to the JVM seeing a large amount of memory and CPUs
being available to it, and trying to use more than is allocated to the container. This pre-allocation can produce
Out of Memory (OOM) errors, which are difficult to debug because the error messages lack detail. Usually you will
137 exit code, which means the process has been
SIGKILLed by the OOM killer (
137 = 128 + "kill -9").
You can see how much memory your container is allocated, and how much it has used, by looking at the following files:
Recent versions of Java (JDK 8u191, and JDK 10 and up) include
UseContainerSupport which defaults on. This flag enables
the JVM to use the CGroup memory constraints available to the container,
rather than the much larger amount of memory on the machine.
Under Docker and other container runtimes, this will let the JVM more accurately
detect memory constraints, and set a default memory usage within those constraints.
You can use the
MaxRAMPercentage flag to customise the fraction of available RAM that is used,
Manual memory limits
Even with cgroup support, the JVM can still use too much memory, e.g. if it executes a worker process pool. To prevent the JVM from pre-allocating too much memory, declare memory limits using Java environment variables. To debug OOM errors, look for the appropriate exit code.
Using Java Environment Variables to Set Memory Limits
You can set several Java environment variables to manage JVM memory usage. These variables have similar names and interact with each other in complicated ways.
The table below shows these environment variables, along with the precedence levels they take when using different build tools. The lower the number, the higher the precedence level, with 0 being the highest.
|Java Environment Variable||Java||Gradle||Maven||Kotlin||Lein|
The above environment variables are listed below, along with details on why to choose one over another.
This environment variable takes precedence over all others. It is read directly by the JVM and overwrites all other Java environment variables, including command-line arguments. Because of this power, consider using a more specific Java environment variable.
_JAVA_OPTIONS is exclusive to Oracle.
If you are using a different runtime,
ensure that you check the name of this variable.
if you are using the IBM Java runtime,
then you would use
This environment variable is a safe choice
for setting Java memory limits.
JAVA_TOOL_OPTIONS can be read by all Java virtual machines,
and you can easily override it
with command-line arguments
or more specific environment variables.
This environment variable is not read by the JVM. Instead, several Java-based tools and languages use it to pass memory limits to the JVM.
This environment variable is exclusive to Clojure.
to pass memory limits to the JVM.
JVM_OPTS does not affect the memory of
nor can it directly pass memory limits to Java.
lein’s available memory,
To directly pass memory limits to Java,
This environment variable is exclusive to
See the Gradle documentation for memory settings.
This environment variable is exclusive to Gradle projects.
to overwrite memory limits set in
See the Maven documentation for memory settings.
This environment variable is exclusive to Apache Maven projects.
to overwrite memory limits set in
Debugging Java OOM Errors
Unfortunately, debugging Java OOM errors often comes down to finding an
code 137 in your error output.
Ensure that your
-Xmx=NN size is large enough for your applications to
completely build, while small enough that other processes can share the remaining memory of your CircleCI
Even if the JVM’s maximum heap size is larger than the job’s limit, the garbage collector may be able to keep up with the allocation rate and avoid your process using too much memory and being killed. The default number of threads allocated to the garbage collector is based on the number of CPUs available, so the cgroup visibility change made on June 3rd 2020 may cause your application to consume more memory than before and be OOM killed. The best fix for this is to configure the maximum heap size within the job’s available RAM, which will cause a full GC to be triggered soon enough to avoid breaching any limits.
If you are still consistently hitting memory limits, consider increasing your jobs’s RAM allocation.