Thread: How to solve consecutive Full GCs?

Full GC is an important event in the garbage collection process. During this full GC phase, garbage is collected from all the regions in the JVM heap (Young, Old, Perm, Metaspace). Full GC tends to evict more objects from memory, as it runs across all generations. A Full GC event has multiple phases. Certain phases of the Full GC event (like ‘initial-mark, ‘remark’, ‘cleanup’,..) pauses all the application threads that are running the JVM. During this period, no customer transactions will be processed. JVM will use all the CPU cycles to perform garbage collection. Due to that CPU consumption will be quite high. Thus in general full GCs aren’t desired. Needless to ask the desirability of consecutive full GCs. Consecutive full GCs will cause following problems:

1.CPU consumption will spike up.

2.Because JVM is paused, the response time of your application transactions will degrade. Thus it will impact your SLAs and cause poor customer experience.

In this article, let’s learn about this consecutive full GCs – What is it? What causes it? How to fix it?

What is Consecutive full GCs?

It’s always easier to explain through an example. So let’s examine the GC log of a real world application, which suffered from this consecutive full GC problem. Below are the graphs generated by GCeasy tool by analyzing garbage collection logs. Notice the highlighted portion of the first graph. You can see the full GCs to be consecutively running (red triangles in the graph indicates full GC). If full GC runs consecutively, then it’s indicative of a problem.

consecutivefullgc-3.jpg
Heap graph generated by GCeasy tool – showing consecutive full GC


consecutivefullgc-4.jpg
Graph generated by GCeasy tool – showing poor reclamation of memory despite full GC

What causes Consecutive full GCs?

Consecutive Full GCs are caused because of one single reason: Under allocation of JVM heap size or Under allocation of Perm Gen/metaspace. It’s indicative of the fact that application needs more memory than what you have allocated. In other words, you are trying to fit in a truck load of objects in a small compact car. So JVM has to work very hard to clean-up the created garbage, to fit in to a small compact car, to make room for actively used objects.

Now you might have a question, my application was running fine all along, why all of a sudden I see this consecutive Full GC problem? That’s a valid question. Answer to this question could be one of the following:

1.Your application’s traffic volume has started to grow since the last time you have tuned the JVM heap size. May be your business is improving, more users have started to use your application.
2.During your peak volume time period, more objects would get created than normal time. May be you didn’t tune your JVM for peak traffic volume or your peak traffic volume has increased since the last time you have tuned the JVM heap size.

How to solve consecutive Full GCs?

Consecutive Full GCs can be solved through one of the following solution:

1. Increase JVM Heap Size
Since consecutive Full GCs runs due to lack of memory, increasing JVM heap size should solve the problem. Say suppose you have allocated JVM heap size to be 2.5GB, then try increasing it to 3 GB and see whether it resolves the problem. JVM heap size can be increased by passing the argument: “-Xmx”. Example:

-Xmx3G

This argument will set the JVM heap size to be 3 GB. If it still doesn’t resolve the problem then try increasing the JVM heap size step by step. Over-allocation of JVM heap is also not good either, it might increase the GC pause time as well.

2. Increase Perm Gen/Metaspace Size
Some times full GCs can run consecutively, if Perm Gen or metaspace is under-allocated. In such circumstance just increase the Perm Gen/Metaspace size.

3. Add more JVM instances
Adding more JVM instances is an another solution to this problem.When you add more JVM instance, then traffic volume will get distributed. The amount of traffic volume handled by one single JVM instance will go down. If less traffic volume is sent to a JVM instance, then less objects will be created. If less objects are created, then you will not run into the consecutive Full GC problems.

Validating the fix
Irrespective of the approach you take to resolve the problem, validate the fix in the test environment before rolling out the change to production. Because any changes to JVM heap settings should be thoroughly tested & validated. To validate that problem doesn’t resurface with new settings, study your GC log with GCeasy tool. It has the intelligence to spot and report whether the application is suffering from consecutive full GC problem or not.


Reference Article: https://blog.gceasy.io/2016/11/22/el...tive-full-gcs/