Caused by: java.lang.NoClassDefFoundError occur

[huange7]

We made changes to the Java agent version 2.11, and since then, when starting the Tomcat service, we occasionally see the following error:

Caused by: java.lang.NoClassDefFoundError: Could not initialize class java.util.concurrent.ThreadLocalRandom

What could be causing this?

Any response would be greatly appreciated. Thank you.

[steverao]

Based on information you provided, it's hard to debug the cause. If vanilla java agent 2.11 is fine, maybe you can rule out the cause from your own modifications. It may occur dependency conflict.

[huange7]

java.util.concurrent.ThreadLocalRandom is a core JDK class. Is it possible for it to cause a dependency conflict? I'm using JDK 8.

[laurit]

Caused by: java.lang.NoClassDefFoundError: Could not initialize class java.util.concurrent.ThreadLocalRandom

This means that the static initializer for ThreadLocalRandom threw an exception. If you are lucky then there is an ExceptionInInitializerError before the NoClassDefFoundError that contains the stack trace to that exception.

[huange7]

I can't find such logs in the log files. We replace the javaagent package during the process runtime, but I understand that this operation should not affect the startup process, correct?

[laurit]

can't find such logs in the log files

Then you have to find an alternative way to debug this. Generally the first step is to figure out how to reproduce the issue. I guess you could try instrumenting the static initializer of ThreadLocalRandom to print the stack trace if there are any exceptions.

[huange7]

Are you referring to instrumenting the static methods of ThreadLocalRandom? Also, could this be related to the way I’m replacing the agent?

[huange7]

Are you referring to instrumenting the static methods of ThreadLocalRandom? Also, could this be related to the way I’m replacing the agent?

@laurit please check this out

[laurit]

Are you referring to instrumenting the static methods of ThreadLocalRandom? Also, could this be related to the way I’m replacing the agent?

no idea

[huange7]

there is a new exception when export metric:

Failed to export metrics. The request could not be executed. Error message: canceled due to java.lang.ClassCastException: 
Cannot cast io.opentelemetry.exporter.internal.otlp.ResourceMarshaler to io.opentelemetry.exporter.internal.otlp.InstrumentationScopeMarshaler
java.io.IOException: canceled due to java.lang.ClassCastException: Cannot cast io.opentelemetry.exporter.internal.otlp.ResourceMarshaler to io.opentelemetry.exporter.internal.otlp.InstrumentationScopeMarshaler

How can I troubleshoot this issue?

[huange7]

May I ask which issue this is related to?

[laurit]

open-telemetry/opentelemetry-java#7019

[huange7]

thanks!

[laurit]

setting otel.java.exporter.memory_mode to immutable_data should also help against that issue

[huange7]

got it

[huange7]

hi @laurit , i got stacktrace like this:

OpenTelemetry Javaagent failed to start
java.lang.IllegalStateException: Could not acquire the circularity lock upon installation.
	at net.bytebuddy.agent.builder.AgentBuilder$Default.doInstall(AgentBuilder.java:11701)
	at net.bytebuddy.agent.builder.AgentBuilder$Default.installOn(AgentBuilder.java:11637)
	at io.opentelemetry.javaagent.tooling.AgentInstaller.installBytebuddyAgent(AgentInstaller.java:202)
	at io.opentelemetry.javaagent.tooling.AgentInstaller.installBytebuddyAgent(AgentInstaller.java:110)
	at io.opentelemetry.javaagent.tooling.AgentStarterImpl.start(AgentStarterImpl.java:101)
	at io.opentelemetry.javaagent.bootstrap.AgentInitializer$1.run(AgentInitializer.java:54)
	at io.opentelemetry.javaagent.bootstrap.AgentInitializer$1.run(AgentInitializer.java:48)
	at io.opentelemetry.javaagent.bootstrap.AgentInitializer.execute(AgentInitializer.java:70)
	at io.opentelemetry.javaagent.bootstrap.AgentInitializer.initialize(AgentInitializer.java:47)
	at io.opentelemetry.javaagent.OpenTelemetryAgent.startAgent(OpenTelemetryAgent.java:58)
	at io.opentelemetry.javaagent.OpenTelemetryAgent.premain(OpenTelemetryAgent.java:45)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at sun.instrument.InstrumentationImpl.loadClassAndStartAgent(InstrumentationImpl.java:386)
	at sun.instrument.InstrumentationImpl.loadClassAndCallPremain(InstrumentationImpl.java:401)

maybe this is the reason that cause the ThreadLocalRandom cannot not initialize, but why this would happen?

[huange7]

here is my jvm args:

-Dserver.port=8080, 
-Xmx4096m, 
-Xms4096m, 
-XX:+HeapDumpOnOutOfMemoryError, 
-XX:HeapDumpPath=/app/logs/, 
-Dspring.profiles.active=prod, 
-Dcom.sun.management.jmxremote, 
-Dcom.sun.management.jmxremote.port=8786, 
-Dcom.sun.management.jmxremote.ssl=false, 
-Dcom.sun.management.jmxremote.authenticate=false, 
-Djute.maxbuffer=104857600, 
-javaagent:opentelemetry-javaagent.jar 
-verbose:gc, 
-XX:+PrintGCDetails, 
-XX:+PrintGCDateStamps, 
-Xloggc:log/gc.log

[huange7]

I have found a similar issue(apache/skywalking#12572). Is it possible that the OpenTelemetry agent might encounter the same problem?

[laurit]

maybe this is the reason that cause the ThreadLocalRandom cannot not initialize, but why this would happen?

Or it could be the other way around. This failure is triggered by failing to initialize ThreadLocalRandom at an earlier time. If you seek to understand why this fails then usually in such cases the key is the ability to reproduce the issue. To reproduce such issues you could runs some sort of script that starts the application and checks whether there was a failure in a loop. Since here starting the agent fails I guess you don't even need to have and application and could run something like java -javaagent:... -version If you can reproduce it you can establish how often it happens and can start adding debug statements or determine whether it is your modifications to the agent that trigger it.

I have found a similar issue(apache/skywalking#12572). Is it possible that the OpenTelemetry agent might encounter the same problem?

It is possible. Without reproducing it is hard to tell. Since you are not using the latest version of the agent it is also possible that this issue has already been worked around in the byte-buddy library. Could also consider adding a call to ThreadLocalRandom.current(); before byte-buddy transformer is installed.

[laurit]

Should be fixed with #14030

[huange7]

Should be fixed with #14030

thanks!