Service shutdown when instrumenting with javaagent

[findingrish]

Tried instrumenting one of the hypertrace java service (https://github.com/hypertrace/hypertrace-service/tree/main/hypertrace-data-config-service) with java agent.
The service was deployed in k8s cluster locally using helm.

Dockerfile used for building the image (locally built the service image to embed agent):

FROM hypertrace/javaagent:latest
ARG COMMIT_SHA=unknown
LABEL commit_sha=${COMMIT_SHA}
ENV COMMIT_SHA=${COMMIT_SHA}
LABEL maintainer="Hypertrace 'https://www.hypertrace.org/'"
#RUN wget https://github.com/hypertrace/javaagent/releases/latest/download/hypertrace-agent-all.jar -O /javaagent.jar ;
WORKDIR /app

COPY scripts/hypertrace-data-config-service run
COPY libs libs
COPY resources resources
COPY classes classes
ENV SERVICE_NAME=hypertrace-data-config-service CLUSTER_NAME=default-cluster POD_NAME=default-pod
ENTRYPOINT ["./run"]

The docker image can be found here: https://hub.docker.com/repository/docker/rishabh691/hypertrace-data-config-service (rishabh691/hypertrace-data-config-service:test1)

Following env vars were passed in k8s deployment.

        - name: JAVA_TOOL_OPTIONS
          value: -XX:InitialRAMPercentage=50.0 -XX:MaxRAMPercentage=75.0 -javaagent:/opt/hypertrace/hypertrace-agent-all.jar
        - name: HT_SERVICE_NAME
          value: data-config-service
        - name: HT_REPORTING_ENDPOINT
          value: http://hypertrace-oc-collector:9411/api/v2/spans

The service crashes with following error:

Picked up JAVA_TOOL_OPTIONS: -XX:InitialRAMPercentage=50.0 -XX:MaxRAMPercentage=75.0 -javaagent:/opt/hypertrace/hypertrace-agent-all.jar
[main] INFO org.hypertrace.agent.core.config.HypertraceConfig - Config loaded: {"serviceName":"data-config-service","reporting":{"endpoint":"http://hypertrace-oc-collector:9411/api/v2/spans","opa":{"endpoint":"http://opa.traceableai:8181/","pollPeriodSeconds":30}},"dataCapture":{"httpHeaders":{"request":true,"response":true},"httpBody":{"request":true,"response":true},"rpcMetadata":{"request":true,"response":true},"rpcBody":{"request":true,"response":true},"bodyMaxSizeBytes":131072},"propagationFormats":["TRACECONTEXT"]}
[opentelemetry.auto.trace 2021-02-07 12:59:11:966 +0000] [main] INFO io.opentelemetry.javaagent.tooling.VersionLogger - opentelemetry-javaagent - version: 0.9.7
[opentelemetry.auto.trace 2021-02-07 12:59:12:169 +0000] [main] INFO org.hypertrace.agent.core.config.HypertraceConfig - Config loaded: {"serviceName":"data-config-service","reporting":{"endpoint":"http://hypertrace-oc-collector:9411/api/v2/spans","opa":{"endpoint":"http://opa.traceableai:8181/","pollPeriodSeconds":30}},"dataCapture":{"httpHeaders":{"request":true,"response":true},"httpBody":{"request":true,"response":true},"rpcMetadata":{"request":true,"response":true},"rpcBody":{"request":true,"response":true},"bodyMaxSizeBytes":131072},"propagationFormats":["TRACECONTEXT"]}
[opentelemetry.auto.trace 2021-02-07 12:59:12:583 +0000] [main] INFO io.opentelemetry.javaagent.tooling.PropagatorsInitializer - Added tracecontext propagator
[opentelemetry.auto.trace 2021-02-07 12:59:14:280 +0000] [main] INFO io.opentelemetry.javaagent.tooling.OpenTelemetryInstaller - Installed span exporter: io.opentelemetry.exporter.zipkin.ZipkinSpanExporter
[opentelemetry.auto.trace 2021-02-07 12:59:17:082 +0000] [main] WARN io.opentelemetry.javaagent.tooling.InstrumentationModule - Helper classes and resources won't be injected if no types are instrumented: methods
Hypertrace agent started, version: 0.9.7
2021-02-07 12:59:30.247 [main] INFO  o.h.c.s.PlatformServiceLauncher - Trying to start PlatformService ...
2021-02-07 12:59:30.355 [main] WARN  o.h.c.s.c.ConfigUtils - Cannot find Property: bootstrap.config.uri in neither JVM parameter nor OS environment
2021-02-07 12:59:30.440 [main] WARN  o.h.c.s.c.ConfigUtils - Cannot find Property: container.name in neither JVM parameter nor OS environment
2021-02-07 12:59:30.440 [main] INFO  o.h.c.s.c.DirectoryBasedConfigClient - Trying to compile configs under directory: /app/resources/configs
2021-02-07 12:59:30.441 [main] INFO  o.h.c.s.c.DirectoryBasedConfigClient - Loading config from path: /app/resources/configs/hypertrace-data-config-service/application.conf
[opentelemetry.auto.trace 2021-02-07 12:59:30:833 +0000] [opentelemetry-exec-shutdown-hook] WARN io.opentelemetry.javaagent.tooling.CommonTaskExecutor - Periodic task scheduler is shutdown. Will not run: cleaner for {}
2021-02-07 12:59:32.357 [main] INFO  o.h.c.s.c.DirectoryBasedConfigClient - Overrided Configs are listed below:
2021-02-07 12:59:32.431 [main] INFO  o.h.c.s.c.ConfigUtils - service.name = Unquoted("hypertrace-data-config-service")
2021-02-07 12:59:32.446 [main] INFO  o.h.c.s.c.ConfigUtils - service.admin.port = ConfigInt(9013)
2021-02-07 12:59:32.533 [main] INFO  o.h.c.s.c.ConfigUtils - main.class = Unquoted("org.hypertrace.service.HypertraceDataConfigService")
2021-02-07 12:59:32.534 [main] INFO  o.h.c.s.c.ConfigUtils - service.port = ConfigInt(9012)
2021-02-07 12:59:33.150 [main] WARN  o.h.c.s.c.ConfigUtils - Cannot find Property: container.name in neither JVM parameter nor OS environment
2021-02-07 12:59:33.150 [main] INFO  o.h.c.s.c.DirectoryBasedConfigClient - Trying to compile configs under directory: /app/resources/configs
2021-02-07 12:59:33.150 [main] INFO  o.h.c.s.c.DirectoryBasedConfigClient - Loading config from path: /app/resources/configs/hypertrace-data-config-service/application.conf
2021-02-07 12:59:33.156 [main] INFO  o.h.c.s.c.DirectoryBasedConfigClient - Overrided Configs are listed below:
2021-02-07 12:59:33.231 [main] INFO  o.h.c.s.c.ConfigUtils - service.name = Unquoted("hypertrace-data-config-service")
2021-02-07 12:59:33.231 [main] INFO  o.h.c.s.c.ConfigUtils - service.admin.port = ConfigInt(9013)
2021-02-07 12:59:33.231 [main] INFO  o.h.c.s.c.ConfigUtils - main.class = Unquoted("org.hypertrace.service.HypertraceDataConfigService")
2021-02-07 12:59:33.233 [main] INFO  o.h.c.s.c.ConfigUtils - service.port = ConfigInt(9012)
2021-02-07 12:59:33.234 [main] INFO  o.h.c.s.PlatformService - Starting the service by using this metrics configuration Config(SimpleConfigObject({}))
2021-02-07 12:59:34.143 [main] INFO  o.h.c.s.m.PlatformMetricsRegistry - Trying to init PrometheusReporter
2021-02-07 12:59:34.547 [main] INFO  o.h.c.s.m.PlatformMetricsRegistry - Setting default tags for all metrics to: {app=hypertrace-data-config-service}
[opentelemetry.auto.trace 2021-02-06 19:35:08:046 +0000] [Thread-0] WARN io.opentelemetry.javaagent.tooling.CommonTaskExecutor - Periodic task scheduler is shutdown. Will not run: cleaner for {}
2021-02-06 19:35:08.133 [Thread-0] ERROR o.h.c.s.PlatformServiceLauncher - Error while calling quitquitquit
org.apache.http.conn.HttpHostConnectException: Connect to 127.0.0.1:15020 [/127.0.0.1] failed: Connection refused (Connection refused)
	at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:156) ~[httpclient-4.5.13.jar:4.5.13]
	at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:376) ~[httpclient-4.5.13.jar:4.5.13]
	at org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:393) ~[httpclient-4.5.13.jar:4.5.13]
	at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:236) ~[httpclient-4.5.13.jar:4.5.13]
	at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:186) ~[httpclient-4.5.13.jar:4.5.13]
	at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) ~[httpclient-4.5.13.jar:4.5.13]

Am I misconfiguring the agent or missing something?

[pavolloffay]

I am not able to reproduce this locally

docker run --rm -it -e JAVA_TOOL_OPTIONS="-XX:InitialRAMPercentage=50.0 -XX:MaxRAMPercentage=75.0   -javaagent:/opt/hypertrace/hypertrace-agent-all.jar" -e HT_SERVICE_NAME=ht-data-service -e HT_REPORTING_ENDPOINT="http://host.docker.internal:9411/api/v2/spans" rishabh691/hypertrace-data-config-service:test1
Picked up JAVA_TOOL_OPTIONS: -XX:InitialRAMPercentage=50.0 -XX:MaxRAMPercentage=75.0   -javaagent:/opt/hypertrace/hypertrace-agent-all.jar
[main] INFO org.hypertrace.agent.core.config.HypertraceConfig - Config loaded: {"serviceName":"ht-data-service","reporting":{"endpoint":"http://host.docker.internal:9411/api/v2/spans","opa":{"endpoint":"http://opa.traceableai:8181/","pollPeriodSeconds":30}},"dataCapture":{"httpHeaders":{"request":true,"response":true},"httpBody":{"request":true,"response":true},"rpcMetadata":{"request":true,"response":true},"rpcBody":{"request":true,"response":true},"bodyMaxSizeBytes":131072},"propagationFormats":["TRACECONTEXT"]}
[opentelemetry.auto.trace 2021-02-08 17:01:37:394 +0000] [main] INFO io.opentelemetry.javaagent.tooling.VersionLogger - opentelemetry-javaagent - version: 0.9.7
[opentelemetry.auto.trace 2021-02-08 17:01:37:436 +0000] [main] INFO org.hypertrace.agent.core.config.HypertraceConfig - Config loaded: {"serviceName":"ht-data-service","reporting":{"endpoint":"http://host.docker.internal:9411/api/v2/spans","opa":{"endpoint":"http://opa.traceableai:8181/","pollPeriodSeconds":30}},"dataCapture":{"httpHeaders":{"request":true,"response":true},"httpBody":{"request":true,"response":true},"rpcMetadata":{"request":true,"response":true},"rpcBody":{"request":true,"response":true},"bodyMaxSizeBytes":131072},"propagationFormats":["TRACECONTEXT"]}
[opentelemetry.auto.trace 2021-02-08 17:01:37:484 +0000] [main] INFO io.opentelemetry.javaagent.tooling.PropagatorsInitializer - Added tracecontext propagator
[opentelemetry.auto.trace 2021-02-08 17:01:37:688 +0000] [main] INFO io.opentelemetry.javaagent.tooling.OpenTelemetryInstaller - Installed span exporter: io.opentelemetry.exporter.zipkin.ZipkinSpanExporter
[opentelemetry.auto.trace 2021-02-08 17:01:38:123 +0000] [main] WARN io.opentelemetry.javaagent.tooling.InstrumentationModule - Helper classes and resources won't be injected if no types are instrumented: methods
Hypertrace agent started, version: 0.9.7
2021-02-08 17:01:40.111 [main] INFO  o.h.c.s.PlatformServiceLauncher - Trying to start PlatformService ...
2021-02-08 17:01:40.128 [main] WARN  o.h.c.s.c.ConfigUtils - Cannot find Property: bootstrap.config.uri in neither JVM parameter nor OS environment
2021-02-08 17:01:40.132 [main] WARN  o.h.c.s.c.ConfigUtils - Cannot find Property: container.name in neither JVM parameter nor OS environment
2021-02-08 17:01:40.132 [main] INFO  o.h.c.s.c.DirectoryBasedConfigClient - Trying to compile configs under directory: /app/resources/configs
2021-02-08 17:01:40.133 [main] INFO  o.h.c.s.c.DirectoryBasedConfigClient - Loading config from path: /app/resources/configs/hypertrace-data-config-service/application.conf
2021-02-08 17:01:40.225 [main] INFO  o.h.c.s.c.DirectoryBasedConfigClient - Overrided Configs are listed below:
2021-02-08 17:01:40.226 [main] INFO  o.h.c.s.c.ConfigUtils - service.name = Unquoted("hypertrace-data-config-service")
2021-02-08 17:01:40.227 [main] INFO  o.h.c.s.c.ConfigUtils - service.admin.port = ConfigInt(9013)
2021-02-08 17:01:40.227 [main] INFO  o.h.c.s.c.ConfigUtils - main.class = Unquoted("org.hypertrace.service.HypertraceDataConfigService")
2021-02-08 17:01:40.228 [main] INFO  o.h.c.s.c.ConfigUtils - service.port = ConfigInt(9012)
2021-02-08 17:01:40.276 [main] WARN  o.h.c.s.c.ConfigUtils - Cannot find Property: container.name in neither JVM parameter nor OS environment
2021-02-08 17:01:40.276 [main] INFO  o.h.c.s.c.DirectoryBasedConfigClient - Trying to compile configs under directory: /app/resources/configs
2021-02-08 17:01:40.277 [main] INFO  o.h.c.s.c.DirectoryBasedConfigClient - Loading config from path: /app/resources/configs/hypertrace-data-config-service/application.conf
2021-02-08 17:01:40.278 [main] INFO  o.h.c.s.c.DirectoryBasedConfigClient - Overrided Configs are listed below:
2021-02-08 17:01:40.278 [main] INFO  o.h.c.s.c.ConfigUtils - service.name = Unquoted("hypertrace-data-config-service")
2021-02-08 17:01:40.279 [main] INFO  o.h.c.s.c.ConfigUtils - service.admin.port = ConfigInt(9013)
2021-02-08 17:01:40.279 [main] INFO  o.h.c.s.c.ConfigUtils - main.class = Unquoted("org.hypertrace.service.HypertraceDataConfigService")
2021-02-08 17:01:40.279 [main] INFO  o.h.c.s.c.ConfigUtils - service.port = ConfigInt(9012)
2021-02-08 17:01:40.279 [main] INFO  o.h.c.s.PlatformService - Starting the service by using this metrics configuration Config(SimpleConfigObject({}))
2021-02-08 17:01:40.313 [main] INFO  o.h.c.s.m.PlatformMetricsRegistry - Trying to init PrometheusReporter
2021-02-08 17:01:40.334 [main] INFO  o.h.c.s.m.PlatformMetricsRegistry - Setting default tags for all metrics to: {app=hypertrace-data-config-service}
2021-02-08 17:01:40.622 [main] WARN  o.h.c.s.c.ConfigUtils - Cannot find Property: container.name in neither JVM parameter nor OS environment
2021-02-08 17:01:40.623 [main] INFO  o.h.c.s.c.DirectoryBasedConfigClient - Trying to compile configs under directory: /app/resources/configs
2021-02-08 17:01:40.623 [main] INFO  o.h.c.s.c.DirectoryBasedConfigClient - Loading config from path: /app/resources/configs/attribute-service/application.conf
2021-02-08 17:01:40.627 [main] INFO  o.h.c.s.c.DirectoryBasedConfigClient - Loading config from path: /app/resources/configs/attribute-service/default-cluster/application.conf
2021-02-08 17:01:40.636 [main] INFO  o.h.c.s.c.DirectoryBasedConfigClient - Overrided Configs are listed below:
2021-02-08 17:01:40.637 [main] INFO  o.h.c.s.c.ConfigUtils - attributes.type.server.port = ConfigInt(9012)
2021-02-08 17:01:40.637 [main] INFO  o.h.c.s.c.ConfigUtils - metrics.reporter.names = [Quoted("prometheus")]
2021-02-08 17:01:40.638 [main] INFO  o.h.c.s.c.ConfigUtils - document.store.postgres.host = Unquoted("postgres")
2021-02-08 17:01:40.639 [main] INFO  o.h.c.s.c.ConfigUtils - metrics.reporter.console.reportInterval = ConfigInt(30)
2021-02-08 17:01:40.639 [main] INFO  o.h.c.s.c.ConfigUtils - document.store.mongo.port = ConfigInt(27017)
2021-02-08 17:01:40.639 [main] INFO  o.h.c.s.c.ConfigUtils - metrics.reporter.prefix = Unquoted("org.hypertrace.core.attribute.service.AttributeService")
2021-02-08 17:01:40.640 [main] INFO  o.h.c.s.c.ConfigUtils - document.store.mongo.host = Unquoted("localhost")
2021-02-08 17:01:40.640 [main] INFO  o.h.c.s.c.ConfigUtils - service.name = Unquoted("attribute-service")
2021-02-08 17:01:40.640 [main] INFO  o.h.c.s.c.ConfigUtils - document.store.postgres.port = ConfigInt(5432)
2021-02-08 17:01:40.640 [main] INFO  o.h.c.s.c.ConfigUtils - service.admin.port = ConfigInt(9013)
2021-02-08 17:01:40.641 [main] INFO  o.h.c.s.c.ConfigUtils - main.class = Unquoted("org.hypertrace.core.attribute.service.AttributeServiceEntry")
2021-02-08 17:01:40.641 [main] INFO  o.h.c.s.c.ConfigUtils - document.store.dataStoreType = Unquoted("mongo")
2021-02-08 17:01:40.784 [main] INFO  o.m.d.cluster - Cluster created with settings {hosts=[localhost:27017], mode=SINGLE, requiredClusterType=UNKNOWN, serverSelectionTimeout='30000 ms'}
2021-02-08 17:01:41.119 [cluster-ClusterId{value='60216e74da74db2c34660eb2', description='null'}-localhost:27017] INFO  o.m.d.cluster - Exception in monitor thread while connecting to server localhost:27017
com.mongodb.MongoSocketOpenException: Exception opening socket
	at com.mongodb.internal.connection.SocketStream.open(SocketStream.java:70) ~[mongodb-driver-core-4.1.1.jar:?]
	at com.mongodb.internal.connection.InternalStreamConnection.open(InternalStreamConnection.java:143) ~[mongodb-driver-core-4.1.1.jar:?]
	at com.mongodb.internal.connection.DefaultServerMonitor$ServerMonitorRunnable.lookupServerDescription(DefaultServerMonitor.java:188) [mongodb-driver-core-4.1.1.jar:?]
	at com.mongodb.internal.connection.DefaultServerMonitor$ServerMonitorRunnable.run(DefaultServerMonitor.java:144) [mongodb-driver-core-4.1.1.jar:?]
	at java.lang.Thread.run(Unknown Source) [?:?]
Caused by: java.net.ConnectException: Connection refused (Connection refused)
	at java.net.PlainSocketImpl.socketConnect(Native Method) ~[?:?]
	at java.net.AbstractPlainSocketImpl.doConnect(Unknown Source) ~[?:?]
	at java.net.AbstractPlainSocketImpl.connectToAddress(Unknown Source) ~[?:?]
	at java.net.AbstractPlainSocketImpl.connect(Unknown Source) ~[?:?]
	at java.net.SocksSocketImpl.connect(Unknown Source) ~[?:?]
	at java.net.Socket.connect(Unknown Source) ~[?:?]
	at com.mongodb.internal.connection.SocketStreamHelper.initialize(SocketStreamHelper.java:78) ~[mongodb-driver-core-4.1.1.jar:?]
	at com.mongodb.internal.connection.SocketStream.initializeSocket(SocketStream.java:79) ~[mongodb-driver-core-4.1.1.jar:?]
	at com.mongodb.internal.connection.SocketStream.open(SocketStream.java:65) ~[mongodb-driver-core-4.1.1.jar:?]
	... 4 more
2021-02-08 17:01:41.530 [main] WARN  o.h.c.s.c.ConfigUtils - Cannot find Property: container.name in neither JVM parameter nor OS environment
2021-02-08 17:01:41.530 [main] INFO  o.h.c.s.c.DirectoryBasedConfigClient - Trying to compile configs under directory: /app/resources/configs
2021-02-08 17:01:41.531 [main] INFO  o.h.c.s.c.DirectoryBasedConfigClient - Loading config from path: /app/resources/configs/entity-service/application.conf
2021-02-08 17:01:41.538 [main] INFO  o.h.c.s.c.DirectoryBasedConfigClient - Loading config from path: /app/resources/configs/entity-service/default-cluster/application.conf
2021-02-08 17:01:41.540 [main] INFO  o.h.c.s.c.DirectoryBasedConfigClient - Overrided Configs are listed below:
2021-02-08 17:01:41.541 [main] INFO  o.h.c.s.c.ConfigUtils - main.class = Unquoted("org.hypertrace.entity.service.EntityService")
2021-02-08 17:01:41.542 [main] INFO  o.h.c.s.c.ConfigUtils - metrics.reporter.names = [Quoted("prometheus")]
2021-02-08 17:01:41.542 [main] INFO  o.h.c.s.c.ConfigUtils - entity.service.config.entity-service.dataStoreType = Unquoted("mongo")
2021-02-08 17:01:41.542 [main] INFO  o.h.c.s.c.ConfigUtils - service.admin.port = ConfigInt(9013)
2021-02-08 17:01:41.542 [main] INFO  o.h.c.s.c.ConfigUtils - entity.service.config.entity-service.postgres.host = Unquoted("postgres")
2021-02-08 17:01:41.542 [main] INFO  o.h.c.s.c.ConfigUtils - metrics.reporter.prefix = Unquoted("org.hypertrace.entity.service.EntityService")
2021-02-08 17:01:41.543 [main] INFO  o.h.c.s.c.ConfigUtils - service.port = ConfigInt(9012)
2021-02-08 17:01:41.543 [main] INFO  o.h.c.s.c.ConfigUtils - entity.service.attributeMap = [{name=Quoted("API.id"), subDocPath=Quoted("entityId"), scope=Quoted("API")}, {subDocPath=Quoted("entityName"), name=Quoted("API.name"), scope=Quoted("API")}, {name=Quoted("API.apiType"), scope=Quoted("API"), subDocPath=Quoted("attributes.API_TYPE")}, {name=Quoted("API.httpMethod"), subDocPath=Quoted("attributes.http_method"), scope=Quoted("API")}, {name=Quoted("API.httpUrl"), subDocPath=Quoted("attributes.API_NAME"), scope=Quoted("API")}, {subDocPath=Quoted("attributes.SERVICE_ID"), name=Quoted("API.serviceId"), scope=Quoted("API")}, {name=Quoted("API.createdTime"), subDocPath=Quoted("createdTime"), scope=Quoted("API")}, {subDocPath=Quoted("attributes.api_discovery_state"), name=Quoted("API.apiDiscoveryState"), scope=Quoted("API")}, {subDocPath=Quoted("entityId"), name=Quoted("SERVICE.id"), scope=Quoted("SERVICE")}, {subDocPath=Quoted("entityName"), scope=Quoted("SERVICE"), name=Quoted("SERVICE.name")}]
2021-02-08 17:01:41.544 [main] INFO  o.h.c.s.c.ConfigUtils - entity.service.config.entity-service.mongo.port = ConfigInt(27017)
2021-02-08 17:01:41.544 [main] INFO  o.h.c.s.c.ConfigUtils - metrics.reporter.console.reportInterval = ConfigInt(30)
2021-02-08 17:01:41.544 [main] INFO  o.h.c.s.c.ConfigUtils - entity.service.config.entity-service.mongo.host = Unquoted("localhost")
2021-02-08 17:01:41.544 [main] INFO  o.h.c.s.c.ConfigUtils - entity.service.config.entity-service.postgres.port = ConfigInt(5432)
2021-02-08 17:01:41.544 [main] INFO  o.h.c.s.c.ConfigUtils - service.name = Unquoted("entity-service")
2021-02-08 17:01:41.546 [main] INFO  o.h.e.s.EntityServiceConfig - Config key postgres.host
2021-02-08 17:01:41.546 [main] INFO  o.h.e.s.EntityServiceConfig - Config key mongo.host
2021-02-08 17:01:41.547 [main] INFO  o.h.e.s.EntityServiceConfig - Config key postgres.port
2021-02-08 17:01:41.547 [main] INFO  o.h.e.s.EntityServiceConfig - Config key mongo.port
2021-02-08 17:01:41.547 [main] INFO  o.h.e.s.EntityServiceConfig - Config key dataStoreType
2021-02-08 17:01:41.547 [main] INFO  o.m.d.cluster - Cluster created with settings {hosts=[localhost:27017], mode=SINGLE, requiredClusterType=UNKNOWN, serverSelectionTimeout='30000 ms'}
2021-02-08 17:01:41.599 [cluster-ClusterId{value='60216e75da74db2c34660eb3', description='null'}-localhost:27017] INFO  o.m.d.cluster - Exception in monitor thread while connecting to server localhost:27017
com.mongodb.MongoSocketOpenException: Exception opening socket
	at com.mongodb.internal.connection.SocketStream.open(SocketStream.java:70) ~[mongodb-driver-core-4.1.1.jar:?]
	at com.mongodb.internal.connection.InternalStreamConnection.open(InternalStreamConnection.java:143) ~[mongodb-driver-core-4.1.1.jar:?]
	at com.mongodb.internal.connection.DefaultServerMonitor$ServerMonitorRunnable.lookupServerDescription(DefaultServerMonitor.java:188) [mongodb-driver-core-4.1.1.jar:?]
	at com.mongodb.internal.connection.DefaultServerMonitor$ServerMonitorRunnable.run(DefaultServerMonitor.java:144) [mongodb-driver-core-4.1.1.jar:?]
	at java.lang.Thread.run(Unknown Source) [?:?]
Caused by: java.net.ConnectException: Connection refused (Connection refused)
	at java.net.PlainSocketImpl.socketConnect(Native Method) ~[?:?]
	at java.net.AbstractPlainSocketImpl.doConnect(Unknown Source) ~[?:?]
	at java.net.AbstractPlainSocketImpl.connectToAddress(Unknown Source) ~[?:?]
	at java.net.AbstractPlainSocketImpl.connect(Unknown Source) ~[?:?]
	at java.net.SocksSocketImpl.connect(Unknown Source) ~[?:?]
	at java.net.Socket.connect(Unknown Source) ~[?:?]
	at com.mongodb.internal.connection.SocketStreamHelper.initialize(SocketStreamHelper.java:78) ~[mongodb-driver-core-4.1.1.jar:?]
	at com.mongodb.internal.connection.SocketStream.initializeSocket(SocketStream.java:79) ~[mongodb-driver-core-4.1.1.jar:?]
	at com.mongodb.internal.connection.SocketStream.open(SocketStream.java:65) ~[mongodb-driver-core-4.1.1.jar:?]
	... 4 more
2021-02-08 17:01:42.500 [main] WARN  o.h.c.s.c.ConfigUtils - Cannot find Property: container.name in neither JVM parameter nor OS environment
2021-02-08 17:01:42.500 [main] INFO  o.h.c.s.c.DirectoryBasedConfigClient - Trying to compile configs under directory: /app/resources/configs
2021-02-08 17:01:42.501 [main] INFO  o.h.c.s.c.DirectoryBasedConfigClient - Loading config from path: /app/resources/configs/config-service/application.conf
2021-02-08 17:01:42.504 [main] INFO  o.h.c.s.c.DirectoryBasedConfigClient - Overrided Configs are listed below:
2021-02-08 17:01:42.505 [main] INFO  o.h.c.s.c.ConfigUtils - generic.config.service.document.store.dataStoreType = Unquoted("mongo")
2021-02-08 17:01:42.505 [main] INFO  o.h.c.s.c.ConfigUtils - service.name = Unquoted("config-service")
2021-02-08 17:01:42.505 [main] INFO  o.h.c.s.c.ConfigUtils - service.admin.port = ConfigInt(50102)
2021-02-08 17:01:42.506 [main] INFO  o.h.c.s.c.ConfigUtils - main.class = Unquoted("org.hypertrace.config.service.ConfigService")
2021-02-08 17:01:42.506 [main] INFO  o.h.c.s.c.ConfigUtils - generic.config.service.document.store.mongo.port = ConfigInt(27017)
2021-02-08 17:01:42.506 [main] INFO  o.h.c.s.c.ConfigUtils - service.port = ConfigInt(50101)
2021-02-08 17:01:42.507 [main] INFO  o.h.c.s.c.ConfigUtils - generic.config.service.document.store.mongo.host = Unquoted("localhost")
2021-02-08 17:01:42.511 [main] INFO  o.m.d.cluster - Cluster created with settings {hosts=[localhost:27017], mode=SINGLE, requiredClusterType=UNKNOWN, serverSelectionTimeout='30000 ms'}
2021-02-08 17:01:42.514 [cluster-ClusterId{value='60216e76da74db2c34660eb4', description='null'}-localhost:27017] INFO  o.m.d.cluster - Exception in monitor thread while connecting to server localhost:27017
com.mongodb.MongoSocketOpenException: Exception opening socket
	at com.mongodb.internal.connection.SocketStream.open(SocketStream.java:70) ~[mongodb-driver-core-4.1.1.jar:?]
	at com.mongodb.internal.connection.InternalStreamConnection.open(InternalStreamConnection.java:143) ~[mongodb-driver-core-4.1.1.jar:?]
	at com.mongodb.internal.connection.DefaultServerMonitor$ServerMonitorRunnable.lookupServerDescription(DefaultServerMonitor.java:188) [mongodb-driver-core-4.1.1.jar:?]
	at com.mongodb.internal.connection.DefaultServerMonitor$ServerMonitorRunnable.run(DefaultServerMonitor.java:144) [mongodb-driver-core-4.1.1.jar:?]
	at java.lang.Thread.run(Unknown Source) [?:?]
Caused by: java.net.ConnectException: Connection refused (Connection refused)
	at java.net.PlainSocketImpl.socketConnect(Native Method) ~[?:?]
	at java.net.AbstractPlainSocketImpl.doConnect(Unknown Source) ~[?:?]
	at java.net.AbstractPlainSocketImpl.connectToAddress(Unknown Source) ~[?:?]
	at java.net.AbstractPlainSocketImpl.connect(Unknown Source) ~[?:?]
	at java.net.SocksSocketImpl.connect(Unknown Source) ~[?:?]
	at java.net.Socket.connect(Unknown Source) ~[?:?]
	at com.mongodb.internal.connection.SocketStreamHelper.initialize(SocketStreamHelper.java:78) ~[mongodb-driver-core-4.1.1.jar:?]
	at com.mongodb.internal.connection.SocketStream.initializeSocket(SocketStream.java:79) ~[mongodb-driver-core-4.1.1.jar:?]
	at com.mongodb.internal.connection.SocketStream.open(SocketStream.java:65) ~[mongodb-driver-core-4.1.1.jar:?]
	... 4 more
2021-02-08 17:01:42.616 [main] INFO  o.h.c.s.PlatformService - Service - hypertrace-data-config-service is initialized.
2021-02-08 17:01:42.616 [main] INFO  o.h.c.s.PlatformService - Trying to start service - hypertrace-data-config-service...
2021-02-08 17:01:42.670 [main] INFO  o.e.j.u.log - Logging initialized @5730ms to org.eclipse.jetty.util.log.Slf4jLog
2021-02-08 17:01:43.133 [Thread-1] INFO  o.h.s.HypertraceDataConfigService - Starting Hypertrace Data Config Service
2021-02-08 17:01:43.376 [main] INFO  o.e.j.s.Server - jetty-9.4.35.v20201120; built: 2020-11-20T21:17:03.964Z; git: bdc54f03a5e0a7e280fab27f55c3c75ee8da89fb; jvm 11.0.10+9-LTS
2021-02-08 17:01:43.493 [main] INFO  o.e.j.s.h.ContextHandler - Started o.e.j.s.ServletContextHandler@43f88150{/,null,AVAILABLE}
2021-02-08 17:01:43.586 [main] INFO  o.e.j.s.AbstractConnector - Started ServerConnector@25a2a71d{HTTP/1.1, (http/1.1)}{0.0.0.0:9013}
2021-02-08 17:01:43.587 [main] INFO  o.e.j.s.Server - Started @6648ms
2021-02-08 17:01:43.587 [main] INFO  o.h.c.s.PlatformService - Started admin service on port: 9013.
2021-02-08 17:01:43.587 [main] INFO  o.h.c.s.PlatformService - Service - hypertrace-data-config-service is started.

[findingrish]

attaching deployment manifest to bring up the container in a Pod.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: hypertrace-data-config-service-test
  labels:
    release: test-release
spec:
  replicas: 1
  selector:
    matchLabels:
      app: hypertrace-data-config-service-test
  template:
    metadata:
      labels:
        release: test-release
        app: hypertrace-data-config-service-test
    spec:
      containers:
        - name: hypertrace-data-config-service-test
          image: rishabh691/hypertrace-data-config-service:test1
          ports:
            - name: grpc-port
              containerPort: 9001
              protocol: TCP
            - name: admin-port
              containerPort: 9002
              protocol: TCP
          env:
            - name: HT_SERVICE_NAME
              value: hypertrace-data-config-service-test
            - name: HT_REPORTING_ENDPOINT
              value: http://hypertrace-oc-collector:9411/api/v2/spans
            - name: JAVA_TOOL_OPTIONS
              value: -javaagent:/opt/hypertrace/hypertrace-agent-all.jar
          livenessProbe:
            initialDelaySeconds: 10
            periodSeconds: 5
            tcpSocket:
              port: grpc-port
          readinessProbe:
            initialDelaySeconds: 2
            periodSeconds: 5
            httpGet:
              path: /health
              port: admin-port