Skip to content

[Bug report] default_catalog is illegal when listing namespaces after adding a catalog with dynamic-config-provider #7395

New issue
New issue
Closed
#8327
Closed
[Bug report] default_catalog is illegal when listing namespaces after adding a catalog with dynamic-config-provider#7395
#8327
Assignees
FANNG1
Labels
1.0.0Release v1.0.0bugSomething isn't working
@helong001

Description

@helong001
helong001
opened on Jun 15, 2025

Version

main branch

Describe what's wrong

Problem Summary

When deploying Gravitino on Kubernetes with the auxiliary Iceberg REST service enabled and using the dynamic-config-provider, everything starts up correctly. A custom catalog (e.g., testCatalog) is successfully added.

However, when a client (such as Spark, Trino, or a REST client) attempts to list namespaces, the request unexpectedly uses default_catalog, which results in a failure. This occurs even if a valid catalog was already created and no such default_catalog was explicitly configured.

Error message and/or stacktrace

Error from web ui

Image

Error log from gravitino pod with auxService iceberg-rest

2025-06-15 08:08:30 INFO  [Gravitino-webserver-36] CatalogOperations:227 - Catalog loaded: test.testCatalog
2025-06-15 08:08:35 INFO  [iceberg-rest-25] IcebergNamespaceOperations:89 - List Iceberg namespaces, catalog: default_catalog, parentNamespace: 
2025-06-15 08:08:35 INFO  [iceberg-rest-25] IcebergExceptionMapper:80 - Iceberg REST server error maybe caused by user request, response http status: 400, exception: class java.lang.IllegalArgumentException, exception message: default_catalog is illegal in gravitino-based-provider
2025-06-15 08:08:35 WARN  [Gravitino-webserver-42] ExceptionHandlers:267 - Failed to operate schema(s) operation [LIST] under catalog [testCatalog], reason [default_catalog is illegal in gravitino-based-provider]
java.lang.IllegalArgumentException: default_catalog is illegal in gravitino-based-provider
        at org.apache.iceberg.rest.ErrorHandlers$DefaultErrorHandler.accept(ErrorHandlers.java:206) ~[?:?]
        at org.apache.iceberg.rest.ErrorHandlers$NamespaceErrorHandler.accept(ErrorHandlers.java:180) ~[?:?]
        at org.apache.iceberg.rest.ErrorHandlers$NamespaceErrorHandler.accept(ErrorHandlers.java:166) ~[?:?]
        at org.apache.iceberg.rest.HTTPClient.throwFailure(HTTPClient.java:211) ~[?:?]
        at org.apache.iceberg.rest.HTTPClient.execute(HTTPClient.java:323) ~[?:?]
        at org.apache.iceberg.rest.HTTPClient.execute(HTTPClient.java:262) ~[?:?]
        at org.apache.iceberg.rest.HTTPClient.get(HTTPClient.java:358) ~[?:?]
        at org.apache.iceberg.rest.RESTClient.get(RESTClient.java:96) ~[?:?]
        at org.apache.iceberg.rest.RESTSessionCatalog.listNamespaces(RESTSessionCatalog.java:562) ~[?:?]
        at org.apache.iceberg.catalog.BaseSessionCatalog$AsCatalog.listNamespaces(BaseSessionCatalog.java:133) ~[?:?]
        at org.apache.iceberg.rest.RESTCatalog.listNamespaces(RESTCatalog.java:228) ~[?:?]
        at org.apache.iceberg.catalog.SupportsNamespaces.listNamespaces(SupportsNamespaces.java:74) ~[?:?]
        at org.apache.iceberg.rest.CatalogHandlers.listNamespaces(CatalogHandlers.java:115) ~[?:?]
        at org.apache.gravitino.iceberg.common.ops.IcebergCatalogWrapper.listNamespace(IcebergCatalogWrapper.java:137) ~[?:?]
        at org.apache.gravitino.catalog.lakehouse.iceberg.IcebergCatalogOperations.listSchemas(IcebergCatalogOperations.java:144) ~[?:?]
        at org.apache.gravitino.catalog.SchemaOperationDispatcher.lambda$listSchemas$0(SchemaOperationDispatcher.java:82) ~[gravitino-core-1.0.0-SNAPSHOT.jar:?]
        at org.apache.gravitino.catalog.CatalogManager$CatalogWrapper.lambda$doWithSchemaOps$0(CatalogManager.java:150) ~[gravitino-core-1.0.0-SNAPSHOT.jar:?]
        at org.apache.gravitino.utils.IsolatedClassLoader.withClassLoader(IsolatedClassLoader.java:86) ~[gravitino-core-1.0.0-SNAPSHOT.jar:?]
        at org.apache.gravitino.catalog.CatalogManager$CatalogWrapper.doWithSchemaOps(CatalogManager.java:145) ~[gravitino-core-1.0.0-SNAPSHOT.jar:?]
        at org.apache.gravitino.catalog.SchemaOperationDispatcher.lambda$listSchemas$1(SchemaOperationDispatcher.java:82) ~[gravitino-core-1.0.0-SNAPSHOT.jar:?]
        at org.apache.gravitino.catalog.OperationDispatcher.doWithCatalog(OperationDispatcher.java:99) ~[gravitino-core-1.0.0-SNAPSHOT.jar:?]
        at org.apache.gravitino.catalog.SchemaOperationDispatcher.lambda$listSchemas$2(SchemaOperationDispatcher.java:80) ~[gravitino-core-1.0.0-SNAPSHOT.jar:?]
        at org.apache.gravitino.lock.TreeLockUtils.doWithTreeLock(TreeLockUtils.java:49) ~[gravitino-core-1.0.0-SNAPSHOT.jar:?]
        at org.apache.gravitino.catalog.SchemaOperationDispatcher.listSchemas(SchemaOperationDispatcher.java:76) ~[gravitino-core-1.0.0-SNAPSHOT.jar:?]
        at org.apache.gravitino.hook.SchemaHookDispatcher.listSchemas(SchemaHookDispatcher.java:54) ~[gravitino-core-1.0.0-SNAPSHOT.jar:?]
        at org.apache.gravitino.catalog.SchemaNormalizeDispatcher.listSchemas(SchemaNormalizeDispatcher.java:48) ~[gravitino-core-1.0.0-SNAPSHOT.jar:?]
        at org.apache.gravitino.listener.SchemaEventDispatcher.listSchemas(SchemaEventDispatcher.java:77) ~[gravitino-core-1.0.0-SNAPSHOT.jar:?]
        at org.apache.gravitino.server.web.rest.SchemaOperations.lambda$listSchemas$0(SchemaOperations.java:85) ~[gravitino-server-1.0.0-SNAPSHOT.jar:?]
        at java.base/java.security.AccessController.doPrivileged(AccessController.java:712) ~[?:?]
        at java.base/javax.security.auth.Subject.doAs(Subject.java:439) ~[?:?]
        at org.apache.gravitino.utils.PrincipalUtils.doAs(PrincipalUtils.java:39) ~[gravitino-core-1.0.0-SNAPSHOT.jar:?]
        at org.apache.gravitino.server.web.Utils.doAs(Utils.java:198) ~[gravitino-server-common-1.0.0-SNAPSHOT.jar:?]
        at org.apache.gravitino.server.web.rest.SchemaOperations.listSchemas(SchemaOperations.java:81) ~[gravitino-server-1.0.0-SNAPSHOT.jar:?]
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:?]
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77) ~[?:?]
        at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:?]
        at java.base/java.lang.reflect.Method.invoke(Method.java:568) ~[?:?]
        at org.glassfish.jersey.server.model.internal.ResourceMethodInvocationHandlerFactory.lambda$static$0(ResourceMethodInvocationHandlerFactory.java:52) ~[jersey-server-2.41.jar:?]
        at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher$1.run(AbstractJavaResourceMethodDispatcher.java:146) ~[jersey-server-2.41.jar:?]
        at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.invoke(AbstractJavaResourceMethodDispatcher.java:189) ~[jersey-server-2.41.jar:?]
        at org.glassfish.jersey.server.model.internal.JavaResourceMethodDispatcherProvider$ResponseOutInvoker.doDispatch(JavaResourceMethodDispatcherProvider.java:176) ~[jersey-server-2.41.jar:?]
        at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.dispatch(AbstractJavaResourceMethodDispatcher.java:93) ~[jersey-server-2.41.jar:?]
        at org.glassfish.jersey.server.model.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:478) ~[jersey-server-2.41.jar:?]
        at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:400) ~[jersey-server-2.41.jar:?]
        at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:81) ~[jersey-server-2.41.jar:?]
        at org.glassfish.jersey.server.ServerRuntime$1.run(ServerRuntime.java:256) ~[jersey-server-2.41.jar:?]
        at org.glassfish.jersey.internal.Errors$1.call(Errors.java:248) ~[jersey-common-2.41.jar:?]
        at org.glassfish.jersey.internal.Errors$1.call(Errors.java:244) ~[jersey-common-2.41.jar:?]
        at org.glassfish.jersey.internal.Errors.process(Errors.java:292) ~[jersey-common-2.41.jar:?]
        at org.glassfish.jersey.internal.Errors.process(Errors.java:274) ~[jersey-common-2.41.jar:?]
        at org.glassfish.jersey.internal.Errors.process(Errors.java:244) ~[jersey-common-2.41.jar:?]
        at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:265) ~[jersey-common-2.41.jar:?]
        at org.glassfish.jersey.server.ServerRuntime.process(ServerRuntime.java:235) ~[jersey-server-2.41.jar:?]
        at org.glassfish.jersey.server.ApplicationHandler.handle(ApplicationHandler.java:684) ~[jersey-server-2.41.jar:?]
        at org.glassfish.jersey.servlet.WebComponent.serviceImpl(WebComponent.java:394) ~[jersey-container-servlet-core-2.41.jar:?]
        at org.glassfish.jersey.servlet.WebComponent.service(WebComponent.java:346) ~[jersey-container-servlet-core-2.41.jar:?]
        at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:358) ~[jersey-container-servlet-core-2.41.jar:?]
        at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:311) ~[jersey-container-servlet-core-2.41.jar:?]
        at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:205) ~[jersey-container-servlet-core-2.41.jar:?]
        at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:799) ~[jetty-servlet-9.4.51.v20230217.jar:9.4.51.v20230217]
        at org.eclipse.jetty.servlet.ServletHandler$ChainEnd.doFilter(ServletHandler.java:1656) ~[jetty-servlet-9.4.51.v20230217.jar:9.4.51.v20230217]
        at org.apache.gravitino.server.authentication.AuthenticationFilter.doFilter(AuthenticationFilter.java:86) ~[gravitino-server-common-1.0.0-SNAPSHOT.jar:?]
        at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193) ~[jetty-servlet-9.4.51.v20230217.jar:9.4.51.v20230217]
        at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1626) ~[jetty-servlet-9.4.51.v20230217.jar:9.4.51.v20230217]
        at org.apache.gravitino.server.web.VersioningFilter.doFilter(VersioningFilter.java:111) ~[gravitino-server-1.0.0-SNAPSHOT.jar:?]
        at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193) ~[jetty-servlet-9.4.51.v20230217.jar:9.4.51.v20230217]
        at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1626) ~[jetty-servlet-9.4.51.v20230217.jar:9.4.51.v20230217]
        at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:552) ~[jetty-servlet-9.4.51.v20230217.jar:9.4.51.v20230217]
        at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143) ~[jetty-server-9.4.51.v20230217.jar:9.4.51.v20230217]
        at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:600) ~[jetty-security-9.4.51.v20230217.jar:9.4.51.v20230217]
        at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127) ~[jetty-server-9.4.51.v20230217.jar:9.4.51.v20230217]
        at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:235) ~[jetty-server-9.4.51.v20230217.jar:9.4.51.v20230217]
        at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1624) ~[jetty-server-9.4.51.v20230217.jar:9.4.51.v20230217]
        at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233) ~[jetty-server-9.4.51.v20230217.jar:9.4.51.v20230217]
        at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1440) ~[jetty-server-9.4.51.v20230217.jar:9.4.51.v20230217]
        at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:188) ~[jetty-server-9.4.51.v20230217.jar:9.4.51.v20230217]
        at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:505) ~[jetty-servlet-9.4.51.v20230217.jar:9.4.51.v20230217]
        at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1594) ~[jetty-server-9.4.51.v20230217.jar:9.4.51.v20230217]
        at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:186) ~[jetty-server-9.4.51.v20230217.jar:9.4.51.v20230217]
        at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1355) ~[jetty-server-9.4.51.v20230217.jar:9.4.51.v20230217]
        at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141) ~[jetty-server-9.4.51.v20230217.jar:9.4.51.v20230217]
        at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:146) ~[jetty-server-9.4.51.v20230217.jar:9.4.51.v20230217]
        at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127) ~[jetty-server-9.4.51.v20230217.jar:9.4.51.v20230217]
        at org.eclipse.jetty.server.Server.handle(Server.java:516) ~[jetty-server-9.4.51.v20230217.jar:9.4.51.v20230217]
        at org.eclipse.jetty.server.HttpChannel.lambda$handle$1(HttpChannel.java:487) ~[jetty-server-9.4.51.v20230217.jar:9.4.51.v20230217]
        at org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:732) [jetty-server-9.4.51.v20230217.jar:9.4.51.v20230217]
        at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:479) [jetty-server-9.4.51.v20230217.jar:9.4.51.v20230217]
        at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:277) [jetty-server-9.4.51.v20230217.jar:9.4.51.v20230217]
        at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:311) [jetty-io-9.4.51.v20230217.jar:9.4.51.v20230217]
        at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:105) [jetty-io-9.4.51.v20230217.jar:9.4.51.v20230217]
        at org.eclipse.jetty.io.ChannelEndPoint$1.run(ChannelEndPoint.java:104) [jetty-io-9.4.51.v20230217.jar:9.4.51.v20230217]
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:883) [jetty-util-9.4.51.v20230217.jar:9.4.51.v20230217]
        at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1034) [jetty-util-9.4.51.v20230217.jar:9.4.51.v20230217]
        at java.base/java.lang.Thread.run(Thread.java:833) [?:?]
(base) ➜  dev-cluster-deploy 
kubectl get svc -n lakehouse-dev

NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
gravitino ClusterIP 10.43.21.104 8090/TCP,9001/TCP 16m
gravitino-mysql ClusterIP 10.43.179.149 3306/TCP 16m
gravitino-mysql-headless ClusterIP None 3306/TCP 16m

How to reproduce

Deployment Context

mysql:
  ## @param mysql.enabled Deploy MySQL container(s)
  ##
  enabled: true

envFrom: []

log4j2Properties:
  rootLoggerLevel: INFO

auxService:
  ## Auxiliary service names, separate by ','
  ##
  names: iceberg-rest

icebergRest:
  ## Iceberg REST service classpath
  ##
  classpath: "iceberg-rest-server/libs, iceberg-rest-server/conf"
  ## Iceberg REST service host
  ##
  host: 0.0.0.0
  ## Iceberg REST service http port
  ##
  httpPort: 9001

# Rest backend configs.
additionalConfigItems:
  ## THE CONFIGURATION EXAMPLE FOR JDBC CATALOG BACKEND WITH S3 SUPPORT
  gravitino.iceberg-rest.io-impl: org.apache.iceberg.aws.s3.S3FileIO
  gravitino.iceberg-rest.s3-endpoint: http://minio.lakehouse-dev.svc.cluster.local:9000
  gravitino.iceberg-rest.s3-region: minio
  gravitino.iceberg-rest.s3-access-key-id: xxx
  gravitino.iceberg-rest.s3-secret-access-key: xxx

  gravitino.iceberg-rest.catalog-backend: jdbc
  gravitino.iceberg-rest.uri: jdbc:mysql://gravitino-mysql.lakehouse-dev.svc.cluster.local:3306/gravitino
  gravitino.iceberg-rest.warehouse: s3://lakehouse-dev-iceberg
  gravitino.iceberg-rest.jdbc-user: gravitino
  gravitino.iceberg-rest.jdbc-password: gravitino
  gravitino.iceberg-rest.jdbc-initialize: "true"
  gravitino.iceberg-rest.jdbc-driver: com.mysql.cj.jdbc.Driver
  

  gravitino.iceberg-rest.catalog-cache-eviction-interval-ms: "100000"
  gravitino.iceberg-rest.catalog-config-provider: dynamic-config-provider
  gravitino.iceberg-rest.gravitino-uri: http://gravitino.lakehouse-dev.svc.cluster.local:8090
  gravitino.iceberg-rest.gravitino-metalake: test

Catalog Configuration

Image

Additional context

No response

Metadata

Metadata

Assignees

Labels

1.0.0Release v1.0.0bugSomething isn't working

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions