Crawler 10.5.0 fails to deploy #415
-
Description - What happened? *Hi, we are running AP 4.1.4 and we are now trying to deploy the catalog-crawler. According to the release notes, this should be on version 10.5.0. I am not familiar with flyway so I am not sure how to fix this. Shouldn't those migrations happen automatically? Please let me know how to proceed with this. Expected Behavior *The crawler deploys and it is able to crawl the connectors registered on the specified environment. Observed Behavior *The crawler fails to deploy Steps to ReproduceNo response Context InformationNo response Relevant log outputBad level value for property: org.eclipse.edc.api.observability.ObservabilityApiController.level
2025-01-17 18:01:31 Configuration file does not exist: dataspaceconnector-configuration.properties. Ignoring.
2025-01-17 18:01:31 Initialized FS Configuration
2025-01-17 18:01:31 Initialized Boot Services
2025-01-17 18:01:31 Initialized JSON-LD Extension
2025-01-17 18:01:31 Initialized FS Vault
2025-01-17 18:01:31 Initialized Core Default Services
2025-01-17 18:01:31 HTTPS enforcement it not enabled, please enable it in a production environment
2025-01-17 18:01:31 HTTPS enforcement it not enabled, please enable it in a production environment
2025-01-17 18:01:31 Initialized Core Services
2025-01-17 18:01:31 Initialized Control Plane Default Services
2025-01-17 18:01:31 Initialized Jetty Service
2025-01-17 18:01:31 Initialized Jersey Web Service
2025-01-17 18:01:31 Protocol API will be available under port=11003, path=/api/dsp
2025-01-17 18:01:31 Initialized Dataspace Protocol API Configuration Extension
2025-01-17 18:01:31 Initialized Transfer Process Default Services
2025-01-17 18:01:31 Initialized org.eclipse.edc.connector.contract.ContractNegotiationDefaultServicesExtension
2025-01-17 18:01:31 Initialized Transfer Core
2025-01-17 18:01:31 Initialized org.eclipse.edc.connector.transfer.TransferProcessCommandExtension
2025-01-17 18:01:31 Initialized Contract Core
2025-01-17 18:01:31 Initialized Contract Negotiation command handlers
2025-01-17 18:01:31 Initialized Observability API
2025-01-17 18:01:31 Initialized Data Plane Selector Default Services
2025-01-17 18:01:31 Initialized Catalog Default Services
2025-01-17 18:01:31 Initialized Catalog Core
2025-01-17 18:01:31 No TransactionContext registered, a no-op implementation will be used, not suitable for production environments
2025-01-17 18:01:31 Initialized Control Plane Services
2025-01-17 18:01:31 Initialized Api Core Default Services
2025-01-17 18:01:31 No AuthenticationService registered, an all-pass implementation will be used, not suitable for production environments
2025-01-17 18:01:31 Management API will be available under port=11002, path=/api/management
2025-01-17 18:01:31 Initialized Management API configuration
2025-01-17 18:01:32 Flyway migration locations for 'flyway_schema_history': [classpath:db-crawler/migration]
2025-01-17 18:01:32 Error booting runtime: Validate failed: Migrations have failed validation Detected resolved migration not applied to database: 1. To fix this error, either run migrate, or set -ignoreMigrationPatterns='*:pending'. Detected resolved migration not applied to database: 2. To fix this error, either run migrate, or set -ignoreMigrationPatterns='*:pending'. Detected resolved migration not applied to database: 3. To fix this error, either run migrate, or set -ignoreMigrationPatterns='*:pending'. Detected resolved migration not applied to database: 4. To fix this error, either run migrate, or set -ignoreMigrationPatterns='*:pending'. Detected resolved migration not applied to database: 5. To fix this error, either run migrate, or set -ignoreMigrationPatterns='*:pending'. Detected resolved migration not applied to database: 6. To fix this error, either run migrate, or set -ignoreMigrationPatterns='*:pending'. Detected resolved migration not applied to database: 7. To fix this error, either run migrate, or set -ignoreMigrationPatterns='*:pending'. Detected resolved migration not applied to database: 8. To fix this error, either run migrate, or set -ignoreMigrationPatterns='*:pending'. Detected resolved migration not applied to database: 9. To fix this error, either run migrate, or set -ignoreMigrationPatterns='*:pending'. Detected resolved migration not applied to database: 10. To fix this error, either run migrate, or set -ignoreMigrationPatterns='*:pending'. Detected resolved migration not applied to database: 12. To fix this error, either run migrate, or set -ignoreMigrationPatterns='*:pending'. Need more flexibility with validation rules? Learn more: https://rd.gt/3AbJUZE
org.flywaydb.core.api.exception.FlywayValidateException: Validate failed: Migrations have failed validation
Detected resolved migration not applied to database: 1.
To fix this error, either run migrate, or set -ignoreMigrationPatterns='*:pending'.
Detected resolved migration not applied to database: 2.
To fix this error, either run migrate, or set -ignoreMigrationPatterns='*:pending'.
Detected resolved migration not applied to database: 3.
To fix this error, either run migrate, or set -ignoreMigrationPatterns='*:pending'.
Detected resolved migration not applied to database: 4.
To fix this error, either run migrate, or set -ignoreMigrationPatterns='*:pending'.
Detected resolved migration not applied to database: 5.
To fix this error, either run migrate, or set -ignoreMigrationPatterns='*:pending'.
Detected resolved migration not applied to database: 6.
To fix this error, either run migrate, or set -ignoreMigrationPatterns='*:pending'.
Detected resolved migration not applied to database: 7.
To fix this error, either run migrate, or set -ignoreMigrationPatterns='*:pending'.
Detected resolved migration not applied to database: 8.
To fix this error, either run migrate, or set -ignoreMigrationPatterns='*:pending'.
Detected resolved migration not applied to database: 9.
To fix this error, either run migrate, or set -ignoreMigrationPatterns='*:pending'.
Detected resolved migration not applied to database: 10.
To fix this error, either run migrate, or set -ignoreMigrationPatterns='*:pending'.
Detected resolved migration not applied to database: 12.
To fix this error, either run migrate, or set -ignoreMigrationPatterns='*:pending'.
Need more flexibility with validation rules? Learn more: https://rd.gt/3AbJUZE
at org.flywaydb.core.Flyway$4.execute(Flyway.java:252)
at org.flywaydb.core.Flyway$4.execute(Flyway.java:242)
at org.flywaydb.core.FlywayExecutor.execute(FlywayExecutor.java:204)
at org.flywaydb.core.Flyway.validate(Flyway.java:242)
at de.sovity.edc.extension.postgresql.FlywayUtils.validate(FlywayUtils.java:65)
at de.sovity.edc.extension.postgresql.FlywayUtils.migrateOrValidate(FlywayUtils.java:59)
at de.sovity.edc.extension.postgresql.FlywayUtils.cleanAndMigrate(FlywayUtils.java:37)
at de.sovity.edc.ext.catalog.crawler.dao.config.FlywayService.validateOrMigrateInTests(FlywayService.java:42)
at de.sovity.edc.ext.catalog.crawler.CrawlerExtensionContextBuilder.buildContext(CrawlerExtensionContextBuilder.java:116)
at de.sovity.edc.ext.catalog.crawler.CrawlerExtension.initialize(CrawlerExtension.java:120)
at org.eclipse.edc.boot.system.injection.lifecycle.InitializePhase.initialize(InitializePhase.java:37)
at org.eclipse.edc.boot.system.injection.lifecycle.ExtensionLifecycleManager.initialize(ExtensionLifecycleManager.java:61)
at java.base/java.util.stream.ReferencePipeline$3$1.accept(Unknown Source)
at java.base/java.util.stream.ReferencePipeline$3$1.accept(Unknown Source)
at java.base/java.util.stream.ReferencePipeline$3$1.accept(Unknown Source)
at java.base/java.util.ArrayList$ArrayListSpliterator.forEachRemaining(Unknown Source)
at java.base/java.util.stream.AbstractPipeline.copyInto(Unknown Source)
at java.base/java.util.stream.AbstractPipeline.wrapAndCopyInto(Unknown Source)
at java.base/java.util.stream.ReduceOps$ReduceOp.evaluateSequential(Unknown Source)
at java.base/java.util.stream.AbstractPipeline.evaluate(Unknown Source)
at java.base/java.util.stream.ReferencePipeline.collect(Unknown Source)
at org.eclipse.edc.boot.system.ExtensionLoader.bootServiceExtensions(ExtensionLoader.java:67)
at org.eclipse.edc.boot.system.runtime.BaseRuntime.bootExtensions(BaseRuntime.java:141)
at org.eclipse.edc.boot.system.runtime.BaseRuntime.boot(BaseRuntime.java:202)
at org.eclipse.edc.boot.system.runtime.BaseRuntime.boot(BaseRuntime.java:84)
at org.eclipse.edc.boot.system.runtime.BaseRuntime.main(BaseRuntime.java:72)ScreenshotsNo response |
Beta Was this translation helpful? Give feedback.
Replies: 11 comments 11 replies
-
|
@ivanov-petar can you please contact Sovity because we need this ASAP? |
Beta Was this translation helpful? Give feedback.
-
|
I'm not an expert on Flyway database migrations, but after internal consultation the log suggests the following:
The error may result from a corrupted database state:
This type of issue can arise if the database state was unintentionally corrupted, potentially due to external factors or incomplete migrations beforehand. Could you please try with a fresh new setup of databases? |
Beta Was this translation helpful? Give feedback.
-
|
Hi Tim, |
Beta Was this translation helpful? Give feedback.
-
Hi! Are you sure the Catalog Crawler is pointing to the correct database? Are you trying to start the Crawler without having the actual portal backend running? |
Beta Was this translation helpful? Give feedback.
-
|
I am deploying the crawler on a local instance to test if it starts up before we deploy on our AP, so I only have the container of the crawler and an empty PostgreSQL database. So there are not tables existing. So I think my answer to all your questions is yes. Shall I deploy additional components to test it properly? (the full AP?) I am kind of afraid on deploying it directly on our working environment. |
Beta Was this translation helpful? Give feedback.
-
|
Ah that explains the problem. To mitigate this, start up the Authority Portal Backend connected to the same database (local is fine, as long as it's the same DB) to let it create all necessary tables and go through the migrations and then start up the Crawler. It should then validate the DB schema successfully. |
Beta Was this translation helpful? Give feedback.
-
|
Thanks @kamilczaja , so just to be sure, only the backend is enough, or all of its other dependencies, other than postgres, should be there as well (KC, DAPS-KC)? Would it work if I import a dump of our database on my local empty database? Perhaps this way would be easier to see if it actually works, my expectation would be to see some crawling in action. Please excuse me for any "silly" questions. |
Beta Was this translation helpful? Give feedback.
-
|
Hey, Backend + Postgres (potentially also IAM Keycloak) should be enough. The backend is needed to populate the DB for the Crawler. Keep in mind though, if you don't spin up a DAPS, the Crawler will start but you won't see it actually working since it needs to talk to other connectors (which means it also needs to talk to the DAPS). Let me know if it worked out for you |
Beta Was this translation helpful? Give feedback.
-
|
Hi @kamilczaja, I I got a dump from our current AP database and restored it on my local DB, then the crawler worked, I see on the logs it is updating connectors. But I have another question. To register the crawler on the DAPS I took a shortcut and registered it as a connector. I suppose this might cause some issue when it is actually deployed on our dataspace, since it will try to harvest itself. What is the proper process to register it? I have Service Provider Admin role on the AP but I cannot register Dataspace components with this role. Is there another role or process for this? Thanks a lot in advance. |
Beta Was this translation helpful? Give feedback.
-
|
You would need the "Operator Admin" role to register it properly. You could also pre-define the client for the crawler in the DAPS with a certificate and everything and always start the Crawler with the same keys, bypassing the need of registering it in the Portal. That said, the former suggestion is probably easier |
Beta Was this translation helpful? Give feedback.
-
|
Is it possible to have both roles on the same time? I mean the "Service Partner Admin" and "Operator Admin". I already have the former plus "Participant Admin". |
Beta Was this translation helpful? Give feedback.
-
|
While you currently cannot assign multiple roles in the portal itself (that feature is on our roadmap), you can actually do that via the Keycloak Admin Interface to avoid having to create another user account. Go to your user -> Groups -> Join Group -> Select |
Beta Was this translation helpful? Give feedback.
-
|
Thanks a lot, that did the trick :) Thanks again a lot for your support! |
Beta Was this translation helpful? Give feedback.
-
|
Also, is there an API request to check what the crawler has harvested? |
Beta Was this translation helpful? Give feedback.
-
|
Unfortunately not, but you can check the logs to see which connectors have been crawled and the data catalog in the portal to see the actual data offers that have been harvested |
Beta Was this translation helpful? Give feedback.
-
|
Thanks I will check that once deployed then. Thanks a lot for your support! Logs look fine on my local, I got lots of this kind of entries |
Beta Was this translation helpful? Give feedback.
-
|
Hey, great to see it solved! I'll just re-open the discussion itself, so it stays on the start page of the discussion tab without changing the filters. One of the answers is marked as "answer" to the original error-log on the top of this discussion :) |
Beta Was this translation helpful? Give feedback.
-
|
Thanks a lot again for support. The crawler is now deployed on our AP deployment, working as a charm so far. I think the documentation could be improved on this issue, I had a guess that the empty database would probably be the issue, but you never know :) There is an issue with the dashboard showing it down but this is low priority for now. I guess this is retrieved from kuma? Also, @kamilczaja said that there are no endpoints, although we open the 11003 port of the connector as per documentation and I see on the logs that the Management API extension is loaded, but when tried to access any of those on 11003 and 11002 (I mapped this as well) I get back a 404. Could you please point me to the codebase of the 10.5.0 version of the crawler? We will need to develop an extension for it and it is a bit difficult to find it after all these repo changes. Thanks a lot in advance. |
Beta Was this translation helpful? Give feedback.
-
|
Hi, on the code-base, I think the 10.5.0 crawler was still in the edc-ce repo at that time, see release-tag and extension there (@kamilczaja correct me if I'm wrong): https://github.com/sovity/edc-ce/tree/v10.5.0/extensions/catalog-crawler |
Beta Was this translation helpful? Give feedback.
-
|
Hi. Up until recently the crawler was an extension of the EDC-CE. Has this changed and how? Is the code of the crawler available now somewhere? |
Beta Was this translation helpful? Give feedback.
-
|
Yes, it has moved to this repository: https://github.com/sovity/dataspace-portal/tree/main/authority-portal-backend/catalog-crawler |
Beta Was this translation helpful? Give feedback.
-
|
Tim, thank you very much for the prompt reply. |
Beta Was this translation helpful? Give feedback.
Ah that explains the problem.
The Crawler validates database migrations but does not migrate on its own. It expects the database to be in an already set up and migrated state.
To mitigate this, start up the Authority Portal Backend connected to the same database (local is fine, as long as it's the same DB) to let it create all necessary tables and go through the migrations and then start up the Crawler. It should then validate the DB schema successfully.