Merge branch 'release/1.1.5' into 1.0-master

Author: paxtonhare

CHANGELOG.md

@@ -1,16 +1,43 @@
 # Change Log
 
-## [v1.1.4](https://github.com/marklogic-community/marklogic-data-hub/tree/v1.1.4)
+## [v1.1.5](https://github.com/marklogic-community/marklogic-data-hub/tree/v1.1.5)
 
-[Full Changelog](https://github.com/marklogic-community/marklogic-data-hub/compare/v2.0.0-rc.1...v1.1.4)
+[Full Changelog](https://github.com/marklogic-community/marklogic-data-hub/compare/v1.1.4...v1.1.5)
+
+**Implemented enhancements:**
+
+- Update Quickstart Ready message [\#478](https://github.com/marklogic-community/marklogic-data-hub/issues/478)
+- Example: Migrating a RDBMS to Data Hub using Spring Batch [\#473](https://github.com/marklogic-community/marklogic-data-hub/issues/473)
 
 **Fixed bugs:**
 
+- Providing different source/dest DB for the hubRunFlow does not work [\#488](https://github.com/marklogic-community/marklogic-data-hub/issues/488)
+- Path to code has wrong slash on Windows [\#467](https://github.com/marklogic-community/marklogic-data-hub/issues/467)
+- Job status stuck on STARTED for Input Flows [\#466](https://github.com/marklogic-community/marklogic-data-hub/issues/466)
 - File not saving properly from quickstart to fix a bug [\#430](https://github.com/marklogic-community/marklogic-data-hub/issues/430)
+- Error on initial run of QuickStart GUI [\#394](https://github.com/marklogic-community/marklogic-data-hub/issues/394)
+- Error saving entity - collation not legal [\#337](https://github.com/marklogic-community/marklogic-data-hub/issues/337)
 
 **Closed issues:**
 
+- collection name is hard coded in online store example [\#474](https://github.com/marklogic-community/marklogic-data-hub/issues/474)
+- add \(rest-extension-user,read\) to XML documents in modules-db [\#470](https://github.com/marklogic-community/marklogic-data-hub/issues/470)
+- column-width, or tooltip with full "Identifier" in traces table [\#469](https://github.com/marklogic-community/marklogic-data-hub/issues/469)
+- Initializing DHF Project against existing DB is dropping indexes [\#468](https://github.com/marklogic-community/marklogic-data-hub/issues/468)
 - Saving the changes in a flow code never finishes \*sometimes\* [\#462](https://github.com/marklogic-community/marklogic-data-hub/issues/462)
+- Sometimes a trace for a failed Harmonize job is not available/not existing [\#461](https://github.com/marklogic-community/marklogic-data-hub/issues/461)
+- Unable to ingest image \(.png\) documents using DHF Quick-Start application [\#420](https://github.com/marklogic-community/marklogic-data-hub/issues/420)
+- Load files into the data hub schemas database [\#288](https://github.com/marklogic-community/marklogic-data-hub/issues/288)
+- Ability to override uri for input flows [\#182](https://github.com/marklogic-community/marklogic-data-hub/issues/182)
+- Create concept of a domain [\#82](https://github.com/marklogic-community/marklogic-data-hub/issues/82)
+
+**Merged pull requests:**
+
+- Feature/473 msb example [\#480](https://github.com/marklogic-community/marklogic-data-hub/pull/480) ([sastafford](https://github.com/sastafford))
+- Some typos and suggestions [\#475](https://github.com/marklogic-community/marklogic-data-hub/pull/475) ([patrickmcelwee](https://github.com/patrickmcelwee))
+
+## [v1.1.4](https://github.com/marklogic-community/marklogic-data-hub/tree/v1.1.4) (2017-07-25)
+[Full Changelog](https://github.com/marklogic-community/marklogic-data-hub/compare/v2.0.0-rc.1...v1.1.4)
 
 ## [v2.0.0-rc.1](https://github.com/marklogic-community/marklogic-data-hub/tree/v2.0.0-rc.1) (2017-07-24)
 [Full Changelog](https://github.com/marklogic-community/marklogic-data-hub/compare/v2.0.0-beta.1...v2.0.0-rc.1)

CONTRIBUTING.md

@@ -71,7 +71,7 @@ plugins {
    ...
 
    // comment out this line. It pulls the version from the cloud
-   // id 'com.marklogic.ml-data-hub' version '1.1.4'
+   // id 'com.marklogic.ml-data-hub' version '1.1.5'
 }
 
 // this tells gradle to apply the plugin you included above in the buildscript section

README.md

@@ -4,8 +4,8 @@ _The Data Hub Framework is free and open source under the [Apache 2 License](htt
 
 | OS | Status |
 | --- | --- |
-| Linux/Mac | [![Build Status](https://travis-ci.org/marklogic-community/marklogic-data-hub.svg?branch=1.0-develop)](https://travis-ci.org/marklogic-community/marklogic-data-hub) |
-| Windows | [![Windows Build status](https://ci.appveyor.com/api/projects/status/kgj0k5na59uhkvbv/branch/1.0-develop?svg=true)](https://ci.appveyor.com/project/paxtonhare/marklogic-data-hub) |
+| Linux/Mac | [![Build Status](https://travis-ci.org/marklogic-community/marklogic-data-hub.svg?branch=1.0-master)](https://travis-ci.org/marklogic-community/marklogic-data-hub) |
+| Windows | [![Windows Build status](https://ci.appveyor.com/api/projects/status/kgj0k5na59uhkvbv/branch/1.0-master?svg=true)](https://ci.appveyor.com/project/paxtonhare/marklogic-data-hub) |
 
 # MarkLogic Data Hub
 
@@ -46,7 +46,7 @@ Alternatively you can include the jar file as a build dependency in your Java pr
 **Gradle**
 
 ```groovy
-compile('com.marklogic:marklogic-data-hub:1.1.4')
+compile('com.marklogic:marklogic-data-hub:1.1.5')
 ```
 
 **Maven**
@@ -55,15 +55,15 @@ compile('com.marklogic:marklogic-data-hub:1.1.4')
 <dependency>
   <groupId>com.marklogic</groupId>
   <artifactId>marklogic-data-hub</artifactId>
-  <version>1.1.4</version>
+  <version>1.1.5</version>
   <type>pom</type>
 </dependency>
 ```
 
 **Ivy**
 
 ```xml
-<dependency org='com.marklogic' name='marklogic-data-hub' rev='1.1.4'>
+<dependency org='com.marklogic' name='marklogic-data-hub' rev='1.1.5'>
   <artifact name='$AID' ext='pom'></artifact>
 </dependency>
 ```
@@ -74,7 +74,7 @@ If you prefer to use gradle for all of your hub interactions then you can includ
 
 ```groovy
 plugins {
-    id 'com.marklogic.ml-data-hub' version '1.1.4'
+    id 'com.marklogic.ml-data-hub' version '1.1.5'
 }
 ```
 

examples/spring-batch/build.gradle

@@ -2,7 +2,7 @@ plugins {
     id 'java'
     id 'application'
     id 'net.saliman.properties' version '1.4.6'
-    id 'com.marklogic.ml-data-hub' version '1.1.4'
+    id 'com.marklogic.ml-data-hub' version '1.1.5'
 }
 
 repositories {
@@ -12,7 +12,7 @@ repositories {
 }
 
 dependencies {
-    compile 'com.marklogic:marklogic-data-hub:1.1.4'
+    compile 'com.marklogic:marklogic-data-hub:1.1.5'
     compile "com.marklogic:marklogic-spring-batch-core:0.7.4"
     compile 'com.marklogic:ml-javaclient-util:4.0.alpha4'
 

gradle.properties

@@ -1 +1 @@
-version=1.1.4
+version=1.1.5

marklogic-data-hub/src/main/java/com/marklogic/hub/HubConfig.java

@@ -140,7 +140,7 @@ public class HubConfig {
 
     public String customForestPath = DEFAULT_CUSTOM_FOREST_PATH;
 
-    public String modulePermissions = "rest-admin,read,rest-admin,update,rest-extension-user,execute";
+    public String modulePermissions = "rest-extension-user,read,rest-extension-user,execute,rest-admin,read,rest-admin,update";
 
     public String projectDir;
 
@@ -281,6 +281,7 @@ public DatabaseClient newStagingClient() {
         return DatabaseClientFactory.newClient(
             host,
             stagingPort,
+            stagingDbName,
             username,
             password,
             DatabaseClientFactory.Authentication.valueOf(stagingAuthMethod.toUpperCase()));
@@ -308,6 +309,7 @@ public DatabaseClient newFinalClient() {
         return DatabaseClientFactory.newClient(
             host,
             finalPort,
+            finalDbName,
             username,
             password,
             DatabaseClientFactory.Authentication.valueOf(finalAuthMethod.toUpperCase()));
@@ -321,6 +323,7 @@ public DatabaseClient newJobDbClient() {
         return DatabaseClientFactory.newClient(
             host,
             jobPort,
+            jobDbName,
             username,
             password,
             DatabaseClientFactory.Authentication.valueOf(jobAuthMethod.toUpperCase()));
@@ -348,6 +351,7 @@ public DatabaseClient newTraceDbClient() {
         return DatabaseClientFactory.newClient(
             host,
             tracePort,
+            traceDbName,
             username,
             password,
             DatabaseClientFactory.Authentication.valueOf(stagingAuthMethod.toUpperCase()));

marklogic-data-hub/src/main/java/com/marklogic/hub/collector/ServerCollector.java

@@ -18,7 +18,6 @@
 import com.fasterxml.jackson.databind.ObjectMapper;
 import com.marklogic.client.DatabaseClient;
 import com.marklogic.hub.HubConfig;
-import com.marklogic.hub.HubDatabase;
 import com.marklogic.hub.plugin.PluginType;
 import com.marklogic.xcc.*;
 import org.slf4j.Logger;
@@ -33,7 +32,6 @@ public class ServerCollector extends AbstractCollector {
     protected final Logger logger = LoggerFactory.getLogger(this.getClass());
     private DatabaseClient client = null;
     private HubConfig hubConfig = null;
-    private HubDatabase sourceDb = null;
 
     private String module;
 
@@ -43,7 +41,6 @@ public ServerCollector(PluginType type, String module) {
     }
 
     public void setHubConfig(HubConfig config) { this.hubConfig = config; }
-    public void setHubDatabase(HubDatabase sourceDb) { this.sourceDb = sourceDb; }
 
     public DatabaseClient getClient() {
         return this.client;
@@ -68,15 +65,8 @@ public void serialize(XMLStreamWriter serializer) throws XMLStreamException {
 
     @Override
     public DiskQueue<String> run(String jobId, int threadCount, Map<String, Object> options) {
-        int port = hubConfig.stagingPort;
-        String dbName = hubConfig.stagingDbName;
-        if (sourceDb.equals(HubDatabase.FINAL)) {
-            port = hubConfig.finalPort;
-            dbName  = hubConfig.finalDbName;
-        }
-
         try {
-            ContentSource cs = ContentSourceFactory.newContentSource(hubConfig.host, port, hubConfig.getUsername(), hubConfig.getPassword(), dbName);
+            ContentSource cs = ContentSourceFactory.newContentSource(client.getHost(), client.getPort(), hubConfig.getUsername(), hubConfig.getPassword(), client.getDatabase());
             Session activeSession = cs.newSession();
             RequestOptions requestOptions = new RequestOptions();
             requestOptions.setCacheResult(false);

marklogic-data-hub/src/main/java/com/marklogic/hub/flow/FlowRunner.java

@@ -1,5 +1,6 @@
 package com.marklogic.hub.flow;
 
+import com.marklogic.client.DatabaseClient;
 import com.marklogic.client.datamovement.JobTicket;
 import com.marklogic.hub.HubDatabase;
 
@@ -11,8 +12,8 @@ public interface FlowRunner {
     FlowRunner withFlow(Flow flow);
     FlowRunner withBatchSize(int batchSize);
     FlowRunner withThreadCount(int threadCount);
-    FlowRunner withSourceDatabase(HubDatabase sourceDatabase);
-    FlowRunner withDestinationDatabase(HubDatabase destinationDatabase);
+    FlowRunner withSourceClient(DatabaseClient sourceClient);
+    FlowRunner withDestinationDatabase(String destinationDatabase);
     FlowRunner withOptions(Map<String, Object> options);
 
     FlowRunner onItemComplete(FlowItemCompleteListener listener);

marklogic-data-hub/src/main/java/com/marklogic/hub/flow/impl/FlowRunnerImpl.java

@@ -32,8 +32,8 @@ public class FlowRunnerImpl implements FlowRunner {
     private Flow flow;
     private int batchSize = DEFAULT_BATCH_SIZE;
     private int threadCount = DEFAULT_THREAD_COUNT;
-    private HubDatabase sourceDatabase = HubDatabase.STAGING;
-    private HubDatabase destinationDatabase = HubDatabase.FINAL;
+    private DatabaseClient sourceClient;
+    private String destinationDatabase;
     private Map<String, Object> options;
     private int previousPercentComplete;
 
@@ -47,6 +47,8 @@ public class FlowRunnerImpl implements FlowRunner {
 
     public FlowRunnerImpl(HubConfig hubConfig) {
         this.hubConfig = hubConfig;
+        this.sourceClient = hubConfig.newStagingClient();
+        this.destinationDatabase = hubConfig.finalDbName;
     }
 
     @Override
@@ -68,13 +70,13 @@ public FlowRunner withThreadCount(int threadCount) {
     }
 
     @Override
-    public FlowRunner withSourceDatabase(HubDatabase sourceDatabase) {
-        this.sourceDatabase = sourceDatabase;
+    public FlowRunner withSourceClient(DatabaseClient sourceClient) {
+        this.sourceClient = sourceClient;
         return this;
     }
 
     @Override
-    public FlowRunner withDestinationDatabase(HubDatabase destinationDatabase) {
+    public FlowRunner withDestinationDatabase(String destinationDatabase) {
         this.destinationDatabase = destinationDatabase;
         return this;
     }
@@ -122,17 +124,6 @@ public void awaitCompletion(long timeout, TimeUnit unit) throws InterruptedExcep
         runningThread.join(unit.convert(timeout, TimeUnit.MILLISECONDS));
     }
 
-    private DatabaseClient getSourceClient() {
-        DatabaseClient srcClient;
-        if (sourceDatabase.equals(HubDatabase.STAGING)) {
-            srcClient = hubConfig.newStagingClient();
-        }
-        else {
-            srcClient = hubConfig.newFinalClient();
-        }
-        return srcClient;
-    }
-
     @Override
     public JobTicket run() {
         String jobId = UUID.randomUUID().toString();
@@ -142,13 +133,10 @@ public JobTicket run() {
             .withJobId(jobId);
         jobManager.saveJob(job);
 
-        DatabaseClient srcClient = getSourceClient();
-
         Collector c = flow.getCollector();
         if (c instanceof ServerCollector) {
             ((ServerCollector)c).setHubConfig(hubConfig);
-            ((ServerCollector)c).setHubDatabase(sourceDatabase);
-            ((ServerCollector)c).setClient(srcClient);
+            ((ServerCollector)c).setClient(sourceClient);
         }
 
         AtomicLong successfulEvents = new AtomicLong(0);
@@ -176,7 +164,7 @@ public JobTicket run() {
 
         ArrayList<String> errorMessages = new ArrayList<>();
 
-        DataMovementManager dataMovementManager = srcClient.newDataMovementManager();
+        DataMovementManager dataMovementManager = sourceClient.newDataMovementManager();
 
         double batchCount = Math.ceil((double)uris.size() / (double)batchSize);
 
@@ -185,7 +173,7 @@ public JobTicket run() {
             .withThreadCount(threadCount)
             .onUrisReady((QueryBatch batch) -> {
                 try {
-                    FlowResource flowRunner = new FlowResource(batch.getClient(), batch.getClient().getDatabase(), flow);
+                    FlowResource flowRunner = new FlowResource(batch.getClient(), destinationDatabase, flow);
                     RunFlowResponse response = flowRunner.run(jobId, batch.getItems(), options);
                     failedEvents.addAndGet(response.errorCount);
                     successfulEvents.addAndGet(response.totalCount - response.errorCount);

marklogic-data-hub/src/main/java/com/marklogic/hub/util/MlcpConsumer.java

@@ -0,0 +1,54 @@
+package com.marklogic.hub.util;
+
+import com.marklogic.hub.flow.FlowStatusListener;
+
+import java.util.List;
+import java.util.concurrent.atomic.AtomicLong;
+import java.util.function.Consumer;
+import java.util.regex.Matcher;
+import java.util.regex.Pattern;
+
+public class MlcpConsumer implements Consumer<String> {
+    private int currentPc = 0;
+    private final Pattern completedPattern = Pattern.compile("^.+completed (\\d+)%$");
+    private final Pattern successfulEventsPattern = Pattern.compile("^.+OUTPUT_RECORDS_COMMITTED:\\s+(\\d+).*$");
+    private final Pattern failedEventsPattern = Pattern.compile("^.+OUTPUT_RECORDS_FAILED\\s+(\\d+).*$");
+    private AtomicLong successfulEvents;
+    private AtomicLong failedEvents;
+    private FlowStatusListener statusListener;
+    private String jobId;
+
+    public MlcpConsumer(AtomicLong successfulEvents, AtomicLong failedEvents, FlowStatusListener statusListener,
+                        String jobId)
+    {
+        this.successfulEvents = successfulEvents;
+        this.failedEvents = failedEvents;
+        this.statusListener = statusListener;
+        this.jobId = jobId;
+    }
+
+    @Override
+    public void accept(String status) {
+        Matcher m = completedPattern.matcher(status);
+        if (m.matches()) {
+            int pc = Integer.parseInt(m.group(1));
+
+            // don't send 100% because more stuff happens after 100% is reported here
+            if (pc > currentPc && pc != 100) {
+                currentPc = pc;
+            }
+        }
+
+        m = successfulEventsPattern.matcher(status);
+        if (m.matches()) {
+            successfulEvents.addAndGet(Long.parseLong(m.group(1)));
+        }
+
+        m = failedEventsPattern.matcher(status);
+        if (m.matches()) {
+            failedEvents.addAndGet(Long.parseLong(m.group(1)));
+        }
+
+        statusListener.onStatusChange(jobId, currentPc, status);
+    }
+}