Merge branch 'release/2.0.3'

Author: aebadirad

.travis.yml

@@ -1,9 +1,6 @@
 matrix:
   include:
-    - env: ML_VERSION=8.0-7
-    - env: ML_VERSION=9.0-1.1
-    - env: ML_VERSION=9.0-2
-    - env: ML_VERSION=9.0-3
+    - env: ML_VERSION=9.0-3.1
 
 dist: trusty
 sudo: required

CHANGELOG.md

@@ -16,13 +16,15 @@ with a fix for the issue you filed.
 You can request a new feature by submitting an issue to our [GitHub Issue Tracker][issue tracker].  If you
 would like to implement a new feature then first create a new issue and discuss it with one of our
 project maintainers.
+#####Note: as of DHF 2.0.3, pull requests will only be accepted for MarkLogic 9. Only critical bug fixes will be accepted for MarkLogic 8 on the legacy 2.0.2 or earlier branches.
 
 ## Building the Framework from Source
 Looking to build the code from source? Look no further.
 
 #### Prerequisites
 You need these to get started
 
+- MarkLogic 9+
 - Java 8 JDK
 - Gradle (3.4 or greater)
 - A decent IDE. IntelliJ is nice.
@@ -73,7 +75,7 @@ plugins {
    ...
 
    // comment out this line. It pulls the version from the cloud
-   // id 'com.marklogic.ml-data-hub' version '2.0.2'
+   // id 'com.marklogic.ml-data-hub' version '2.0.3'
 }
 
 // this tells gradle to apply the plugin you included above in the buildscript section
@@ -287,7 +289,22 @@ Say you want to run FlowRunnerTest.
 $ ./gradlew -Dtest.single=FlowRunner test
 ```
 
-Make sure that all tests pass. Please, do not submit patches that fail.
+If you want to run just the Quick-Start UI End to End tests, you will need nodejs 8.9.1 or later installed:
+```jshelllanguage
+gradlew bootrun
+cd quick-start
+npm install
+npm install -g protractor
+npm run webdriver-update
+npm run e2e 
+```
+*Note for e2e tests, the datahub must be running and so must a MarkLogic instance with appservers for 8010-8014 free.*
+
+For those that want to run the E2E tests from Intellij or another IDE to fullstack debug you can have add a run/debug
+task that runs the script "e2e". Make sure to add a 'before launch' task as folows: npm run "webdriver-update".
+
+
+**Make sure that all tests pass. Please, do not submit patches that fail.**
 
 #### Push your changes
 

CONTRIBUTING.md

@@ -1,11 +1,10 @@
- 
+
 Copyright © 2017 MarkLogic Corporation.
-This project and its code and functionality is not representative of MarkLogic Server and is not supported by MarkLogic.
- 
+
 Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at
- 
+
 http://www.apache.org/licenses/LICENSE-2.0
- 
+
 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.
 
                                  Apache License

LICENSE

@@ -1,97 +1,17 @@
-_The MarkLogic Data Hub Framework is a data integration framework and tool-set to quickly and efficiently integrate data from many sources into a single MarkLogic database, and expose that data._
-
-_The Data Hub Framework is free and open source under the [Apache 2 License](https://github.com/marklogic-community/marklogic-data-hub/blob/master/LICENSE) and is supported by the community of developers who build and contribute to it. Please note that this open source project and its code and functionality is not representative of MarkLogic Server and is not supported by MarkLogic._
-
-| OS | Status |
-| --- | --- |
-| Linux/Mac | [![Build Status](https://travis-ci.org/marklogic-community/marklogic-data-hub.svg?branch=master)](https://travis-ci.org/marklogic-community/marklogic-data-hub) |
-| Windows | [![Windows Build status](https://ci.appveyor.com/api/projects/status/kgj0k5na59uhkvbv/branch/master?svg=true)](https://ci.appveyor.com/project/paxtonhare/marklogic-data-hub) |
-
-_This is the official branch of The MarkLogic Data Hub Framework. It is compatible with MarkLogic 8 or 9._
-
-_If you are looking for the Legacy 1.0 branch ( MarkLogic 8 compatible only ) go [here](https://github.com/marklogic-community/marklogic-data-hub/tree/1.0-master)._
-
 # MarkLogic Data Hub
 
-Go from nothing to Operational Data Hub in a matter of minutes.  
+Go from nothing to an Operational Data Hub in a matter of minutes.  
 
-This project allows you to deploy a skeleton Data Hub into MarkLogic. With some basic configuration you will be running an Operational Data Hub in no time.
+The MarkLogic Data Hub Framework is a data integration framework and tool-set to quickly and efficiently integrate data from many sources into a single MarkLogic database and then expose that data.
 
 # Getting Started
+Grab the [latest release](https://github.com/marklogic-community/marklogic-data-hub/releases) then visit our [Data Hub Framework website](https://marklogic-community.github.io/marklogic-data-hub/) to get started.
 
-### Prerequisites
-
-You need these to get started
-
-- Java 8 JDK
-- MarkLogic 8.0-7 or greater, or 9.0-1.1 or greater
-- Gradle 3.4 or greater **(Optional)**
-
-### TL;DR
-
-Head over to our [Getting Started Tutorial](https://marklogic-community.github.io/marklogic-data-hub/) to get up and running with the Data Hub.
-
-Or watch the [MarkLogic University - Data Hub Framework On Demand Video Courses](http://mlu.marklogic.com/ondemand/index.xqy?q=Series%3A%22Operational%20Data%20Hubs%22)
-
-### The Easiest Way
-
-To use the Data Hub Framework you should download the latest [quickstart.war](https://github.com/marklogic-community/marklogic-data-hub/releases/download/v2.0.2/quick-start-2.0.2.war).
-
-Then Run the war like so:
-
-```bash
-java -jar quick-start-2.0.2.war
-```
-
-### Using the Hub in your existing Java project
-
-Alternatively you can include the jar file as a build dependency in your Java project. Make sure you reference the latest version.
-
-**Gradle**
-
-```groovy
-compile('com.marklogic:marklogic-data-hub:2.0.2')
-```
-
-**Maven**
-
-```xml
-<dependency>
-  <groupId>com.marklogic</groupId>
-  <artifactId>marklogic-data-hub</artifactId>
-  <version>2.0.2</version>
-  <type>pom</type>
-</dependency>
-```
-
-**Ivy**
-
-```xml
-<dependency org='com.marklogic' name='marklogic-data-hub' rev='2.0.2'>
-  <artifact name='$AID' ext='pom'></artifact>
-</dependency>
-```
-
-### Command Line Ninjas
-
-If you prefer to use gradle for all of your hub interactions then you can include the ml-data-hub gradle plugin in your build.gradle file:
-
-```groovy
-plugins {
-    id 'com.marklogic.ml-data-hub' version '2.0.2'
-}
-```
-
-Now you have full access to the Data Hub tasks. To see all available tasks run:
-
-```bash
-./gradlew tasks
-```
-
-There is a sample barebones project [here](https://github.com/marklogic-community/marklogic-data-hub/tree/master/examples/barebones)
 
-# Building From Source
+##Advanced Hub Usage
+Ready to go beyond QuickStart and start using DHF in your own Java or Gradle project? The [Data Hub Framework website](https://marklogic-community.github.io/) also provides that.
 
-Feeling intrepid? Want to contrubute to the Data Hub Framework? Perhaps you just want to poke the code?
+## Don't get this code
+You don't need to checkout this repo unless you plan to actively contribute changes to the Data Hub Framework. Go to [our website](https://marklogic-community.github.io/marklogic-data-hub/) to get started with DHF.
 
-Look at our [CONTRIBUTING.md](https://github.com/marklogic-community/marklogic-data-hub/blob/master/CONTRIBUTING.md#building-the-framework-from-source) file for details on building from source.
+If you really do want to contribute, see our [Contributing Guide](https://github.com/marklogic-community/marklogic-data-hub/blob/master/CONTRIBUTING.md) to get started.

README.md

@@ -0,0 +1,29 @@
+#!/bin/sh
+
+# This script will build the javadocs for the DHF jar file
+# and push them into the docs branch to be magically published
+# via travis to the https://marklogic-community.github.io/marklogic-data-hub site
+
+# run this after you update gradle.properties to contain the new version
+# preferable after you do a release
+#
+# this script will create a new branch add-${hubversion}-javadocs
+# you will need to create a Pull Request and review it/approve/merge
+# travis will then update the docs site
+
+tmpdir=$(mktemp -d)
+hubversion=$(cat gradle.properties | awk -F"=" '{print $2}')
+echo "tmpdir: ${tmpdir}"
+echo "version: ${hubversion}"
+git clone -b docs git@github.com:marklogic-community/marklogic-data-hub.git ${tmpdir}/docs
+./gradlew javadoc
+mkdir -p ${tmpdir}/docs/javadocs/${hubversion}
+cp -R marklogic-data-hub/build/docs/javadoc ${tmpdir}/docs/javadocs/${hubversion}
+
+cd ${tmpdir}/docs
+branch_name=add-${hubversion}-javadocs
+git checkout -b ${branch_name}
+git add javadocs/${hubversion}
+git commit -m "adding ${hubversion} javadocs"
+git push origin ${branch_name}
+rm -rf ${tmpdir}

build-docs.sh

@@ -0,0 +1,46 @@
+# development and build time 
+- junit 4 & 5 -> www.junit.org
+- gradle 3.4+ -> www.gradle.org
+- gradle properties plugin -> https://github.com/stevesaliman/gradle-properties-plugin
+- gradle node plugin -> https://github.com/srs/gradle-node-plugin
+- xmlunit 1.3 -> http://www.xmlunit.org/
+- jsonassert 1.5.0 -> https://github.com/skyscreamer/JSONassert
+- hamcrest-junit 2.0.0.0 -> https://github.com/hamcrest/hamcrest-junit
+- easymock 3.4 -> http://easymock.org/
+- logback classis 1.1.11 -> https://logback.qos.ch/
+- mlcp 9.0.4 -> https://github.com/marklogic/marklogic-contentpump
+- spock 1.1-groovy-2.4 -> http://spockframework.org/
+
+# DHF Runtime
+- ml-gradle 3.4 -> https://github.com/marklogic-community/ml-gradle
+- ml-app-deployer 3.4 -> https://github.com/marklogic-community/ml-app-deployer
+- ml-javaclient-util 3.3.1 -> https://github.com/marklogic-community/ml-javaclient-util
+- marklogic-client-api 4.0.3.1 -> https://github.com/marklogic/java-client-api
+- mlcp-util 0.2.0 -> https://github.com/marklogic-community/mlcp-util
+- apache commons io 2.4 -> https://commons.apache.org/proper/commons-io/
+- apache commons text 1.1 -> https://commons.apache.org/proper/commons-text/
+- apache commons csv 1.4 -> https://commons.apache.org/proper/commons-csv/
+-
+
+# Quickstart Runtime
+- spring boot 1.5.6.RELEASE -> https://projects.spring.io/spring-boot/
+- aspectj 1.8.9 -> https://www.eclipse.org/aspectj/
+- spring messaging -> https://github.com/spring-projects/spring-framework/tree/master/spring-messaging/src/main/java/org/springframework/messaging
+- angular-mdl 4.0.8 -> https://github.com/mseemann/angular2-mdl
+- angular 4.2.4 -> https://angular.io/
+- classlist.js 1.1.20150312 -> https://www.npmjs.com/package/classList.js
+- clipboard 1.7.1 -> https://www.npmjs.com/package/clipboard
+- codemirror 5.30.0 -> https://www.npmjs.com/package/codemirror
+- core-js 2.5.1 -> https://www.npmjs.com/package/core-js
+- date-fns 1.29.0 -> https://www.npmjs.com/package/date-fns
+- font-awesome 4.7.0 -> https://www.npmjs.com/package/font-awesome
+- lodash 4.17.4 -> https://www.npmjs.com/package/lodash
+- material-design-icons-iconfont 3.0.3 -> https://www.npmjs.com/package/material-design-icons-iconfont
+- material-design-lite 1.3.0 -> https://www.npmjs.com/package/material-design-lite
+- mdi 2.0.46 -> https://www.npmjs.com/package/mdi
+- ng2-device-detector 1.0.0 -> https://www.npmjs.com/package/ng2-device-detector
+- rxjs 5.4.3 -> https://www.npmjs.com/package/rxjs
+- semver 5.4.1 -> https://www.npmjs.com/package/semver
+- sockjs-client 1.1.4 -> https://www.npmjs.com/package/sockjs-client
+- stompjs 2.3.3 -> https://www.npmjs.com/package/stompjs
+- zone.js 0.8.18 -> https://www.npmjs.com/package/zone.js

dependencies.md

@@ -13,7 +13,7 @@ plugins {
     // it includes ml-gradle. This plugin is what lets you
     // run DHF (Data Hub Framework) tasks from the
     // command line
-    id 'com.marklogic.ml-data-hub' version '2.0.2'
+    id 'com.marklogic.ml-data-hub' version '2.0.3'
 }
 ```
 

examples/barebones/README.md

@@ -7,5 +7,5 @@ plugins {
     // it includes ml-gradle. This plugin is what lets you
     // run DHF (Data Hub Framework) tasks from the
     // command line
-    id 'com.marklogic.ml-data-hub' version '2.0.2'
+    id 'com.marklogic.ml-data-hub' version '2.0.3'
 }

examples/barebones/build.gradle

@@ -15,9 +15,9 @@ The sample data is located in the input/ folder.
 ```
 
 # TLDR; How do I run it?
-1. Download the [latest quick-start war](https://github.com/marklogic-community/marklogic-data-hub/releases/download/v2.0.2/quick-start-2.0.2.war) into this folder.
+1. Download the [latest quick-start war](https://github.com/marklogic-community/marklogic-data-hub/releases/download/v2.0.3/quick-start-2.0.3.war) into this folder.
 
-1. Run the quick-start jar `java -jar quick-start-2.0.2.war`
+1. Run the quick-start jar `java -jar quick-start-2.0.3.war`
 
 1. Open your web browser to [http://localhost:8080](http://localhost:8080).
 
@@ -39,7 +39,7 @@ The sample data is located in the input/ folder.
 
 1. Browse the Input Files to **input/hl7**.
 
-1. Set the Data Format to **Documents**.
+1. Set the Input File Type to **Documents**.
 
 1. Now Click **RUN IMPORT**.
 
@@ -51,29 +51,27 @@ The sample data is located in the input/ folder.
 
 1. Browse the Input Files to **input/nppes**.
 
-1. Set the Data Format to **Delimited Text**.
+1. Set the Input File Type to **Delimited Text**.
 
 1. Now Click **RUN IMPORT**.
 
 ## Browsing the Staging Data
 
-1. At this point you have loaded the sample data. You can browse the data by clicking on the Browse tab at the top menu.
+1. At this point you have loaded the sample data. You can browse the data by clicking on the Browse Data tab at the top menu.
 
 ## Harmonizing Data
 
 1. Click on the **Flows** tab.
 
 1. Click on the **final** flow.
 
-1. To run the harmonize flows simply press the **Run** button on the info screen.
+1. To run the harmonize flows simply press the **Run Harmonize** button on the info screen.
 
 ## Browsing the Harmonized Data
 
-1. Now you have harmonized the data into your final database. You can browse the harmonized data by clicking on the Browse tab at the top menu.
+1. Now you have harmonized the data into your final database. You can browse the harmonized data by clicking on the Browse Data tab at the top menu.
 
-1. Next change the database to **FINAL**.
-
-1. Click Search.
+1. Next change the database to **FINAL**. Quick Start will automatically run a search against the Final database.
 
 # Entities
 Entities represent the data you are modeling. For this example we provide the **Patients** entity. Inside this entity definition you will find all of the example flows.
@@ -87,14 +85,14 @@ Flows are sets of plugins that work together to create an envelope document.
 ## Input Flows
 
 ### hl7
-The hl7 Flow is intended to ingest C-CDA C32 Hl7 XML files. When running the hl7 flow simply point it at input/hl7. Set the document type to **Document**.
+The hl7 Flow is intended to ingest C-CDA C32 Hl7 XML files. When running the hl7 flow simply point it at input/hl7. Set the Input Document Type to **Document**.
 
 ### nppes
-The nppes Flow is intended to ingest NPPES csv files. This flow will split each row of the NPPES file into a separate XML document in the staging database. When running the hl7 flow simply point it at input/nppes. Set the document type to **Delimited Text**.
+The nppes Flow is intended to ingest NPPES csv files. This flow will split each row of the NPPES file into a separate XML document in the staging database. When running the hl7 flow simply point it at input/nppes. Set the Input Document Type to **Delimited Text**.
 
 ## Harmonize Flows
 
-There is only one harmonize flow provided. This final flow will create a harmonized XML document that contains the original C32 xml as the content of an envelope. It will also extract various data from th3 C32 into the header section for easier queryability.
+There is only one harmonize flow provided. This final flow will create a harmonized XML document that contains the original C32 xml as the content of an envelope. It will also extract various data from the C32 into the header section for easier queryability.
 
 ## Final REST services