Merge branch 'release/1.0.0-rc.3'

Author: paxtonhare

CHANGELOG.md

@@ -1,7 +1,30 @@
 # Change Log
 
-## [v1.0.0-rc.2](https://github.com/marklogic/marklogic-data-hub/tree/v1.0.0-rc.2)
+## [v1.0.0-rc.3](https://github.com/marklogic/marklogic-data-hub/tree/v1.0.0-rc.3)
 
+[Full Changelog](https://github.com/marklogic/marklogic-data-hub/compare/v1.0.0-rc.2...v1.0.0-rc.3)
+
+**Fixed bugs:**
+
+- Better support for mixed flow and file type [\#228](https://github.com/marklogic/marklogic-data-hub/issues/228)
+
+**Closed issues:**
+
+- better support for binaries [\#259](https://github.com/marklogic/marklogic-data-hub/issues/259)
+- Get traces is super slow [\#258](https://github.com/marklogic/marklogic-data-hub/issues/258)
+- Job Output window stuck and not close-able [\#256](https://github.com/marklogic/marklogic-data-hub/issues/256)
+- mlcp uri replace is busted when run from cygwin [\#255](https://github.com/marklogic/marklogic-data-hub/issues/255)
+- Enable tracing by default [\#254](https://github.com/marklogic/marklogic-data-hub/issues/254)
+- Clear modules / Redeploy [\#252](https://github.com/marklogic/marklogic-data-hub/issues/252)
+- quickstart should be deployable to an appserver under a subdir [\#249](https://github.com/marklogic/marklogic-data-hub/issues/249)
+- Install status window too narrow [\#248](https://github.com/marklogic/marklogic-data-hub/issues/248)
+- gradle tasks need to force deploy content [\#246](https://github.com/marklogic/marklogic-data-hub/issues/246)
+- Running MLCP on non-existing flow causes error [\#243](https://github.com/marklogic/marklogic-data-hub/issues/243)
+- Loading GlobalCorp dataset results in deadlocks [\#225](https://github.com/marklogic/marklogic-data-hub/issues/225)
+- Deploy modules button should clear out the modules first [\#214](https://github.com/marklogic/marklogic-data-hub/issues/214)
+- Better error handling for failed hub install [\#176](https://github.com/marklogic/marklogic-data-hub/issues/176)
+
+## [v1.0.0-rc.2](https://github.com/marklogic/marklogic-data-hub/tree/v1.0.0-rc.2) (2016-08-10)
 [Full Changelog](https://github.com/marklogic/marklogic-data-hub/compare/v1.0.0-rc.1...v1.0.0-rc.2)
 
 **Closed issues:**

examples/healthcare/README.md

@@ -21,9 +21,13 @@ The sample data is located in the input/ folder.
 
 1. Open your web browser to [http://localhost:8080](http://localhost:8080).
 
-1. Point the Login box to your MarkLogic installation.
+1. Browse to this folder from the login screen.
 
-1. Deploy your modules by pressing the **Deploy Modules** button.
+1. Initialize the project (if necessary)
+
+1. Login with your MarkLogic credentials
+
+1. Install the Hub into MarkLogic (if necessary)
 
 1. Load hl7 data by pressing the **Load Data** button next to hl7. When prompted Set the Path to **input/hl7**. Set the collection to **hl7**. Set the Data Format to **Documents**. Now Press **Submit**.
 

examples/load-binaries/README.md

@@ -0,0 +1,67 @@
+# Example Binary Loading
+This example shows how to load binary documents with the Hub Framework.
+
+# TLDR; How do I run it?
+1. Download the latest quick-start jar from the [releases page](https://github.com/marklogic/marklogic-data-hub/releases) into this folder.
+
+1. Run the quick-start jar `java -jar quick-start.jar`
+
+1. Open your web browser to [http://localhost:8080](http://localhost:8080).
+
+1. Browse to this folder from the login screen.
+
+1. Initialize the project (if necessary)
+
+1. Login with your MarkLogic credentials
+
+1. Install the Hub into MarkLogic (if necessary)
+
+1. Load the sample pdf.
+
+  1. Click on the Guides entity on the left.
+  1. Click on the "LoadAsXml" input flow.
+  1. Click on the "Run Flow" button.
+  1. Browse to the input folder.
+  1. Expand the "General Options" section.
+  1. Add ,\\.pdf,'.xml' to the end of "Output URI Replace". It should look something like: /Users/yourname/data-hub/examples/load-binaries,'','\\.pdf','.xml'
+  1. Change "Document Type" to "binary".
+  1. Scroll down and press the "RUN IMPORT" button.
+
+1. At this point you have loaded the sample data. You can browse the data via [QConsole](http://localhost:8000/qconsole) or by searching the REST endpoint on the Staging Http Server [http://localhost:8010/v1/search](http://localhost:8010/v1/search). *Your port may be different if you changed it during setup*
+
+# Ingesting Binaries
+
+When you ingest a binary via the Quick Start MLCP process you will want to do a few things.
+
+1. Change the file extension to xml or json via "Output URI Replace".
+
+This is because we will be manually storing the binary an using the ingest for creating XML or JSON metadata.
+
+1. Change the document type to binary via the "Document Type" option. This is necessary because MLCP might think this document is an XML or JSON after we changed the file extension above.
+
+1. Store the binary manually and return the metatada. In your content.xqy put this:
+
+```xquery
+declare function plugin:create-content(
+  $id as xs:string,
+  $raw-content as node()?,
+  $options as map:map) as node()?
+{
+  (: name the binary uri with a pdf extension :)
+  let $binary-uri := fn:replace($id, ".xml", ".pdf")
+
+  (: stash the binary uri in the options map for later:)
+  let $_ := map:put($options, 'binary-uri', $binary-uri)
+
+  (: save the incoming binary as a pdf :)
+  return
+    xdmp:document-insert($binary-uri, $raw-content),
+
+  (: extract the contents of the pdf and return them
+   : as the content for the envelope
+   :)
+  xdmp:document-filter($raw-content)
+};
+```
+
+Now you have loaded the binary manually and returned xhtml content to store in the envelope.

examples/load-binaries/input/qconsole.pdf

@@ -0,0 +1,8 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!--This file is autogenerated. Please don't edit.-->
+<flow xmlns="http://marklogic.com/data-hub">
+  <complexity>simple</complexity>
+  <data-format>application/xml</data-format>
+  <plugins/>
+</flow>
+

examples/load-binaries/plugins/entities/Guides/input/LoadAsXml/LoadAsXml.xml

@@ -0,0 +1,37 @@
+xquery version "1.0-ml";
+
+module namespace plugin = "http://marklogic.com/data-hub/plugins";
+
+declare namespace envelope = "http://marklogic.com/data-hub/envelope";
+
+declare option xdmp:mapping "false";
+
+(:~
+ : Create Content Plugin
+ :
+ : @param $id          - the identifier returned by the collector
+ : @param $raw-content - the raw content being loaded.
+ : @param $options     - a map containing options. Options are sent from Java
+ :
+ : @return - your transformed content
+ :)
+declare function plugin:create-content(
+  $id as xs:string,
+  $raw-content as node()?,
+  $options as map:map) as node()?
+{
+  (: name the binary uri with a pdf extension :)
+  let $binary-uri := fn:replace($id, ".xml", ".pdf")
+
+  (: stash the binary uri in the options map for later:)
+  let $_ := map:put($options, 'binary-uri', $binary-uri)
+
+  (: save the incoming binary as a pdf :)
+  return
+    xdmp:document-insert($binary-uri, $raw-content),
+
+  (: extract the contents of the pdf and return them
+   : as the content for the envelope
+   :)
+  xdmp:document-filter($raw-content)
+};

examples/load-binaries/plugins/entities/Guides/input/LoadAsXml/content/content.xqy

@@ -0,0 +1,27 @@
+xquery version "1.0-ml";
+
+module namespace plugin = "http://marklogic.com/data-hub/plugins";
+
+declare namespace envelope = "http://marklogic.com/data-hub/envelope";
+
+declare option xdmp:mapping "false";
+
+(:~
+ : Create Headers Plugin
+ :
+ : @param $id      - the identifier returned by the collector
+ : @param $content - the output of your content plugin
+ : @param $options - a map containing options. Options are sent from Java
+ :
+ : @return - zero or more header nodes
+ :)
+declare function plugin:create-headers(
+  $id as xs:string,
+  $content as node()?,
+  $options as map:map) as node()*
+{
+  (: put the binary uri in the header of the envelope :)
+  (
+    <binary-uri>{map:get($options, "binary-uri")}</binary-uri>
+  )
+};

examples/load-binaries/plugins/entities/Guides/input/LoadAsXml/headers/headers.xqy

@@ -0,0 +1,26 @@
+xquery version "1.0-ml";
+
+module namespace plugin = "http://marklogic.com/data-hub/plugins";
+
+declare namespace envelope = "http://marklogic.com/data-hub/envelope";
+
+declare option xdmp:mapping "false";
+
+(:~
+ : Create Triples Plugin
+ :
+ : @param $id      - the identifier returned by the collector
+ : @param $content - the output of your content plugin
+ : @param $headers - the output of your headers plugin
+ : @param $options - a map containing options. Options are sent from Java
+ :
+ : @return - zero or more triples
+ :)
+declare function plugin:create-triples(
+  $id as xs:string,
+  $content as node()?,
+  $headers as node()*,
+  $options as map:map) as sem:triple*
+{
+  ()
+};

examples/load-binaries/plugins/entities/Guides/input/LoadAsXml/triples/triples.xqy

@@ -30,6 +30,7 @@ dependencies {
     testCompile 'org.springframework.batch:spring-batch-test:3.0.6.RELEASE'
     testCompile 'junit:junit:4.12'
     testCompile 'xmlunit:xmlunit:1.3'
+    testCompile 'org.skyscreamer:jsonassert:1.3.0'
     testCompile 'org.hamcrest:hamcrest-junit:2.0.0.0'
     testCompile 'org.easymock:easymock:3.4'
 }

marklogic-data-hub/build.gradle

@@ -33,6 +33,6 @@ mlSchemasDbName=data-hub-SCHEMAS
 
 hubModulesPath=examples/hr-hub/plugins
 
-version=1.0.0-rc.2
+version=1.0.0-rc.3