Skip to content

Add the Jelly output format #258

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 4 commits into from
Jun 26, 2025
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 3 additions & 0 deletions .gitattributes
Original file line number Diff line number Diff line change
@@ -1 +1,4 @@
docs/apidocs/* binary

# Mark Jelly files as binary
*.jelly binary
59 changes: 31 additions & 28 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -56,7 +56,7 @@ and see [Usage](#cli) on how to use the commandline interface!
- For examples on how to use functions within RML mapping documents, you can have a look at the [RML+FnO test cases](https://github.com/RMLio/rml-fno-test-cases)
- configuration file
- metadata generation
- output formats: nquads (default), turtle, trig, trix, jsonld, hdt
- output formats: nquads (default), turtle, trig, trix, jsonld, hdt, [jelly](https://w3id.org/jelly)
- join conditions
- targets:
- local file
Expand Down Expand Up @@ -104,7 +104,7 @@ The following options are most common.

- `-m, --mapping <arg>`: one or more mapping file paths and/or strings (multiple values are concatenated).
- `-o, --output <arg>`: path to output file
- `-s,--serialization <arg>`: serialization format (nquads (default), trig, trix, jsonld, hdt)
- `-s,--serialization <arg>`: serialization format (nquads (default), trig, trix, jsonld, hdt, [jelly](https://w3id.org/jelly))

All options can be found when executing `java -jar rmlmapper.jar --help`,
that output is found below.
Expand Down Expand Up @@ -148,7 +148,7 @@ options:
passwords, certificates, etc.
-s,--serialization <arg> serialization format (nquads
(default), turtle, trig, trix,
jsonld, hdt)
jsonld, hdt, jelly)
--strict Enable strict mode. In strict mode,
the mapper will fail on invalid IRIs
instead of skipping them.
Expand Down Expand Up @@ -307,31 +307,34 @@ The tests will fail otherwise, as Testcontainers can't spin up the container.

## Dependencies

| Dependency | License |
|:----------------------------------------------:|--------------------------------------------------------------------|
| ch.qos.logback logback-classic | Eclipse Public License 1.0 & GNU Lesser General Public License 2.1 |
| com.github.fnoio function-agent-java | MIT |
| com.github.fnoio grel-functions-java | MIT |
| com.github.fnoio idlab-functions-java | MIT |
| com.github.rdfhdt hdt-java | GNU Lesser General Public License v3.0 |
| com.github.tomakehurst:wiremock-jre8 | Apache License 2.0 |
| com.microsoft.sqlserver mssql-jdbc | MIT |
| com.mysql mysql-connector-java | GNU General Public License v2.0 |
| com.oracle.database.jdbc:ojdbc11 | Oracle Free Use Terms and Conditions |
| net.minidev json-smart | Apache License 2.0 |
| org.apache.jena fuseki-main | Apache License 2.0 |
| org.eclipse.rdf4j rdf4j-client | Eclipse Distribution License v1.0 |
| org.junit.jupiter junit-jupiter-api | Eclipse Public License v2.0 |
| org.junit.jupiter junit-jupiter-engine | Eclipse Public License v2.0 |
| org.junit.jupiter junit-jupiter-params | Eclipse Public License v2.0 |
| org.junit.vintage junit-vintage-engine | Eclipse Public License v2.0 |
| org.postgresql postgresql | BSD |
| org.testcontainers jdbc | MIT |
| org.testcontainers junit-jupiter | MIT |
| org.testcontainers mssqlserver | MIT |
| org.testcontainers mysql | MIT |
| org.testcontainers oracle-xe | MIT |
| org.testcontainers postgresql | MIT |
| Dependency | License |
|:--------------------------------------:|--------------------------------------------------------------------|
| ch.qos.logback logback-classic | Eclipse Public License 1.0 & GNU Lesser General Public License 2.1 |
| com.github.fnoio function-agent-java | MIT |
| com.github.fnoio grel-functions-java | MIT |
| com.github.fnoio idlab-functions-java | MIT |
| com.github.rdfhdt hdt-java | GNU Lesser General Public License v3.0 |
| com.github.tomakehurst:wiremock-jre8 | Apache License 2.0 |
| com.google.protobuf protobuf-java | BSD 3-clause |
| com.microsoft.sqlserver mssql-jdbc | MIT |
| com.mysql mysql-connector-java | GNU General Public License v2.0 |
| com.oracle.database.jdbc:ojdbc11 | Oracle Free Use Terms and Conditions |
| eu.neverblink.jelly jelly-core | Apache License 2.0 |
| eu.neverblink.jelly jelly-rdf4j | Apache License 2.0 |
| net.minidev json-smart | Apache License 2.0 |
| org.apache.jena fuseki-main | Apache License 2.0 |
| org.eclipse.rdf4j rdf4j-client | Eclipse Distribution License v1.0 |
| org.junit.jupiter junit-jupiter-api | Eclipse Public License v2.0 |
| org.junit.jupiter junit-jupiter-engine | Eclipse Public License v2.0 |
| org.junit.jupiter junit-jupiter-params | Eclipse Public License v2.0 |
| org.junit.vintage junit-vintage-engine | Eclipse Public License v2.0 |
| org.postgresql postgresql | BSD |
| org.testcontainers jdbc | MIT |
| org.testcontainers junit-jupiter | MIT |
| org.testcontainers mssqlserver | MIT |
| org.testcontainers mysql | MIT |
| org.testcontainers oracle-xe | MIT |
| org.testcontainers postgresql | MIT |

## Commercial Support

Expand Down
5 changes: 2 additions & 3 deletions buildNumber.properties
Original file line number Diff line number Diff line change
@@ -1,4 +1,3 @@
#maven.buildNumber.plugin properties file
#Tue Aug 13 08:29:04 GMT 2024
buildNumber0=373

#Sun Jun 08 18:28:21 CEST 2025
buildNumber0=378
12 changes: 12 additions & 0 deletions pom.xml
Original file line number Diff line number Diff line change
Expand Up @@ -39,6 +39,7 @@
<maven.compiler.source>17</maven.compiler.source>
<maven.compiler.target>17</maven.compiler.target>
<jena.version>5.0.0</jena.version>
<jelly.version>3.2.0</jelly.version>
<testcontainers.version>1.18.3</testcontainers.version>
</properties>

Expand Down Expand Up @@ -263,6 +264,17 @@
</exclusion>
</exclusions>
</dependency>
<dependency>
<groupId>eu.neverblink.jelly</groupId>
<artifactId>jelly-rdf4j</artifactId>
<version>${jelly.version}</version>
<exclusions> <!-- Exclude RDF4J, use the version from the rdf4j-client dependency -->
<exclusion>
<groupId>org.eclipse.rdf4j</groupId>
<artifactId>*</artifactId>
</exclusion>
</exclusions>
</dependency>
<dependency>
<groupId>be.ugent.idlab.knows</groupId>
<artifactId>function-agent-java</artifactId>
Expand Down
15 changes: 5 additions & 10 deletions src/main/java/be/ugent/rml/cli/Main.java
Original file line number Diff line number Diff line change
Expand Up @@ -127,7 +127,7 @@ public static void run(String[] args, String basePath) throws Exception {
.build();
Option serializationFormatOption = Option.builder("s")
.longOpt("serialization")
.desc("serialization format (nquads (default), turtle, trig, trix, jsonld, hdt)")
.desc("serialization format (nquads (default), turtle, trig, trix, jsonld, hdt, jelly)")
.hasArg()
.build();
Option jdbcDSNOption = Option.builder("dsn")
Expand Down Expand Up @@ -545,12 +545,7 @@ private static void writeOutputTargets(Map<Term, QuadStore> targets, QuadStore r
String serializationFormat = target.getSerializationFormat();
OutputStream output = target.getOutputStream();
store.addQuads(target.getMetadata());

// Set character encoding
try (Writer out = new BufferedWriter(new OutputStreamWriter(output, Charset.defaultCharset()))) {
// Write store to target
store.write(out, serializationFormat);
}
store.write(output, serializationFormat);
// Close OS resources
target.close();
logger.debug("Exporting to Target: {}", target);
Expand Down Expand Up @@ -637,7 +632,7 @@ private static File writeOutputUncompressed(QuadStore store, String outputFile,
logger.info("{} quad was generated for default Target", store.size());
}

Writer out = null;
OutputStream out = null;
try {

String doneMessage = null;
Expand All @@ -653,11 +648,11 @@ private static File writeOutputUncompressed(QuadStore store, String outputFile,

doneMessage = "Writing to " + targetFile.getPath() + " is done.";

out = Files.newBufferedWriter(targetFile.toPath(), StandardCharsets.UTF_8);
out = Files.newOutputStream(targetFile.toPath());

} else {
isSystemOut = true;
out = new BufferedWriter(new OutputStreamWriter(System.out, StandardCharsets.UTF_8));
out = System.out;
}

store.write(out, format);
Expand Down
24 changes: 13 additions & 11 deletions src/main/java/be/ugent/rml/store/QuadStore.java
Original file line number Diff line number Diff line change
Expand Up @@ -127,29 +127,31 @@ public abstract class QuadStore {
public abstract void read(InputStream is, String base, RDFFormat format) throws Exception;

/**
* Write out the QuadStore in given format
* TODO use class or enum for output format
* Write out the QuadStore in given format.
* Deprecated, use write(OutputStream out, String format) instead.
* TODO remove this method in future versions, use write(OutputStream out, String format) instead.
*
* @param out Writer output location
* @param format QuadStore format (.TTL)
* @throws Exception
*/
@Deprecated
public abstract void write(Writer out, String format) throws Exception;

// END OF ABSTRACT METHODS

// following final methods use the abstract methods to provide additional functionality or helper functions
/**
* Helper function
* Write out the QuadStore in given format, using a binary output stream.
* Override this method if you want to support binary formats like Jelly.
* TODO use class or enum for output format
*
* @param out
* @param format
* @param out OutputStream to write to
* @param format QuadStore format (.TTL)
* @throws Exception
*/
public final void write(ByteArrayOutputStream out, String format) throws Exception {
write(new BufferedWriter(new OutputStreamWriter(out)), format);
}
public abstract void write(OutputStream out, String format) throws Exception;

// END OF ABSTRACT METHODS

// following final methods use the abstract methods to provide additional functionality or helper functions
/**
* Helper function
*
Expand Down
24 changes: 22 additions & 2 deletions src/main/java/be/ugent/rml/store/RDF4JStore.java
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,9 @@
import be.ugent.rml.term.Literal;
import be.ugent.rml.term.NamedNode;
import be.ugent.rml.term.Term;
import eu.neverblink.jelly.convert.rdf4j.rio.JellyFormat;
import eu.neverblink.jelly.convert.rdf4j.rio.JellyWriterSettings;
import eu.neverblink.jelly.core.JellyOptions;
import org.eclipse.rdf4j.model.*;
import org.eclipse.rdf4j.model.impl.*;
import org.eclipse.rdf4j.model.util.Models;
Expand All @@ -15,8 +18,7 @@
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;

import java.io.InputStream;
import java.io.Writer;
import java.io.*;
import java.util.ArrayList;
import java.util.List;
import java.util.Optional;
Expand Down Expand Up @@ -172,11 +174,29 @@ public void write(Writer out, String format) throws Exception {
case "ntriples":
Rio.write(model, out, RDFFormat.NTRIPLES);
break;
case "jelly":
throw new UnsupportedOperationException(
"Writing Jelly format to a Writer is not supported. Use OutputStream instead."
);
default:
throw new Exception("Serialization " + format + " not supported");
}
}

@Override
public void write(OutputStream out, String format) throws Exception {
// Jelly is the only format that requires a binary output stream
if (format.equals("jelly")) {
var settings = JellyWriterSettings.empty();
// Mark RDF-star as not used
settings.setJellyOptions(JellyOptions.BIG_STRICT);
Rio.write(model, out, JellyFormat.JELLY, settings);
} else {
// For all other formats, fall back to using the Writer
write(new BufferedWriter(new OutputStreamWriter(out)), format);
}
}

public String getBase() {
Optional<Namespace> base = model.getNamespace("");
if (base.isPresent()) {
Expand Down
5 changes: 5 additions & 0 deletions src/main/java/be/ugent/rml/store/SimpleQuadStore.java
Original file line number Diff line number Diff line change
Expand Up @@ -118,6 +118,11 @@ public void write(Writer out, String format) throws IOException {
}
}

@Override
public void write(OutputStream out, String format) throws IOException {
write(new BufferedWriter(new OutputStreamWriter(out)), format);
}

@Override
public boolean equals(Object o) {
throw new UnsupportedOperationException("Method not implemented.");
Expand Down
22 changes: 22 additions & 0 deletions src/test/java/be/ugent/rml/ArgumentsTest.java
Original file line number Diff line number Diff line change
Expand Up @@ -378,6 +378,28 @@ public void outputHDT() throws Exception {
}
}

@Test
public void outputJelly() throws Exception {
String cwd = Utils.getFile("argument").getAbsolutePath();
String mappingFilePath = (new File(cwd, "mapping.ttl")).getAbsolutePath();
String actualJellyPath = (new File("./generated_output.jelly")).getAbsolutePath();
String expectedJellyPath = Utils.getFile( "argument/output-jelly/target_output.jelly").getAbsolutePath();

Main.run(new String[]{"-m" , mappingFilePath , "-o" , actualJellyPath , "-s", "jelly"}, cwd);
compareFiles(
expectedJellyPath,
actualJellyPath,
false
);

try {
File outputFile = Utils.getFile(actualJellyPath);
assertTrue(outputFile.delete());
} catch (Exception e) {
e.printStackTrace();
}
}


@Test
public void quoteInLiteral() throws Exception {
Expand Down
3 changes: 3 additions & 0 deletions src/test/java/be/ugent/rml/TestCore.java
Original file line number Diff line number Diff line change
Expand Up @@ -12,6 +12,7 @@
import be.ugent.rml.target.TargetFactory;
import be.ugent.rml.term.NamedNode;
import be.ugent.rml.term.Term;
import eu.neverblink.jelly.convert.rdf4j.rio.JellyFormat;
import org.eclipse.rdf4j.rio.RDFFormat;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
Expand Down Expand Up @@ -351,6 +352,8 @@ protected QuadStore filePathToStore(String path) throws Exception {
store = QuadStoreFactory.read(outputFile, RDFFormat.JSONLD);
} else if (path.endsWith(".trig")) {
store = QuadStoreFactory.read(outputFile, RDFFormat.TRIG);
} else if (path.endsWith(".jelly")) {
store = QuadStoreFactory.read(outputFile, JellyFormat.JELLY);
} else {
store = QuadStoreFactory.read(outputFile);
}
Expand Down
3 changes: 1 addition & 2 deletions src/test/java/be/ugent/rml/readme/ReadmeFunctionTest.java
Original file line number Diff line number Diff line change
Expand Up @@ -49,8 +49,7 @@ public void function() {
QuadStore result = executor.execute(null).get(new NamedNode("rmlmapper://default.store"));

// Output the result
BufferedWriter out = new BufferedWriter(new OutputStreamWriter(System.out));
result.write(out, "turtle");
result.write(System.out, "turtle");
} catch (Exception e) {
fail("No exception was expected.");
}
Expand Down
3 changes: 1 addition & 2 deletions src/test/java/be/ugent/rml/readme/ReadmeTest.java
Original file line number Diff line number Diff line change
Expand Up @@ -44,8 +44,7 @@ public void standard() {
QuadStore result = executor.execute(null).get(new NamedNode("rmlmapper://default.store"));

// Output the result
BufferedWriter out = new BufferedWriter(new OutputStreamWriter(System.out));
result.write(out, "turtle");
result.write(System.out, "turtle");
} catch (Exception e) {
fail("No exception was expected.");
}
Expand Down
21 changes: 21 additions & 0 deletions src/test/resources/argument/output-jelly/target_output.jelly
Original file line number Diff line number Diff line change
@@ -0,0 +1,21 @@
�

H�P�X px
Rhttp://example.com/10/
JVenus
Rhttp://xmlns.com/foaf/0.1/
Jname

*Z
Venusen
Rhttp://example.com/
Jid

*Z
10
/R-+http://www.w3.org/1999/02/22-rdf-syntax-ns#
Jtype

JPerson

*J