Do you have plan to support ProtocolBuffer serialization in pangool?

[dongpf]

Hi, @pereferrera @epalace @ivanprado

Right now, pangool has three serialization way, including hadoop, protostuff and thrift serializations, but does not support protobuffer, which is also widely used.

Do you have plans to add protobuffer serialization in pangool in the near future?

[ivanprado]

Hi Pegfei Dong,

Do you need support for ProtocolBuffers in Pangool? Although it is not yet
implemented, it is possible to develop ProtocolBuffers support in Pangool.
Would you like to contribute an implementation? We can give you some
guidance ...

Anyway, it should be possible to use elephant-bird Hadoop integration
https://github.com/kevinweil/elephant-bird with Pangool. Because this class
is Writable (
https://github.com/kevinweil/elephant-bird/blob/master/core/src/main/java/com/twitter/elephantbird/mapreduce/io/ProtobufWritable.java)
it would work perfectly inside Pangool.

2013/1/23 Pengfei Dong notifications@github.com

Hi, @pereferrera https://github.com/pereferrera @epalacehttps://github.com/epalace
@ivanprado https://github.com/ivanprado

Right now, pangool has three serialization way, including hadoop,
protostuff and thrift serializations, but does not support protobuffer,
which is also widely used.

Do you have plans to add protobuffer serialization in pangool in the near
future?

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/27.

Iván de Prado
CEO & Co-founder
www.datasalt.com

[dongpf]

Thanks for your guidance!
I will try elephant-bird ProtobufWritable in some tests.
But I think add a builtin Protobuf Serialization support in pangool is even better.
I'd like to contribute an implementation.

[pereferrera]

Perfect! We look forward to contributions!

[ivanprado]

There are two ways of adding new serialization to Pangool:

By adding the serialization support to Hadoop itself.

See the following URL for an example:
https://github.com/datasalt/pangool/blob/master/core/src/main/java/com/datasalt/pangool/serialization/ThriftSerialization.java

This kind of serialization is only possible the whole family of objects for which this serialization will apply inherits from a common ancestor, and this ancestor can be use to identify the objects. For example, in the case of Thrift that can be done because all Thirft objects inherits from TBase.

One drawback of that kind of serialization, is that you cannot specify specific configuration per field. Only can be configured globally through Hadoop Configuration. That can be a problem, as can be seen in #19

By creating a new custom field serialization.

See http://pangool.net/userguide/custom_serialization.html
Also you can check the class https://github.com/datasalt/pangool/blob/master/core/src/main/java/com/datasalt/pangool/tuplemr/serialization/AvroFieldSerialization.java
And an example of use in a test case: https://github.com/datasalt/pangool/blob/master/core/src/test/java/com/datasalt/pangool/BaseTest.java

This kind of serialization is Pangool specific, and has the advantage that can be associated with every field at the schema configuration. That means two things:

1. You can associate field specific metadata. For example, one field could be serialized using ProtocolBuffers using the binary format, but other field could be serialized with the same serializer but using other format.
2. The metadata is store withing the schema, so when you read a written file using this schema, the Tuple will be deserialized properly (would solve the Thrift using different serialization protocols #19 problem).

The idea would be to have some properties to configure which binary format to use.

ProtocolBuffers

The following class could serve as guidance to see how to serialize/deserialize ProtocolBuffers https://github.com/kevinweil/elephant-bird/blob/master/core/src/main/java/com/twitter/elephantbird/mapreduce/io/BinaryWritable.java

#29 is related with this ticket, as #29 could be used to be more efficient when writing PBuf objects as buffer size would be provided. By now, let's ignore that #29 will be present in the future.

Recommendation

I recommend to implement a field specific serializer for ProtocolBuffers (Option 2). The idea would be to submit a new class with the serializer, and a corresponding test case assuring that the it is working.