Partial message support

[danielmotaleite]

From what i see, your support for partial messages is to just forward the partial flag to the elasticsearch... but that way is not easy to rebuild the full message, even in logstash.

IMHO, it would be better that you could flat the log plugin to rebuild the messages itself by merging the partial messages before sending to the elasticsearch (or logstash). This would also help later parse the multiline as all the logs would be complete

Of course, by doing this, you should put another variable to limit the size of partial messages, go avoid a runaway log that have no newline from eating all the ram trying to find the final log event

So i suggest 2 new options:

--log-opt merge-partial=true  (defaults should be false)
--log-opt merge-limit=1M      (default should be maybe 64k, 3 partial + final 16k docker size log)

[rchicoli]

Great idea @danielmotaleite ! Somehow I need to add some labels and prioritize what will be done first before going stable. I will add this to the list. Cool.

[megastef]

+1 for a better Elasticsearch driver in golang

Nice plugin, i see on your roadmap many features, which are already in https://github.com/sematext/sematext-agent-docker - it supports Elasticsearch bulk indexing, parses multiline logs, recognizes many application logs OOTB and enriches logs with labels and environment with special support for Swarm and Kuberntes environments. Most of the features have been developed on customer demand.