Skip to content

Commit 2cef9b9

Browse files
authored
In memory support (#27)
* updating latest release version * adding support for in memory buffer, changing tests to run once for in memory buffer and once for on disk buffer * adding check for buffer threshold size * make start function shorter * deleting abstract test + adding parameterized * naming buffer -> queueu, update readme * use new java sender(1.0.16) * change in printing messages * adding description to deprecated and taking some error prints out to higher level function * fix travis * refactoring tests + adding more tests for in-memory * change naming in some tests * adding log count limit + test * shade guava * add validation for log count * update readme * change term limit to capacity + fix tests naming
1 parent 50ea111 commit 2cef9b9

File tree

6 files changed

+399
-162
lines changed

6 files changed

+399
-162
lines changed

README.md

+24-7
Original file line numberDiff line numberDiff line change
@@ -6,7 +6,7 @@ Log4j 2 Appender that ships logs using HTTPS bulk
66
This appender sends logs to your [Logz.io](http://logz.io) account, using non-blocking threading, bulks, and HTTPS encryption. Please note that this appender requires log4j version 2.7 and up, and java 8 and up.
77

88
### Technical Information
9-
This appender uses [LogzioSender](https://github.com/logzio/logzio-java-sender) implementation. Once you send a log, it will be enqueued in the buffer and 100% non-blocking. There is a background task that will handle the log shipment for you. This jar is an "Uber-Jar" that shades both LogzioSender, BigQueue, Gson and Guava to avoid "dependency hell".
9+
This appender uses [LogzioSender](https://github.com/logzio/logzio-java-sender) implementation. Once you send a log, it will be enqueued in the queue and 100% non-blocking. There is a background task that will handle the log shipment for you. This jar is an "Uber-Jar" that shades both LogzioSender, BigQueue, Gson and Guava to avoid "dependency hell".
1010

1111
### Installation from maven
1212
```xml
@@ -40,16 +40,31 @@ This appender uses [LogzioSender](https://github.com/logzio/logzio-java-sender)
4040
| **logzioToken** | *None* | Your Logz.io token, which can be found under "settings" in your account, If the value begins with `$` then the appender looks for an environment variable or system property with the name specified. For example: `$LOGZIO_TOKEN` will look for environment variable named `LOGZIO_TOKEN` |
4141
| **logzioType** | *java* | The [log type](https://support.logz.io/hc/en-us/articles/209486049-What-is-Type-) for that appender, it must not contain any spaces |
4242
| **logzioUrl** | *https://listener.logz.io:8071* | The url that the appender sends to. If your account is in the EU you must use https://listener-eu.logz.io:8071 |
43-
| **drainTimeoutSec** | *5* | How often the appender should drain the buffer (in seconds) |
44-
| **fileSystemFullPercentThreshold** | *98* | The percent of used file system space at which the appender will stop buffering. When we will reach that percentage, the file system in which the buffer is stored will drop all new logs until the percentage of used space drops below that threshold. Set to -1 to never stop processing new logs |
45-
| **bufferDir** | *System.getProperty("java.io.tmpdir")* | Where the appender should store the buffer |
43+
| **drainTimeoutSec** | *5* | How often the appender should drain the queue (in seconds) |
4644
| **socketTimeoutMs** | *10 * 1000* | The socket timeout during log shipment |
4745
| **connectTimeoutMs** | *10 * 1000* | The connection timeout during log shipment |
4846
| **addHostname** | *false* | Optional. If true, then a field named 'hostname' will be added holding the host name of the machine. If from some reason there's no defined hostname, this field won't be added |
4947
| **additionalFields** | *None* | Optional. Allows to add additional fields to the JSON message sent. The format is "fieldName1=fieldValue1;fieldName2=fieldValue2". You can optionally inject an environment variable value using the following format: "fieldName1=fieldValue1;fieldName2=$ENV_VAR_NAME". In that case, the environment variable should be the only value. In case the environment variable can't be resolved, the field will be omitted. |
5048
| **debug** | *false* | Print some debug messages to stdout to help to diagnose issues |
5149
| **compressRequests** | *false* | Boolean. `true` if logs are compressed in gzip format before sending. `false` if logs are sent uncompressed. |
5250

51+
#### Parameters for in-memory queue
52+
| Parameter | Default | Explained |
53+
| ------------------ | ------------------------------------ | ----- |
54+
| **inMemoryQueueCapacityBytes** | *1024 * 1024 * 100* | The amount of memory(bytes) we are allowed to use for the memory queue. If the value is -1 the sender will not limit the queue size.|
55+
| **inMemoryLogsCountCapacity** | *-1* | Number of logs we are allowed to have in the queue before dropping logs. If the value is -1 the sender will not limit the number of logs allowed.|
56+
| **inMemoryQueue** | *false* | Set to true if the appender uses in memory queue. By default the appender uses disk queue|
57+
58+
59+
#### Parameters for disk queue
60+
| Parameter | Default | Explained |
61+
| ------------------ | ------------------------------------ | ----- |
62+
| **fileSystemFullPercentThreshold** | *98* | The percent of used file system space at which the sender will stop queueing. When we will reach that percentage, the file system in which the queue is stored will drop all new logs until the percentage of used space drops below that threshold. Set to -1 to never stop processing new logs |
63+
| **gcPersistedQueueFilesIntervalSeconds** | *30* | How often the disk queue should clean sent logs from disk |
64+
| **bufferDir**(deprecated, use queueDir) | *System.getProperty("java.io.tmpdir")* | Where the appender should store the queue |
65+
| **queueDir** | *System.getProperty("java.io.tmpdir")* | Where the appender should store the queue |
66+
67+
5368

5469
### Code Example
5570
```java
@@ -78,7 +93,7 @@ public class LogzioLog4j2Example {
7893

7994
public static void main(String[] args) {
8095
Logger logger = LogManager.getLogger(LogzioLog4j2Example.class);
81-
96+
8297
ThreadContext.put("Key", "Value");
8398
logger.info("This log will hold the MDC data as well");
8499
}
@@ -106,7 +121,6 @@ public class LogzioLog4j2Example {
106121

107122
public static void main(String[] args) {
108123
Logger logger = LogManager.getLogger(LogzioLog4j2Example.class);
109-
110124
Marker marker = MarkerManager.getMarker("Fatal");
111125
logger.error(marker, "This line has a fatal error");
112126
}
@@ -123,8 +137,11 @@ Will send a log to Logz.io that looks like this:
123137
```
124138

125139
### Release notes
140+
- 1.0.11
141+
- add in memory queue option
142+
- change bufferDir(deprecated) to queueDir
126143
- 1.0.10
127-
- Allow to set type using enviroment variables.
144+
- Allow to set type using environment variables.
128145
- 1.0.9
129146
- fix guava shaded dependency.
130147
- 1.0.8

pom.xml

+13-1
Original file line numberDiff line numberDiff line change
@@ -36,7 +36,7 @@
3636
</scm>
3737

3838
<properties>
39-
<logzio-sender-version>1.0.11</logzio-sender-version>
39+
<logzio-sender-version>1.0.17</logzio-sender-version>
4040
<log4j2-version>2.8.2</log4j2-version>
4141
</properties>
4242

@@ -119,9 +119,16 @@
119119
<artifactSet>
120120
<includes>
121121
<include>io.logz.sender:logzio-sender</include>
122+
<include>com.google.guava:guava</include>
122123
</includes>
123124
</artifactSet>
124125
<createDependencyReducedPom>true</createDependencyReducedPom>
126+
<relocations>
127+
<relocation>
128+
<pattern>com.google.common</pattern>
129+
<shadedPattern>io.logz.log4j2appender.com.google.common</shadedPattern>
130+
</relocation>
131+
</relocations>
125132
<filters>
126133
<filter>
127134
<artifact>*:*</artifact>
@@ -165,5 +172,10 @@
165172
<version>${logzio-sender-version}</version>
166173
<scope>test</scope>
167174
</dependency>
175+
<dependency>
176+
<groupId>com.google.guava</groupId>
177+
<artifactId>guava</artifactId>
178+
<version>25.0-jre</version>
179+
</dependency>
168180
</dependencies>
169181
</project>

0 commit comments

Comments
 (0)