Skip to content

Commit 4304248

Browse files
wanghangyu817wanghangyu817
authored
Merge branch 'zendesk:master' into master
2 parents 0baa746 + 88b36ed commit 4304248

File tree

12 files changed

+130
-82
lines changed

12 files changed

+130
-82
lines changed

‎CHANGELOG.md

Lines changed: 17 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,22 @@
11
# Maxwell changelog
22

3+
### [v1.39.4](https://github.com/zendesk/maxwell/releases/tag/v1.39.4)
4+
5+
- Fix bugs with older versions of mariadb (<10.4)
6+
7+
8+
9+
_Released 2022-12-07_
10+
11+
### [v1.39.3](https://github.com/zendesk/maxwell/releases/tag/v1.39.3)
12+
13+
- some bugfixes for 1.39.2 and google pubsub
14+
- couple of security upgrades, including in the docker image
15+
16+
17+
18+
_Released 2022-12-04_
19+
320
### [v1.39.2](https://github.com/zendesk/maxwell/releases/tag/v1.39.2)
421

522
this is a bug-fix release. some upgrades broke maxwell's http interface and there's

‎Dockerfile

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,5 @@
11
FROM maven:3.8-jdk-11 as builder
2-
ENV MAXWELL_VERSION=1.39.2 KAFKA_VERSION=1.0.0
2+
ENV MAXWELL_VERSION=1.39.4 KAFKA_VERSION=1.0.0
33

44
RUN apt-get update \
55
&& apt-get -y upgrade \

‎README.md

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -9,7 +9,7 @@ search indexing and inter-service communication. Maxwell gives you some of the
99
benefits of event sourcing without having to re-architect your entire platform.
1010

1111
<b>Download:</b><br>
12-
[https://github.com/zendesk/maxwell/releases/download/v1.39.2/maxwell-1.39.2.tar.gz](https://github.com/zendesk/maxwell/releases/download/v1.39.2/maxwell-1.39.2.tar.gz)
12+
[https://github.com/zendesk/maxwell/releases/download/v1.39.4/maxwell-1.39.4.tar.gz](https://github.com/zendesk/maxwell/releases/download/v1.39.4/maxwell-1.39.4.tar.gz)
1313
<br/>
1414
<b>Source:</b><br>
1515
[https://github.com/zendesk/maxwell](https://github.com/zendesk/maxwell)

‎docs/docs/quickstart.md

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -1,13 +1,13 @@
11
# Download
22
***
3-
- Download binary distro: [https://github.com/zendesk/maxwell/releases/download/v1.39.2/maxwell-1.39.2.tar.gz](https://github.com/zendesk/maxwell/releases/download/v1.39.2/maxwell-1.39.2.tar.gz)
3+
- Download binary distro: [https://github.com/zendesk/maxwell/releases/download/v1.39.4/maxwell-1.39.4.tar.gz](https://github.com/zendesk/maxwell/releases/download/v1.39.4/maxwell-1.39.4.tar.gz)
44
- Sources and bug tracking is available on github: [https://github.com/zendesk/maxwell](https://github.com/zendesk/maxwell)
55

66
**curl**:
77
```
8-
curl -sLo - https://github.com/zendesk/maxwell/releases/download/v1.39.2/maxwell-1.39.2.tar.gz \
8+
curl -sLo - https://github.com/zendesk/maxwell/releases/download/v1.39.4/maxwell-1.39.4.tar.gz \
99
| tar zxvf -
10-
cd maxwell-1.39.2
10+
cd maxwell-1.39.4
1111
```
1212

1313
**docker**:

‎pom.xml

Lines changed: 66 additions & 66 deletions
Original file line numberDiff line numberDiff line change
@@ -4,7 +4,7 @@
44

55
<groupId>com.zendesk</groupId>
66
<artifactId>maxwell</artifactId>
7-
<version>1.39.2</version>
7+
<version>1.39.4</version>
88
<packaging>jar</packaging>
99

1010
<name>maxwell</name>
@@ -225,70 +225,6 @@
225225
<version>0.6.0</version>
226226
</dependency>
227227

228-
<!-- metrics -->
229-
<dependency>
230-
<groupId>io.dropwizard.metrics</groupId>
231-
<artifactId>metrics-core</artifactId>
232-
<version>4.1.17</version>
233-
</dependency>
234-
<dependency>
235-
<groupId>io.dropwizard.metrics</groupId>
236-
<artifactId>metrics-servlets</artifactId>
237-
<version>4.1.17</version>
238-
</dependency>
239-
<dependency>
240-
<groupId>io.dropwizard.metrics</groupId>
241-
<artifactId>metrics-jmx</artifactId>
242-
<version>4.1.17</version>
243-
</dependency>
244-
<dependency>
245-
<groupId>org.eclipse.jetty</groupId>
246-
<artifactId>jetty-server</artifactId>
247-
<version>10.0.12</version>
248-
</dependency>
249-
<dependency>
250-
<groupId>org.eclipse.jetty</groupId>
251-
<artifactId>jetty-servlet</artifactId>
252-
<version>10.0.12</version>
253-
</dependency>
254-
<dependency>
255-
<groupId>com.viafoura</groupId>
256-
<artifactId>metrics-datadog</artifactId>
257-
<version>2.0.0-RC3</version>
258-
</dependency>
259-
<dependency>
260-
<groupId>io.prometheus</groupId>
261-
<artifactId>simpleclient_dropwizard</artifactId>
262-
<version>0.9.0</version>
263-
</dependency>
264-
<dependency>
265-
<groupId>io.prometheus</groupId>
266-
<artifactId>simpleclient_servlet</artifactId>
267-
<version>0.9.0</version>
268-
</dependency>
269-
<!-- OpenCensus used for pushing metrics to stackdriver -->
270-
<dependency>
271-
<groupId>io.opencensus</groupId>
272-
<artifactId>opencensus-contrib-dropwizard</artifactId>
273-
<version>${opencensus.version}</version>
274-
</dependency>
275-
<dependency>
276-
<groupId>io.opencensus</groupId>
277-
<artifactId>opencensus-api</artifactId>
278-
<version>${opencensus.version}</version>
279-
</dependency>
280-
<dependency>
281-
<groupId>io.opencensus</groupId>
282-
<artifactId>opencensus-impl</artifactId>
283-
<version>${opencensus.version}</version>
284-
<scope>runtime</scope>
285-
</dependency>
286-
<dependency>
287-
<groupId>io.opencensus</groupId>
288-
<artifactId>opencensus-exporter-stats-stackdriver</artifactId>
289-
<version>${opencensus.version}</version>
290-
</dependency>
291-
292228
<!-- producer libs -->
293229
<dependency>
294230
<groupId>com.google.cloud</groupId>
@@ -338,7 +274,7 @@
338274
<dependency>
339275
<groupId>com.google.cloud</groupId>
340276
<artifactId>google-cloud-pubsub</artifactId>
341-
<version>1.115.1</version>
277+
<version>1.120.24</version>
342278
</dependency>
343279
<dependency>
344280
<groupId>io.nats</groupId>
@@ -355,6 +291,70 @@
355291
<artifactId>jedis</artifactId>
356292
<version>3.5.1</version>
357293
</dependency>
294+
295+
<!-- metrics -->
296+
<dependency>
297+
<groupId>io.dropwizard.metrics</groupId>
298+
<artifactId>metrics-core</artifactId>
299+
<version>4.1.17</version>
300+
</dependency>
301+
<dependency>
302+
<groupId>io.dropwizard.metrics</groupId>
303+
<artifactId>metrics-servlets</artifactId>
304+
<version>4.1.17</version>
305+
</dependency>
306+
<dependency>
307+
<groupId>io.dropwizard.metrics</groupId>
308+
<artifactId>metrics-jmx</artifactId>
309+
<version>4.1.17</version>
310+
</dependency>
311+
<dependency>
312+
<groupId>org.eclipse.jetty</groupId>
313+
<artifactId>jetty-server</artifactId>
314+
<version>10.0.12</version>
315+
</dependency>
316+
<dependency>
317+
<groupId>org.eclipse.jetty</groupId>
318+
<artifactId>jetty-servlet</artifactId>
319+
<version>10.0.12</version>
320+
</dependency>
321+
<dependency>
322+
<groupId>com.viafoura</groupId>
323+
<artifactId>metrics-datadog</artifactId>
324+
<version>2.0.0-RC3</version>
325+
</dependency>
326+
<dependency>
327+
<groupId>io.prometheus</groupId>
328+
<artifactId>simpleclient_dropwizard</artifactId>
329+
<version>0.9.0</version>
330+
</dependency>
331+
<dependency>
332+
<groupId>io.prometheus</groupId>
333+
<artifactId>simpleclient_servlet</artifactId>
334+
<version>0.9.0</version>
335+
</dependency>
336+
<!-- OpenCensus used for pushing metrics to stackdriver -->
337+
<dependency>
338+
<groupId>io.opencensus</groupId>
339+
<artifactId>opencensus-contrib-dropwizard</artifactId>
340+
<version>${opencensus.version}</version>
341+
</dependency>
342+
<dependency>
343+
<groupId>io.opencensus</groupId>
344+
<artifactId>opencensus-api</artifactId>
345+
<version>${opencensus.version}</version>
346+
</dependency>
347+
<dependency>
348+
<groupId>io.opencensus</groupId>
349+
<artifactId>opencensus-impl</artifactId>
350+
<version>${opencensus.version}</version>
351+
<scope>runtime</scope>
352+
</dependency>
353+
<dependency>
354+
<groupId>io.opencensus</groupId>
355+
<artifactId>opencensus-exporter-stats-stackdriver</artifactId>
356+
<version>${opencensus.version}</version>
357+
</dependency>
358358

359359
<!-- test -->
360360
<dependency>

‎src/main/antlr4/imports/mysql_alter_database.g4

Lines changed: 3 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -5,5 +5,7 @@ import mysql_literal_tokens, mysql_idents;
55
alter_database: ALTER (DATABASE | SCHEMA) name? alter_database_definition;
66
alter_database_definition:
77
(default_character_set | default_collation)+
8-
| UPGRADE DATA DIRECTORY NAME;
8+
| UPGRADE DATA DIRECTORY NAME
9+
| alter_encryption;
910

11+
alter_encryption: ENCRYPTION '='? string_literal;

‎src/main/java/com/zendesk/maxwell/Maxwell.java

Lines changed: 3 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -353,13 +353,14 @@ public void run() {
353353
} catch ( URISyntaxException e ) {
354354
// catch URISyntaxException explicitly as well to provide more information to the user
355355
LOGGER.error("Syntax issue with URI, check for misconfigured host, port, database, or JDBC options (see RFC 2396)");
356-
LOGGER.error("URISyntaxException: " + e.getLocalizedMessage());
356+
LOGGER.error("URISyntaxException: " + e.getLocalizedMessage(), e);
357357
System.exit(1);
358358
} catch ( ServerException e ) {
359-
LOGGER.error("Maxwell couldn't find the requested binlog, exiting...");
359+
LOGGER.error("Maxwell couldn't find the requested binlog, exiting...", e);
360360
System.exit(2);
361361
} catch ( Exception e ) {
362362
e.printStackTrace();
363+
LOGGER.error("Maxwell saw an exception and is exiting...", e);
363364
System.exit(1);
364365
}
365366
}

‎src/main/java/com/zendesk/maxwell/MaxwellConfig.java

Lines changed: 4 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1110,7 +1110,7 @@ private void setup(OptionSet options, Properties properties) {
11101110
this.pubsubMessageOrderingKey = fetchStringOption("pubsub_message_ordering_key", options, properties, null);
11111111
this.pubsubPublishDelayThreshold = Duration.ofMillis(fetchLongOption("pubsub_publish_delay_threshold", options, properties, 1L));
11121112
this.pubsubRetryDelay = Duration.ofMillis(fetchLongOption("pubsub_retry_delay", options, properties, 100L));
1113-
this.pubsubRetryDelayMultiplier = fetchFloatOption("pubsub_retry_delay_multiplier", options, properties, 1.0f);
1113+
this.pubsubRetryDelayMultiplier = fetchFloatOption("pubsub_retry_delay_multiplier", options, properties, 1.3f);
11141114
this.pubsubMaxRetryDelay = Duration.ofSeconds(fetchLongOption("pubsub_max_retry_delay", options, properties, 60L));
11151115
this.pubsubInitialRpcTimeout = Duration.ofSeconds(fetchLongOption("pubsub_initial_rpc_timeout", options, properties, 5L));
11161116
this.pubsubRpcTimeoutMultiplier = fetchFloatOption("pubsub_rpc_timeout_multiplier", options, properties, 1.0f);
@@ -1379,6 +1379,9 @@ public void validate() {
13791379
} else if (this.producerType.equals("sns") && this.snsTopic == null) {
13801380
usageForOptions("please specify a topic ARN for SNS", "sns_topic");
13811381
} else if (this.producerType.equals("pubsub")) {
1382+
if (this.pubsubProjectId == null)
1383+
usageForOptions("please specify --pubsub_project_id.", "--pubsub_project_id");
1384+
13821385
if (this.pubsubRequestBytesThreshold <= 0L)
13831386
usage("--pubsub_request_bytes_threshold must be > 0");
13841387
if (this.pubsubMessageCountBatchSize <= 0L)

‎src/main/java/com/zendesk/maxwell/schema/SchemaCapturer.java

Lines changed: 23 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -10,6 +10,7 @@
1010
import org.slf4j.LoggerFactory;
1111

1212
import java.sql.Connection;
13+
import java.sql.DatabaseMetaData;
1314
import java.sql.PreparedStatement;
1415
import java.sql.ResultSet;
1516
import java.sql.Statement;
@@ -137,8 +138,9 @@ public Schema capture() throws SQLException {
137138

138139
Schema s = new Schema(databases, captureDefaultCharset(), this.sensitivity);
139140
try {
140-
if ( isMariaDB() )
141+
if ( isMariaDB() && mariaSupportsJSON()) {
141142
detectMariaDBJSON(s);
143+
}
142144
} catch ( InvalidSchemaError e ) {
143145
e.printStackTrace();
144146
}
@@ -176,17 +178,34 @@ private boolean isMySQLAtLeast56() throws SQLException {
176178
if ( isMariaDB() )
177179
return true;
178180

179-
java.sql.DatabaseMetaData meta = connection.getMetaData();
181+
DatabaseMetaData meta = connection.getMetaData();
180182
int major = meta.getDatabaseMajorVersion();
181183
int minor = meta.getDatabaseMinorVersion();
182184
return ((major == 5 && minor >= 6) || major > 5);
183185
}
184186

185187
private boolean isMariaDB() throws SQLException {
186-
java.sql.DatabaseMetaData meta = connection.getMetaData();
188+
DatabaseMetaData meta = connection.getMetaData();
187189
return meta.getDatabaseProductVersion().toLowerCase().contains("maria");
188190
}
189191

192+
static final String MARIA_VERSION_REGEX = "[\\d\\.]+-(\\d+)\\.(\\d+)";
193+
private boolean mariaSupportsJSON() throws SQLException {
194+
DatabaseMetaData meta = connection.getMetaData();
195+
String versionString = meta.getDatabaseProductVersion();
196+
Pattern pattern = Pattern.compile(MARIA_VERSION_REGEX);
197+
Matcher m = pattern.matcher(versionString);
198+
199+
if ( m.find() ) {
200+
int major = Integer.parseInt(m.group(1));
201+
int minor = Integer.parseInt(m.group(2));
202+
203+
return major >= 10 && minor > 1;
204+
} else { // shrugging purple lady
205+
return false;
206+
}
207+
}
208+
190209
private void captureTables(Database db, HashMap<String, Table> tables) throws SQLException {
191210
columnPreparedStatement.setString(1, db.getName());
192211

@@ -299,7 +318,7 @@ public void close() throws SQLException {
299318
private void detectMariaDBJSON(Schema schema) throws SQLException, InvalidSchemaError {
300319
String checkConstraintSQL = "SELECT CONSTRAINT_SCHEMA, TABLE_NAME, CONSTRAINT_NAME, CHECK_CLAUSE " +
301320
"from INFORMATION_SCHEMA.CHECK_CONSTRAINTS " +
302-
"where LEVEL='column' and CHECK_CLAUSE LIKE 'json_valid(%)'";
321+
"where CHECK_CLAUSE LIKE 'json_valid(%)'";
303322

304323
String regex = "json_valid\\(`(.*)`\\)";
305324
Pattern pattern = Pattern.compile(regex);

‎src/main/java/com/zendesk/maxwell/util/AbstractConfig.java

Lines changed: 3 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -114,7 +114,9 @@ protected Properties readPropertiesEnv(String envConfig) {
114114
Properties properties = new Properties();
115115
for (Map.Entry<String, Object> entry : stringMap.entrySet()) {
116116
LOGGER.debug("Got env_config key: {}", entry.getKey());
117-
properties.put(entry.getKey(), entry.getValue().toString());
117+
if (entry.getKey() != null && entry.getValue() != null) {
118+
properties.put(entry.getKey(), entry.getValue().toString());
119+
}
118120
}
119121
return properties;
120122
} catch (JsonProcessingException e) {

‎src/test/java/com/zendesk/maxwell/MaxwellConfigTest.java

Lines changed: 4 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -12,6 +12,7 @@
1212
import org.junit.contrib.java.lang.system.EnvironmentVariables;
1313

1414
import java.nio.file.Paths;
15+
import java.util.HashMap;
1516
import java.util.Map;
1617

1718
import static org.junit.Assert.*;
@@ -88,12 +89,14 @@ public void testEnvVarConfigViaConfigFile() {
8889

8990
@Test
9091
public void testEnvJsonConfig() throws JsonProcessingException {
91-
Map<String, String> configMap = ImmutableMap.<String, String>builder()
92+
Map<String, String> nonNullconfigMap = ImmutableMap.<String, String>builder()
9293
.put("user", "foo")
9394
.put("password", "bar")
9495
.put("host", "remotehost")
9596
.put("kafka.retries", "100")
9697
.build();
98+
HashMap<String, String> configMap = new HashMap<>(nonNullconfigMap);
99+
configMap.put("ignore.me", null);
97100
ObjectMapper mapper = new ObjectMapper();
98101
String jsonConfig = mapper.writeValueAsString(configMap);
99102
environmentVariables.set("MAXWELL_JSON", " " + jsonConfig);

‎src/test/java/com/zendesk/maxwell/schema/ddl/DDLParserTest.java

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -260,7 +260,8 @@ public void testParsingSomeAlters() {
260260
"ALTER TABLE t1 DROP PARTITION IF EXISTS p3", // some mariada-fu
261261
"ALTER TABLE t1 DROP CONSTRAINT ck",
262262
"ALTER TABLE t1 DROP CHECK ck",
263-
"create table test ( i float default -1. )"
263+
"create table test ( i float default -1. )",
264+
"alter database d ENCRYPTION='Y'"
264265
};
265266

266267
for ( String s : testSQL ) {

0 commit comments

Comments
 (0)