You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
|`topic`|_Required_<br><br>**Description:**The Pulsar topic where records are published. to.|
103
-
|`url`|<br><br>**Description:**The service URL for the Pulsar cluster.<br><br>**Default**: `"pulsar://localhost:6650"`|
104
-
|`defaultHeaders`|**Description**: Default headers to include in all outgoing messages along with KurrentDB system headers. Prefix header names with `defaultHeaders:` followed by the header key name.<br><br>**Example**: `"defaultHeaders:AppName": "Kurrent"` <br><br>**Default**: None|
105
-
|`authentication:token`|**Description:**A JSON web token for authenticating the connector with Pulsar. |
|`partitionKeyExtraction:source`|**Description:**The source for extracting the partition key.<br><br>**Accepted Values:**`stream`, `streamSuffix`, `headers`<br><br>**Default**: `PartitionKey`|
115
-
|`partitionKeyExtraction:expression`|**Description:**A regex (for `stream` source) or a comma-separated list of header keys (for `headers` source) used to extract or combine values for the partition key. When using headers, values are concatenated with a hyphen (for example, `value1-value2`).|
|`partitionKeyExtraction:source`|**Description:**<br>The source for extracting the partition key.<br><br>**Accepted Values:**`"stream"`, `"streamSuffix"`, `"headers"`<br><br>**Default**: `"partitionKey"`|
60
+
|`partitionKeyExtraction:expression`|**Description:**<br>A regex (for `stream` source) or a comma-separated list of header keys (for `headers` source) used to extract or combine values for the partition key. When using headers, values are concatenated with a hyphen (for example, `value1-value2`). |
116
61
117
62
### Resilience
118
63
119
-
These settings customize the connector’s behavior in handling message failures and retries provided by Apache Pulsar.
64
+
The Pulsar sink connector relies on its own retry mechanism and doesn't include the configuration from [Resilience configuration](../settings.md#resilience-configuration).
|`resilience:enabled`|**Description:**<br>Enables resilience features for message handling.<br><br>**Default**: `"true"`|
69
+
|`resilience:retryInterval`|**Description:**<br>Retry interval in seconds. Must be greater than 0.<br><br>**Format:** seconds or `"HH:MM:SS"`.<br><br> **Default:**`"00:00:03"`|
70
+
71
+
## Headers
72
+
73
+
The Kafka sink connector lets you include custom headers in the message headers
74
+
it sends to your topic. To add custom headers, use the `defaultHeaders` setting
75
+
in your connector configuration. Each custom header should be specified with the
76
+
prefix `defaultHeaders:` followed by the header name.
These headers will be included in every message sent by the connector, in addition to the [default headers](../features.md#headers) automatically added by the connector's plugin.
|`resilience:enabled`|**Description:** Enables resilience features for message handling.<br><br>**Default**: `"true"`|
124
-
|`resilience:retryInterval`|**Description:** Retry interval in seconds. Must be greater than 0.<br><br>**Format:** seconds or `"HH:MM:SS"`.<br><br> **Default:**`"00:00:03"`|
125
94
126
95
## Examples
127
96
128
-
These examples demonstrate how to configure partitioning, security, and other practical scenarios.
97
+
### Partitioning
98
+
99
+
The Pulsar sink connector allows customizing the partition keys that are sent
100
+
with the message.
101
+
102
+
By default, it will use `"partitionKey"` and the message will be distributed
103
+
using round-robin partitioning across the available partitions in the topic.
129
104
130
-
### Partition using Stream ID
105
+
**Partition using Stream ID**
106
+
107
+
You can extract part of the stream name using a regular expression (regex) to
108
+
define the partition key. The expression is optional and can be customized based
109
+
on your naming convention. In this example, the expression captures the stream
110
+
name up to `_data`.
111
+
112
+
```http
113
+
PUT /connectors/{{id}}/settings
114
+
Host: localhost:2113
115
+
Content-Type: application/json
131
116
132
-
Extract part of a stream name using a regex. In this example, the regex captures everything up to `_data`.
133
117
134
-
```json
135
118
{
136
119
"partitionKeyExtraction:enabled": "true",
137
120
"partitionKeyExtraction:source": "stream",
@@ -143,7 +126,12 @@ Alternatively, if you only need the last segment of the stream name (after a
143
126
hyphen), you can use the `streamSuffix` source. This
144
127
doesn't require an expression since it automatically extracts the suffix.
145
128
146
-
```json
129
+
```http
130
+
PUT /connectors/{{id}}/settings
131
+
Host: localhost:2113
132
+
Content-Type: application/json
133
+
134
+
147
135
{
148
136
"partitionKeyExtraction:enabled": "true",
149
137
"partitionKeyExtraction:source": "streamSuffix"
@@ -154,31 +142,22 @@ The `streamSuffix` source is useful when stream names follow a structured
154
142
format, and you want to use only the trailing part as the partition key. For
155
143
example, if the stream is named `user-123`, the partition key would be `123`.
156
144
157
-
### Partition using header values
145
+
**Partition using header values**
146
+
147
+
You can create partition keys by combining values from a record's metadata.
158
148
159
-
Combine multiple header values to form the partition key. This example concatenates header values `key1` and `key2` using a hyphen.
149
+
```http
150
+
PUT /connectors/{{id}}/settings
151
+
Host: localhost:2113
152
+
Content-Type: application/json
160
153
161
-
```json
162
154
{
163
155
"partitionKeyExtraction:enabled": "true",
164
156
"partitionKeyExtraction:source": "headers",
165
157
"partitionKeyExtraction:expression": "key1,key2"
166
158
}
167
159
```
168
160
169
-
The `Headers` source allows you to pull values from the event's metadata. The
170
-
`documentId:expression` field lists the header keys (in this case, `key1` and
171
-
`key2`), and their values are concatenated to generate the partition key.
172
-
173
-
::: details Click here to see an example
161
+
Specify the header keys you want to use in the `partitionKeyExtraction:expression` field (e.g., `key1,key2`). The connector will concatenate the header values with a hyphen (`-`) to create the partition key.
174
162
175
-
```json
176
-
{
177
-
"key1": "value1",
178
-
"key2": "value2"
179
-
}
180
-
181
-
// outputs "value1-value2"
182
-
```
183
-
184
-
:::
163
+
For example, if your event has headers `key1: regionA` and `key2: zone1`, the partition key will be `regionA-zone1`.
0 commit comments