Skip to content

Commit ba36e00

Browse files
committed
docs: tweaks to include subscribe based sinks
1 parent 7c2004f commit ba36e00

File tree

4 files changed

+55
-26
lines changed

4 files changed

+55
-26
lines changed

doc/user/content/concepts/sinks.md

Lines changed: 10 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -12,14 +12,15 @@ aliases:
1212

1313
## Overview
1414

15-
Sinks are the inverse of sources and represent a connection to an external stream
16-
where Materialize outputs data. When a user defines a sink over a materialized view,
17-
source, or table, Materialize automatically generates the required schema and writes down
18-
the stream of changes to that view or source. In effect, Materialize sinks act as
19-
change data capture (CDC) producers for the given source or view.
15+
Sinks are the inverse of sources and represent a connection to an external
16+
stream where Materialize outputs data. You can sink data from a **materialized**
17+
view, a source, or a table.
2018

21-
Currently, Materialize only supports sending sink data to Kafka. See
22-
the [Kafka sink documentation](/sql/create-sink/kafka) for details.
19+
## Sink methods
20+
21+
To create a sink, you can:
22+
23+
{{< yaml-table data="sink_external_systems" >}}
2324

2425
## Clusters and sinks
2526

@@ -29,7 +30,8 @@ See also [Operational guidelines](/manage/operational-guidelines/).
2930

3031
## Hydration considerations
3132

32-
During creation, sinks need to load an entire snapshot of the data in memory.
33+
During creation, Kafka sinks need to load an entire snapshot of the data in
34+
memory.
3335

3436
## Related pages
3537

doc/user/content/serve-results/sink/_index.md

Lines changed: 5 additions & 16 deletions
Original file line numberDiff line numberDiff line change
@@ -11,31 +11,20 @@ menu:
1111
---
1212

1313
A [sink](/concepts/sinks/) describes the external system you want Materialize to
14-
write data to and details the encoding of that data.
14+
write data to and details the encoding of that data. You can sink data from a
15+
**materialized** view, a source, or a table.
1516

16-
### Creating a sink
17+
## Sink methods
1718

18-
When a user defines a sink over a **materialized** view, source, or table,
19-
Materialize automatically generates the required schema and writes down the
20-
stream of changes to that view or source. In effect, Materialize sinks act as
21-
change data capture (CDC) producers for the given source or view.
19+
To create a sink, you can:
2220

23-
During creation, sinks need to load an entire snapshot of the data in memory.
21+
{{< yaml-table data="sink_external_systems" >}}
2422

2523
### Operational guideline
2624

2725
- Avoid putting sinks on the same cluster that hosts sources to allow for
2826
[blue/green deployment](/manage/dbt/blue-green-deployments).
2927

30-
### Available guides
31-
32-
The following guides are available for sinking results from Materialize to external systems:
33-
34-
- [Sinking results to Amazon S3](/serve-results/sink/s3/)
35-
- [Sinking results to Census](/serve-results/sink/census/)
36-
- [Sinking results to Kafka/Redpanda](/serve-results/sink/kafka/)
37-
- [Sinking results to Snowflake](/serve-results/sink/snowflake/)
38-
3928
### Troubleshooting
4029

4130
For help, see [Troubleshooting

doc/user/content/serve-results/sink/kafka.md

Lines changed: 9 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -15,7 +15,10 @@ menu:
1515
## Connectors
1616

1717
Materialize bundles a **native connector** that allow writing data to Kafka and
18-
Redpanda.
18+
Redpanda. When a user defines a sink to Kafka/Redpanda, Materialize
19+
automatically generates the required schema and writes down the stream of
20+
changes to that view or source. In effect, Materialize sinks act as change data
21+
capture (CDC) producers for the given source or view.
1922

2023
For details on the connector, including syntax, supported formats and examples,
2124
refer to [`CREATE SINK`](/sql/create-sink/kafka).
@@ -24,10 +27,14 @@ refer to [`CREATE SINK`](/sql/create-sink/kafka).
2427

2528
Redpanda uses the same syntax as Kafka [`CREATE SINK`](/sql/create-sink/kafka).
2629

27-
{{</ tip >}}
30+
{{< /tip >}}
2831

2932
## Features
3033

34+
### Memory use during creation
35+
36+
During creation, sinks need to load an entire snapshot of the data in memory.
37+
3138
### Automatic topic creation
3239

3340
If the specified Kafka topic does not exist, Materialize will attempt to create
Lines changed: 31 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,31 @@
1+
columns:
2+
- column: Method
3+
- column: External system
4+
- column: Guide(s) or Example(s)
5+
6+
rows:
7+
- External system: "Amazon S3 or S3-compatible storage"
8+
Method: "Use `COPY TO` command"
9+
Guide(s) or Example(s): |
10+
- [Sink to Amazon S3](/serve-results/sink/s3/)
11+
12+
- External system: "Census supported destinations"
13+
Method: "Use Census as an intermediate step"
14+
Guide(s) or Example(s): |
15+
- [Sink to Census](/serve-results/sink/census/)
16+
17+
- External system: "Snowflake and other systems that can read from S3"
18+
Method: "Use `COPY TO` S3 or S3-compatible storage as an intermediate step"
19+
Guide(s) or Example(s): |
20+
- [Sink to Snowflake](/serve-results/sink/snowflake/)
21+
22+
- External system: "Kafka/Redpanda"
23+
Method: "Use a native connector"
24+
Guide(s) or Example(s): |
25+
- [Sink to Kafka/Redpanda](/serve-results/sink/kafka/)
26+
27+
- External system: "Various"
28+
Method: "Use `SUBSCRIBE`"
29+
Guide(s) or Example(s): |
30+
- [Sink to Postgres](https://github.com/MaterializeInc/mz-catalog-sync)
31+
- [Sink to Redis](https://github.com/MaterializeIncLabs/mz-redis-sync)

0 commit comments

Comments
 (0)