diff --git a/doc/user/content/concepts/sinks.md b/doc/user/content/concepts/sinks.md index ddb51a3f05e22..2a589331aea66 100644 --- a/doc/user/content/concepts/sinks.md +++ b/doc/user/content/concepts/sinks.md @@ -12,14 +12,15 @@ aliases: ## Overview -Sinks are the inverse of sources and represent a connection to an external stream -where Materialize outputs data. When a user defines a sink over a materialized view, -source, or table, Materialize automatically generates the required schema and writes down -the stream of changes to that view or source. In effect, Materialize sinks act as -change data capture (CDC) producers for the given source or view. +Sinks are the inverse of sources and represent a connection to an external +stream where Materialize outputs data. You can sink data from a **materialized** +view, a source, or a table. -Currently, Materialize only supports sending sink data to Kafka. See -the [Kafka sink documentation](/sql/create-sink/kafka) for details. +## Sink methods + +To create a sink, you can: + +{{< yaml-table data="sink_external_systems" >}} ## Clusters and sinks @@ -29,7 +30,8 @@ See also [Operational guidelines](/manage/operational-guidelines/). ## Hydration considerations -During creation, sinks need to load an entire snapshot of the data in memory. +During creation, Kafka sinks need to load an entire snapshot of the data in +memory. ## Related pages diff --git a/doc/user/content/serve-results/sink/_index.md b/doc/user/content/serve-results/sink/_index.md index 81068b9f896a6..10a053706d3c2 100644 --- a/doc/user/content/serve-results/sink/_index.md +++ b/doc/user/content/serve-results/sink/_index.md @@ -11,31 +11,20 @@ menu: --- A [sink](/concepts/sinks/) describes the external system you want Materialize to -write data to and details the encoding of that data. +write data to and details the encoding of that data. You can sink data from a +**materialized** view, a source, or a table. -### Creating a sink +## Sink methods -When a user defines a sink over a **materialized** view, source, or table, -Materialize automatically generates the required schema and writes down the -stream of changes to that view or source. In effect, Materialize sinks act as -change data capture (CDC) producers for the given source or view. +To create a sink, you can: -During creation, sinks need to load an entire snapshot of the data in memory. +{{< yaml-table data="sink_external_systems" >}} ### Operational guideline - Avoid putting sinks on the same cluster that hosts sources to allow for [blue/green deployment](/manage/dbt/blue-green-deployments). -### Available guides - -The following guides are available for sinking results from Materialize to external systems: - -- [Sinking results to Amazon S3](/serve-results/sink/s3/) -- [Sinking results to Census](/serve-results/sink/census/) -- [Sinking results to Kafka/Redpanda](/serve-results/sink/kafka/) -- [Sinking results to Snowflake](/serve-results/sink/snowflake/) - ### Troubleshooting For help, see [Troubleshooting diff --git a/doc/user/content/serve-results/sink/kafka.md b/doc/user/content/serve-results/sink/kafka.md index 50ca189673b69..97b888823bb77 100644 --- a/doc/user/content/serve-results/sink/kafka.md +++ b/doc/user/content/serve-results/sink/kafka.md @@ -15,7 +15,10 @@ menu: ## Connectors Materialize bundles a **native connector** that allow writing data to Kafka and -Redpanda. +Redpanda. When a user defines a sink to Kafka/Redpanda, Materialize +automatically generates the required schema and writes down the stream of +changes to that view or source. In effect, Materialize sinks act as change data +capture (CDC) producers for the given source or view. For details on the connector, including syntax, supported formats and examples, refer to [`CREATE SINK`](/sql/create-sink/kafka). @@ -24,10 +27,14 @@ refer to [`CREATE SINK`](/sql/create-sink/kafka). Redpanda uses the same syntax as Kafka [`CREATE SINK`](/sql/create-sink/kafka). -{{}} +{{< /tip >}} ## Features +### Memory use during creation + +During creation, sinks need to load an entire snapshot of the data in memory. + ### Automatic topic creation If the specified Kafka topic does not exist, Materialize will attempt to create diff --git a/doc/user/data/sink_external_systems.yml b/doc/user/data/sink_external_systems.yml new file mode 100644 index 0000000000000..ad31dfab10e58 --- /dev/null +++ b/doc/user/data/sink_external_systems.yml @@ -0,0 +1,31 @@ +columns: + - column: Method + - column: External system + - column: Guide(s) or Example(s) + +rows: + - External system: "Amazon S3 or S3-compatible storage" + Method: "Use `COPY TO` command" + Guide(s) or Example(s): | + - [Sink to Amazon S3](/serve-results/sink/s3/) + + - External system: "Census supported destinations" + Method: "Use Census as an intermediate step" + Guide(s) or Example(s): | + - [Sink to Census](/serve-results/sink/census/) + + - External system: "Snowflake and other systems that can read from S3" + Method: "Use `COPY TO` S3 or S3-compatible storage as an intermediate step" + Guide(s) or Example(s): | + - [Sink to Snowflake](/serve-results/sink/snowflake/) + + - External system: "Kafka/Redpanda" + Method: "Use a native connector" + Guide(s) or Example(s): | + - [Sink to Kafka/Redpanda](/serve-results/sink/kafka/) + + - External system: "Various" + Method: "Use `SUBSCRIBE`" + Guide(s) or Example(s): | + - [Sink to Postgres](https://github.com/MaterializeInc/mz-catalog-sync) + - [Sink to Redis](https://github.com/MaterializeIncLabs/mz-redis-sync)