I wish Redpanda Connect had a modular design

[static-moonlight]

Not sure this will ever be possible. I just wish Redpanda Connect (aka Benthos) would be modular, so I could tailor it down to the 1-2% of functionality I actually need, resulting in a lightweight deployment, instead of dealing with a 200+ MB all-in-one monster (which adds up, when deploying a lot of instances).

[mihaitodor]

Hey @static-moonlight that’s possible right now if you’re willing to maintain your own binary. Have a look here: https://github.com/redpanda-data/redpanda-connect-plugin-example. While that repo only shows you how to use the bundle imports, you can actually import them selectively as needed (see https://github.com/redpanda-data/connect/tree/main/public/components).

[static-moonlight]

A little more complicated than i had hoped, but I will look into that. Thanks.

One follow-up question though. It doesn't look like a 1:1 structure of the components, described in the documentation. How do I know what to include, to get a certain component, like the "rate limit" for example.

[mihaitodor]

“rate limit” is a component type. There can be one or more implementations. We did a best-effort approach to package components by technology, so if you need the Redis rate limit, you’ll have to import all the Redis components. From a binary size perspective, it won’t make a huge difference. The biggest impact is including the client libraries for a specific technology in the first place. The embedded docs that we add for each component do take a bit of space as well and we’re discussing how we might want to streamline that in the future.

[static-moonlight]

Oh, I wasn't aware, that things like rate limit are technology specific (looks generic).

Anyway, thank you. That was helpful.

[mihaitodor]

You can see here https://docs.redpanda.com/redpanda-connect/components/rate_limits/about/ in the accordion menu that there are 2 implementations: local and redis. The local one comes from the Benthos framework and, if you do fine-grained selective imports, you’ll have to make sure that the io and pure components are also brought in https://github.com/redpanda-data/benthos/tree/main/public/components (normally they’re included by the Connect packages that you’ll import, but you’ll have to check).

[static-moonlight]

Awesome