Capabilities of Testcontainers modules

[HofmeisterAn]

Testcontainers modules allow different kinds of implementations. From a very pure module that only contains the bare minimum configurations, up to a more advanced module that allows extensive customization of the container, application, or services running inside.

Moreover, a module can extend beyond container configurations and include third-party dependencies, offering developers specific APIs to interact with the container, application, or service.

However, we believe that providing the option to configure every small detail of a container, application, or service may not add much value. Ideally, developers should be able to start modules and use them without the need for specific configurations. Modules should cover common use cases, incorporate best practices, and facilitate fast startup of test dependencies.

We believe that sticking to basic modules offers several advantages. Often, there is no need to configure every small detail, for example:

• Configuring ports for applications or services running within containers is not necessary, as developers can access the application or service through the random assigned host port or network alias in container-to-container communication.
• Excluding third-party dependencies can prevent developers from encountering dependency clashes. It is not appropriate to force developers to use specific dependency versions.

We are interested in hearing the thoughts and opinions of our Testcontainers community regarding their vision for modules. What do you consider to be the crucial features required for working effectively with modules?

Note: We understand that companies or applications may have certain restrictions, and that certain features may be essential for some use cases but not for others. However, we believe that the Testcontainers module API provides the flexibility for users to create their own custom implementation to cater to their specific needs.

[HofmeisterAn]

I really enjoy maintaining Testcontainers for .NET and have learned a lot over the years. Lately, I've noticed it's getting harder to keep up with the library, especially with Docker.DotNet (my main focus) and many of the modules.

Modules that I don't know deeply or that aren't widely used can become a real burden. Often, after the initial PR, they lack someone who consistently maintains them: updating code, fixing CI, improving docs, and being a point of contact.

I want to use this discussion to reach out to the community and encourage more developers to get involved, take over modules, and become part of this project ❤️.

[digital88]

What modules do you think are most in the need of maintainer?

[HofmeisterAn]

Thanks @digital88, your contributions have already been a huge help!

The Pulsar bug is really annoying 😆.

I think it's hard to assign responsibility for just a few modules. Right now, I'm either missing or don't have time for:

• Making sure modules match other Testcontainers language implementations in features and behavior.
• Reviewing open PRs to ensure they're solid, follow standards, and run reliably.
• @kiview's idea is great. We could move some modules or PRs to community maintenance to reduce the load, and promote them to official modules once they're stable. Maybe we could even set up a process for this. It's something worth elaborating.
• Cleaning up unnecessary modules or replacing them with better, widely used alternatives.
  Some niche modules are probably better community maintained.
• Keeping docs up-to-date (https://dotnet.testcontainers.org/modules/), including module specific docs.
  @mdelapenya automated a lot in Go, AFAIK 🥳.

Overall, it's about continuous improvement. With so many modules, finding maintainers for each is tough.

[DDtKey]

The approach with community modules is indeed very helpful. It divides both responsibilities and codebase, which significantly reduces the cognitive load for maintainers. Additionally, it provides a natural flow for the core part, which evolves independently with feedback from users & maintainers of the modules.

However, based on our experience, I recommend establishing a team of a few individuals who are prepared to maintain the modules. Since the community is spontaneous, you may receive numerous pull requests for new modules. However, when any issues arise, it might be challenging to find someone willing to fix or debug them. If you’re unfamiliar with the module sufficiently, you’ll need to research the underlying Docker image behavior, logs, parameters, and so on. For instance, you may encounter test flakiness (it might even be client/test side issue).

All these factors require you to allocate sufficient time or create an issue to request the community’s assistance in addressing them. Therefore, there should be at least one person who is prepared to handle unexpected issues and can at least dedicate time to describing them to ask the community for a help.

Additionally, I recommend designing CI for the community modules repository in a way that isolates them. For example, in PRs you can run tests only for the changed parts and have runs of the entire test suite in the main branch only (on push or nightly).

[mdelapenya]

Keeping docs up-to-date (https://dotnet.testcontainers.org/modules/), including module specific docs.
@mdelapenya automated a lot in Go, AFAIK

For that, I created a modulegen tool (for tc-go) that creates all the scaffolding for a module: CI automation, docs, tests, and testable examples. Specifically for docs, it helps a lot in defining a consistent structure for laying the documentation about a given module.

If there is interest in that, I could picture a generic module generator that integrates on each language 🤔 But of course, it would need maintenance 😅

[mdelapenya]

Additionally, I recommend designing CI for the community modules repository in a way that isolates them. For example, in PRs you can run tests only for the changed parts and have runs of the entire test suite in the main branch only (on push or nightly).

Indeed! We adopted that in tc-go about 8 months ago, and it is super helpful. A change in a module runs the CI build for that specific module, and a change in the core runs everything. Our PRs as maintainers usually run all, but eventual contributors has a much better experience waiting for their builds to pass/fail. It also reduces the flakiness coming from other modules.

I helped @cristianrgreco with that for tc-node, so I'm happy to help here in .NET or Rust