Acceptance Testing

[alexluong]

I'm creating this issue to start the discussion around acceptance testing. My objectives from this discussion are:

1. a documentation guide on acceptance testing so contributors can use it in their development process
2. decision on how to use acceptance testing in CI/CD

What

First, what the acceptance test suite does: it takes some TF code, provisions the resources on Hookdeck, runs assertions, and then destroys the created resources.

Given Hookdeck's models and APIs, we are able to perform the acceptance test suite in a way that everything is cleaned up very well.

⭐ Running acceptance testing will result in deleted resources in Hookdeck's core. I'm gonna assume these deleted resources are insignificant for our system to handle. Just want to call this out for confirmation before we proceed.

How

The main requirements to run the acceptance test suite is a HOOKDECK_API_KEY. I used a project in my personal Hookdeck account for all my testing so far.

The test command is quite similar to Outpost's (actually Outpost's test setup is inspired by this).

# make sure HOOKDECK_API_KEY env exists; for local dev, we load .env.testing automatically

# run the full suite, including unit tests & acceptance tests
make testacc

# or run a specific resource suite
TEST=./internal/provider/connection make testacc

# or run a specific test in the suite
TEST=./internal/provider/connection RUN=TestAccConnectionResourceWithRules make testacc

We can also run the suite against another environment (staging/local) using HOOKDECK_API_BASE env.

CI

It should be fairly straightforward for us to run the acceptance test since the only requirement is HOOKDECK_API_KEY. The full suite may take up to 1min right now with tests on connections & transformations. To support more resources like sources and destinations, it can take up to 2-3min in total. It's not the fastest but I think it's not terrible either.

❓ Where should we run this acceptance test suite? During PR? Before release? Or maybe both?