using schemathesis or similar tools for testing purposes

[sameh-farouk]

Description

I saw these two issues raised recently,

1. Sometimes our API specifications get outdated and out-of-sync with our implementation. we should always respect this contract, but how to make sure it is always respected and updated?
2. We need to test our API functionality regularly to ensure it is working properly, a PR was submitted to run basic tests in a GitHub action. it will add one more thing to the stack that is needed also to keep in sync with every change.

I want to point out that there are many tools that can generate and run tests from the API specification.
e.g, schemathesis is an API testing tool that could run thousands of generated test cases given just the API schema.

This could solve these two issues and provides added benefits:

• The API specification is the source of truth, this will help us to improve the API and make sure that API specification remains in sync with the implementation. issue no1. solved.
• No tests code to write or maintain, less work. issue no.2 solved.
• Better coverage. the tool will run thousands of tests.
• GitHub action is available out of the box (for the suggested tool).

It is expected to have many falling tests when we start to use this tool, but I see this as a pro and a way to improve and fix invisible issues.

[ramezsaeed]

@A-Harby please verify and update this issue

[A-Harby]

As @sameh-farouk said old PR is a hard coded endpoint tests so it's hard to maintain updated with every new changes happens on the grid proxy.
And I did some researches on schema/api testing tools and schemathesis was the most suitable tool to be used cause it have the ability to generate and easily customize theses tests.

The issue is that when the schema was tested, it reported a numerous amount of errors which will make it hard to debug and fix.
so with using schemathesis docs I found a way to use a python code to specify each endpoint and checks (mentioned below) I want the tool to test and I'm trying to separates theses errors as I could, so it could be more easier to identify and report.

There are five built-in checks you can use:

• not_a_server_error. The response has 5xx HTTP status;
• status_code_conformance. The response status is not defined in the API schema;
• content_type_conformance. The response content type is not defined in the API schema;
• response_schema_conformance. The response content does not conform to the schema defined for this specific response;
• response_headers_conformance. The response headers does not contain all defined headers.

[Stranger6667]

Hey, Schemathesis author here. Folks, if you'd need any help or something won't work as expected with Schemathesis or you'd like to propose some feature request - do not hesitate to ping me :)

[xmonader]

What is the status here? please make sure it's updated and merged or let's close the issue