Extensive memory consumption when using OpenAPI 3.1 spec

[tarnowskijan]

Q&A (please complete the following information)

• OS: macOS
• Environment: Node.js v10.9.2
• Method of installation: npm
• Swagger-Client version: 3.34.0
• Swagger/OpenAPI version: OpenAPI 3.1

Content & configuration

Swagger/OpenAPI definition:
https://petstore31.swagger.io/api/v31/openapi.json

Swagger-Client usage:

const swaggerClient = require('swagger-client')

const specFile = require('./petstore-spec')

async function main() {
    const specVersion = process.argv[2]
    const executions = Number(process.argv[3] || '10')
    console.log(`Using spec version: ${specVersion}, executions: ${executions}`)

    const iterator = Array.from(Array(executions).keys());
    const spec = Object.assign(specFile, {openapi: specVersion})
    const specClients = await Promise.all(
        iterator.map(idx => swaggerClient({
            spec: Object.assign(
                {...spec},
                {servers: [{url: 'http://localhost:8989/example-api-' + idx}]}
            )
        }))
    )
    return specClients.length
}

main()
    .then(res => {
        console.log(res);
    })
    .catch(err => {
        console.error(err);
    })

Describe the bug you're encountering

After migration to OpenAPI 3.1 specs we are experiencing memory consumption spikes leading to OOMs in our Node.js applications. Our investigation revealed that creating Swagger clients from OpenAPI 3.1 spec requires about 20% more memory than it was for 3.0.1 spec. For large specs (186 kB) it's even six times more and we have plenty of such specs. Unfortunately, we cannot execute promises one-by-one to limit the consumption because it would take ages to build all clients (sometimes even 50).

Petstore spec:

% /usr/bin/time -l node app.js 3.0.1 40 
Using spec version: 3.0.1, executions: 40
40
        0.70 real         0.35 user         0.09 sys
           138412032  maximum resident set size

% /usr/bin/time -l node app.js 3.1.0 40
Using spec version: 3.1.0, executions: 40
40
        1.39 real         1.59 user         0.07 sys
           160399360  maximum resident set size

Large spec:

% /usr/bin/time -l node app.js 3.0.1 40
Using spec version: 3.0.1, executions: 40
40
        0.85 real         0.60 user         0.09 sys
           144490496  maximum resident set size

% /usr/bin/time -l node app.js 3.1.0 40
Using spec version: 3.1.0, executions: 40
40
       30.93 real        32.17 user         0.48 sys
           903905280  maximum resident set size

To reproduce...

Steps to reproduce the behavior:

1. (Optional) Replace petstore-spec file with some large spec file
2. Execute /usr/bin/time -l node app.js 3.0.1 40 to get results for 3.0.1 spec
3. Execute /usr/bin/time -l node app.js 3.1.0 40 to get results for 3.1.0 spec

Expected behavior

Memory consumption for large 3.1.0 specs is as close as possible to 3.0.1 specs

Additional context or thoughts

Is there a way to configure swagger-client so it consumes less memory for 3.1.0 specs?

[ponelat]

Relevant: swagger-api/apidom#385

[glowcloud]

After investigating, we have determined that the cause lies within the ApiDOM structure.

The result of refracting a definition of 10 MB size will leave 100 MB of memory footprint for the JSON structure. For the ApiDOM AST interpretation, the footprint will be 200 MB, and the dehydrated result will take 400 MB.

We need to further investigate the possibilities of amending the ApiDOM and its elements to address this issue.

[char0n]

ApiDOM is unfortunately memory inefficient. We're already trying to figure out how to address this.