Connections to the Firecracker unix socket are never closed

[yitsushi]

When we call the API to see what's the status of the MicroVM, the connection stays there and it's never closed (only when we restart the application).

Here is an isolated demonstration: https://github.com/yitsushi/firecracker-go-sdk-connection-leak-poc

Sample code:

func callAPI() {
	fcClient := client.NewHTTPClient(strfmt.NewFormats())
	logger := logrus.NewEntry(logrus.New())

	socketPath := "/tmp/firecracker.socket"
	transport := firecracker.NewUnixSocketTransport(
		socketPath,
		logger,
		true,
	)

	fcClient.SetTransport(transport)

	resp, err := fcClient.Operations.DescribeInstance(
		operations.NewDescribeInstanceParams(),
	)
	if err != nil {
		logrus.Error(err.Error())

		return
	}

	logrus.
		WithField("state", *resp.Payload.State).
		Info("Firecracker API response")
}

In the demo (link above) we are using github.com/firecracker-microvm/firecracker-go-sdk v0.22.0.

It's a very painful behavior because

1. We have unused resources in the system piling up
2. Firecracker limitations on the connection pool and we are hitting 503 errors

References:

• Source issue: Delete microvm fails because of 'too many open connections' liquidmetal-dev/flintlock#266
• Demo: https://github.com/yitsushi/firecracker-go-sdk-connection-leak-poc
• Firecracker server (const MAX_CONNECTIONS): https://github.com/firecracker-microvm/micro-http/blob/36e59a083e76a2449e0f58e4283d201bc72fdf13/src/server.rs#L23