Close the old connection to make sure the broker drops the producer on its side

[wenbingshen]

Master Issue: #676

Motivation

When reconnecting to the server, the old connection should be closed.

Modifications

Close the old connection to make sure the broker drops the producer on its side

Verifying this change

• Make sure that the change passes the CI checks.

(Please pick either of the following options)

This change is a trivial rework / code cleanup without any test coverage.

(or)

This change is already covered by existing tests, such as (please describe tests).

(or)

This change added tests and can be verified as follows:

(example:)

• Added integration tests for end-to-end deployment with large payloads (10MB)
• Extended integration test for recovery after broker failure

Does this pull request potentially affect one of the following parts:

If yes was chosen, please highlight the changes

• Dependencies (does it add or upgrade a dependency): (yes / no)
• The public API: (yes / no)
• The schema: (yes / no / don't know)
• The default values of configurations: (yes / no)
• The wire protocol: (yes / no)

Documentation

• Does this pull request introduce a new feature? (yes / no)
• If yes, how is the feature documented? (not applicable / docs / GoDocs / not documented)
• If a feature is not applicable for documentation, explain why?
• If a feature is not documented yet in this PR, please create a followup issue for adding the documentation

[wenbingshen]

@cckellogg PTAL, thanks

[cckellogg]

How does changing the state here to connectionClosing and then changing it to connectionClosed later fix the issue?

[wenbingshen]

This is not to say that the problem is solved, but first change the state to connectionClosing , the logic is more rigorous, I modified the description of the PR, it does not mean that the issue is fixed, but it is related to investigating this issue.

[wenbingshen]

If there is no intermediate connectionClosing state, other threads may start to create new connections in GetConnection before the current cnx is really closed. This is a mistake in itself. State transitions should be stricter. You can see that the network connection state transition of the java client is very rigorous.

If we can ensure that the state transition is more rigorous, set the state to connectionClosing before cnx closed, which can prevent other threads to create a new connection. This may be able to solve the issue #676 that the old connection was not closed, and the new connection has initiated a request, even if this is not the root cause of the problem.