bug: Query service client not stable by SLO tests

[asmyasnikov]

Ideas about reason:

• deadlock in internal/pool
• something else

[adwski]

I did some tests and post below my findings.

It seems like issue is not a deadlock inside pool, but timeouts in query client (possibly because of DB overload?).

All tests were done by slo tester build from this PR like this:

docker buildx build --platform linux/amd64 -t go-sdk-test:v2 -f Dockerfile --build-arg SRC_PATH=native/query --build-arg JOB_NAME=workload-native-query ../..

k8s cluster has 9 nodes, with 2 cpus, 4 mem and 0.5 cpu fraction (not much but the goal wasn't to achieve some exact performance numbers). To bootstrap ydb cluster I used manifests from slo repo.

Tests were performed manually without introducing any artificial failures (as automated slo test does).

Tests with native-query (YDB_GO_SDK_QUERY_SERVICE_USE_SESSION_POOL is NOT set)

1. read rps 500, write rps 100

logs

program started
db init ok
get entry error: non-retryable error occurred on attempt No.1 (idempotent=true): context deadline exceeded at `github.com/ydb-platform/ydb-go-sdk/v3/internal/query.(*result).nextResultSet(result.go:174)` at `github.com/ydb-platform/ydb-go-sdk/v3/internal/query.do.func1(client.go:122)` at `github.com/ydb-platform/ydb-go-sdk/v3/internal/query.(*poolStub).With.func2(client.go:83)` at `github.com/ydb-platform/ydb-go-sdk/v3/retry.Retry.func1(retry.go:264)` at `github.com/ydb-platform/ydb-go-sdk/v3/retry.opWithRecover(retry.go:411)` at `github.com/ydb-platform/ydb-go-sdk/v3/retry.RetryWithResult(retry.go:356)` at `github.com/ydb-platform/ydb-go-sdk/v3/retry.Retry(retry.go:270)` at `github.com/ydb-platform/ydb-go-sdk/v3/internal/query.(*poolStub).With(client.go:89)` at `github.com/ydb-platform/ydb-go-sdk/v3/internal/query.do(client.go:130)`
get entry error: non-retryable error occurred on attempt No.1 (idempotent=true): context deadline exceeded at `github.com/ydb-platform/ydb-go-sdk/v3/internal/query.do.func1(client.go:122)` at `github.com/ydb-platform/ydb-go-sdk/v3/internal/query.(*poolStub).With.func2(client.go:83)` at `github.com/ydb-platform/ydb-go-sdk/v3/retry.Retry.func1(retry.go:264)` at `github.com/ydb-platform/ydb-go-sdk/v3/retry.opWithRecover(retry.go:411)` at `github.com/ydb-platform/ydb-go-sdk/v3/retry.RetryWithResult(retry.go:356)` at `github.com/ydb-platform/ydb-go-sdk/v3/retry.Retry(retry.go:270)` at `github.com/ydb-platform/ydb-go-sdk/v3/internal/query.(*poolStub).With(client.go:89)` at `github.com/ydb-platform/ydb-go-sdk/v3/internal/query.do(client.go:130)`
get entry error: non-retryable error occurred on attempt No.1 (idempotent=true): context deadline exceeded at `github.com/ydb-platform/ydb-go-sdk/v3/internal/query.(*result).nextResultSet(result.go:174)` at `github.com/ydb-platform/ydb-go-sdk/v3/internal/query.do.func1(client.go:122)` at `github.com/ydb-platform/ydb-go-sdk/v3/internal/query.(*poolStub).With.func2(client.go:83)` at `github.com/ydb-platform/ydb-go-sdk/v3/retry.Retry.func1(retry.go:264)` at `github.com/ydb-platform/ydb-go-sdk/v3/retry.opWithRecover(retry.go:411)` at `github.com/ydb-platform/ydb-go-sdk/v3/retry.RetryWithResult(retry.go:356)` at `github.com/ydb-platform/ydb-go-sdk/v3/retry.Retry(retry.go:270)` at `github.com/ydb-platform/ydb-go-sdk/v3/internal/query.(*poolStub).With(client.go:89)` at `github.com/ydb-platform/ydb-go-sdk/v3/internal/query.do(client.go:130)`
workers close ok
program finished

2. 100 read rps, 100 write rps

logs

program started
db init ok
workers close ok
program finished

3. 1000 read rps, 400 write rps

logs

program started
db init ok
get entry error: ["attempt No.1: context deadline exceeded","connError{node_id:50001,address:'database-sample-1.database-sample-interconnect.default.svc.cluster.local:2135'}: transport/DeadlineExceeded (code = 4, source error = \"rpc error: code = DeadlineExceeded desc = context deadline exceeded\", address: \"database-sample-1.database-sample-interconnect.default.svc.cluster.local:2135\", traceID: \"366d6940-5052-4cc1-9879-47fa9c276fdd\") at `github.com/ydb-platform/ydb-go-sdk/v3/internal/conn.invoke(conn.go:356)` at `github.com/ydb-platform/ydb-go-sdk/v3/internal/balancer.(*Balancer).wrapCall(balancer.go:345)` at `github.com/ydb-platform/ydb-go-sdk/v3/internal/query.createSession(session.go:107)` at `github.com/ydb-platform/ydb-go-sdk/v3/internal/query.New.func1(client.go:380)` at `github.com/ydb-platform/ydb-go-sdk/v3/internal/query.(*poolStub).With.func2(client.go:75)` at `github.com/ydb-platform/ydb-go-sdk/v3/retry.Retry.func1(retry.go:264)` at `github.com/ydb-platform/ydb-go-sdk/v3/retry.opWithRecover(retry.go:411)`"] at `github.com/ydb-platform/ydb-go-sdk/v3/retry.RetryWithResult(retry.go:371)` at `github.com/ydb-platform/ydb-go-sdk/v3/retry.Retry(retry.go:270)` at `github.com/ydb-platform/ydb-go-sdk/v3/internal/query.(*poolStub).With(client.go:89)` at `github.com/ydb-platform/ydb-go-sdk/v3/internal/query.do(client.go:130)`
get entry error: non-retryable error occurred on attempt No.1 (idempotent=true): context deadline exceeded at `github.com/ydb-platform/ydb-go-sdk/v3/internal/query.do.func1(client.go:122)` at `github.com/ydb-platform/ydb-go-sdk/v3/internal/query.(*poolStub).With.func2(client.go:83)` at `github.com/ydb-platform/ydb-go-sdk/v3/retry.Retry.func1(retry.go:264)` at `github.com/ydb-platform/ydb-go-sdk/v3/retry.opWithRecover(retry.go:411)` at `github.com/ydb-platform/ydb-go-sdk/v3/retry.RetryWithResult(retry.go:356)` at `github.com/ydb-platform/ydb-go-sdk/v3/retry.Retry(retry.go:270)` at `github.com/ydb-platform/ydb-go-sdk/v3/internal/query.(*poolStub).With(client.go:89)` at `github.com/ydb-platform/ydb-go-sdk/v3/internal/query.do(client.go:130)`
get entry error: non-retryable error occurred on attempt No.1 (idempotent=true): context deadline exceeded at `github.com/ydb-platform/ydb-go-sdk/v3/internal/query.(*result).nextResultSet(result.go:174)` at `github.com/ydb-platform/ydb-go-sdk/v3/internal/query.do.func1(client.go:122)` at `github.com/ydb-platform/ydb-go-sdk/v3/internal/query.(*poolStub).With.func2(client.go:83)` at `github.com/ydb-platform/ydb-go-sdk/v3/retry.Retry.func1(retry.go:264)` at `github.com/ydb-platform/ydb-go-sdk/v3/retry.opWithRecover(retry.go:411)` at `github.com/ydb-platform/ydb-go-sdk/v3/retry.RetryWithResult(retry.go:356)` at `github.com/ydb-platform/ydb-go-sdk/v3/retry.Retry(retry.go:270)` at `github.com/ydb-platform/ydb-go-sdk/v3/internal/query.(*poolStub).With(client.go:89)` at `github.com/ydb-platform/ydb-go-sdk/v3/internal/query.do(client.go:130)`
get entry error: non-retryable error occurred on attempt No.1 (idempotent=true): context deadline exceeded at `github.com/ydb-platform/ydb-go-sdk/v3/internal/query.(*result).nextResultSet(result.go:174)` at `github.com/ydb-platform/ydb-go-sdk/v3/internal/query.do.func1(client.go:122)` at `github.com/ydb-platform/ydb-go-sdk/v3/internal/query.(*poolStub).With.func2(client.go:83)` at `github.com/ydb-platform/ydb-go-sdk/v3/retry.Retry.func1(retry.go:264)` at `github.com/ydb-platform/ydb-go-sdk/v3/retry.opWithRecover(retry.go:411)` at `github.com/ydb-platform/ydb-go-sdk/v3/retry.RetryWithResult(retry.go:356)` at `github.com/ydb-platform/ydb-go-sdk/v3/retry.Retry(retry.go:270)` at `github.com/ydb-platform/ydb-go-sdk/v3/internal/query.(*poolStub).With(client.go:89)` at `github.com/ydb-platform/ydb-go-sdk/v3/internal/query.do(client.go:130)`
get entry error: non-retryable error occurred on attempt No.1 (idempotent=true): context deadline exceeded at `github.com/ydb-platform/ydb-go-sdk/v3/internal/query.(*result).nextResultSet(result.go:174)` at `github.com/ydb-platform/ydb-go-sdk/v3/internal/query.do.func1(client.go:122)` at `github.com/ydb-platform/ydb-go-sdk/v3/internal/query.(*poolStub).With.func2(client.go:83)` at `github.com/ydb-platform/ydb-go-sdk/v3/retry.Retry.func1(retry.go:264)` at `github.com/ydb-platform/ydb-go-sdk/v3/retry.opWithRecover(retry.go:411)` at `github.com/ydb-platform/ydb-go-sdk/v3/retry.RetryWithResult(retry.go:356)` at `github.com/ydb-platform/ydb-go-sdk/v3/retry.Retry(retry.go:270)` at `github.com/ydb-platform/ydb-go-sdk/v3/internal/query.(*poolStub).With(client.go:89)` at `github.com/ydb-platform/ydb-go-sdk/v3/internal/query.do(client.go:130)`
get entry error: non-retryable error occurred on attempt No.1 (idempotent=true): context deadline exceeded at `github.com/ydb-platform/ydb-go-sdk/v3/internal/query.(*result).nextResultSet(result.go:174)` at `github.com/ydb-platform/ydb-go-sdk/v3/internal/query.do.func1(client.go:122)` at `github.com/ydb-platform/ydb-go-sdk/v3/internal/query.(*poolStub).With.func2(client.go:83)` at `github.com/ydb-platform/ydb-go-sdk/v3/retry.Retry.func1(retry.go:264)` at `github.com/ydb-platform/ydb-go-sdk/v3/retry.opWithRecover(retry.go:411)` at `github.com/ydb-platform/ydb-go-sdk/v3/retry.RetryWithResult(retry.go:356)` at `github.com/ydb-platform/ydb-go-sdk/v3/retry.Retry(retry.go:270)` at `github.com/ydb-platform/ydb-go-sdk/v3/internal/query.(*poolStub).With(client.go:89)` at `github.com/ydb-platform/ydb-go-sdk/v3/internal/query.do(client.go:130)`
get entry error: non-retryable error occurred on attempt No.1 (idempotent=true): context deadline exceeded at `github.com/ydb-platform/ydb-go-sdk/v3/internal/query.(*result).nextResultSet(result.go:174)` at `github.com/ydb-platform/ydb-go-sdk/v3/internal/query.do.func1(client.go:122)` at `github.com/ydb-platform/ydb-go-sdk/v3/internal/query.(*poolStub).With.func2(client.go:83)` at `github.com/ydb-platform/ydb-go-sdk/v3/retry.Retry.func1(retry.go:264)` at `github.com/ydb-platform/ydb-go-sdk/v3/retry.opWithRecover(retry.go:411)` at `github.com/ydb-platform/ydb-go-sdk/v3/retry.RetryWithResult(retry.go:356)` at `github.com/ydb-platform/ydb-go-sdk/v3/retry.Retry(retry.go:270)` at `github.com/ydb-platform/ydb-go-sdk/v3/internal/query.(*poolStub).With(client.go:89)` at `github.com/ydb-platform/ydb-go-sdk/v3/internal/query.do(client.go:130)`
get entry error: non-retryable error occurred on attempt No.1 (idempotent=true): context deadline exceeded at `github.com/ydb-platform/ydb-go-sdk/v3/internal/query.(*result).nextResultSet(result.go:174)` at `github.com/ydb-platform/ydb-go-sdk/v3/internal/query.do.func1(client.go:122)` at `github.com/ydb-platform/ydb-go-sdk/v3/internal/query.(*poolStub).With.func2(client.go:83)` at `github.com/ydb-platform/ydb-go-sdk/v3/retry.Retry.func1(retry.go:264)` at `github.com/ydb-platform/ydb-go-sdk/v3/retry.opWithRecover(retry.go:411)` at `github.com/ydb-platform/ydb-go-sdk/v3/retry.RetryWithResult(retry.go:356)` at `github.com/ydb-platform/ydb-go-sdk/v3/retry.Retry(retry.go:270)` at `github.com/ydb-platform/ydb-go-sdk/v3/internal/query.(*poolStub).With(client.go:89)` at `github.com/ydb-platform/ydb-go-sdk/v3/internal/query.do(client.go:130)`
get entry error: non-retryable error occurred on attempt No.1 (idempotent=true): context deadline exceeded at `github.com/ydb-platform/ydb-go-sdk/v3/internal/query.(*result).nextResultSet(result.go:174)` at `github.com/ydb-platform/ydb-go-sdk/v3/internal/query.do.func1(client.go:122)` at `github.com/ydb-platform/ydb-go-sdk/v3/internal/query.(*poolStub).With.func2(client.go:83)` at `github.com/ydb-platform/ydb-go-sdk/v3/retry.Retry.func1(retry.go:264)` at `github.com/ydb-platform/ydb-go-sdk/v3/retry.opWithRecover(retry.go:411)` at `github.com/ydb-platform/ydb-go-sdk/v3/retry.RetryWithResult(retry.go:356)` at `github.com/ydb-platform/ydb-go-sdk/v3/retry.Retry(retry.go:270)` at `github.com/ydb-platform/ydb-go-sdk/v3/internal/query.(*poolStub).With(client.go:89)` at `github.com/ydb-platform/ydb-go-sdk/v3/internal/query.do(client.go:130)`
error when stop 'write' worker: non-retryable error occurred on attempt No.1 (idempotent=true): context deadline exceeded at `github.com/ydb-platform/ydb-go-sdk/v3/internal/query.do.func1(client.go:122)` at `github.com/ydb-platform/ydb-go-sdk/v3/internal/query.(*poolStub).With.func2(client.go:83)` at `github.com/ydb-platform/ydb-go-sdk/v3/retry.Retry.func1(retry.go:264)` at `github.com/ydb-platform/ydb-go-sdk/v3/retry.opWithRecover(retry.go:411)` at `github.com/ydb-platform/ydb-go-sdk/v3/retry.RetryWithResult(retry.go:356)` at `github.com/ydb-platform/ydb-go-sdk/v3/retry.Retry(retry.go:270)` at `github.com/ydb-platform/ydb-go-sdk/v3/internal/query.(*poolStub).With(client.go:89)` at `github.com/ydb-platform/ydb-go-sdk/v3/internal/query.do(client.go:130)`
workers close ok
program finished

Tests with native-query (YDB_GO_SDK_QUERY_SERVICE_USE_SESSION_POOL is set)

1. read rps 300, write rps 100

logs

program started
db init ok

// after calls finished, test froze here and error flood on ydb's side has stoped

get entry error: non-retryable error occurred on attempt No.1 (idempotent=true): connError{node_id:50005,address:'database-sample-4.database-sample-interconnect.default.svc.cluster.local:2135'}: operation/TIMEOUT (code = 400090, address = database-sample-4.database-sample-interconnect.default.svc.cluster.local:2135, issues = [{'Query did not complete within specified timeout, session id ydb://session/3?node_id=50000&id=YmQyMWZiNTEtZmQ2NzM0NWItYzE1ODliODYtNGNiNGU3ZmI='}]) at `github.com/ydb-platform/ydb-go-sdk/v3/internal/conn.(*grpcClientStream).RecvMsg(grpc_client_stream.go:178)` at `github.com/ydb-platform/ydb-go-sdk/v3/internal/query.nextPart(result.go:143)` at `github.com/ydb-platform/ydb-go-sdk/v3/internal/query.newResult(result.go:101)` at `github.com/ydb-platform/ydb-go-sdk/v3/internal/query.execute(execute_query.go:74)` at `github.com/ydb-platform/ydb-go-sdk/v3/internal/query.(*Session).Execute(session.go:316)` at `github.com/ydb-platform/ydb-go-sdk/v3/internal/query.do.func1(client.go:122)` at `github.com/ydb-platform/ydb-go-sdk/v3/internal/pool.(*Pool).try(pool.go:465)` at `github.com/ydb-platform/ydb-go-sdk/v3/internal/pool.(*Pool).With.func2(pool.go:493)` at `github.com/ydb-platform/ydb-go-sdk/v3/retry.Retry.func1(retry.go:264)` at `github.com/ydb-platform/ydb-go-sdk/v3/retry.opWithRecover(retry.go:411)` at `github.com/ydb-platform/ydb-go-sdk/v3/retry.RetryWithResult(retry.go:356)` at `github.com/ydb-platform/ydb-go-sdk/v3/retry.Retry(retry.go:270)` at `github.com/ydb-platform/ydb-go-sdk/v3/internal/pool.(*Pool).With(pool.go:505)` at `github.com/ydb-platform/ydb-go-sdk/v3/internal/query.do(client.go:130)`
workers close ok
program finished

1. read rps 50, write rps 400

logs

program started
db init ok
workers close ok
program finished

2. 500 read rps, 100 write rps

logs

program started
db init ok
get entry error: non-retryable error occurred on attempt No.1 (idempotent=true): context deadline exceeded at `github.com/ydb-platform/ydb-go-sdk/v3/internal/query.(*result).nextResultSet(result.go:174)` at `github.com/ydb-platform/ydb-go-sdk/v3/internal/query.do.func1(client.go:122)` at `github.com/ydb-platform/ydb-go-sdk/v3/internal/pool.(*Pool).try(pool.go:465)` at `github.com/ydb-platform/ydb-go-sdk/v3/internal/pool.(*Pool).With.func2(pool.go:493)` at `github.com/ydb-platform/ydb-go-sdk/v3/retry.Retry.func1(retry.go:264)` at `github.com/ydb-platform/ydb-go-sdk/v3/retry.opWithRecover(retry.go:411)` at `github.com/ydb-platform/ydb-go-sdk/v3/retry.RetryWithResult(retry.go:356)` at `github.com/ydb-platform/ydb-go-sdk/v3/retry.Retry(retry.go:270)` at `github.com/ydb-platform/ydb-go-sdk/v3/internal/pool.(*Pool).With(pool.go:505)` at `github.com/ydb-platform/ydb-go-sdk/v3/internal/query.do(client.go:130)`
get entry error: non-retryable error occurred on attempt No.1 (idempotent=true): context deadline exceeded at `github.com/ydb-platform/ydb-go-sdk/v3/internal/query.(*result).nextResultSet(result.go:174)` at `github.com/ydb-platform/ydb-go-sdk/v3/internal/query.do.func1(client.go:122)` at `github.com/ydb-platform/ydb-go-sdk/v3/internal/pool.(*Pool).try(pool.go:465)` at `github.com/ydb-platform/ydb-go-sdk/v3/internal/pool.(*Pool).With.func2(pool.go:493)` at `github.com/ydb-platform/ydb-go-sdk/v3/retry.Retry.func1(retry.go:264)` at `github.com/ydb-platform/ydb-go-sdk/v3/retry.opWithRecover(retry.go:411)` at `github.com/ydb-platform/ydb-go-sdk/v3/retry.RetryWithResult(retry.go:356)` at `github.com/ydb-platform/ydb-go-sdk/v3/retry.Retry(retry.go:270)` at `github.com/ydb-platform/ydb-go-sdk/v3/internal/pool.(*Pool).With(pool.go:505)` at `github.com/ydb-platform/ydb-go-sdk/v3/internal/query.do(client.go:130)`

// froze here

get entry error: non-retryable error occurred on attempt No.2 (idempotent=true): connError{node_id:50000,address:'database-sample-0.database-sample-interconnect.default.svc.cluster.local:2135'}: operation/TIMEOUT (code = 400090, address = database-sample-0.database-sample-interconnect.default.svc.cluster.local:2135, issues = [{'Query did not complete within specified timeout, session id ydb://session/3?node_id=50001&id=ZGI2ZWJiYjctNTgwMjI2ZDEtNDJkY2Q5YjAtNjExMzlkY2U='}]) at `github.com/ydb-platform/ydb-go-sdk/v3/internal/conn.(*grpcClientStream).RecvMsg(grpc_client_stream.go:178)` at `github.com/ydb-platform/ydb-go-sdk/v3/internal/query.nextPart(result.go:143)` at `github.com/ydb-platform/ydb-go-sdk/v3/internal/query.newResult(result.go:101)` at `github.com/ydb-platform/ydb-go-sdk/v3/internal/query.execute(execute_query.go:74)` at `github.com/ydb-platform/ydb-go-sdk/v3/internal/query.(*Session).Execute(session.go:316)` at `github.com/ydb-platform/ydb-go-sdk/v3/internal/query.do.func1(client.go:122)` at `github.com/ydb-platform/ydb-go-sdk/v3/internal/pool.(*Pool).try(pool.go:465)` at `github.com/ydb-platform/ydb-go-sdk/v3/internal/pool.(*Pool).With.func2(pool.go:493)` at `github.com/ydb-platform/ydb-go-sdk/v3/retry.Retry.func1(retry.go:264)` at `github.com/ydb-platform/ydb-go-sdk/v3/retry.opWithRecover(retry.go:411)` at `github.com/ydb-platform/ydb-go-sdk/v3/retry.RetryWithResult(retry.go:356)` at `github.com/ydb-platform/ydb-go-sdk/v3/retry.Retry(retry.go:270)` at `github.com/ydb-platform/ydb-go-sdk/v3/internal/pool.(*Pool).With(pool.go:505)` at `github.com/ydb-platform/ydb-go-sdk/v3/internal/query.do(client.go:130)`
get entry error: non-retryable error occurred on attempt No.1 (idempotent=true): connError{node_id:50004,address:'database-sample-2.database-sample-interconnect.default.svc.cluster.local:2135'}: operation/TIMEOUT (code = 400090, address = database-sample-2.database-sample-interconnect.default.svc.cluster.local:2135, issues = [{'Query did not complete within specified timeout, session id ydb://session/3?node_id=50004&id=ZGMyYjllZTItZTZhY2FhODEtYzVhN2E3ZmYtMTBhYjQwN2M='}]) at `github.com/ydb-platform/ydb-go-sdk/v3/internal/conn.(*grpcClientStream).RecvMsg(grpc_client_stream.go:178)` at `github.com/ydb-platform/ydb-go-sdk/v3/internal/query.nextPart(result.go:143)` at `github.com/ydb-platform/ydb-go-sdk/v3/internal/query.newResult(result.go:101)` at `github.com/ydb-platform/ydb-go-sdk/v3/internal/query.execute(execute_query.go:74)` at `github.com/ydb-platform/ydb-go-sdk/v3/internal/query.(*Session).Execute(session.go:316)` at `github.com/ydb-platform/ydb-go-sdk/v3/internal/query.do.func1(client.go:122)` at `github.com/ydb-platform/ydb-go-sdk/v3/internal/pool.(*Pool).try(pool.go:465)` at `github.com/ydb-platform/ydb-go-sdk/v3/internal/pool.(*Pool).With.func2(pool.go:493)` at `github.com/ydb-platform/ydb-go-sdk/v3/retry.Retry.func1(retry.go:264)` at `github.com/ydb-platform/ydb-go-sdk/v3/retry.opWithRecover(retry.go:411)` at `github.com/ydb-platform/ydb-go-sdk/v3/retry.RetryWithResult(retry.go:356)` at `github.com/ydb-platform/ydb-go-sdk/v3/retry.Retry(retry.go:270)` at `github.com/ydb-platform/ydb-go-sdk/v3/internal/pool.(*Pool).With(pool.go:505)` at `github.com/ydb-platform/ydb-go-sdk/v3/internal/query.do(client.go:130)`
get entry error: non-retryable error occurred on attempt No.1 (idempotent=true): connError{node_id:50005,address:'database-sample-4.database-sample-interconnect.default.svc.cluster.local:2135'}: operation/TIMEOUT (code = 400090, address = database-sample-4.database-sample-interconnect.default.svc.cluster.local:2135, issues = [{'Query did not complete within specified timeout, session id ydb://session/3?node_id=50005&id=MmUzMDU3MjktMzA0M2RhYmEtZDNhNDFkN2QtOTEyM2JjN2Y='}]) at `github.com/ydb-platform/ydb-go-sdk/v3/internal/conn.(*grpcClientStream).RecvMsg(grpc_client_stream.go:178)` at `github.com/ydb-platform/ydb-go-sdk/v3/internal/query.nextPart(result.go:143)` at `github.com/ydb-platform/ydb-go-sdk/v3/internal/query.newResult(result.go:101)` at `github.com/ydb-platform/ydb-go-sdk/v3/internal/query.execute(execute_query.go:74)` at `github.com/ydb-platform/ydb-go-sdk/v3/internal/query.(*Session).Execute(session.go:316)` at `github.com/ydb-platform/ydb-go-sdk/v3/internal/query.do.func1(client.go:122)` at `github.com/ydb-platform/ydb-go-sdk/v3/internal/pool.(*Pool).try(pool.go:465)` at `github.com/ydb-platform/ydb-go-sdk/v3/internal/pool.(*Pool).With.func2(pool.go:493)` at `github.com/ydb-platform/ydb-go-sdk/v3/retry.Retry.func1(retry.go:264)` at `github.com/ydb-platform/ydb-go-sdk/v3/retry.opWithRecover(retry.go:411)` at `github.com/ydb-platform/ydb-go-sdk/v3/retry.RetryWithResult(retry.go:356)` at `github.com/ydb-platform/ydb-go-sdk/v3/retry.Retry(retry.go:270)` at `github.com/ydb-platform/ydb-go-sdk/v3/internal/pool.(*Pool).With(pool.go:505)` at `github.com/ydb-platform/ydb-go-sdk/v3/internal/query.do(client.go:130)`
workers close ok
program finished

3. 1000 read rps, 100 write rps

logs

program started
db init ok
get entry error: non-retryable error occurred on attempt No.1 (idempotent=true): context deadline exceeded at `github.com/ydb-platform/ydb-go-sdk/v3/internal/query.(*result).nextResultSet(result.go:174)` at `github.com/ydb-platform/ydb-go-sdk/v3/internal/query.do.func1(client.go:122)` at `github.com/ydb-platform/ydb-go-sdk/v3/internal/pool.(*Pool).try(pool.go:465)` at `github.com/ydb-platform/ydb-go-sdk/v3/internal/pool.(*Pool).With.func2(pool.go:493)` at `github.com/ydb-platform/ydb-go-sdk/v3/retry.Retry.func1(retry.go:264)` at `github.com/ydb-platform/ydb-go-sdk/v3/retry.opWithRecover(retry.go:411)` at `github.com/ydb-platform/ydb-go-sdk/v3/retry.RetryWithResult(retry.go:356)` at `github.com/ydb-platform/ydb-go-sdk/v3/retry.Retry(retry.go:270)` at `github.com/ydb-platform/ydb-go-sdk/v3/internal/pool.(*Pool).With(pool.go:505)` at `github.com/ydb-platform/ydb-go-sdk/v3/internal/query.do(client.go:130)`
get entry error: non-retryable error occurred on attempt No.1 (idempotent=true): context deadline exceeded at `github.com/ydb-platform/ydb-go-sdk/v3/internal/query.(*result).nextResultSet(result.go:174)` at `github.com/ydb-platform/ydb-go-sdk/v3/internal/query.do.func1(client.go:122)` at `github.com/ydb-platform/ydb-go-sdk/v3/internal/pool.(*Pool).try(pool.go:465)` at `github.com/ydb-platform/ydb-go-sdk/v3/internal/pool.(*Pool).With.func2(pool.go:493)` at `github.com/ydb-platform/ydb-go-sdk/v3/retry.Retry.func1(retry.go:264)` at `github.com/ydb-platform/ydb-go-sdk/v3/retry.opWithRecover(retry.go:411)` at `github.com/ydb-platform/ydb-go-sdk/v3/retry.RetryWithResult(retry.go:356)` at `github.com/ydb-platform/ydb-go-sdk/v3/retry.Retry(retry.go:270)` at `github.com/ydb-platform/ydb-go-sdk/v3/internal/pool.(*Pool).With(pool.go:505)` at `github.com/ydb-platform/ydb-go-sdk/v3/internal/query.do(client.go:130)`
get entry error: non-retryable error occurred on attempt No.1 (idempotent=true): context deadline exceeded at `github.com/ydb-platform/ydb-go-sdk/v3/internal/query.(*result).nextResultSet(result.go:174)` at `github.com/ydb-platform/ydb-go-sdk/v3/internal/query.do.func1(client.go:122)` at `github.com/ydb-platform/ydb-go-sdk/v3/internal/pool.(*Pool).try(pool.go:465)` at `github.com/ydb-platform/ydb-go-sdk/v3/internal/pool.(*Pool).With.func2(pool.go:493)` at `github.com/ydb-platform/ydb-go-sdk/v3/retry.Retry.func1(retry.go:264)` at `github.com/ydb-platform/ydb-go-sdk/v3/retry.opWithRecover(retry.go:411)` at `github.com/ydb-platform/ydb-go-sdk/v3/retry.RetryWithResult(retry.go:356)` at `github.com/ydb-platform/ydb-go-sdk/v3/retry.Retry(retry.go:270)` at `github.com/ydb-platform/ydb-go-sdk/v3/internal/pool.(*Pool).With(pool.go:505)` at `github.com/ydb-platform/ydb-go-sdk/v3/internal/query.do(client.go:130)`

// froze here

get entry error: non-retryable error occurred on attempt No.1 (idempotent=true): connError{node_id:50002,address:'database-sample-5.database-sample-interconnect.default.svc.cluster.local:2135'}: operation/TIMEOUT (code = 400090, address = database-sample-5.database-sample-interconnect.default.svc.cluster.local:2135, issues = [{'Query did not complete within specified timeout, session id ydb://session/3?node_id=50003&id=YjBlM2JjOTUtYWZkZGJlMDUtOWE1MTI0MDctOTJhZTEzMmU='}]) at `github.com/ydb-platform/ydb-go-sdk/v3/internal/conn.(*grpcClientStream).RecvMsg(grpc_client_stream.go:178)` at `github.com/ydb-platform/ydb-go-sdk/v3/internal/query.nextPart(result.go:143)` at `github.com/ydb-platform/ydb-go-sdk/v3/internal/query.newResult(result.go:101)` at `github.com/ydb-platform/ydb-go-sdk/v3/internal/query.execute(execute_query.go:74)` at `github.com/ydb-platform/ydb-go-sdk/v3/internal/query.(*Session).Execute(session.go:316)` at `github.com/ydb-platform/ydb-go-sdk/v3/internal/query.do.func1(client.go:122)` at `github.com/ydb-platform/ydb-go-sdk/v3/internal/pool.(*Pool).try(pool.go:465)` at `github.com/ydb-platform/ydb-go-sdk/v3/internal/pool.(*Pool).With.func2(pool.go:493)` at `github.com/ydb-platform/ydb-go-sdk/v3/retry.Retry.func1(retry.go:264)` at `github.com/ydb-platform/ydb-go-sdk/v3/retry.opWithRecover(retry.go:411)` at `github.com/ydb-platform/ydb-go-sdk/v3/retry.RetryWithResult(retry.go:356)` at `github.com/ydb-platform/ydb-go-sdk/v3/retry.Retry(retry.go:270)` at `github.com/ydb-platform/ydb-go-sdk/v3/internal/pool.(*Pool).With(pool.go:505)` at `github.com/ydb-platform/ydb-go-sdk/v3/internal/query.do(client.go:130)`
error when stop 'write' worker: non-retryable error occurred on attempt No.1 (idempotent=true): connError{node_id:50002,address:'database-sample-5.database-sample-interconnect.default.svc.cluster.local:2135'}: operation/TIMEOUT (code = 400090, address = database-sample-5.database-sample-interconnect.default.svc.cluster.local:2135, issues = [{'Query did not complete within specified timeout, session id ydb://session/3?node_id=50004&id=ZWY2NDQ4ZWQtYmM1ODEwOTgtYmNjZjliZGMtOTAzNTAyZTM='}]) at `github.com/ydb-platform/ydb-go-sdk/v3/internal/conn.(*grpcClientStream).RecvMsg(grpc_client_stream.go:178)` at `github.com/ydb-platform/ydb-go-sdk/v3/internal/query.nextPart(result.go:143)` at `github.com/ydb-platform/ydb-go-sdk/v3/internal/query.newResult(result.go:101)` at `github.com/ydb-platform/ydb-go-sdk/v3/internal/query.execute(execute_query.go:74)` at `github.com/ydb-platform/ydb-go-sdk/v3/internal/query.(*Session).Execute(session.go:316)` at `github.com/ydb-platform/ydb-go-sdk/v3/internal/query.do.func1(client.go:122)` at `github.com/ydb-platform/ydb-go-sdk/v3/internal/pool.(*Pool).try(pool.go:465)` at `github.com/ydb-platform/ydb-go-sdk/v3/internal/pool.(*Pool).With.func2(pool.go:493)` at `github.com/ydb-platform/ydb-go-sdk/v3/retry.Retry.func1(retry.go:264)` at `github.com/ydb-platform/ydb-go-sdk/v3/retry.opWithRecover(retry.go:411)` at `github.com/ydb-platform/ydb-go-sdk/v3/retry.RetryWithResult(retry.go:356)` at `github.com/ydb-platform/ydb-go-sdk/v3/retry.Retry(retry.go:270)` at `github.com/ydb-platform/ydb-go-sdk/v3/internal/pool.(*Pool).With(pool.go:505)` at `github.com/ydb-platform/ydb-go-sdk/v3/internal/query.do(client.go:130)`
get entry error: non-retryable error occurred on attempt No.1 (idempotent=true): connError{node_id:50001,address:'database-sample-1.database-sample-interconnect.default.svc.cluster.local:2135'}: operation/TIMEOUT (code = 400090, address = database-sample-1.database-sample-interconnect.default.svc.cluster.local:2135, issues = [{'Query did not complete within specified timeout, session id ydb://session/3?node_id=50005&id=ZjU2NjIyM2ItYWM2ZTNhNTMtZjZkODZkODQtNGY4YTJlMTY='}]) at `github.com/ydb-platform/ydb-go-sdk/v3/internal/conn.(*grpcClientStream).RecvMsg(grpc_client_stream.go:178)` at `github.com/ydb-platform/ydb-go-sdk/v3/internal/query.nextPart(result.go:143)` at `github.com/ydb-platform/ydb-go-sdk/v3/internal/query.newResult(result.go:101)` at `github.com/ydb-platform/ydb-go-sdk/v3/internal/query.execute(execute_query.go:74)` at `github.com/ydb-platform/ydb-go-sdk/v3/internal/query.(*Session).Execute(session.go:316)` at `github.com/ydb-platform/ydb-go-sdk/v3/internal/query.do.func1(client.go:122)` at `github.com/ydb-platform/ydb-go-sdk/v3/internal/pool.(*Pool).try(pool.go:465)` at `github.com/ydb-platform/ydb-go-sdk/v3/internal/pool.(*Pool).With.func2(pool.go:493)` at `github.com/ydb-platform/ydb-go-sdk/v3/retry.Retry.func1(retry.go:264)` at `github.com/ydb-platform/ydb-go-sdk/v3/retry.opWithRecover(retry.go:411)` at `github.com/ydb-platform/ydb-go-sdk/v3/retry.RetryWithResult(retry.go:356)` at `github.com/ydb-platform/ydb-go-sdk/v3/retry.Retry(retry.go:270)` at `github.com/ydb-platform/ydb-go-sdk/v3/internal/pool.(*Pool).With(pool.go:505)` at `github.com/ydb-platform/ydb-go-sdk/v3/internal/query.do(client.go:130)`
get entry error: non-retryable error occurred on attempt No.1 (idempotent=true): connError{node_id:50002,address:'database-sample-5.database-sample-interconnect.default.svc.cluster.local:2135'}: operation/TIMEOUT (code = 400090, address = database-sample-5.database-sample-interconnect.default.svc.cluster.local:2135, issues = [{'Query did not complete within specified timeout, session id ydb://session/3?node_id=50000&id=MjgzOTg1ODYtOTJlZGExMmUtNDhiNTI3YTYtNmQ2Y2FmNWE='}]) at `github.com/ydb-platform/ydb-go-sdk/v3/internal/conn.(*grpcClientStream).RecvMsg(grpc_client_stream.go:178)` at `github.com/ydb-platform/ydb-go-sdk/v3/internal/query.nextPart(result.go:143)` at `github.com/ydb-platform/ydb-go-sdk/v3/internal/query.newResult(result.go:101)` at `github.com/ydb-platform/ydb-go-sdk/v3/internal/query.execute(execute_query.go:74)` at `github.com/ydb-platform/ydb-go-sdk/v3/internal/query.(*Session).Execute(session.go:316)` at `github.com/ydb-platform/ydb-go-sdk/v3/internal/query.do.func1(client.go:122)` at `github.com/ydb-platform/ydb-go-sdk/v3/internal/pool.(*Pool).try(pool.go:465)` at `github.com/ydb-platform/ydb-go-sdk/v3/internal/pool.(*Pool).With.func2(pool.go:493)` at `github.com/ydb-platform/ydb-go-sdk/v3/retry.Retry.func1(retry.go:264)` at `github.com/ydb-platform/ydb-go-sdk/v3/retry.opWithRecover(retry.go:411)` at `github.com/ydb-platform/ydb-go-sdk/v3/retry.RetryWithResult(retry.go:356)` at `github.com/ydb-platform/ydb-go-sdk/v3/retry.Retry(retry.go:270)` at `github.com/ydb-platform/ydb-go-sdk/v3/internal/pool.(*Pool).With(pool.go:505)` at `github.com/ydb-platform/ydb-go-sdk/v3/internal/query.do(client.go:130)`
get entry error: non-retryable error occurred on attempt No.2 (idempotent=true): connError{node_id:50003,address:'database-sample-3.database-sample-interconnect.default.svc.cluster.local:2135'}: operation/TIMEOUT (code = 400090, address = database-sample-3.database-sample-interconnect.default.svc.cluster.local:2135, issues = [{'Query did not complete within specified timeout, session id ydb://session/3?node_id=50001&id=YzA5ODk0ZWEtYWViNzM0ZjgtN2NkZGFjYzQtNmI3MmEwZjI='}]) at `github.com/ydb-platform/ydb-go-sdk/v3/internal/conn.(*grpcClientStream).RecvMsg(grpc_client_stream.go:178)` at `github.com/ydb-platform/ydb-go-sdk/v3/internal/query.nextPart(result.go:143)` at `github.com/ydb-platform/ydb-go-sdk/v3/internal/query.newResult(result.go:101)` at `github.com/ydb-platform/ydb-go-sdk/v3/internal/query.execute(execute_query.go:74)` at `github.com/ydb-platform/ydb-go-sdk/v3/internal/query.(*Session).Execute(session.go:316)` at `github.com/ydb-platform/ydb-go-sdk/v3/internal/query.do.func1(client.go:122)` at `github.com/ydb-platform/ydb-go-sdk/v3/internal/pool.(*Pool).try(pool.go:465)` at `github.com/ydb-platform/ydb-go-sdk/v3/internal/pool.(*Pool).With.func2(pool.go:493)` at `github.com/ydb-platform/ydb-go-sdk/v3/retry.Retry.func1(retry.go:264)` at `github.com/ydb-platform/ydb-go-sdk/v3/retry.opWithRecover(retry.go:411)` at `github.com/ydb-platform/ydb-go-sdk/v3/retry.RetryWithResult(retry.go:356)` at `github.com/ydb-platform/ydb-go-sdk/v3/retry.Retry(retry.go:270)` at `github.com/ydb-platform/ydb-go-sdk/v3/internal/pool.(*Pool).With(pool.go:505)` at `github.com/ydb-platform/ydb-go-sdk/v3/internal/query.do(client.go:130)`
get entry error: non-retryable error occurred on attempt No.1 (idempotent=true): connError{node_id:50000,address:'database-sample-0.database-sample-interconnect.default.svc.cluster.local:2135'}: operation/TIMEOUT (code = 400090, address = database-sample-0.database-sample-interconnect.default.svc.cluster.local:2135, issues = [{'Query did not complete within specified timeout, session id ydb://session/3?node_id=50004&id=YzQ2ZGMwNjUtMzI5ZDc0ZWYtYjAxMWViNjktNWY3NDQ3ZTI='}]) at `github.com/ydb-platform/ydb-go-sdk/v3/internal/conn.(*grpcClientStream).RecvMsg(grpc_client_stream.go:178)` at `github.com/ydb-platform/ydb-go-sdk/v3/internal/query.nextPart(result.go:143)` at `github.com/ydb-platform/ydb-go-sdk/v3/internal/query.newResult(result.go:101)` at `github.com/ydb-platform/ydb-go-sdk/v3/internal/query.execute(execute_query.go:74)` at `github.com/ydb-platform/ydb-go-sdk/v3/internal/query.(*Session).Execute(session.go:316)` at `github.com/ydb-platform/ydb-go-sdk/v3/internal/query.do.func1(client.go:122)` at `github.com/ydb-platform/ydb-go-sdk/v3/internal/pool.(*Pool).try(pool.go:465)` at `github.com/ydb-platform/ydb-go-sdk/v3/internal/pool.(*Pool).With.func2(pool.go:493)` at `github.com/ydb-platform/ydb-go-sdk/v3/retry.Retry.func1(retry.go:264)` at `github.com/ydb-platform/ydb-go-sdk/v3/retry.opWithRecover(retry.go:411)` at `github.com/ydb-platform/ydb-go-sdk/v3/retry.RetryWithResult(retry.go:356)` at `github.com/ydb-platform/ydb-go-sdk/v3/retry.Retry(retry.go:270)` at `github.com/ydb-platform/ydb-go-sdk/v3/internal/pool.(*Pool).With(pool.go:505)` at `github.com/ydb-platform/ydb-go-sdk/v3/internal/query.do(client.go:130)`
get entry error: non-retryable error occurred on attempt No.1 (idempotent=true): connError{node_id:50002,address:'database-sample-5.database-sample-interconnect.default.svc.cluster.local:2135'}: operation/TIMEOUT (code = 400090, address = database-sample-5.database-sample-interconnect.default.svc.cluster.local:2135, issues = [{'Query did not complete within specified timeout, session id ydb://session/3?node_id=50000&id=OTU4YzIyY2MtOTZmZGEwYTMtMzNhNGJlM2EtYjE1MjdkOTE='}]) at `github.com/ydb-platform/ydb-go-sdk/v3/internal/conn.(*grpcClientStream).RecvMsg(grpc_client_stream.go:178)` at `github.com/ydb-platform/ydb-go-sdk/v3/internal/query.nextPart(result.go:143)` at `github.com/ydb-platform/ydb-go-sdk/v3/internal/query.newResult(result.go:101)` at `github.com/ydb-platform/ydb-go-sdk/v3/internal/query.execute(execute_query.go:74)` at `github.com/ydb-platform/ydb-go-sdk/v3/internal/query.(*Session).Execute(session.go:316)` at `github.com/ydb-platform/ydb-go-sdk/v3/internal/query.do.func1(client.go:122)` at `github.com/ydb-platform/ydb-go-sdk/v3/internal/pool.(*Pool).try(pool.go:465)` at `github.com/ydb-platform/ydb-go-sdk/v3/internal/pool.(*Pool).With.func2(pool.go:493)` at `github.com/ydb-platform/ydb-go-sdk/v3/retry.Retry.func1(retry.go:264)` at `github.com/ydb-platform/ydb-go-sdk/v3/retry.opWithRecover(retry.go:411)` at `github.com/ydb-platform/ydb-go-sdk/v3/retry.RetryWithResult(retry.go:356)` at `github.com/ydb-platform/ydb-go-sdk/v3/retry.Retry(retry.go:270)` at `github.com/ydb-platform/ydb-go-sdk/v3/internal/pool.(*Pool).With(pool.go:505)` at `github.com/ydb-platform/ydb-go-sdk/v3/internal/query.do(client.go:130)`
get entry error: non-retryable error occurred on attempt No.2 (idempotent=true): connError{node_id:50003,address:'database-sample-3.database-sample-interconnect.default.svc.cluster.local:2135'}: operation/TIMEOUT (code = 400090, address = database-sample-3.database-sample-interconnect.default.svc.cluster.local:2135, issues = [{'Query did not complete within specified timeout, session id ydb://session/3?node_id=50001&id=ODQ3OTdkNTAtMzg4NWUxYWQtMzhlMzE2NDMtZDg5MmExNWE='}]) at `github.com/ydb-platform/ydb-go-sdk/v3/internal/conn.(*grpcClientStream).RecvMsg(grpc_client_stream.go:178)` at `github.com/ydb-platform/ydb-go-sdk/v3/internal/query.nextPart(result.go:143)` at `github.com/ydb-platform/ydb-go-sdk/v3/internal/query.newResult(result.go:101)` at `github.com/ydb-platform/ydb-go-sdk/v3/internal/query.execute(execute_query.go:74)` at `github.com/ydb-platform/ydb-go-sdk/v3/internal/query.(*Session).Execute(session.go:316)` at `github.com/ydb-platform/ydb-go-sdk/v3/internal/query.do.func1(client.go:122)` at `github.com/ydb-platform/ydb-go-sdk/v3/internal/pool.(*Pool).try(pool.go:465)` at `github.com/ydb-platform/ydb-go-sdk/v3/internal/pool.(*Pool).With.func2(pool.go:493)` at `github.com/ydb-platform/ydb-go-sdk/v3/retry.Retry.func1(retry.go:264)` at `github.com/ydb-platform/ydb-go-sdk/v3/retry.opWithRecover(retry.go:411)` at `github.com/ydb-platform/ydb-go-sdk/v3/retry.RetryWithResult(retry.go:356)` at `github.com/ydb-platform/ydb-go-sdk/v3/retry.Retry(retry.go:270)` at `github.com/ydb-platform/ydb-go-sdk/v3/internal/pool.(*Pool).With(pool.go:505)` at `github.com/ydb-platform/ydb-go-sdk/v3/internal/query.do(client.go:130)`
get entry error: non-retryable error occurred on attempt No.1 (idempotent=true): connError{node_id:50004,address:'database-sample-2.database-sample-interconnect.default.svc.cluster.local:2135'}: operation/TIMEOUT (code = 400090, address = database-sample-2.database-sample-interconnect.default.svc.cluster.local:2135, issues = [{'Query did not complete within specified timeout, session id ydb://session/3?node_id=50004&id=OTEwMzc3NTUtYzM4ZjViZGMtM2JkN2Y2MDMtN2YwZWRhMGM='}]) at `github.com/ydb-platform/ydb-go-sdk/v3/internal/conn.(*grpcClientStream).RecvMsg(grpc_client_stream.go:178)` at `github.com/ydb-platform/ydb-go-sdk/v3/internal/query.nextPart(result.go:143)` at `github.com/ydb-platform/ydb-go-sdk/v3/internal/query.newResult(result.go:101)` at `github.com/ydb-platform/ydb-go-sdk/v3/internal/query.execute(execute_query.go:74)` at `github.com/ydb-platform/ydb-go-sdk/v3/internal/query.(*Session).Execute(session.go:316)` at `github.com/ydb-platform/ydb-go-sdk/v3/internal/query.do.func1(client.go:122)` at `github.com/ydb-platform/ydb-go-sdk/v3/internal/pool.(*Pool).try(pool.go:465)` at `github.com/ydb-platform/ydb-go-sdk/v3/internal/pool.(*Pool).With.func2(pool.go:493)` at `github.com/ydb-platform/ydb-go-sdk/v3/retry.Retry.func1(retry.go:264)` at `github.com/ydb-platform/ydb-go-sdk/v3/retry.opWithRecover(retry.go:411)` at `github.com/ydb-platform/ydb-go-sdk/v3/retry.RetryWithResult(retry.go:356)` at `github.com/ydb-platform/ydb-go-sdk/v3/retry.Retry(retry.go:270)` at `github.com/ydb-platform/ydb-go-sdk/v3/internal/pool.(*Pool).With(pool.go:505)` at `github.com/ydb-platform/ydb-go-sdk/v3/internal/query.do(client.go:130)`
get entry error: non-retryable error occurred on attempt No.2 (idempotent=true): connError{node_id:50003,address:'database-sample-3.database-sample-interconnect.default.svc.cluster.local:2135'}: operation/TIMEOUT (code = 400090, address = database-sample-3.database-sample-interconnect.default.svc.cluster.local:2135, issues = [{'Query did not complete within specified timeout, session id ydb://session/3?node_id=50002&id=NDA5ZGEyNmMtNzczYzRhMDUtMjdlMWRiZmUtZmVhMzIzNzQ='}]) at `github.com/ydb-platform/ydb-go-sdk/v3/internal/conn.(*grpcClientStream).RecvMsg(grpc_client_stream.go:178)` at `github.com/ydb-platform/ydb-go-sdk/v3/internal/query.nextPart(result.go:143)` at `github.com/ydb-platform/ydb-go-sdk/v3/internal/query.newResult(result.go:101)` at `github.com/ydb-platform/ydb-go-sdk/v3/internal/query.execute(execute_query.go:74)` at `github.com/ydb-platform/ydb-go-sdk/v3/internal/query.(*Session).Execute(session.go:316)` at `github.com/ydb-platform/ydb-go-sdk/v3/internal/query.do.func1(client.go:122)` at `github.com/ydb-platform/ydb-go-sdk/v3/internal/pool.(*Pool).try(pool.go:465)` at `github.com/ydb-platform/ydb-go-sdk/v3/internal/pool.(*Pool).With.func2(pool.go:493)` at `github.com/ydb-platform/ydb-go-sdk/v3/retry.Retry.func1(retry.go:264)` at `github.com/ydb-platform/ydb-go-sdk/v3/retry.opWithRecover(retry.go:411)` at `github.com/ydb-platform/ydb-go-sdk/v3/retry.RetryWithResult(retry.go:356)` at `github.com/ydb-platform/ydb-go-sdk/v3/retry.Retry(retry.go:270)` at `github.com/ydb-platform/ydb-go-sdk/v3/internal/pool.(*Pool).With(pool.go:505)` at `github.com/ydb-platform/ydb-go-sdk/v3/internal/query.do(client.go:130)`
get entry error: non-retryable error occurred on attempt No.1 (idempotent=true): connError{node_id:50004,address:'database-sample-2.database-sample-interconnect.default.svc.cluster.local:2135'}: operation/TIMEOUT (code = 400090, address = database-sample-2.database-sample-interconnect.default.svc.cluster.local:2135, issues = [{'Query did not complete within specified timeout, session id ydb://session/3?node_id=50002&id=MmQ4OTJlNDQtNjE3NWZjOWItNGViM2Q0MWEtNGYxNDViMzM='}]) at `github.com/ydb-platform/ydb-go-sdk/v3/internal/conn.(*grpcClientStream).RecvMsg(grpc_client_stream.go:178)` at `github.com/ydb-platform/ydb-go-sdk/v3/internal/query.nextPart(result.go:143)` at `github.com/ydb-platform/ydb-go-sdk/v3/internal/query.newResult(result.go:101)` at `github.com/ydb-platform/ydb-go-sdk/v3/internal/query.execute(execute_query.go:74)` at `github.com/ydb-platform/ydb-go-sdk/v3/internal/query.(*Session).Execute(session.go:316)` at `github.com/ydb-platform/ydb-go-sdk/v3/internal/query.do.func1(client.go:122)` at `github.com/ydb-platform/ydb-go-sdk/v3/internal/pool.(*Pool).try(pool.go:465)` at `github.com/ydb-platform/ydb-go-sdk/v3/internal/pool.(*Pool).With.func2(pool.go:493)` at `github.com/ydb-platform/ydb-go-sdk/v3/retry.Retry.func1(retry.go:264)` at `github.com/ydb-platform/ydb-go-sdk/v3/retry.opWithRecover(retry.go:411)` at `github.com/ydb-platform/ydb-go-sdk/v3/retry.RetryWithResult(retry.go:356)` at `github.com/ydb-platform/ydb-go-sdk/v3/retry.Retry(retry.go:270)` at `github.com/ydb-platform/ydb-go-sdk/v3/internal/pool.(*Pool).With(pool.go:505)` at `github.com/ydb-platform/ydb-go-sdk/v3/internal/query.do(client.go:130)`
get entry error: non-retryable error occurred on attempt No.1 (idempotent=true): connError{node_id:50000,address:'database-sample-0.database-sample-interconnect.default.svc.cluster.local:2135'}: operation/TIMEOUT (code = 400090, address = database-sample-0.database-sample-interconnect.default.svc.cluster.local:2135, issues = [{'Query did not complete within specified timeout, session id ydb://session/3?node_id=50005&id=Y2YxYjFiYzMtNTk0NmNjMTMtOThlNmQ5OTQtNGY4MWQzMDk='}]) at `github.com/ydb-platform/ydb-go-sdk/v3/internal/conn.(*grpcClientStream).RecvMsg(grpc_client_stream.go:178)` at `github.com/ydb-platform/ydb-go-sdk/v3/internal/query.nextPart(result.go:143)` at `github.com/ydb-platform/ydb-go-sdk/v3/internal/query.newResult(result.go:101)` at `github.com/ydb-platform/ydb-go-sdk/v3/internal/query.execute(execute_query.go:74)` at `github.com/ydb-platform/ydb-go-sdk/v3/internal/query.(*Session).Execute(session.go:316)` at `github.com/ydb-platform/ydb-go-sdk/v3/internal/query.do.func1(client.go:122)` at `github.com/ydb-platform/ydb-go-sdk/v3/internal/pool.(*Pool).try(pool.go:465)` at `github.com/ydb-platform/ydb-go-sdk/v3/internal/pool.(*Pool).With.func2(pool.go:493)` at `github.com/ydb-platform/ydb-go-sdk/v3/retry.Retry.func1(retry.go:264)` at `github.com/ydb-platform/ydb-go-sdk/v3/retry.opWithRecover(retry.go:411)` at `github.com/ydb-platform/ydb-go-sdk/v3/retry.RetryWithResult(retry.go:356)` at `github.com/ydb-platform/ydb-go-sdk/v3/retry.Retry(retry.go:270)` at `github.com/ydb-platform/ydb-go-sdk/v3/internal/pool.(*Pool).With(pool.go:505)` at `github.com/ydb-platform/ydb-go-sdk/v3/internal/query.do(client.go:130)`
get entry error: non-retryable error occurred on attempt No.1 (idempotent=true): connError{node_id:50002,address:'database-sample-5.database-sample-interconnect.default.svc.cluster.local:2135'}: operation/TIMEOUT (code = 400090, address = database-sample-5.database-sample-interconnect.default.svc.cluster.local:2135, issues = [{'Query did not complete within specified timeout, session id ydb://session/3?node_id=50001&id=NWVmNWNiNmItMmNhYjBkNTMtYzFiODUxM2MtZWYzYzdiN2Y='}]) at `github.com/ydb-platform/ydb-go-sdk/v3/internal/conn.(*grpcClientStream).RecvMsg(grpc_client_stream.go:178)` at `github.com/ydb-platform/ydb-go-sdk/v3/internal/query.nextPart(result.go:143)` at `github.com/ydb-platform/ydb-go-sdk/v3/internal/query.newResult(result.go:101)` at `github.com/ydb-platform/ydb-go-sdk/v3/internal/query.execute(execute_query.go:74)` at `github.com/ydb-platform/ydb-go-sdk/v3/internal/query.(*Session).Execute(session.go:316)` at `github.com/ydb-platform/ydb-go-sdk/v3/internal/query.do.func1(client.go:122)` at `github.com/ydb-platform/ydb-go-sdk/v3/internal/pool.(*Pool).try(pool.go:465)` at `github.com/ydb-platform/ydb-go-sdk/v3/internal/pool.(*Pool).With.func2(pool.go:493)` at `github.com/ydb-platform/ydb-go-sdk/v3/retry.Retry.func1(retry.go:264)` at `github.com/ydb-platform/ydb-go-sdk/v3/retry.opWithRecover(retry.go:411)` at `github.com/ydb-platform/ydb-go-sdk/v3/retry.RetryWithResult(retry.go:356)` at `github.com/ydb-platform/ydb-go-sdk/v3/retry.Retry(retry.go:270)` at `github.com/ydb-platform/ydb-go-sdk/v3/internal/pool.(*Pool).With(pool.go:505)` at `github.com/ydb-platform/ydb-go-sdk/v3/internal/query.do(client.go:130)`
get entry error: non-retryable error occurred on attempt No.3 (idempotent=true): connError{node_id:50003,address:'database-sample-3.database-sample-interconnect.default.svc.cluster.local:2135'}: operation/TIMEOUT (code = 400090, address = database-sample-3.database-sample-interconnect.default.svc.cluster.local:2135, issues = [{'Query did not complete within specified timeout, session id ydb://session/3?node_id=50004&id=YzgxYzdhZDktMzIzNjI5NzQtZDgyODg0OTQtM2FhNzg1MjQ='}]) at `github.com/ydb-platform/ydb-go-sdk/v3/internal/conn.(*grpcClientStream).RecvMsg(grpc_client_stream.go:178)` at `github.com/ydb-platform/ydb-go-sdk/v3/internal/query.nextPart(result.go:143)` at `github.com/ydb-platform/ydb-go-sdk/v3/internal/query.newResult(result.go:101)` at `github.com/ydb-platform/ydb-go-sdk/v3/internal/query.execute(execute_query.go:74)` at `github.com/ydb-platform/ydb-go-sdk/v3/internal/query.(*Session).Execute(session.go:316)` at `github.com/ydb-platform/ydb-go-sdk/v3/internal/query.do.func1(client.go:122)` at `github.com/ydb-platform/ydb-go-sdk/v3/internal/pool.(*Pool).try(pool.go:465)` at `github.com/ydb-platform/ydb-go-sdk/v3/internal/pool.(*Pool).With.func2(pool.go:493)` at `github.com/ydb-platform/ydb-go-sdk/v3/retry.Retry.func1(retry.go:264)` at `github.com/ydb-platform/ydb-go-sdk/v3/retry.opWithRecover(retry.go:411)` at `github.com/ydb-platform/ydb-go-sdk/v3/retry.RetryWithResult(retry.go:356)` at `github.com/ydb-platform/ydb-go-sdk/v3/retry.Retry(retry.go:270)` at `github.com/ydb-platform/ydb-go-sdk/v3/internal/pool.(*Pool).With(pool.go:505)` at `github.com/ydb-platform/ydb-go-sdk/v3/internal/query.do(client.go:130)`
get entry error: non-retryable error occurred on attempt No.1 (idempotent=true): connError{node_id:50001,address:'database-sample-1.database-sample-interconnect.default.svc.cluster.local:2135'}: operation/TIMEOUT (code = 400090, address = database-sample-1.database-sample-interconnect.default.svc.cluster.local:2135, issues = [{'Query did not complete within specified timeout, session id ydb://session/3?node_id=50003&id=M2Q1ZGI2ZDYtNGU3NTdkOTgtYmE4YmUyOTQtZGVlZTRlYmU='}]) at `github.com/ydb-platform/ydb-go-sdk/v3/internal/conn.(*grpcClientStream).RecvMsg(grpc_client_stream.go:178)` at `github.com/ydb-platform/ydb-go-sdk/v3/internal/query.nextPart(result.go:143)` at `github.com/ydb-platform/ydb-go-sdk/v3/internal/query.newResult(result.go:101)` at `github.com/ydb-platform/ydb-go-sdk/v3/internal/query.execute(execute_query.go:74)` at `github.com/ydb-platform/ydb-go-sdk/v3/internal/query.(*Session).Execute(session.go:316)` at `github.com/ydb-platform/ydb-go-sdk/v3/internal/query.do.func1(client.go:122)` at `github.com/ydb-platform/ydb-go-sdk/v3/internal/pool.(*Pool).try(pool.go:465)` at `github.com/ydb-platform/ydb-go-sdk/v3/internal/pool.(*Pool).With.func2(pool.go:493)` at `github.com/ydb-platform/ydb-go-sdk/v3/retry.Retry.func1(retry.go:264)` at `github.com/ydb-platform/ydb-go-sdk/v3/retry.opWithRecover(retry.go:411)` at `github.com/ydb-platform/ydb-go-sdk/v3/retry.RetryWithResult(retry.go:356)` at `github.com/ydb-platform/ydb-go-sdk/v3/retry.Retry(retry.go:270)` at `github.com/ydb-platform/ydb-go-sdk/v3/internal/pool.(*Pool).With(pool.go:505)` at `github.com/ydb-platform/ydb-go-sdk/v3/internal/query.do(client.go:130)`
get entry error: non-retryable error occurred on attempt No.1 (idempotent=true): connError{node_id:50004,address:'database-sample-2.database-sample-interconnect.default.svc.cluster.local:2135'}: operation/TIMEOUT (code = 400090, address = database-sample-2.database-sample-interconnect.default.svc.cluster.local:2135, issues = [{'Query did not complete within specified timeout, session id ydb://session/3?node_id=50002&id=ZGY1YzBjMWItNGQ1MTkwNGItM2UxYTdiNWItMTY3MTE5N2M='}]) at `github.com/ydb-platform/ydb-go-sdk/v3/internal/conn.(*grpcClientStream).RecvMsg(grpc_client_stream.go:178)` at `github.com/ydb-platform/ydb-go-sdk/v3/internal/query.nextPart(result.go:143)` at `github.com/ydb-platform/ydb-go-sdk/v3/internal/query.newResult(result.go:101)` at `github.com/ydb-platform/ydb-go-sdk/v3/internal/query.execute(execute_query.go:74)` at `github.com/ydb-platform/ydb-go-sdk/v3/internal/query.(*Session).Execute(session.go:316)` at `github.com/ydb-platform/ydb-go-sdk/v3/internal/query.do.func1(client.go:122)` at `github.com/ydb-platform/ydb-go-sdk/v3/internal/pool.(*Pool).try(pool.go:465)` at `github.com/ydb-platform/ydb-go-sdk/v3/internal/pool.(*Pool).With.func2(pool.go:493)` at `github.com/ydb-platform/ydb-go-sdk/v3/retry.Retry.func1(retry.go:264)` at `github.com/ydb-platform/ydb-go-sdk/v3/retry.opWithRecover(retry.go:411)` at `github.com/ydb-platform/ydb-go-sdk/v3/retry.RetryWithResult(retry.go:356)` at `github.com/ydb-platform/ydb-go-sdk/v3/retry.Retry(retry.go:270)` at `github.com/ydb-platform/ydb-go-sdk/v3/internal/pool.(*Pool).With(pool.go:505)` at `github.com/ydb-platform/ydb-go-sdk/v3/internal/query.do(client.go:130)`
get entry error: non-retryable error occurred on attempt No.1 (idempotent=true): connError{node_id:50004,address:'database-sample-2.database-sample-interconnect.default.svc.cluster.local:2135'}: operation/TIMEOUT (code = 400090, address = database-sample-2.database-sample-interconnect.default.svc.cluster.local:2135, issues = [{'Query did not complete within specified timeout, session id ydb://session/3?node_id=50005&id=ZDNlMzk2MS05MTYwZmU4Mi04NWNkOTZhNy1iNWZkZGVlMg=='}]) at `github.com/ydb-platform/ydb-go-sdk/v3/internal/conn.(*grpcClientStream).RecvMsg(grpc_client_stream.go:178)` at `github.com/ydb-platform/ydb-go-sdk/v3/internal/query.nextPart(result.go:143)` at `github.com/ydb-platform/ydb-go-sdk/v3/internal/query.newResult(result.go:101)` at `github.com/ydb-platform/ydb-go-sdk/v3/internal/query.execute(execute_query.go:74)` at `github.com/ydb-platform/ydb-go-sdk/v3/internal/query.(*Session).Execute(session.go:316)` at `github.com/ydb-platform/ydb-go-sdk/v3/internal/query.do.func1(client.go:122)` at `github.com/ydb-platform/ydb-go-sdk/v3/internal/pool.(*Pool).try(pool.go:465)` at `github.com/ydb-platform/ydb-go-sdk/v3/internal/pool.(*Pool).With.func2(pool.go:493)` at `github.com/ydb-platform/ydb-go-sdk/v3/retry.Retry.func1(retry.go:264)` at `github.com/ydb-platform/ydb-go-sdk/v3/retry.opWithRecover(retry.go:411)` at `github.com/ydb-platform/ydb-go-sdk/v3/retry.RetryWithResult(retry.go:356)` at `github.com/ydb-platform/ydb-go-sdk/v3/retry.Retry(retry.go:270)` at `github.com/ydb-platform/ydb-go-sdk/v3/internal/pool.(*Pool).With(pool.go:505)` at `github.com/ydb-platform/ydb-go-sdk/v3/internal/query.do(client.go:130)`
get entry error: non-retryable error occurred on attempt No.2 (idempotent=true): connError{node_id:50003,address:'database-sample-3.database-sample-interconnect.default.svc.cluster.local:2135'}: operation/TIMEOUT (code = 400090, address = database-sample-3.database-sample-interconnect.default.svc.cluster.local:2135, issues = [{'Query did not complete within specified timeout, session id ydb://session/3?node_id=50005&id=NjcyMmRjNWQtY2Y5MWNlZTEtMTZhMDdmMWYtYTRkZjM2NA=='}]) at `github.com/ydb-platform/ydb-go-sdk/v3/internal/conn.(*grpcClientStream).RecvMsg(grpc_client_stream.go:178)` at `github.com/ydb-platform/ydb-go-sdk/v3/internal/query.nextPart(result.go:143)` at `github.com/ydb-platform/ydb-go-sdk/v3/internal/query.newResult(result.go:101)` at `github.com/ydb-platform/ydb-go-sdk/v3/internal/query.execute(execute_query.go:74)` at `github.com/ydb-platform/ydb-go-sdk/v3/internal/query.(*Session).Execute(session.go:316)` at `github.com/ydb-platform/ydb-go-sdk/v3/internal/query.do.func1(client.go:122)` at `github.com/ydb-platform/ydb-go-sdk/v3/internal/pool.(*Pool).try(pool.go:465)` at `github.com/ydb-platform/ydb-go-sdk/v3/internal/pool.(*Pool).With.func2(pool.go:493)` at `github.com/ydb-platform/ydb-go-sdk/v3/retry.Retry.func1(retry.go:264)` at `github.com/ydb-platform/ydb-go-sdk/v3/retry.opWithRecover(retry.go:411)` at `github.com/ydb-platform/ydb-go-sdk/v3/retry.RetryWithResult(retry.go:356)` at `github.com/ydb-platform/ydb-go-sdk/v3/retry.Retry(retry.go:270)` at `github.com/ydb-platform/ydb-go-sdk/v3/internal/pool.(*Pool).With(pool.go:505)` at `github.com/ydb-platform/ydb-go-sdk/v3/internal/query.do(client.go:130)`
get entry error: non-retryable error occurred on attempt No.1 (idempotent=true): connError{node_id:50001,address:'database-sample-1.database-sample-interconnect.default.svc.cluster.local:2135'}: operation/TIMEOUT (code = 400090, address = database-sample-1.database-sample-interconnect.default.svc.cluster.local:2135, issues = [{'Query did not complete within specified timeout, session id ydb://session/3?node_id=50003&id=MmEzOTA5MjQtNTRhNzRjYjAtZTYzMzQ0OTItOGFlZTU4MzE='}]) at `github.com/ydb-platform/ydb-go-sdk/v3/internal/conn.(*grpcClientStream).RecvMsg(grpc_client_stream.go:178)` at `github.com/ydb-platform/ydb-go-sdk/v3/internal/query.nextPart(result.go:143)` at `github.com/ydb-platform/ydb-go-sdk/v3/internal/query.newResult(result.go:101)` at `github.com/ydb-platform/ydb-go-sdk/v3/internal/query.execute(execute_query.go:74)` at `github.com/ydb-platform/ydb-go-sdk/v3/internal/query.(*Session).Execute(session.go:316)` at `github.com/ydb-platform/ydb-go-sdk/v3/internal/query.do.func1(client.go:122)` at `github.com/ydb-platform/ydb-go-sdk/v3/internal/pool.(*Pool).try(pool.go:465)` at `github.com/ydb-platform/ydb-go-sdk/v3/internal/pool.(*Pool).With.func2(pool.go:493)` at `github.com/ydb-platform/ydb-go-sdk/v3/retry.Retry.func1(retry.go:264)` at `github.com/ydb-platform/ydb-go-sdk/v3/retry.opWithRecover(retry.go:411)` at `github.com/ydb-platform/ydb-go-sdk/v3/retry.RetryWithResult(retry.go:356)` at `github.com/ydb-platform/ydb-go-sdk/v3/retry.Retry(retry.go:270)` at `github.com/ydb-platform/ydb-go-sdk/v3/internal/pool.(*Pool).With(pool.go:505)` at `github.com/ydb-platform/ydb-go-sdk/v3/internal/query.do(client.go:130)`
get entry error: non-retryable error occurred on attempt No.1 (idempotent=true): connError{node_id:50003,address:'database-sample-3.database-sample-interconnect.default.svc.cluster.local:2135'}: operation/TIMEOUT (code = 400090, address = database-sample-3.database-sample-interconnect.default.svc.cluster.local:2135, issues = [{'Query did not complete within specified timeout, session id ydb://session/3?node_id=50001&id=NzllNGI5ZC00MDdkMTJmLTdjNmViOGM4LTQ4Y2YxYjE1'}]) at `github.com/ydb-platform/ydb-go-sdk/v3/internal/conn.(*grpcClientStream).RecvMsg(grpc_client_stream.go:178)` at `github.com/ydb-platform/ydb-go-sdk/v3/internal/query.nextPart(result.go:143)` at `github.com/ydb-platform/ydb-go-sdk/v3/internal/query.newResult(result.go:101)` at `github.com/ydb-platform/ydb-go-sdk/v3/internal/query.execute(execute_query.go:74)` at `github.com/ydb-platform/ydb-go-sdk/v3/internal/query.(*Session).Execute(session.go:316)` at `github.com/ydb-platform/ydb-go-sdk/v3/internal/query.do.func1(client.go:122)` at `github.com/ydb-platform/ydb-go-sdk/v3/internal/pool.(*Pool).try(pool.go:465)` at `github.com/ydb-platform/ydb-go-sdk/v3/internal/pool.(*Pool).With.func2(pool.go:493)` at `github.com/ydb-platform/ydb-go-sdk/v3/retry.Retry.func1(retry.go:264)` at `github.com/ydb-platform/ydb-go-sdk/v3/retry.opWithRecover(retry.go:411)` at `github.com/ydb-platform/ydb-go-sdk/v3/retry.RetryWithResult(retry.go:356)` at `github.com/ydb-platform/ydb-go-sdk/v3/retry.Retry(retry.go:270)` at `github.com/ydb-platform/ydb-go-sdk/v3/internal/pool.(*Pool).With(pool.go:505)` at `github.com/ydb-platform/ydb-go-sdk/v3/internal/query.do(client.go:130)`
get entry error: non-retryable error occurred on attempt No.2 (idempotent=true): connError{node_id:50002,address:'database-sample-5.database-sample-interconnect.default.svc.cluster.local:2135'}: operation/TIMEOUT (code = 400090, address = database-sample-5.database-sample-interconnect.default.svc.cluster.local:2135, issues = [{'Query did not complete within specified timeout, session id ydb://session/3?node_id=50002&id=OTYwY2U1NC1kZmQ5NTkyNi0zOTVhMWM1MC1hZDY0NzMxZA=='}]) at `github.com/ydb-platform/ydb-go-sdk/v3/internal/conn.(*grpcClientStream).RecvMsg(grpc_client_stream.go:178)` at `github.com/ydb-platform/ydb-go-sdk/v3/internal/query.nextPart(result.go:143)` at `github.com/ydb-platform/ydb-go-sdk/v3/internal/query.newResult(result.go:101)` at `github.com/ydb-platform/ydb-go-sdk/v3/internal/query.execute(execute_query.go:74)` at `github.com/ydb-platform/ydb-go-sdk/v3/internal/query.(*Session).Execute(session.go:316)` at `github.com/ydb-platform/ydb-go-sdk/v3/internal/query.do.func1(client.go:122)` at `github.com/ydb-platform/ydb-go-sdk/v3/internal/pool.(*Pool).try(pool.go:465)` at `github.com/ydb-platform/ydb-go-sdk/v3/internal/pool.(*Pool).With.func2(pool.go:493)` at `github.com/ydb-platform/ydb-go-sdk/v3/retry.Retry.func1(retry.go:264)` at `github.com/ydb-platform/ydb-go-sdk/v3/retry.opWithRecover(retry.go:411)` at `github.com/ydb-platform/ydb-go-sdk/v3/retry.RetryWithResult(retry.go:356)` at `github.com/ydb-platform/ydb-go-sdk/v3/retry.Retry(retry.go:270)` at `github.com/ydb-platform/ydb-go-sdk/v3/internal/pool.(*Pool).With(pool.go:505)` at `github.com/ydb-platform/ydb-go-sdk/v3/internal/query.do(client.go:130)`
get entry error: non-retryable error occurred on attempt No.1 (idempotent=true): connError{node_id:50003,address:'database-sample-3.database-sample-interconnect.default.svc.cluster.local:2135'}: operation/TIMEOUT (code = 400090, address = database-sample-3.database-sample-interconnect.default.svc.cluster.local:2135, issues = [{'Query did not complete within specified timeout, session id ydb://session/3?node_id=50003&id=YTc2Zjg2MWUtNGY0NDhhNzEtZGZmYjNjYmYtNjJhZDBjODI='}]) at `github.com/ydb-platform/ydb-go-sdk/v3/internal/conn.(*grpcClientStream).RecvMsg(grpc_client_stream.go:178)` at `github.com/ydb-platform/ydb-go-sdk/v3/internal/query.nextPart(result.go:143)` at `github.com/ydb-platform/ydb-go-sdk/v3/internal/query.newResult(result.go:101)` at `github.com/ydb-platform/ydb-go-sdk/v3/internal/query.execute(execute_query.go:74)` at `github.com/ydb-platform/ydb-go-sdk/v3/internal/query.(*Session).Execute(session.go:316)` at `github.com/ydb-platform/ydb-go-sdk/v3/internal/query.do.func1(client.go:122)` at `github.com/ydb-platform/ydb-go-sdk/v3/internal/pool.(*Pool).try(pool.go:465)` at `github.com/ydb-platform/ydb-go-sdk/v3/internal/pool.(*Pool).With.func2(pool.go:493)` at `github.com/ydb-platform/ydb-go-sdk/v3/retry.Retry.func1(retry.go:264)` at `github.com/ydb-platform/ydb-go-sdk/v3/retry.opWithRecover(retry.go:411)` at `github.com/ydb-platform/ydb-go-sdk/v3/retry.RetryWithResult(retry.go:356)` at `github.com/ydb-platform/ydb-go-sdk/v3/retry.Retry(retry.go:270)` at `github.com/ydb-platform/ydb-go-sdk/v3/internal/pool.(*Pool).With(pool.go:505)` at `github.com/ydb-platform/ydb-go-sdk/v3/internal/query.do(client.go:130)`
get entry error: non-retryable error occurred on attempt No.1 (idempotent=true): connError{node_id:50004,address:'database-sample-2.database-sample-interconnect.default.svc.cluster.local:2135'}: operation/TIMEOUT (code = 400090, address = database-sample-2.database-sample-interconnect.default.svc.cluster.local:2135, issues = [{'Query did not complete within specified timeout, session id ydb://session/3?node_id=50004&id=ZTJlYjNkMTMtYjcyNGFjMTUtZWZhZDc0N2UtYzcwY2RhNGQ='}]) at `github.com/ydb-platform/ydb-go-sdk/v3/internal/conn.(*grpcClientStream).RecvMsg(grpc_client_stream.go:178)` at `github.com/ydb-platform/ydb-go-sdk/v3/internal/query.nextPart(result.go:143)` at `github.com/ydb-platform/ydb-go-sdk/v3/internal/query.newResult(result.go:101)` at `github.com/ydb-platform/ydb-go-sdk/v3/internal/query.execute(execute_query.go:74)` at `github.com/ydb-platform/ydb-go-sdk/v3/internal/query.(*Session).Execute(session.go:316)` at `github.com/ydb-platform/ydb-go-sdk/v3/internal/query.do.func1(client.go:122)` at `github.com/ydb-platform/ydb-go-sdk/v3/internal/pool.(*Pool).try(pool.go:465)` at `github.com/ydb-platform/ydb-go-sdk/v3/internal/pool.(*Pool).With.func2(pool.go:493)` at `github.com/ydb-platform/ydb-go-sdk/v3/retry.Retry.func1(retry.go:264)` at `github.com/ydb-platform/ydb-go-sdk/v3/retry.opWithRecover(retry.go:411)` at `github.com/ydb-platform/ydb-go-sdk/v3/retry.RetryWithResult(retry.go:356)` at `github.com/ydb-platform/ydb-go-sdk/v3/retry.Retry(retry.go:270)` at `github.com/ydb-platform/ydb-go-sdk/v3/internal/pool.(*Pool).With(pool.go:505)` at `github.com/ydb-platform/ydb-go-sdk/v3/internal/query.do(client.go:130)`
get entry error: non-retryable error occurred on attempt No.1 (idempotent=true): connError{node_id:50001,address:'database-sample-1.database-sample-interconnect.default.svc.cluster.local:2135'}: operation/TIMEOUT (code = 400090, address = database-sample-1.database-sample-interconnect.default.svc.cluster.local:2135, issues = [{'Query did not complete within specified timeout, session id ydb://session/3?node_id=50001&id=NzI0MzhlOGUtM2I4ZTRmYzAtODQ2YzgxZGYtNjA4NTljYmE='}]) at `github.com/ydb-platform/ydb-go-sdk/v3/internal/conn.(*grpcClientStream).RecvMsg(grpc_client_stream.go:178)` at `github.com/ydb-platform/ydb-go-sdk/v3/internal/query.nextPart(result.go:143)` at `github.com/ydb-platform/ydb-go-sdk/v3/internal/query.newResult(result.go:101)` at `github.com/ydb-platform/ydb-go-sdk/v3/internal/query.execute(execute_query.go:74)` at `github.com/ydb-platform/ydb-go-sdk/v3/internal/query.(*Session).Execute(session.go:316)` at `github.com/ydb-platform/ydb-go-sdk/v3/internal/query.do.func1(client.go:122)` at `github.com/ydb-platform/ydb-go-sdk/v3/internal/pool.(*Pool).try(pool.go:465)` at `github.com/ydb-platform/ydb-go-sdk/v3/internal/pool.(*Pool).With.func2(pool.go:493)` at `github.com/ydb-platform/ydb-go-sdk/v3/retry.Retry.func1(retry.go:264)` at `github.com/ydb-platform/ydb-go-sdk/v3/retry.opWithRecover(retry.go:411)` at `github.com/ydb-platform/ydb-go-sdk/v3/retry.RetryWithResult(retry.go:356)` at `github.com/ydb-platform/ydb-go-sdk/v3/retry.Retry(retry.go:270)` at `github.com/ydb-platform/ydb-go-sdk/v3/internal/pool.(*Pool).With(pool.go:505)` at `github.com/ydb-platform/ydb-go-sdk/v3/internal/query.do(client.go:130)`
get entry error: non-retryable error occurred on attempt No.1 (idempotent=true): connError{node_id:50001,address:'database-sample-1.database-sample-interconnect.default.svc.cluster.local:2135'}: operation/TIMEOUT (code = 400090, address = database-sample-1.database-sample-interconnect.default.svc.cluster.local:2135, issues = [{'Query did not complete within specified timeout, session id ydb://session/3?node_id=50001&id=YTU5OWJmNDMtM2JlNWJjZTMtODNjYjM3MzEtNzU1ZDJhYmU='}]) at `github.com/ydb-platform/ydb-go-sdk/v3/internal/conn.(*grpcClientStream).RecvMsg(grpc_client_stream.go:178)` at `github.com/ydb-platform/ydb-go-sdk/v3/internal/query.nextPart(result.go:143)` at `github.com/ydb-platform/ydb-go-sdk/v3/internal/query.newResult(result.go:101)` at `github.com/ydb-platform/ydb-go-sdk/v3/internal/query.execute(execute_query.go:74)` at `github.com/ydb-platform/ydb-go-sdk/v3/internal/query.(*Session).Execute(session.go:316)` at `github.com/ydb-platform/ydb-go-sdk/v3/internal/query.do.func1(client.go:122)` at `github.com/ydb-platform/ydb-go-sdk/v3/internal/pool.(*Pool).try(pool.go:465)` at `github.com/ydb-platform/ydb-go-sdk/v3/internal/pool.(*Pool).With.func2(pool.go:493)` at `github.com/ydb-platform/ydb-go-sdk/v3/retry.Retry.func1(retry.go:264)` at `github.com/ydb-platform/ydb-go-sdk/v3/retry.opWithRecover(retry.go:411)` at `github.com/ydb-platform/ydb-go-sdk/v3/retry.RetryWithResult(retry.go:356)` at `github.com/ydb-platform/ydb-go-sdk/v3/retry.Retry(retry.go:270)` at `github.com/ydb-platform/ydb-go-sdk/v3/internal/pool.(*Pool).With(pool.go:505)` at `github.com/ydb-platform/ydb-go-sdk/v3/internal/query.do(client.go:130)`
workers close ok
program finished

Overall

Seems like test freezes during closing after it sends every read/write. It freezes in conn.(*grpcClientStream).RecvMsg.

I also suspect that freezes are not caused by pool directly but it could be side effect of the fact that connections are long lived in case of pool.

I also noticed that increased amount of reads results in more errors (in logs) than writes.

There is no errors (or freezes) on client-side if load in small.

In all cases on database's side there's flood of errors like this:

2024-06-24T09:55:40.590560Z :RPC_REQUEST WARN: Client lost
2024-06-24T09:55:40.613093Z :KQP_SESSION ERROR: SessionId: ydb://session/3?node_id=50000&id=Yjk3ODI0NDItOTdjMTQ5NDktNzc1MzFmYTEtYjViMTJhMmM=, ActorId: [50000:7384006305595171918:5537], ActorState: ReadyState, Internal error, message: TKqpSessionActor in state ReadyState received unexpected event NActors::IEventHandle(0x108c0001) sender: [50005:8320808721877066593:7169396]
2024-06-24T09:55:40.613395Z :KQP_PROXY NOTICE: Session not found: ydb://session/3?node_id=50000&id=Yjk3ODI0NDItOTdjMTQ5NDktNzc1MzFmYTEtYjViMTJhMmM=

and some time after the test there're many of following errors:

2024-06-23T19:27:56.284095Z :KQP_PROXY ERROR: Unknown sender for proxy response, requestId: 187771
2024-06-23T19:27:57.313717Z :KQP_PROXY ERROR: Unknown sender for proxy response, requestId: 187891
2024-06-23T19:27:58.399784Z :KQP_PROXY ERROR: Unknown sender for proxy response, requestId: 188008
2024-06-23T19:27:58.559155Z :KQP_PROXY ERROR: Unknown sender for proxy response, requestId: 188030

Which makes me wonder if test client is doing everything ok.

Open questions (for me at least):

1. Is there some blocking points in internal/conn that we're not canceling with context at the moment?

2. Should we add circuit breaking for s.stream.RecvMsg(m) in func (s *grpcClientStream) RecvMsg?

3. Does YDB errors indicate that slo test is doing something wrong?