Skip to content

overdue: Sync.initSync misses retryable query #997

@oxzi

Description

@oxzi

Describe the bug

After updating and restarting almost every package, a race between the relational database and Icinga DB resulted in a failing query which is currently not being retried.

FATAL   icingadb        read tcp [::1]:6464->[::1]:5432: read: connection reset by peer
can't perform "SELECT id FROM service_state WHERE is_overdue='y'"
github.com/icinga/icinga-go-library/database.CantPerformQuery
        github.com/icinga/icinga-go-library@v0.7.2/database/utils.go:19
github.com/icinga/icingadb/pkg/icingadb/overdue.Sync.initSync
        github.com/icinga/icingadb/pkg/icingadb/overdue/sync.go:89
github.com/icinga/icingadb/pkg/icingadb/overdue.Sync.Sync.func2
        github.com/icinga/icingadb/pkg/icingadb/overdue/sync.go:53
golang.org/x/sync/errgroup.(*Group).add.func1
        golang.org/x/sync@v0.15.0/errgroup/errgroup.go:128
runtime.goexit
        runtime/asm_amd64.s:1699
can't sync overdue indicators
github.com/icinga/icingadb/pkg/icingadb/overdue.Sync.Sync
        github.com/icinga/icingadb/pkg/icingadb/overdue/sync.go:57
main.run.func6.3
        github.com/icinga/icingadb/cmd/icingadb/main.go:215
golang.org/x/sync/errgroup.(*Group).add.func1
        golang.org/x/sync@v0.15.0/errgroup/errgroup.go:128
runtime.goexit
        runtime/asm_amd64.s:1699

var rows []v1.IdMeta
query := fmt.Sprintf("SELECT id FROM %s_state WHERE is_overdue='y'", objectType)
if err := s.db.SelectContext(ctx, &rows, query); err != nil {
return database.CantPerformQuery(err, query)
}

Expected behavior

Same retry logic as everywhere else.

Your Environment

  • Icinga DB version: 1.4.0

Additional context

In general, we should scan the code for SQL queries not wrapped in our retry logic. I would guess there are some more which can be wrapped safely.

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't workingcrash

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions