[UNHEALTHY] supabase/postgres-meta:v0.88.9

[Shabinder]

Bug report

• I confirm this is a bug with Supabase, not with my own application.
• I confirm I have searched the Docs, GitHub Discussions, and Discord.

Describe the bug

I am seeing frequent downtimes, and upon checking I find postgres-meta to be unhealthy. with following trace:

#  node[1]: uint64_t node::Histogram::RecordDelta() at ../src/histogram-inl.h:87
  #  Assertion failed: (time) >= (prev_)
----- Native stack trace -----
 1: 0xcc7f77 node::Assert(node::AssertionInfo const&) [node]
 2: 0xd41c02  [node]
 3: 0x18c23f1  [node]
 4: 0x18c61d4 uv_run [node]
 5: 0xbd3be6 node::SpinEventLoopInternal(node::Environment*) [node]
 6: 0xd17a71 node::NodeMainInstance::Run() [node]
 7: 0xc7eb0f node::Start(int, char**) [node]
 8: 0x751a53d5624a  [/lib/x86_64-linux-gnu/libc.so.6]
 9: 0x751a53d56305 __libc_start_main [/lib/x86_64-linux-gnu/libc.so.6]
10: 0xbd181e _start [node]
FATAL ERROR: v8::HandleScope::CreateHandle() Cannot create a handle without a HandleScope
----- Native stack trace -----
 1: 0xb8cdc8 node::OnFatalError(char const*, char const*) [node]
 2: 0xf03d16 v8::Utils::ReportApiFailure(char const*, char const*) [node]
 3: 0x10a9f92 v8::internal::HandleScope::Extend(v8::internal::Isolate*) [node]
 4: 0xf055f8 v8::EscapableHandleScope::EscapableHandleScope(v8::Isolate*) [node]
 5: 0xcc7ec6 node::GetCurrentStackTrace(v8::Isolate*, int) [node]
 6: 0xc1973b node::DumpJavaScriptBacktrace(_IO_FILE*) [node]
 7: 0xcc7f83 node::Assert(node::AssertionInfo const&) [node]
 8: 0xd41c02  [node]
 9: 0x18c23f1  [node]
10: 0x18c61d4 uv_run [node]
11: 0xbd3be6 node::SpinEventLoopInternal(node::Environment*) [node]
12: 0xd17a71 node::NodeMainInstance::Run() [node]
13: 0xc7eb0f node::Start(int, char**) [node]
14: 0x751a53d5624a  [/lib/x86_64-linux-gnu/libc.so.6]
15: 0x751a53d56305 __libc_start_main [/lib/x86_64-linux-gnu/libc.so.6]
16: 0xbd181e _start [node]

System information

• OS: Ubuntu 24.04.2 LTS x86_64
• Kernel: 6.8.0-58-generic
• Shell: bash 5.2.21
• CPU: AMD EPYC 9354P (8) @ 3.249GHz
• Memory: 6349MiB / 32094MiB

by mistake had opened this in main repo, reference link: supabase/supabase#35451

[avallete]

Looking at the error message, it looks like an issue with the time of the container vs time on the machine or the container time itself drifting away for some reason.

I cannot reproduce but a few things come to mind:

1. Try to sync the timezone between host/container with something like:

    volumes:
      - /etc/localtime:/etc/localtime:ro
      - /etc/timezone:/etc/timezone:ro

2. Maybe try to check the time on the pg_meta container when the error occurs (something like docker exec <pg_metacontainer_name> date)