Skip to content

Conversation

aylei
Copy link
Collaborator

@aylei aylei commented Aug 29, 2025

Measures:

  • DB histogram (ops, duration, percentiles) (not the SQL, just entire function)
  • Background request execution histogram
  • Event loop lag

test:

image

Draft dashboard json: SkyPilot API Server Edit-1756489275861.json

Next step:

  • measure the sync block in async function
  • measure the RSS and heap size of each process
  • Update default dashboard

Tested (run the relevant ones):

  • Code formatting: install pre-commit (auto-check on commit) or bash format.sh
  • Any manual or new tests for this PR (please specify below)
  • All smoke tests: /smoke-test (CI) or pytest tests/test_smoke.py (local)
  • Relevant individual tests: /smoke-test -k test_name (CI) or pytest tests/test_smoke.py::test_name (local)
  • Backward compatibility: /quicktest-core (CI) or pytest tests/smoke_tests/test_backward_compat.py (local)

aylei added 3 commits August 29, 2025 13:13
Signed-off-by: Aylei <rayingecho@gmail.com>
Signed-off-by: Aylei <rayingecho@gmail.com>
Signed-off-by: Aylei <rayingecho@gmail.com>
@aylei
Copy link
Collaborator Author

aylei commented Aug 29, 2025

/smoke-test --kubernetes

@aylei
Copy link
Collaborator Author

aylei commented Aug 29, 2025

/smoke-test -k test_long_setup_run_script--kubernetes

@aylei
Copy link
Collaborator Author

aylei commented Aug 29, 2025

/smoke-test -k test_long_setup_run_script --kubernetes

aylei added 4 commits August 30, 2025 00:33
Signed-off-by: Aylei <rayingecho@gmail.com>
Signed-off-by: Aylei <rayingecho@gmail.com>
Signed-off-by: Aylei <rayingecho@gmail.com>
Signed-off-by: Aylei <rayingecho@gmail.com>
@aylei aylei changed the title Support loop lag monitoring Add more server metrics for loop lag monitoring Aug 29, 2025
aylei added 2 commits August 30, 2025 01:12
Signed-off-by: Aylei <rayingecho@gmail.com>
Signed-off-by: Aylei <rayingecho@gmail.com>
Copy link
Collaborator

@SeungjinYang SeungjinYang left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

will approve once PR is out of draft

Signed-off-by: Aylei <rayingecho@gmail.com>
@aylei aylei marked this pull request as ready for review August 29, 2025 17:41
Signed-off-by: Aylei <rayingecho@gmail.com>
Copy link
Collaborator

@SeungjinYang SeungjinYang left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @aylei! LGTM

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants