refactor: migrate the job gc to mananger gc server

[chlins]

This pull request introduces a new garbage collection (GC) framework for the Dragonfly Manager, replacing the previous implementation. It adds modular GC tasks for audits and jobs, refactors related code, and enhances maintainability. The most important changes are grouped into the following themes:

New Garbage Collection Framework:

• Introduced a new GC framework with a GC struct to manage tasks and their execution. Added support for registering and running multiple GC tasks (manager/gc/gc.go).
• Added NewAuditGCTask and NewJobGCTask to define GC tasks for audits and jobs, including their intervals, timeouts, and batch sizes (manager/gc/audit.go, manager/gc/job.go). [1] [2]

Task Recording and Results:

• Implemented a jobRecorder to record the results of GC tasks, including purged items and errors, with support for task initialization (manager/gc/recorder.go).

Integration with Manager:

• Integrated the new GC framework into the Server struct in manager/manager.go, replacing the old GC implementation. GC tasks are now started and stopped alongside the server (manager/manager.go). [1] [2] [3] [4]

Removal of Old GC Implementation:

• Removed the previous GC implementation from manager/job/gc.go and related references in manager/job/job.go. This eliminates the hardcoded GC logic and replaces it with the new modular approach. [1] [2] [3]

Refactoring Constants in Audit Service:

• Refactored constants in manager/service/audit.go for better readability and consistency, including AuditBufferSize, AuditInsertBatchSize, and AuditInsertInterval. Updated their usage throughout the file. [1] [2] [3] [4]

Description

Related Issue

#3811

Motivation and Context

Screenshots (if appropriate)

Types of changes

• Bug fix (non-breaking change which fixes an issue)
• New feature (non-breaking change which adds functionality)
• Breaking change (fix or feature that would cause existing functionality to change)
• Documentation Update (if none of the other choices apply)

Checklist

• My change requires a change to the documentation.
• I have updated the documentation accordingly.
• I have read the CONTRIBUTING document.
• I have added tests to cover my changes.

[codecov]

Codecov Report

Attention: Patch coverage is 0% with 95 lines in your changes missing coverage. Please review.

Project coverage is 34.34%. Comparing base (09d99d3) to head (e300dc3).
Report is 4 commits behind head on main.

Files with missing lines	Patch %	Lines
manager/gc/job.go	0.00%	44 Missing ⚠️
manager/gc/recorder.go	0.00%	31 Missing ⚠️
manager/manager.go	0.00%	12 Missing ⚠️
manager/service/audit.go	0.00%	8 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #3991      +/-   ##
==========================================
- Coverage   34.40%   34.34%   -0.06%     
==========================================
  Files         341      342       +1     
  Lines       40084    40153      +69     
==========================================
  Hits        13789    13789              
- Misses      25389    25456      +67     
- Partials      906      908       +2     

Flag	Coverage Δ
unittests	34.34% <0.00%> (-0.06%)	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

Files with missing lines	Coverage Δ
manager/job/job.go	18.36% <ø> (+2.29%)	⬆️
manager/service/audit.go	0.00% <0.00%> (ø)
manager/manager.go	0.00% <0.00%> (ø)
manager/gc/recorder.go	0.00% <0.00%> (ø)
manager/gc/job.go	0.00% <0.00%> (ø)

... and 6 files with indirect coverage changes

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[gaius-qi]

LGTM

[Liam-Zhao]

LGTM