Skip to content

feat: add experimental monitor mode to BasicCrawler#3553

Open
hiepau1231 wants to merge 9 commits intoapify:masterfrom
hiepau1231:feature/monitor-mode
Open

feat: add experimental monitor mode to BasicCrawler#3553
hiepau1231 wants to merge 9 commits intoapify:masterfrom
hiepau1231:feature/monitor-mode

Conversation

@hiepau1231
Copy link
Copy Markdown

Fixes #2680

Continues the work from #2692, addressing all reviewer feedback and adding missing tests.

What changed

packages/core/src/crawlers/monitor.ts

  • Guard ETA against going negative when finished > total (approximate counts from RequestManagerTandem) — Math.max(0, remaining)
  • Show CPU: N/A on Windows where os.loadavg() always returns [0,0,0]
  • Call .unref() on the interval so the monitor cannot prevent process exit

packages/basic-crawler/src/internals/basic-crawler.ts

  • Add monitorOptions?: MonitorOptions to BasicCrawlerOptions so callers can configure intervalSecs and other options without monkey-patching
  • Pass this.monitorOptions through to the Monitor constructor
  • Skip the statusMessageLoggingInterval periodic logger when monitor: true is active to avoid plain log lines being interleaved with ANSI cursor-control sequences

test/core/crawlers/monitor.test.ts

  • 4 new unit tests: negative-ETA guard, Windows CPU N/A label, intervalSecs config, improved cursor-overwrite assertion (15 total, all passing)

All 65 existing basic_crawler.test.ts tests continue to pass.

Usage

const crawler = new BasicCrawler({
  monitor: true,
  monitorOptions: { intervalSecs: 2 },
  async requestHandler({ request }) { /* ... */ },
});
await crawler.run(['https://example.com']);

hiepau1231 and others added 9 commits April 4, 2026 15:01
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
…olved)

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Adds a new Monitor class that renders a compact real-time status block
to process.stderr during a crawl. In TTY mode the block overwrites
itself in-place; in non-TTY mode (CI, pipes) it prints plain lines.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
12 tests covering: construction, start/stop lifecycle, buildLines()
output content, TTY vs non-TTY rendering, and stderr write behavior.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Adds a `monitor` boolean option to BasicCrawlerOptions. When true,
a Monitor instance is created at the start of `run()` and stopped in the
`finally` block, ensuring it always cleans up even if the crawl throws.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
3 integration tests verifying that:
- crawl completes normally with monitor: true
- crawl completes normally with monitor: false (default)
- errors still propagate to failedRequestHandler when monitor is active

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
… CPU, unref timer, 4 new tests

- Add `monitorOptions?: MonitorOptions` to `BasicCrawlerOptions` and pass
  it through to the `Monitor` constructor so callers can configure
  `intervalSecs` without monkey-patching.
- Skip the periodic status-message logger when `monitor: true` is set to
  prevent ANSI cursor-control sequences from being interleaved with plain
  log lines.
- Guard ETA against negative values when `finished > total` (approximate
  counts from `RequestManagerTandem`): clamp remaining to ≥ 0.
- Show `CPU: N/A` on Windows where `os.loadavg()` always returns zeros.
- Call `.unref()` on the interval so the monitor cannot prevent process exit.
- Remove internal planning docs (spec/plan) from the repository.
- Add 4 tests: negative-ETA guard, Windows CPU label, intervalSecs config,
  and improve cursor-overwrite assertion.

Closes apify#2680

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Monitor mode

3 participants