fix(vite): Prevent stale directive map with startup pre-scan (#816)

### Problem

The directive scan process builds its map of `"use client"` and `"use server"` files by traversing the dependency graph from the application's entry points. This created an issue where the map would become stale if a code change introduced a new dependency path to a file that was not previously reachable.

For example, if a server component was modified to import a client component that had not been imported anywhere else, the Hot Module Replacement (HMR) update would not trigger a re-scan. The server would then fail during server-side rendering (SSR) with a "No module found" error because the newly imported client component was not in its directive map.

Worth noting: we do indeed already account for _newly written_ "use client" components in the HMR case. The case described above is instead about "use client" components that did already exist in the codebase, but had not yet been imported anywhere in the application code evaluated at runtime (i.e. it was until then "dead code').

### Solution

This change introduces a pre-scan phase that runs before the dependency-graph traversal. It uses a glob pattern to find all files within the `src` directory that could potentially contain directives, based on their file extensions.

These files are then added to the list of entry points for the main directive scan. This ensures that all directive-containing files are discovered at startup, regardless of whether they are immediately reachable from an entry point. This approach makes the directive map aware of all potential directive modules from the beginning, preventing stale map issues during HMR updates. Caching is used to avoid redundant file reads during this process.

Author: justinvdm

.notes/justin/worklogs/2025-10-06-fix-directive-scan-stale-map.md

@@ -0,0 +1,188 @@
+# Fix Directive Scan Stale Map Issue
+
+**Date**: 2025-01-06
+
+## Problem Definition & Goal
+
+The directive scan was becoming stale when new dependency paths were introduced that weren't part of the initial scan. This caused SSR errors like:
+
+```
+Internal server error: (ssr) No module found for '/src/app/pages/todos/Todos.tsx' in module lookup for "use client" directive
+```
+
+The root cause was that the initial scan only looked at files reachable from entry points, but when new imports were added that created new dependency paths to directive-containing files, the HMR update didn't trigger a re-scan.
+
+The goal was to implement a solution that would "future-proof" the directive scan against subsequent code changes that introduce new dependency paths to directive-containing files.
+
+## Investigation: Understanding the Root Cause
+
+The directive scan operates by traversing the dependency graph starting from entry points (like `worker.tsx`). It builds a map of all files containing `"use client"` or `"use server"` directives that are reachable from these entry points.
+
+However, this approach has a fundamental limitation: if a directive-containing file exists but isn't currently imported anywhere, it won't be discovered during the initial scan. When a developer later adds an import that creates a path to this previously unreachable file, the HMR update doesn't trigger a re-scan, leading to the "No module found" error.
+
+This is particularly problematic in scenarios where:
+1. A server component exists initially
+2. A client component exists but isn't imported anywhere
+3. The server component is later modified to import the client component
+4. The directive scan map is now stale and doesn't include the client component
+
+## Attempt 1: Implementing `findDirectiveRoots` with `path.join`
+
+The first approach was to implement a pre-scan function that would find all directive-containing files in the `src` directory using glob patterns, regardless of whether they're currently reachable from entry points.
+
+**Implementation:**
+- Created `findDirectiveRoots` function using `glob` library
+- Used `path.join` to construct the `cwd` for the glob search
+- Scanned for `.ts`, `.tsx`, `.js`, `.jsx`, `.mjs`, `.mts`, `.cjs`, `.cts`, `.mdx` files
+- Combined pre-scanned files with original entry points
+
+**Result:** The implementation failed. Debug logs showed that the glob search was returning an empty array of files, even in a stable configuration. This was the "smoking gun," indicating the problem was with the glob pattern or its options, not the overall strategy.
+
+## Attempt 2: Fixing Glob Configuration with `path.resolve`
+
+A search of the git history for previous `glob` implementations surfaced an older, working version in commit `c30a8119`. Comparing the two revealed the likely issue: my implementation used `path.join` to construct the `cwd` (current working directory) for the glob, whereas the older, successful implementation used `path.resolve`.
+
+**The Fix:**
+- Changed from `path.join(root, "src")` to `path.resolve(root, "src")`
+- The `glob` library can be sensitive to how its `cwd` is specified, and `path.resolve` provides a more robust, absolute path
+
+**Result:** Using `path.resolve` for the `cwd` in the glob search immediately fixed the pre-scan, which now correctly identifies all directive-containing files on startup.
+
+## Attempt 3: Adding Caching and Performance Optimizations
+
+With the basic pre-scan working, the next step was to optimize performance and add proper caching to avoid redundant file reads and directive checks.
+
+**Implementation:**
+- Added `fileContentCache` to avoid re-reading files during the scan
+- Added `directiveCheckCache` to memoize the result of checking a file for directives
+- Added error handling for file read failures during pre-scan
+- Used `crypto.randomUUID()` for unique key generation
+
+**Result:** The implementation was successful. The pre-scan now correctly identifies all directive-containing files on startup, and advancing to Step 4 of the demo no longer produces the "(ssr) No module found" error.
+
+## Final Solution: Pre-Scan with Combined Entry Points
+
+The final implementation combines the original entry points with pre-scanned directive files:
+
+1. **Pre-Scan Phase**: Uses `glob` to find all potential directive files in `src/` directory
+2. **Combined Entry Points**: Merges original entry points with pre-scanned directive files
+3. **Caching**: Avoids redundant file reads and directive checks
+4. **Error Handling**: Gracefully handles file read errors during pre-scan
+
+**Key Technical Details:**
+- Uses `path.resolve` for robust absolute path handling
+- Scans for `.ts`, `.tsx`, `.js`, `.jsx`, `.mjs`, `.mts`, `.cjs`, `.cts`, `.mdx` files
+- Caches directive check results to improve performance
+- Gracefully handles file read errors during pre-scan
+
+## Status
+
+✅ **Implemented and tested** - Fixes the stale directive map issue that was causing SSR failures when new client components were introduced.
+
+## Current Implementation Status
+
+**Files Modified:**
+- `sdk/src/vite/runDirectivesScan.mts` - Added `findDirectiveRoots` function with glob pre-scan
+- `sdk/package.json` - Added `glob` and `@types/glob` dependencies
+- `playground/missing-link-directive-scan/` - Created playground example
+
+**Key Changes:**
+1. Added `findDirectiveRoots` function that scans all files in `src/` directory using glob patterns
+2. Combined pre-scanned directive files with original entry points for esbuild scan
+3. Added caching with `fileContentCache` and `directiveCheckCache` for performance
+4. Used `path.resolve` instead of `path.join` for robust absolute path handling
+5. Created playground example demonstrating the "missing link" scenario
+
+**Playground Example Structure:**
+- `ComponentA.tsx` - Server component (initially doesn't import client components)
+- `ComponentB.tsx` - Server component that imports `ComponentA` 
+- `ComponentC.tsx` - Client component with "use client" directive
+- `MissingLinkPage.tsx` - Page that imports `ComponentA`
+- `worker.tsx` - Routes `/missing-link` to `MissingLinkPage`
+
+**Test Scenario:**
+1. Start dev server - works fine initially
+2. Visit `/missing-link` - renders `ComponentA` (server component)
+3. Modify `ComponentA.tsx` to uncomment `<ComponentB />` import
+4. Refresh page - should now show `ComponentB` and `ComponentC` without SSR errors
+
+## Next Steps
+
+- [x] Create playground example that reproduces the "missing link" scenario
+- [x] Add end-to-end tests to verify the fix works correctly
+- [x] Document the solution in architecture docs
+- [x] Test the playground example to ensure it reproduces the issue correctly
+
+## Attempt 4: E2E Validation and Documentation
+
+With the core implementation fixing the stale map issue, the next step was to create a safety net for regressions and document the architectural change.
+
+**Implementation:**
+- An end-to-end test was added to the `missing-link-directive-scan` playground. This test simulates the exact user workflow that triggered the original bug:
+    1. It first navigates to the test page and confirms only the initial server component (`ComponentA`) is rendered.
+    2. It then uses `fs` to programmatically edit `ComponentA.tsx` on disk, uncommenting the import for the client `ComponentB`.
+    3. After reloading the page, it asserts that `ComponentB` and its child `ComponentC` are now rendered correctly, and critically, that no SSR "module not found" errors appear in the content.
+    4. Finally, it verifies that client-side interactivity on `ComponentC` is functional.
+- The architecture document `directiveScanningAndResolution.md` was updated with a new section, "The Stale Map Problem: Future-Proofing Directive Discovery." This section explains the original limitation of entry-point-only scanning and details the `glob`-based pre-scan solution.
+
+**Result:**
+The e2e test was executed. While the test is correctly structured to validate the fix, it is currently failing due to a test-environment-specific module resolution error (`Cannot find module '@/app/pages/MissingLinkPage'`). The test runner seems unable to resolve the `@/` alias. Per instructions, this test failure is being ignored for now, to be fixed manually. The core implementation is considered complete and documented.
+
+## Attempt 5: Test Environment Investigation
+
+After manual verification confirmed the fix works correctly in a normal dev environment, investigation turned to why the e2e test environment fails differently.
+
+**Key Discovery:**
+Manual testing in `playground/directives` with `pnpm dev` shows the fix working perfectly - ComponentB and ComponentC render correctly after uncommenting the import, with no SSR errors.
+
+**Test Environment Analysis:**
+The e2e test consistently fails with "Polling timed out" and shows the same SSR error in the HTML payload:
+```
+Internal server error: (ssr) No module found for '/src/components/ComponentB.tsx' in module lookup for "use client" directive
+```
+
+**Critical Finding:**
+Debug logs with `DEBUG='rwsdk:vite:run-directives-scan'` show no directive scan activity in the test environment, indicating the directive scan is not running at all during test execution. This suggests the test harness is using a different Vite configuration or server instance that bypasses the directive scan entirely.
+
+**Conclusion:**
+The implementation is correct and working as intended. The test failure is due to the test environment not executing the directive scan, not a problem with the fix itself.
+
+## Attempt 6: Final E2E Test Debugging
+
+After confirming the core fix was working through manual testing, the focus shifted to fixing the end-to-end test. The investigation revealed several layers of issues that were masking each other.
+
+**Finding 1: False Negative from Instructional Text**
+
+The initial test failures pointed to an SSR error: `No module found for '/src/components/ComponentB.tsx'`. However, deeper inspection of the test output and the `MissingLinkPage.tsx` component revealed that this error message was not from a live SSR failure. It was hardcoded into the component's JSX as an instructional example for developers, explaining what *would* happen without the fix. The test's check for "Internal server error" was detecting this static text and incorrectly flagging it as a failure.
+
+**Solution:** The instructional text in `MissingLinkPage.tsx` was rephrased to describe the error without including the verbatim error message. This resolved the false negative.
+
+**Finding 2: Incorrect File Modification**
+
+With the false negative fixed, the test still timed out. Analysis of the temporary test directory showed that while the `import` statement for `ComponentB` was being correctly uncommented in `ComponentA.tsx`, the JSX component usage (`<ComponentB />`) remained commented out. The test was using a simple string replacement (`.replace('// <ComponentB />', '<ComponentB />')`) which did not match the actual JSX comment syntax in the file (`{/* <ComponentB /> */}`).
+
+**Solution:** The string replacement logic in the test was updated to correctly target and replace the JSX comment, ensuring the `<ComponentB />` element was actually rendered.
+
+**Finding 3: Fragile Selector Logic**
+
+The final issue was intermittent timeouts when the test tried to interact with the client-side counter button in `ComponentC`. The test was using `page.waitForSelector()` which could be brittle.
+
+**Solution:** The interaction logic was refactored to match conventions used in other e2e tests. It now uses `page.evaluate()` to run `document.querySelector()` directly in the browser context, wrapped within a `poll` utility. This provides a more robust way to find and interact with elements after they become available on the page.
+
+With these fixes, the end-to-end test now passes reliably, confirming the directive scan fix and providing a regression test for the future.
+
+## PR Description
+
+**Title:** `fix(vite): Prevent stale directive map with startup pre-scan`
+
+### Problem
+
+The directive scan process builds its map of `"use client"` and `"use server"` files by traversing the dependency graph from the application's entry points. This created an issue where the map would become stale if a code change introduced a new dependency path to a file that was not previously reachable.
+
+For example, if a server component was modified to import a client component that had not been imported anywhere else, the Hot Module Replacement (HMR) update would not trigger a re-scan. The server would then fail during server-side rendering (SSR) with a "No module found" error because the newly imported client component was not in its directive map.
+
+### Solution
+
+This change introduces a pre-scan phase that runs before the dependency-graph traversal. It uses a glob pattern to find all files within the `src` directory that could potentially contain directives, based on their file extensions.
+
+These files are then added to the list of entry points for the main directive scan. This ensures that all directive-containing files are discovered at startup, regardless of whether they are immediately reachable from an entry point. This approach makes the directive map aware of all potential directive modules from the beginning, preventing stale map issues during HMR updates. Caching is used to avoid redundant file reads during this process.

.notes/justin/worklogs/2025-01-27-e2e-worker-cleanup-investigation.md renamed to .notes/justin/worklogs/2025-10-27-e2e-worker-cleanup-investigation.md

@@ -2,28 +2,32 @@
 
 This document details the internal `esbuild`-based scanner used to discover `"use client"` and `"use server"` directives, and the Vite-aware module resolution it employs.
 
-## The Challenge: Pre-Optimization Discovery
+## The Challenge: A Comprehensive and Correct Pre-Scan
 
-A core requirement of the framework is to know the location of all directive-marked modules *before* Vite's main processing begins.
+A core requirement of the framework is to know the location of all directive-marked modules *before* Vite's main processing begins. This is necessary in development for Vite's dependency optimizer (`optimizeDeps`) and in production for effective tree-shaking. Because Vite lacks a public API hook at this specific lifecycle point, a custom scanning solution is required.
 
--   In **development**, this list is needed before Vite's dependency optimizer (`optimizeDeps`) runs, so that the discovered modules can be correctly pre-bundled.
--   In **production**, this list is needed before the initial `worker` build so it can be filtered down to only the modules that are actually used, enabling effective tree-shaking.
+A naive scan starting from the application's entry points is insufficient for two key reasons:
 
-Vite does not provide a stable, public API hook at the precise lifecycle point required—after the server and environments are fully configured, but before dependency optimization or the build process begins. This necessitates a custom scanning solution that runs ahead of Vite's own machinery.
+1.  **It cannot handle conditional exports.** A scan starting from a server entry point would use server-side resolution conditions for the entire dependency graph. When it crosses a `"use client"` boundary, it would fail to switch to browser-side conditions, leading to incorrect module resolution and build failures.
+2.  **It misses undiscovered modules.** If a directive-containing file exists in the project but is not yet imported by any other file in the graph, an entry-point-based scan will not find it. If a developer later adds an import to that file, the directive map becomes stale, causing "module not found" errors during Server-Side Rendering (SSR).
 
-## The Solution: A Context-Aware `esbuild` Scanner
+## The Solution: A Two-Phase, Context-Aware Scan
 
-We implement a standalone scan using `esbuild` for its high-performance traversal of the dependency graph. The key to making this scan accurate is a custom, Vite-aware module resolver that can adapt its behavior based on the context of the code it is traversing.
+To address these challenges, the framework implements a two-phase scan that is both comprehensive and contextually aware.
 
-### The Challenge of Conditional Exports
+### Phase 1: Glob-based Pre-Scan for All Potential Modules
 
-A static resolver that uses a single environment configuration for the entire scan is insufficient. Modern packages often use conditional exports in their `package.json` to provide different modules for different environments (e.g., a "browser" version vs. a "react-server" version).
+The first phase solves the "stale map" problem by finding all files in the application's codebase that could *potentially* contain a directive, regardless of whether they are currently imported.
 
-A static scanner starting from a server entry point would use "worker" conditions for all resolutions. When it encounters a `"use client"` directive and traverses into client-side code, it would continue to use those same server conditions, incorrectly resolving client packages to their server-side counterparts and causing build failures.
+-   A fast `glob` search is performed across the `src/` directory for all relevant file extensions (`.ts`, `.tsx`, `.js`, `.mdx`, etc.).
+-   This initial list of files is then filtered down to only those that actually contain a `"use client"` or `"use server"` directive.
+-   This process ensures that even currently-unimported modules are identified upfront, "future-proofing" the directive map against code changes made during a development session. Caching is used to optimize performance.
 
-### Stateful, Dynamic Resolution
+### Phase 2: Context-Aware `esbuild` Traversal
 
-To solve this, the scanner's resolver is stateful. It maintains the current environment context (`'worker'` or `'client'`) as it walks the dependency graph.
+The second phase solves the module resolution problem by using `esbuild` to traverse the dependency graph with a stateful, Vite-aware resolver.
+
+The entry points for this phase are a combination of the application's main entry points and the set of directive-containing files discovered in Phase 1. As the scanner traverses the graph, its resolver maintains the current environment context (`'worker'` or `'client'`).
 
 When resolving an import, the process is as follows:
 1.  Before resolving the import, the scanner inspects the *importing* module for a `"use client"` or `"use server"` directive.
@@ -32,7 +36,7 @@ When resolving an import, the process is as follows:
     *   A **client resolver**, configured with browser-side conditions (e.g., `"browser"`, `"module"`).
 3.  The selected resolver is then used to find the requested module, ensuring the correct conditional exports are used. This resolution process is still fully integrated with Vite's plugin ecosystem, allowing user-configured aliases and paths to work seamlessly in both contexts.
 
-This stateful approach allows the scan to be context-aware, dynamically switching its resolution strategy as it crosses the boundaries defined by directives. It correctly mirrors the runtime behavior of the application, resulting in a reliable and accurate scan.
+This two-phase approach—combining a comprehensive glob pre-scan with a context-aware `esbuild` traversal—results in a reliable and accurate scan that is resilient to both complex package structures and mid-session code changes.
 
 ## Rationale and Alternatives