📦 Project Complexity Management - Simplification and Organization

[ootakazuhiko]

📦 Project Complexity Management - Simplification and Organization

Priority: MEDIUM - Maintainability and Developer Experience

Overview

The ae-framework has grown into a comprehensive system with significant complexity that impacts maintainability and onboarding experience. Strategic simplification is needed to improve developer productivity.

Complexity Indicators

1. Package.json Overload

• Current State: 230+ npm scripts across multiple categories
• Categories:
  • Core build/test: ~30 scripts
  • Quality gates: ~40 scripts
  • Agent/MCP servers: ~25 scripts
  • Benchmarking/performance: ~20 scripts
  • Formal verification: ~15 scripts
  • CI/automation: ~35 scripts
  • Specialized tools: ~65 scripts

2. ESLint Configuration Complexity

• Broad Exclusions: 15+ directories excluded for "Verify Lite stability"
• Multiple Rule Sets: Base, type-checked, escalated rules by directory
• Technical Debt: Temporary exclusions becoming permanent

3. Project Structure Scale

• Source Structure: 30+ top-level src directories
• Package Count: 3+ workspace packages
• Configuration Files: 15+ config files in root
• CI Workflows: 35 GitHub Actions workflows

Impact Analysis

Developer Onboarding

• Learning Curve: High cognitive load for new contributors
• Documentation Debt: Config scattered across multiple files
• Tool Conflicts: Multiple overlapping tools for similar purposes

Maintenance Overhead

• Dependency Management: Complex pnpm workspace coordination
• Version Alignment: Multiple TypeScript configs with different targets
• Build Complexity: Multi-stage builds with interdependencies

CI/CD Pipeline

• Workflow Proliferation: 35 workflows with complex triggering logic
• Resource Usage: Heavy CI resource consumption
• Debug Difficulty: Complex failure diagnosis across workflows

Proposed Solutions

Phase 1: Script Consolidation (Weeks 1-2)

1. Package.json Reorganization

   {
     "scripts": {
       // Core (keep as-is)
       "build": "...",
       "test": "...",
       "dev": "...",
       
       // Consolidated categories
       "quality": "node scripts/quality/run-all.mjs",
       "formal": "node scripts/formal/run-suite.mjs", 
       "agents": "node scripts/agents/manage.mjs",
       "ci": "node scripts/ci/orchestrate.mjs"
     }
   }

2. Script Categorization

   • Create scripts/ directory structure by purpose
   • Implement unified CLI for script management
   • Add interactive script discovery: pnpm run help

Phase 2: Configuration Simplification (Weeks 2-4)

1. ESLint Cleanup Strategy

   // Gradual inclusion approach
   export default ts.config(
     {
       ignores: [
         // Temporary exclusions with removal timeline
         'src/ui/**', // TODO: Enable by Q2 2025
         'src/cli/**', // TODO: Enable by Q1 2025
       ]
     }
   );

2. Config File Consolidation

   • Move configs to configs/ directory
   • Create master config with includes
   • Standardize config naming patterns

Phase 3: Architecture Simplification (Month 2)

1. Workflow Consolidation

   # Master workflow with matrix strategy
   name: Unified CI
   strategy:
     matrix:
       suite: [fast, security, formal, ui, benchmarks]
       include:
         - suite: fast
           runs-on: ubuntu-latest
           timeout: 10
         - suite: security  
           runs-on: ubuntu-latest
           timeout: 30

2. Package Architecture Review

   • Evaluate package boundaries
   • Consider monorepo vs. workspace tradeoffs
   • Optimize dependency graphs

Phase 4: Developer Experience (Month 2-3)

1. Unified CLI Interface

   # Single entry point for all operations
   ae-framework --help
   ae-framework quality --run all
   ae-framework ci --local --fast
   ae-framework setup --interactive

2. Interactive Setup and Discovery

   • Guided project setup wizard
   • Interactive script discovery
   • Contextual help system

Implementation Roadmap

Week 1-2: Quick Wins

• Audit and categorize all package.json scripts
• Create scripts/ directory structure
• Implement script consolidation for top 5 categories
• Document script migration plan

Week 3-4: Configuration Cleanup

• Move configuration files to configs/
• Create ESLint inclusion roadmap with timelines
• Standardize config file naming and structure
• Update documentation references

Month 2: Workflow Optimization

• Analyze workflow overlap and redundancy
• Design unified CI workflow strategy
• Implement workflow consolidation (5-10 workflows target)
• Test and validate consolidated approach

Month 3: Developer Experience

• Create unified CLI interface
• Implement interactive setup wizard
• Add contextual help and discovery tools
• Update onboarding documentation

Success Metrics

Quantitative Goals

• Package.json scripts: 230+ → <100
• ESLint exclusions: 15+ directories → <5
• CI workflows: 35 → <15
• Onboarding time: Reduce by 50%

Qualitative Improvements

• Clear command discovery path
• Consistent configuration patterns
• Simplified mental model for contributors
• Reduced maintenance overhead

Risk Mitigation

1. Backward Compatibility

   • Maintain script aliases during transition
   • Gradual deprecation with clear timelines
   • Comprehensive testing of consolidated scripts

2. Feature Preservation

   • Ensure no functionality loss during consolidation
   • Maintain all quality gates and checks
   • Preserve specialized tool capabilities

3. Team Coordination

   • Communicate changes clearly to contributors
   • Provide migration guides for existing workflows
   • Maintain documentation during transition

Related Documentation

• Project Organization Guide
• Development Baseline
• Agent Working Rules