Research: stdlib stub docstring enrichment from typeshed and CPython sources

[Copilot]

Research options to enrich MicroPython stdlib stubs with docstrings from CPython/typeshed without overwriting MicroPython-specific documentation.

Findings

BasedPyright/Pyright Integration

• Typeshed stubs bundled at node_modules/basedpyright/dist/typeshed/stdlib/
• High-quality type annotations, but intentionally exclude docstrings (maintenance policy)
• Extractable programmatically via npm package

CPython Docstring Extraction

• Runtime introspection via inspect module provides rich documentation
• Successfully tested on json and sys modules (41 functions extracted)

Existing Infrastructure

• merge_docstub.py codemod handles type-rich stub merging
• Test case typeshed_incomplete_pyi demonstrates CPython-like stub integration
• MicroPython-specific docstrings well-maintained in RST-generated stubs

Recommendation: Hybrid Approach

Extract type information from typeshed + docstrings from CPython runtime, merge with strict preservation:

# Before: MicroPython stub
def dumps(obj) -> str: ...

# After: enriched with types + docstrings
def dumps(obj: Any, separators: tuple[str, str] | None = ...) -> str:
    """
    Serialize ``obj`` to a JSON formatted ``str``.
    
    If ``separators`` is specified, it should be a tuple of 
    (item_separator, key_separator).
    
    Note: MicroPython has limited support for some JSON features.
    """
    ...

Critical Rules:

• NEVER overwrite MicroPython-specific docstrings
• ADD CPython docstrings only where none exist
• FLAG conflicts for manual review

Deliverables

Documentation (33KB)

• docs/README_stdlib_docstrings.md - Index
• docs/research_summary_stdlib_docstrings.md - Executive summary
• docs/research_stdlib_docstrings.md - Technical deep-dive (13KB)
• docs/implementation_guide_stdlib_docstrings.md - Implementation roadmap

Proof-of-Concept Scripts (tested ✅)

• /tmp/extract_typeshed_poc.py - Extracts from basedpyright npm package
• /tmp/extract_cpython_docstrings.py - Extracts CPython docstrings via runtime

Implementation Phases

1. PoC ✅ Complete - Research and validation
2. Automation - Enhance merge_docstub.py, add preservation logic (2-3 weeks)
3. Integration - CI/CD, optional feature flag (1-2 weeks)
4. Maintenance - Regular updates, conflict review

Decision Points

• Approve hybrid approach?
• Initial module scope? (recommend: json, os, sys, re)
• Optional or default feature?
• Authorize Phase 2 implementation?

Original prompt

---

This section details on the original issue you should resolve

<issue_title>Add more docstrings to stdlib stubs</issue_title>
<issue_description>reasearch to options to do this by 👍🏻

• checking : https://docs.basedpyright.com/dev/development/internals/
• or is this somethang that we can extract already as a .pyi from the node pagage ?
• then merging the relevant docstrings , but not overwriting the micropython specific ones
  </issue_description>

<agent_instructions>Only research and create a plan - do not start implementation yet</agent_instructions>

Comments on the Issue (you are @copilot in this section)

• Fixes Add more docstrings to stdlib stubs #698

---

✨ Let Copilot coding agent set things up for you — coding agent works faster and does higher quality work when set up for your repo.