Robust Execution and Adaptive Timeout: implement adaptive timeout and robust retry #725
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Adaptive Timeout
Modes: off / basic / auto; supports base + per-1k with EMA profiling; default slow-model multiplier = 10, persisted across retries; global cap protection.
Resilient Retry & Key Rotation
Unified error classification and mapping; exponential backoff with jitter; API key rotation on 429/503 triggers; masked key display in logs; full attempt history retained.
Streaming Robustness
Forced flush of trailing packets to ensure no tokens are lost; shard-based parsing compatible with reasoning tags and native reasoning channels.
Tool JSON Parsing
Strict JSON → DirtyJson fault tolerance → “final balanced object” recovery stage.
Safer Python Execution
AST.parse validation; safe injection (via json.dumps); narrow-scope base64 fallback for triple-quote anomalies; isolated namespace execution.
Settings & UI Consistency
Only four adaptive global override keys are exposed:
• chat_timeout_base
• chat_timeout_per_1k
• util_timeout_base
• util_timeout_per_1k
UX Noise Reduction
Recoverable “minor JSON misformat” is downgraded to non-error diagnostics: not written to history, not logged as an error, and does not trigger UI red dots.
Memory & Consolidation
Merging process adopts adaptive timeout; parsing made more robust; conservative REPLACE threshold with concurrent re-validation; temporal metrics recorded.
⸻
Tests Completed
• Adaptive timeout + slow-model multiplier persistence (default 10): Done
• Retry + jitter + API key rotation: Done
• Streaming tail flush: Done
• Log noise reduction & key masking: Done
• Layered JSON parsing & repair: Done
• Hardened Python execution: Done
• Prevent delegation loops & preserve memory sequence: Done
• Misformatted JSON warnings downgraded (no history red dots): Done