feat: Support parallel asynchronous tool calls with task ids

prompts/agent.system.main.md

@@ -8,4 +8,6 @@
 
 {{ include "./agent.system.main.solving.md" }}
 
+{{ include "./agent.system.parallel_tools_workflow.md" }}
+
 {{ include "./agent.system.main.tips.md" }}

prompts/agent.system.main.solving.md

@@ -11,14 +11,36 @@ agentic mode active
 2 break task into subtasks if needed
 
 3 solve or delegate
-tools solve subtasks
+tools solve subtasks - use run_task for parallel execution
 you can use subordinates for specific subtasks
 call_subordinate tool
 use prompt profiles to specialize subordinates
 never delegate full to subordinate of same profile as you
 always describe role for new subordinate
 they must execute their assigned tasks
 
+### Parallel Tool Execution Workflow:
+use run_task tool to wrap other tools for background execution
+direct tool calls execute synchronously (blocking)
+**IMPORTANT**: after starting background tasks, CONTINUE your monologue - don't stop thinking
+use wait_for_tasks tool with task IDs to retrieve actual results when ready
+you can start multiple background tasks for efficiency, then collect all results
+
+**Example workflow:**
+- "I need to search and analyze code, let me start both tasks"
+- call run_task(tool_name="search_engine", args='{"query":"test"}') → get task ID abc123
+- "Now starting code analysis while search runs"
+- call run_task(tool_name="code_exe", args='{"code":"analysis"}') → get task ID def456
+- "Both tools running, let me collect results"
+- call wait_for_tasks with ["abc123", "def456"] → get both results
+- "Based on the search results and code analysis..."
+
+**After starting background tasks:**
+1. KEEP THINKING - your monologue continues
+2. You can start additional background tasks if needed
+3. When ready, use wait_for_tasks to get results
+4. Only use response tool for final answer to user
+
 4 complete task
 focus user task
 present results verify with tools

prompts/agent.system.parallel_tools_workflow.md

@@ -0,0 +1,151 @@
+# Parallel Tools Workflow Instructions
+
+## Key Concept: Explicit Parallel Execution with run_task
+You now have full control over which tools run in parallel. Use the `run_task` wrapper tool to execute any tool in the background. All tools execute synchronously by default unless wrapped with `run_task`.
+
+## Important Workflow Rules
+
+### **1. Understanding Parallel Execution**
+- **Direct tool calls** (like `search_engine:search`) execute synchronously (blocking)
+- **Wrapped tool calls** (like `run_task` with `search_engine`) execute in background
+- Background tasks run in isolated temporary contexts that auto-clean up
+- You receive a **task ID** immediately for background tasks
+- **IMPORTANT**: after starting a background task, CONTINUE your monologue - don't stop thinking
+
+### **2. Two Execution Methods**
+
+#### Synchronous (Direct) Execution
+```json
+{
+  "tool_name": "search_engine",
+  "tool_args": {"query": "test"}
+}
+```
+- Blocks until completion
+- Returns actual result immediately
+- Use for simple, quick operations
+
+#### Asynchronous (Background) Execution
+```json
+{
+  "tool_name": "run_task",
+  "tool_args": {
+    "tool_name": "search_engine",
+    "method": "search",
+    "args": "{\"query\": \"test\"}"
+  }
+}
+```
+- Returns task ID immediately
+- Runs in isolated background context
+- Use for parallel/long-running operations
+
+### **3. When to Use Each Method**
+
+#### Use Direct Execution For:
+- Quick, simple operations
+- Single tool calls where you need immediate results
+- Final tools like `response`
+
+#### Use Background Execution (`run_task`) For:
+- Multiple parallel research tasks
+- Long-running operations (code execution, complex searches)
+- When you want to continue thinking while tool runs
+- Gathering information from multiple sources simultaneously
+
+### **4. Correct Parallel Workflow Example**
+
+```
+Agent: "I need comprehensive research on this topic."
+
+🔧 run_task(tool_name="search_engine", method="search", args='{"query": "topic A"}')
+→ Returns: "Task 'search_engine:search' running with ID: abc123..."
+
+Agent: "While that search runs, let me start researching related aspects."
+
+🔧 run_task(tool_name="search_engine", method="search", args='{"query": "topic B"}')
+→ Returns: "Task 'search_engine:search' running with ID: def456..."
+
+Agent: "I'll also check for code examples in parallel."
+
+🔧 run_task(tool_name="code_exe", method="execute", args='{"language": "python", "code": "example_code"}')
+→ Returns: "Task 'code_exe:execute' running with ID: ghi789..."
+
+Agent: "Now I'll collect all the results."
+
+🔧 wait_for_tasks(tool_call_ids="abc123,def456,ghi789")
+→ Returns: Full results from all three tasks
+
+Agent: "Based on the comprehensive research and code execution results..."
+
+🔧 response(message="Here's my complete analysis...")
+```
+
+### **5. Wrong vs Right Behavior**
+
+#### ❌ Wrong: Stopping After Background Task Start
+```
+Agent: "I'll search for information"
+🔧 run_task(tool_name="search_engine", args='{"query": "test"}')
+→ Returns: "Task running with ID: abc123..."
+[AGENT STOPS - WRONG!]
+```
+
+#### ✅ Right: Continuing After Background Task Start
+```
+Agent: "I'll search for information"
+🔧 run_task(tool_name="search_engine", args='{"query": "test"}')
+→ Returns: "Task running with ID: abc123..."
+Agent: "While that runs, let me also check documentation"
+🔧 run_task(tool_name="search_engine", args='{"query": "documentation"}')
+→ Returns: "Task running with ID: def456..."
+Agent: "Now I'll retrieve both results"
+🔧 wait_for_tasks(tool_call_ids="abc123,def456")
+```
+
+### **6. Mixed Execution Strategy**
+You can combine both approaches:
+
+```
+Agent: "Let me start some background research while I do quick analysis"
+
+🔧 run_task(tool_name="search_engine", args='{"query": "complex topic"}')
+→ Background task started: abc123
+
+Agent: "While that runs, let me do a quick calculation"
+
+🔧 code_exe(language="python", code="2 + 2")
+→ Immediate result: 4
+
+Agent: "Now let me get the background research"
+
+🔧 wait_for_tasks(tool_call_ids="abc123")
+→ Retrieved background results
+```
+
+## Critical Success Rules
+
+### **Rule 1: Choose Execution Method Wisely**
+- Use `run_task` for parallel/long operations
+- Use direct calls for immediate needs
+
+### **Rule 2: Continue Your Monologue**
+After starting background tasks, **keep thinking and planning**. Don't wait immediately.
+
+### **Rule 3: Use Parallel Execution for Efficiency**
+Start multiple independent background tasks simultaneously.
+
+### **Rule 4: Batch Result Collection**
+Collect multiple task results in one `wait_for_tasks` call when possible.
+
+### **Rule 5: Plan Your Workflow**
+Think about which operations can run in parallel before starting them.
+
+## Tool Reference
+
+### Core Tools for Parallel Workflow:
+- **`run_task`** - Wraps any tool for background execution
+- **`wait_for_tasks`** - Collects results from background tasks
+- **`response`** - Provides final answer (always synchronous)
+
+**Remember**: You now have explicit control over parallel execution. Use `run_task` when you want background execution, and direct tool calls when you need immediate results!

prompts/agent.system.tool.response.md

@@ -2,6 +2,11 @@
 final answer to user
 ends task processing use only when done or no task active
 put result in text arg
+
+**Important for parallel tools workflow:**
+- This tool is synchronous and ends your monologue
+- Use only after collecting all needed results with wait_for_tasks
+- Continue your monologue after starting async tools, only use response for final answer
 usage:
 ~~~json
 {

prompts/agent.system.tool.run_task.md

@@ -0,0 +1,35 @@
+### run_task Tool
+
+## Description
+The `run_task` tool executes other tools in isolated background contexts.
+
+#### Batched Tool Calls (required)
+- Use `tool_calls` to pass a JSON object mapping `call_id` to a tool call object
+- Each `call_id` must be a non-empty string with no whitespace characters
+- Each tool call object must include:
+  - `tool_name` (required): Name of the tool to execute
+  - `method` (optional): Method to call on the tool
+  - `args` (optional): Object or JSON string of arguments for the tool
+  - `message` (optional): Message context for the tool
+- The tool starts all calls and returns a mapping from `call_id` to task UUID or an error string
+- Response format: one pair per line as `call_id: <uuid>` or `call_id: Error: <text>`
+
+##### Example
+```json
+{
+  "tool_name": "run_task",
+  "tool_args": {
+    "tool_calls": "{\"search1\": {\"tool_name\": \"search_engine\", \"method\": \"search\", \"args\": {\"query\": \"vector databases\"}}, \"code1\": {\"tool_name\": \"code_exe\", \"method\": \"execute\", \"args\": {\"language\": \"python\", \"code\": \"print('hi')\"}} }"
+  }
+}
+```
+
+#### Behavior
+- Each task runs in an isolated temporary BACKGROUND context and is cleaned up after completion
+- Results are stored and can be retrieved later with `wait_for_tasks`
+- On input or runtime error for a call, the returned value is `Error: <text>` for that `call_id`
+
+#### Notes
+- Avoid whitespace in `call_id` (no spaces, tabs, or newlines)
+- Prefer batching independent tool invocations to reduce overhead
+- Use `wait_for_tasks` with task IDs returned in the response to collect results

prompts/agent.system.tool.wait_for_tasks.md

@@ -0,0 +1,68 @@
+### wait_for_tasks
+
+#### Description
+The `wait_for_tasks` tool allows you to retrieve results from background tasks started with the `run_task` tool. When you use `run_task` to wrap other tools, they run asynchronously in isolated temporary contexts and return a task ID immediately. Use this tool to collect the actual results from those background tasks.
+
+#### Usage
+```json
+{
+  "tool_name": "wait_for_tasks",
+  "tool_args": {
+    "tool_call_ids": "task-id-1,task-id-2,task-id-3"
+  }
+}
+```
+
+#### Parameters
+- **tool_call_ids** (required): Comma-separated list of task IDs to wait for and retrieve results from
+
+#### Behavior
+- This tool executes synchronously in the current context
+- It retrieves results from background tasks started with `run_task`
+- It will wait for the specified tasks to complete if they are still running
+- Returns the results of all specified tasks (results are preserved even after context cleanup)
+- Tasks that are already completed will return their cached results immediately
+- Tasks that don't exist will be reported as "not found"
+- Temporary contexts are automatically cleaned up after tool execution
+
+#### Example Workflow
+1. Start background tasks using run_task:
+   ```json
+   {"tool_name": "run_task", "tool_args": {"tool_name": "code_exe", "method": "execute", "args": "{\"code\": \"print('Task 1')\"}"}}
+   ```
+   → Returns: "Task started with ID: abc123..."
+
+2. Start another background task:
+   ```json
+   {"tool_name": "run_task", "tool_args": {"tool_name": "search_engine", "method": "search", "args": "{\"query\": \"python asyncio\"}"}}
+   ```
+   → Returns: "Task started with ID: def456..."
+
+3. Retrieve all results:
+   ```json
+   {"tool_name": "wait_for_tasks", "tool_args": {"tool_call_ids": "abc123,def456"}}
+   ```
+   → Returns results from both background tasks
+
+#### Critical Workflow Instructions
+- **After starting background tasks**: CONTINUE YOUR MONOLOGUE - don't stop thinking
+- **Your role continues**: The system expects you to keep reasoning and planning
+- **When to collect**: Use `wait_for_tasks` when you're ready to use the results
+- **Multiple tasks**: You can start several background tasks with `run_task`, then collect all results together
+- **Keep working**: After getting results, continue analysis and provide final response
+
+#### Important Notes
+- Always use `wait_for_tasks` to get actual tool results
+- Task IDs are UUIDs generated automatically
+- You can wait for multiple tasks at once for efficiency
+- Tasks are tracked in the system prompt under "Tasks in Progress"
+- Each tool runs in its own isolated temporary context (including subordinate agents)
+- Contexts are automatically cleaned up - only results are preserved
+- This prevents context pollution and ensures clean execution environments
+
+#### Expected Agent Behavior After Background Task Start
+1. **Background task starts** → You see "Task 'X:method' running with task ID: abc123"
+2. **KEEP THINKING** → Continue your monologue, don't stop here
+3. **Plan next steps** → Consider starting more background tasks or preparing for results
+4. **Collect when ready** → Use wait_for_tasks to retrieve actual results
+5. **Continue working** → Analyze results and provide final answer

python/extensions/message_loop_prompts_after/_80_track_tool_tasks.py

@@ -0,0 +1,48 @@
+from typing import Any
+from python.helpers.extension import Extension
+from agent import Agent, LoopData
+
+
+class TrackToolTasks(Extension):
+    """
+    Extension that tracks active tool tasks and adds them to the system prompt extras.
+    This allows the agent to see what tool calls are currently in progress.
+    """
+
+    async def execute(self, loop_data: LoopData = LoopData(), **kwargs: Any):
+        """
+        Add information about active tool tasks to the system prompt extras.
+        """
+
+        # Get active tasks from agent data
+        active_tasks = self.agent.get_data("active_tool_tasks") or {}
+
+        if not active_tasks:
+            # No active tasks, nothing to add
+            return
+
+        # Build the tasks in progress section
+        tasks_info = []
+        tasks_info.append("## Tasks in Progress")
+        tasks_info.append("")
+        tasks_info.append("The following tool calls are currently running in parallel in isolated temporary contexts:")
+        tasks_info.append("")
+
+        for task_id, task_info in active_tasks.items():
+            tool_name = task_info.get("tool_name", "unknown")
+            started_at = task_info.get("started_at", "unknown")
+
+            tasks_info.append(f"- **Task ID**: `{task_id}`")
+            tasks_info.append(f"  - **Tool**: {tool_name}")
+            tasks_info.append(f"  - **Started**: {started_at}")
+            tasks_info.append(f"  - **Context**: Isolated temporary context (auto-cleanup)")
+            tasks_info.append("")
+
+        tasks_info.append("**Important**: Each tool runs in its own temporary context that is automatically")
+        tasks_info.append("cleaned up after execution. Only the results are preserved.")
+        tasks_info.append("")
+        tasks_info.append("To retrieve results from these tasks, use the `wait_for_tasks` tool with the task IDs.")
+        tasks_info.append("Example: `wait_for_tasks` with `tool_call_ids` parameter containing comma-separated task IDs.")
+
+        # Add to extras_temporary so it appears in the system prompt
+        loop_data.extras_temporary["tasks_in_progress"] = "\n".join(tasks_info)

python/extensions/monologue_start/_10_memory_init.py

@@ -1,13 +1,14 @@
 from python.helpers.extension import Extension
-from agent import LoopData
+from agent import LoopData, AgentContextType
 from python.helpers import memory
 import asyncio
 
 
 class MemoryInit(Extension):
 
     async def execute(self, loop_data: LoopData = LoopData(), **kwargs):
-        db = await memory.Memory.get(self.agent)
-
-
-
+        # Avoid blocking startup for background agents; initialize lazily
+        if self.agent.context.type == AgentContextType.BACKGROUND:
+            asyncio.create_task(memory.Memory.get(self.agent))
+            return
+        await memory.Memory.get(self.agent)