[BUG] Flyte subworkflow nodes can be stuck even after referenced launchplans executions are complete

[sshardool]

Flyte & Flytekit version

flyte: v1.16.3
flytekit: 1.16.12

Describe the bug

Summary
Flyte subworkflow nodes with reference launchplans can stay in the Running state long after the target execution has completed.

Behavior
The screenshots below once such example which can consistently reproduce the issue.

• Parent workflow triggers a reference launch plan execution, node n7 - another_sleepy_lp
• The child reference launchplan executions finishes in under 3.5 minutes.
• Corresponding node n7 in the parent executions takes over 1 hour for completion.

Note that the problematic behavior is not seen for all workflows and is highly dependent on the dag-structure and the run-times of individual nodes in the dag.

Expected behavior

Parent execution's reference launchplan node n7 should get updated within about 30 seconds for child's completion (workflow evaluation re-enqueue frequency)

Additional context to reproduce

The wokflows defined here can consistenly reproduce the issue: https://github.com/sshardool/flyte/blob/4feb41fd0036dde78009bc9962ef269546e54f08/workflows/max-parallel/max_parallelism_workflows.py

Note that structure of the dags is carefully crafted to trigger the issue with nodes appearing in a very specific order.

Screenshots

Are you sure this issue hasn't been raised already?

• Yes

Have you read the Code of Conduct?

• Yes

[runllm]

It could be possible that this issue is related to how Flyte handles the synchronization and status propagation of subworkflow nodes that reference launch plans. According to the Flyte documentation, when a workflow node references a launch plan, it triggers a new workflow execution with its own execution context, and the parent node should update its status based on the child execution's completion. However, the actual update timing may depend on the workflow evaluation re-enqueue frequency and the orchestration logic in FlytePropeller, which can be affected by DAG structure and node run-times. There is no explicit mention in the provided data sources of a known bug causing subworkflow nodes to remain in the Running state long after the referenced launch plan execution is complete, but the orchestration complexity and event handling described could contribute to such behavior in edge cases (tasks, workflows, and launch plans, performance benchmarking).

Though I'm not entirely certain, it might be that the issue is related to how FlytePropeller tracks and reconciles node states, especially when dealing with high parallelism or specific DAG structures. The eventing and state update system in Flyte is designed to eventually reconcile the desired and observed states, but delays can occur due to factors like Kubernetes watch latency, max parallelism settings, or scheduling bottlenecks (performance benchmarking). There is no direct reference to a fix or workaround for this specific scenario in the data sources. Would you like a more detailed breakdown of the orchestration flow and possible debugging steps?

Sources:

• tasks, workflows, and launch plans
• performance benchmarking

---

This is an AI-generated response and your feedback is appreciated! Please leave a 👍 if this is helpful and 👎 if it is not.

Mention @RunLLM to ask follow-up questions. See runllm.com for more!

[sshardool]

Root Cause
The issue happens for a couple of reasons all of them related to the recursive workflow dag evaluation in RecursiveNodeHandler within nodes/executor.go

Primary reason : max-parallelism implementation for limiting the number of concurrently running nodes is not correct and can lead to a situation where a node can start execution following which other nodes start and due to the depth-first nature of the traversal the nodes started subsequently are picked up first for evaluation. This leads to the previously started node getting starved within RecursiveNodeHandler and the child workflow status is not checked until another parallelism slot becomes available.
Example workflows that reproduce the issue here

[sshardool]

We have a fix for this applied within our internal fork (at LinkedIn).
Planning to create a PR later in the week. Feel free to assign the issue to me.