[Bug][tasks] Duplicate subtasks running

[kostas-petrakis]

Search before asking

• I had searched in the issues and found no similar issues.

What happened

I cancelled a transformation pipeline but the pipeline remained hanging. I eventually killed the pod, once restarted Apache Devlake resumed (although cancelled) the pipeline, but now I see two tasks running.

What do you expect to happen

Pipeline is cancelled

How to reproduce

Not sure, as I never experienced this issue before. I was running a pipeline which was cancelled because I needed to change the scope config.
Cancelling the pipeline resulted in a stuck state, I had to actually kill the pod, since I cannot access the DB directly.
Restarting the pod, resulted in the pipeline resuming but I see now two task ID's with the exact same finished_records column updates.

Anything else

No response

Version

v1.0.2-beta4

Are you willing to submit PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct

[dosubot]

The issue you're experiencing with duplicate tasks running after a pod restart might be related to how Apache DevLake handles task resumption and cancellation. When a pod is killed and restarted, the pipeline status is marked as interrupted. If the RESUME_PIPELINES configuration is set to true, the pipeline is set to resume, which might lead to the observed behavior of duplicate tasks if the cancellation process wasn't fully completed before the pod was killed ^[1][2].

To address this, you might want to check the RESUME_PIPELINES setting in your configuration. If it's set to true, consider setting it to false to prevent automatic resumption of tasks after a pod restart. Additionally, ensure that the task cancellation process is completed before killing the pod to avoid leaving tasks in an inconsistent state.

If this issue persists, it might be beneficial to investigate further into the task management and cancellation logic to identify any potential edge cases or race conditions that could lead to duplicate task entries.

To continue talking to Dosu, mention @dosu.

---

^{Help Dosu learn! Give it feedback: Great Response | Irrelevant Answer | Incorrect Sources | Too Verbose | Hallucination | Bug Report | Other}

[kostas-petrakis]

@dosu the resume is expected, the issue is that there are two entries in the _devlake_subtasks table for the same subtask.

[dosubot]

Apache DevLake ensures the uniqueness of subtask entries in the _devlake_subtasks table by using a composite primary key consisting of the Plugin, Subtask, and Params fields. This setup should prevent duplicate entries for the same subtask ^[1].

If you're seeing duplicate entries, it might indicate a race condition or an issue in the task management logic that wasn't fully addressed. Unfortunately, I couldn't find specific information on known issues or race conditions that could lead to this situation. It might be helpful to review the task management and resumption logic to ensure that the uniqueness constraints are being properly enforced during pipeline resumption.

To continue talking to Dosu, mention @dosu.

---

^{Help Dosu learn! Give it feedback: Great Response | Irrelevant Answer | Incorrect Sources | Too Verbose | Hallucination | Bug Report | Other}

[kostas-petrakis]

Adding some extra context, the issue has auto resolved, pipelines finished and Apache Devlake resumed as normal. Feel free to keep this open if you would like to test this corner case.