[DSIP-95][API] Complete the functionality of using dependencies in the complement data

[det101]

Search before asking

• I had searched in the DSIP and found no similar DSIP.

Motivation

Current dependency mode configuration in frontend cannot connect the complement data dependency enabling action with downstream workflows. We want to improve the Complement Data process so that, it correctly triggers the downstream workflows according to the selected dependency mode. This enhancement will support two dependency modes.

The current DSIP replaces and close #17741

Design Detail

• Add implementation for frontend dependency mode switching
• Supports a toggle to trigger either all downstream dependencies or only direct downstream. Default is off, triggering only the next layer; when on, recursively triggers all downstream workflow levels.
• Implement traversal logic for recursive triggering, with protection against cycles and duplicate triggering.
• Update UI to reflect mode and ensure users can select preferred behavior.
• Add tests for both triggering modes.

Compatibility, Deprecation, and Migration Plan

This improvement does not introduce breaking changes to existing workflows, but recursive triggering should be idempotent to prevent duplicate executions. Cycle detection will be implemented to protect against circular dependencies.

Test Plan

• Verify direct and recursive triggering work correctly for various workflow dependency graphs.
• Add automated tests on workflow DAGs for both dependency modes.
• Manual test for large-scale workflow with complex dependencies to ensure reliability and performance.
• Test cycle detection and duplicate triggering prevention.

Code of Conduct

• I agree to follow this project's Code of Conduct

[SbloodyS]

We want to improve the rerun workflow so that it correctly triggers the downstream workflows after backfill, supporting two dependency modes.

I'm a little confused. Do you want to optimize the complement process or rerun the workflow process?

[det101]

We want to improve the rerun workflow so that it correctly triggers the downstream workflows after backfill, supporting two dependency modes.

I'm a little confused. Do you want to optimize the complement process or rerun the workflow process?

It should be "improve the complement data process," the wording was incorrect and has been corrected. @SbloodyS

[SbloodyS]

It should be "improve the complement data process," the wording was incorrect and has been corrected. @SbloodyS

Thanks for clarification. Before PR, there are some details needs to be discuss.

1. Is it better to use provide the user with the specified level in the basic mode and set 1 as the default value? @det101

[det101]

It should be "improve the complement data process," the wording was incorrect and has been corrected. @SbloodyS

Thanks for clarification. Before PR, there are some details needs to be discuss.

1. Is it better to use provide the user with the specified level in the basic mode and set 1 as the default value? @det101

That sounds great. I’ll update the design so users can specify the recursion depth and we’ll keep the default at 1.

[SbloodyS]

I have no more questions. Let's see if others have other opinions.

[ruanwenjun]

Does recursive mode with user-defined depth is really useful? This might be difficult to use.

[det101]

Does recursive mode with user-defined depth is really useful? This might be difficult to use.

After discussion and careful consideration, we found that in complex dependency scenarios, specifying a custom level may trigger dependency chains that end up waiting on one another. Supporting this feature would likely require recursively checking whether upstream nodes are part of the dependency chain, which is significantly more complex than initially anticipated. I recommend implementing the existing functionality first and discussing the custom-level feature separately. @ruanwenjun @SbloodyS