Luigi Parameters Unable to Be Set Programmatically

[DanielScholtyssek]

Issue Description:

We are encountering a significant issue with setting configuration parameters programmatically or globally in the cls/cls_luigi framework, which is built on top of the luigi. The standard method for defining parameters in luigi does not seem to function correctly within cls_luigi due to the pipeline synthesis process.

Problem Details:

• The typical approach for defining and setting parameters in luigi (using luigi.Parameter) is ineffective in cls_luigi.
• The pipeline appears to gather parameters at the time of creating the RepoMeta, which complicates the ability to parameterize tasks dynamically.
• This limitation restricts our ability to manage configurations efficiently, particularly when task parameters need to be adjusted based on different conditions or environments.

Suggested Improvements:

1. Refactor Pipeline Creation: Modify the pipeline synthesis process to allow for more flexible parameter management, enabling the use of luigi.Parameter as intended.
2. Global Configuration Management: Implement a system that allows for global configuration settings to be accessed and modified programmatically across different tasks.
3. Documentation and Examples: Provide clear documentation or examples demonstrating how to achieve dynamic configuration settings within the cls_luigi framework.