Add option to pass config as env variable, not just as file path to local file

[MPettersen]

Not all our deployments have access to local file storage, so needed to pass the config as a local file is a bit cumbersome and limits when we are able to use this package.

Hub post:
https://hub.cognite.com/developer-community-427/the-extractor-utils-requires-a-config-file-path-no-way-to-pass-the-config-object-directly-5181

Regards,
Markus Pettersen
Aker BP - Senior Technical Domain Architect - CDF Platform

[eighty20results]

Hi @MPettersen.

I think the biggest challenge with implementing something like this would be the environment variable limits of the underlying operating system the deployment is running on (which varies).

From a utility perspective, I can't think of a uniform approach to solving this problem since we would need to support potentially long inclusion and exclusion lists (for some extractors), safely encode and decode regular expressions (for others), etc, etc.

But I'm only a single individual, perhaps some of the contributors to this package can think of something (@mathialo, @ozangoktan, @toondaey, @einarmo)?

[einarmo]

In the new unstable module of the utils, which are going to supersede the current utils in the future, we do support loading the initial connection config from env (only the configuration that is necessary to load the real config file from CDF)

[eighty20results]

Thanks @einarmo!

Also, once we release our updated approach to configuration - which the ENV based connection config updates in the unstable module are meant to support, management, and observability of the on-premises extractors, the remaining configuration will be something that can be edited and loaded from CDF (should address the cases you're mentioning above @MPettersen).