feat: timeouts

[tristan-f-r]

Adds timeout to algorithms as a demonstration of passing through errors. Closes #316.

Caveat: ML requires at least one pathway, and failing pathways can break ML-work. How do we want to handle downstream analysis when errors occur (including in the future heuristic errors?)

[read-the-docs-community]

Documentation build overview

📚 spras | 🛠️ Build #32419831 | 📁 Comparing 7a0c4f3 against latest (87b314c)

  🔍 Preview build  

6 files changed · + 2 added · ± 4 modified

+ Added

• timeout.html
• design/errors.html

± Modified

• genindex.html
• index.html
• fordevs/modules.html
• fordevs/spras.html

[ntalluri]

This does not work with singularity (singularity has no docker wait equivalent and to implement timeouts in singularity would probably require constant polling of a detached thread)

This is a problem; if this is something we are going to use for the benchmarking study we need this to work with singularity because CHTC only uses singularity/apptainer.

[tristan-f-r]

I originally assumed this was a more esoteric PR to test the error-handling workflow. Though, based on the meeting just now, I'll look into a nice way to get this working with Singularity.

[ntalluri]

Here is my first round review. I like the empty files if an error occurs, and it is good to have an associated log explaining why.
(Adding this comment again here) My only issue with this is the definition of "error." If a parameter combination fails the heuristics check, I do not want the output to be empty. I want the output to reflect what was actually produced, so I do not have to rerun that combination even though it "failed" the heuristics. I should be able to freely update the heuristics and have that output counted if it now passes, without rerunning combinations that previously produced empty output. In short, a parameter combination failing the heuristics should not be classified as an error as defined in this PR.

This PR not working with singularity is a very big problem because of the chtc integration.

"ML requires at least one pathway, and failing pathways can break ML-work. How do we want to handle downstream analysis when errors occur." This seems like a separate problem that I will fix internally in the ML code (I want to still make figures if we only have one pathway or a set of empty pathways).

[tristan-f-r]

My only issue with this is the definition of "error." If a parameter combination fails the heuristics check, I do not want the output to be empty. I want the output to reflect what was actually produced, so I do not have to rerun that combination even though it "failed" the heuristics.

The small part here that is adaptable is that errors, in this PR, are only made in the reconstruct rule. We can define other errors in some heuristics rule, and we can rewire other rules to depend on the success of heuristics instead of on reconstruct (via that resource_info = rules.reconstruct.output.resource_info input rule above, but instead we would say rules.heuristics.output.resource_info instead.)

[tristan-f-r]

This works with apptainer now, but this now touches an untested part of profiling, so that part needs a review from @jhiemstrawisc.

[ntalluri]

here is half of a review on this pr

[ntalluri]

can we make this a separate section that explains each error instead of putting it in parentheses (even though right now it is only timeouts). We can expand this in the future.

[ntalluri]

Suggested change

	<https://pypi.org/project/pytimeparse/>`__, which allows for more
	<https://pypi.org/project/pytimeparse/>`__, (examples linked here). This allows for more

[ntalluri]

Suggested change

	SPRAS allows for per-algorithm timeouts, specified under the global
	SPRAS allows for optional per-algorithm timeouts, specified under the global

[ntalluri]

can you also add a sentence on what happens when timeout is not included?

[ntalluri]

could we convert this back into other time scales (if I put 5.6 days I don't want to have to read that as 483840 seconds).

Suggested change

	self.message = f"Timed out after {timeout}s."
	self.message = f"Timed out after {timeout} seconds."

[ntalluri]

f"Timed out after {timeout} seconds; {hours} hours; {days} days ."

Something like this?

[ntalluri]

(only do this if it is easy to do)