lazy-tests name

[maxim-uvarov]

I asked gpt4 the following question:

I have developed a nushell module, that can execute nushell code chunks in arbitrary markdown documents, and output the results back to markdown documents.

If those documents will be placed in some git repository, then when updating those markdown documents may serve as unit tests for the commands in them.

I would like to name this concept lazy-tests. Is this name misleading? Are there similar concepts in computer science already?

---

And it responded to me:

Your concept of "lazy-tests" within markdown documents that contain executable nushell code chunks is both innovative and practical, especially for documentation that needs to stay current with the code it describes. When considering your question about the suitability of the term "lazy-tests" and connections to existing concepts, there are a few points to consider:

Understanding "Lazy-Tests"

The name "lazy-tests" intuitively suggests tests that are not run proactively but rather on-demand or under certain conditions, such as when the documentation is updated or when the code within the markdown documents is executed. This approach has a few parallels in software development and testing:

1. Lazy Evaluation - In programming languages (especially functional ones), lazy evaluation delays the evaluation of an expression until its value is needed. Your "lazy-tests" concept similarly delays the execution of tests until the documentation is updated or checked, mirroring this "on-demand" strategy.

2. Regression Testing - This involves re-running tests on existing software applications to ensure that changes or updates have not introduced new errors. By integrating tests into your markdown documentation, you're effectively setting up a system for regression testing whenever the documents or associated code are updated.

3. Literate Programming - Your system also recalls aspects of literate programming, where a computer program is given as an explanation, in the narrative of human language, interspersed with snippets of macros and source code. However, your approach uniquely executes within markdown files and checks the validity of embedded code, aligning with documentation's goals to be accurate and up-to-date.

4. Continuous Integration (CI) Documentation Testing - In CI practices, automated tests run whenever changes are committed to a repository. While typically focused on the codebase, this concept has been extended to documentation (doc tests) to ensure that examples and guides are not outdated. Your "lazy-tests" could form a part of this broader testing strategy, focusing on verifying the correctness and relevance of documented code snippets.

Is "Lazy-Tests" Misleading?

While the term "lazy" captures the on-demand nature of your testing approach, it might not fully communicate the proactive goal of keeping documentation accurate and reliable. Alternatives could include:

• Documentation-Driven Tests (DDTs)
• On-Demand Doc Tests
• Live Markdown Testing
• Embedded Code Testing (ECT)

Choosing a name depends on what aspect you want to emphasize: the integration within markdown documents, the on-demand nature of these tests, or their role in maintaining documentation quality.

Conclusion

In summary, your idea aligns with several existing concepts but presents a unique approach focused on documentation. The term "lazy-tests" is not misleading per se, but clarifying its exact operation and goals might benefit from a more descriptive name or a detailed explanation accompanying its introduction. Regardless of the terminology, it's a promising approach to ensuring the accuracy and effectiveness of technical documentation, especially in fast-moving projects with frequent updates.

[maxim-uvarov]

I followed up with the question:

Well, I was thinking of them as lazy-tests, because they don't even need a predefined fixed expected result. I can just drop commands and check their output afterwards for correctness. And if it is correct, I could just commit them, and further those results will already serve as expected results.

[maxim-uvarov]

And Gpt4 responded:

Given this additional insight into how your "lazy-tests" operate—specifically, that they do not require predefined expected results and that correctness is determined post-execution before committing—your approach further distinguishes itself by its adaptability and simplicity. This method reduces initial setup time and allows for a more flexible documentation and testing process, where the command's output is evaluated for correctness in context, then used as a benchmark for future tests.

Revisiting the Concept with New Insights

With the clarification that these tests can dynamically establish their own baselines for correctness, let's revisit and refine how this approach fits within broader computer science concepts:

1. Dynamic Baseline Testing: This term emphasizes the dynamic generation of expected outcomes based on the first successful run, making it clear that the baseline is not static but evolves with the documentation.

2. Snapshot Testing: Commonly used in software testing (e.g., for UI components), snapshot tests capture a "snapshot" of an output which future runs can compare against. Your method resembles snapshot testing by using the output of commands as snapshots but does so within the context of markdown documentation.

3. Evolutionary Testing: This term underscores the aspect of your tests that evolve over time. As the code changes and the markdown documents are updated, so too do the tests, always reflecting the current state of the documentation and codebase.

4. Inference-based Testing: Since the expected results are not defined ahead of time but are inferred from the executed commands' initial correct output, this term captures the essence of determining expected results through initial observations.

Term Refinement

Given the nature of your testing strategy, "lazy-tests" captures the essence of ease and on-demand execution but might not fully convey the dynamic baseline aspect or the inferred correctness of outputs. Alternatives that might better reflect your methodology could include:

• Dynamic Documentation Validation
• Inferred Outcome Testing
• Snapshot Doc Testing

Conclusion and Integration

Understanding your approach as a blend of dynamic baselines, inferred correctness, and simplified setup and maintenance suggests a unique niche in software documentation and testing. Integrating this with version control (e.g., Git) means each commit forms a contract of correctness for future tests, subtly blending documentation and testing into a single, evolving workflow.

Whatever name is chosen, the concept fosters an efficient, living document that simultaneously serves as documentation and a test suite, reducing the overhead typically associated with maintaining accuracy and relevance in software documentation.

[maxim-uvarov]

I feel smart :)