Add end-to-end (e2e) tests for Apify CLI

[patrikbraborec]

Summary

Adding e2e tests will help iterate much faster and catch regressions across the full CLI surface.

Action Items

• Set up e2e test infrastructure
• Add e2e tests covering core CLI workflows

Reference

• mcpc implementation for inspiration on e2e test setup

[janbuchar]

When this happens, we can remove the beta specifier from crawlee e2e tests https://github.com/apify/crawlee/blob/9586a6aa231c679b00bb527087e9ccfa7bf0f49c/test/e2e/tools.mjs#L74

[l2ysho]

I have build simple matrix of where/how/what is tested right now https://www.notion.so/apify/CLI-State-of-tests-33cf39950a2280cfb73dcfc6eb78945f + what are blind spots

1. drop cucumber e2e ? (now real coverage, running only manually) or we can continue cucumber e2e but probably there is a reason why is it not used :] cc @vladfrangu @B4nan
2. if so, how we want to do e2e?

• there is this bash approach from mcpc
• or we can still use vitest + execa or similar tool

  const { stdout, stderr, exitCode } = await execa('node', ['dist/apify.js', 'push', '--force']);

3. we have som blind spots in testing, so with e2e also api and unit tests should be updated

In general this looks too big to make it in one shot, so I would prefer to choose a way how to do it and pick some part to demonstrate it in this task. Other parts can be done in few iterations and multiple people can "touch" e2e.

[patrikbraborec]

Thanks for the analysis! We can break it down to separate issues. I suggest to cover the most used commands, and then we can slowly add coverage for the rest. We will probably add new commands during upcoming weeks and months, so it would be great to kick this project off so we can cover the new commands with e2e tests.

The analytics of most used commands:

I think we can start with apify run, apify push, apify call, apify login.

[B4nan]

drop cucumber e2e ? (now real coverage, running only manually) or we can continue cucumber e2e but probably there is a reason why is it not used :] cc @vladfrangu @B4nan

Honestly, I am fine dropping those, unless @patrikbraborec would like to start contributing those himself. If there are missing spots that were only covered through them, we should improve the regular tests we have.

[patrikbraborec]

I am ok to drop them as it is not used anyway on CI (if I understand it correctly) = not used at all :)

[vladfrangu]

Cucumber was meant to be much simpler to write many tests that act like the CLI is ran in a terminal instead of having to deal with coding (that said between documenting it better and claude existing we might be able to just...not need it)

If we don't see any benefits lets nuke em. Or we use the harness already there adapted for e2e testing instead (so it deals with installing cli and running it from npm and bundle)

[l2ysho]

ok here is the deal

• kill cucumber
• vitest + execa to run (as we already have it in dependecies)
  • locally we can use /dist
  • in CI/CD last beta release
• migrate old cucumber to this approach (just few tests but already covers push and auth)

I create a draft with changes and we can discuss there and iterate (if you are ok with this approach) @B4nan @vladfrangu @patrikbraborec

[patrikbraborec]

Sounds good, go for it.