Skip to content

reduce eyes parallelism from 79 to 20 to deflake drone#72312

Merged
davidsbailey merged 3 commits intostagingfrom
decrease-eyes-parallelism
Apr 23, 2026
Merged

reduce eyes parallelism from 79 to 20 to deflake drone#72312
davidsbailey merged 3 commits intostagingfrom
decrease-eyes-parallelism

Conversation

@davidsbailey
Copy link
Copy Markdown
Member

@davidsbailey davidsbailey commented Apr 23, 2026

Sauce Labs is dropping our concurrency from 200 to 65 soon. As part of the work to move some of our UI test load to device farm, I noticed that our DTT currently maxes out our current Sauce Labs concurrency, using 120 + 79 = 199 at the beginning of the DTT:

  • Starting 738 dashboard UI tests in 120 threads...
  • Starting 79 dashboard Eyes tests in 172 threads... (this is 79x not 172x parallel)

We also use saucelabs during CI, but we use saucelabs priority option to give higher priority to DTT:

Saucelabs docs explain that pending jobs with lower priority number are always selected before jobs with higher priority number. This could be creating a situation where saucelabs is entirely unavailable for CI jobs for a period of minutes at the start of each DTT.

Therefore this PR accomplishes 2 goals:

  • start preparing us for the upcoming drop in saucelabs concurrency
  • see if we can fix some hard-to-diagnose issues in drone

Speculation

This might explain why we have UI tests failing in drone with the following mysterious errors:

This is highly speculative and may be hard to verify, but it seems worth a shot at potentially fixing this class of errors.

Testing story

I patched this change into the test machine, and found that it will not have any impact on DTT runtimes, because eyes tests still finish faster than regular UI tests.

before:

79 passed. 0 failed. Test count: 79. Duration: 9:39 minutes. Total successful reruns of flaky tests: 0.
738 passed. 0 failed. Test count: 738. Duration: 29:45 minutes. Total successful reruns of flaky tests: 5.

after:

78 passed. 1 failed. Test count: 79. Duration: 17:09 minutes. Total successful reruns of flaky tests: 0.
741 passed. 0 failed. Test count: 741. Duration: 30:17 minutes. Total successful reruns of flaky tests: 2.

Deployment notes

Deploy independently of any Device Farm changes to help reduce drone flakiness

@davidsbailey davidsbailey marked this pull request as ready for review April 23, 2026 18:05
@davidsbailey davidsbailey changed the title reduce eyes parallelism from 79 to 20 reduce eyes parallelism from 79 to 20 to deflake drone Apr 23, 2026
@davidsbailey davidsbailey merged commit ab643cd into staging Apr 23, 2026
7 of 8 checks passed
@davidsbailey davidsbailey deleted the decrease-eyes-parallelism branch April 23, 2026 22:51
@davidsbailey davidsbailey restored the decrease-eyes-parallelism branch April 23, 2026 22:51
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants