API key expiry time drift on laptops due to mismatching sleep methods

[MoralCode]

While testing, I noticed that the collectoss github api-keys CLI would show all configured API keys as fully reset, but collection would still be sleeping (as calculated based on the timestamps and sleep times printed in the logs.

Im still not 100% certain of the cause, but my best theory so far (found with the help of sonnet 4.6 via cursor) is this:

CollectOSS uses two different types of waiting as it relates to API keys. Keyman itself stores the reset time for the key as a unix timestamp so that, the next time a request comes in, all expired keys get re-evaluated and added back to the valid pool when those times have elapsed. This essentially uses real/wall clock time.

The keyman clients, on the other hand, will wait using time.sleep(), which relies on monotonic time (essentially a counter).

According to Sonnet, on linux devices, the monotonic clock does not advance when the machine (such as a laptop) is asleep.

This means a key could be 15 minutes away from getting reset, but if the laptop goes to sleep and wakes up 30 minutes later, the collection process that was waiting still has to wait for the final 15 minutes of that time.sleep() to finish.

The part of the problem I cant explain is that this resetting behavior seems like it might persist from iteration to iteration, meaning a laptop going to sleep could mean affected collection tasks are now permanently "behind" the actual key reset time by some amount of time thats based on how long/when the laptop sleep happened and when in the key reset cycle it was.

The impact of this is low, but I think it may still be important to fix eventually, especially for the development experience or people running instances on laptops.

[MoralCode]

My personal proposed solution is to have a small custom sleep function that closely mimics the behavior of an acutal wall clock sleep (which i dont think exists in the python standard library). Something like this:

def sleep_until(epoch: float, check_interval: int = 60) -> None:
    while True:
        remaining = epoch - time.time()
        if remaining <= 0:
            return
        time.sleep(min(remaining, check_interval))

This would sleep as long as needed, but with an occasional wakeup to check the current time and decide whether to continue sleeping.

[MoralCode]

This might also help us improve on our ability to get collection going again if a new API key is added to keyman mid-collection (because all the collection services will be polling every 60 sec and so some of them will pick up on the new key and will be able to continue)

[MoralCode]

I wonder if this could also be caused by secondary rate limits, which dont show up in the CLI key checker but may be enforced by github/trigger keyman

https://docs.github.com/en/rest/using-the-rest-api/rate-limits-for-the-rest-api?apiVersion=2026-03-10#about-secondary-rate-limits

[Dipro-cyber]

@MoralCode sir taking your concerns in account, I would like to take up this issue.

[MoralCode]

I'm not fully confident in my root cause analysis, especially regarding how the key timing offset persists across multiple key resets.

Unifying the timing might help but I think more testing/problem analysis is needed to confirm the source of the issue first

My attempt at repro steps from memory when I witnessed this general odd behavior:

1. Start a collectoss instance
2. Load a few large repos (rook/rook, ansible/ansible, kubernetes/kubermetes). Repos should be large enough to collectively not be able to be collected within 3-4 API key reset cycles
3. Observe the collection activity drop substantially (via logs or flower), confirm that this is due to key expiry via logs
4. Record number of seconds until key is reset
5. Record key reset time from CLI: collectoss github api-keys
6. Put laptop to sleep from somewhere between 1-50 minutes, waking laptop before(? May also need to test after) the keys reset
7. Check CLI again to confirm whether actual key reset time changed
8. Wait and Observe collection activity resume after duration found in step 4
9. Repeat step 7 to check how long keys have to expire.
10. Wait for collection activity to stop due to key expiry
11. Record number of seconds until reset and expiry time from CLI
12. Wait and verify collection resumes at key reset time (without sleeping the laptop in between).