Skip to content

fix(cobuilds): add socket + connect timeout#5797

Open
aramissennyeydd wants to merge 4 commits intomicrosoft:mainfrom
aramissennyeydd:sennyeya/fix-redis-reconnect
Open

fix(cobuilds): add socket + connect timeout#5797
aramissennyeydd wants to merge 4 commits intomicrosoft:mainfrom
aramissennyeydd:sennyeya/fix-redis-reconnect

Conversation

@aramissennyeydd
Copy link
Copy Markdown
Contributor

Summary

With #5325, we're seeing more recoveries of redis issues but also more hanging agents that are waiting for the redis reconnect to happen. Adding in a socket timeout to prevent that hanging behavior.

Details

We'd have logs like

==[ @doordash/project (build) ]===========================[ 287 of 296 ]==                                                                       
  Redis client error: Socket closed unexpectedly                                                                                                        
  Redis client reconnecting attempt #0                                                                                                                  
  Redis client error: Socket closed unexpectedly                                                                                                        
  Redis client reconnecting attempt #0                                                                                                                  
  Error getting cache entry: Error: read ECONNRESET                                                                                                     
  "@doordash/project (build)" completed successfully in 41 minutes 7.7 seconds..

when there were network issues. The build time is 38 minutes longer than it usually takes. As you can see in the above, we sometimes reconnect but generally, the build agent hangs indefinitely.

How it was tested

Impacted documentation

n/a

Signed-off-by: Aramis Sennyey <aramissennyeydd@users.noreply.github.com>
Signed-off-by: Aramis Sennyey <aramissennyeydd@users.noreply.github.com>
Signed-off-by: Aramis Sennyey <aramissennyeydd@users.noreply.github.com>
Signed-off-by: Aramis Sennyey <aramissennyeydd@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

Status: Needs triage

Development

Successfully merging this pull request may close these issues.

1 participant