Skip to content

Parallelized prlist function#11

Draft
zqumei0 wants to merge 1 commit intoawslabs:masterfrom
zqumei0:feature/parallel-requests
Draft

Parallelized prlist function#11
zqumei0 wants to merge 1 commit intoawslabs:masterfrom
zqumei0:feature/parallel-requests

Conversation

@zqumei0
Copy link
Contributor

@zqumei0 zqumei0 commented Jan 9, 2022

Issue #, if available:
N/A

Description of changes:
Parallelized for loop in prlist() responsible for making awscli calls and formatting pr lines decreasing runtime of prlist operation.

Runtime measurement below were measure as follows:

START=`date +%s`
# Switch case statement
END=`date +%s`
runtime=$(echo "$END - $START" | bc -l)
echo "Runtime: $runtime seconds"

Runtime test output for sequential prlist():

PR #       Title                                                        Branch                                                                                                                              
12         PR Branch 5                                                  bug/broken-print <- test_branch5 (qumei-Isengard)                                                                                   
11         PR 4                                                         test_branch3 <- test_branch4 (qumei-Isengard)                                                                                       
10         PR 3                                                         feature/cpp-hello <- test_branch3 (qumei-Isengard)                                                                                  
9          Test PR 1                                                    main <- test-branch1 (qumei-Isengard)                                                                                               
8          Feature test 2                                               main <- test_branch2 (qumei-Isengard)                                                                                               
6          Feature Request - Cpp Hello                                  main <- feature/cpp-hello (qumei-Isengard)
5          Bug Fix - Broken Print                                       main <- bug/broken-print (qumei-Isengard)
4          Feature Request - Python Hello                               main <- feature/python-hello (qumei-Isengard)
Runtime: 13 seconds

Runtime test output for parallel prlist():

PR #       Title                                                        Branch                                                                                               
12         PR Branch 5                                                  bug/broken-print <- test_branch5 (qumei-Isengard)
9          Test PR 1                                                    main <- test-branch1 (qumei-Isengard)
11         PR 4                                                         test_branch3 <- test_branch4 (qumei-Isengard)
8          Feature test 2                                               main <- test_branch2 (qumei-Isengard)
4          Feature Request - Python Hello                               main <- feature/python-hello (qumei-Isengard)
6          Feature Request - Cpp Hello                                  main <- feature/cpp-hello (qumei-Isengard)
5          Bug Fix - Broken Print                                       main <- bug/broken-print (qumei-Isengard)
10         PR 3                                                         feature/cpp-hello <- test_branch3 (qumei-Isengard)
Runtime: 5 seconds

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

Copy link
Contributor

@rkeene rkeene left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @zqumei0 this is much faster !

One thing that I've noticed is that this can make too many queries per second to the AWS CodeCommit endpoint if you have a lot of open PRs.

What might be useful here is to limit the number of inflight API calls.

This can be done by collecting the background process IDs ("$!" after ) &) and once the list of those grows beyond a certain number, wait, clear the list, and continue.

@zqumei0 zqumei0 marked this pull request as draft February 4, 2022 22:59
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants