Affinity group creation sometimes fails due to race condition

[mrog]

ISSUE TYPE

• Bug Report

COMPONENT NAME

API

CLOUDSTACK VERSION

4.14.1.0

CONFIGURATION

N/A

OS / ENVIRONMENT

CentOS 7, but this is likely N/A.

SUMMARY

When using the API to create affinity groups, it usually works. On rare instances, a race condition causes a NullPointerException.

STEPS TO REPRODUCE

Use the API to create an affinity group.  This may take many tries to reproduce the issue.

EXPECTED RESULTS

The affinity group should be created.

ACTUAL RESULTS

There is sometimes a NullPointerException.

See the attached log. It shows that an async job is created and runs. The output from the async job includes the message Update db status for job-131043, which indicates that the transaction has started which will set the removed date for the async job. Then when ApiServer.getBaseAsyncCreateResponse() looks for the job on line 662, it gets a null value because the record's removed value isn't null. This causes the NPE to happen on line 663.

NullPointerException.log

[shwstppr]

@mrog how to reproduce this? I tested with 4.17 branch env, trying to create 50 groups simultaneously and it seem to work okay with following script using cmk,

#!/bin/bash

create_ag_func(){
  echo "Create AG $1 times..."
  cmk create affinitygroup name="abcd${1}" type='host affinity' &
}

for i in {1..50}
do
        create_ag_func $i &
done
wait
echo "All done"

[mrog]

@shwstppr It might be hard to reproduce. Creating affinity groups usually works, but one of our developers saw this issue 1 out of 4 tries yesterday.

We have the CloudStack management service and MySQL running on the same machine. It has 2 CPU cores, and CPU utilization was around 60-65% at the time of the error. It makes me wonder if the error will only happen if the machine is fairly busy.

[mrog]

If ApiServer.getBaseAsyncCreateResponse() was able to find jobs that have been removed, that would fix this issue. But I don't know if it would cause other problems. Is there a reason why it shouldn't be allowed to find removed jobs?

[shwstppr]

@mrog that change is already there in 4.17.0
293517ad728
You will have to upgrade or create your own packages for 4.14

[shwstppr]

@mrog I'm closing this for now as the issue has been fixed in 4.17.0.
Please re-open if it is reproducible there.

[mrog]

Thank you!