st2-pkg instances are not terminated on AWS

[arm4b]

Opened it here instead of https://github.com/StackStorm/st2cicd/issues/56

Still seeing this behavior from time to time. VMs are not destroyed all the time, CNAME is not removed.

Probably bug somewhere in the st2ci.st2_pkg_e2e_test / st2cd.create_vm_role workflows.
Good to keep an eye on it.

[arm4b]

Experienced this again.
Not sure if related or just coincidence, but this time AWS node didn't clean up when Slack Post message workflow task failed.

[arm4b]

Confirming the cause. So when Slack post message task failed (due to random Slack 50x response) in the workflow, destroy_vm is not triggered as a fallback:

[warrenvw]

Here's more data from occurrence today:

warrenvw@st2cicd033:~$ st2 execution get 5b8ed6370a8b8a078446c116
id: 5b8ed6370a8b8a078446c116
action.ref: st2cd.create_vm_role
parameters: 
  distro: pkg_RHEL6
  dns_zone: uswest2.stackstorm.net
  environment: sandbox
  hostname: st2-pkg-unstable-el6
  instance_type: c4.large
  key_name: st2_deploy
  keyfile: /home/stanley/.ssh/stanley_rsa
  role: cislave
status: failed (10s elapsed)
result_task: get_ami
result: 
  exit_code: 0
  result: ami-ea096e8a
  stderr: ''
  stdout: ''
start_timestamp: Tue, 04 Sep 2018 19:00:07 UTC
end_timestamp: Tue, 04 Sep 2018 19:00:17 UTC
+--------------------------+------------------------+---------------+------------------------+--------------------------+
| id                       | status                 | task          | action                 | start_timestamp          |
+--------------------------+------------------------+---------------+------------------------+--------------------------+
| 5b8ed6390a8b8a078446c11e | succeeded (2s elapsed) | get_subnet_id | st2.kv.get             | Tue, 04 Sep 2018         |
|                          |                        |               |                        | 19:00:09 UTC             |
| 5b8ed63b0a8b8a078446c12a | succeeded (2s elapsed) | get_ami       | st2.kv.get             | Tue, 04 Sep 2018         |
|                          |                        |               |                        | 19:00:11 UTC             |
| 5b8ed63e0a8b8a078446c132 | succeeded (2s elapsed) | check_cname   | aws.r53_zone_get_cname | Tue, 04 Sep 2018         |
|                          |                        |               |                        | 19:00:14 UTC             |
+--------------------------+------------------------+---------------+------------------------+--------------------------+
warrenvw@st2cicd033:~$ st2 execution get 5b8ed63e0a8b8a078446c132
id: 5b8ed63e0a8b8a078446c132
status: succeeded (2s elapsed)
parameters: 
  name: st2-pkg-unstable-el6.uswest2.stackstorm.net
  zone: uswest2.stackstorm.net
result: 
  exit_code: 0
  result:
  - alias_dns_name: null
    alias_evaluate_target_health: null
    alias_hosted_zone_id: null
    failover: null
    health_check: null
    identifier: null
    name: st2-pkg-unstable-el6.uswest2.stackstorm.net.
    region: null
    resource_records:
    - ip-10-0-3-67.us-west-2.compute.internal.
    ttl: '60'
    type: CNAME
    weight: null
  stderr: ''
  stdout: ''