Users are not added to new group until 2nd convergence of recipe

[poliva83]

chef_group will create new group in your organization but the users do not appear to be added to group until 2nd chef-client run.

recipe snippet

url_with_org = {
    :chef_server_url => URI.join(chef_server_url, "/organizations/blackberry").to_s,
    :options => {
      :client_name => "pivotal",
      :signing_key_filename => "/etc/opscode/pivotal.pem" } }

chef_group 'ops' do
      users  ["ops-blackberry", "anbrown"]
      complete true
      chef_server url_with_org
      action :create
end

1st chef-client run

* chef_group[ops] action create
 - create group ops at https://127.0.0.1/organizations/blackberry
 -   update users from [] to ["ops-blackberry", "anbrown"]

knife group show ops -c ~/.chef/knife-chef12.rb  (via knife-acl)

actors:
annotated_groups:
clients:
groupname:        ops
groups:
name:             ops
orgname:          blackberry
users:

2nd chef-client run

* chef_group[ops] action create
 - update group ops at https://127.0.0.1/organizations/blackberry
 -   update users from [] to ["ops-blackberry", "anbrown"]
 -   remove orgname

knife group show ops -c ~/.chef/knife-chef12.rb (via knife-acl)

actors:
  anbrown
  ops-blackberry
annotated_groups:
clients:
groupname:        ops
groups:
name:             ops
orgname:          blackberry
users:
  anbrown
  ops-blackberry

[poliva83]

@jkeiser @tyler-ball We discussed this issue today I'd rate this issue higher on priority as needing to run chef-client twice when adding new group is less then ideal.

[nlam79]

Phil had mentioned that this issue that i'm seeing with kitchen could be related.
When converging the backend recipe in test-kitchen the following error was observed. Reconverging with no changes resulted in a pass:

Manual run:

• sudo chef-server-ctl user-list
• ERROR: Server returned error 500 for https://127.0.0.1/users, retrying 1/5 in 3s
• Pivotal

Test-Kitchen run:

* chef_user[nlam] action create

           ================================================================================
           Error executing action `create` on resource 'chef_user[nlam]'
           ================================================================================

           Net::HTTPFatalError
           -------------------
           500 "Internal Server Error"

           Resource Declaration:
           ---------------------
           # In /tmp/kitchen/cache/cookbooks/bb_chef_server_cluster/recipes/manage_rbac.rb

            43:   chef_user user do
            44:     display_name "#{values['display_name']}"
            45:     email "#{values['email']}"
            46:     password "#{values['password']}"
            47:     source_key "#{values['source_key']}"
            48:     chef_server url_root
            49:     action :nothing
            50:     not_if { values['source_key'].nil? }
            51:     subscribes :create, 'ruby_block[only modify rbac content on master backend]', :immediately
            52:   end
            53: end

           Compiled Resource:
           ------------------
           # Declared in /tmp/kitchen/cache/cookbooks/bb_chef_server_cluster/recipes/manage_rbac.rb:43:in `block in from_file'

           chef_user("nlam") do
             action [:nothing]
             retries 0
             retry_delay 2
             default_guard_interpreter :default
             chef_server {:chef_server_url=>"https://127.0.0.1:443", :options=>{:client_name=>"pivotal", :signing_key_filename=>"/etc/opscode/pivotal.pem", :api_version=>"0"}}
             declared_type :chef_user
             cookbook_name "bb_chef_server_cluster"
             recipe_name "manage_rbac"
             display_name "Nhid Lam"
             email "nlam@blackberry.com"
             password "passw0rd"
             source_key "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDU+yPffx9TRlctRhja2hBi3780L+xhBlxxi6apkKN6UdlXgBiFNPVIXhkAgOUWRMpbigNOlF+tQ2se9ItG1uMkJkmArp4TvgCckiXMTro8ZD7cU4yvGwLuziBINdaCFXi9Xvjm8HYc3jXQh2oPAEbmheGd+KIVBMaC/9cWAtsddUpelVaaBHnLmm472sxcvfZFdwRlV5MorJeGvUfC0XUbg+sp1Qufdvpd6TBD4qM2TUDl0loUrjR/5AzUvzNulr+R2+EKFPetj8U/DS9v/5m0lxyRCBZORpGYhuVO5SY2ounSKORwvAtzdNE9/5LWpOEz3YD3/Deb0RbOhMMsmhVV nlam@nlam-chef-provisioning-ws"
             not_if { #code block }
           end


       Running handlers:
       [2015-11-02T20:01:37+00:00] ERROR: Running exception handlers
       Running handlers complete
       [2015-11-02T20:01:37+00:00] ERROR: Exception handlers complete
       Chef Client failed. 70 resources updated in 331.736216745 seconds
       [2015-11-02T20:01:37+00:00] FATAL: Stacktrace dumped to /tmp/kitchen/cache/chef-stacktrace.out
       [2015-11-02T20:01:37+00:00] ERROR: chef_user[nlam] (bb_chef_server_cluster::manage_rbac line 43) had an error: Net::HTTPFatalError: 500 "Internal Server Error"
       [2015-11-02T20:01:38+00:00] FATAL: Chef::Exceptions::ChildConvergeError: Chef run process exited unsuccessfully (exit code 1)
>>>>>> Converge failed on instance <backend-ubuntu-1404>.
>>>>>> Please see .kitchen/logs/backend-ubuntu-1404.log for more details
>>>>>> ------Exception-------
>>>>>> Class: Kitchen::ActionFailed
>>>>>> Message: SSH exited (1) for command: [sh -c '

sudo -E /opt/chef/bin/chef-client --local-mode --config /tmp/kitchen/client.rb --log_level auto --force-formatter --no-color --json-attributes /tmp/kitchen/dna.json --chef-zero-port 8889
']
>>>>>> ----------------------

cat /tmp/kitchen/cache/chef-stacktrace.out
Generated at 2015-11-02 20:01:37 +0000
Net::HTTPFatalError: chef_user[nlam] (bb_chef_server_cluster::manage_rbac line 43) had an error: Net::HTTPFatalError: 500 "Internal Server Error"
/opt/chef/embedded/lib/ruby/2.1.0/net/http/response.rb:119:in `error!'
/opt/chef/embedded/apps/chef/lib/chef/http.rb:145:in `request'
/opt/chef/embedded/apps/chef/lib/chef/http.rb:126:in `post'
/opt/chef/embedded/lib/ruby/gems/2.1.0/gems/cheffish-1.5.0/lib/cheffish/actor_provider_base.rb:30:in `block in create_actor'
/opt/chef/embedded/apps/chef/lib/chef/mixin/why_run.rb:52:in `call'
/opt/chef/embedded/apps/chef/lib/chef/mixin/why_run.rb:52:in `add_action'
/opt/chef/embedded/apps/chef/lib/chef/provider.rb:175:in `converge_by'
/opt/chef/embedded/lib/ruby/gems/2.1.0/gems/cheffish-1.5.0/lib/cheffish/actor_provider_base.rb:29:in `create_actor'
/opt/chef/embedded/lib/ruby/gems/2.1.0/gems/cheffish-1.5.0/lib/chef/provider/chef_user.rb:15:in `block in <class:ChefUser>'
/opt/chef/embedded/apps/chef/lib/chef/provider/lwrp_base.rb:160:in `instance_eval'
/opt/chef/embedded/apps/chef/lib/chef/provider/lwrp_base.rb:160:in `block in action'
/opt/chef/embedded/apps/chef/lib/chef/provider.rb:140:in `run_action'
/opt/chef/embedded/apps/chef/lib/chef/resource.rb:584:in `run_action'
/opt/chef/embedded/apps/chef/lib/chef/runner.rb:49:in `run_action'
/opt/chef/embedded/apps/chef/lib/chef/runner.rb:57:in `block in run_action'
/opt/chef/embedded/apps/chef/lib/chef/runner.rb:55:in `each'
/opt/chef/embedded/apps/chef/lib/chef/runner.rb:55:in `run_action'
/opt/chef/embedded/apps/chef/lib/chef/runner.rb:81:in `block (2 levels) in converge'
/opt/chef/embedded/apps/chef/lib/chef/runner.rb:81:in `each'
/opt/chef/embedded/apps/chef/lib/chef/runner.rb:81:in `block in converge'
/opt/chef/embedded/apps/chef/lib/chef/resource_collection/resource_list.rb:83:in `block in execute_each_resource'
/opt/chef/embedded/apps/chef/lib/chef/resource_collection/stepable_iterator.rb:116:in `call'
/opt/chef/embedded/apps/chef/lib/chef/resource_collection/stepable_iterator.rb:116:in `call_iterator_block'
/opt/chef/embedded/apps/chef/lib/chef/resource_collection/stepable_iterator.rb:85:in `step'
/opt/chef/embedded/apps/chef/lib/chef/resource_collection/stepable_iterator.rb:104:in `iterate'
/opt/chef/embedded/apps/chef/lib/chef/resource_collection/stepable_iterator.rb:55:in `each_with_index'
/opt/chef/embedded/apps/chef/lib/chef/resource_collection/resource_list.rb:81:in `execute_each_resource'
/opt/chef/embedded/apps/chef/lib/chef/runner.rb:80:in `converge'
/opt/chef/embedded/apps/chef/lib/chef/client.rb:654:in `block in converge'
/opt/chef/embedded/apps/chef/lib/chef/client.rb:649:in `catch'
/opt/chef/embedded/apps/chef/lib/chef/client.rb:649:in `converge'
/opt/chef/embedded/apps/chef/lib/chef/client.rb:688:in `converge_and_save'
/opt/chef/embedded/apps/chef/lib/chef/client.rb:269:in `run'
/opt/chef/embedded/apps/chef/lib/chef/application.rb:270:in `block in fork_chef_client'
/opt/chef/embedded/apps/chef/lib/chef/application.rb:258:in `fork'
/opt/chef/embedded/apps/chef/lib/chef/application.rb:258:in `fork_chef_client'
/opt/chef/embedded/apps/chef/lib/chef/application.rb:224:in `block in run_chef_client'
/opt/chef/embedded/apps/chef/lib/chef/local_mode.rb:39:in `with_server_connectivity'
/opt/chef/embedded/apps/chef/lib/chef/application.rb:212:in `run_chef_client'
/opt/chef/embedded/apps/chef/lib/chef/application/client.rb:402:in `block in interval_run_chef_client'
/opt/chef/embedded/apps/chef/lib/chef/application/client.rb:392:in `loop'
/opt/chef/embedded/apps/chef/lib/chef/application/client.rb:392:in `interval_run_chef_client'
/opt/chef/embedded/apps/chef/lib/chef/application/client.rb:382:in `run_application'
/opt/chef/embedded/apps/chef/lib/chef/application.rb:60:in `run'
/opt/chef/embedded/apps/chef/bin/chef-client:26:in `<top (required)>'
/opt/chef/bin/chef-client:55:in `load'

[jkeiser]

@poliva83 what Chef server are you using? I'm seeing things that make me wonder if perhaps the issue is server-side.

[poliva83]

@jkeiser We are seeing this on 12.2.0-1 from stable channel.

[poliva83]

@jkeiser Not sure this matters but worth mentioning its using a HA topology setup (BYOHA model).

[jkeiser]

I see the above bug (chef/chef-server#605) on Hosted; not 100% sure if it's the same bug, because it would imply that both create and update were busted. Will investigate more.

[poliva83]

@jkeiser Did your investigation bear fruit? Any news on this issue?

[pburkholder]

@poliva83 We're following up on this in chef/chef-server#605

[jkeiser]

I can work around it in cheffish by having it create and then immediately update the group.