check_point.gaia.cp_gaia_user_facts error 500 General Exception

[chuegel]

We are using this collection to query the users on gaia gateways

- name: Gather current Gaia users
  check_point.gaia.cp_gaia_user_facts:
  register: current_users
  tags:
    - always

- name: List of current Gaia users
  ansible.builtin.debug:
    msg: |
      Current users on the system:
      {% for user in current_users.ansible_facts.objects %}
      - Username: {{ user.name }} - (Role(s): {{ user.roles | join(', ') }})
      {% endfor %}
  tags:
    - debug

fails with following error:

[fatal: [fwinternet1]: FAILED! => changed=false 
  msg: 'Checkpoint device returned error 500 with message {''code'': ''generic_error'', ''errors'': ''1'', ''msg'': ''General Exception''}'](msg: 'Checkpoint device returned error 500 with message {''code'': ''generic_error'', ''errors'': ''1'', ''msg'': ''General Exception''}')

Other tasks are working fine.

Is this related to: #60 ?

Versions used:

check_point.gaia              5.0.1

ansible [core 2.15.12]
python version = 3.9.19 (main, May 16 2024, 08:45:40) [GCC 8.5.0 20210514 (Red Hat 8.5.0-22)]
jinja version = 3.1.4
libyaml = True

OS: Gaia R81.20
HW: QLS250

[duanetoler]

Check your /var/log/gaia_api_server.log for internal details on this. The real error is in this log file. It's probably the same bug with Gaia API 1.7 and lower.

Likewise, make sure you have a recent Jumbo HFA which includes internal Gaia CONFD (Clish) fixes for running commands via Ansible. There was an issue dealing with the internal CONFD database lock between sessions via the Ansible Gaia modules. Similarly, there is an issue in older versions with inconsistency applying changes for both static routes and dynamic routing processes.

Your Gaia API module collection is also outdated. You should update that as well:

https://galaxy.ansible.com/ui/repo/published/check_point/gaia

[chuegel]

03/03/25 07:59:54: MainThread: infra.pipeline: INFO: Handling output
03/03/25 07:59:54: MainThread: infra.utils: INFO: Server IP initialized for the first time
03/03/25 07:59:54: MainThread: objects.sessions: INFO: Remote authentication succeed for user:ansible_user
03/03/25 07:59:54: MainThread: infra.utils: INFO: Server Port initialized for the first time
03/03/25 07:59:54: MainThread: server_util.udsListener: INFO: Request for endpoint /login [method: POST], for source 172.xxx.xxx.xxx, SUCCEEDED [duration 200ms]
03/03/25 07:59:54: MainThread: server_util.udsListener: INFO: remote_addr IP = 172.xxx.xxx.xxx
03/03/25 07:59:54: MainThread: infra.urlGeneratorIS: INFO: Handle request: <class 'requests.sessions.LogoutRequest'>
03/03/25 07:59:54: MainThread: infra.urlGeneratorIS: INFO: Execute validators
03/03/25 07:59:54: MainThread: infra.urlGeneratorIS: INFO: Verify permissions
03/03/25 07:59:54: MainThread: infra.pipeline: INFO: Execute commit function for class <class 'requests.sessions.LogoutRequest'>
03/03/25 07:59:54: MainThread: server_util.udsListener: INFO: Request for endpoint /logout [method: POST], for source 172.xxx.xxx.xxx, SUCCEEDED [duration 1ms]
03/03/25 08:01:26: MainThread: server_util.udsListener: INFO: remote_addr IP = 172.xxx.xxx.xxx
03/03/25 08:01:26: MainThread: infra.urlGeneratorIS: INFO: Handle request: <class 'requests.sessions.LoginRequest'>
03/03/25 08:01:26: MainThread: infra.urlGeneratorIS: INFO: Execute validators
03/03/25 08:01:26: MainThread: infra.urlGeneratorIS: INFO: Verify permissions
03/03/25 08:01:26: MainThread: infra.pipeline: INFO: Execute commit function for class <class 'requests.sessions.LoginRequest'>
03/03/25 08:01:27: MainThread: infra.pipeline: INFO: Handling output
03/03/25 08:01:27: MainThread: objects.sessions: INFO: Remote authentication succeed for user:ansible_user
03/03/25 08:01:27: MainThread: server_util.udsListener: INFO: Request for endpoint /login [method: POST], for source 172.xxx.xxx.xxx SUCCEEDED [duration 180ms]
03/03/25 08:01:27: MainThread: server_util.udsListener: INFO: remote_addr IP = 172.xxx.xxx.xxx
03/03/25 08:01:27: MainThread: infra.urlGeneratorIS: INFO: Handle request: <class 'requests.users.ShowUsers'>
03/03/25 08:01:27: MainThread: infra.urlGeneratorIS: INFO: Execute validators
03/03/25 08:01:27: MainThread: infra.urlGeneratorIS: INFO: Verify permissions
03/03/25 08:01:27: MainThread: objects.users: INFO: Collecting users data from the system
03/03/25 08:01:27: MainThread: objects.users: INFO: Processing users data from the system
03/03/25 08:01:32: MainThread: objects.users: INFO: Information collected and processed successfully
03/03/25 08:01:32: MainThread: objects.users: INFO: Collecting users data from the system
03/03/25 08:01:32: MainThread: objects.users: INFO: Processing users data from the system
03/03/25 08:01:37: MainThread: objects.users: INFO: Information collected and processed successfully
03/03/25 08:01:37: MainThread: objects.users: INFO: Collecting users data from the system
03/03/25 08:01:38: MainThread: objects.users: INFO: Processing users data from the system
03/03/25 08:01:42: MainThread: objects.users: INFO: Information collected and processed successfully
03/03/25 08:01:43: MainThread: objects.users: INFO: Collecting users data from the system
03/03/25 08:01:43: MainThread: objects.users: INFO: Processing users data from the system
03/03/25 08:01:48: MainThread: objects.users: INFO: Information collected and processed successfully
03/03/25 08:01:48: MainThread: objects.users: INFO: Collecting users data from the system
03/03/25 08:01:48: MainThread: objects.users: INFO: Processing users data from the system
03/03/25 08:01:53: MainThread: objects.users: INFO: Information collected and processed successfully
03/03/25 08:01:53: MainThread: objects.users: INFO: Collecting users data from the system
03/03/25 08:01:54: MainThread: objects.users: INFO: Processing users data from the system
03/03/25 08:01:58: MainThread: infra.pipeline: ERROR: Failed to handle request, reason: '1'
03/03/25 08:01:58: MainThread: infra.pipeline: ERROR: 1
Traceback (most recent call last):
  File "/rest_api/ckp/infra/urlGeneratorIS.py", line 67, in actor
    res = execute(inst, requestData)
  File "/rest_api/ckp/infra/pipeline.py", line 262, in execute
    ret = inst.Handler_RESPONSE().serialize()
  File "/rest_api/ckp/objects/users.py", line 560, in __init__
    self.populate()
  File "/rest_api/ckp/objects/users.py", line 577, in populate
    self.users.append(UserV18().populate(userDict.get(NAME)))
  File "/rest_api/ckp/objects/users.py", line 770, in populate
    User.populateFromUserFactory(self, name)
  File "/rest_api/ckp/objects/users.py", line 708, in populateFromUserFactory
    self.userFactory = UserFactory()
  File "/rest_api/ckp/objects/users.py", line 171, in __init__
    self.processData()
  File "/rest_api/ckp/objects/users.py", line 479, in processData
    XMLResult = clish.runClish("show user {} lock-out".format(user), lock, clish.outXml, False)
  File "/rest_api/libs/clish.py", line 177, in runClish
    returnCode, output, error = command_factory.execFactoryExecutor(clishCmd, ws.get_user_env())
  File "/rest_api/libs/command_factory.py", line 89, in execFactoryExecutor
    tup = p.communicate()
  File "/opt/CPsuite-R81.20/fw1/Python/lib/python3.7/subprocess.py", line 964, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/opt/CPsuite-R81.20/fw1/Python/lib/python3.7/subprocess.py", line 1715, in _communicate
    ready = selector.select(timeout)
  File "/opt/CPsuite-R81.20/fw1/Python/lib/python3.7/selectors.py", line 415, in select
    fd_event_list = self._selector.poll(timeout)
  File "/opt/CPsuite-R81.20/fw1/Python/lib/python3.7/site-packages/gunicorn/workers/base.py", line 192, in handle_abort
    sys.exit(1)
SystemExit: 1
03/03/25 08:01:58: MainThread: server_util.udsListener: INFO: Request for endpoint /show-users [method: POST], for source 172.xxx.xxx.xxx, FAILED [duration 30946ms]
03/03/25 08:01:58 [ INFO] MainThread:__init__(): **********************     Init Gaia API Logger - New Run     **********************
03/03/25 08:01:58: MainThread: infra.vsnext_utils: INFO: VSNext status: off
03/03/25 08:01:58: MainThread: infra.vsnext_utils: INFO: VSNext status: off
03/03/25 08:01:58: MainThread: infra.annotations: INFO: Setting End-Point for URL: /v1.4/run-reboot, methods: ['POST']
03/03/25 08:01:58: MainThread: infra.annotations: INFO: Setting End-Point for URL: /v1.5/run-reboot, methods: ['POST']
03/03/25 08:01:58: MainThread: infra.annotations: INFO: Setting End-Point for URL: /v1.6/run-reboot, methods: ['POST']
03/03/25 08:01:58: MainThread: infra.annotations: INFO: Setting End-Point for URL: /v1.7/run-reboot, methods: ['POST']
03/03/25 08:01:58: MainThread: infra.annotations: INFO: Setting End-Point for URL: /v1.8/run-reboot, methods: ['POST']
03/03/25 08:01:58: MainThread: infra.annotations: INFO: Setting none version End-Point for URL: /run-reboot
03/03/25 08:01:58: MainThread: infra.annotations: INFO: Setting Permissions for system
03/03/25 08:01:58: MainThread: infra.annotations: INFO: Setting End-Point for URL: /v1.1/set-radius, methods: ['POST']
03/03/25 08:01:58: MainThread: infra.annotations: INFO: Setting End-Point for URL: /v1.2/set-radius, methods: ['POST']
03/03/25 08:01:58: MainThread: infra.annotations: INFO: Setting End-Point for URL: /v1.3/set-radius, methods: ['POST']
03/03/25 08:01:58: MainThread: infra.annotations: INFO: Setting End-Point for URL: /v1.4/set-radius, methods: ['POST']
03/03/25 08:01:58: MainThread: infra.annotations: INFO: Setting End-Point for URL: /v1.5/set-radius, methods: ['POST']
03/03/25 08:01:58: MainThread: infra.annotations: INFO: Setting End-Point for URL: /v1.6/set-radius, methods: ['POST']
03/03/25 08:01:58: MainThread: infra.annotations: INFO: Setting End-Point for URL: /v1.7/set-radius, methods: ['POST']
03/03/25 08:01:58: MainThread: infra.annotations: INFO: Setting Permissions for aaa
03/03/25 08:01:58: MainThread: infra.annotations: INFO: Setting End-Point for URL: /v1.8/set-radius, methods: ['POST']
03/03/25 08:01:58: MainThread: infra.annotations: INFO: Setting none version End-Point for URL: /set-radius
03/03/25 08:01:58: MainThread: infra.annotations: INFO: Setting End-Point for URL: /v1/show-radius, methods: ['POST']
03/03/25 08:01:58: MainThread: infra.annotations: INFO: Setting End-Point for URL: /v1.1/show-radius, methods: ['POST']
03/03/25 08:01:58: MainThread: infra.annotations: INFO: Setting End-Point for URL: /v1.2/show-radius, methods: ['POST']
03/03/25 08:01:58: MainThread: infra.annotations: INFO: Setting End-Point for URL: /v1.3/show-radius, methods: ['POST']
03/03/25 08:01:58: MainThread: infra.annotations: INFO: Setting End-Point for URL: /v1.4/show-radius, methods: ['POST']
03/03/25 08:01:58: MainThread: infra.annotations: INFO: Setting End-Point for URL: /v1.5/show-radius, methods: ['POST']
03/03/25 08:01:58: MainThread: infra.annotations: INFO: Setting End-Point for URL: /v1.6/show-radius, methods: ['POST']
03/03/25 08:01:58: MainThread: infra.annotations: INFO: Setting End-Point for URL: /v1.7/show-radius, methods: ['POST']
03/03/25 08:01:58: MainThread: infra.annotations: INFO: Setting End-Point for URL: /v1.8/show-radius, methods: ['POST']
03/03/25 08:01:58: MainThread: infra.annotations: INFO: Setting none version End-Point for URL: /show-radius
03/03/25 08:01:58: MainThread: infra.annotations: INFO: Setting End-Point for URL: /v1/set-tacacs, methods: ['POST']
03/03/25 08:01:58: MainThread: infra.annotations: INFO: Setting End-Point for URL: /v1.1/set-tacacs, methods: ['POST']
03/03/25 08:01:58: MainThread: infra.annotations: INFO: Setting End-Point for URL: /v1.2/set-tacacs, methods: ['POST']
03/03/25 08:01:58: MainThread: infra.annotations: INFO: Setting End-Point for URL: /v1.3/set-tacacs, methods: ['POST']
03/03/25 08:01:58: MainThread: infra.annotations: INFO: Setting End-Point for URL: /v1.4/set-tacacs, methods: ['POST']
03/03/25 08:01:58: MainThread: infra.annotations: INFO: Setting End-Point for URL: /v1.5/set-tacacs, methods: ['POST']
03/03/25 08:01:58: MainThread: infra.annotations: INFO: Setting End-Point for URL: /v1.6/set-tacacs, methods: ['POST']
03/03/25 08:01:58: MainThread: infra.annotations: INFO: Setting End-Point for URL: /v1.7/set-tacacs, methods: ['POST']
03/03/25 08:01:58: MainThread: infra.annotations: INFO: Setting End-Point for URL: /v1.8/set-tacacs, methods: ['POST']
03/03/25 08:01:58: MainThread: infra.annotations: INFO: Setting none version End-Point for URL: /set-tacacs

--snip--
03/03/25 08:02:01: MainThread: server_util.udsListener: INFO: Generate Post reboot tasks
03/03/25 08:02:01: MainThread: postRebootHandler: INFO: Loading all tasks
03/03/25 08:02:01: MainThread: postRebootHandler: INFO: Clear tasks cache
03/03/25 08:02:01: MainThread: server_util.udsListener: INFO: Start server...
03/03/25 08:02:01: Thread-1: server_util.udsListener: INFO: Setting UDS for requests
03/03/25 08:02:01: Thread-1: server_util.udsListener: INFO: Start listening to UDS connections
03/03/25 08:02:01: MainThread: server_util.udsListener: INFO: remote_addr IP = 172.xxx.xxx.xxx
03/03/25 08:02:01: MainThread: sessions_manager: WARNING: Got unauthorized sid, remote ip:172.xxx.xxx.xxx.
03/03/25 08:02:01: MainThread: server_util.udsListener: INFO: Request for endpoint /logout [method: POST], for source 172.xxx.xxx.xxx, FAILED [duration 1ms]
03/03/25 08:02:01: MainThread: server_util.udsListener: INFO: remote_addr IP = 172.xxx.xxx.xxx
03/03/25 08:02:01: MainThread: infra.urlGeneratorIS: INFO: Handle request: <class 'requests.sessions.LoginRequest'>
03/03/25 08:02:01: MainThread: infra.urlGeneratorIS: INFO: Execute validators
03/03/25 08:02:01: MainThread: infra.urlGeneratorIS: INFO: Verify permissions
03/03/25 08:02:01: MainThread: infra.pipeline: INFO: Execute commit function for class <class 'requests.sessions.LoginRequest'>
03/03/25 08:02:02: MainThread: infra.pipeline: INFO: Handling output
03/03/25 08:02:02: MainThread: infra.utils: INFO: Server IP initialized for the first time
03/03/25 08:02:02: MainThread: objects.sessions: INFO: Remote authentication succeed for user:ansible_user
03/03/25 08:02:02: MainThread: infra.utils: INFO: Server Port initialized for the first time
03/03/25 08:02:02: MainThread: server_util.udsListener: INFO: Request for endpoint /login [method: POST], for source 172.xxx.xxx.xxx, SUCCEEDED [duration 198ms]
03/03/25 08:02:02: MainThread: server_util.udsListener: INFO: remote_addr IP = 172.xxx.xxx.xxx
03/03/25 08:02:02: MainThread: infra.urlGeneratorIS: INFO: Handle request: <class 'requests.sessions.LogoutRequest'>
03/03/25 08:02:02: MainThread: infra.urlGeneratorIS: INFO: Execute validators
03/03/25 08:02:02: MainThread: infra.urlGeneratorIS: INFO: Verify permissions
03/03/25 08:02:02: MainThread: infra.pipeline: INFO: Execute commit function for class <class 'requests.sessions.LogoutRequest'>
03/03/25 08:02:02: MainThread: server_util.udsListener: INFO: Request for endpoint /logout [method: POST], for source 172.xxx.xxx.xxx, SUCCEEDED [duration 1ms]
```

We will upgrade the gateways asap.
The Ansible Collections have also been upgraded but the error remains

[chuegel]

The error above is from a R81.10 Jumbo Hotfix Take 172 so it's the last recommended from 81.10

 gaia_api status

API Status:
---------------------
Build: cp991255275
Uptime: 0:37:48
Current Sessions: 0
Latest Version: 1.8

Processes:

Name           State        PID
---------------------------------
GAIA_API       Started      22860
GAIA_API_DOCS  Started      22848
APACHE         Started      7376
CONFD          Started      7373
CLISHD         Started      7463 5149
CELERY         Started      22847
REDIS          Started      7473

Port Details:
-------------------
APACHE Gaia Port:         443

--------------------------------------------
Overall API Status: Started
--------------------------------------------

 cat /rest_api/rest-api-server.conf
# This file holds the gunicorn configuration setting
import gunicorn
import os
#Note: For gunicorn versions upper than 20.1.0 you need to rename the variable to gunicorn.SERVER instead of gunicorn.SERVER_SOFTWARE
gunicorn.SERVER_SOFTWARE = 'CPWS'

pidfile = '/tmp/gaia_api.pid'
#errorlog = '/var/tmp/gaiaRestServer.log'
loglevel = 'info'
bind = '127.0.0.1:9092'
#threads = 2
group = 'config'
if os.path.isfile('/etc/.scalable_platform') or os.path.isfile('/etc/.scalable_platform_mho'):
        timeout = 300

Whats weird is:

03/03/25 08:01:58: MainThread: infra.pipeline: ERROR: Failed to handle request, reason: '1'
03/03/25 08:01:58: MainThread: infra.pipeline: ERROR: 1
Traceback (most recent call last):
  File "/rest_api/ckp/infra/urlGeneratorIS.py", line 67, in actor
    res = execute(inst, requestData)
  File "/rest_api/ckp/infra/pipeline.py", line 262, in execute
    ret = inst.Handler_RESPONSE().serialize()
  File "/rest_api/ckp/objects/users.py", line 560, in __init__
    self.populate()
  File "/rest_api/ckp/objects/users.py", line 577, in populate
    self.users.append(UserV18().populate(userDict.get(NAME)))
  File "/rest_api/ckp/objects/users.py", line 770, in populate
    User.populateFromUserFactory(self, name)
  File "/rest_api/ckp/objects/users.py", line 708, in populateFromUserFactory
    self.userFactory = UserFactory()
  File "/rest_api/ckp/objects/users.py", line 171, in __init__
    self.processData()
  File "/rest_api/ckp/objects/users.py", line 479, in processData
    XMLResult = clish.runClish("show user {} lock-out".format(user), lock, clish.outXml, False)
  File "/rest_api/libs/clish.py", line 177, in runClish
    returnCode, output, error = command_factory.execFactoryExecutor(clishCmd, ws.get_user_env())
  File "/rest_api/libs/command_factory.py", line 89, in execFactoryExecutor
    tup = p.communicate()
  File "/opt/CPsuite-R81.20/fw1/Python/lib/python3.7/subprocess.py", line 964, in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
  File "/opt/CPsuite-R81.20/fw1/Python/lib/python3.7/subprocess.py", line 1715, in _communicate
    ready = selector.select(timeout)
  File "/opt/CPsuite-R81.20/fw1/Python/lib/python3.7/selectors.py", line 415, in select
    fd_event_list = self._selector.poll(timeout)
  File "/opt/CPsuite-R81.20/fw1/Python/lib/python3.7/site-packages/gunicorn/workers/base.py", line 192, in handle_abort
    sys.exit(1)
SystemExit: 1
03/03/25 08:01:58: MainThread: server_util.udsListener: INFO: Request for endpoint /show-users [method: POST], for source 172.xxx.xxx.xxx, FAILED [duration 30946ms]
03/03/25 08:01:58 [ INFO] MainThread:__init__(): **********************     Init Gaia API Logger - New Run     **********************

[duanetoler]

Yeah that's interesting. It died on running the CLISH command "show user lock-out" for the list of users. This command is being ran so that Gaia API can return a bunch of useful information for each of the users since you did "cp_gaia_user_facts" to get a list of all the users. (you probably knew this already)

You can run these commands manually yourself to "follow the trail", and you might be able to find the missing piece. Login to the gateway, go to CLISH, then run "show users". For each of your users, run "show user lock-out".

I looked over the R81.10 JHF notes and didn't see anything interesting in Take 173 that might've already been fixed, either.

[chuegel]

As a workaround:

---
- name: Gather current Gaia users
  check_point.gaia.cp_gaia_user_facts:
    version: 1.7
  register: current_users
  tags:
    - always

specifying the lower version (1.7 in this case) worked. So there must be something fishy in the 1.8 version.

[duanetoler]

Indeed, or at least v1.8 on R81.10. You still might want to check /config/active and still run the CLISH commands manually to see if you can find it before you open a TAC case.

Another point to consider: I saw your Ansible playbook is using the username "ansible_user" (perfectly fine), but does this "ansible_user" have read-write and adminRole permissions? I wonder if there's something odd in the RBA configuration when this user runs these commands, versus "admin" (for example).

Either way, you have found some sort of issue, and it warrants additional review by TAC, since it works on API v1.7 and not v1.8.
"Congrats!" :)

[chuegel]

@duanetoler yes, that user has the adminRole (I just redacted the name)
Should I open a TAC?

[duanetoler]

Ok, good, just wanted to be sure; TAC will ask you about that. :) You're welcome to do a TAC case, as R81.10 still supported. However, be aware that R81.10 is going EoL in July 2025 so you'll be better served updating to R81.20 if you can.

[mib1185]

we're affected by the same issue and have already raised an TAC case. It is confirmed as an issue in the gaia api server, as there is a fixed timeout of 30s per api call, but the "show users" api call queries each user one-by-one via an clish command which is quiet slow (approx. 3-5s per user). We're on R81.20 with gaia-api-server 1.8