diff --git a/doc/deployment/Creating-a-release.md b/doc/deployment/Creating-a-release.md index 2bb23f2d8..33bc15cde 100644 --- a/doc/deployment/Creating-a-release.md +++ b/doc/deployment/Creating-a-release.md @@ -60,7 +60,7 @@ is recent enough to include the most recent changes in the 1. Do the same for the `EPICS32` 32-bit build, replacing `EPICS` with `EPICS32` everywhere in the commands above. 1. Check release is now listed in [`https://control-svcs.isis.cclrc.ac.uk/git/?a=project_list;pf=releases`](https://control-svcs.isis.cclrc.ac.uk/git/?a=project_list;pf=releases) -1. Run the [`instrument_deploy.yaml` ansible playbook](https://github.com/ISISComputingGroup/ansible-playbooks/tree/main?tab=readme-ov-file#instrument_deployyaml) on any test machines which will be used for manual system testing. +1. Run the {ref}`instrumentdeployyaml` playbook on any test machines which will be used for manual system testing. {#creating_release_testing_steps} diff --git a/doc/deployment/Deploy.md b/doc/deployment/Deploy.md index 0fe2bda1d..dbf559200 100644 --- a/doc/deployment/Deploy.md +++ b/doc/deployment/Deploy.md @@ -1,9 +1,12 @@ # Deployment +We use a combination of {ref}`ansible` followed by [`ibex_utils`](https://github.com/ISISComputingGroup/IBEX_utils) for deployment. + ```{toctree} :glob: :maxdepth: 1 +deploy/ansible deploy/Updating-Instrument-Machines deploy/Deployment-on-an-Instrument-Control-PC deploy/Configure-Mini-Inst @@ -16,4 +19,4 @@ deploy/Reviewing-Deploy-Tickets deploy/Upgrade-ISISICP deploy/Upgrade-Java deploy/Installing-and-Upgrading-MySQL -``` \ No newline at end of file +``` diff --git a/doc/deployment/deploy/Deployment-on-an-Instrument-Control-PC.md b/doc/deployment/deploy/Deployment-on-an-Instrument-Control-PC.md index 225aaf2ee..bd6f36909 100644 --- a/doc/deployment/deploy/Deployment-on-an-Instrument-Control-PC.md +++ b/doc/deployment/deploy/Deployment-on-an-Instrument-Control-PC.md @@ -4,7 +4,7 @@ This document describes the steps necessary to install/upgrade IBEX on an Instru ## Preparatory Steps for Client and Server -Before the group starts any manual deployments, one developer should run the [`instrument_deploy.yaml` ansible playbook](https://github.com/ISISComputingGroup/ansible-playbooks/tree/main?tab=readme-ov-file#instrument_deployyaml) which currently installs the JDK on instruments. This should be run on the relevant deploy group - the playbook will prompt for the hosts to run it on to avoid accidentally deploying to all instruments. As an example, to deploy to the winter group, enter `winter_deploy` (as specified in the [inventory file](https://github.com/ISISComputingGroup/ansible-playbooks/blob/main/hosts.yaml)). +Before the group starts any manual deployments, one developer should run the {ref}`instrumentdeployyaml` playbook which currently installs the JDK on instruments. This should be run on the relevant deploy group - the playbook will prompt for the hosts to run it on to avoid accidentally deploying to all instruments. As an example, to deploy to the winter group, enter `winter_deploy` (as specified in the [inventory file](https://github.com/ISISComputingGroup/ansible-playbooks/blob/main/hosts.yaml)). This playbook has been designed so it is idempotent, ie. if a step has already occurred such as deploying the JDK it will not be repeated. - Inform the instrument scientist that you are going to upgrade the instrument in 5 minutes so that they are not surprised when you remote desktop to the instrument, include a link to the release notes of the latest release in this email. Wait 5 minutes. diff --git a/doc/deployment/deploy/ansible.md b/doc/deployment/deploy/ansible.md new file mode 100644 index 000000000..357e1fbf9 --- /dev/null +++ b/doc/deployment/deploy/ansible.md @@ -0,0 +1,213 @@ +{#ansible} +# Ansible + +We are moving towards using [Ansible](https://github.com/ISISComputingGroup/IBEX_utils) for both initial configuration of machines as well as deployment. + +The main repository for this is [here](https://github.com/ISISComputingGroup/ansible-playbooks) which contains our configuration, inventories, playbooks, roles and tasks. + +[Ansible playbooks](https://docs.ansible.com/projects/ansible/latest/playbook_guide/playbooks_intro.html) used for remotely automating tasks on NDX machines and other machines we look after. + +These run on your own computer (the control node) and orchestrate steps on remote machines. + +Playbooks should be idempotent, ie. they can be run multiple times and will always result in the same output - this is marked in Ansible as `OK`/`Changed` if a task is skipped or changed respectively. For example, checking that a dependency is a certain version and skipping a set of tasks to update it (which may remove the previous version) and so on. This means that the "deploy" playbook can be run absolutely everywhere, and machines that are already up to date will get most tasks skipped completely. Most Ansible modules are built to be idempotent by default. + +We are working towards these playbooks being the declarative steps to provision an instrument machine so it can easily be (re)created. As a developer you should keep this in mind - Ansible playbooks _can_ be used for one-shot roll outs of a specific set of tasks, but we should make them repeatable. This repo should not become a 'dumping ground' of playbooks we've written to roll out a certain change - they should be applied to the [blueprints of an instrument machine](https://github.com/ISISComputingGroup/ansible-playbooks/blob/main/windows/instrument_deploy.yaml) so we can use them again. + +## Setup +Preliminary steps to run these: +1. If you haven't already, set up a keeper account with access to our group's passwords. +1. Make sure your ssh public keys (which should be stored [here](https://github.com/ISISComputingGroup/keys)) are deployed to instruments - see below for the playbook that does this +1. Set up the WSL if you're using a Windows control node - [Ansible does not support running on a windows control node natively](https://docs.ansible.com/projects/ansible/latest/installation_guide/intro_installation.html#control-node-requirements). You will need your [SSH keys registered in the WSL](https://devblogs.microsoft.com/commandline/sharing-ssh-keys-between-windows-and-wsl-2/). If you are running Linux you can install Ansible natively. +1. Install Ansible, including plugins we require, by: + 1. `sudo snap install astral-uv --classic` + 1. `uv venv` & `source .venv/bin/activate` + 1. `uv pip install -r requirements.txt` +1. Install the galaxy collections and roles by running `ansible-galaxy install -r requirements.yml` +1. Add the DNS search suffix (`isis.cclrc.ac.uk`) to `/etc/resolv.conf` (to edit the file, use e.g. `nano /etc/resolv.conf`) by adding the following line: + +``` +... +search isis.cclrc.ac.uk +... +``` + +To test if this works, run an {ref}`ansibleadhoccommand` and use the module `ping` (for linux machines) or `win_ping` (for windows machines ie. NDXes) to test for aliveness. + +PRs will be linted by CI. To run this locally run `ansible-lint` - this is included in `requirements.txt`. Configuration is set by `.ansible-lint`. + +## `Hosts.yaml` +This is the main [inventory file](https://docs.ansible.com/projects/ansible/latest/inventory_guide/intro_inventory.html) which lists hosts in groups such as by target station and/or by deploy group. It is contained in the inventory directory. + +to limit running a playbook to just TS1 instruments, for example, you can use `ansible-playbook playbook.yaml --limit ts1`. This works because `ts1` is a host group in the inventory file. + + +## Windows-only playbooks (`windows/`) + +### Notes + +These playbooks will prompt for a host group to run on, equivalent to (and taking precedence over) `--limit` as a command line argument. + +As an example, to run the playbook on all NDXes other than `NDXENGINX`, enter `ndxes,!NDXENGINX` - this syntax is documented [here](https://docs.ansible.com/projects/ansible/latest/inventory_guide/intro_patterns.html#common-patterns) + +### `truncate_databases.yaml` + +This performs a backup and truncation of the local databases on instruments. It will prompt for hosts so to run use: + +`ansible-playbook windows/truncate_databases.yaml` + +{#instrumentdeployyaml} +### `instrument_deploy.yaml` + +This is the main playbook for deploying software to NDXes. Currently this stops the server if it is running and installs the JDK using the `jdk` role. + +To use this you need to run `ansible-playbook windows/instrument_deploy.yaml` - it should prompt for hosts. + +{#ansibleupdatingjdk} +#### Updating JDK version + +To update the JDK version, you will need to set the vars in `roles\defaults\main.yaml`. +These are: + +- `jdk_major_ver` - the major version (excluding .min.patch) of the JDK. +- `jdk_full_ver` - the full version string of the JDK. +- `jdk_url` - the URL to download the JDK from. +- `jdk_checksum` - the checksum for the downloaded JDK. +- (optionally) `jdk_force_update` - whether to overwrite the current version if it already exists + + +### `deploy_ssh_keys.yaml` + +This is the windows equivalent of the Linux workflow for deploying [the group's ssh keys](https://github.com/ISISComputingGroup/keys) and turning off password auth in favour of public key auth. + +To run this on all NDXes run `ansible-playbook windows/deploy_ssh_keys.yaml`. To limit to certain hosts/host groups use `--limit` ie. +`--limit NDXHRPD_SETUP` to just run on `NDXHRPD_SETUP` or `--limit muons` to only run on muon NDXes. + +### `nsclient.yaml` + +Install `nsclient++` monitor program. Needs to be run with + +`ansible-playbook --ask-vault-password windows/nsclient.yaml` + +There is a host group `nsclient` that is used to hold encrypted passwords and can also be used +for deploying + + +### `instrument_deploy.yaml` + +This is for deploying software to NDXes. Currently this stops the server if it is running and installs the JDK using the `jdk` role. + +To use this you need to run `ansible-playbook windows/instrument_deploy.yaml` - it should prompt for hosts. + +### `deploy_wincred.yaml` + +This is for deploying a new set of credentials to NDXes. The credentials should be updated in keeper first; this playbook reads the credentials from keeper. + +To use this you need to run `ansible-playbook windows/deploy_wincred.yaml --ask-vault-pass` - it should prompt for a vault password (found in keeper - search for `ansible`) and hosts to target. + + +## Linux-only playbooks (`linux/`) + +### Initial SSH set up + +This involves: +1) copying over the keys in [this repo](https://github.com/ISISComputingGroup/keys) +2) disabling OpenSSH password-authentication (ie. you can only use pub/priv keys!) + +It can be run multiple times if for example you wanted to update the deployed keys if someone new joins the team or someone gets a new computer with a different public key. + +#### Deploying ssh keys (`playbook_deploy_ssh_keys.yaml`) + +This can be run on multiple hosts by using `ansible-playbook playbook_deploy_ssh_keys.yaml` if you already have ssh key-based auth set up (ie. if you wanted to _update_ the list of keys because someone's public key changed), but if this is a new machine you should use `--ask-pass` (to ask for the ssh password) along with `--limit` to limit the single host (as your ssh passwords may/should be different between machines) + +For example, to run this on `madara`: + +`ansible-playbook playbook_deploy_ssh_keys.yaml --ask-pass --limit madara` +which will prompt for `madara`s ssh password. + +This step also prompts for a personal access token to access the `keys` repo, which is in Keeper. + +#### Turning off `sshd` password authentication (`playbook_turn_off_ssh_passwd_auth.yaml`) + +This is fairly obvious but to do this you need to have done the above step otherwise you'll lock yourself out. + +to run on all hosts, use: + +`ansible-playbook playbook_turn_off_ssh_passwd_auth.yaml` + +This will prompt for the vault password, which is in keeper (`ds-config ansible vault`) + +### General system updates + +`playbook_system_updates.yaml` exists to update the system packages. + +### Kafka cluster provisioning + +`deploy_kafka.yaml` and `deploy_redpanda_console.yaml` exist to provision a Kafka cluster, currently on the test machines in R55. + +## Notes + +### Setting up a windows hyper-v virtual machine for testing + +To test playbooks on a local virtual machine running in Hyper-V, you need to set the following up: + +1) A virtual machine itself. You can use the [evaluation `.iso` images for this](https://www.microsoft.com/en-gb/evalcenter/) +2) If using the `Default switch` on the VM, you need to forward the WSL network so it can reach the VM. To do this run `Get-NetIPInterface | where {$_.InterfaceAlias -eq 'vEthernet (WSL (Hyper-V firewall))' -or $_.InterfaceAlias -eq 'vEthernet (Default Switch)'} | Set-NetIPInterface -Forwarding Enabled -Verbose` from an _elevated_ powershell window. Note that by default most server windows images do not respond to ping requests. You can enable this by enabling the inbound rule `File and Printer Sharing (Echo Request - ICMPv4-In)` in the firewall settings. +3) [OpenSSH set up](https://learn.microsoft.com/en-us/windows-server/administration/openssh/openssh_install_firstuse) on the VM + + +{#ansibleadhoccommand} +### Ad-hoc commands +To run an [adhoc command](https://docs.ansible.com/projects/ansible/latest/command_guide/intro_adhoc.html) to ie. `win_ping` to `TS2` machines to check they're reachable, you can run: + +`ansible ts2 -m win_ping` + +This can also be used to perform one-line shell/bash commands remotely. + +#### Chaining commands + +There is an issue with passing environment variables on windows specifically between chained commands. +If you wanted to run `config_env.bat` to define `MYPVPREFIX` then call `caput` after, you need to use a batch script and copy it across instead. + +This can be done with a playbook similar to the following: + +
+Show playbook + +```yaml +--- +- name: set some PVs + hosts: ndxes + tasks: + - name: create temp dir + ansible.windows.win_tempfile: + state: directory + register: tmpdir + - name: copy bat over + ansible.windows.win_copy: + src: mybat.bat + dest: "{{tmpdir.path}}" + - name: call bat + ansible.windows.win_command: "{{tmpdir.path}}\\mybat.bat" + register: output + - name: print output + debug: + var: output + +``` + +which copies over a batch script `mybat.bat` to a temporary dir, calls it, then Ansible cleans up the temp dir containing the batch script. + +Example batch script, which resets the power check with a for loop: + +```bat +call \instrument\apps\epics\config_env.bat +@echo on +FOR /L %%v IN (1, 1, 9) DO ( + echo running caput %MYPVPREFIX%MOT:DMC0%%v:PWRDET:RESET:SP 1 +caput %MYPVPREFIX%MOT:DMC0%%v:PWRDET:RESET:SP 1 +) + +exit /B 0 +``` + +
diff --git a/doc/processes/dev_processes/Dependency-Updates.md b/doc/processes/dev_processes/Dependency-Updates.md index bb0cba913..b93f1f5a8 100644 --- a/doc/processes/dev_processes/Dependency-Updates.md +++ b/doc/processes/dev_processes/Dependency-Updates.md @@ -160,7 +160,7 @@ In `c:\instrument\apps\epics\support\mysql\master\MySQLCppApp\src\mysql-connecto * ALARM * IOCLOG - Check that CS Studio IDE is loaded correctly -- Update the Ansible playbook with the new version as per [these instructions](https://github.com/ISISComputingGroup/ansible-playbooks/tree/main?tab=readme-ov-file#updating-jdk-version). +- Update the Ansible playbook with the new version as per {ref}`ansibleupdatingjdk` ### Maven - There are two versions of maven, this is to update the one following the format `maven-X.x.x` diff --git a/doc/processes/dev_processes/wsl.md b/doc/processes/dev_processes/wsl.md index 68dc77d4d..a8f285229 100644 --- a/doc/processes/dev_processes/wsl.md +++ b/doc/processes/dev_processes/wsl.md @@ -16,50 +16,8 @@ The WSL filesystem can be accessed using `\\wsl$\` on your native Windows machin ## SSH After you have {external+sysadmin:doc}`generated an SSH key `, you may copy both the public -key and the (encrypted with passphrase) private key to `~/.ssh/` in your WSL instance. +key and the (encrypted with passphrase) private key to your WSL instance. -### Disabling host-key checking for Ansible +See instructions [on this page.](https://devblogs.microsoft.com/commandline/sharing-ssh-keys-between-windows-and-wsl-2) -To avoid Ansible asking for host-key verification for each machine it connects to, you can disable the SSH host-key check prompt. - -You can use either: -```bash -export ANSIBLE_HOST_KEY_CHECKING=False -``` -in `~/.bashrc` to disable host-key checking only in Ansible, or - -``` -Host * - StrictHostKeyChecking no -``` -in `~/.ssh/config` to disable it everywhere. - -### Adding SSH key to SSH agent automatically on WSL startup - -Add the following lines to `~/.bashrc`: - -```bash -env=~/.ssh/agent.env - -agent_load_env () { test -f "$env" && . "$env" >| /dev/null ; } - -agent_start () { - (umask 077; ssh-agent >| "$env") - . "$env" >| /dev/null ; } - -agent_load_env - -# agent_run_state: 0=agent running w/ key; 1=agent w/o key; 2=agent not running -agent_run_state=$(ssh-add -l >| /dev/null 2>&1; echo $?) - -if [ ! "$SSH_AUTH_SOCK" ] || [ $agent_run_state = 2 ]; then - agent_start - ssh-add -elif [ "$SSH_AUTH_SOCK" ] && [ $agent_run_state = 1 ]; then - ssh-add -fi - -unset env -``` - -This will start an agent automatically on login if one is not already started, and add your SSH key to it. +This will start an agent automatically on login if one is not already started, and add your SSH key to it, prompting for the passphrase the first time you use it for the first time since the WSL has started.