Ansible and Rundeck

Rundeck with Ansible

How to build a NICE Ansible environment? Let’s use Ansible with a dynamic inventory, with Rundeck as a GUI (graphical User Interface) and vscode server for file editing directly on the server! I am providing installation scripts for Rundeck with Ansible with vscode server on my Github repository

Nana’s video about Ansible is here

Watch the video on YouTube

Click to view the entire transcript

There are so many videos about Ansible already on YouTube and many of them follow this scheme: They quickly explain what Ansible is, then they show you how to write a rudimentary inventory with nano or vi or any other command line editor, then you’d write a simple playbook and run it on the command line. Come on really? That’s where they leave off. But none of the videos that I watched actually showed me how to build a nice usable environment for my everyday’s needs. This is how I use Ansible with Rundeck at home. I can browse all nodes in my network in a nice inventory. Be it Wifi routers, Proxmox Containers, Raspberry Pis, Virtual Machines, Windows laptops – whatever. This inventory is dynamic. So whenever I add a host to my network, it is added here automatically. Plus - I can nicely put the target nodes in groups with these tags. And then I have jobs – stuff that I want to be done. Such as “run a backup” or whatever. I can run these jobs ad hoc on demand or schedule them – on a single host, a group of hosts or all hosts. The jobs themselves are actually single or multiple ansible playbooks. If I want to modify my playbooks then I can do that from my workstation using my editor of choice such as vscode in my case – no fiddling around with the command line or nano or vi really. Let me show you how to build this. Just very quickly: If you have no idea what Ansible is and you want to learn the basics then please check out this great video by fellow youtuber Nana at “Techworld with Nana” – She’s a devops engineer and explains in 17 minutes how ansible works, what playbooks are and the like. Definitely worth watching. Alternatively, just watch and I’ll explain stuff as we go. (Installation) Let’s start with the installation. I am providing Installation scripts on my github repository for you. There is a bash script for Debian and there is a Dockerfile and a Docker-compose File if you want to try it out or run it in a Docker Container. If you want to run it in a Proxmox Container, then create a Debian 11 container, give it at least 2 Gigs, better 4 GB of RAM, one or two cores and 10 or better 12 GB of disk. Run the installation script and it will take care of everything. If you want to use the Docker version then just grab the files from my repo, run the installation script which will in fact create a hidden .env file which contains all the parameters for the containers. After that, just do a docker-compose up and that will spawn two containers – one with rundeck and ansible and a second one with a mariadb.The Dockerfile is built on the official rundeck container and just adds Ansible to it really. I found that no one on YouTube really explains why they install ansible the way they do. As you can see in the script and in the Dockerfile, I am using pip to install ansible. Why? Ansible is a python application really. So to me the most natural and portable way to install it is through pypi or pip. That can be done on all Linuxes and does not depend on a specific distribution. So why do I install it as root? Usually Python packages should be installed in a virtual environment in a user context? Well, I’ve done that before and it just turned out that it’s much easier to have the same python libraries, modules and binaries in the right path for everyone on the machine rather than two users having different versions. Of course this is only true if you install into a container or VM. On a server with other python applications this would not be the best solution. We will however see that I put the ansible configuration files into the rundeck user’s home directory rather than into /etc – why? That will make it much easier to remotely edit those files. We will see that in a minute. Cool, that’s it – we’re done with the command line. (How to edit files) Let’s quickly talk about the client setup. Our workstation. We will need to give ansible a couple of configuration files, but we don’t really want to edit those with nano or vi on the command line over ssh. I want to show you two ways how you can use a graphical editor from your workplace in order to manage those files. First method: I personally think that a nice way to remote edit files on Linux systems is to use Winscp or FileZilla. So what I can do here is that I remote connect to the server using the rundeck user and the Password that I have specified and I can now copy, create, modify or delete files. When I want to edit a file, then I just do a right click, then view/edit and my preferred editor comes up which in my case is Visual Studio Code. But you could use anything else here. I just like vscode for development. When I change a File, then Filezilla takes note of that and asks me if I want to update it on the server. Now for a second method. (vscode server) My scripts also install a server version of vscode on the rundeck server. So with this you can use a web browser version of Visual Studio code directly in a browser from your workstation. The advantage of this is that you run the editor on the rundeck / ansible server and you can add all the nice vscode plugins here. Let me do that. I log into vscode server on my rundeck server and I want so install some extensions here. Let me install the github extension and the extension for yaml files. Now I can go to the source control menu here and clone Marc’s example files from the OneMarcFifty github repository. Takes a second – here it is. This enables you to use github in the Rundeck Ansible GUI directly on the server. Furthermore, the web version of vscode can be used to create, move, copy and delete files on the server without any need for ssh, FileZilla, Winscp or the like. If I need a shell then I can also open that directly in the browser here in the Terminal menu. Just be advised that this has not been optimized for security in any way. You might want to hide it behind an NGINx for example. (Non-Root-User advantages) I told you that I prefer to install Ansible as root, but that I would rather have the Ansible config files under the home directory of the rundeck user. Here is why. By default, root ssh access with user name and password to a Debian Linux is deactivated. You could of course use a private and public key and I strongly recommend you do that. However, if I connect to the rundeck/Ansible server as root and edit files as root, then they would not be editable to the rundeck user. By default however I can ssh and therefore also connect with Filezilla for example as a non-root user and more easily edit the files there. This does not work with the Docker version as there is no ssh daemon running in the docker container. The home directory of the rundeck user which I want to use for all ansible actions as well is located in /var/lib/rundeck. In the docker version it is /home/rundeck. You can see that hidden .ansible directory here. Let’s ignore that for the time being and create a new directory called ansible – without the dot. This is where we put our stuff in. The docker version already has that directory which is in fact a named docker volume so that you don’t loose your work if ever you would need to recreate the container. Under that directory we create a sub-directory called playbooks and another one called inventory. You can do this remotely with WinSCP or Filezilla. Or you can do this with the Web browser version of vscode. That’s how you would do it in the docker version anyhow. (Config Files and Inventory) Before we can use Ansible and Rundeck, we need to give Ansible a couple of config files. First the generic configuration file. When you launch ansible, then it searches for config files in the following order: First in the $ANSIBLE_CONFIG directory if that environment variable is set, then it looks for a file called ansible.cfg if it’s in the current directory. If it can’t find that then it looks for a file called .ansible.cfg in the user’s home directory and last but not least for /etc/ansible/ansible.cfg, the default config file. My suggestion is to just put it into the .ansible.cfg file in the rundeck user’s home directory. This way the rundeck user will always find it. So I go to the /var/lib/rundeck directory and create the new file .ansible.cfg and add the following content into it: Write “defaults” in square brackets and below you type inventory=~/ansible/inventory. This line just tells Ansible where to find the inventory, so the list of hosts that we can do stuff with. The tilda stands for the user’s home directory. You could as well write /var/lib/rundeck or /home/rundeck instead. Save the file. So here’s another subtle nuance of Ansible which the other videos just don’t talk about. If I just pointed the inventory to a file rather than to a directory, then I could only have one single inventory. That means that if I have three machines, then I need to write their names into that file. If I add a fourth machine to my environment, then I would have to change that file. Very annoying and not very agile. Here’s a better solution. I can add multiple inventory files into the inventory directory now. Those could be yaml files describing static hosts or groups, they can be yaml files referring to inventory plugins or they can be scripts that give back a list as a json file. They will be treated alphabetically. That’s why I called them 10-something, 20-something etc. just to make sure that they are treated in the right order. The first file is called 10-static and mainly just defines groups of hosts and variables that are different for every group such as a different ssh port or user etc. The second file which is called 20-nmap.yaml just scans my entire network for hosts and adds them to my inventory. There are a couple more files here. One uses the Proxmox Api to get a list of all proxmox containers and VMs and the other one connects to my Zabbix Server and pulls the inventory from there. The last file which is called 99-construct.yaml now dispatches the hosts to the right group. Let’s quickly see what that looks like in the inventory. You can run ansible-inventory –graph and see which hosts are assigned to which group. Nice. Let’s go and tell Rundeck about the inventory. (Adding the inventory to rundeck) Rundeck can use the Ansible inventory. All that we have to do is define a job that actually does that. When we first log into rundeck then we can create a new project here. Let me call it Ansible. The next thing rundeck wants to know is where we get the nodes from. Here we click on “add a new node source”. And of course we use the Ansible Resource Model Source. The only thing that I really need to specify here is the Ansible config file path. Plus – I prefer to uncheck “gather facts” as I do have nodes without python as well. Now I click on “Jobs” – “create a new job” – I’ll call it update nodes – and just under the Workflow section here I need to specify the Refresh Project Nodes Type under Workflow steps. Save. When I now say “Run this job now” then rundeck will query the ansible inventory and update the nodes accordingly. If you use nmap to create the inventory then this may take a while as the network will need to be scanned. You can now verify if the nodes have been added by checking in to the nodes section here. (deploying ansible – prepare hosts) Two more steps to do before we can really use the Rundeck Ansible GUI. And these steps are related to how Ansible works really. You probably know that Ansible uses ssh. That means that it connects to other nodes using ssh. And it is agentless. But that’s not quite the whole truth. Ansible needs a little bit more than just plain ssh on the remote host so that you can use it properly. On the remote machine we need three things in order to enjoy full Ansible functionality: A non-root user which is able to become root as needed with sudo, we need python – preferrably python3, and we should also be able to use a private key to connect to ssh. So let me quickly do this. First – I want to create an ssh key which I will use in the future to connect to the remote machines. I can do this with vscode server. If I select the vscode menu here, then terminal, new Terminal, I actually get a shell on the rundeck server inside my web browser. Inside the terminal I just type ssh-keygen in order to create a public/private key pair. By default, this uses rsa. If you want to use eliptic curves such as ed25519 which needs much smaller keys for the same level of security then just specify this with -t ed25519. I use rsa here for compatibility reasons. You may give the key a password – we’ll see in a minute how the Rundeck Ansible GUI handles passwords and other secrets. I can now trash this terminal and find the newly created key here under .ssh. We’ll use it in a minute. Just a quick remark here with regards to the docker version from my Github: the key is now stored inside the container. If you scrap the container, then the key is gone. Either store it in the rundeck vault like we will do in a minute or copy it over to the /home/rundeck/ansible directory because this is a persistent volume in the stack. But first I have to prepare the target node. I do need some initial ssh or console access to the target node for that unfortunately. But I’ll show you in a minute how we can actually automate this step with Rundeck and Ansible. But first let’s do this by hand. So I ssh into the target node or open a console for example with Proxmox if my target node would be an LXC container and do the following. I create a new user, I add it to the sudo group, I then give it a password. Next I want to make sure that python3 is installed. Last but not least I need to add the public key to that user. So from the rundeck node I type ssh-copy-id and then the newly created user at the new target node and then the path to the key which I had just created. As you can see, this is quite cumbersome. Also I might need additional configuration, for example if I don’t want sudo to ask for a password. Let’s see how we can automate this with Rundeck and Ansible. Let’s see how we can deploy Ansible with Ansible. (Deploy Ansible with Ansible – rundeck key store) In my github repository you can find a couple of example playbooks. Let’s use this deploy-ansible sample in order to automate the steps which we have just done by hand. I quickly copy it to my playbooks directory with the vscode server app. Now I log into the Rundeck Ansible GUI and configure the keys and passwords. That’s one big advantage of using rundeck together with ansible. You can safely store passwords and keys in rundeck and use them with ansible. Let’s go to project settings down here in the left corner and then key storage. The default path for the key is set to keys/project/ and then the name of the project which in my case is Ansible. If I click on “add or upload a key” then I can add private keys, public keys or passwords to the vault. If you are using ansible vault then you could add the password to your ansible vault here. What I want to do is that I add some passwords for initial ssh connection where I don’t have private/public key authentication set up yet. I also want to add the password to the ansible rsa key which I have created in the previous chapter. Now I can create my first job by clicking on jobs here. Actually my second one because I already have the inventory job here. I click on New job, call the job deploy ansible, and then I click on the Workflow tab here. There’s loads of options here – in the interest of time we can’t look at all of them here. Below here we can find the section “Add a step”. What we want is a node step, that means a step that is executed on every single node that we specify. Under the node steps plugins here we select “Ansible Playbook Workflow Node Step”. Now we can put in all the parameters for ansible and also the variables for the playbook. Another great advantage if you use rundeck. You can use ansible variables in your playbooks and specify them here in various different formats. Further down we can now specify the connection parameters for ssh and select the passwords for the connection or for the key to use. Further down you can also specify options for superuser interactions, if for example you need to specify a password in order to sudo. Quick remark - The way I execute this job in this example – with root and a password – does only work if you allow root login to ssh with a password on your node. Also – in my playbook I have defined a variable called ansible_user_password. That’s the password that will be assigned to the ansiblessh user. In order to pass this on, I need to do two things. First I need to define an option for that value on the job by cicking on that “Add an option” button up here. I call it userpass. The input type shall be secure. And here I can select the key from the rundeck vault. In my case that’s the sudo password. In the extra variables I can now select this option to be passed to the ansible playbook. On the node tab let’s just tell rundeck that this job will be dispatched to nodes. We can select them at runtime. Cool – let’s run that job. Here I can change the nodes and actually tell rundeck to maybe only execute on one specific node or on a group of Machines. Once I click on run now then it goes off and executes the job on the target nodes. We can see that it does the first step which is just raw execution of apt update and apt install python3 and then it does the other steps. From now on I can connect with the new ansiblessh user and use the id_rsa key for authentication. So – I am sure that you get the idea on why and how to use rundeck together with Ansible. A couple of features are really nice here, especially with regards to storing keys, integrating the inventory, passing on options as variables and so on. Feel free to use the examples which I provide on github. Just before we close – what other alternatives are there if you were looking for an Ansible GUI? In enterprise environments you would presumably use Tower or AWX. Why do I not use them? Well, I found that they are a bit over sized for my small environment. Also – everything seems to go Kubernetes which I don’t have at home. Another GUI is Semaphore. What I like with Semaphore is that it is free and open source, it’s easy to use and it has a very clean and simple User Interface. I just felt that rundeck would give me more features and possibilities. But Semaphore is definitely something to watch. That’s it folks. Leave me a comment on YouTube if you want more. Thanks so much for watching. Stay safe, stay healthy, bye for now.