Some time ago, we were asked (twice) to add a module for managing HTTP proxies in Foreman to the Foreman Ansible Modules collection. Given the limited scope of the module, it’s a perfect fit for a guide how to write new modules utilizing the abstraction we have developed in Foreman Ansible Modules!
What does the
fox API say?
The first step when planning to implement a new module should always be to consult the API documentation. In this case, we’re interested in the HTTP Proxies endpoint which describes how to show, create, update and destroy HTTP Proxies in Foreman, especially in the fields required when creating a new or update an existing proxy:
name, required for create, optional for update
url, required for create, optional for update
We also see that there are no other actions besides list, show, create, update, and delete.
Creating a minimal module
A regular Ansible module always starts with an instance of
In Foreman Ansible Modules, we have several classes based on this basic one, so we need to decide which base class from
foreman_helper we should take.
HTTP Proxies inside of Foreman are stateful (can be created and deleted) entities that can have taxonomies (Organizations and Locations), so we’re taking the
ForemanTaxonomicEntityAnsibleModule class, which will provide us with the
locations parameters and automatic handling of those. Please refer to the
foreman_helper documentation for other possible base classes and their use-cases.
from ansible_collections.theforeman.foreman.plugins.module_utils.foreman_helper import ForemanTaxonomicEntityAnsibleModule
As the helper decides which API endpoint to talk to based on the class name, we create a minimal class:
class ForemanHttpProxyModule(ForemanTaxonomicEntityAnsibleModule): pass
Now we need to add the other fields the API supports as parameters. We’ll mark the
name parameter as
required because that’s the one we’ll be using for finding existing proxies – the API has it as
optional because it uses the ID of the proxy as the primary identification, but nobody wants to write static ids in Ansible playbooks. The
url parameter is only required for new proxies, so we’ll mark it as
optional in the module parameters and document the requirement for creation. As our modules already have
password parameters for authenticating against Foreman, we’ll have to name the parameters for those fields differently –
proxy_ seems like a good prefix to add here. And last but not least,
password parameters should be marked as
no_log=True so that Ansible skips them when printing debugging information about the module invocation:
module = ForemanHttpProxyModule( foreman_spec=dict( name=dict(required=True), url=dict(), proxy_username=dict(flat_name='username'), proxy_password=dict(no_log=True, flat_name='password'), ), )
flat_name is telling our abstraction layer that the field name in the API differs from the Ansible parameter name. See the specification of the
foreman_spec for details how it works and extends Ansible’s
At this point, we can add the code for the API connection and parameter processing and would have a working module:
with module.api_connection(): module.run()
However, as noted above, the
url parameter is only optional when modifying an existing proxy, but we marked it as optional for any module invocation because Ansible doesn’t know if it’s creating a new proxy or modifying an existing one before the API connection has been established. To accommodate for that, we need to manually execute the “search for entity” step and then raise the appropriate error message if the user is trying to create a proxy without specifying the URL:
with module.api_connection(): entity = module.lookup_entity('entity') if not module.desired_absent: if 'url' not in module.foreman_params: if not entity: module.fail_json(msg="The 'url' parameter is required when creating a new HTTP Proxy.") else: module.foreman_params['url'] = entity['url'] module.run()
And that’s it from a code perspective, we now have a working module to manage HTTP Proxies in Foreman!
Adding Ansible documentation to the code
While the code is done, the documentation is not. And documentation is important! Our CI runs
ansible-test sanity on the modules, and that requires all parameters to be properly documented.
The most interesting part of the
DOCUMENTATION stanza is the
extends_documentation_fragment section. As we have several parameters coming from our base class, we also have documentation fragments that document these parameters:
extends_documentation_fragment: - theforeman.foreman.foreman - theforeman.foreman.foreman.entity_state - theforeman.foreman.foreman.taxonomy
theforeman.foreman.foreman adds the most basic parameters for the API connection,
theforeman.foreman.foreman.entity_state adds the
state parameter for our entity and finally
locations parameters to describe the taxonomy.
The rest of the
DOCUMENTATION section is basic Ansible module documentation, so we’ll omit it here.
Writing tests is tedious and annoying, but at the same time, a lot of bugs can be prevented if you have tests. Or could have been prevented, if there would have been tests ;-)
In Foreman Ansible Modules, we mostly do integration tests – unit testing modules is not really feasible. Those integration tests consist of an Ansible playbook that executes the module multiple times and analyses the results and a set of recorded API calls that can be used to replay the tests without having a full Foreman running, which saves a lot of resources when running the tests and allows the tests to run in parallel.
To write a test, just drop a new playbook into
<module_name>.yml – so in our case
http_proxy.yml. As this playbook will call the same module over and over again, we have established the practice to have a
tasks/<module_name>.yml file with the whole module invocation and then simply include it in the main playbook using
For the HTTP Proxy module, the tasks file looks like this:
--- - name: "Create/Update/Delete HTTP Proxy" vars: http_proxy_state: "present" http_proxy: username: "" password: "" server_url: "" validate_certs: "" name: "" url: "" proxy_username: "" proxy_password: "" locations: "" organizations: "" state: "" register: result - assert: fail_msg: "Ensuring http_proxy is failed! (expected_change: )" that: - result.changed == expected_change when: expected_change is defined ...
In this task file, there are two tasks. The first one calls the
http_proxy module, passing in the connection information and the module-specific parameters, most of which have
default(omit) specified, which means these parameters won’t be passed to the module if the variable isn’t set – super useful when you need to test the module with different sets of parameters. And the second one that ensures that the module reported the
changed attribute correctly.
Now the main playbook (
http_proxy.yml) is a bit longer, so we won’t paste it verbatim here, but discuss the individual sections.
First, it has a play with
hosts: localhost to prepare the test environment: setup organizations, locations and whatever else might be needed.
This is followed by a long play with
hosts: tests that contains the actual test steps.
Those steps follow a pattern: do something and expect a change to happen (
expected_change: true), do the same thing again and expect no change to happen (
expected_change: false). This ensures idempotency of the module.
The “things to do” again follow a pattern: create a proxy with minimal information, update a proxy, delete a proxy. This ensures the common workflows of a user are covered.
At last, it has another play with
hosts: localhost to tear down anything that has been prepared for this specific test to run.
The differentiation between
hosts: localhost and
hosts: tests allows us to execute the
localhost steps only when we’re recording a new set of API interactions and the
tests steps always, as there is no need for the setup and teardown when running the tests against the recorded data.
The last missing piece between us and working tests is a symlink. To keep the recorded API data clean of the big apidoc JSON and also allow easier updates when the apidoc changes, we keep copies of the JSON in
tests/fixtures/apidoc. However, we keep only
luna.json as real files while all the others are symlinks. As the HTTP Proxy is a Foreman feature, we will create a symlink from
foreman.json. Would it have been a Katello feature, the link would have been to
katello.json. And for any other plugin, it’s
luna.json as that’s essentially “Foreman with all the nice plugins”.
Having all pieces in place (and a running Foreman instance somewhere), we can run
make record_http_proxy which it will execute our playbook and record the API interactions in
The record step can be re-run as often as required. You can also execute
make livetest_http_proxy to execute the whole test playbook without recording the interactions.
This post only contains code snippets, you can read the full code of the added
http_proxy module on GitHub.