Foreman Community Newsletter - September 2018

2018-09-30T13:59:04+00:00

Releases

Since 1.19.0 went out right at the end of August, it’s a bit quiet for releases just now. However, 1.18.2 has been released, containing 9 bug fixes for the 1.18 series. Do upgrade if you’re still on 1.18!

Also, as mentioned in the previous newsletter, Katello 3.8.0 was indeed close to release - in fact, it was made available on Sep 6th! With ~50 improvements, and of course compatibility with Foreman 1.19, you’ll want to get that upgrade in as well.

As ever, many thanks to the contributors to these releases - coders, RC testers, documentation writers, and bug/feature reporters. Foreman wouldn’t be what it is without you! If you do find issues with, please do report them!

Critical Security issue in Remote Execution plugin

If you use the Remote Execution plugin, you’ll want to make sure you’ve seen CVE-2018-14643 and the associated updates on the forum. This is an arbitrary code execution flaw on hosts managed by Foreman, so please do take this seriously.

Updates are available for the affected packages from Foreman 1.15+, see the forum post for details.

RFCs for comment

As usual, I like to call out some RFCs for wider attention. Here’s this month’s selection:

In addition, I’ll again mention the large set of RFCs regarding our approach to making Foreman able to run in a (set of) container(s) - lots of good discussion is happening here, but more voices are welcome. Start with the overview and then dive into the details RFCs from there.

New Community stats dashboard

One of my long-term to-do items as community lead has been to create a metrics dashboard that we can use to track various things that matter to us as a community. That’s been on the back burner for a while, but I’ve now had time to get started on it. You can find the current version here.

The current version is somewhat alpha, and only covers:

Total Open / Total Closed / Net Open bug (all projects or for one project)
Open bugs by project / category / triage status
Community demographics (age of Redmine account for people who created/commented on a bug in the last 6 months)

Already this helps us to see the state of the community, track it’s evolution, and hopefully focus on certain bug categories. I have more charts and interactive tools in development - if you know the “R” language and would like to help, I’m reachable here :)

Hacktoberfest (via DigitalOcean and GitHub)

If you’re looking for that extra incentive to get you contributing, it seems that DigitalOcean & GitHub are again running a Hacktoberfest Event. You can get the full details over there but essentially, 5 separate (and not spammy) PRs to any GitHub repos will count. So, that means our repos too :). It appears they have 50,000 t-shirts to give away to those who manage 5 PRs during the month of October. Get contributing!

Upcoming Events

All Foreman events are available via the forum calendar - hit the Subscribe button (which will give you an ICS link). You can also use the Add To Calendar button on specific events if you don’t wish to import the whole thing.

Future community demos will now be created much further in advance, so you should be able to keep up with upcoming demos there.

Community Demo #51

All the usual goodness from the community, rounded up and presented for your viewing pleasure.

Open Source Automation Day

On October 16, 2018 the Open Source Automation Day of ATIX AG will take place in the Mercedes-Benz Tower in Munich.

At one of Munich’s largest open source conferences we offer you an appealing program with lectures, live demos, talks and discussions. We focus on a mix of business and technical topics. The focus here is on possibilities for the simple and automatic operation of data centres. A special focus is on the latest technologies.

Foreman TechDay - after OSAD

We at ATIX think, that it’s a good time to hold a Foreman TechDay the day after the OSAD conference in Munich. The Foreman TechDay is targeted at software developers who already have knowledge of software development in Foreman or similar solutions like Ansible module development. Feel free to add suggestions for topics!

Open Source Summit, Europe

I’ll be at OSS-EU on Oct 22-24th - I have two talks there in the community track (which aren’t really about Foreman, but about community work in general), and I’ll spend a lot of my time hanging out at the Red Hat stand. Come find me if you want to chat! :)

Past Events

Community Demo #50

Hammer, Ansible variables, content changes, and a quick overview of the new stats dashboard. Click the link to see the full agenda!

Plugin news

New Plugins:

Foreman M2

Integration of the M2 (Malleable Metal as a Service) with Foreman. Check out the forum post for more details.

Updated plugins:

foreman_ansible updated to 2.2.9
foreman_chef updated to 0.8.1
foreman_openscap updated to 0.10.3
foreman_remote_execution updated to 1.6.3
foreman_userdatato 0.1.0
hammer_cli_foreman_tasks updated to 0.0.13
hammer_cli_foreman_templates updated to 0.1.2
hammer_cli_katello updated to 0.14.1

Acknowledgements

Thanks to everyone who contributes to the Foreman community - patches, plugins, testing, bug reports, translations, interviews, presentations, meetups and everything else. Your efforts are very much appreciated!

Foreman Community Newsletter - August 2018

2018-08-31T15:17:00+00:00

Releases for everyone!

We’ve had a lot of releases this month, so I’ll group them all together. We’ve released:

Foreman 1.19.0 was released on the very last day of August, featuring support for Ubuntu Bionic, syslog logging, reporting failed builds, and more.
1.17.3 and 1.17.4 bug fix releases. If you are still on 1.17, be sure to upgrade to 1.17.4 before upgrading to 1.18!
Katello 3.8 RC 2 & 3 - do give the RC a try and let us know how you find it :)
Katello 3.8 appears to be in the release process right now so stay tuned for that!

With the release of 1.19.0, this marks 1.17.X as now unmaintained - 1.17.4 will be the last release in the 1.17 series. Please do upgrade as soon as you can.

Big thanks also to Ondrej, Ewoud, and Tomer for their effort in bringing these releases out at this rate, there’s been a lot of work behind the scenes.

New RFCs for container work

In the survey analysis I posted earlier this year, I called attention to the idea of supporting Foreman being run in a container, and that more would come on this later. That time is now :)

Eric has spent a lot of time working through the implications of this, and has now posted a set of RFCs to the Development board for review. There’s a lot to read here, so it’s broken down into multiple posts. You can get the top-level over on this forum post and then dive into details from there.

I’d recommend for anyone with an interest in the future direction of Foreman to read over these, and give their thoughts on any part of it which interests you. Thanks!

Foreman at FrOSCon

Thanks to the awesome work of Matthias, Bernhard, and the rest of the ATIX team, Foreman had a community presence at FrOSCon at the University of Applied Sciences Bonn-Rhein-Sieg this year. Whilst I didn’t get to go myself, I heard that it was a great event, lots of traffic to the booth, and good discussions with other collaborators.

You can read all about it on the ATIX blog post! Thanks as always to you all for your support!

Redmine upgrade (continued)

Whilst the Redmine upgrade was completed last month, Ewoud and I have continued to improve it. This month we cleaned up the project hierarchy, moving plugins to their own top-level project, and keeping just “core” projects grouped under Foreman. The main reason for this is so that we can re-enable Version sharing without cluttering the bug tracker.

A consequence of this affect the roles that users have on each project. If you had Developer or Manager roles on a project, then you should refer to this post for instructions on what to check.

Upcoming Events

The Foreman Community has now fully embraced Discourse as the source of its event data - if you’ve been using the old /events/all.ics in your own calendar, you’ll need to head over to the forum calendar and hit the Subscribe button (which will give you an ICS link). You can also use the Add To Calendar button on specific events if you don’t wish to import the whole thing.

Future community demos will now be created much further in advance, so you should be able to keep up with upcoming demos there.

Community Demo #50

All the usual goodness from the community, rounded up and presented for your viewing pleasure.

Past Events

Community Demo #48

A short demo, with Docker, dev changes, an analysis of the impact Discourse has had, and some chat about how to improve our docs.

Community Demo #49

A busy and Katello-centric demo containing lots of new content changes. Also seeding and Bash completion in Hammer!

Plugin news

Updated plugins:

foreman_ansible updated to 2.2.6
foreman_discovery updated to 13.0.1
foreman_host_extra_validator updated to 0.1.0
foreman_remote_execution updated to 1.6.1
foreman_scc_managerto 1.4.0
foreman_tasks updated to 0.14.0
hammer_cli_foreman updated to 0.14.0
hammer_cli updated to 0.14.0
katello_host_tools updated to 3.3.5
smart_proxy_omaha updated to 0.0.5

Acknowledgements

Deploying ESXi through Foreman

2018-08-09T14:25:45+00:00

At the time of writing this (Foreman v1.18), ESXi deployment through Foreman is not supported natively. The problem is exacerbated by the fact that the ESXi Legacy BIOS bootloader depends on an older Syslinux (v3.86) and GRUB2 not being able to chainload the ESXi EFI bootloader. However there are quite a few articles on the Internet that have attempted to work around these issues.
This post is an attempt to bring all the information together in one place, simplify the process and most importantly, support both Legacy BIOS and UEFI modes of installation.
Huge thanks to the foreman developers for helping me out whenever I was stuck.

Requirements

A fully working Foreman instance. Follow the manual to install Foreman. After installation, please configure Foreman so that a Linux OS (for e.g. Centos) can be deployed successfully through Foreman.

Please make sure the Bootfile Handoff section in dhcpd.conf looks like this:

  # Bootfile Handoff
  next-server 127.0.0.1;
  option architecture code 93 = unsigned integer 16 ;
  if option architecture = 00:06 {
    filename "grub2/bootia32.efi";
  } elsif option architecture = 00:07 {
    filename "grub2/bootx64.efi";
  } elsif option architecture = 00:09 {
    filename "grub2/bootx64.efi";
  } else {
    filename "pxelinux.0";
  }

Foreman-Hooks. Run the following command to install:

foreman-installer --enable-foreman-plugin-hooks
Syslinux version 3.86.
An ESXi ISO. For this demo we will use VMware-VMvisor-Installer-6.7.0-8169922.x86_64.iso but the same process will work on any version of ESXi.

Note: This demo is run on a Ubuntu system. The paths may be different for other OSes. Please refer to the documentation for details.

1: Create an entry for the ESXi OS in Foreman

1.1: Create the OS Medium

Open https://{foreman-url}/media and click on Create Medium.
Now create a dummy entry for ESXi since each OS entry in Foreman needs an installation media.

1.2: Create the OS

Open https://{foreman-url}/operatingsystems and click on Create Operating System
Under Operating System, enter the following:
Name - ESXi-6.7.0-8169922 (ESXi-{OS Version}-{Build Number})
Major version - 6
Minor version - 7
Family - Redhat
Root pass hash - SHA512
Architectures - x86_64
Partition Table - Kickstart default
Note: We are using Kickstart default for the sake of simplicity. The actual partitioning will be controlled from the kickstart file. It is also possible to create a new ptable for ESXi and use it here.
Installation Media - Select the dummy media created above
Press Submit

1.3: Create provisioning templates

1.3.1: Create the PXELinux template
- Open https://{foreman-url}/templates/provisioning_templates and click on Create Template
- Name - ESXi-6.7.0-8169922
- Template contents:
```
DEFAULT ESXi
NOHALT 1
LABEL ESXi
KERNEL ../boot/ESXi-6.7.0-8169922/mboot.c32
APPEND -c ../boot-ESXi-6.7.0-8169922.cfg
IPAPPEND 2
```
Note: The KERNEL and APPEND lines start with “../”. I will explain why in Section 2.4. These values will be used when we actually mount the ISO image.
- Template Type - PXELinux Template
- Template Association - Select “ESXi 6.7 Build 8169922”
- Click Submit

1.3.2: Create the Kickstart Template

Click on Create Template
Name - ESXi Minimal Kickstart

A sample template:

vmaccepteula
keyboard 'US Default'
reboot
rootpw --iscrypted <%= root_pass %>
install --firstdisk --overwritevmfs --novmfsondisk

# Set the network to DHCP on the first network adapter
network --bootproto=dhcp --device=<%= @host.mac %>

%post --interpreter=busybox
# Add temporary DNS resolution so the foreman call works
echo "nameserver <%= @host.subnet.dns_primary %>" >> /etc/resolv.conf

# Inform Foreman that we are done.
wget -O /dev/null <%= foreman_url('built') %>
echo "Done with Foreman call"

Template Type - Provisioning template
Template Association - Select “ESXi 6.7 Build 8169922”
Click Submit

1.4: Set default provisioning templates
Now that the templates are created, we need to set those as default.

Navigate to https://{foreman-url}/operatingsystems and click on the newly created ESXi OS “ESXi 6.7 Build 8169922”
Open the “Templates” tab
Select the newly created provisioning templates from the drop down boxes and press Submit.

2: ESXi installation in Legacy BIOS mode

The ESXi PXELinux mboot.c32 module only supports syslinux bootloader (pxelinux.0) version 3.86. You may replace the newer pxelinux.0 file that comes with the default Foreman installation with pxelinux.0 v3.86 but then you might face issues with newer OSes. So we will use a different approach.

2.1: Prepare the bootloader
We will create a new directory under the tftp root directory (typically /var/lib/tftpboot) and place the syslinux v3.86 pxelinux.0 along with the c32 modules there.

  cd /tmp/

  wget https://mirrors.edge.kernel.org/pub/linux/utils/boot/syslinux/3.xx/syslinux-3.86.tar.gz

  tar xvf syslinux-3.86.tar.gz

  mkdir /var/lib/tftpboot/syslinux386

  cp syslinux-3.86/core/pxelinux.0 /var/lib/tftpboot/syslinux386/

  find syslinux-3.86/com32/ -name \*.c32 -exec cp {} /var/lib/tftpboot/syslinux386 \;

  # Create a symlink to the default pxelinux.cfg directory inside new directory
  cd /var/lib/tftpboot/

  ln -s ../pxelinux.cfg syslinux386/

2.2: Create the bootloader entry in Foreman
Now comes the interesting part. We will create a new PXELoader entry in Foreman for pxelinux-3.86.0. Huge thanks to Lukáš Zapletal for pointing this out.

2.2.1: Edit the file - /usr/share/foreman/app/models/concerns/pxe_loader_support.rb


def all_loaders_map(precision = 'x64')
  {
    "None" => "",
    "PXELinux BIOS" => "pxelinux.0",
    "PXELinux UEFI" => "pxelinux.efi",
    "PXELinux Alt BIOS" => "syslinux386/pxelinux.0", # Add this line
    ...
</pre>

2.2.2: Restart httpd/apache2 for the changes to be reflected.

2.3: Mount the ESXi ISO
We will mount the ESXi ISO under /var/lib/tftpboot/boot/ESXi-6.7.0-8169922. Note that this path corresponds to the KERNEL entry in the PXELinux template.

  cd /var/lib/tftpboot/

  mkdir -p boot/ESXi-6.7.0-8169922

  mount ~/Desktop/VMware-VMvisor-Installer-6.7.0-8169922.x86_64.iso boot/ESXi-6.7.0-8169922

2.4: Edit boot.cfg
Copy the boot.cfg file from the mountpoint to tftproot and edit it to add a prefix to the ISO mountpoint. Make sure to use the same filename as the one specified in the APPEND line of the PXELinux template.

  cd /var/lib/tftpboot/

  cp boot/ESXi-6.7.0-8169922/boot.cfg boot-ESXi-6.7.0-8169922.cfg

  sed -e "s#/##g" -e "3s#^#prefix=../boot/ESXi-6.7.0-8169922\n#" -i boot-ESXi-6.7.0-8169922.cfg

Note:

Since the host boots into syslinux386/pxelinux.0, the TFTP root dir will be set to /var/lib/tftpboot/syslinux386. So all paths need to be relative to this path. This is why all paths need to be prefixed with “../”.
Instead of mounting the ISO, you can also extract the ISO contents to the directory and edit the boot.cfg directly inside that directory. Remember to change the PXELinux template accordingly. I chose this approach because it is a tad cleaner.

Now we are ready to deploy this OS on to a host.

2.5: Open https://{foreman-url}/hosts/ and edit the host on which you want to deploy ESXi.

2.6: Under the ‘Operating System’ tab, select the following values:

Operating System : ESXi 6.7 Build 8169922
Media : ESXi Dummy
Partition table: Kickstart default
PXE Loader : PXELinux Alt BIOS

2.7: Press ‘Resolve’ beside Provisioning templates and you should see the ESXi PXELinux and Kickstart templates.

2.8: Press ‘Build’ and reboot the host to PXE. If all goes well, you should see ESXi installer load up and perform an automated installation.

Once installation is complete, the host will reboot to PXE and will be presented with the default PXE menu.

Upon timeout, it will attempt a Localboot and boot to ESXi.

3: ESXi installation in UEFI mode

3.1: Prepare the bootloader
In UEFI mode the bootloader is mboot.efi which is just a renamed version of efi/boot/bootx64.efi from the ESXi ISO image. The same mboot.efi can be used for booting different ESXi versions but it is recommended to use the latest one.
Let’s create the bootloader.

Assuming the ESXi ISO is mounted under /var/lib/tftpboot/boot/ESXi-6.7.0-8169922:
```
cp /var/lib/tftpboot/boot/ESXi-6.7.0-8169922/efi/boot/bootx64.efi /var/lib/tftpboot/mboot.efi
```
3.2: Create the bootloader entry in Foreman
Now we have to create an entry for this bootloader in the Foreman UI just like we did in Legacy BIOS mode.
- Edit the file: /usr/share/foreman/app/models/concerns/pxe_loader_support.rb
```
def all_loaders_map(precision = 'x64')
  {
    "None" => "",
    "PXELinux BIOS" => "pxelinux.0",
    "PXELinux UEFI" => "pxelinux.efi",
    "PXELinux Alt BIOS" => "syslinux386/pxelinux.0",
    "mboot UEFI" => "mboot.efi", # Add this line
    ...
</pre>
```
- Restart httpd/apache2 to register the changes

3.3: Once the host loads mboot.efi, it will look for a boot.cfg file under tftproot/01-{MAC address of host}. Once it is located, the installer loads up and performs the installation. So each time we need to deploy ESXi in UEFI mode we need to create this directory and copy over the boot.cfg file. This can be quite tedious when we are deploying at scale.

Here is where Foreman-Hooks comes in handy. From the Foreman-Hooks Github repo:

Foreman Hooks allows you to trigger scripts and commands on the Foreman server at any point in an object’s lifecycle in Foreman. This lets you run scripts when a host is created, or finishes provisioning etc.

Once the host is set to build mode, the MAC directory along with the boot.cfg file should be created automatically. For this we need to hook into the after_build event.

cd /usr/share/foreman/config/hooks

mkdir -p host/managed/after_build

cd host/managed/after_build

# Symlink the hook_functions file.
ln -s /usr/share/foreman/vendor/ruby/2.3.0/gems/foreman_hooks-0.3.14/examples/bash/hook_functions.sh .

# Now create the script
cat  <<EOT >> /host/managed/after_build/01-prep-esxi-uefimode.sh
#!/bin/bash

# Import the functions
. $(dirname $0)/hook_functions.sh

# event name (create, before_destroy etc.)
# orchestration hooks must obey this to support rollbacks (create/update/destroy)
event=${HOOK_EVENT}

# to_s representation of the object, e.g. host's fqdn
object=${HOOK_OBJECT}

exec >> /tmp/${event}.log
exec 2>&1

system_name=$(hook_data host.host.name) # Yes it is host.host.<attribute> instead of host.<attribute> due to this open issue: https://github.com/theforeman/foreman_hooks/issues/46
system_mac=$(hook_data host.host.mac)
system_pxe_loader=$(hook_data host.host.pxe_loader)
system_operatingsystem_name=$(hook_data host.host.operatingsystem_name)

echo "$(date): received ${event} on ${object}"
echo "${system_name} ${system_mac} ${system_operatingsystem_name} ${system_pxe_loader}"
if [[ $system_operatingsystem_name == ESXi* ]] && [[ $system_pxe_loader == "mboot UEFI" ]]; then
    echo "ESXi UEFI mode detected"

    # Create MAC Address directory under tftproot after substituting : with -
    macdir="01-${system_mac//:/-}"
    mkdir -p /var/lib/tftpboot/${macdir}

    # Copy the boot.cfg file from the ISO mountpoint to tftproot and edit it to add a prefix to the ISO mountpoint.
    cp boot/ESXi-6.7.0-8169922/boot.cfg $macdir/
    # The mountpoint can also be parsed from $system_operatingsystem_name. Useful for deploying multiple ESXi builds.
    sed -e "s#/##g" -e "3s#^#prefix=/boot/ESXi-6.7.0-8169922\n#" -i $macdir/boot.cfg
fi
# exit code is important on orchestration tasks
exit 0
EOT

chmod u+x 01-prep-esxi-uefimode.sh

# Make it accessible to Foreman
cd /usr/share/foreman/config/hooks
chown -R foreman:foreman host/

# Register the hook script in Foreman by restarting Apache
service apache2 restart

Important Note: after_build hook scripts are not run on newly created hosts. These are only run when build mode is enabled on an existing host.

3.4: Now we are ready to deploy the host. Edit the host entry in Foreman and set the following:

Operating System : ESXi 6.7 Build 8169922
Media : ESXi Dummy
Partition table: Kickstart default
PXE Loader : mboot UEFI
3.5: Press ‘Resolve’ beside Provisioning templates and you should see the ESXi PXELinux and Kickstart templates.

Note: The PXELinux template is irrelevant as it does not play any role in UEFI mode. So we are keeping it as is.
3.6: Press ‘Build’, change the host’s boot mode to UEFI and reboot the host to PXE. You should see the ESXi installer load up.

Once the installation is complete and the host reboots, it will load mboot.efi from the DHCP Server again and the installation will restart. However since the host is not in build mode, it will not be able to fetch the kickstart file from Foreman and will throw an error and freeze at the installer screen.
To solve this we have to manually change the host PXE Loader to None in Foreman.
OR we can use Foreman-Hooks to do it automatically!

3.7: Using Foreman-Hooks to solve host not able to perform local boot from PXE

Once the host informs Foreman that build is completed, we will run a script that checks the host’s OS and PXELoader and changes the PXELoader to None. For this we need to hook into the before_provision event.

cd /usr/share/foreman/config/hooks

mkdir -p host/managed/before_provision

cd host/managed/before_provision

# Symlink the hook_functions file.
ln -s /usr/share/foreman/vendor/ruby/2.3.0/gems/foreman_hooks-0.3.14/examples/bash/hook_functions.sh .

# Now create the script
cat  <<EOT >> 01-esxi-unset-pxeloaders.sh
#!/bin/bash
. $(dirname $0)/hook_functions.sh
event=${HOOK_EVENT}
object=${HOOK_OBJECT}

exec >> /tmp/${event}.log
exec 2>&1

system_name=$(hook_data host.host.name)
system_mac=$(hook_data host.host.mac)
system_id=$(hook_data host.host.id)
system_pxe_loader=$(hook_data host.host.pxe_loader)
system_operatingsystem_name=$(hook_data host.host.operatingsystem_name)

echo "$(date): received ${event} on ${object}"
echo "${system_name} ${system_mac} ${system_operatingsystem_name} ${system_pxe_loader}"

if [[ $system_operatingsystem_name == ESXi* ]] && [[ $system_pxe_loader == "mboot UEFI" ]]; then
    echo "ESXi in UEFI mode detected. Changing PXE Loader to None"
    hammer -u admin -p foreman host update --id $system_id --pxe-loader 'None'  # Change the credentials
fi

exit 0
EOT

chmod u+x 01-esxi-unset-pxeloaders.sh

# Make it accessible to Foreman
cd /usr/share/foreman/config/hooks
chown -R foreman:foreman host/

# Register the hook script in Foreman by restarting Apache
service apache2 restart

Check production.log to verify if the hook script is loaded.

Now when a host finishes OS installation and sends a build complete call to Foreman, its PXE Loader will be automatically changed to “None”. So when the host boots it will try to load ‘grub2/bootx64.efi’ from the DHCP server. Since this bootloader does not exist (unless you have modified a default installation), it will boot into the next entry which ideally will be the ESXi boot medium.

With these steps you can have a fully automated ESXi deployment through Foreman in both Legacy BIOS and UEFI modes.

Foreman Community Newsletter - July 2018

2018-07-31T15:17:00+00:00

Foreman 1.18.0 released

The next stable release of Foreman (1.18.0) was released on July 19th, with many new features and bug fixes. Everyone is encouraged to schedule their upgrades, especially if you’re still on the (now unmaintained) 1.16 series.

As ever, many thanks to the contributors to this release - coders, RC testers, documentation writers, and bug/feature reporters. Foreman wouldn’t be what it is without you! If you do find issues with, please do report them!

Katello 3.7 released

Hard on the heels of the 1.18 release of Foreman core, the Katello team brought you version 3.7 on Jul 27th, with far too many features and bug fixes to list. If you’re upgrading a Katello install, do be sure to read the upgrade notes.

Huge thanks to all the contributors to this release - as with core, we couldn’t put out these releases without you! Further issues with Katello should be reported, as always.

1.19 RC1, Katello 3.8 RC1, and test week event

Since we’re trying to make up time for the slow releases over the last year or so, the next releases of Foreman and Katello are already being prepared. Release candidates for 1.19 and 3.8 are already available, and as always we’d like you help to test them!

To give a little structure to this, we’ve again organised a test week event during which we’d like to see as many of our end-user scenarios tested as possible. If any of these fit your workflow, and you’re willing to give the RC a spin, please do! Be sure to let us know how it went and of course report any bugs that you find. Thanks!

Foreman’s 9th Birthday

July is our birthday month, this year we are 9! Everyone who’s a part of this lovely community deserves a $beverage-of-choice, we wouldn’t be here today without you.

To celebrate, we teamed up with our friends at Atix for a 9th birthday party hackday. You can read all about the good times over on their blog

6 months of Discourse

The start of this month also marked a full 6 months since our migration to Discourse. It’s had a noticeable effect on both the user and dev communities, and in a good way. To mark the occasion, I wrote a short blog post analysing some of the data Discourse generates, looking at trends in community growth and amount of discussion. Check it out

Redmine upgrade

The Redmine instance completed its upgrade path last week, we are now fully up to date. Thanks to everyone for their patience during the chaos that caused.

RFCs needing attention

Here’s this month’s selection of open discussion areas:

Upcoming Events

Future community demos will now be created much further in advance, so you should be able to keep up with upcoming demos there.

Community Demo #48

All the usual goodness from the community, rounded up and presented for your viewing pleasure.

DevConf India 2018

Foreman Project will have a huge appearance at DevConf India ’18. We have two talks lined up on the very first day of the conference, as well as a booth at the conference, so drop by and say hello! We’ll have very exciting goodies and t-shirts at the Foreman booth, make sure you grab them before they are gone :slight_smile:

Past Events

Community Demo #47

Subscriptions, Jenkins, live images, and an awesome new reporting engine from Marek. Do watch!

Plugin news

New plugins:

hammer_cli_foreman_ansible

Dropped plugins (unmaintained, no longer working):

ABRT
Azure

Updated plugins:

hammer_cli_katello updated to 0.13.4
hammer_cli_foreman updated to 0.13.1
foreman_discovery updated to 12.0.2
foreman_salt updated to 10.1.0
foreman_templatesto 6.0.3
foreman_remote_execution updated to 1.5.4
foreman_openscap updated to 0.10.2
foreman_ansible updated to 2.2.5
foreman_maintain updated to 0.2.5
smart_proxy_ansible updated to 2.0.3

Acknowledgements

Recap: The Foreman Birthday Party @ATIX

2018-07-31T14:25:45+00:00

This year it was our pleasure to host Foreman’s 9th birthday party.

Once upon a time …

On the 13th of July in 2009 Ohad Levy gave birth to a new project which was later to be named “The Foreman”

Nine years later The Foreman is grown up and famous for managing and automating a huge number of IT infrastructures with several thousands of servers. Today, Foreman is also a major part of enterprise system management solutions like Red Hat Satellite 6 and orcharhino.

We’re celebrating

The party started at 10 am with birthday presents for all party guests. Everybody got t-shirts, stickers and foreman party helmets. After a short introduction we started our hack and try sessions. Powered by mate–tea and cola, three groups worked very successfully on foreman-ansible, foreman-probing and chat integration for foreman messaging. Only interrupted by a very short lunch break the sessions went until 14:30 when the „official“ part started.

This time we could listen to very interesting talks: Bernhard and Markus from ATIX showed us the possibilities of securing software repositories with gpg keys. Dirk Götz from NETWAYS gave a very good overview and demo about his favorite foreman plugins. Foreman Community-Developer Adam presented the latest enhancements of the remote execution plugin and Ansible integration.

What’s a birthday party without a cake? Nothing!

The Foreman turned nine – so we had a big birthday cake with nine candles! After the Happy birthday song, Adam was chosen to blow out the candles and make some well wishes for the birthday boy.

The party went on with interesting discussions over pizza, beer.

The end…

All guest enjoyed the party and working together with other community members. New friendships were made and everybody is living happily ever after :)

Discourse, 6 months on: Impact Assessment

2018-07-26T00:00:00+00:00

It’s been 6 months since the Foreman community switched to Discourse, so I wanted to revisit some of the graphs I drew in November (when I was justifying the migration) and see if its had a positive impact. Spoiler alert! It has :)

Back in November, I made a big deal about not only how our mailing list was failing, but also about how a more modern medium could help us achieve our goals better. Six months in, it seemed appropriate to try and evaluate a few of those things. We’ll take a look at overall post count, active user count, and the interaction between the two.

The data

Thanks to importing all the mailing list posts into Discourse, we have post/topic/user data going right back to the creation of the mailing list some 8 years ago. Discourse itself generates far more metrics we could look at, but since we want to compare to the old lists, we’ll have to restrict ourselves to these.

Since we want to compare like to like, we’ll also only use the first 6 months of 2017. Obviously we have all of 2017, but there’s complications in using it. Firstly, it’s a different time period, so seasonal effects may come into play (e.g. it’s always quieter at Christmas), but more importantly, Discourse trials begin in November, which would make interpretation very hard. So, we’ll use 2017-01-01 to 2017-06-30 (and likewise for 2018).

We do have the data per day, but that causes zero-inflation issues - our community is always very quiet on Saturdays, and somewhat quiet on Fridays and Sundays too. To avoid modelling issues with this, we’ll aggregate the data by week - look at this graph to see the difference:

As you can see, grouping by week makes for a more “normally distributed” data set, which is better for modelling (yes, I could use a Poisson GLM or something here, but I don’t think it affects the result).

So, we’ll have 26 data points per year, one for each week, and both the count forum posts for that week, and the number of active users for that week (this isn’t a simple sum, but a form of SELECT DISTINCT on all the post authors for each week).

We can break this down in a few ways - the whole data set, two groups by year, two groups by list (foreman-users and foreman-dev), or a 2x2 matrix of both.

Post and user count

Let’s start by seeing the raw post count by week for each of the lists. This is simply the forum_posts column, one graph per channel, and with the two year’s data overlaid:

The lines are a linear fit, the shaded area is the uncertainty in that fit.

We can see that in both cases, the 2017 data shows the mailing lists were not doing well - dev was growing slowly but users was declining. 2018 looks much better! We can also look at this from an averages view:

data %>% group_by(year,buffer) %>% summarise(mean_posts_per_day = mean(forum_posts)) %>% kable()

year	buffer	mean posts per day
2017	theforeman	8.311111
2018	theforeman	17.928177
2017	theforeman-dev	4.153846
2018	theforeman-dev	9.213483

In both cases, the mean posts per day has more than doubled, which is a big deal, I think. However, there’s a catch….

We also talked about attracting a wider community as part of our move, so let’s look at this data and include the number of users too. You might well expect to see the number of posts grow if the number of users is growing. Here’s the same plot, but now we’ll make the “point size” relative to the number of active users that week:

I’ve just plotted 2018 here, for clarity, but 2017 is similar. So, yes! We do see a rise in the number of users - this is great news! We wanted to widen the community, and we seem to be achieving that. We can actually test that, for foreman-users:

t.test(d_18_usr$forum_users, d_17_usr$forum_users, paired = F, var.equal = F)

t = 6.6282, df = 49.737, p-value = 2.344e-08
alternative hypothesis: true difference in means is not equal to 0
95 percent confidence interval:
  5.709454 10.675162
sample estimates:
mean of x mean of y
 21.42308  13.23077

The things to focus on here are:

The averages (means) at the end - 2018 has nearly double the mean number of post authors
The confidence interval for the difference of means
- it doesn’t contain 0, so it’s very unlikely that this shift is due to chance

We can do the same for the foreman-dev:

t.test(d_18_dev$forum_users, d_17_dev$forum_users, paired = F, var.equal = F)


t = 6.6509, df = 44.602, p-value = 3.5e-08
alternative hypothesis: true difference in means is not equal to 0
95 percent confidence interval:
 10.08103 18.84204
sample estimates:
mean of x mean of y
 43.26923  28.80769

Same results - large difference in mean, and the confidence interval is even further from zero.

Accounting for users

So we know the number of users has increased - what does this mean for the number of posts. Happily, we can use some statistical modelling to account for that. We’ll use a model like this:

forum_posts ~ week_num + forum_users

So we’re saying that posts is some combination of the time (week_num) and the number of authors (forum_users), plus some constant. Using this model, we can try to make a statement about what we would expect the effect of week_num to be while holding the number of users constant. Let’s try that.

Here’s a graph of forum_posts divided by forum_users for each week - that’s not quite the same as the model above, but it gives us a way to view the data. Over the top I’ve laid a simple linear fit to that normalized post count; you can see that for 3 of the graphs, it’s pretty much flat:

However, the Dev 2018 graph is not flat. If we investigate the model more closely there, we can see an interesting result. Here’s the model output:

lm(formula = forum_posts ~ week_num + forum_users, data = d_18_dev)

Coefficients:
            Estimate Std. Error t value Pr(>|t|)
(Intercept) -46.3843    10.7805  -4.303 0.000265 ***
week_num      0.8893     0.2915   3.051 0.005674 **
forum_users   4.5491     0.5196   8.755 8.83e-09 ***

I’ve trimmed the output to just the coefficients. What this tells us is that we expect to see an increase in the number of posts to the Development category over time, even while holding the number of users constant. Specifically, we expect between 0.3 and 1.5 more posts week-on-week, even if the number of users doesn’t change. That’s to a 95% confidence level (i.e. there’s a 5% chance I’m wrong, and this is all by chance).

Conclusions

There’s clear evidence of an increase in posts and number of users since the move to Discourse. I don’t think that’s surprising - I’ve certainly felt the impact, and I believe others have too. I’ve spent much more of my time replying to posts than I used to on the mailing list. However, that the mean number of posts has more than doubled for both categories is a surprise even to me. In terms of raw numbers, that’s quite a turnaround (and less than 1/1,000,000 chance of it being a freak accident and nothing has really changed).

However, it’s the result for the Development category that I really like. One of the arguments I made for Discourse that seems to resonate with people is that it would bring better tools for the developers to discuss the project and come to consensus on how to move forward. While this data says nothing about consensus specifically (we could all be arguing more :P), it does show that we’re all talking to each other more as time goes on. Speaking as a long-time remotee, I like it when I see communication between people increasing :)

Can we attribute this to Discourse? Well, there’s an age-old saying that correlation does not mean causation. However, thanks to Hill’s criteria for causal relationships, we can still draw some conclusions. Most applicable is the fact that no other large event took place in our community that could plausibly cause such a change, and the mechanism by which Discourse could cause a change is also fairly clear. I’m going to go out on a limb and say these two things are linked :)

So, six months in and it’s looking pretty good. How are you finding Discourse?

P.S. If you want to check my working, I’ll happily share the code :)

Monitoring and telemetry of Foreman 1.18

2018-07-20T13:26:15+00:00

Performance Co-Pilot (PCP) is an open source framework and toolkit for monitoring and analyzing live and historical system performance. It provides high-resolution live monitoring from local or remote hosts (with optional auto-discovery) and instrumented software. PCP can be integrated with monitoring solutions like Graphite (Carbon/Whisper), InfluxDB, Zabbix or Nagios, or configured to store historical data into archive files which is a unique feature we want to take advantage from.

PCP follows UNIX tradition by providing a relatively large collection of small utilities which do their job well. There are several APIs available to write collecting agents called PMDAs or to instrument existing applications to gather “telemetry” data. APIs are available mostly in C, Python and Perl out of box with extra wrappers or native libraries in Java, Rust and Golang. PCP is a compact package (under one megabyte) with minimum dependencies and part of standard RHEL7 base repositories.

The article covers PCP installation, basic operating system monitoring configuration with extra monitoring of key processes and PostgreSQL and Apache httpd. It also shows how to setup Foreman Ruby on Rails monitoring to read internal metrics from Foreman 1.18 core application. Major areas covered:

basic system monitoring (load, memory, IO)
hot processes (memory and cpu utilization)
Apache httpd monitoring
PostgreSQL monitoring
Foreman Rails application instrumentation
all relevant metrics archival for retrospective troubleshooting

Second part of the article covers analysis of results using PCP CLI and web UI tools and retrospective analysis.

If you’d like to watch a video of this process, you can view that on YouTube

The article assumes a Red Hat based distro (Centos etc), version 7.6+. It will likely work for older versions, or other distros such as Debian, with appropriate changes to the paths (e.g. where the Apache config files are located, etc.). Due to a known bug PCP 4.1+ is recommended.

The following packages needs to be installed and enabled:

yum -y install pcp pcp-pmda-apache pcp-pmda-postgresql

Make sure there is enough space for /var/log/pcp. Our testing showed about 100-500 MB per archive file (per day) but on busy servers this can go up to several gigabytes per day. Default PCP retention policy is to keep 14 days of compressed archives, the article shows how to change this setting below. We recommend at least 20 GB of free space and continuous monitoring for the first few weeks.

By default, PCP collects basic system metrics but not all processes, to get memory information from key processes let’s configure hotproc monitoring which will include CPU time spent, RSS, swap and virtual memory, file descriptors opened and bytes read and written:

cat >/var/lib/pcp/pmdas/proc/hotproc.conf <<EOF
#pmdahotproc
Version 1.0

fname == "java" ||
fname ~ /(qdrouterd|qpidd)/ ||
(fname == "postgres" && psargs ~ /-D/) ||
fname == "mongod" ||
fname ~ /^dynflow/ ||
psargs ~ /Passenger RackApp/ ||
fname ~ /^wsgi:pulp/ ||
psargs ~ /celery (beat|worker)/ ||
psargs ~ /pulp_streamer/ ||
psargs ~ /smart-proxy/ ||
psargs ~ /squid.conf/
EOF

In the next step, we will make sure those metrics also gets logged into archive files.

mkdir -p /var/lib/pcp/config/pmlogconf/foreman-hotproc
cat >/var/lib/pcp/config/pmlogconf/foreman-hotproc/summary << EOF
#pmlogconf-setup 2.0
ident   foreman hotproc metrics
probe   hotproc.control.config != "" ? include : exclude
        hotproc.psinfo.psargs
        hotproc.psinfo.cnswap
        hotproc.psinfo.nswap
        hotproc.psinfo.rss
        hotproc.psinfo.vsize
        hotproc.psinfo.cstime
        hotproc.psinfo.cutime
        hotproc.psinfo.stime
        hotproc.psinfo.utime
        hotproc.io.write_bytes
        hotproc.io.read_bytes
        hotproc.schedstat.cpu_time
        hotproc.fd.count
EOF

Enable proc agent to collection more per-process details.

cd /var/lib/pcp/pmdas/proc
./Install

The above configuration only stores interesting process data, it does not archive any details from Apache or PostgreSQL agents. Use pminfo to list all available metrics and add those to the summary configuration file when needed.

Apache agent collects basic information from Apache, but it needs the mod_status plugin to be enabled in Apache (which requires restart of the service). PostgreSQL agent gets basic information about SQL server health, it does not need any restart to be enabled.

cd /var/lib/pcp/pmdas/apache
./Install
cd /var/lib/pcp/pmdas/postgresql
./Install

For Apache PMDA to work, extended status Apache module must be loaded. Only perform this if you installed the apache PMDA above:

cat >/etc/httpd/conf.d/01-status.conf <<EOF
ExtendedStatus On
LoadModule status_module modules/mod_status.so

<Location "/server-status">
PassengerEnabled off
SetHandler server-status
Order deny,allow
Deny from all
Allow from localhost
</Location>
EOF

Foreman installer (Puppet) will delete this file, to prevent this from happening add the following line into /etc/foreman-installer/custom-hiera.yaml:

apache::purge_configs: false

Restart of Apache is needed for this, again this can be skipped if Apache PMDA is not being installed.

systemctl restart httpd pmcd pmlogger

Foreman Rails application provides telemetry data via two endpoints - prometheus and statsd. The former one can be only used in single process environment at the moment, in our case we will configure Rails workers to send metrics via statsd protocol into pcp-mmvstatsd daemon which will provide aggregation and export into PCP MMV API. Monitoring will work the following way:

pmcd -> pmda-mmv -> memory mapped file <- pcp-mmvstatsd <- PassengerWorker(s)

Install required packages:

yum -y install foreman-telemetry pcp-mmvstatsd

Enable telemetry support in Foreman settings file /etc/foreman/settings.yaml:

:telemetry:
  :prefix: 'fm_rails'
  :statsd:
    :enabled: true
    :host: '127.0.0.1:8125'
    :protocol: 'statsd'
  :prometheus:
    :enabled: false
  :logger:
    :enabled: false
    :level: 'INFO'

Keep other exporters (prometheus, logger) disabled and note the port is 8125. Do not use host names in the host setting, make sure it’s set to localhost via IP address 127.0.0.1 for optimal performance. Starting from Foreman 1.19, foreman-installer is getting puppet parameters and CLI options for setting up all options in the settings.yaml file.

Enable and start the statsd aggregation service:

systemctl enable pcp-mmvstatsd pmcd pmlogger
systemctl start pcp-mmvstatsd

Apache httpd (Passenger) may now be restarted to reload Foreman new configuration - at this point, telemetry data is started to be collected.

systemctl restart httpd

Note the pcp-mmvstatsd daemon listens on all interfaces by default, this can be further hardened via /etc/default/pcp-mmvstatsd or via firewall.

Dynamic metrics won’t appear automatically in archive files until pmlogger is told to start logging them. This can be performed daily, it’s a fast operation. We recommend to run this on a daily basis:

cat >/etc/cron.daily/refresh_mmv <<EOF
#!/bin/bash
echo "log mandatory on 1 minute mmv" | /usr/bin/pmlc -P
EOF
chmod +x /etc/cron.daily/refresh_mmv

User interface and exporting options

Metrics can now be listed via pminfo command. Along with other system metrics like CPU, memory, kernel, xfs, disk or network, there are the following metrics which were configured above:

hotproc.* - basic metrics of key processes configured above
apache.* - apache httpd server metrics (if configured)
postgresql.* - basic postgresql statistics (if configured)
mmv.fm_rails_* - Foreman Rails metrics

Optionally, to get lists of metrics reported from Rails with descriptions:

foreman-rake telemetry:metrics

Resulting table markdown-formatted, here is the result:

Metric name	Labels	Type	Description
fm_rails_activerecord_instances	class	counter	Number of instances of ActiveRecord models
fm_rails_bruteforce_locked_ui_logins		counter	Number of blocked logins via bruteforce protection
fm_rails_failed_ui_logins		counter	Number of failed logins in total
fm_rails_http_request_db_duration	controller,action	histogram	Time spent in database for a request
fm_rails_http_request_total_duration	controller,action	histogram	Total duration of controller action
fm_rails_http_request_view_duration	controller,action	histogram	Time spent in view for a request
fm_rails_http_requests	controller,action	counter	A counter of HTTP requests made
fm_rails_importer_facts_count_input	type	counter	Number of facts before imports starts per importer type
fm_rails_importer_facts_count_interfaces	type	counter	Number of changed interfaces per importer type
fm_rails_importer_facts_count_processed	type,action	counter	Number of facts processed per importer type
fm_rails_importer_facts_import_duration	type	histogram	Duration of fact import (ms) per importer type
fm_rails_importer_facts_populate_duration	type	histogram	Duration of fields population (ms) per importer type
fm_rails_proxy_api_duration	method	histogram	Time spent waiting for Proxy (ms)
fm_rails_proxy_api_response_code	code	counter	Number of Proxy API responses per HTTP code
fm_rails_ruby_gc_allocated_objects	controller,action	counter	Ruby GC statistics per req (total_allocated_objects)
fm_rails_ruby_gc_count	controller,action	counter	Ruby GC statistics per request (count)
fm_rails_ruby_gc_freed_objects	controller,action	counter	Ruby GC statistics per request (total_freed_objects)
fm_rails_ruby_gc_major_count	controller,action	counter	Ruby GC statistics per request (major_gc_count)
fm_rails_ruby_gc_minor_count	controller,action	counter	Ruby GC statistics per request (minor_gc_count)
fm_rails_successful_ui_logins		counter	Number of successful logins in total

There are three types of metrics reported by Rails (this only applies to mmv.* metrics):

counter - increasing counter, PCP converts this to rate (hits per second)
gauge - value observation
duration - approximation of time spent using HDR histogram algorithm with instances: mean, min, max, variance and standard_deviation

Use PCP tools to monitor deployment live or offline via recorded archive files. A couple of examples now follow. To see instance creation rate of ActiveRecord object User at rate of instancer per second (counter metric):

pmval mmv.fm_rails_activerecord_instances.User

To see minimum, maximum and average (mean) total duration of dashboard controller index action (duration metric):

pmval mmv.fm_rails_http_request_total_duration.dashboard_controller.index

To access remotely, a port needs to be opened:

firewall-cmd --add-port=44321/tcp
firewall-cmd --permanent --add-port=44321/tcp

Then connect remotely (no authorization by default - SSL or password can be set):

pminfo -h remote.host.example.com

There are several existing web apps available for PCP to monitor live or archived data using standard web browser: Grafana and Vector. Although PCP core application is in base repository, the web applications are in “optional” repository:

subscription-manager repos --enable rhel-7-server-optional-rpms
yum -y install pcp-webapi pcp-webapp-grafana pcp-webapp-vector
systemctl start pmwebd
systemctl enable pmwebd
firewall-cmd --add-port=44323/tcp
firewall-cmd --permanent --add-port=44323/tcp

To access the applications, visit the following URLs:

http://foreman:44323/grafana
http://foreman:44323/vector

All three applications are static JavaScript only apps accessing PCP archival data via a web API - it will only show metrics which are configured to be archived via pmlogger. Beware, Foreman sets HTST flag which forces browsers to HTTPS which is not supported by the built in PCP web server. To workaround that, use different browser or make a DNS alias in order to access those web apps.

Vector app connect to PCP daemon directly and only shows live data while Grafana app reads archive files and shows historical data up to one year back. For this reason, there is a slight lag of several minutes until values appear in Grafana. Dynamic metrics (MMV) will not appear in archives until refresh_mmv cron job mentioned above is executed.

PCP data can be exported into external Graphite, InfluxDB or Zabbix applications, consult RHEL and PCP.IO documentation on how to configure that (packages named pcp-export-pcp2graphite, pcp-export-pcp2influxdb and pcp-export-zabbix-agent).

There is a powerful engine called pmie for taking actions on various events (e.g. process is utilizing CPU or memory too much). This can be used to send e-mails or trigger events (e.g. restart process). Read more about it in the PCP documentation (man pmie).

Collector daemon and all agents are logging into individual files, check those for errors. Note these log files are not archives of metric data, these are text files where you can find possible configuration errors or warnings:

tail /var/log/pcp/pmcd/*.log

Logging daemon also has its own log file:

less /var/log/pcp/pmlogger/$(hostname)/pmlogger.log

Let’s summarize all key daemons which were configured so far:

systemctl status pmcd pmlogger pcp-mmvstatsd pmwebd

These are:

pmcd - collector daemon
pmlogger - archiving daemon
pcp-mmvstatsd - statsd to MMV PCP aggregator
pmwebd - web API with Grafana/Vector (optional)

The rest of the article introduces PCP and its tools in more detail.

Introduction to PCP and its tools

By default, the PCP pmcd daemon collects a reasonable number of system metrics (load, memory, processes, I/O, XFS). Other collector agents (also called PMDAs) can be installed as separate packages. The default probing rate is one second for live monitoring, but when no client utility is connected pmcd idles in the background and consumes effectively no resources.

When pmlogger daemon is running, the most important metrics are being archived into /var/log/pcp directory, for each day new archive file is created and old files are rotated. One day archive is roughly 25 MB in size (uncompressed data) with most metrics being sampled at one minute interval by default. By default PCP compresses the data and rotates archives every two weeks, highly loaded servers may consume up to 10 GB of historical data.

To display PCP version, list of enabled agents and active logging archive:

pcp

To display all currently available metrics from the current host:

pminfo

All metrics are stored in a tree-like structure, to view available metrics under specific subtree, let’s say all network related information, provide node on the command line:

pminfo network

Each metric can be either a single value or a list of values called instances. Kernel load is typical example of metric with three instances (1, 5 and 15 minute average load). To display detailed information about a metric including its types and instances:

pminfo -dfmtT kernel.all.load

Unique feature of PCP is metric metadata, extra information which is available for every metric. This includes expected things like description or data type, but also units and dimensions. This metadata allows one to correctly display data or create correct graphs combining various metrics together. For example, network interface speed has a unit of megabytes per second:

pminfo -dfmtT network.interface.speed

A counter is monotonic increasing or decreasing integer. Thanks to the metadata, PCP can even convert counter values into rates. To monitor live disk write operations per partition using fixed point notation as a rate (operations per second):

pmval -f 1 disk.partitions.write

To see raw counter values instead of rates:

pmval -r disk.partitions.write

Process data is captured by the proc PMDA (installed and enabled by default). Process ID is the instance name in this case. To monitor the number of open file descriptors of the process with PID 1 every two seconds:

pmval -t 2sec 'proc.fd.count[1]'

Let’s take a look on other utilities from the additional package called pcp-system-tools. Monitor system metrics in a top-like window:

yum -y install pcp-system-tools
pcp atop

Monitor CPU-related system metrics and memory in a sar-like (System Activity Report) manner:

pcp atopsar -cm

Monitor system metrics in a sar-like fashion with two second interval:

pmstat -t 2sec

Monitor system metrics in an iostat-like fashion with two second interval:

pmiostat -t 2sec

Most of these tools operate on archives too, which is a unique feature. It is so easy to “go back in time” and inspect a system with pmstat-like or iostat-like experience easily. The key argument to remember is –archive or -a:

pmstat -a /var/log/pcp/pmlogger/$(hostname)/20180503.08.58
pcp atop --archive /var/log/pcp/pmlogger/$(hostname)/20180503.08.58

Another interesting feature leveraged for monitoring of Foreman servers is “hotproc”, or hot processes monitoring. In short, it is possible to set up rules to start monitoring some processes in greater detail. The conditions can be as simple as “all java applications” or “all processes consuming more than 2 GB of memory”. There is a configuration file to store these rules, but it is also possible to enable these on the fly without restarting the PCP daemon:

pmstore hotproc.control.config 'fname == "java"'
pminfo -f hotproc

Hot process monitoring and logging into archives are flexible features of PCP. An example configuration below starts watching of all java applications running on the server and storing one key metric in archives (RSS memory):

cat >/var/lib/pcp/pmdas/proc/hotproc.conf <EOF
#pmdahotproc
Version 1.0
fname == "java"
EOF

mkdir -p /var/lib/pcp/config/pmlogconf/my-hotproc
cat >/var/lib/pcp/config/pmlogconf/my-hotproc/summary << EOF
#pmlogconf-setup 2.0
ident   my-hotproc metrics
probe   hotproc.control.config != "" ? include : exclude
        hotproc.psinfo.rss
EOF

systemctl restart pmcd pmlogger

To see a list of currently archived metrics, do:

less /var/log/pcp/pmlogger/$(hostname)/pmlogger.log

The log file also contains estimation on size, in the example above it is around 13 megabytes per day of uncompressed data. To identify currently active archive file, simply start command “pcp” which also shows other interesting information like number of CPUs, memory, pcp version and timezone.

There is a low-level control utility for pmlogger that can interactively query current status or change what should be logged:

echo status | pmlc -P
echo "query mmv" | pmlc -P

Logging interval, retention and compression

The default logging (sampling) interval is 60 seconds. This interval was chosen as a balance between logging volume and the level of detail needed for long term monitoring. To change that setting, edit /etc/pcp/pmlogger/control.d/local and append -t XXs to the line for LOCALHOSTNAME, and restart the pmlogger service.

The default retention policy is to keep archives for the last 14 days, compressing archives older than one day with xz. To change these settings, edit /etc/cron.d/pcp-pmlogger file and use -x and -k options. For example, “-x 4 -k 7” options on the first line will compress archives after 4 days and delete after 7 days.

Retrospective Performance Analysis

PCP archive logs are located under /var/log/pcp/pmlogger/$(hostname) and they can be easily transferred for offline inspection. A simple utility which is able to dump raw data from archives is available. To check the host and the time period an archive covers:

AVE=/var/log/pcp/pmlogger/$(hostname)/20170918.13.59
pmdumplog -l $AVE

Check PCP configuration at the time when an archive was created:

pcp -a $AVE

Display all enabled performance metrics at the time when an archive was created:

pminfo -a $AVE

Display detailed information about a performance metric at the time when an archive was created:

pminfo -a $AVE -df mem.freemem

Dump past disk write operations per partition in an archive using fixed point notation:

pmval -a $AVE -f1 disk.partitions.write

Dump RSS memory consumed by two particular processes configured via hotproc above (the PIDs will be likely different):

pmval -a $AVE hotproc.psinfo.rss -i 000768 -i 000977

Replay past disk write operations per partition in an archive with two second interval using fixed point notation between 14:00 and 14:15:

pmval -a $AVE -d -t 2sec -f 3 disk.partitions.write -S @14:00 -T @14:15

Print subnet controller index action average total processing time with number of instances created for Subnet and Setting models and total number of Ruby allocation objects for the same controller and action (all on one line)

pmrep mmv.statsd.fm_rails_http_request_total_duration.subnets_controller.index mmv.statsd.fm_rails_activerecord_instances.Subnet mmv.statsd.fm_rails_activerecord_instances.Setting mmv.statsd.fm_rails_ruby_gc_allocated_objects.subnets_controller.index

Calculate average values of performance metrics in a time interval using tabular formatting including the time of minimum/maximum value and the actual minimum/maximum value:

pmlogsummary -HlfiImM -S @14:00 -T @14:30 $AVE disk.partitions.write mem.freemem

Replay past system metrics in an archive in a top-like window starting at 2 PM:

pcp -a $AVE -S @14:00 atop

Most of the PCP utilities supports -a (–archive) option, including pmstat, pmiostat or pmchart, an interactive GUI application written in QT with plotting capabilities. A REST API exposing live or historical data can be installed for integration with other applications.

The utility atop also supports hot processes via an explicit option which is very handy back-in-time top-like experience:

pcp -a $AVE -S @14:00 atop --hotproc

One important command can deliver quick calculation of average, min and max values:

pmlogsummary archive_name

The pmdiff utility uses pmlogsummary to compare archives or time windows of interest, and reports those metric whose values have changed the most in the times being compared.

PCP ships with a powerful GUI utility for plotting live or offline graphs called pmchart. It works under Linux and MacOS.

We have just scratched the surface, PCP offers derived metrics via simple arithmetics, PMIE (metrics inference engine) for alerting (sending emails or executing arbitrary commands), application instrumentation and tracing APIs. For more information, use the official RHEL or PCP documentation.

https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/7/html/performance_tuning_guide/sect-red_hat_enterprise_linux-performance_tuning_guide-performance_monitoring_tools-performance_co_pilot_pcp
https://access.redhat.com/articles/1145953
http://pcp.io/docs/guide.html
https://access.redhat.com/articles/2450251
https://access.redhat.com/videos/898313

Contributing to the Katello API

2018-07-03T12:00:00+00:00

Introduction

If you have used the Katello API a lot, you have almost certainly found areas where the API is difficult to use. Fortunately, it is easy to add a new API call, or add features to an existing call.

This tutorial will show how to add a new API, and how to add a new asynchrnous task. Adding an asynchronous task is not always needed, but is a common requirement.

You’ll need to write tests for any code you commit, which is outside the scope of this document.

Initial setup

You’ll need to run through a forklift development setup. Instructions are located in the forklift README. You’ll be running foreman start in one terminal window, and making edits using another terminal window.

Making an API call

The first order of business is to make an existing API call in your development environment. This will ensure that your development environment is working, and it will show an example of how to make API calls with cURL.

To make the call, simply run curl -s -k https://admin:changeme@localhost/katello/api/v2/ping | python -mjson.tool. You should get this back:

{
    "services": {
        "candlepin": {
            "duration_ms": "36",
            "status": "ok"
        },
        "candlepin_auth": {
            "duration_ms": "38",
            "status": "ok"
        },
        "foreman_tasks": {
            "duration_ms": "862",
            "status": "ok"
        },
        "pulp": {
            "duration_ms": "141",
            "status": "ok"
        },
        "pulp_auth": {
            "duration_ms": "124",
            "status": "ok"
        }
    },
    "status": "ok"
}

The python -mjson.tool command simply reformats the JSON output into a more human-readable form.

Adding code to an existing API call

Now that we know everything is working, we can modify an existing call. If you are new at this, it’s good to work through everything one step at a time. This makes it easy to figure out what happened if something is broken. Let’s add a logging statement, and then see if it worked.

In the Katello project, edit ./app/controllers/katello/api/v2/ping_controller.rb and add a logging statement like so:

    api :GET, "/ping", N_("Shows status of system and it's subcomponents")
    description N_("This service is only available for authenticated users")
    def index
      Rails.logger.warn("ping!") # this is the line we added
      respond_for_show :resource => Ping.ping
    end

After you make this change, you’ll need to hit ^C on your rails development server and reload it for the change to take effect.

After the restart is complete, run the same curl command again. You’ll see this in the development server output:

18:26:45 rails.1   | 2018-07-03T18:26:45 [W|app|9a453] ping!

Adding a new API call

We were able to add some example code to an existing API call. Let’s now add a new API call in the same controller.

NOTE: This example is a bit different since we are adding a URL to the top-level, as a peer to “/ping”. You’ll usually want to add something underneath your controller’s top level URL, like “/ping/example” instead of “/example”.

Pull up ./app/controllers/katello/api/v2/ping_controller.rb again, and add this. It is copied from the index method, and we simply changed some references from “index” to “example”:

    api :GET, "/example", N_("An example API call")
    description N_("an example API call"
    def example
      Rails.logger.warn("example!")
      respond_for_show :resource => Ping.ping
    end

Save the file, restart your dev server, and run curl -s -k https://admin:changeme@localhost/katello/api/v2/example. Surprise! You got a 404. We need to also tell Katello about a new route for your new API call.

Edit config/routes/api/v2.rb and make a change like so:

        api_resources :ping, :only => [:index]
        match "/status" => "ping#server_status", :via => :get
        match "/example" => "ping#example", :via => :get # this is the line we added

Now, restart your development server once more, run your curl command and it should return some json, along with printing the message example! in the log file. Congrats!

Creating a task

You can perform work either directly in Katello’s API code which will return synchronously, or via an asynchronous task. This section describes how to make a task and have it execute when your API is called.

NOTE: If you are unsure if you need to create a task or not, feel free to ask on the Community Forums.

Executing an existing task via rake console

Let’s kick off a task via the rake console. Just run rails console from the ~/foreman directory on your development environment.

First, let’s try to find a host that we can reference later. Try running ::Host::Managed.first. It will either raise an error, or return something like this:

=> #<Host::Managed id: 1, name: "centos7-devel.example.com", last_compile: "2018-06-22 12:57:44", last_report: nil, updated_at: "2018-06-22 12:57:50", created_at: "2018-06-15 14:45:39", root_pass: nil, architecture_id: 1, operatingsystem_id: nil, environment_id: nil, ptable_id: nil, medium_id: nil, build: false, comment: nil, disk: nil, installed_at: nil, model_id: 1, hostgroup_id: nil, owner_id: 4, owner_type: "User", enabled: true, puppet_ca_proxy_id: nil, managed: false, use_image: nil, image_file: nil, uuid: nil, compute_resource_id: nil, puppet_proxy_id: nil, certname: nil, image_id: nil, organization_id: 1, location_id: 2, type: "Host::Managed", otp: nil, realm_id: nil, compute_profile_id: nil, provision_method: nil, grub_pass: "", content_view_id: nil, lifecycle_environment_id: nil, global_status: 1, lookup_value_matcher: "fqdn=centos7-devel.example.com", pxe_loader: nil>

If you get an error, it probably means that you simply need to register a host. Once a host is registered, it should work.

Now, let’s kick off a task in the console:

ForemanTasks.async_task(::Actions::Katello::Host::GenerateApplicability, [::Host::Managed.first], false)

You should see a lot of output. If you go into the “tasks” page in the web UI, you’ll see that the task executed. Very cool!

Adding your own task

Now that we know how to execute tasks, we can create our own and try to run it. Create a file in app/lib/actions/katello/content_view/example.rb that looks something like this:

    module Actions
      module Katello
        module ContentView
          class Example < Actions::Base
            def plan
              Rails.logger.warn("an example!")
              # without this line, the task will not go into 'run' phase after planning.
              plan_self
            end
     
            def humanized_name
              _("Example")
            end
          end
        end
      end
    end

Then, try this from the console:

ForemanTasks.async_task(::Actions::Katello::ContentView::Example)

If all went well, you should see your task get executed.

Putting it all together

Now that you can create an API call, create a task and execute a task, you can put it all together into an API call that executes a task. This is left as an exercise to the reader, but one way to write the API method would be like this:

    def example
      Rails.logger.warn("example!")
      task = async_task(::Actions::Katello::ContentView::Example)
      respond_for_async :resource => task
    end

Try using your curl call on this, and the task should fire. Congratulations!

Foreman Community Newsletter - June 2018

2018-06-28T10:17:00+00:00

Foreman 1.18 RC1 and RC2 (and test week event) -> RC3 next week

The next major release for Foreman is undergoing final testing, and the first release candidate came out on June 12th, with a ton of fixes and updates. We then held a successful 1.18 test week (kindly organised by Lukas) which aimed to cover a lot of the common scenarios for RC testing. This was largely successful, and many bugs were logged and looked at. Thanks to everyone for helping out!

We’ve just released 1.18 RC2 for the next round of testing, so give it a spin and let us know how you get on - and do report issues as you find them! We couldn’t do this without the testing we get from the community, so thanks very much to you all!

Late-breaking news! I’m told that RC3 will be out early next week, so you may want to hold off a few days!

Katello 3.7 RC1

To go with 1.18 RC1/2, there is also an RC1 release of the next Katello version. Again, lots of bug fixes and new features are in this release, so please do check it out, and report issues if you find any. Thanks!

New Release Manager

It’s no secret that build stability has been an issue for some time in the nightlies, and that’s had a knock-on effect on release schedules, etc.

To try to address this, Tomer Brisker has accepted the role of Release Manager, and will be actively working to improve the state of the build pipeline. You can read all about his new role here

Tomer’s long-time experience in the Foreman project, as his attention to detail, is a welcome addition to this area of the project - I’m sure he’ll have an impact. Thanks Tomer!

Survey analysis

Earlier in the year we asked you all to fill in our survey, as we do every year. I’ve had chance to at least do some descriptive analysis of the raw data, and do some comparisons to the 2017 data. If you’re interested in the results, check them out here

Blog & Forum integration

Staying with the blog, we’ve also enabled comments on the blog again (after a long absence), which now uses our [forum][https://community.theforeman.org/c/blog] to store the comments. As a side effect, you can now read the blog on our forum too, if that’s more your style. Scroll to the bottom of this post to check it out!

Redmine upgrade

In a short while, we’ll start the long-delayed upgrade to our bug tracker. This has blocked for a long time because of the unmaintained plugins we were using, but there is now a plan in place to move past this. Work is scheduled to start soon, so you may see some instability in the bug tracker at times - details will be posted to the forum, of course.

Plugin Maintainers wanted!

Many of our plugins suffer compatibility issues and/or changes in the projects they integrate with over time - when Foreman changes, or (for example) Salt released a new version, plugins can be affected. Sadly, several plugins hit by this have not been maintained, and are now facing being retired from the plugin repository.

If you’re interesting in helping contribute to the community by helping to fix a plugin you rely on, please get in touch! Currently we’d love some help with:

ABRT
Chef
Salt
Compute resources:
- Azure
- GCE
- EC2
- Rackspace

If these plugins matter to you (or you want to contribute to another plugin!), but don’t know where to start, come talk to me and I’ll help you get started. You can also join the [dev discussion][plugins].

RFCs needing attention

I’m going to start highlighting open RFCs that might benefit from more eyeballs. Here’s this month’s selection:

Upcoming Events

Future community demos will now be created much further in advance, so you should be able to keep up with upcoming demos there (currently I’ve created topics out to August :P)o

Foreman’s 9th Birthday @ ATIX

A one-day unconference / hack session in honour of our birthday, hosted by our good friends at ATIX - thanks!

We’d like some more speakers for this, so please get in touch if you want to go along!

CFP closing for Open Source Summit EU, Edinburgh, October

There’s going to be some kind of Foreman presence at OSS-EU this year, but the CFP for presentations closes this week (SUnday 1st) - so if you want to get your entry paid for, get your submissions in right away!

Past Events

OpenSourceCamp.de

Our first ever Foreman-headlined conference was held earlier this month, and was great fun - thanks to our partners at NETWAYS and Graylog for helping to make this event happen. A full day of talks, hacking, and discussion, thoroughly enjoyed by all (so I am told :P)

Community Demo - 31st May

Containers, backups, and oVirt, oh my! A return to force for the demo, with great content from all the presenters.

Community Demo - 21st June

A packed demo, covering the Redmine and Survey results mentioned above, as well as foreman-hooks for content, more Docker support, better state handling for unattended installs (awesome!) and more. One to watch.

UX Dive on the Audits page

Tereza Novotna took us through the proposed wireframes for the old Audits UI, which has been in need of love for some time. We also had time for a solid discussion around filtering vs sorting, and tables vs list views.

Plugin news

Updated plugins:

hammer_cli_foreman_remote_execution updated to 0.1.0
foreman_chef updated to 0.8.0
foreman_tasks updated to 0.13.2
foreman_remote_executionto 1.5.3
foreman_discovery updated to 12.0.1
foreman_docker updated to 4.1.0
foreman_openscap updated to 0.10.1
hammer_cli_katello updated to 0.13.2
foreman_memcache updated to 0.1.1
smart_proxy_monitoring updated to 0.1.2
smart_proxy_omaha updated to 0.0.4
foreman_virt_who_configure updated to 0.2.2

Acknowledgements

Building Ubuntu Using Katello File Repo

2018-06-20T00:00:00+00:00

NOTE: This blog post describes how to use Katello 3.5 to host Ubuntu repos, and is aimed at those wanting to get Apt repo support going without upgrading. If you are using a later version of Katello, it may already support deb packages natively.

Building Ubuntu Using Katello File Repo

I have an offline network and need to build both RPM based systems and DEB based systems. Instead of installing Katello to handle rpms, and something else to handle debs, I set up Katello to handle both.

Get Local Copy of Repo

I created a script to do this for me, with following assumptions:

Using box that has debmirror installed
- Installing this on a Fedora box required me to remove /etc/debmirror.conf for this script to work
Has rsync installed
Media is mounted at /mnt has enough space to copy the repo
The example is downloading xenial, but can be duplicated to do other releases.

/mnt has a copy of file_repogen.rb

#!/bin/bash
 arch=amd64
 section=main,restricted,universe,multiverse,main/debain-installer,universe/debian-installer,multiver/debian-installer,restricted/debian-installer
 release=xenial,xenial-updates
 server=us.archive.ubuntu.com
 inpath=/ubuntu
 outpath=/mnt/ubuntu/xenial

 debmirror --arch=$arch \
 	    --no-source \
 	    --section=$section \
 	    --host=$server \
 	    --dist=$release \
 	    --di-dist=dists \
 	    --di-arch=arches \
 	    --root=$inpath \
 	    --progress \
 	    --ignore-release-gpg \
 	    --no-check-gpg \
 	    --exclude-deb-section=games \
 	    --exclude=sid \
 	    --method=rsync \
 	    $outpath

for i in $(echo $release | sed “s/,/ /g”); do rsync -L –progress –exclude=*i386 -a rsync://$server/ubuntu/dists/$i $outpath/dists/ done /mnt/file_repogen.rb /mnt/ubuntu/xenial

Import into Katello

Your repo has been downloaded and prepped with a pulp_manifest so it’s ready to be imported into Katello.

I created a product called Ubuntu and made a repository for each Ubuntu Distro I was trying to use. In this case, I have a repo called xenial. The type is file and the url is pointed to the media I have made available on my system. Then I start the sync and slowly sync the repo.

Issues: /var/spool/pulp stores a copy of the files as it syncs, which I did not give enough space to since it was not listed in the katello installer documentation. Due to this, it takes many many failed syncs to sync all the files.

Set up the Foreman Info

Update the Ubuntu Mirror Installation media to http://foreman/pulp/isos/Default_Organization-Ubuntu-$release/
Add your operating system to foreman

Build a Host

You should now be able to build a host selecting Ubuntu as an operating system and it will build off your local media.