This plugin is a platform for running, monitoring, debugging and auditing any kind of task within Foreman or its plugins. By task we mean a job, a process, an action or whatever that deserves to have its own name, progress or should simply keep history. Both synchronous and asynchronous tasks are supported.
There are two types of users that use this plugin, first is a developer, who needs a better control of their task flow, second is a regular Foreman user, who wants to know, what’s happening in their Foreman instance. This manual focuses on both groups, keeping the info in separate chapters. We highly recommend developers to start with user guide first.
The plugin is compatible with all platforms supported by Foreman. Since foreman-tasks depends on Dynflow, Ruby 1.9+ is required.
Note that since Foreman 1.14, foreman-tasks became a dependency of the core project and is installed automatically.
You can install the plugin using foreman-installer like this
The installer will install the package and setup the foreman-tasks service which allows background processing of tasks.
In the case you want to install the package manually on an existing Foreman, use the following steps.
Install the packages appropriate for your operating system:
yum install tfm-rubygem-foreman-tasks
apt-get install ruby-foreman-tasks
Foreman tasks provides a service that is used for background processing, make sure you have it running by executing following
service foreman-tasks start
One of the most important plugin page is list of all tasks. You can find it in Monitor -> Tasks. The table lists all tasks with their statuses. Note that the task name can vary based on how it was triggered. Each task type has unique label, which can be used to search all tasks of a given type. The search query might look like this
label = Actions::Foreman::Report::Import
You can also search by task status and result
state = paused state = running state = stopped result = error result = pending result = warning result = success state = stopped and result = failed
as well as start and end time, to find tasks started after December 1st or finished after the same date, you can use following
start_at > 2016-12-01 ended_at > 2016-12-01
To get a quick overview of tasks status, you can use two dashboard widgets provided by the plugin.
Each task has following attributes
Execution type - possible values are immediate and delayed, delayed tasks are usually scheduled to start at specific time in future, to determine the execution type, attributes start_at and started_at are compared
Start before - indicates the deadline after which the task should not start if it hasn’t started yet
Note that you get more information by hovering some of the attribute values.
A: When a task is paused it means it encountered an error which it can’t solve on its own and needs the user to either cancel it or fix the issue and resume it. Not every task is pause on error, only tasks that were programmed to pause do. To find more information, check what you see in Errors field on task page or red box under the Errors tab.
A: While the task is in progress we can’t tell what the result will be, in such case the result is denoted as pending. E.g. if we’re in the middle of puppet report importing, we can’t tell if it will succeed or fail until it’s fully processed. If we ran out of HDD space before the last line was saved in DB, it would fail at the very last step.
A: The best place to look is the task detail page. In the first tab, you will see the human readable message based on the exception that was raised. The format depends on author of the task. If the task is based on Dynflow, you can find more information by navigating to the Errors tab on the same page. If you report an issue, make sure you attach the content of this tab with other logs. Note that the input and output fields may contain sensitive data so make sure you clean all such data while uploading it.
There are more tabs in task detail page. They are mostly used during debugging.
If you have a task that’s in a paused state because of some error, you can fix the error and resume the task execution. After the error is fixed and the task is resumed, the task will continue to run from the place, where the error occurred. If the error prevails, chances are that the task will fail again with the same error. To resume paused task, navigate to its detail page and click the resume button. If the button is disabled, the state is no longer paused. Stopped tasks can’t be resumed, they are finished completely. The error might appear because of some unexpected conditions, e.g. connectivity problems during communication with external service, or simply because there’s a bug in the task itself. If it’s the later case, it’s up to the author of the task to fix the code, deliver the new version of the task definition to the user who can then resume the execution.
Some tasks can be cancelled. Cancellation support is determined by the author and so far only Dynflow based tasks can be cancelled. Most of the tasks can be cancelled if they were scheduled in future and they were not started yet or they are running and the author of the task allowed cancelling. How exactly cancellation behaves is depending on task type, common practice is that it cancels all steps that have not been executed yet.
As mentioned above, foreman-tasks supports locking mechanism. This is useful if tasks can’t access the same resource at the same time, e.g. if they would write the same file. Each task can have multiple locks, each lock having its own name (usually derived from the operation) and resource. Exclusive locks prevent all other tasks to lock the same resource for the same operation. Task that destroy some resource usually lock all operations on a resource being deleted, since we don’t want any other task to touch deleted resource.
Non-exclusive lock is a link that only adds a reference to the resource, it does not prevent any other task from locking or linking the same resource. Each task usually has a link to the user who triggered it.
Now think of a situation where a task locks a resource and fails because of some error. Such task will keep the resource locked preventing all new tasks to work with this resource, since when they try to lock it, they will receive a conflict. To fix this state, the original task must finish first so it unlocks the resource. See Resuming a task section for more details about resuming paused tasks. If you for some reason can’t fix the failing task, you can force the unlock of locked resources. Since this is potentially dangerous action which will leave the system in inconsistent state, it’s not enabled by default. It can be enabled by dynflow_allow_dangerous_actions setting. For more information about settings, please see Settings chapter.
After you enable dangerous action you can see two new buttons in task detail - Unlock and Force Unlock. They are only active under specific circumstances. You can use Unlock only for paused task. After clicking the button and confirming you understand the consequences, the task is moved to stopped state. This effectively unlocks all resources locked by the task. The task can still be resumed later even it’s state is stopped, this allows you to recover from inconsistent state if the original issue was fixed. If the task is resumed, it can fail again, remain paused and therefore lock the resource again. Force Unlock is similar to Unlock but can be used for tasks in any state other than stopped. This can be useful for tasks that got stuck in running state and will never stop which happens very rarely.
If the task uses dynflow as a an execution backend you might navigate to corresponding dynflow task from the task detail page. This is only possible if dynflow_enable_console setting is enabled. Please note that the dynflow console is low-level debugging tool and should not be used or exposed to regular user. If you enable the dynflow console, it’s a good idea to also set dynflow_console_require_auth to Yes, which allows only admin users to access it. In dynflow console you can find information about the dynflow tasks, dynflow execution plans and executor statuses.
Some tasks can be set to repeat based on regular basis. Recurring logic is the definition of such repeating. Note that the task is a single run, recurring logic creates new task instance for every iteration. Existing recurring logics can be found under Monitor -> Recurring Logics. Following list explains the logic attributes.
State - one of Active, Finished, Cancelled, Failed. Normally the recurring logic starts in Active state, once the last iteration is finished it moves to Finished. There can be a logic without any stop date so it can remain Active forever. If user decides to stop recurring before the logic would finish, he can cancel the logic. In this case the logic state is set to Cancelled. Failed state means that there was some internal failure.
Foreman tasks plugin adds new permissions to Foreman core permission system. The new resources you can find in filter form are Foreman tasks/recurring logic and Foreman tasks/task. Both support granular filters so you can specify a subset of resources that the permission is granted for. E.g. to create a filter permitting to view and edit only tasks of a specific type, you can use a filter with condition such as
label = Actions::Foreman::Report::Import. The filter search field uses the same syntax as task searching so for more examples, see Listing tasks. The edit tasks permission allows resuming, cancelling, unlocking and dynflow console access.
Under Administer -> Settings you can find ForemanTasks tab which allows you to customize plugin behavior. Here is the list of settings and their meaning.
|dynflow_allow_dangerous_actions||Enables unlocking tasks. Note that this might leave the system in inconsistent state and is highly discouraged. Please see locking explanation chapter before allowing this.|
|dynflow_console_require_auth||If dynflow console is enabled, require user to be authenticated as user with admin rights when accessing dynflow console|
|dynflow_enable_console||Enable the dynflow console (/foreman_tasks/dynflow) for debugging|
|foreman_tasks_proxy_action_retry_count||Number of attempts to start a task on the smart proxy before failing|
|foreman_tasks_proxy_action_retry_interval||Time in seconds between retries|
|foreman_tasks_proxy_action_start_timeout||Time in second during which a task has to be started on the proxy|
Although, the history of tasks has an auditing value, some kinds of tasks can grow up in number quite soon. Therefore there is a mechanism how to clean the tasks, using a rake command. When running without any arguments, the tasks are deleted based on the default parameters defined in the code.
To see what tasks would be deleted, without actually deleting the records, you can run
foreman-rake foreman_tasks:cleanup NOOP=true
By default, only the actions explicitly defined with expiration time in the code, will get cleaned. One can configure new actions, or override the default configuration inside the configuration config/settings.plugins.d/foreman_tasks.yaml, such as:
:foreman-tasks: :cleanup: # the period after witch to delete all the tasks (by default all tasks are not being deleted after some period) :after: 365d # per action settings to override the default defined in the actions (cleanup_after method) :actions: - :name: Actions::Foreman::Host::ImportFacts :after: 10d
The foreman_tasks:cleanup script also accepts additional parameters to specify the search criteria for the cleanup manually:
To see the current configuration (what actions get cleaned automatically and what is their after period), this script can be used:
If you want to share the the history of tasks with someone else who does not have access to your Foreman instance, there’s a tool to do the export. This is especially useful when developers asks you for some debugging data. To do the export, use to following rake task provided by the plugin
This will create gzipped tar file in temp directory containing html page for each exported task. The path to the file is printed after the export is finished. These data contains tasks information stored in database from last 7 days by default. To change how old tasks should be part of the export, use following variables when running the export.
To explicitly name the export, you can use TASK_FILE variable. The following example would export all facts importing tasks from last week into file /tmp/last_week_importing.tar.gz
foreman-rake foreman_tasks:export_tasks TASK_SEARCH='label = "Actions::Foreman::Host::ImportFacts"' \ TASK_FILE=/tmp/last_week_importing.tar.gz TASK_DAYS=7
If you prefer CSV format instead of HTML pages, you use use TASK_FORMAT variable and change to format to csv. Only html and csv formats are available at the moment. The CSV format does not generate the gzipped tarball but plain CSV file that contains limited information of matched tasks. The information includes task id, state, type, label, result, parent_task_id, started_at and ended_at attributes.
We don’t have the developer guide yet. At the moment we recommend to read the code in the repository. Please consider contributing.
Please follow our standard procedures and contacts.
If you find a bug, please file it in Redmine.
See the troubleshooting section in the Foreman manual for more info.
Follow the same process as Foreman for contributing.