Now that we have katello being tested alongside foreman for pull requests, we need to make some more formal statements about how we test other plugins. We need to define:

1) What plugins we support
2) At what point in the cycle they get their own tests run (to check for plugin breakage)
3) At what point in the cycle the foreman tests are run with the plugin (to check for core breakage)
4) How we notify authors

For my own opinions, I feel:

1) Anything under the "theforeman" Github username
2) After a PR is merged (plugin tests are usually pretty fast)
3) After a PR is merged

4) is the tricky one. With the exception of katello, I'm not sure it's right to block PRs or merges on failed plugin tests - to do so implies to me that this plugin is so important, it should be merged into core. Perhaps we need a separate set of jobs which can be non-blocking downstream jobs, and plugin authors can then subscribe to the RSS feed for the job, to be alerted to failing tests of either nature.