Project

General

Profile

Actions
Copy link

Support #21107

closed

Puppet Run - 500 server error

Added by Jeff Sparrow over 6 years ago. Updated over 6 years ago.

Status:
Resolved
Priority:
Normal
Assignee:
-
Category:
Puppet integration
Target version:
-
Triaged:
Fixed in Releases:
Found in Releases:
1.12.4

Description

Your typical 500 server error with foreman-proxy and puppet run (under mco). Proxy up and running, settings all seem correct. I believe I am running into the following issue, to which I have tried numerous different sudo settings. None have worked. Does anyone know how to resolve this?

When I execute the command I am going to assume the 500 failure is due to the following:

Sep 26 07:13:49 puppet sudo: pam_unix(sudo:auth): conversation failed
Sep 26 07:13:49 puppet sudo: pam_unix(sudo:auth): auth could not identify password for [foreman-proxy]
Sep 26 07:13:49 puppet sudo: foreman-proxy : command not allowed ; TTY=unknown ; PWD=/ ; USER=mcollective ; COMMAND=/opt/puppetlabs/bin/mco puppet runonce -I ppt-2016-1.lab.beer.town

----------------------

[*NEXT* root@puppet 1 /etc/sudoers.d]# sudo -U foreman-proxy -l
Matching Defaults entries for foreman-proxy on puppet:
    !visiblepw, always_set_home, match_group_by_gid, env_reset, env_keep="COLORS DISPLAY HOSTNAME HISTSIZE KDEDIR LS_COLORS", env_keep+="MAIL PS1 PS2 QTDIR USERNAME LANG LC_ADDRESS LC_CTYPE", env_keep+="LC_COLLATE LC_IDENTIFICATION LC_MEASUREMENT LC_MESSAGES",
    env_keep+="LC_MONETARY LC_NAME LC_NUMERIC LC_PAPER LC_TELEPHONE", env_keep+="LC_TIME LC_ALL LANGUAGE LINGUAS _XKB_CHARSET XAUTHORITY", secure_path=/sbin\:/bin\:/usr/sbin\:/usr/bin, !requiretty

User foreman-proxy may run the following commands on puppet:
    (root) NOPASSWD: /opt/puppetlabs/bin/mco *

-----------------------

[*NEXT* root@puppet 0 /etc/sudoers.d]# sudo -U mcollective -l
Matching Defaults entries for mcollective on puppet:
    !visiblepw, always_set_home, match_group_by_gid, env_reset, env_keep="COLORS DISPLAY HOSTNAME HISTSIZE KDEDIR LS_COLORS", env_keep+="MAIL PS1 PS2 QTDIR USERNAME LANG LC_ADDRESS LC_CTYPE", env_keep+="LC_COLLATE LC_IDENTIFICATION LC_MEASUREMENT LC_MESSAGES",
    env_keep+="LC_MONETARY LC_NAME LC_NUMERIC LC_PAPER LC_TELEPHONE", env_keep+="LC_TIME LC_ALL LANGUAGE LINGUAS _XKB_CHARSET XAUTHORITY", secure_path=/sbin\:/bin\:/usr/sbin\:/usr/bin

User mcollective may run the following commands on puppet:
    (ALL) ALL
    (root) NOPASSWD: /opt/puppetlabs/bin/puppet *, /opt/puppetlabs/bin/mco *

What I dont understand is why I get the error, when I am explicitly giving sudoer access to foreman-proxy and mcollective users. Hoping someone has an answer.

Actions
Copy link
 #1

Updated by Jeff Sparrow over 6 years ago

Adding this in case search engine users land here, as no other resolutions for ERF12-4252 have worked.

2017-09-26 07:53:05 [app] [W] Unable to execute puppet run
 | ProxyAPI::ProxyException: ERF12-4252 [ProxyAPI::ProxyException]: Unable to execute Puppet run ([RestClient::InternalServerError]: 500 Internal Server Error) for proxy https://puppet:8443/puppet
 | /usr/share/foreman/lib/proxy_api/puppet.rb:41:in `rescue in run'
 | /usr/share/foreman/lib/proxy_api/puppet.rb:39:in `run'
 | /usr/share/foreman/app/models/host/managed.rb:660:in `puppetrun!'
 | /opt/rh/sclo-ror42/root/usr/share/gems/gems/activerecord-4.2.5.1/lib/active_record/relation/delegation.rb:46:in `map'
 | /opt/rh/sclo-ror42/root/usr/share/gems/gems/activerecord-4.2.5.1/lib/active_record/relation/delegation.rb:46:in `map'
 | /usr/share/foreman/app/controllers/hosts_controller.rb:572:in `update_multiple_puppetrun'
 | /opt/rh/sclo-ror42/root/usr/share/gems/gems/actionpack-4.2.5.1/lib/action_controller/metal/implicit_render.rb:4:in `send_action'
 | /opt/rh/sclo-ror42/root/usr/share/gems/gems/actionpack-4.2.5.1/lib/abstract_controller/base.rb:198:in `process_action'
 | /opt/rh/sclo-ror42/root/usr/share/gems/gems/actionpack-4.2.5.1/lib/action_controller/metal/rendering.rb:10:in `process_action'
 | /opt/rh/sclo-ror42/root/usr/share/gems/gems/actionpack-4.2.5.1/lib/abstract_controller/callbacks.rb:20:in `block in process_action'
 | /opt/rh/sclo-ror42/root/usr/share/gems/gems/activesupport-4.2.5.1/lib/active_support/callbacks.rb:117:in `call'
 | /opt/rh/sclo-ror42/root/usr/share/gems/gems/activesupport-4.2.5.1/lib/active_support/callbacks.rb:117:in `call'
 | /opt/rh/sclo-ror42/root/usr/share/gems/gems/activesupport-4.2.5.1/lib/active_support/callbacks.rb:555:in `block (2 levels) in compile'
 | /opt/rh/sclo-ror42/root/usr/share/gems/gems/activesupport-4.2.5.1/lib/active_support/callbacks.rb:505:in `call'
 | /opt/rh/sclo-ror42/root/usr/share/gems/gems/activesupport-4.2.5.1/lib/active_support/callbacks.rb:505:in `call'
 | /opt/rh/sclo-ror42/root/usr/share/gems/gems/activesupport-4.2.5.1/lib/active_support/callbacks.rb:498:in `block (2 levels) in around'
 | /opt/rh/sclo-ror42/root/usr/share/gems/gems/activesupport-4.2.5.1/lib/active_support/callbacks.rb:313:in `call'
 | /opt/rh/sclo-ror42/root/usr/share/gems/gems/activesupport-4.2.5.1/lib/active_support/callbacks.rb:313:in `block (2 levels) in halting'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/rails-observers-0.1.2/lib/rails/observers/action_controller/caching/sweeping.rb:73:in `around'
 | /opt/rh/sclo-ror42/root/usr/share/gems/gems/activesupport-4.2.5.1/lib/active_support/callbacks.rb:455:in `public_send'
 | /opt/rh/sclo-ror42/root/usr/share/gems/gems/activesupport-4.2.5.1/lib/active_support/callbacks.rb:455:in `block in make_lambda'
 | /opt/rh/sclo-ror42/root/usr/share/gems/gems/activesupport-4.2.5.1/lib/active_support/callbacks.rb:312:in `call'
 | /opt/rh/sclo-ror42/root/usr/share/gems/gems/activesupport-4.2.5.1/lib/active_support/callbacks.rb:312:in `block in halting'
 | /opt/rh/sclo-ror42/root/usr/share/gems/gems/activesupport-4.2.5.1/lib/active_support/callbacks.rb:497:in `call'
 | /opt/rh/sclo-ror42/root/usr/share/gems/gems/activesupport-4.2.5.1/lib/active_support/callbacks.rb:497:in `block in around'
 | /opt/rh/sclo-ror42/root/usr/share/gems/gems/activesupport-4.2.5.1/lib/active_support/callbacks.rb:505:in `call'
 | /opt/rh/sclo-ror42/root/usr/share/gems/gems/activesupport-4.2.5.1/lib/active_support/callbacks.rb:505:in `call'
 | /opt/rh/sclo-ror42/root/usr/share/gems/gems/activesupport-4.2.5.1/lib/active_support/callbacks.rb:498:in `block (2 levels) in around'
 | /opt/rh/sclo-ror42/root/usr/share/gems/gems/activesupport-4.2.5.1/lib/active_support/callbacks.rb:313:in `call'
 | /opt/rh/sclo-ror42/root/usr/share/gems/gems/activesupport-4.2.5.1/lib/active_support/callbacks.rb:313:in `block (2 levels) in halting'
 | /usr/share/foreman/app/controllers/concerns/application_shared.rb:13:in `set_timezone'
 | /opt/rh/sclo-ror42/root/usr/share/gems/gems/activesupport-4.2.5.1/lib/active_support/callbacks.rb:432:in `block in make_lambda'
 | /opt/rh/sclo-ror42/root/usr/share/gems/gems/activesupport-4.2.5.1/lib/active_support/callbacks.rb:312:in `call'
 | /opt/rh/sclo-ror42/root/usr/share/gems/gems/activesupport-4.2.5.1/lib/active_support/callbacks.rb:312:in `block in halting'
 | /opt/rh/sclo-ror42/root/usr/share/gems/gems/activesupport-4.2.5.1/lib/active_support/callbacks.rb:497:in `call'
 | /opt/rh/sclo-ror42/root/usr/share/gems/gems/activesupport-4.2.5.1/lib/active_support/callbacks.rb:497:in `block in around'
 | /opt/rh/sclo-ror42/root/usr/share/gems/gems/activesupport-4.2.5.1/lib/active_support/callbacks.rb:505:in `call'
 | /opt/rh/sclo-ror42/root/usr/share/gems/gems/activesupport-4.2.5.1/lib/active_support/callbacks.rb:505:in `call'
 | /opt/rh/sclo-ror42/root/usr/share/gems/gems/activesupport-4.2.5.1/lib/active_support/callbacks.rb:498:in `block (2 levels) in around'
 | /opt/rh/sclo-ror42/root/usr/share/gems/gems/activesupport-4.2.5.1/lib/active_support/callbacks.rb:313:in `call'
 | /opt/rh/sclo-ror42/root/usr/share/gems/gems/activesupport-4.2.5.1/lib/active_support/callbacks.rb:313:in `block (2 levels) in halting'
 | /usr/share/foreman/app/models/concerns/foreman/thread_session.rb:32:in `clear_thread'
 | /opt/rh/sclo-ror42/root/usr/share/gems/gems/activesupport-4.2.5.1/lib/active_support/callbacks.rb:432:in `block in make_lambda'
 | /opt/rh/sclo-ror42/root/usr/share/gems/gems/activesupport-4.2.5.1/lib/active_support/callbacks.rb:312:in `call'
 | /opt/rh/sclo-ror42/root/usr/share/gems/gems/activesupport-4.2.5.1/lib/active_support/callbacks.rb:312:in `block in halting'
 | /opt/rh/sclo-ror42/root/usr/share/gems/gems/activesupport-4.2.5.1/lib/active_support/callbacks.rb:497:in `call'
 | /opt/rh/sclo-ror42/root/usr/share/gems/gems/activesupport-4.2.5.1/lib/active_support/callbacks.rb:497:in `block in around'
 | /opt/rh/sclo-ror42/root/usr/share/gems/gems/activesupport-4.2.5.1/lib/active_support/callbacks.rb:505:in `call'
 | /opt/rh/sclo-ror42/root/usr/share/gems/gems/activesupport-4.2.5.1/lib/active_support/callbacks.rb:505:in `call'
 | /opt/rh/sclo-ror42/root/usr/share/gems/gems/activesupport-4.2.5.1/lib/active_support/callbacks.rb:498:in `block (2 levels) in around'
 | /opt/rh/sclo-ror42/root/usr/share/gems/gems/activesupport-4.2.5.1/lib/active_support/callbacks.rb:313:in `call'
 | /opt/rh/sclo-ror42/root/usr/share/gems/gems/activesupport-4.2.5.1/lib/active_support/callbacks.rb:313:in `block (2 levels) in halting'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/rails-observers-0.1.2/lib/rails/observers/action_controller/caching/sweeping.rb:73:in `around'
 | /opt/rh/sclo-ror42/root/usr/share/gems/gems/activesupport-4.2.5.1/lib/active_support/callbacks.rb:455:in `public_send'
 | /opt/rh/sclo-ror42/root/usr/share/gems/gems/activesupport-4.2.5.1/lib/active_support/callbacks.rb:455:in `block in make_lambda'
 | /opt/rh/sclo-ror42/root/usr/share/gems/gems/activesupport-4.2.5.1/lib/active_support/callbacks.rb:312:in `call'
 | /opt/rh/sclo-ror42/root/usr/share/gems/gems/activesupport-4.2.5.1/lib/active_support/callbacks.rb:312:in `block in halting'
 | /opt/rh/sclo-ror42/root/usr/share/gems/gems/activesupport-4.2.5.1/lib/active_support/callbacks.rb:497:in `call'
 | /opt/rh/sclo-ror42/root/usr/share/gems/gems/activesupport-4.2.5.1/lib/active_support/callbacks.rb:497:in `block in around'
 | /opt/rh/sclo-ror42/root/usr/share/gems/gems/activesupport-4.2.5.1/lib/active_support/callbacks.rb:505:in `call'
 | /opt/rh/sclo-ror42/root/usr/share/gems/gems/activesupport-4.2.5.1/lib/active_support/callbacks.rb:505:in `call'
 | /opt/rh/sclo-ror42/root/usr/share/gems/gems/activesupport-4.2.5.1/lib/active_support/callbacks.rb:92:in `__run_callbacks__'
 | /opt/rh/sclo-ror42/root/usr/share/gems/gems/activesupport-4.2.5.1/lib/active_support/callbacks.rb:778:in `_run_process_action_callbacks'
 | /opt/rh/sclo-ror42/root/usr/share/gems/gems/activesupport-4.2.5.1/lib/active_support/callbacks.rb:81:in `run_callbacks'
 | /opt/rh/sclo-ror42/root/usr/share/gems/gems/actionpack-4.2.5.1/lib/abstract_controller/callbacks.rb:19:in `process_action'
 | /opt/rh/sclo-ror42/root/usr/share/gems/gems/actionpack-4.2.5.1/lib/action_controller/metal/rescue.rb:29:in `process_action'
 | /opt/rh/sclo-ror42/root/usr/share/gems/gems/actionpack-4.2.5.1/lib/action_controller/metal/instrumentation.rb:32:in `block in process_action'
 | /opt/rh/sclo-ror42/root/usr/share/gems/gems/activesupport-4.2.5.1/lib/active_support/notifications.rb:164:in `block in instrument'
 | /opt/rh/sclo-ror42/root/usr/share/gems/gems/activesupport-4.2.5.1/lib/active_support/notifications/instrumenter.rb:20:in `instrument'
 | /opt/rh/sclo-ror42/root/usr/share/gems/gems/activesupport-4.2.5.1/lib/active_support/notifications.rb:164:in `instrument'
 | /opt/rh/sclo-ror42/root/usr/share/gems/gems/actionpack-4.2.5.1/lib/action_controller/metal/instrumentation.rb:30:in `process_action'
 | /opt/rh/sclo-ror42/root/usr/share/gems/gems/actionpack-4.2.5.1/lib/action_controller/metal/params_wrapper.rb:250:in `process_action'
 | /opt/rh/sclo-ror42/root/usr/share/gems/gems/activerecord-4.2.5.1/lib/active_record/railties/controller_runtime.rb:18:in `process_action'
 | /opt/rh/sclo-ror42/root/usr/share/gems/gems/actionpack-4.2.5.1/lib/abstract_controller/base.rb:137:in `process'
 | /opt/rh/sclo-ror42/root/usr/share/gems/gems/actionview-4.2.5.1/lib/action_view/rendering.rb:30:in `process'
 | /opt/rh/sclo-ror42/root/usr/share/gems/gems/actionpack-4.2.5.1/lib/action_controller/metal.rb:196:in `dispatch'
 | /opt/rh/sclo-ror42/root/usr/share/gems/gems/actionpack-4.2.5.1/lib/action_controller/metal/rack_delegation.rb:13:in `dispatch'
 | /opt/rh/sclo-ror42/root/usr/share/gems/gems/actionpack-4.2.5.1/lib/action_controller/metal.rb:237:in `block in action'
 | /opt/rh/sclo-ror42/root/usr/share/gems/gems/actionpack-4.2.5.1/lib/action_dispatch/routing/route_set.rb:74:in `call'
 | /opt/rh/sclo-ror42/root/usr/share/gems/gems/actionpack-4.2.5.1/lib/action_dispatch/routing/route_set.rb:74:in `dispatch'
 | /opt/rh/sclo-ror42/root/usr/share/gems/gems/actionpack-4.2.5.1/lib/action_dispatch/routing/route_set.rb:43:in `serve'
 | /opt/rh/sclo-ror42/root/usr/share/gems/gems/actionpack-4.2.5.1/lib/action_dispatch/journey/router.rb:43:in `block in serve'
 | /opt/rh/sclo-ror42/root/usr/share/gems/gems/actionpack-4.2.5.1/lib/action_dispatch/journey/router.rb:30:in `each'
 | /opt/rh/sclo-ror42/root/usr/share/gems/gems/actionpack-4.2.5.1/lib/action_dispatch/journey/router.rb:30:in `serve'
 | /opt/rh/sclo-ror42/root/usr/share/gems/gems/actionpack-4.2.5.1/lib/action_dispatch/routing/route_set.rb:815:in `call'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/apipie-rails-0.3.6/lib/apipie/static_dispatcher.rb:65:in `call'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/apipie-rails-0.3.6/lib/apipie/extractor/recorder.rb:132:in `call'
 | /opt/rh/sclo-ror42/root/usr/share/gems/gems/actionpack-4.2.5.1/lib/action_dispatch/middleware/static.rb:116:in `call'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/apipie-rails-0.3.6/lib/apipie/middleware/checksum_in_headers.rb:27:in `call'
 | /opt/rh/sclo-ror42/root/usr/share/gems/gems/rack-1.6.2/lib/rack/etag.rb:24:in `call'
 | /opt/rh/sclo-ror42/root/usr/share/gems/gems/rack-1.6.2/lib/rack/conditionalget.rb:38:in `call'
 | /opt/rh/sclo-ror42/root/usr/share/gems/gems/rack-1.6.2/lib/rack/head.rb:13:in `call'
 | /opt/rh/sclo-ror42/root/usr/share/gems/gems/actionpack-4.2.5.1/lib/action_dispatch/middleware/params_parser.rb:27:in `call'
 | /usr/share/foreman/lib/middleware/catch_json_parse_errors.rb:9:in `call'
 | /opt/rh/sclo-ror42/root/usr/share/gems/gems/actionpack-4.2.5.1/lib/action_dispatch/middleware/flash.rb:260:in `call'
 | /opt/rh/sclo-ror42/root/usr/share/gems/gems/rack-1.6.2/lib/rack/session/abstract/id.rb:225:in `context'
 | /opt/rh/sclo-ror42/root/usr/share/gems/gems/rack-1.6.2/lib/rack/session/abstract/id.rb:220:in `call'
 | /opt/rh/sclo-ror42/root/usr/share/gems/gems/actionpack-4.2.5.1/lib/action_dispatch/middleware/cookies.rb:560:in `call'
 | /opt/rh/sclo-ror42/root/usr/share/gems/gems/activerecord-4.2.5.1/lib/active_record/query_cache.rb:36:in `call'
 | /opt/rh/sclo-ror42/root/usr/share/gems/gems/activerecord-4.2.5.1/lib/active_record/connection_adapters/abstract/connection_pool.rb:653:in `call'
 | /opt/rh/sclo-ror42/root/usr/share/gems/gems/actionpack-4.2.5.1/lib/action_dispatch/middleware/callbacks.rb:29:in `block in call'
 | /opt/rh/sclo-ror42/root/usr/share/gems/gems/activesupport-4.2.5.1/lib/active_support/callbacks.rb:88:in `__run_callbacks__'
 | /opt/rh/sclo-ror42/root/usr/share/gems/gems/activesupport-4.2.5.1/lib/active_support/callbacks.rb:778:in `_run_call_callbacks'
 | /opt/rh/sclo-ror42/root/usr/share/gems/gems/activesupport-4.2.5.1/lib/active_support/callbacks.rb:81:in `run_callbacks'
 | /opt/rh/sclo-ror42/root/usr/share/gems/gems/actionpack-4.2.5.1/lib/action_dispatch/middleware/callbacks.rb:27:in `call'
 | /opt/rh/sclo-ror42/root/usr/share/gems/gems/actionpack-4.2.5.1/lib/action_dispatch/middleware/remote_ip.rb:78:in `call'
 | /opt/rh/sclo-ror42/root/usr/share/gems/gems/actionpack-4.2.5.1/lib/action_dispatch/middleware/debug_exceptions.rb:17:in `call'
 | /opt/rh/sclo-ror42/root/usr/share/gems/gems/actionpack-4.2.5.1/lib/action_dispatch/middleware/show_exceptions.rb:30:in `call'
 | /opt/rh/sclo-ror42/root/usr/share/gems/gems/railties-4.2.5.1/lib/rails/rack/logger.rb:38:in `call_app'
 | /opt/rh/sclo-ror42/root/usr/share/gems/gems/railties-4.2.5.1/lib/rails/rack/logger.rb:22:in `call'
 | /opt/rh/sclo-ror42/root/usr/share/gems/gems/actionpack-4.2.5.1/lib/action_dispatch/middleware/request_id.rb:21:in `call'
 | /opt/rh/sclo-ror42/root/usr/share/gems/gems/rack-1.6.2/lib/rack/methodoverride.rb:22:in `call'
 | /opt/rh/sclo-ror42/root/usr/share/gems/gems/rack-1.6.2/lib/rack/runtime.rb:18:in `call'
 | /opt/rh/sclo-ror42/root/usr/share/gems/gems/activesupport-4.2.5.1/lib/active_support/cache/strategy/local_cache_middleware.rb:28:in `call'
 | /opt/rh/sclo-ror42/root/usr/share/gems/gems/actionpack-4.2.5.1/lib/action_dispatch/middleware/static.rb:116:in `call'
 | /opt/rh/sclo-ror42/root/usr/share/gems/gems/rack-1.6.2/lib/rack/sendfile.rb:113:in `call'
 | /opt/rh/sclo-ror42/root/usr/share/gems/gems/railties-4.2.5.1/lib/rails/engine.rb:518:in `call'
 | /opt/rh/sclo-ror42/root/usr/share/gems/gems/railties-4.2.5.1/lib/rails/application.rb:165:in `call'
 | /opt/rh/sclo-ror42/root/usr/share/gems/gems/railties-4.2.5.1/lib/rails/railtie.rb:194:in `public_send'
 | /opt/rh/sclo-ror42/root/usr/share/gems/gems/railties-4.2.5.1/lib/rails/railtie.rb:194:in `method_missing'
 | /opt/rh/sclo-ror42/root/usr/share/gems/gems/rack-1.6.2/lib/rack/urlmap.rb:66:in `block in call'
 | /opt/rh/sclo-ror42/root/usr/share/gems/gems/rack-1.6.2/lib/rack/urlmap.rb:50:in `each'
 | /opt/rh/sclo-ror42/root/usr/share/gems/gems/rack-1.6.2/lib/rack/urlmap.rb:50:in `call'
 | /usr/share/gems/gems/passenger-4.0.18/lib/phusion_passenger/rack/thread_handler_extension.rb:77:in `process_request'
 | /usr/share/gems/gems/passenger-4.0.18/lib/phusion_passenger/request_handler/thread_handler.rb:140:in `accept_and_process_next_request'
 | /usr/share/gems/gems/passenger-4.0.18/lib/phusion_passenger/request_handler/thread_handler.rb:108:in `main_loop'
 | /usr/share/gems/gems/passenger-4.0.18/lib/phusion_passenger/request_handler.rb:441:in `block (3 levels) in start_threads'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/logging-1.8.2/lib/logging/diagnostic_context.rb:323:in `call'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/logging-1.8.2/lib/logging/diagnostic_context.rb:323:in `block in create_with_logging_context'
2017-09-26 07:53:05 [app] [I] Redirected to https://puppet/hosts
2017-09-26 07:53:05 [app] [I] Completed 302 Found in 103ms (ActiveRecord: 1.1ms)
Actions
Copy link
 #2

Updated by Jeff Sparrow over 6 years ago

We are starting to think this might be the cause: ??

Sep 26 13:26:40 puppet sudo: pam_localuser(sudo:auth): checking "root:x:0:0:root:/root:/bin/bash#012" 
Sep 26 13:26:40 puppet sudo: pam_localuser(sudo:auth): checking "bin:x:1:1:bin:/bin:/sbin/nologin#012" 
Sep 26 13:26:40 puppet sudo: pam_localuser(sudo:auth): checking "daemon:x:2:2:daemon:/sbin:/sbin/nologin#012" 
Sep 26 13:26:40 puppet sudo: pam_localuser(sudo:auth): checking "adm:x:3:4:adm:/var/adm:/sbin/nologin#012" 
Sep 26 13:26:40 puppet sudo: pam_localuser(sudo:auth): checking "lp:x:4:7:lp:/var/spool/lpd:/sbin/nologin#012" 
Sep 26 13:26:40 puppet sudo: pam_localuser(sudo:auth): checking "sync:x:5:0:sync:/sbin:/bin/sync#012" 
Sep 26 13:26:40 puppet sudo: pam_localuser(sudo:auth): checking "shutdown:x:6:0:shutdown:/sbin:/sbin/shutdown#012" 
Sep 26 13:26:40 puppet sudo: pam_localuser(sudo:auth): checking "halt:x:7:0:halt:/sbin:/sbin/halt#012" 
Sep 26 13:26:40 puppet sudo: pam_localuser(sudo:auth): checking "mail:x:8:12:mail:/var/spool/mail:/sbin/nologin#012" 
Sep 26 13:26:40 puppet sudo: pam_localuser(sudo:auth): checking "operator:x:11:0:operator:/root:/sbin/nologin#012" 
Sep 26 13:26:40 puppet sudo: pam_localuser(sudo:auth): checking "games:x:12:100:games:/usr/games:/sbin/nologin#012" 
Sep 26 13:26:40 puppet sudo: pam_localuser(sudo:auth): checking "ftp:x:14:50:FTP User:/var/ftp:/sbin/nologin#012" 
Sep 26 13:26:40 puppet sudo: pam_localuser(sudo:auth): checking "nobody:x:99:99:Nobody:/:/sbin/nologin#012" 
Sep 26 13:26:40 puppet sudo: pam_localuser(sudo:auth): checking "avahi-autoipd:x:170:170:Avahi IPv4LL Stack:/var/lib/avahi-autoipd:/sbin/nologin#012" 
Sep 26 13:26:40 puppet sudo: pam_localuser(sudo:auth): checking "dbus:x:81:81:System message bus:/:/sbin/nologin#012" 
Sep 26 13:26:40 puppet sudo: pam_localuser(sudo:auth): checking "polkitd:x:999:998:User for polkitd:/:/sbin/nologin#012" 
Sep 26 13:26:40 puppet sudo: pam_localuser(sudo:auth): checking "tss:x:59:59:Account used by the trousers package to sandbox the tcsd daemon:/dev/null:/sbin/nologin#012" 
Sep 26 13:26:40 puppet sudo: pam_localuser(sudo:auth): checking "postfix:x:89:89::/var/spool/postfix:/sbin/nologin#012" 
Sep 26 13:26:40 puppet sudo: pam_localuser(sudo:auth): checking "sshd:x:74:74:Privilege-separated SSH:/var/empty/sshd:/sbin/nologin#012" 
Sep 26 13:26:40 puppet sudo: pam_localuser(sudo:auth): checking "esadmin:x:1000:1000:esadmin:/home/esadmin:/bin/bash#012" 
Sep 26 13:26:40 puppet sudo: pam_localuser(sudo:auth): checking "nscd:x:28:28:NSCD Daemon:/:/sbin/nologin#012" 
Sep 26 13:26:40 puppet sudo: pam_localuser(sudo:auth): checking "nslcd:x:65:55:LDAP Client User:/:/sbin/nologin#012" 
Sep 26 13:26:40 puppet sudo: pam_localuser(sudo:auth): checking "systemd-bus-proxy:x:998:996:systemd Bus Proxy:/:/sbin/nologin#012" 
Sep 26 13:26:40 puppet sudo: pam_localuser(sudo:auth): checking "systemd-network:x:997:995:systemd Network Management:/:/sbin/nologin#012" 
Sep 26 13:26:40 puppet sudo: pam_localuser(sudo:auth): checking "rpc:x:32:32:Rpcbind Daemon:/var/lib/rpcbind:/sbin/nologin#012" 
Sep 26 13:26:40 puppet sudo: pam_localuser(sudo:auth): checking "rpcuser:x:29:29:RPC Service User:/var/lib/nfs:/sbin/nologin#012" 
Sep 26 13:26:40 puppet sudo: pam_localuser(sudo:auth): checking "nfsnobody:x:65534:65534:Anonymous NFS User:/var/lib/nfs:/sbin/nologin#012" 
Sep 26 13:26:40 puppet sudo: pam_localuser(sudo:auth): checking "sssd:x:996:994:User for sssd:/:/sbin/nologin#012" 
Sep 26 13:26:40 puppet sudo: pam_localuser(sudo:auth): checking "puppet:x:52:52:puppetserver daemon:/opt/puppetlabs/server/data/puppetserver:/sbin/nologin#012" 
Sep 26 13:26:40 puppet sudo: pam_localuser(sudo:auth): checking "ntp:x:38:38::/etc/ntp:/sbin/nologin#012" 
Sep 26 13:26:40 puppet sudo: pam_localuser(sudo:auth): checking "foreman:x:995:992:Foreman:/usr/share/foreman:/bin/false#012" 
Sep 26 13:26:40 puppet sudo: pam_localuser(sudo:auth): checking "apache:x:48:48:Apache:/usr/share/httpd:/sbin/nologin#012" 
Sep 26 13:26:40 puppet sudo: pam_localuser(sudo:auth): checking "mysql:x:27:27:MariaDB Server:/var/lib/mysql:/sbin/nologin#012" 
Sep 26 13:26:40 puppet sudo: pam_localuser(sudo:auth): checking "foreman-proxy:x:994:991:Foreman Proxy account:/usr/share/foreman-proxy:/bin/false#012" 
Sep 26 13:26:40 puppet sudo: pam_unix(sudo:auth): conversation failed
Sep 26 13:26:40 puppet sudo: pam_unix(sudo:auth): unable to obtain a password
Sep 26 13:26:40 puppet sudo: pam_unix(sudo:auth): auth could not identify password for [foreman-proxy]
Sep 26 13:26:40 puppet sudo: foreman-proxy : command not allowed ; TTY=unknown ; PWD=/ ; USER=mcollective ; COMMAND=/opt/puppetlabs/bin/mco puppet runonce -I ppt-2016-1.lab.beer.town

Still digging though, but one this is, pam.d doesnt seem happy.

Actions
Copy link
 #3

Updated by Jeff Sparrow over 6 years ago

So this ticket is a bit different because I am using the new choria.io app provided by puppet/mcollective - https://choria.io/

Thanks to the amazing help of gwmngilfen in IRC, I was able to use the help/suggestions to get this working. Although, I did run in to more issues (below), which were even more difficult to troubleshoot, due to lack of logging. I didnt end up getting it to work with the user that I wanted, but I found a workaround which we are happy with:

One of the problems was the settings for /etc/sudoers.d/foreman-proxy, to which I had:

foreman-proxy ALL = (root) NOPASSWD : /opt/puppetlabs/bin/mco *
Defaults:foreman-proxy !requiretty

we changed it to - (root) to (ALL):

foreman-proxy ALL = (ALL) NOPASSWD : /opt/puppetlabs/bin/mco *
Defaults:foreman-proxy !requiretty

to allow the user that foreman-proxy was calling (user mcollective) to run the command. This allowed the commands to be processed, but calling mco in choria requires ssl certs. The only way I could see this was failing, as its not logged anywhere (aside from the standard 500 error in proxy.log), I had to open the shell as foreman-proxy 

sudo -u foreman-proxy -s /bin/bash

then run the commands as the user that foreman-proxy was trying to make the call as:
sudo -u mcollective /opt/puppetlabs/bin/mco help

this led to even more confusion, since calling mco help doesnt require a cert check. So once you run an actual mco command, you can see the failure:

bash-4.2$ sudo -u mcollective /opt/puppetlabs/bin/mco ping
debug 2017/09/27 07:07:18: pluginmanager.rb:167:in `loadclass' Loading Mcollective::Facts::Yaml_facts from mcollective/facts/yaml_facts.rb
debug 2017/09/27 07:07:18: pluginmanager.rb:44:in `<<' Registering plugin facts_plugin with class MCollective::Facts::Yaml_facts single_instance: true
debug 2017/09/27 07:07:18: pluginmanager.rb:167:in `loadclass' Loading Mcollective::Connector::Nats from mcollective/connector/nats.rb
debug 2017/09/27 07:07:18: cache.rb:117:in `block in ttl' Cache miss on 'ddl' key 'connector/nats'
debug 2017/09/27 07:07:18: base.rb:94:in `block in findddlfile' Found nats ddl at /opt/puppetlabs/mcollective/plugins/mcollective/connector/nats.ddl
debug 2017/09/27 07:07:18: pluginmanager.rb:44:in `<<' Registering plugin connector_plugin with class MCollective::Connector::Nats single_instance: true
debug 2017/09/27 07:07:18: pluginmanager.rb:167:in `loadclass' Loading Mcollective::Security::Choria from mcollective/security/choria.rb
debug 2017/09/27 07:07:18: pluginmanager.rb:44:in `<<' Registering plugin security_plugin with class MCollective::Security::Choria single_instance: true
debug 2017/09/27 07:07:18: pluginmanager.rb:167:in `loadclass' Loading Mcollective::Registration::Agentlist from mcollective/registration/agentlist.rb
debug 2017/09/27 07:07:18: pluginmanager.rb:44:in `<<' Registering plugin registration_plugin with class MCollective::Registration::Agentlist single_instance: true
debug 2017/09/27 07:07:18: pluginmanager.rb:47:in `<<' Registering plugin global_stats with class MCollective::RunnerStats single_instance: true
info 2017/09/27 07:07:18: config.rb:167:in `loadconfig' The Marionette Collective version 2.10.5 started by /opt/puppetlabs/bin/mco using config file /etc/puppetlabs/mcollective/client.cfg
debug 2017/09/27 07:07:18: pluginmanager.rb:167:in `loadclass' Loading MCollective::Application::Ping from mcollective/application/ping.rb
debug 2017/09/27 07:07:18: pluginmanager.rb:44:in `<<' Registering plugin ping_application with class MCollective::Application::Ping single_instance: true
debug 2017/09/27 07:07:18: pluginmanager.rb:80:in `[]' Returning new plugin ping_application with class MCollective::Application::Ping
debug 2017/09/27 07:07:18: pluginmanager.rb:80:in `[]' Returning new plugin connector_plugin with class MCollective::Connector::Nats
info 2017/09/27 07:07:18: nats.rb:15:in `initialize' Choria NATS.io connector using pure ruby nats/io/client 0.2.4 with protocol version 1
debug 2017/09/27 07:07:18: pluginmanager.rb:80:in `[]' Returning new plugin security_plugin with class MCollective::Security::Choria
debug 2017/09/27 07:07:18: pluginmanager.rb:83:in `[]' Returning cached plugin global_stats with class MCollective::RunnerStats

The ping application failed to run, use -v for full error backtrace details: No such file or directory @ rb_sysopen - /home/mcollective/.puppetlabs/etc/puppet/ssl/certs/foreman-proxy.mcollective.pem
debug 2017/09/27 07:07:18: pluginmanager.rb:83:in `[]' Returning cached plugin connector_plugin with class MCollective::Connector::Nats

You can see the next issue, after fixing the sudoer.d privileges, is chorio/nats/mcollective looking for cert (2nd to last line). There is a signed cert for mcollective, but this was trying to look for another cert of foreman-proxy (the user calling the command through mcollective, whereas I would expect it to look for mcollective's cert, it did not). So I signed a new cert through choria.io, for foreman-proxy user:

sudo -u foreman-proxy /opt/puppetlabs/bin/mco choria request_cert

which led to another issue of not having the rights to do that as foreman-proxy:

[*NEXT* root@puppet 1 ~]# sudo -u foreman-proxy /opt/puppetlabs/bin/mco choria request_cert
debug 2017/09/27 07:15:26: pluginmanager.rb:167:in `loadclass' Loading Mcollective::Facts::Yaml_facts from mcollective/facts/yaml_facts.rb
debug 2017/09/27 07:15:26: pluginmanager.rb:44:in `<<' Registering plugin facts_plugin with class MCollective::Facts::Yaml_facts single_instance: true
debug 2017/09/27 07:15:26: pluginmanager.rb:167:in `loadclass' Loading Mcollective::Connector::Nats from mcollective/connector/nats.rb
debug 2017/09/27 07:15:26: cache.rb:117:in `block in ttl' Cache miss on 'ddl' key 'connector/nats'
debug 2017/09/27 07:15:26: base.rb:94:in `block in findddlfile' Found nats ddl at /opt/puppetlabs/mcollective/plugins/mcollective/connector/nats.ddl
debug 2017/09/27 07:15:26: pluginmanager.rb:44:in `<<' Registering plugin connector_plugin with class MCollective::Connector::Nats single_instance: true
debug 2017/09/27 07:15:26: pluginmanager.rb:167:in `loadclass' Loading Mcollective::Security::Choria from mcollective/security/choria.rb
debug 2017/09/27 07:15:26: pluginmanager.rb:44:in `<<' Registering plugin security_plugin with class MCollective::Security::Choria single_instance: true
debug 2017/09/27 07:15:26: pluginmanager.rb:167:in `loadclass' Loading Mcollective::Registration::Agentlist from mcollective/registration/agentlist.rb
debug 2017/09/27 07:15:26: pluginmanager.rb:44:in `<<' Registering plugin registration_plugin with class MCollective::Registration::Agentlist single_instance: true
debug 2017/09/27 07:15:26: pluginmanager.rb:47:in `<<' Registering plugin global_stats with class MCollective::RunnerStats single_instance: true
info 2017/09/27 07:15:26: config.rb:167:in `loadconfig' The Marionette Collective version 2.10.5 started by /opt/puppetlabs/bin/mco using config file /etc/puppetlabs/mcollective/client.cfg
debug 2017/09/27 07:15:26: pluginmanager.rb:167:in `loadclass' Loading MCollective::Application::Choria from mcollective/application/choria.rb
debug 2017/09/27 07:15:26: pluginmanager.rb:44:in `<<' Registering plugin choria_application with class MCollective::Application::Choria single_instance: true
debug 2017/09/27 07:15:26: pluginmanager.rb:80:in `[]' Returning new plugin choria_application with class MCollective::Application::Choria
debug 2017/09/27 07:15:26: pluginmanager.rb:167:in `loadclass' Loading MCollective::Util::Choria from mcollective/util/choria.rb
debug 2017/09/27 07:15:26: pluginmanager.rb:80:in `[]' Returning new plugin connector_plugin with class MCollective::Connector::Nats
info 2017/09/27 07:15:26: nats.rb:15:in `initialize' Choria NATS.io connector using pure ruby nats/io/client 0.2.4 with protocol version 1

The choria application failed to run, use -v for full error backtrace details: Permission denied @ dir_s_mkdir - /usr/share/foreman-proxy/.puppetlabs/etc/puppet/ssl
debug 2017/09/27 07:15:26: pluginmanager.rb:83:in `[]' Returning cached plugin connector_plugin with class MCollective::Connector::Nats

I manually made the directory structure, and assigned the owner to be foreman-proxy, then re-ran the choria request_cert command and it worked. I then attempted to do another remote puppet run, which failed again, so I ended up changing the user in

/etc/foreman-proxy/settings.d/puppet_proxy_mcollective.yml to:

:user: foreman-proxy

restarted the proxy service and now everything works.

Given, I couldn't figure out how to get it all working as the original, mcollective choria user I created, it works under foreman-proxy and thats good enough for me.

I hope all this information helps someone that might be finding this from a search engine.
Good luck.

Actions
Copy link
 #4

Updated by Jeff Sparrow over 6 years ago

  • Status changed from New to Resolved
Actions
Copy link

Also available in: Atom PDF