Bug #37352
open
Foreman unable to launch virtual console from Foreman provisioned guests on EL9 KVM/Libvirt host set_password failure
Description
Foreman 3.9.1 deployed on EL8.
provisioning virtual machines using the libvirt compute resource to a KVM host running EL9.3 successfully provisions the guest, but fails to support remote virtual console connection.
Launching the virtual console during or post provision generates the on screen error message.
"Failed to set console: Call to virDomainUpdateDeviceFlags failed: internal error: unable to execute QEMU command 'set_password': Could not set password"
the EL9.3 KVM host is running the following component versions
libvirt-libs-9.5.0-7.2.el9_3.x86_64
libvirt-client-9.5.0-7.2.el9_3.x86_64
ibvirt-daemon-9.5.0-7.2.el9_3.x86_64
libvirt-9.5.0-7.2.el9_3.x86_64
Foreman logging supports the error message with the following output on a provisioned guest referenced as "will' for a hostname
2024-04-15T10:53:19 [I|app|c0e262a8] Started GET "/hosts/will.no-dns.co.uk/console" for 10.11.216.109 at 2024-04-15 10:53:19 +0100
2024-04-15T10:53:19 [I|app|c0e262a8] Processing by HostsController#console as HTML
2024-04-15T10:53:19 [I|app|c0e262a8] Parameters: {"id"=>"will.no-dns.co.uk"}
2024-04-15T10:53:20 [W|app|c0e262a8] Failed to set console
2024-04-15T10:53:20 [I|app|c0e262a8] Backtrace for 'Failed to set console' error (Libvirt::Error): Call to virDomainUpdateDeviceFlags failed: internal error: unable to execute QEMU command 'set_password': Could not set password
/usr/share/gems/gems/fog-libvirt-0.12.0/lib/fog/libvirt/requests/compute/update_display.rb:22:in `update_device'
/usr/share/gems/gems/fog-libvirt-0.12.0/lib/fog/libvirt/requests/compute/update_display.rb:22:in `update_display'
/usr/share/gems/gems/fog-libvirt-0.12.0/lib/fog/libvirt/models/compute/server.rb:230:in `update_display'
/usr/share/foreman/app/models/compute_resources/foreman/model/libvirt.rb:184:in `console'
/usr/share/foreman/app/controllers/hosts_controller.rb:357:in `console'
/usr/share/gems/gems/actionpack-6.1.7.6/lib/action_controller/metal/basic_implicit_render.rb:6:in `send_action'
/usr/share/gems/gems/actionpack-6.1.7.6/lib/abstract_controller/base.rb:228:in `process_action'
/usr/share/gems/gems/actionpack-6.1.7.6/lib/action_controller/metal/rendering.rb:30:in `process_action'
/usr/share/gems/gems/actionpack-6.1.7.6/lib/abstract_controller/callbacks.rb:42:in `block in process_action'
/usr/share/gems/gems/activesupport-6.1.7.6/lib/active_support/callbacks.rb:117:in `block in run_callbacks'
/usr/share/foreman/app/controllers/concerns/foreman/controller/timezone.rb:10:in `set_timezone'
/usr/share/gems/gems/activesupport-6.1.7.6/lib/active_support/callbacks.rb:126:in `block in run_callbacks'
/usr/share/foreman/app/models/concerns/foreman/thread_session.rb:32:in `clear_thread'
/usr/share/gems/gems/activesupport-6.1.7.6/lib/active_support/callbacks.rb:126:in `block in run_callbacks'
/usr/share/foreman/app/controllers/concerns/foreman/controller/topbar_sweeper.rb:12:in `set_topbar_sweeper_controller'
/usr/share/gems/gems/activesupport-6.1.7.6/lib/active_support/callbacks.rb:126:in `block in run_callbacks'
/usr/share/gems/gems/audited-5.4.2/lib/audited/sweeper.rb:16:in `around'
/usr/share/gems/gems/activesupport-6.1.7.6/lib/active_support/callbacks.rb:126:in `block in run_callbacks'
/usr/share/gems/gems/audited-5.4.2/lib/audited/sweeper.rb:16:in `around'
/usr/share/gems/gems/activesupport-6.1.7.6/lib/active_support/callbacks.rb:126:in `block in run_callbacks'
/usr/share/gems/gems/activesupport-6.1.7.6/lib/active_support/callbacks.rb:137:in `run_callbacks'
/usr/share/gems/gems/actionpack-6.1.7.6/lib/abstract_controller/callbacks.rb:41:in `process_action'
/usr/share/gems/gems/actionpack-6.1.7.6/lib/action_controller/metal/rescue.rb:22:in `process_action'
/usr/share/gems/gems/actionpack-6.1.7.6/lib/action_controller/metal/instrumentation.rb:34:in `block in process_action'
/usr/share/gems/gems/activesupport-6.1.7.6/lib/active_support/notifications.rb:203:in `block in instrument'
/usr/share/gems/gems/activesupport-6.1.7.6/lib/active_support/notifications/instrumenter.rb:24:in `instrument'
/usr/share/gems/gems/activesupport-6.1.7.6/lib/active_support/notifications.rb:203:in `instrument'
/usr/share/gems/gems/actionpack-6.1.7.6/lib/action_controller/metal/instrumentation.rb:33:in `process_action'
/usr/share/gems/gems/actionpack-6.1.7.6/lib/action_controller/metal/params_wrapper.rb:249:in `process_action'
/usr/share/gems/gems/activerecord-6.1.7.6/lib/active_record/railties/controller_runtime.rb:27:in `process_action'
/usr/share/gems/gems/actionpack-6.1.7.6/lib/abstract_controller/base.rb:165:in `process'
/usr/share/gems/gems/actionview-6.1.7.6/lib/action_view/rendering.rb:39:in `process'
/usr/share/gems/gems/actionpack-6.1.7.6/lib/action_controller/metal.rb:190:in `dispatch'
/usr/share/gems/gems/actionpack-6.1.7.6/lib/action_controller/metal.rb:254:in `dispatch'
/usr/share/gems/gems/actionpack-6.1.7.6/lib/action_dispatch/routing/route_set.rb:50:in `dispatch'
/usr/share/gems/gems/actionpack-6.1.7.6/lib/action_dispatch/routing/route_set.rb:33:in `serve'
/usr/share/gems/gems/actionpack-6.1.7.6/lib/action_dispatch/journey/router.rb:50:in `block in serve'
/usr/share/gems/gems/actionpack-6.1.7.6/lib/action_dispatch/journey/router.rb:32:in `each'
/usr/share/gems/gems/actionpack-6.1.7.6/lib/action_dispatch/journey/router.rb:32:in `serve'
/usr/share/gems/gems/actionpack-6.1.7.6/lib/action_dispatch/routing/route_set.rb:842:in `call'
/usr/share/gems/gems/apipie-dsl-2.6.1/lib/apipie_dsl/static_dispatcher.rb:67:in `call'
/usr/share/gems/gems/apipie-rails-1.2.3/lib/apipie/static_dispatcher.rb:68:in `call'
/usr/share/gems/gems/actionpack-6.1.7.6/lib/action_dispatch/middleware/static.rb:24:in `call'
/usr/share/gems/gems/actionpack-6.1.7.6/lib/action_dispatch/middleware/static.rb:24:in `call'
/usr/share/gems/gems/actionpack-6.1.7.6/lib/action_dispatch/middleware/static.rb:24:in `call'
/usr/share/gems/gems/actionpack-6.1.7.6/lib/action_dispatch/middleware/static.rb:24:in `call'
/usr/share/gems/gems/actionpack-6.1.7.6/lib/action_dispatch/middleware/static.rb:24:in `call'
/usr/share/gems/gems/actionpack-6.1.7.6/lib/action_dispatch/middleware/static.rb:24:in `call'
/usr/share/gems/gems/actionpack-6.1.7.6/lib/action_dispatch/middleware/static.rb:24:in `call'
/usr/share/foreman/lib/foreman/middleware/libvirt_connection_cleaner.rb:9:in `call'
/usr/share/foreman/lib/foreman/middleware/telemetry.rb:10:in `call'
/usr/share/gems/gems/apipie-rails-1.2.3/lib/apipie/middleware/checksum_in_headers.rb:27:in `call'
/usr/share/gems/gems/rack-2.2.8/lib/rack/tempfile_reaper.rb:15:in `call'
/usr/share/gems/gems/rack-2.2.8/lib/rack/etag.rb:27:in `call'
/usr/share/gems/gems/rack-2.2.8/lib/rack/conditional_get.rb:27:in `call'
/usr/share/gems/gems/rack-2.2.8/lib/rack/head.rb:12:in `call'
/usr/share/gems/gems/actionpack-6.1.7.6/lib/action_dispatch/http/permissions_policy.rb:22:in `call'
/usr/share/gems/gems/actionpack-6.1.7.6/lib/action_dispatch/http/content_security_policy.rb:19:in `call'
/usr/share/foreman/lib/foreman/middleware/logging_context_session.rb:22:in `call'
/usr/share/gems/gems/rack-2.2.8/lib/rack/session/abstract/id.rb:266:in `context'
/usr/share/gems/gems/rack-2.2.8/lib/rack/session/abstract/id.rb:260:in `call'
/usr/share/gems/gems/actionpack-6.1.7.6/lib/action_dispatch/middleware/cookies.rb:697:in `call'
/usr/share/gems/gems/actionpack-6.1.7.6/lib/action_dispatch/middleware/callbacks.rb:27:in `block in call'
/usr/share/gems/gems/activesupport-6.1.7.6/lib/active_support/callbacks.rb:98:in `run_callbacks'
/usr/share/gems/gems/actionpack-6.1.7.6/lib/action_dispatch/middleware/callbacks.rb:26:in `call'
/usr/share/gems/gems/actionpack-6.1.7.6/lib/action_dispatch/middleware/actionable_exceptions.rb:18:in `call'
/usr/share/gems/gems/actionpack-6.1.7.6/lib/action_dispatch/middleware/debug_exceptions.rb:29:in `call'
/usr/share/gems/gems/actionpack-6.1.7.6/lib/action_dispatch/middleware/show_exceptions.rb:33:in `call'
/usr/share/gems/gems/railties-6.1.7.6/lib/rails/rack/logger.rb:37:in `call_app'
/usr/share/gems/gems/railties-6.1.7.6/lib/rails/rack/logger.rb:28:in `call'
/usr/share/gems/gems/sprockets-rails-3.4.2/lib/sprockets/rails/quiet_assets.rb:13:in `call'
/usr/share/foreman/lib/foreman/middleware/logging_context_request.rb:11:in `call'
/usr/share/gems/gems/actionpack-6.1.7.6/lib/action_dispatch/middleware/remote_ip.rb:81:in `call'
/usr/share/gems/gems/request_store-1.5.1/lib/request_store/middleware.rb:19:in `call'
/usr/share/gems/gems/actionpack-6.1.7.6/lib/action_dispatch/middleware/request_id.rb:26:in `call'
/usr/share/gems/gems/rack-2.2.8/lib/rack/method_override.rb:24:in `call'
/usr/share/gems/gems/rack-2.2.8/lib/rack/runtime.rb:22:in `call'
/usr/share/gems/gems/activesupport-6.1.7.6/lib/active_support/cache/strategy/local_cache_middleware.rb:29:in `call'
/usr/share/gems/gems/actionpack-6.1.7.6/lib/action_dispatch/middleware/executor.rb:14:in `call'
/usr/share/gems/gems/rack-2.2.8/lib/rack/sendfile.rb:110:in `call'
/usr/share/gems/gems/actionpack-6.1.7.6/lib/action_dispatch/middleware/ssl.rb:77:in `call'
/usr/share/gems/gems/actionpack-6.1.7.6/lib/action_dispatch/middleware/host_authorization.rb:142:in `call'
/usr/share/gems/gems/secure_headers-6.5.0/lib/secure_headers/middleware.rb:11:in `call'
/usr/share/gems/gems/railties-6.1.7.6/lib/rails/engine.rb:539:in `call'
/usr/share/gems/gems/railties-6.1.7.6/lib/rails/railtie.rb:207:in `public_send'
/usr/share/gems/gems/railties-6.1.7.6/lib/rails/railtie.rb:207:in `method_missing'
/usr/share/gems/gems/rack-2.2.8/lib/rack/urlmap.rb:74:in `block in call'
/usr/share/gems/gems/rack-2.2.8/lib/rack/urlmap.rb:58:in `each'
/usr/share/gems/gems/rack-2.2.8/lib/rack/urlmap.rb:58:in `call'
/usr/share/gems/gems/puma-6.4.0/lib/puma/configuration.rb:272:in `call'
/usr/share/gems/gems/puma-6.4.0/lib/puma/request.rb:100:in `block in handle_request'
/usr/share/gems/gems/puma-6.4.0/lib/puma/thread_pool.rb:378:in `with_force_shutdown'
/usr/share/gems/gems/puma-6.4.0/lib/puma/request.rb:99:in `handle_request'
/usr/share/gems/gems/puma-6.4.0/lib/puma/server.rb:443:in `process_client'
/usr/share/gems/gems/puma-6.4.0/lib/puma/server.rb:241:in `block in run'
/usr/share/gems/gems/puma-6.4.0/lib/puma/thread_pool.rb:155:in `block in spawn_thread'
/usr/share/gems/gems/logging-2.3.1/lib/logging/diagnostic_context.rb:474:in `block in create_with_logging_context'
2024-04-15T10:53:20 [E|app|c0e262a8] Failed to save:
2024-04-15T10:53:20 [I|app|c0e262a8] Redirected to https://jarvis.no-dns.co.uk/new/hosts/will.no-dns.co.uk
2024-04-15T10:53:20 [I|app|c0e262a8] Completed 302 Found in 843ms (ActiveRecord: 170.9ms | Allocations: 11671)
2024-04-15T10:53:20 [I|app|7ceb1129] Started GET "/new/hosts/will.no-dns.co.uk" for 10.11.216.109 at 2024-04-15 10:53:20 +0100
2024-04-15T10:53:20 [I|app|7ceb1129] Processing by ReactController#index as HTML
2024-04-15T10:53:20 [I|app|7ceb1129] Parameters: {"id"=>"will.no-dns.co.uk"}
2024-04-15T10:53:20 [I|app|47bf21cb] Started GET "/notification_recipients" for 10.11.216.109 at 2024-04-15 10:53:20 +0100
2024-04-15T10:53:20 [I|app|7ceb1129] Rendered react/index.html.erb within layouts/react_application (Duration: 0.9ms | Allocations: 134)
2024-04-15T10:53:20 [I|app|47bf21cb] Processing by NotificationRecipientsController#index as JSON
2024-04-15T10:53:20 [I|app|47bf21cb] Completed 200 OK in 106ms (Views: 0.2ms | ActiveRecord: 88.1ms | Allocations: 4109)
2024-04-15T10:53:20 [I|app|7ceb1129] Rendered layouts/base.html.erb (Duration: 245.3ms | Allocations: 43497)
2024-04-15T10:53:20 [I|app|7ceb1129] Rendered layout layouts/react_application.html.erb (Duration: 248.2ms | Allocations: 44041)
2024-04-15T10:53:20 [I|app|7ceb1129] Completed 200 OK in 315ms (Views: 246.7ms | ActiveRecord: 17.3ms | Allocations: 51221)
2024-04-15T10:53:21 [I|app|2d8c6e13] Started GET "/notification_recipients" for 10.11.216.109 at 2024-04-15 10:53:21 +0100
2024-04-15T10:53:21 [I|app|87bb4e08] Started GET "/api/v2/remote_execution_features" for 10.11.216.109 at 2024-04-15 10:53:21 +0100
2024-04-15T10:53:21 [I|app|475beac8] Started GET "/hosts/will.no-dns.co.uk/statuses" for 10.11.216.109 at 2024-04-15 10:53:21 +0100
2024-04-15T10:53:21 [I|app|92f7ddb6] Started GET "/api/hosts/will.no-dns.co.uk?show_hidden_parameters=true" for 10.11.216.109 at 2024-04-15 10:53:21 +0100
2024-04-15T10:53:21 [I|app|2d8c6e13] Processing by NotificationRecipientsController#index as JSON
2024-04-15T10:53:21 [I|app|d53ad1be] Started GET "/api/audits?search=host%3Dwill.no-dns.co.uk&per_page=3" for 10.11.216.109 at 2024-04-15 10:53:21 +0100
2024-04-15T10:53:21 [I|app|dbbfc1ac] Started GET "/api/hosts/will.no-dns.co.uk/power?timeout=30" for 10.11.216.109 at 2024-04-15 10:53:21 +0100
2024-04-15T10:53:21 [I|app|475beac8] Processing by HostsController#statuses as JSON
2024-04-15T10:53:21 [I|app|475beac8] Parameters: {"id"=>"will.no-dns.co.uk"}
2024-04-15T10:53:21 [I|app|87bb4e08] Processing by Api::V2::RemoteExecutionFeaturesController#index as JSON
2024-04-15T10:53:21 [I|app|87bb4e08] Parameters: {"apiv"=>"v2"}
2024-04-15T10:53:21 [I|app|d53ad1be] Processing by Api::V2::AuditsController#index as JSON
2024-04-15T10:53:21 [I|app|d53ad1be] Parameters: {"search"=>"host=will.no-dns.co.uk", "per_page"=>"3", "apiv"=>"v2"}
2024-04-15T10:53:21 [I|app|92f7ddb6] Processing by Api::V2::HostsController#show as JSON
2024-04-15T10:53:21 [I|app|92f7ddb6] Parameters: {"show_hidden_parameters"=>"true", "apiv"=>"v2", "id"=>"will.no-dns.co.uk"}
2024-04-15T10:53:21 [I|app|dbbfc1ac] Processing by Api::V2::HostsController#power_status as JSON
2024-04-15T10:53:21 [I|app|dbbfc1ac] Parameters: {"timeout"=>"30", "apiv"=>"v2", "id"=>"will.no-dns.co.uk"}
2024-04-15T10:53:21 [I|app|2d8c6e13] Completed 200 OK in 96ms (Views: 0.2ms | ActiveRecord: 76.2ms | Allocations: 4114)
2024-04-15T10:53:21 [I|app|d53ad1be] Rendered api/v2/audits/index.json.rabl within api/v2/layouts/index_layout (Duration: 185.1ms | Allocations: 19039)
2024-04-15T10:53:21 [I|app|d53ad1be] Rendered layout api/v2/layouts/index_layout.json.erb (Duration: 211.5ms | Allocations: 26100)
2024-04-15T10:53:21 [I|app|d53ad1be] Completed 200 OK in 332ms (Views: 156.1ms | ActiveRecord: 143.3ms | Allocations: 33935)
2024-04-15T10:53:21 [I|app|475beac8] Completed 200 OK in 349ms (Views: 0.6ms | ActiveRecord: 202.0ms | Allocations: 31962)
2024-04-15T10:53:21 [I|app|87bb4e08] Rendered /usr/share/gems/gems/foreman_remote_execution-12.0.5/app/views/api/v2/remote_execution_features/index.json.rabl within api/v2/layouts/index_layout (Duration: 261.3ms | Allocations: 23396)
2024-04-15T10:53:21 [I|app|87bb4e08] Rendered layout api/v2/layouts/index_layout.json.erb (Duration: 269.3ms | Allocations: 29380)
2024-04-15T10:53:21 [I|app|87bb4e08] Completed 200 OK in 393ms (Views: 125.6ms | ActiveRecord: 237.4ms | Allocations: 36172)
2024-04-15T10:53:21 [I|app|dbbfc1ac] Completed 200 OK in 822ms (Views: 0.4ms | ActiveRecord: 99.1ms | Allocations: 13833)
2024-04-15T10:53:22 [I|app|92f7ddb6] Rendered api/v2/hosts/show.json.rabl (Duration: 889.5ms | Allocations: 127613)
2024-04-15T10:53:22 [I|app|92f7ddb6] Completed 200 OK in 1087ms (Views: 465.1ms | ActiveRecord: 548.3ms | Allocations: 141855)
2024-04-15T10:53:22 [I|app|d4261536] Started GET "/api/v2/hosts/170/available_remote_execution_features" for 10.11.216.109 at 2024-04-15 10:53:22 +0100
2024-04-15T10:53:22 [I|app|638825d1] Started GET "/job_invocations/preview_job_invocations_per_host?host_id=170&status=failed+or+status%3D+succeeded&limit=3" for 10.11.216.109 at 2024-04-15 10:53:22 +0100
2024-04-15T10:53:22 [I|app|d4261536] Processing by Api::V2::RemoteExecutionFeaturesController#available_remote_execution_features as JSON
2024-04-15T10:53:22 [I|app|d4261536] Parameters: {"apiv"=>"v2", "id"=>"170"}
2024-04-15T10:53:22 [I|app|638825d1] Processing by JobInvocationsController#preview_job_invocations_per_host as JSON
2024-04-15T10:53:22 [I|app|638825d1] Parameters: {"host_id"=>"170", "status"=>"failed or status= succeeded", "limit"=>"3"}
2024-04-15T10:53:22 [I|app|638825d1] Completed 200 OK in 100ms (Views: 0.3ms | ActiveRecord: 63.9ms | Allocations: 8416)
2024-04-15T10:53:22 [I|app|d4261536] Rendered /usr/share/gems/gems/foreman_remote_execution-12.0.5/app/views/api/v2/remote_execution_features/available_remote_execution_features.json.rabl (Duration: 156.3ms | Allocations: 24716)
on the KVM host, the only output referencing the guest is a deprecated video card driver, which is useful to know, but I believe not part of this bug, foreman needs to be changed to use a different default virtual video card when provisioning guests.
2024-04-02T16:15:50.209668Z qemu-kvm: -device {"driver":"cirrus-vga","id":"video0","bus":"pcie.0","addr":"0x1"}: warning: 'cirrus-vga' is deprecated, please use a different VGA card instead
Updated by Jan Loeser 16 days ago
Matt Darcy wrote:
Foreman 3.9.1 deployed on EL8.
provisioning virtual machines using the libvirt compute resource to a KVM host running EL9.3 successfully provisions the guest, but fails to support remote virtual console connection.
QEMU only allows changing VNC passphrase if enabled qemu ... -vnc :1,password=on
http://patchwork.ozlabs.org/project/qemu-devel/patch/1386777271-12667-1-git-send-email-kraxel@redhat.com/
https://qemu-project.gitlab.io/qemu/system/vnc-security.html#with-passwords
QEMU is started by libvirt. Foreman/OR generated domain XML definition for VNC is:
<graphics type='vnc' port='-1' autoport='yes' listen='0.0.0.0'>
resulting in QEMU process:
qemu ... -vnc 0.0.0.0:2,audiodev=audio1
No passphrase set -> no passphrase at all -> passphrase can't be changed.
If any passphrase is provided in domain XML definition:
<graphics type='vnc' port='-1' autoport='yes' listen='0.0.0.0' passwd='foobar'>
resulting in QEMU process:
qemu ... -vnc 0.0.0.0:2,password=on,audiodev=audio1
Having this, VNC passphrase can be changed by Foreman/OR.
Workaround is to set/enable default passphrase in /etc/libvirt/qemu.conf.
IMO fog-libvirt should take care of creating domain XML definition with a passphrase set if checkbox "Console Passwords" is checked that it can change passphrases later on.
Updated by Matt Darcy 8 days ago
really useful context, especially the redhat thread stating 'this isn't backward compatible but it's the right thing to do' which explains why it has worked and isn't working, I'll find the rpm this update was pushed out in.
I acknowledge the work around, that's fine for my current development hosts, and I acknowledge fog-libvirt should handle this better.
Feels like this isn't a foreman problem to fix, unless foreman wants to generate the XML definition for a guest that forces the password to be set, and forces qemu to launch with the correct parameters.