Bug #34193: Slow DNS query getresources after upgrading tfm-runtime package - Smart Proxy - Foreman

Actions

Copy link

Bug #34193

closed

Slow DNS query getresources after upgrading tfm-runtime package

Added by F. Zicklam over 2 years ago. Updated over 2 years ago.

Status:

Closed

Priority:

Normal

Assignee:

Category:

DNS

Target version:

Difficulty:

Triaged:

Bugzilla link:

Pull request:

Fixed in Releases:

Found in Releases:

Foreman - 2.5.0, Foreman - 2.5.1, Foreman - 2.5.2, Foreman - 2.5.3, Foreman - 2.5.4, Foreman - 3.0.0, Foreman - 3.0.1, Foreman - 3.0.2

Red Hat JIRA:

Description

OS TheForeman: CentOS 7.9.2009
OS DNS Server: RHEL 7.9
Bind Version: 9.11.4.26.P2
Foreman Version: 3.0.1

We experience following issue after updating the additional packages for foreman-smart-proxy from repository version 2.5 and higher.

This warning message is after provisioning a new server on following steps:

Create IPv4 DNS record for servername1.domain.local

2021-12-22T11:47:35 48d893f0 [I] Started POST /dns/ 
2021-12-22T11:48:01 48d893f0 [W] Slow DNS query getresources for ["servername1.domain.local", Resolv::DNS::Resource::IN::A] took 26021.57 ms
2021-12-22T11:48:01 48d893f0 [D] Resolver used: #<Resolv::DNS:0x00007faf241e3798 @mutex=#<Thread::Mutex:0x00007faf241e3770>, @config=#<Resolv::DNS::Config:0x00007faf241e3748 @mutex=#<Thread::Mutex:0x00007faf241e3720>, @config_info={:nameserver=>"127.0.0.1"}, @initialized=true, @timeouts=[5, 8, 13], @nameserver_port=[["127.0.0.1", 53]], @search=[[]], @ndots=1>, @initialized=true>
2021-12-22T11:48:01 48d893f0 [D] running /usr/bin/nsupdate -k /etc/foreman-proxy/foreman_dns.key 
2021-12-22T11:48:01 48d893f0 [D] nsupdate: executed - server 127.0.0.1
2021-12-22T11:48:01 48d893f0 [D] nsupdate: executed - update add servername1.domain.local. 60 A 10.40.15.118
2021-12-22T11:48:01 48d893f0 [I] Finished POST /dns/ with 200 (26053.1 ms)

Create Reverse IPv4 DNS record for servername1.domain.local

2021-12-22T11:48:01 48d893f0 [I] Started POST /dns/ 
2021-12-22T11:48:27 48d893f0 [W] Slow DNS query getresources for ["118.15.40.10.in-addr.arpa", Resolv::DNS::Resource::IN::PTR] took 26027.94 ms
2021-12-22T11:48:27 48d893f0 [D] Resolver used: #<Resolv::DNS:0x00007faf2423dfb8 @mutex=#<Thread::Mutex:0x00007faf2423df90>, @config=#<Resolv::DNS::Config:0x00007faf2423df68 @mutex=#<Thread::Mutex:0x00007faf2423df40>, @config_info={:nameserver=>"127.0.0.1"}, @initialized=true, @timeouts=[5, 8, 13], @nameserver_port=[["127.0.0.1", 53]], @search=[[]], @ndots=1>, @initialized=true>
2021-12-22T11:48:27 48d893f0 [D] running /usr/bin/nsupdate -k /etc/foreman-proxy/foreman_dns.key 
2021-12-22T11:48:27 48d893f0 [D] nsupdate: executed - server 127.0.0.1
2021-12-22T11:48:27 48d893f0 [D] nsupdate: executed - update add 118.15.40.10.in-addr.arpa. 60 PTR servername1.domain.local
2021-12-22T11:48:27 48d893f0 [I] Finished POST /dns/ with 200 (26054.26 ms)

This issue exist, when using the Smart Proxy Modul "DNS for nsupdate" after Upgrading the Smart Proxy above Version 2.5.

We tried various versions to get this issue fixed, but it is not related to the foreman-proxy package, it is caused after upgrading the package tfm-runtime

Every Foreman SmartProxy Version work (2.5 - 3.0) until you upgrade the tfm-runtime package to version "tfm-runtime-7.0-4.el7.x86_64".

You can reproduce this behavior with downgrading back to "tfm-runtime-6.1-4.el7.x86_64" (working) or upgrade to "tfm-runtime-7.0-4.el7.x86_64" (not working or slow dns queries).

Someone else reported this behavior already on community forum: https://community.theforeman.org/t/foreman-domains-dns-proxy-slow/22558/3

Actions

Copy link

Updated by F. Zicklam over 2 years ago

Subject changed from Slow DNS query getresources Resolv::DNS::Resource::IN::A (tfm-runtime) to Slow DNS query getresources after upgrading tfm-runtime package

Actions

Copy link

Updated by F. Zicklam over 2 years ago

All packages are from official repositories.

Working package is in 2.4 repository:

https://yum.theforeman.org/releases/2.4/el7/x86_64/

Not working in all higher then 2.5 repositories with tfm-runtime version higher 7.0.0

Actions

Copy link

Updated by Lukas Zapletal over 2 years ago

Hello,

that error means that Ruby runtime method getresrouces (https://devdocs.io/ruby~2.7/resolv/dns#method-i-getresources) was slow and perhaps did not return any result due to timeout. Ruby uses DNS stub implementation which actually performs the DNS lookup by itself not utilizing libc. From logs it looks like it tries to contact localhost on 53.

Do you have IPv6 on that host? I remember there were some bugs around Ruby DNS resolving but I can't recall what was it. But IPv6 in /etc/hosts or resolv.conf was breaking Ruby DNS IIRC.

Can you reproduce this behavior if you create a Ruby script trying to resolve the very same name on that host? I am afraid this looks like a Ruby bug, a reproducer would be necessary and then creating BZ for Red Hat would be better.

Actions

Copy link

Updated by F. Zicklam over 2 years ago

Hello Lukas,

thank you for your help.

I tried to change some settings, removed unused hosts at /etc/hosts and disabled IPv6 on my interfaces, now the duration decreased from 26 seconds to 3 seconds for this step/request.

With the old tfm-runtime package it just took 1.6 milliseconds for the same request.

If i will find more tuning options, i will let you know.

Thank you,
regards
Flo

Actions

Copy link