Project

General

Profile

Actions

Bug #10651

closed

Excessive failed unlink calls to /usr/share/foreman/tmp/cache

Added by Paul Arnold almost 9 years ago. Updated almost 7 years ago.

Status:
Resolved
Priority:
Normal
Assignee:
-
Category:
Rails
Target version:
-
Difficulty:
Triaged:
Fixed in Releases:
Found in Releases:

Description

We have a stringent requirement for audit logging, and Foreman has been producing a lot of failed unlink events, such as:

[root@xxxxx ~]# ausearch -a 2193623 -i
----
node=xxxxx type=PATH msg=audit(05/27/2015 00:23:00.557:2193623) : item=0 name=/usr/share/foreman/tmp/cache/ inode=261691 dev=08:05 mode=dir,750 ouid=foreman ogid=foreman rdev=00:00 obj=unconfined_u:object_r:foreman_var_run_t:s0 nametype=PARENT
node=xxxxx type=CWD msg=audit(05/27/2015 00:23:00.557:2193623) : cwd=/usr/share/foreman
node=xxxxx type=SYSCALL msg=audit(05/27/2015 00:23:00.557:2193623) : arch=x86_64 syscall=unlink success=no exit=-2(No such file or directory) a0=0x7f3aff591b00 a1=0x7f3afd034578 a2=0x0 a3=0x1c08 items=1 ppid=1 pid=25198 auid=xxxxx uid=foreman gid=foreman euid=foreman suid=foreman fsuid=foreman egid=foreman sgid=foreman fsgid=foreman tty=(none) ses=1301 comm=ruby exe=/opt/rh/ruby193/root/usr/bin/ruby subj=unconfined_u:system_r:httpd_t:s0 key=delete

We can work around this by excluding this comm on this directory for the time-being, but it really is producing a lot of events:
[root@xxxxx ~]# ausearch -i -m SYSCALL -sc unlink -sv no -c ruby -f '/usr/share/foreman/tmp/cache/' -ui foreman -ts yesterday | grep -c 'type=SYSCALL'
217690

('ausearch' has a ton of options: this is interpret syscall messages, only unlinks, only failures, only from ruby, only in the cache path, only from user foreman, only yesterday)

For reference, the entirety of failed SYSCALLs yesterday:
[root@xxxxx ~]# ausearch -i -m SYSCALL -sv no -ts yesterday | grep -c 'type=SYSCALL'
252182

Some system info:

[root@xxxxx ~]# rpm -qa | grep -i foreman | sort
foreman-1.8.1-1.el6.noarch
foreman-cli-1.8.1-1.el6.noarch
foreman-debug-1.8.1-1.el6.noarch
foreman-postgresql-1.8.1-1.el6.noarch
foreman-proxy-1.8.1-1.el6.noarch
foreman-release-1.8.1-1.el6.noarch
foreman-selinux-1.8.1-1.el6.noarch
ruby193-rubygem-foreman_openscap-0.4.0-1.el6.noarch
ruby193-rubygem-foreman_setup-2.1.1-1.el6.noarch
ruby193-rubygem-puppetdb_foreman-0.1.2-1.el6.noarch
rubygem-hammer_cli_foreman-0.2.0-1.el6.noarch
[root@xxxxx ~]# cat /etc/redhat-release && uname -r
Red Hat Enterprise Linux Server release 6.6 (Santiago)
2.6.32-504.16.2.el6.x86_64

Using Apache+Passenger: Apache/2.2.15 (Unix) DAV/2 mod_nss/2.2.15 NSS/3.15.1 Basic ECC Phusion_Passenger/5.0.6 configured

Actions #1

Updated by Dominic Cleal almost 9 years ago

  • Category set to Rails

It's likely that the cause of this is in Rails, either how the cache implementation is written or in some Rails component that uses the cache heavily. Foreman itself only uses it a bit, and only deletes entries on a couple of pages (assuming that unlink is from a Rails.cache.delete call).

You might find behaviour changes as we change Rails versions (e.g. Rails 4 in the next couple of versions) or you could even swap the cache to something like memcached with https://github.com/theforeman/foreman_memcache.

Actions #2

Updated by Anonymous almost 7 years ago

  • Status changed from New to Resolved

should hopefully be better with rails 4.2

Actions

Also available in: Atom PDF