We already do on RPM, but not on Debian.

Quoting Ewoud from https://github.com/theforeman/forklift/pull/1339:

So perhaps this is a time to normalize this. We have https://github.com/theforeman/foreman-packaging/blob/rpm/develop/packages/foreman/foreman/foreman.tmpfiles but is that something that should live in foreman.git's extras or do we not need it at all?

Also, systemd can create a `/run` directory for us if need it. Perhaps that's enough. It will mean that the caches are flushed on service restart, but perhaps that's good? What I see now on my install:

# find /run/foreman/cache/ -type f
/run/foreman/cache/A5E/FE0/settings%2Foauth_map_users
/run/foreman/cache/90B/200/settings%2Foauth_active
/run/foreman/cache/CDB/C60/settings%2Fssl_client_verify_env
/run/foreman/cache/BF4/FC0/settings%2Fssl_client_cert_env
/run/foreman/cache/C07/D50/settings%2Fwebsockets_ssl_cert
/run/foreman/cache/BA2/ED0/settings%2Fwebsockets_ssl_key
/run/foreman/cache/568/6C0/notification-3
/run/foreman/cache/7D5/D00/template_kind_names
/run/foreman/cache/156/E01/failed_login_SOME_IP
/run/foreman/cache/08F/951/failed_login_SOME_IP
/run/foreman/cache/C95/C70/hosts_count%2Foperatingsystem%2F3
/run/foreman/cache/973/190/c40a9904-384e-41a3-ad61-ad0d96e8b3e8
/run/foreman/cache/A32/E20/settings%2Fsafemode_render
/run/foreman/cache/952/2C0/8665fc9d-00f1-4d19-a660-7b958bca904f
/run/foreman/cache/A39/010/settings%2Fmanage_puppetca
/run/foreman/cache/9F5/030/settings%2Ftoken_duration
/run/foreman/cache/98C/460/name_generator_register
/run/foreman/cache/029/AA1/settings%2Fupdate_environment_from_facts
/run/foreman/cache/0EC/8E1/failed_login_SOME_IP
/run/foreman/cache/917/2C0/879f167a-702f-4f7e-9478-e8c248967bf8
/run/foreman/cache/1A4/961/failed_login_SOME_IP
/run/foreman/cache/A88/C01/settings%2Fauthorize_login_delegation_auth_source_user_autocreate
/run/foreman/cache/8BB/560/settings%2Fbcrypt_cost
/run/foreman/cache/17A/E01/failed_login_SOME_IP
/run/foreman/cache/93D/C90/c5537768-8cbe-45c8-99ca-e6ed78075971
/run/foreman/cache/93E/6C0/ba92ba47-0086-4cc6-9324-936d02ebcc17
/run/foreman/cache/8B1/850/26e0824b-0d80-43f7-8040-f43a624768ca
/run/foreman/cache/0A7/FE1/failed_login_SOME_IP
/run/foreman/cache/A2E/EF0/49c12cc4-4c02-43cd-b7e5-cb0fbfe5f48c
/run/foreman/cache/185/231/failed_login_SOME_IP
/run/foreman/cache/0CD/911/failed_login_SOME_IP
/run/foreman/cache/8B8/560/123658a0-6af8-4000-929c-41a9e87646bd
/run/foreman/cache/E05/ED0/settings%2Fmonitoring_create_action
/run/foreman/cache/870/4F0/settings%2Fhttp_proxy
/run/foreman/cache/D1F/460/settings%2Fproxy_request_timeout
/run/foreman/cache/F94/FA0/settings%2Fappend_domain_name_for_hosts
/run/foreman/cache/857/150/settings%2Fhost_owner
/run/foreman/cache/BF7/030/settings%2Fname_generator_type
/run/foreman/cache/E6D/DB0/settings%2Fuse_uuid_for_certificates
/run/foreman/cache/B5C/500/settings%2Fhost_power_status
/run/foreman/cache/DE6/340/settings%2Fall_out_of_sync_disabled
/run/foreman/cache/9E5/C10/settings%2Finstance_title
/run/foreman/cache/8FC/FB0/settings%2Flab_features
/run/foreman/cache/84C/9B0/settings%2Flogin_text
/run/foreman/cache/D80/AC0/settings%2Fignore_facts_for_domain
/run/foreman/cache/176/961/settings%2Fignore_facts_for_operatingsystem
/run/foreman/cache/F5F/670/settings%2Fupdate_hostgroup_from_facts
/run/foreman/cache/C68/D40/settings%2Fdefault_organization
/run/foreman/cache/B21/250/settings%2Forganization_fact
/run/foreman/cache/AAC/8F0/settings%2Fdefault_location
/run/foreman/cache/965/440/settings%2Flocation_fact
/run/foreman/cache/9CD/EF0/settings%2Fexcluded_facts
/run/foreman/cache/E31/EE0/settings%2Fmaximum_structured_facts
/run/foreman/cache/40E/101/settings%2Fcreate_new_host_when_facts_are_uploaded
/run/foreman/cache/BBD/B80/settings%2Foutofsync_interval
/run/foreman/cache/A71/4D0/settings%2Fpuppet_interval
/run/foreman/cache/F4B/890/settings%2Fpuppet_out_of_sync_disabled
/run/foreman/cache/A59/6A0/settings%2Fenc_environment
/run/foreman/cache/32C/C71/settings%2Fignore_puppet_facts_for_provisioning
/run/foreman/cache/017/731/settings%2Finterpolate_erb_in_parameters
/run/foreman/cache/8A9/160/settings%2Fforeman_url
/run/foreman/cache/82D/AE0/settings%2Frss_enable
/run/foreman/cache/719/A00/settings%2Frss_url
/run/foreman/cache/0F0/B51/settings%2Fmonitoring_affect_global_status
/run/foreman/cache/7E9/D60/settings%2Froot_pass
/run/foreman/cache/913/A80/settings%2Fidle_timeout
/run/foreman/cache/CE0/390/settings%2Foauth_consumer_secret
/run/foreman/cache/BA3/E40/settings%2Foauth_consumer_key
/run/foreman/cache/EDD/530/settings%2Fauthorize_login_delegation
/run/foreman/cache/B60/5B0/failed_login_SOME_IP
/run/foreman/cache/F3B/E00/settings%2Ffailed_login_attempts_limit
/run/foreman/cache/9AA/2B0/settings%2Ftrusted_hosts
/run/foreman/cache/EAC/3A0/settings%2Frequire_ssl_smart_proxies
/run/foreman/cache/1FB/711/settings%2Frestrict_registered_smart_proxies
/run/foreman/cache/AAB/C90/settings%2Fentries_per_page

Failed logins probably make sense to keep around. Settings are probably not that important and cheap, with perhaps the exception of bcrypt_cost. All the others are ActiveRecord caches.

Foreman really does surprisingly little caching. That makes me think we should keep tmpfiles around and align Debian to what RPMs already do. Thoughts?

Note that I think it doesn't have to block this PR since we won't have immediate cherry picks, but if we agree on that we can open an issue and link it here why we need the workaround.

This is a tracker -- will need changes to installer, packaging and core git.