将设为首页浏览此站
开启辅助访问 天气与日历 收藏本站联系我们切换到窄版

易陆发现论坛

 找回密码
 开始注册
查看: 178|回复: 2
收起左侧

system hang after hostname systemd: Started Cleanup of Temporary Directories.

[复制链接]
发表于 2021-12-16 13:57:57 | 显示全部楼层 |阅读模式

马上注册,结交更多好友,享用更多功能,让你轻松玩转社区。

您需要 登录 才可以下载或查看,没有帐号?开始注册

x
system hang after Started Cleanup of Temporary Directories compute04 systemd: Started Cleanup of Temporary Directories.* f" [! V, i. e# N5 F8 X8 t

* k1 K3 e5 \: w

( \4 m+ l  P) ESep  6 16:58:27 compute04 systemd: Starting Cleanup of Temporary Directories...0 i0 i3 {8 ~/ i6 j0 N& w, K
Sep  6 16:58:27 compute04 systemd: Started Cleanup of Temporary Directories.( |, D. D4 y+ }2 S
Sep  6 17:01:01 compute04 systemd: Started Session 541 of user root.: R8 H/ d. U5 F! Q0 Q! c) v' k' z$ ~8 V
Sep  6 18:01:01 compute04 systemd: Started Session 543 of user root.) n, m$ R: F( s
Sep  6 19:01:01 compute04 systemd: Started Session 545 of user root.) [5 Z+ q# j, `5 C  K' Y
Sep  6 20:01:01 compute04 systemd: Started Session 547 of user root.+ c" [! o( e: S& w7 V0 L' N
Sep  6 21:01:01 compute04 systemd: Started Session 549 of user root.% p; n! a0 m" w; \8 A( d
Sep  6 22:01:01 compute04 systemd: Started Session 551 of user root.
1 r' J" r( u+ T: V6 mDec 16 13:37:13 compute04 kernel: Initializing cgroup subsys cpuset  c# @2 t0 W$ g
Dec 16 13:37:13 compute04 kernel: Initializing cgroup subsys cpu
3 s) n2 g6 E  Q: C4 v: D7 Z8 V1 w" Z0 e& V$ a
. N6 Z2 N4 F) Z3 c0 b

/ w1 j  [/ O4 Y. g' C2 G0 |6 e$ z# w" E# @. j6 V3 {- H
系统无辜夯死:# N& t6 }; c6 r/ A9 a3 b

0 W5 Q0 L# D- [) d" t  kat least, please test with v238 or newer, set SYSTEMD_LOG_LEVE=debug environment variable, and provide relevant debug logs
* J+ b% N$ k+ e6 ]$ _
0 E/ {- Z4 e( e/ j9 F
 楼主| 发表于 2021-12-16 13:59:57 | 显示全部楼层

Have you ever used a temporary directory? I'm guessing if you use a computer, you've used one of these. It's a core feature of nearly every operating-system.

To ensure system stability, you should always check that filesystems on which a temporary directory resides don't get full --- running out of space can quickly bring your system to a grinding halt.

One method to prevent running out of space could be to place those directories on a dedicated partition, but no matter the solution, it is a best practice to clean those directories periodically, based on your/your app's needs.

Red Hat Enterprise Linux (and Fedora as well) provides a more structured and configurable method to manage temporary directories and files: systemd-tmpfiles.

Once a Red Hat-like system boots up, a special unit file is executed: systemd-tmpfiles-setup, this unit will execute the systemd-tmpfile --create --remove command.

You can take a look at the systemd unit file by executing the following commands:

[root@freddy ~]# systemctl status systemd-tmpfiles-setup.service● systemd-tmpfiles-setup.service - Create Volatile Files and Directories   Loaded: loaded (/usr/lib/systemd/system/systemd-tmpfiles-setup.service; static; vendor preset: disabled)   Active: active (exited) since Thu 2016-08-18 20:06:04 CEST; 2 weeks 2 days ago     Docs: man:tmpfiles.d(5)           man:systemd-tmpfiles(8) Main PID: 1007 (code=exited, status=0/SUCCESS)    Tasks: 0 (limit: 512)   CGroup: /system.slice/systemd-tmpfiles-setup.serviceAug 18 20:06:04 freddy systemd[1]: Starting Create Volatile Files and Directories...Aug 18 20:06:04 freddy systemd[1]: Started Create Volatile Files and Directories.[root@freddy ~]# cat /usr/lib/systemd/system/systemd-tmpfiles-setup.service#  This file is part of systemd.##  systemd is free software; you can redistribute it and/or modify it#  under the terms of the GNU Lesser General Public License as published by#  the Free Software Foundation; either version 2.1 of the License, or#  (at your option) any later version.[Unit]Description=Create Volatile Files and DirectoriesDocumentation=man:tmpfiles.d(5) man:systemd-tmpfiles(8)DefaultDependencies=noConflicts=shutdown.targetAfter=local-fs.target systemd-sysusers.serviceBefore=sysinit.target shutdown.targetRefuseManualStop=yes[Service]Type=oneshotRemainAfterExit=yesExecStart=/usr/bin/systemd-tmpfiles --create --remove --boot --exclude-prefix=/dev

The previous command reads all the configuration files and execute directories' creation/clean up as specified.

Scheduled cleaning

The systemd-tmpfile service execution, at boot time, will ensure that all the required temporary directories are created --- but what about any scheduled clean up?

Systemd gives you the timer unit for controlling scheduled and cyclic actions to be performed in the running system.

To quote the systemd manual:

A unit configuration file whose name ends in ".timer" encodes information about a timer controlled and supervised by systemd, for timer-based activation.

For each timer file, a matching unit file must exist, describing the unit to activate when the timer elapses. By default, a service by the same name as the timer (except for the suffix) is activated. Example: a timer file foo.timer activates a matching service foo.service. The unit to activate may be controlled by Unit=.

You can inspect the behaviour of the timer unit by querying the systemd daemon:

[root@freddy ~]# systemctl status systemd-tmpfiles-clean.timer● systemd-tmpfiles-clean.timer - Daily Cleanup of Temporary Directories   Loaded: loaded (/usr/lib/systemd/system/systemd-tmpfiles-clean.timer; static; vendor preset: disabled)   Active: active (waiting) since Thu 2016-08-18 20:06:04 CEST; 2 weeks 2 days ago     Docs: man:tmpfiles.d(5)           man:systemd-tmpfiles(8)Aug 18 20:06:04 freddy systemd[1]: Started Daily Cleanup of Temporary Directories.[root@freddy ~]# cat /usr/lib/systemd/system/systemd-tmpfiles-clean.timer#  This file is part of systemd.##  systemd is free software; you can redistribute it and/or modify it#  under the terms of the GNU Lesser General Public License as published by#  the Free Software Foundation; either version 2.1 of the License, or#  (at your option) any later version.[Unit]Description=Daily Cleanup of Temporary DirectoriesDocumentation=man:tmpfiles.d(5) man:systemd-tmpfiles(8)[Timer]OnBootSec=15minOnUnitActiveSec=1d

As you'll find on the timer unit, it will run just 15 minutes after systemd start, and then every 24 hours from that time onward. The command, in this case, will only affect files/directories purging/cleaning.

Configuration files

Let's discuss systemd-tmpfiles configuration files.

Typically, for unix-like services, configuration files are located in different places and they have a hierarchical priority process. The configuration locations for systemd-tmpfiles service have the following order priority (Highest to lower):

  • /etc/tmpfiles.d/*.conf
  • /run/tmpfiles.d/*.conf
  • /usr/lib/tmpfiles.d/*.conf
    $ U' H/ |. Y( `' A

Supposing that a configuration file with the same name is placed under all the three configuration directories, the file with highest priority will be the one in /etc. The configuration files placed under /run are created at runtime by services/daemons to control temporary directory cleaning processes. And, as usual, configuration files under /usr/lib/* should be never edited directly because they're vendor-provided --- you should override them using this priority mechanism, placing a custom file under /etc/tmpfiles.d/*.

Looking at configuration file syntax, a systemd-tmpfiles configuration contains:

Type, Path, Mode, UID, GID, Age, and Arguments

You can inspect the installed configuration file and understand their behaviour looking at the systemd-tmpfiles man page.

We'll now explore some existing configuration files in the /usr/lib/tmpdfiles.d directory:

Directory creation[root@freddy ~]# cat /usr/lib/tmpfiles.d/httpd.confd /run/httpd 710 root apached /run/httpd/htcacheclean 700 apache apache

In this example, systemd will create the /run/httpd and /run/httpd/htcacheclean, if they don't yet exist, assigning them the given permission, UID and GID.

File removal[root@freddy ~]# cat /usr/lib/tmpfiles.d/rpm.confr /var/lib/rpm/__db.*

In this example, systemd will periodically clean up RPM database library files.

Symlink creation and recursive file copy[root@freddy ~]# cat /usr/lib/tmpfiles.d/etc.conf#  This file is part of systemd.##  systemd is free software; you can redistribute it and/or modify it#  under the terms of the GNU Lesser General Public License as published by#  the Free Software Foundation; either version 2.1 of the License, or#  (at your option) any later version.# See tmpfiles.d(5) for detailsL /etc/os-release - - - - ../usr/lib/os-releaseL /etc/localtime - - - - ../usr/share/zoneinfo/UTCL+ /etc/mtab - - - - ../proc/self/mountsC /etc/nsswitch.conf - - - -C /etc/pam.d - - - -

In this example, systemd will ensure the creation of symbolic links for /etc/os-release, /etc/localtime and /etc/mtab.

If suffixed with '+', and a file already exists where the symlink is to be created, the file will be removed and be replaced by the symlink.

Lastly ,"C" types for files/directories /etc/nsswitch.conf and /etc/pam.d, quoting the tmpfiles.d manual section:

Recursively copy a file or directory, if the destination files or directories do not exist yet. Note that this command will not descend into subdirectories if the destination directory already exists. Instead, the entire copy operation is skipped. If the argument is omitted, files from the source directory /usr/share/factory/ with the same name are copied. Does not follow symlinks.

Time option usage[root@freddy ~]# cat /usr/lib/tmpfiles.d/cups.conf# See tmpfiles.d(5) for detailsd /run/cups 0755 root lp -d /run/cups/certs 0511 lp sys -d /var/spool/cups/tmp - - - 30d

Finally, in this example, you can see the usage of Time options. Each file older than 30 days on /var/spool/cups/tmp will be deleted.

Note that the Time option designates that a file will be considered unused only if atime, mtime and ctime are all older than the specified time.

If you want to dig more on the available configuration types, just type "man 5 tmpfiles.d" on your shell, and in case you want to test just one configuration file at a time, you can place it as an argument to the systemd-tmpfiles command.

Feel free to leave comments or ask questions! Thanks for reading.

Last updated: October 18, 20186 n, M3 j, u' P" n8 d% a) W
 楼主| 发表于 2021-12-16 14:00:30 | 显示全部楼层
How to clean temporary files automatically using systemd-tmpfiles in RHEL 7 / CentOS 75 K- y2 n1 J$ i( A' e8 C
Have you also come across a situation when your filesystem gets piled up with unwanted files and you have to step in manually clean up all the unwanted temporary files and directories.
0 B4 f: B. Q, \9 Z- l- i. LWith Red Hat Enterprise Linux 7 we have a new systemd unit file (systemd-tmpfiles) introduced which can do the dirty work for you automatically.
" U: R& U  ~" P6 z3 SIn RHEL 6 we had a similar solution namely tmpwatch which used to clean up temporary files, the tmpwatch utility recursively searches through specified directories and removes files which have not been accessed in a specified period of time.
6 u' B5 e' l* v( Xsystemd-tmpfiles creates, deletes, and cleans up volatile and temporary files and directories, based on the configuration file format and location specified in tmpfiles.d9 o1 p! k* a( M! X  f7 T* j3 J
In my last article I have given an overview of systemd with examples.( I+ d/ ~% m# J. Q' ?
The configuration files are located in different places and they have a hierarchical priority process. The configuration locations for systemd-tmpfiles service have the following order priority (Highest to lower):- X6 }+ [3 ~1 m/ ^9 K
/etc/tmpfiles.d/*.conf
- Q! ~. J) O0 A# D/ A& c/run/tmpfiles.d/*.conf
0 L& R8 \: r$ r1 W0 q+ o6 d/usr/lib/tmpfiles.d/*.conf
$ V: |6 }8 c" i1 nSupposing that a configuration file with the same name is placed under all the three configuration directories, the file with highest priority will be the one in /etc.4 _. ?$ N4 M* D
The configuration files placed under /run are created at runtime by services/daemons to control temporary directory cleaning processes.
4 M: G3 O. ^3 fAnd, as usual, configuration files under /usr/lib/* should be never edited directly because they’re vendor-provided you should override them using this priority mechanism, placing a custom file under /etc/tmpfiles.d/*.) o( S  t# l! \3 ^) h, B" b
Looking at configuration file syntax, a systemd-tmpfiles configuration contains:
* p0 \3 v  `. o) i( TType, Path, Mode, UID, GID, Age, and Arguments
1 ]+ r  a: J. c6 q& ]! g. i* a8 Yr    Remove a file or directory if it exists. This may not be used to remove non-empty directories, use R for that. Lines of this type accept shell-style globs in place of normal path names.- F" f& v1 Y  I  U1 w/ q* X
For example I create a configuration file inside “/etc/tmpfiles.d“
* a  |- P* \9 \# cat tmp.conf
5 \/ y* c2 E( s' [2 I. K% @/ c5 d4 d# Remove /tmp/test directory and its content
/ K6 d: u3 ]$ o+ \1 h: O4 _r /tmp/test/
7 f7 v$ f: r1 k  V# F+ u7 E9 ?Before running this I will create a directory. A" m" }& l- a- q7 m2 X+ j' N
# mkdir /tmp/test
  W' r# o' M" R8 c3 R# XLets manually run systemd-tmpfiles to check if this work8 q, Y) t1 ~* N
# env SYSTEMD_LOG_LEVEL=debug systemd-tmpfiles –remove% s: B, o  l5 b; A
Skipping overridden file: /usr/lib/tmpfiles.d/tmp.conf.
1 }/ V& \. {9 V* IReading config file “/etc/tmpfiles.d/tmp.conf”.7 e9 ?8 s! v1 u) a- J  V
Running remove action for entry r /tmp/test; S5 f3 S7 e, ~& t' ?* P
The ‘test’ directory is removed
, b1 x* D8 s4 E% k' t# ls /tmp/test
7 b  K( x& p2 i8 v, @1 [ls: cannot access /tmp/test: No such file or directory
" I9 G: M3 l( K( FBut what if the directory has some content
$ O( R: Z: l3 E! t2 b0 \( [  j% C# env SYSTEMD_LOG_LEVEL=debug systemd-tmpfiles –remove+ F/ }9 }2 ^, i0 {# ]
Skipping overridden file: /usr/lib/tmpfiles.d/tmp.conf.
' a3 Y4 a* T" ~Reading config file “/etc/tmpfiles.d/tmp.conf”.
: b7 \% l2 M- k1 l, Y$ oRunning remove action for entry r /tmp/test/ e3 V( G% F; z4 S
rm(/tmp/test): Directory not empty
, g, {; S+ X5 }* }/ @: `For this situation we need different variable
: n1 L, [9 j9 c- rR    Recursively remove a path and all its subdirectories (if it is a directory). Lines of this type accept shell-style globs in place of normal path names.
* x7 E4 I. P! H9 TLooks like it works4 Q# w0 n7 T' ^
# env SYSTEMD_LOG_LEVEL=debug systemd-tmpfiles –remove
3 `! [* R; G" i- `% K/ C: ASkipping overridden file: /usr/lib/tmpfiles.d/tmp.conf.
7 d. s' D0 Z: o3 A9 AReading config file “/etc/tmpfiles.d/tmp.conf”.( |8 ]9 r/ ~& n% S2 g1 ^
Running remove action for entry R /tmp/test7 @" |% X' l  }  D: U4 S; Q4 c, q4 G
rm -rf “/tmp/test”
: N% Q8 ~/ Z" G6 u" R1 fWhen does the clean up happens automatically?* ~( f! L- c9 E
Systemd gives you the timer unit for controlling scheduled and cyclic actions to be performed in the running system.$ J6 Y; c; s3 v' c' K0 a
You can inspect the behaviour of the timer unit by querying the systemd daemon
' I2 W5 I+ g5 i" X) b# systemctl status systemd-tmpfiles-clean.timer
0 y0 f  h* C# E' Hâ systemd-tmpfiles-clean.timer – Daily Cleanup of Temporary Directories
6 Y4 V4 W# `. v   Loaded: loaded (/usr/lib/systemd/system/systemd-tmpfiles-clean.timer; static; vendor preset: disabled): k1 \  Z" y% \/ x& [9 O7 y. T
   Active: active (waiting) since Sat 2018-01-20 23:55:24 IST; 15h ago
1 f' u. F* i. t2 S: J+ ^1 r     Docs: man:tmpfiles.d(5)
2 i' P3 i4 a/ R" Z! E4 W           man:systemd-tmpfiles(8)
- c; d. I( `( J/ x+ Q+ DJan 20 23:55:24 Ban17-rds01-b systemd[1]: Started Daily Cleanup of Temporary Directories.
( s1 y3 _* I, L0 ?/ p* o% l# oJan 20 23:55:24 Ban17-rds01-b systemd[1]: Starting Daily Cleanup of Temporary Directories.
. d& D- _3 l4 n! y3 W( F/ }If you check the systemd unit file for this service it will give you more details
$ ?  p6 @: k) T) f1 B# systemctl cat systemd-tmpfiles-clean.timer" C- W* }& k6 p1 O. ^
# /usr/lib/systemd/system/systemd-tmpfiles-clean.timer+ Q- ~/ J: Z6 L% f
#  This file is part of systemd.
' z& U7 K9 o, Q- H$ d! j#
( D0 o2 M4 c* g- S#  systemd is free software; you can redistribute it and/or modify it
/ d, W9 a& Q1 b#  under the terms of the GNU Lesser General Public License as published by) W" W7 l# s8 H/ C# \% F/ L' [. a
#  the Free Software Foundation; either version 2.1 of the License, or/ `$ v9 I) g& c0 n6 e
#  (at your option) any later version.7 D+ k8 P2 k6 ], _
[Unit]8 `5 ~+ |" ~$ r8 i, |' g
Description=Daily Cleanup of Temporary Directories
) p0 y3 W6 F7 @# m2 N; K  |8 f7 p+ K  yDocumentation=man:tmpfiles.d(5) man:systemd-tmpfiles(8)3 q% t  x. }" @
[Timer]0 n# ?, K% P. t9 }
OnBootSec=15min
3 {9 M' `, g3 QOnUnitActiveSec=1d( S8 @' X. m: W" N4 G
As you’ll find on the timer unit, it will run just 15 minutes after systemd start, and then every 24 hours from that time onward. The command, in this case, will only affect files/directories purging/cleaning.
$ ]# F4 Z; g% I( J6 ~; h7 r" wOnce the system boots up, a special unit file is executed: systemd-tmpfiles-setup, this unit will execute the systemd-tmpfile –create –remove command.
; u' k% O. @0 J/ x: Z6 BI will create a test directory again under /tmp and create a configuration file to clear this directory as above* c. \) d# `" R8 s
# cat /etc/tmpfiles.d/tmp.conf
+ L( R* L: p$ _( D" W# Remove /tmp/test directory and its content0 Q& T' D2 k! J
r /tmp/test/2 g8 ]9 Q5 s! G
Next reboot my system to validate the changes! R9 t5 [- }! q+ k9 k0 g/ D- `
# reboot3 n* F% j. A, d! O- [' a
Next monitor the logs- f: q, Q$ l( b
Jan 20 16:44:24 Ban17-adm-a systemd: Stopped Daily Cleanup of Temporary Directories.& N$ w" @+ p7 @% [
Jan 20 16:44:24 Ban17-adm-a systemd: Stopping Daily Cleanup of Temporary Directories.
1 L, U& {6 L6 G, ^. mWhile starting up the system
3 I; ^1 j$ L$ v4 p* PJan 20 16:49:14 Ban17-adm-a systemd: Started Daily Cleanup of Temporary Directories.
" w5 [- i( H, ~Jan 20 16:49:14 Ban17-adm-a systemd: Starting Daily Cleanup of Temporary Directories.
0 B2 A$ K0 F2 WOnce the system is accessible again lets check if ‘/tmp/test’ exists/ t8 a0 H+ y+ d# ]; y
# ls /tmp/test3 A5 K9 }% w) K& i7 {( K
ls: cannot access /tmp/test: No such file or directory
! R% Z8 P, E2 s, B$ ESo this is cleaned automatically.( L$ n7 k" n+ w$ [( K
I hope the article was useful.
, C. H6 @: k  z$ ^& v
您需要登录后才可以回帖 登录 | 开始注册

本版积分规则

关闭

站长推荐上一条 /4 下一条

如有购买积分卡请联系497906712

QQ|返回首页|Archiver|手机版|小黑屋|易陆发现 点击这里给我发消息

GMT+8, 2022-1-22 12:37 , Processed in 0.043063 second(s), 22 queries .

Powered by LR.LINUX.cloud bbs168x X3.2 Licensed

© 2012-2022 Comsenz Inc.

快速回复 返回顶部 返回列表