Systemd

List all units and their status

  • All running
systemctl
  • List only services
systemctl list-units --type=service
  • All available
systemctl list-unit-files

List all failed services

systemctl --failed

Start / Stop service

  • All services can be found in /usr/lib/systemd/system
systemctl [start|stop] sshd.service

Activate service on boot

systemctl enable sshd.service

Show status of a service

systemctl status sshd.service

List all available targets (runlevels)

systemctl list-units --type=target

Change default target (runlevel)

systemctl set-default multi-user.target
  • or
rm /etc/systemd/system/default.target
ln -sf /lib/systemd/system/multi-user.target /etc/systemd/system/default.target

Persistent logs

  • Normally journald logs to /run/log/journal this is a tmpfs and logs are deleted on reboot
  • To avoid this edit /etc/systemd/journald.conf
[Journal]
Storage=persistent
  • If that’s not working try
mkdir /var/log/journal
chgrp systemd-journal /var/log/journal
chmod 775 /var/log/journal

Filtering logs

  • Use the force of TAB completion!
journalctl <TAB>
journalctl _COMM=<TAB>
  • Errors since last boot
journalctl -b -p err
  • List all reboots
journalctl --list-boots
  • Since today
journalctl --since today
  • One hour ago
journalctl --since -1h
  • Or timerange
journalctl --since=2012-10-15 --until="2011-10-16 23:59:59"
  • For a specific file
journalctl /some/file
  • Tailed
journalctl -f
  • For a single pid
journalctl _PID=123
  • For a single unit (service)
journalctl -u <servicename>
  • For kernel messages
journalctl -k
  • For network stuff
journalctl _COMM=network
  • For a SELinux context
journalctl _SELINUX_CONTEXT=<security context>
  • For a single user
journalctl _UID=<userid>
  • Full output for last 10 messages
journalctl -l -o verbose -n 10
  • Where to find the log files?
cd /var/log/journal
  • How to configure max hd space for logs? Edit /etc/systemd/journald.conf
SystemMaxUse=100M
  • Log rotation (/etc/systemd/journald.conf)
MaxRetentionSec=1day
MaxFileSec=1month
  • How to log to syslog (edit /etc/systemd/journald.conf)
ForwardToSyslog=yes
  • Export log as JSON
-o json

Remote logging

  • Install systemd-journal-gateway
  • On server edit /etc/systemd/journal-remote.conf and start service systemd-journal-remote
  • On log client edit /etc/systemd/journal-upload.conf, to URL to http://<ip_of_logserver>:19531 and start service systemd-journal-upload

Journald Web Gateway

  • Install systemd-journal-gateway
  • Start service systemd-journal-gateways
  • Connect your browser to http://<ip>:19531
  • To get an endless stream http://<ip>:19531/entries?follow
  • To pull remote journal log an save it to a text file
nohup curl --silent -o some-host.log 'http://<ip>:19531/entries?follow' &
  • Or to pull it in the original journal format
nohup curl --silent -H'Accept: application/vnd.fdo.journal' -o some-host.log 'http://<ip>:19531/entries?follow' &

Rescue Mode / Debugging

  • On Grub prompt try to set one of the following kernel parameter
systemd.unit=rescue.target      # (single user mode)
systemd.unit=emergency.target   # (only shell)
  • Ask before starting a servce

    systemd.confirm_spawn=1

  • Give me more log output

systemd.log_target=kmsg systemd.log_level=debug
  • Get console output of legacy sysv init scripts
systemd.sysv_console=1
  • Which units want which target?
systemctl show -p "Wants" multi-user.target
  • To analyze which services was slow
systemd-analyze blame

What services do get started?

systemctl list-dependencies multi-user.target

Change runlevel

systemctl isolate <newtarget e.g. rescue.target or mutli-user.target>

Changing the default runlevel

ln -sf /usr/lib/systemd/system/multi-user.target /etc/systemd/system/default.target

An example service

[Unit]
Description=Just a simple test
After=syslog.target

[Service]
ExecStart=/bin/some-daemon
Type=forking
CPUShares=1500
MemoryLimit=1G
BlockIOWeight=500

[Install]
WantedBy=multi-user.target
  • Afterwards exec
systemctl daemon-reload
systemctl start test.service
systemctl status test.service

Power management

systemctl suspend
systemctl hibernate

Chrooting

  • Set up chroot environment with yum or debootstrap or whatever
  • Old school with chroot()
[Service]
RootDirectory=/srv/chroot/foobar
  • New age with kernel namespaces
systemd-nspawn -D <chroot_dir> <command>

More security options

  • Disable networking
PrivateNetwork=yes
  • Isolate tmp dir
PrivateTmp=yes
  • Read-only or inaccessible directories
InaccessibleDirectories=/home
ReadOnlyDirectories=/var
  • Use capabilities (see man capabilities)
CapabilityBoundingSet=CAP_CHOWN CAP_KILL
  • Use process limits
LimitNPROC=1
LimitFSIZE=0
  • Limit device usage

    DeviceAllow=/dev/null rw

  • Run as a specific user / group

User=nobody
Group=nobody

Only start a service if a specific device is found

BindToDevice=dev-sda5.device

I want more gettys / text consoles

ln -sf /usr/lib/systemd/system/getty@.service /etc/systemd/system/getty.target.wants/getty@tty9.service

Custom kernel

  • CONFIG_DEVTMPFS
  • CONFIG_CGROUPS (it is OK to disable all controllers)
  • CONFIG_INOTIFY_USER
  • CONFIG_SIGNALFD
  • CONFIG_TIMERFD
  • CONFIG_EPOLL
  • CONFIG_NET
  • CONFIG_SYSFS
  • CONFIG_PROC_FS
  • CONFIG_FHANDLE (libudev, mount and bind mount handling)
  • CONFIG_SYSFS_DEPRECATED=n
  • CONFIG_UEVENT_HELPER_PATH=”“
  • CONFIG_FW_LOADER_USER_HELPER=n
  • CONFIG_DMIID
  • CONFIG_BLK_DEV_BSG
  • CONFIG_NET_NS
  • CONFIG_IPV6
  • CONFIG_AUTOFS4_FS
  • CONFIG_TMPFS_POSIX_ACL
  • CONFIG_TMPFS_XATTR
  • CONFIG_SECCOMP
  • CONFIG_CGROUP_SCHED
  • CONFIG_FAIR_GROUP_SCHED
  • CONFIG_SCHEDSTATS
  • CONFIG_SCHED_DEBUG