Systemd

List all units and their status

  • All running

systemctl
  • List only services

systemctl list-units --type=service
  • All available

systemctl list-unit-files

List all failed services

systemctl --failed

Start / Stop service

  • All services can be found in /usr/lib/systemd/system

systemctl [start|stop] sshd.service

Activate service on boot

systemctl enable sshd.service

Show status of a service

systemctl status sshd.service

List all available targets (runlevels)

systemctl list-units --type=target

Change default target (runlevel)

systemctl set-default multi-user.target
  • or

rm /etc/systemd/system/default.target
ln -sf /lib/systemd/system/multi-user.target /etc/systemd/system/default.target

Persistent logs

  • Normally journald logs to /run/log/journal this is a tmpfs and logs are deleted on reboot

  • To avoid this edit /etc/systemd/journald.conf

[Journal]
Storage=persistent
  • If that’s not working try

mkdir /var/log/journal
chgrp systemd-journal /var/log/journal
chmod 775 /var/log/journal

Filtering logs

  • Use the force of TAB completion!

journalctl <TAB>
journalctl _COMM=<TAB>
  • Errors since last boot

journalctl -b -p err
  • List all reboots

journalctl --list-boots
  • Since today

journalctl --since today
  • One hour ago

journalctl --since -1h
  • Or timerange

journalctl --since=2012-10-15 --until="2011-10-16 23:59:59"
  • For a specific file

journalctl /some/file
  • Tailed

journalctl -f
  • For a single pid

journalctl _PID=123
  • For a single unit (service)

journalctl -u <servicename>
  • For kernel messages

journalctl -k
  • For network stuff

journalctl _COMM=network
  • For a SELinux context

journalctl _SELINUX_CONTEXT=<security context>
  • For a single user

journalctl _UID=<userid>
  • Full output for last 10 messages

journalctl -l -o verbose -n 10
  • Where to find the log files?

cd /var/log/journal
  • How to configure max hd space for logs? Edit /etc/systemd/journald.conf

SystemMaxUse=100M
  • Log rotation (/etc/systemd/journald.conf)

MaxRetentionSec=1day
MaxFileSec=1month
  • How to log to syslog (edit /etc/systemd/journald.conf)

ForwardToSyslog=yes
  • Export log as JSON

-o json

Remote logging

  • Install systemd-journal-gateway

  • On server edit /etc/systemd/journal-remote.conf and start service systemd-journal-remote

  • On log client edit /etc/systemd/journal-upload.conf, to URL to http://<ip_of_logserver>:19531 and start service systemd-journal-upload

Journald Web Gateway

  • Install systemd-journal-gateway

  • Start service systemd-journal-gateways

  • Connect your browser to http://<ip>:19531

  • To get an endless stream http://<ip>:19531/entries?follow

  • To pull remote journal log an save it to a text file

nohup curl --silent -o some-host.log 'http://<ip>:19531/entries?follow' &
  • Or to pull it in the original journal format

nohup curl --silent -H'Accept: application/vnd.fdo.journal' -o some-host.log 'http://<ip>:19531/entries?follow' &

Rescue Mode / Debugging

  • On Grub prompt try to set one of the following kernel parameter

systemd.unit=rescue.target      # (single user mode)
systemd.unit=emergency.target   # (only shell)
  • Ask before starting a servce

    systemd.confirm_spawn=1

  • Give me more log output

systemd.log_target=kmsg systemd.log_level=debug
  • Get console output of legacy sysv init scripts

systemd.sysv_console=1
  • Which units want which target?

systemctl show -p "Wants" multi-user.target
  • To analyze which services was slow

systemd-analyze blame

What services do get started?

systemctl list-dependencies multi-user.target

Change runlevel

systemctl isolate <newtarget e.g. rescue.target or mutli-user.target>

Changing the default runlevel

ln -sf /usr/lib/systemd/system/multi-user.target /etc/systemd/system/default.target

An example service

[Unit]
Description=Just a simple test
After=syslog.target

[Service]
ExecStart=/bin/some-daemon
Type=forking
CPUShares=1500
MemoryLimit=1G
BlockIOWeight=500

[Install]
WantedBy=multi-user.target
  • Afterwards exec

systemctl daemon-reload
systemctl start test.service
systemctl status test.service

Power management

systemctl suspend
systemctl hibernate

Using Resolved

  • List interfaces and their attached dns server

resolvectl dns
  • List interfaces and their dns search domain(s)

resolvectl domain
  • Do a DNS query

resolvectl query <host>
  • Debugging

resolvectl log-level debug

Disabling Resolved

systemctl disable systemd-resolved
systemctl stop systemd-resolved
  • Edit /etc/NetworkManager/NetworkManager.conf and add dns=default in the [main] section

rm /etc/resolv.conf
systemctl restart NetworkManager

Use systemd as inetd

Chrooting

  • Set up chroot environment with yum or debootstrap or whatever

  • Old school with chroot()

[Service]
RootDirectory=/srv/chroot/foobar
  • New age with kernel namespaces

systemd-nspawn -D <chroot_dir> <command>

More security options

  • Disable networking

PrivateNetwork=yes
  • Isolate tmp dir

PrivateTmp=yes
  • Read-only or inaccessible directories

InaccessibleDirectories=/home
ReadOnlyDirectories=/var
  • Use capabilities (see man capabilities)

CapabilityBoundingSet=CAP_CHOWN CAP_KILL
  • Use process limits

LimitNPROC=1
LimitFSIZE=0
  • Limit device usage

    DeviceAllow=/dev/null rw

  • Run as a specific user / group

User=nobody
Group=nobody

Only start a service if a specific device is found

BindToDevice=dev-sda5.device

I want more gettys / text consoles

ln -sf /usr/lib/systemd/system/getty@.service /etc/systemd/system/getty.target.wants/getty@tty9.service

Python Coding

Custom kernel

  • CONFIG_DEVTMPFS

  • CONFIG_CGROUPS (it is OK to disable all controllers)

  • CONFIG_INOTIFY_USER

  • CONFIG_SIGNALFD

  • CONFIG_TIMERFD

  • CONFIG_EPOLL

  • CONFIG_NET

  • CONFIG_SYSFS

  • CONFIG_PROC_FS

  • CONFIG_FHANDLE (libudev, mount and bind mount handling)

  • CONFIG_SYSFS_DEPRECATED=n

  • CONFIG_UEVENT_HELPER_PATH=””

  • CONFIG_FW_LOADER_USER_HELPER=n

  • CONFIG_DMIID

  • CONFIG_BLK_DEV_BSG

  • CONFIG_NET_NS

  • CONFIG_IPV6

  • CONFIG_AUTOFS4_FS

  • CONFIG_TMPFS_POSIX_ACL

  • CONFIG_TMPFS_XATTR

  • CONFIG_SECCOMP

  • CONFIG_CGROUP_SCHED

  • CONFIG_FAIR_GROUP_SCHED

  • CONFIG_SCHEDSTATS

  • CONFIG_SCHED_DEBUG