{"id":64,"date":"2020-09-16T17:40:08","date_gmt":"2020-09-16T17:40:08","guid":{"rendered":"http:\/\/blogs.gentoo.org\/marecki\/?p=64"},"modified":"2020-09-17T09:35:42","modified_gmt":"2020-09-17T09:35:42","slug":"console-bound-systemd-services-the-right-way","status":"publish","type":"post","link":"https:\/\/blogs.gentoo.org\/marecki\/2020\/09\/16\/console-bound-systemd-services-the-right-way\/","title":{"rendered":"Console-bound systemd services, the right way"},"content":{"rendered":"<p>Let&#8217;s say that you need to run on your system some sort server software which instead of daemonising, has a command console permanently attached to standard input. Let us also say that said console is the only way for the administrator to interact with the service, including requesting its orderly shutdown &#8211; whoever has written it has not implemented any sort of signal handling so sending SIGTERM to the service process causes it to simply drop dead, potentially losing data in the process. And finally, let us say that the server in question is proprietary software so it isn&#8217;t really possible for you to fix any of the above in the source code (yes, I <em>am<\/em> talking about a specific piece of software &#8211; which by the way is very much alive and kicking as of late 2020). What do you do?<\/p>\n<p>According to the collective wisdom of World Wide Web, the answer to this question is &#8220;use a terminal multiplexer like <a href=\"https:\/\/tmux.github.io\/\">tmux<\/a> or <a href=\"https:\/\/www.gnu.org\/software\/screen\/\">screen<\/a>&#8220;, or at the very least a stripped-down variant of same such as <a href=\"https:\/\/github.com\/crigler\/dtach\">dtach<\/a>. OK, that sort of works &#8211; what if you want to run it as a proper system-managed service under <em>e.g.<\/em> OpenRC? The answer of the Stack Exchange crowd: have your init script invoke the terminal multiplexer. Oooooookay, how about under systemd, which actually prefers services it manages not to daemonise by itself? Nope, still &#8220;use a terminal multiplexer&#8221;.<\/p>\n<p>What follows is my attempt to run a service like this under systemd more efficiently and elegantly, or at least with no extra dependencies beyond basic Unix shell commands.<\/p>\n<p>Let us have a closer look at what systemd does with standard I\/O of processes it spawns. The man page <a href=\"https:\/\/www.freedesktop.org\/software\/systemd\/man\/systemd.exec.html\">systemd.exec(5)<\/a> tells us that what happens here is controlled by the directives <em>StandardInput<\/em>, <em>StandardOutput<\/em> and <em>StandardError<\/em>. By default the former is assigned to null while the latter two get piped to the journal, there are however quite a few other options here. According to the documentation, here is what systemd allows us to connect to standard input:<\/p>\n<ul>\n<li style=\"list-style-type: none\">\n<ul>\n<li>we are not interested in <em>null<\/em> (for obvious reasons) or any of the <em>tty<\/em> options (the whole point of this exercise is to run fully detached from any terminals);<\/li>\n<li><em>data<\/em> would work if we needed to feed some commands to the service when it starts but is useless for triggering a shutdown;<\/li>\n<li><em>file<\/em> looks promising &#8211; just point it to a FIFO on the file system and we&#8217;re all set &#8211; but it doesn&#8217;t actually take care of creating the FIFO for us. While we could in theory work around that by invoking <em>mkfifo (<\/em>and possibly <em>chown<\/em> if the service is to run as a specific user) in <em>ExecStartPre<\/em>, let&#8217;s see if we can find a better option<\/li>\n<li><em>socket<\/em> &#8220;is valid in socket-activated services only&#8221; and the corresponding socket unit must &#8220;have <em>Accept=yes<\/em> set&#8221;. What we want is the opposite, i.e. for the service to create its socket<\/li>\n<li>finally, there is <em>fd<\/em> &#8211; which seems to be exactly what we need. According to the documentation all we have to do is write <a href=\"http:\/\/www.freedesktop.org\/software\/systemd\/man\/systemd.socket.html\">a socket unit<\/a> creating a FIFO with appropriate ownership and permissions, make it a dependency of our service using the <em>Sockets<\/em> directive, and assign the corresponding named file descriptor to standard input.<\/li>\n<\/ul>\n<\/li>\n<\/ul>\n<p>Let&#8217;s try it out. To begin with, our socket unit &#8220;proprietarycrapd.socket&#8221;. Note that I have successfully managed to get this to work using unit templates as well, %i expansion works fine both here and while specifying unit or file-descriptor names in the service unit &#8211; but in order to avoid any possible confusion caused by the fact socket-activated services explicitly require being defined with templates, I have based my example on static units:<\/p>\n<blockquote><p>[Unit]<br \/>\nDescription=Command FIFO for proprietarycrapd<\/p>\n<p>[Socket]<br \/>\nListenFIFO=\/run\/proprietarycrapd\/pcd.control<br \/>\nDirectoryMode=0700<br \/>\nSocketMode=0600<br \/>\nSocketUser=pcd<br \/>\nSocketGroup=pcd<br \/>\nRemoveOnStop=true<\/p><\/blockquote>\n<p>Apart from the fact the unit in question has got no <em>[Install]<\/em> section (which makes sense given we want this socket to only be activated by the corresponding service, not by systemd itself), nothing out of the ordinary here. Note that since we haven&#8217;t used the directive <em>FileDescriptorName<\/em>, systemd will apply default behaviour and give the file descriptor associated with the FIFO the name of the socket unit itself.<\/p>\n<p>And now, our service unit &#8220;proprietarycrapd.service&#8221;:<\/p>\n<blockquote><p>[Unit]<br \/>\nDescription=proprietarycrap daemon<br \/>\nAfter=network.target<\/p>\n<p>[Service]<br \/>\nUser=pcd<br \/>\nGroup=pcd<br \/>\nSockets=proprietarycrapd.socket<br \/>\nStandardInput=socket<br \/>\nStandardOutput=journal<br \/>\nStandardError=journal<br \/>\nExecStart=\/opt\/proprietarycrap\/bin\/proprietarycrapd<br \/>\nExecStop=\/usr\/local\/sbin\/proprietarycrapd-stop<\/p>\n<p>[Install]<br \/>\nWantedBy=multi-user.target<\/p><\/blockquote>\n<p><em>StandardInput=socket<\/em>??? Whatever&#8217;s happened to <em>StandardInput=fd:proprietarycrapd.socket<\/em>??? Here is an odd thing. If I use the latter on my system, the service starts fine and gets the FIFO attached to its standard input &#8211; but when I try to stop the service the journal shows &#8220;Failed to load a named file descriptor: No such file or directory&#8221;, the <em>ExecStop<\/em> command is not run and systemd immediately fires a SIGTERM at the process. No idea why. Anyway, through trial and error I have found out that <em>StandardInput=socket<\/em> not only works fine in spite of being used in a service that is not socket-activated but actually does exactly what I wanted to achieve &#8211; so that is what I have ended up using.<\/p>\n<p>Which brings us to the final topic, the <em>ExecStop<\/em> command. There are three reasons why I have opted for putting all the commands required to shut the server down in a shell script:<\/p>\n<ul>\n<li style=\"list-style-type: none\">\n<ul>\n<li>first and foremost, writing the shutdown command to the FIFO will return right away even if the service takes time to shut down. systemd sends SIGTERM to the unit process as soon as the last <em>ExecStop<\/em> command has exited so we have to follow the <em>echo<\/em> with something that waits for the server process to finish (see below)<\/li>\n<li>systemd does <em>not<\/em> execute <em>Exec<\/em> commands in a shell so simply running <em>echo &gt; \/run\/proprietarycrapd\/pcd.control<\/em> doesn&#8217;t work, we would have to wrap the <em>echo<\/em> call in an explicit invocation of a shell<\/li>\n<li>between the aforementioned two reasons and the fact the particular service for which I have created these units actually requires <em>several<\/em> commands in order to execute an orderly shutdown, I have decided that putting all those command in a script file instead of cramming them into the unit would be much cleaner.<\/li>\n<\/ul>\n<\/li>\n<\/ul>\n<p>The shutdown script itself is mostly unremarkable so I&#8217;ll only quote the bit responsible for waiting for the server to actually shut down. At present I am still looking for doing it in blocking fashion without adding more dependencies (<em>wait<\/em> only works on child processes of the current shell, the server in question does not create any lock files to which I could attach <em>inotifywait<\/em>, and attaching the latter to the relevant directory in \/proc does not work) but in the meantime, the loop<\/p>\n<blockquote><p>while kill -0 &#8220;${MAINPID}&#8221; 2&gt; \/dev\/null; do<br \/>\nsleep 1s<br \/>\ndone<\/p><\/blockquote>\n<p>keeps the script ticking along until either the process has exited or the script has timed out (see the <em>TimeoutStopSec<\/em> directive in systemd.service(5)) and systemd has killed both it and the service itself.<\/p>\n<p><em>Acknowledgements: with many thanks to steelman for having figured out the StandardInput=socket bit in particular and having let me bounce my ideas off him in general.<\/em><\/p>\n","protected":false},"excerpt":{"rendered":"<p>Let&#8217;s say that you need to run on your system some sort server software which instead of daemonising, has a command console permanently attached to standard input. Let us also say that said console is the only way for the administrator to interact with the service, including requesting its orderly shutdown &#8211; whoever has written &hellip; <a href=\"https:\/\/blogs.gentoo.org\/marecki\/2020\/09\/16\/console-bound-systemd-services-the-right-way\/\" class=\"more-link\">Continue reading <span class=\"screen-reader-text\">Console-bound systemd services, the right way<\/span><\/a><\/p>\n","protected":false},"author":167,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"spay_email":""},"categories":[1],"tags":[15,7,16,14],"jetpack_featured_media_url":"","jetpack_shortlink":"https:\/\/wp.me\/p8unF2-12","_links":{"self":[{"href":"https:\/\/blogs.gentoo.org\/marecki\/wp-json\/wp\/v2\/posts\/64"}],"collection":[{"href":"https:\/\/blogs.gentoo.org\/marecki\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/blogs.gentoo.org\/marecki\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/blogs.gentoo.org\/marecki\/wp-json\/wp\/v2\/users\/167"}],"replies":[{"embeddable":true,"href":"https:\/\/blogs.gentoo.org\/marecki\/wp-json\/wp\/v2\/comments?post=64"}],"version-history":[{"count":7,"href":"https:\/\/blogs.gentoo.org\/marecki\/wp-json\/wp\/v2\/posts\/64\/revisions"}],"predecessor-version":[{"id":71,"href":"https:\/\/blogs.gentoo.org\/marecki\/wp-json\/wp\/v2\/posts\/64\/revisions\/71"}],"wp:attachment":[{"href":"https:\/\/blogs.gentoo.org\/marecki\/wp-json\/wp\/v2\/media?parent=64"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/blogs.gentoo.org\/marecki\/wp-json\/wp\/v2\/categories?post=64"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/blogs.gentoo.org\/marecki\/wp-json\/wp\/v2\/tags?post=64"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}