Posts on Michał Góral Recent content in Posts on Michał Góral Hugo -- pl-PL Fri, 28 Feb 2020 00:00:00 +0000 I Blocked Bots Fri, 28 Feb 2020 00:00:00 +0000 <p>Yesterday I started blocking crawlers via robots.txt. I was forced to because it started getting out of hand. I&rsquo;ll observe traffic for a few days and will block them on level of HTTP server or fail2ban/iptables if it turns out that they don&rsquo;t respect robots.txt, contrary to the claims of the biggest offenders.</p> <p>This is mostly low-profile site. Me, my family and maybe some of my friends are doing at most 500-600 requests a day, a little more if I&rsquo;m tinkering with server.</p> <figure class="center"> <img src="" alt="14 days goaccess log"/> <figcaption> <p>14 days goaccess log</p> </figcaption> </figure> <p>There are 2 main contributors to the increased traffic: <a href="">MJ12</a> and <a href="">SemRush</a>. Above is part of goaccess&rsquo; report from last 14 days. It is clear that these 2 bots alone do over 11000 requests a day, which is insane for a site known by maybe 10 people on Earth. It&rsquo;s nowhere near the capacity of the server, but it translates to 2 GB of transferred data every month. Wasted 2 GB, because those bots give me nothing. I don&rsquo;t even think I&rsquo;m serving that many pages in total, including some dynamically generated.</p> <p>Cherry on top: I&rsquo;m not the only one who hates these bots. <a href="">Wikipedia</a> hates MJ12 especially as well. And <a href="">others</a> <a href="">too</a>.</p> <h2 id="update-2020-03-03">Update: 2020-03-03</h2> <figure class="center"> <img src=""/> </figure> <p>After few days it seems that robots.txt trick worked as things calmed down. I had to add robots.txt to every subdomain though, some disallowing all bots and some merely delaying requests (via non-standard <em>Crawl-Delay</em> directive). Still, there are more requests than I&rsquo;d expect, but I have to evaluate those before saying anything else.</p> Synchronization Sat, 15 Feb 2020 00:00:00 +0000 <p>I synchronize a lot of things. My personal knowledge base, calendar (via vdirsyncer), Qalculate exchange rates, ledger and budget repository, dotfiles, e-mail, tasks&hellip; How do I keep up with all of those things?</p> <p>As a systemd user, it&rsquo;s natural for me to set up systemd timers for some of those repetitive tasks. Unfortunately creating a timer for each small task is quite frustrating due to very explicit nature of system timers architecture: one have to create a service unit, timer unit, connect them, enable them, start them, think about their dependencies&hellip; That&rsquo;s too much work!</p> <p>So I created a single shell script (and also set up systemd timer for it) which launches all registered sync functions in parallel, with help of great <a href="">GNU Parallel</a>. It looks like this:</p> <div class="highlight"><pre class="chroma"><code class="language-bash" data-lang="bash"><span class="cp">#!/bin/bash </span><span class="cp"></span> sync_cal<span class="o">()</span> <span class="o">{</span> <span class="nb">echo</span> <span class="s2">&#34;----- [ </span><span class="si">${</span><span class="nv">FUNCNAME</span><span class="p">[0]</span><span class="si">}</span><span class="s2"> ] -----&#34;</span> <span class="nb">set</span> -e vdirsyncer sync khal2rem -o <span class="nv">$HOME</span>/.khal.rem touch ~/.reminders <span class="nb">set</span> +e <span class="o">}</span> sync_qalc<span class="o">()</span> <span class="o">{</span> <span class="nb">echo</span> <span class="s2">&#34;----- [ </span><span class="si">${</span><span class="nv">FUNCNAME</span><span class="p">[0]</span><span class="si">}</span><span class="s2"> ] -----&#34;</span> <span class="nb">set</span> -e qalc &lt; &lt;<span class="o">(</span><span class="nb">echo</span> <span class="s2">&#34;exrates&#34;</span><span class="o">)</span> <span class="nb">set</span> +e <span class="o">}</span> sync_tasks<span class="o">()</span> <span class="o">{</span> <span class="nb">echo</span> <span class="s2">&#34;----- [ </span><span class="si">${</span><span class="nv">FUNCNAME</span><span class="p">[0]</span><span class="si">}</span><span class="s2"> ] -----&#34;</span> <span class="nb">set</span> -e <span class="nb">local</span> <span class="nv">h</span><span class="o">=</span><span class="s2">&#34;</span><span class="k">$(</span>hostname<span class="k">)</span><span class="s2">&#34;</span> <span class="k">if</span> <span class="o">[[</span> <span class="s2">&#34;</span><span class="nv">$h</span><span class="s2">&#34;</span> <span class="o">==</span> <span class="s2">&#34;hostname_behind_a_proxy&#34;</span> <span class="o">]]</span><span class="p">;</span> <span class="k">then</span> proxychains task sync <span class="k">else</span> task sync <span class="k">fi</span> <span class="nb">set</span> +e <span class="o">}</span> <span class="nv">fns</span><span class="o">=()</span> register<span class="o">()</span> <span class="o">{</span> <span class="nb">export</span> -f <span class="nv">$1</span> <span class="nv">fns</span><span class="o">+=(</span><span class="s2">&#34;</span><span class="nv">$1</span><span class="s2">&#34;</span><span class="o">)</span> <span class="o">}</span> register sync_cal register sync_qalc register sync_tasks <span class="c1"># Handle arguments passed to the script</span> <span class="k">if</span> <span class="o">[[</span> <span class="nv">$#</span> -gt <span class="m">0</span> <span class="o">]]</span><span class="p">;</span> <span class="k">then</span> <span class="nv">fns</span><span class="o">=()</span> <span class="k">for</span> fn in <span class="s2">&#34;</span><span class="nv">$@</span><span class="s2">&#34;</span><span class="p">;</span> <span class="k">do</span> <span class="nv">fns</span><span class="o">+=(</span><span class="s2">&#34;sync_</span><span class="si">${</span><span class="nv">fn</span><span class="si">}</span><span class="s2">&#34;</span><span class="o">)</span> <span class="k">done</span> <span class="k">fi</span> parallel -j0 <span class="s1">&#39;{}&#39;</span> ::: <span class="s2">&#34;</span><span class="si">${</span><span class="nv">fns</span><span class="p">[@]</span><span class="si">}</span><span class="s2">&#34;</span> </code></pre></div><p>I called it <em>mg-sync</em> (I prefix all of my personal scripts with <em>mg-</em>, which makes them easier to find with shell tab-completion. Recently I even wrote a wrapper which instead of e.g. <em>mg-sync</em> allows me to write <em>mg sync</em>, the same way as e.g. Git works). Thanks to this script I can run in parallel all registered functions (when it&rsquo;s called with no arguments) or only selected ones (when it&rsquo;s called with arguments).</p> <p>Even though it&rsquo;s short and concise, there are few things to remember. First, each Bash function must be exported (<code>export -f</code>), otherwise it will be invisible to Parallel. Second, GNU Parallel spawns a separate job for each function (<code>-j0</code>), where full command of each job is a single argument from the argument list following triple colon. I am also explicit about command substitution (<code>{}</code>), but Parallel would work even without it: <code>parallel -j0 ::: &quot;${fns[@]}&quot;</code>.</p> <p>I like this a lot because it handles parallelism for me, waits for each ran command and correctly reports command errors. My systemd timer runs this script every few hours, but I configured a shortcut key which starts systemd service immediately if I need to quickly synchronize some data.</p> Cross References in Notes Wed, 12 Feb 2020 00:00:00 +0000 <p>My <a href="">notes</a> are continuous struggle. Because I render my notes with Hugo, which is a fabulous static site generator, I kept links in Hugo-friendly form: <code>[foo]({{&lt; ref &quot;; &gt;}})</code>. Unfortunately this is only understood by Hugo and sometimes I want to read and navigate notes from my phone. But fear no more - I fixed it!</p> <p>I create Makefiles for almost all of my projects. They&rsquo;re great: they provide easy to remember facade to complex tasks and have shell completion for their targets, so choosing what to run usually boils down to a little tabbing through the list. How does it relates to having and not having Hugo-like references? What should we do to have our cake and eat it? Add a little preprocessing in Makefile of course!</p> <p>First of all I needed to convert back all of my Hugo-like refs to ordinary markdown links. I used some magic sed for it which worked for ~95% of all original refs the rest I fixed manually. It was one time thing and I don&rsquo;t remember the exact regular expression unfortunately, but it was something like <code>sed -i -E -e 's/{{&lt; ref &quot;(; &gt;}}/\1/g'</code>. Unhandled cases? <code>relref</code> shortcode (obviously, I just HAD to use it at least once somewhere) and several refs in a single line (<code>.+</code> is a greedy match).</p> <p>Next, I wanted Hugo to understand that any links to <em>.md</em> files are in fact cross-references, which means that we must somehow replace bare file name with a shortcode, without changing &ndash; without even touching a working copy (because who likes when his text editor screams at you that file has changed when it hasn&rsquo;t).</p> <p>First I thought that I&rsquo;ll be able to use these fancy <a href="">pipelines</a> which Hugo uses to pre-process assets, but apparently they&rsquo;re special snowflakes, specialized for minifying Javascript and compiling SASS to CSS. Not good. So I decided to do the simplest thing ever: copy the whole content aside, run sed through it and use it to generate my site.</p> <div class="highlight"><pre class="chroma"><code class="language-make" data-lang="make"><span class="nv">PUBDIR</span> <span class="o">:=</span> /var/www/docs <span class="nv">CONTENT_TMP</span> <span class="o">:=</span> /tmp/docs_content_tmp <span class="nf">build</span><span class="o">:</span> @rm -rf <span class="s2">&#34;</span><span class="k">$(</span>PUBDIR<span class="k">)</span><span class="s2">&#34;</span> @mkdir -p <span class="s2">&#34;</span><span class="k">$(</span>PUBDIR<span class="k">)</span><span class="s2">&#34;</span> @rm -rf <span class="s2">&#34;</span><span class="k">$(</span>CONTENT_TMP<span class="k">)</span><span class="s2">&#34;</span> cp -r content <span class="s2">&#34;</span><span class="k">$(</span>CONTENT_TMP<span class="k">)</span><span class="s2">&#34;</span> sed -i -E<span class="se">\ </span><span class="se"></span> -e <span class="s1">&#39;s,]\(([^).]\),]({{&lt; ref &#34;\1&#34; &gt;}}),g&#39;</span> <span class="se">\ </span><span class="se"></span> -e <span class="s1">&#39;s,]\((files/[^)]+)\),]({{&lt; file &#34;\1&#34; &gt;}}),g&#39;</span> <span class="se">\ </span><span class="se"></span> <span class="sb">`</span>find <span class="s2">&#34;</span><span class="k">$(</span>CONTENT_TMP<span class="k">)</span><span class="s2">&#34;</span> -name <span class="s2">&#34;*.md&#34;</span> -type f<span class="sb">`</span> hugo -d <span class="s2">&#34;</span><span class="k">$(</span>PUBDIR<span class="k">)</span><span class="s2">&#34;</span> -c <span class="s2">&#34;</span><span class="k">$(</span>CONTENT_TMP<span class="k">)</span><span class="s2">&#34;</span> @rm -rf <span class="s2">&#34;</span><span class="k">$(</span>CONTENT_TMP<span class="k">)</span><span class="s2">&#34;</span> <span class="nf">.PHONY</span><span class="o">:</span> <span class="n">build</span> </code></pre></div><p>I hate that sed, but it does the job and probably it&rsquo;s not the worst one I&rsquo;ve ever written. And hey, it even works for several links in a single line, which wasn&rsquo;t that trivial (see that <code>[^)]</code> block? That&rsquo;s a hack for catching all unicode characters in a non-greedy way, which is generally unsupported by BRE and ERE used by GNU tools. Or it depends on locale. Or something else obscure enough that I don&rsquo;t want to think about it).</p> <p>Second <code>-e</code> script additionally detects file attachments, which are stored inside each repo in <em>files</em> directory, and wraps them with a custom <code>{{&lt; file &gt;}}</code> shortcode which handles conversions to absolute paths. So for example image <code>![foo](files/bar.jpg)</code> gets converted to <code>![foo]({{&lt; &quot;files/bar.jpg&quot; &gt;}})</code>, which in turn is generated to <code>&lt;img src=&quot;http://localhost/docs/0/files/bar.jpg&quot; /&gt;</code>.</p> <p>Shortcode implementation is one-liner:</p> <div class="highlight"><pre class="chroma"><code class="language-txt" data-lang="txt">{{-/* printf &#34;%s/%s/%s&#34; .Site.BaseURL .Page.Section (.Get 0) */-}} </code></pre></div><p>I like this approach a lot because finally content of my notes is independent from any external system/renderer. And now they work rather nicely with <a href="">Markor</a>.</p> <p>One thing which I&rsquo;ll for sure improve in future: I won&rsquo;t blindly copy the <em>whole</em> content directory, but only Markdown files. All other files (&ldquo;attachments&rdquo;) will be symlinked. It should save a little disk space and a little time spent on unnecessary copying.</p> Officially I hate my notes Mon, 03 Feb 2020 00:00:00 +0000 <p>Recently I came across an interesting way of working with personal notes called <a href="">Zettelkasten</a>. I resisted it for a while, but finally jumped the bandwagon and converted my whole knowledge database to zettels, which is a German word for small post-it notes, usually kept in a box.</p> <p>I am plaintext junkie. New fancy formats come and go and basic plaintext files have stayed with me for my whole life. When I started the process my whole knowledge database was kept in roughly 20-30 unordered pages, with strong &ldquo;unmaintained wiki&rdquo; vibes. A month later I have ~150 notes in 2 &ldquo;notebooks&rdquo;, 3 scripts which help me handling renaming of my notes and synchronizing them, <a href="">Hugo</a> theme, <a href="">ctags language definition</a>, freaking custom git-submodules replacement, a few custom Vim functions and mappings on top of that and strong WTF vibes. Everything resides comfortably (yeah, right) in 3 or 4 different git repositories, which is probably the biggest downside of my setup.</p> <p>And I hate it. And then I love it. And then I hate it again. But ultimately I love it.</p> <p>The hatred comes probably from my anxiety which I get when I start looking at it. It is complex and hacky and most probably would call it unnecessary or &ldquo;you don&rsquo;t have anything to do with your free time, don&rsquo;t you&rdquo;. But at the end of the day it just works for me. It even works on my phone terminal, which is a great achievement of modern-day technology, but not very useful as I consume my notes differently on laptop and phone. *Nively put, but as young Cerro said to King Vridank on their wedding night: &ldquo;does it have any practical uses?&quot;<sup id="fnref:1"><a href="#fn:1" class="footnote-ref" role="doc-noteref">1</a></sup></p> <p>Now it&rsquo;s so easy to take the notes that I actually started taking them. And thanks to Hugo, when I&rsquo;m not in a plaintext mood and want to just watch these not-so-beautiful images I embed from time to time to some of my notes, I can just generate my custom not-very-unpretty static webpage in a matter of milliseconds, copy them to <em>/var/www</em>, where I have Python&rsquo;s simple HTTP server<sup id="fnref:2"><a href="#fn:2" class="footnote-ref" role="doc-noteref">2</a></sup> constantly running on port 9999, type that 9999 in a web browser and bam! watch my glorious notes.</p> <h2 id="structure">Structure</h2> <p>I store notes in Git, which is the only system I wholeheartedly trust to help me recover from my stupidity. I believe that my SSD will die sooner than any of my Git repos lose integrity and thanks to its decentralized nature I usually have copies on 2, 3 or 4 different, geographically distant devices. I already deleted some of my notes by accident but thanks to Git they&rsquo;re versioned, so it was trivial to recover them.</p> <p>In the past, when I keptn notes synchronized with syncthing I sometimes run to synchronization conflicts. <em>These weren&rsquo;t that bad.</em> Bad was when empty copy of single file overrided copies with content and OF COURSE I haven&rsquo;t thought about enabling Syncthing&rsquo;s file versioning.</p> <p>My notes are stored in several notebooks and each one of those is a separate Git repo which contains notes only, optionally attachments. All notebooks are bundled inside a parent Git repo with Hugo theme and configuration files. It looks like this:</p> <div class="highlight"><pre class="chroma"><code class="language-text" data-lang="text">./ &lt;parent repo&gt; content/ &lt;main page of generated site&gt; 0/ &lt;personal git repo&gt; files/ &lt;attachments&gt; other/ &lt;work git repo&gt; files/ &lt;attachments&gt; </code></pre></div><p>Sounds like a job for <a href="">Git Submodules</a>, right? Yeah, almost. There are 3 reasons why I don&rsquo;t want submodules:</p> <ol> <li>They complicate the workflow of notes syncing because I have to remember to set-up branch-tracking submodules, update them and then update them (commit + push) in parent repo.</li> <li>I want to have <em>conditional</em> submodules. Why? Because I want to keep my work-related notes only on one laptop and specifically keep them as far as I can from my phone to avoid any IP theft accusations.</li> <li>My work forces me to connect to internet via a proxy, which only opens ports 80 and 443, so connecting to git via ssh is no-go at work but it is my preferred method on any other machine.</li> </ol> <p>So I created <em>csync</em> script which reads submodule definitions from a file named <em>modules</em>. It is a very simple text file which contains 3 columns: <em>hostname</em>, <em>local path</em> and <em>repo URL</em>. <em>csync</em> iterates this file and when hostname matches with machine&rsquo;s hostname, it synchronizes URL under the requested path. For example:</p> <div class="highlight"><pre class="chroma"><code class="language-text" data-lang="text">work content/0 https://personal.git any content/0 ssh://personal.git </code></pre></div><p>And <em>csync</em>:</p> <div class="highlight"><pre class="chroma"><code class="language-bash" data-lang="bash"><span class="cp">#!/bin/bash </span><span class="cp"></span> <span class="nv">synchronized</span><span class="o">=()</span> <span class="k">while</span> <span class="nv">IFS</span><span class="o">=</span><span class="s1">&#39; &#39;</span> <span class="nb">read</span> -r host dir url <span class="k">do</span> <span class="c1"># skip if host doesn&#39;t match; &#34;any&#34; is special host which always matches</span> <span class="k">if</span> <span class="o">[[</span> <span class="s2">&#34;</span><span class="k">$(</span>hostname<span class="k">)</span><span class="s2">&#34;</span> !<span class="o">=</span> <span class="s2">&#34;</span><span class="nv">$host</span><span class="s2">&#34;</span> <span class="o">]]</span> <span class="o">&amp;&amp;</span> <span class="o">[[</span> <span class="s2">&#34;</span><span class="nv">$host</span><span class="s2">&#34;</span> !<span class="o">=</span> <span class="s2">&#34;any&#34;</span> <span class="o">]]</span><span class="p">;</span> <span class="k">then</span> <span class="k">continue</span> <span class="k">fi</span> <span class="c1"># skip if directory was already synchronized</span> <span class="k">if</span> <span class="o">[[</span> <span class="s2">&#34; </span><span class="si">${</span><span class="nv">synchronized</span><span class="p">[@]</span><span class="si">}</span><span class="s2"> &#34;</span> <span class="o">=</span>~ <span class="s2">&#34; </span><span class="nv">$dir</span><span class="s2"> &#34;</span> <span class="o">]]</span><span class="p">;</span> <span class="k">then</span> <span class="k">continue</span> <span class="k">fi</span> <span class="k">if</span> <span class="o">[[</span> ! -e <span class="s2">&#34;</span><span class="nv">$dir</span><span class="s2">&#34;</span> <span class="o">]]</span><span class="p">;</span> <span class="k">then</span> git clone <span class="s2">&#34;</span><span class="nv">$url</span><span class="s2">&#34;</span> <span class="s2">&#34;</span><span class="nv">$dir</span><span class="s2">&#34;</span> <span class="o">&amp;&amp;</span> <span class="nv">synchronized</span><span class="o">+=</span><span class="s2">&#34;</span><span class="nv">$dir</span><span class="s2">&#34;</span> <span class="k">else</span> <span class="nb">pushd</span> <span class="s2">&#34;</span><span class="nv">$dir</span><span class="s2">&#34;</span> &gt; /dev/null <span class="c1"># Better use git-sync instead; read note below!</span> git pull --rebase --autostash <span class="o">&amp;&amp;</span> <span class="nv">synchronized</span><span class="o">+=</span><span class="s2">&#34;</span><span class="nv">$dir</span><span class="s2">&#34;</span> <span class="k">if</span> <span class="o">[</span> -n <span class="s2">&#34;`git status --porcelain`&#34;</span> <span class="o">]</span><span class="p">;</span> <span class="k">then</span> git add -A <span class="o">&amp;&amp;</span> git commit -m <span class="s2">&#34;sync from `hostname`&#34;</span> <span class="o">&amp;&amp;</span> git push <span class="k">fi</span> <span class="nb">popd</span> &gt; /dev/null <span class="k">fi</span> <span class="k">done</span> &lt; <span class="s2">&#34;modules&#34;</span> </code></pre></div><p><em>2020-02-10: Instead of manual git operations as presented above, which are far from being perfect (I once ended with a conflict report instead of file), I switched to the superior <a href="">git-sync</a> script.</em></p> <h2 id="cross-references">Cross References</h2> <p><em>2020-02-12: I changed my approach for Cross References a little. You can read about it in my [later post][notes-2]</em>.</p> <p>One of central ideas of Zettelkasten is creating and maintaining a graph of connections between notes. One way to do it is by tagging each note. I use literal tags for this purpose: ctags, which I already described in my <a href="">other post</a>. Because 95% of them live in notes&rsquo; frontmatters, they are also understood by Hugo, which correctly renders them the to HTML.</p> <p>Notes cross referencing is equally important. Markdown allows creating hyperlinks in form of <code>[descr](</code>. It&rsquo;s super handy when I read it in Vim, because I can place cursor over <code></code> part, press <code>gf</code> and Vim opens that file for me. That&rsquo;s all blows and whistles, but URLs written this way don&rsquo;t go along with Hugo because they&rsquo;re interpreted as relative paths. That&rsquo;s why I use <a href="">shortcodes for cross references</a> provided by Hugo:</p> <div class="highlight"><pre class="chroma"><code class="language-text" data-lang="text">[descr]({{&lt; ref &#34;; &gt;}}) </code></pre></div><p>I combine them with <a href="">UltiSnips</a>, because shortcode syntax is a little too much writing for me:</p> <div class="highlight"><pre class="chroma"><code class="language-text" data-lang="text">snippet ref &#34;Hugo reference&#34; [$1]({{&lt; ref &#34;$2&#34; &gt;}})$0 endsnippet </code></pre></div><p>Additionally I created a markdown-local mapping which helps me with <code>gf</code> part:</p> <div class="highlight"><pre class="chroma"><code class="language-vim" data-lang="vim"><span class="c">&#34; after/ftplugin/markdown.vim</span><span class="err"> </span><span class="err"></span><span class="nx">nnoremap</span> <span class="p">&lt;</span><span class="nx">buffer</span><span class="p">&gt;</span> <span class="p">&lt;</span><span class="nx">cr</span><span class="p">&gt;</span> <span class="nx">f</span>&#34;<span class="nx">gf</span><span class="err"> </span></code></pre></div><h2 id="permalinks">Permalinks</h2> <p>Cross-linking cannot work without permalinks, period. Niklas Luhmann, who initially developed the system, assigned subsequent letters and numbers, indicating positions of notes in a box (like <em>21/3d7</em>, <em>21/3a1</em>). Today people usually recommend using some kind of timestamp for note IDs, which I tried and hated. After a quick brainstorm I invented my own format: <em>some-title-{8-digits-of-UUID}</em>. For example: <em></em>, <em></em> etc. Pretty unconventional, I say, but I had good reasons to do it this way:</p> <ol> <li>File names must start with human readable string (note topic). That&rsquo;s because sometimes I want to browse them from the phone (with limited search capabilities) or with a file manager (usually some kind of netrw clone to quickly go to the <em>next</em> note on the topic).</li> <li>Files must be unique. I don&rsquo;t want conflicts when I create 2 notes on a single topic on 2 different devices.</li> <li>File names must not be too long or unreadable. Timestamps look horrible, but UUIDs have a nice property of being a mix between letters and numbers which is easier for eye in my opinion.</li> </ol> <p>That&rsquo;s why I use first 8 hexadecimal digits of generated random UUID. It allows storing up to 16<sup>8</sup> notes of the same topic which is unique enough for me<sup id="fnref:3"><a href="#fn:3" class="footnote-ref" role="doc-noteref">3</a></sup>. Even if I run into a collision (I doubt it), it will be spotted by me so I can just generate next short UUID.</p> <p>Of course I don&rsquo;t create those UUIDs by hand. I wrote a vim function which does that for me. For example I can write <code>:Note foo</code> and it&rsquo;ll create a file named <em>0/aaa-{short-uuid}</em> (<em>0</em> is my default notebook, but the function detects if I specified any other notebook). When available, it uses <code>uuidgen</code> and Python <code>uuid.uuid4()</code> as a fallback.</p> <p>To ease referencing newly created notes in other notes I also created a command which yanks basename of current buffer: <em>YankNoteName</em>. The command name itself doesn&rsquo;t look very friendly, but actually is the only user command on my system which starts with <em>Y</em>, so I only have to type <code>:Y&lt;Tab&gt;&lt;CR&gt;</code> to call it.</p> <p>I admit that I&rsquo;m no vimscript pro and that I hate every moment spent writing vimscript, so the code might be below any standards. But hey, it works so let&rsquo;s call it a day!</p> <div class="highlight"><pre class="chroma"><code class="language-vim" data-lang="vim"><span class="c">&#34; 99notes.vim</span><span class="err"> </span><span class="err"></span><span class="nx">func</span><span class="p">!</span> <span class="nx">s</span>:<span class="nx">shortuuid</span><span class="p">()</span><span class="err"> </span><span class="err"></span> <span class="k">let</span> <span class="nx">l</span>:<span class="nx">uuid</span> <span class="p">=</span> <span class="s2">&#34;&#34;</span><span class="err"> </span><span class="err"></span> <span class="k">if</span> <span class="nx">executable</span><span class="p">(</span><span class="s1">&#39;uuidgen&#39;</span><span class="p">)</span><span class="err"> </span><span class="err"></span> <span class="k">let</span> <span class="nx">l</span>:<span class="nx">uuid</span><span class="p">=</span><span class="nx">system</span><span class="p">(</span><span class="s1">&#39;uuidgen&#39;</span><span class="p">)</span>[:<span class="m">-2</span>]<span class="err"> </span><span class="err"></span> <span class="k">else</span><span class="err"> </span><span class="err"></span><span class="k">python</span> <span class="o">&lt;&lt;</span> ENDPY <span class="kn">import</span> <span class="nn">vim</span> <span class="kn">from</span> <span class="nn">uuid</span> <span class="kn">import</span> <span class="n">uuid4</span> <span class="n">vim</span><span class="o">.</span><span class="n">command</span><span class="p">(</span><span class="s2">&#34;let l:uuid = &#39;</span><span class="si">%s</span><span class="s2">&#39;&#34;</span><span class="o">%</span> <span class="nb">str</span><span class="p">(</span><span class="n">uuid4</span><span class="p">()))</span> ENDPY<span class="err"> </span><span class="err"></span> <span class="k">endif</span><span class="err"> </span><span class="err"> </span><span class="err"></span> <span class="k">let</span> <span class="nx">l</span>:<span class="nx">spl</span> <span class="p">=</span> <span class="nx">split</span><span class="p">(</span><span class="nx">l</span>:<span class="nx">uuid</span><span class="p">,</span> <span class="s1">&#39;-&#39;</span><span class="p">)</span><span class="err"> </span><span class="err"></span> <span class="nx">return</span> <span class="nx">tolower</span><span class="p">(</span><span class="nx">l</span>:<span class="nx">spl</span>[<span class="m">0</span>]<span class="p">)</span><span class="err"> </span><span class="err"></span><span class="k">endfunction</span><span class="err"> </span><span class="err"> </span><span class="err"></span><span class="nx">func</span><span class="p">!</span> <span class="nx">custom</span>#<span class="m">99</span>notes#<span class="nx">edit</span><span class="p">(</span><span class="nx">name</span><span class="p">)</span><span class="err"> </span><span class="err"></span> <span class="k">let</span> <span class="nx">l</span>:<span class="nx">uuid</span> <span class="p">=</span> <span class="nx">s</span>:<span class="nx">shortuuid</span><span class="p">()</span><span class="err"> </span><span class="err"></span><span class="c"> </span><span class="c"> &#34; assign note to a default notebook</span><span class="err"> </span><span class="err"></span> <span class="k">let</span> <span class="nx">l</span>:<span class="nx">default_notebook</span> <span class="p">=</span> <span class="s2">&#34;0/&#34;</span><span class="err"> </span><span class="err"></span> <span class="k">let</span> <span class="nx">l</span>:<span class="nx">name</span> <span class="p">=</span> <span class="nx">a</span>:<span class="nx">name</span> <span class="p">=~</span> <span class="s1">&#39;.\+\/.\+&#39;</span> ? <span class="nx">a</span>:<span class="nx">name</span> : <span class="nx">l</span>:<span class="nx">default_notebook</span> . <span class="nx">a</span>:<span class="nx">name</span><span class="err"> </span><span class="err"></span> <span class="k">let</span> <span class="nx">l</span>:<span class="nx">fname</span> <span class="p">=</span> <span class="nx">expand</span><span class="p">(</span><span class="s2">&#34;~/docs/content/&#34;</span><span class="p">)</span> . <span class="nx">l</span>:<span class="nx">name</span> . <span class="s2">&#34;-&#34;</span> . <span class="nx">l</span>:<span class="nx">uuid</span> . <span class="s2">&#34;.md&#34;</span><span class="err"> </span><span class="err"></span> <span class="nx">exec</span> <span class="s2">&#34;e &#34;</span> . <span class="nx">l</span>:<span class="nx">fname</span><span class="err"> </span><span class="err"></span> <span class="nx">exec</span> <span class="s2">&#34;keepalt r &#34;</span> . <span class="nx">expand</span><span class="p">(</span><span class="s2">&#34;~/docs/archetypes/;</span><span class="p">)</span><span class="err"> </span><span class="err"></span> <span class="nx">exec</span> <span class="s2">&#34;normal ggdd&#34;</span><span class="err"> </span><span class="err"></span><span class="nx">endfunc</span><span class="err"> </span><span class="err"> </span><span class="err"></span><span class="nx">func</span><span class="p">!</span> <span class="nx">custom</span>#<span class="m">99</span>notes#<span class="nx">copynotename</span><span class="p">()</span><span class="err"> </span><span class="err"></span> <span class="k">let</span> @<span class="s2">&#34; = expand(&#34;</span>%:<span class="nx">t</span>:<span class="nx">r</span>&#34;<span class="p">)</span><span class="err"> </span><span class="err"></span><span class="nx">endfunc</span><span class="err"> </span><span class="err"> </span><span class="err"></span><span class="nx">command</span><span class="p">!</span> <span class="p">-</span><span class="nx">nargs</span><span class="p">=</span><span class="m">1</span> <span class="nx">Note</span> <span class="nx">call</span> <span class="nx">custom</span>#<span class="m">99</span>notes#<span class="nx">edit</span><span class="p">(&lt;</span><span class="nx">q</span><span class="p">-</span><span class="nx">args</span><span class="p">&gt;)</span><span class="err"> </span><span class="err"></span><span class="nx">command</span><span class="p">!</span> <span class="nx">YankNoteName</span> <span class="nx">call</span> <span class="nx">custom</span>#<span class="m">99</span>notes#<span class="nx">copynotename</span><span class="p">(&lt;</span><span class="nx">f</span><span class="p">-</span><span class="nx">args</span><span class="p">&gt;)</span><span class="err"> </span></code></pre></div><p>But even with all of that sometimes I might have to rename a note. For example, when I create a note from <a href="">Markor</a>. That&rsquo;s when <em>rename</em> script comes into play. It not only renames files, but also searches and replaces all their back-references:</p> <div class="highlight"><pre class="chroma"><code class="language-bash" data-lang="bash"><span class="cp">#!/bin/bash </span><span class="cp"></span> <span class="nv">from</span><span class="o">=</span><span class="nv">$1</span> <span class="nv">to</span><span class="o">=</span><span class="nv">$2</span> <span class="nv">frombase</span><span class="o">=</span><span class="si">${</span><span class="nv">from</span><span class="p">##*/</span><span class="si">}</span> <span class="c1"># remove largest prefix</span> <span class="nv">tobase</span><span class="o">=</span><span class="si">${</span><span class="nv">to</span><span class="p">##*/</span><span class="si">}</span> <span class="nv">dir</span><span class="o">=</span><span class="s2">&#34;</span><span class="k">$(</span>dirname <span class="s2">&#34;</span><span class="nv">$from</span><span class="s2">&#34;</span><span class="k">)</span><span class="s2">&#34;</span> <span class="k">for</span> cf in content/*<span class="p">;</span> <span class="k">do</span> <span class="k">if</span> <span class="o">[[</span> ! -d <span class="s2">&#34;</span><span class="nv">$cf</span><span class="s2">&#34;</span> <span class="o">]]</span><span class="p">;</span> <span class="k">then</span> <span class="k">continue</span> <span class="k">fi</span> <span class="nv">files</span><span class="o">=</span><span class="k">$(</span>git -C <span class="s2">&#34;</span><span class="nv">$cf</span><span class="s2">&#34;</span> ls-files <span class="s2">&#34;*.md&#34;</span><span class="k">)</span> <span class="nb">pushd</span> <span class="s2">&#34;</span><span class="nv">$cf</span><span class="s2">&#34;</span> &gt; /dev/null sed -i <span class="s2">&#34;s/</span><span class="nv">$frombase</span><span class="s2">/</span><span class="nv">$tobase</span><span class="s2">/g&#34;</span> <span class="nv">$files</span> <span class="nb">popd</span> &gt; /dev/null <span class="k">done</span> git -C <span class="s2">&#34;</span><span class="nv">$dir</span><span class="s2">&#34;</span> mv <span class="s2">&#34;</span><span class="nv">$frombase</span><span class="s2">&#34;</span> <span class="s2">&#34;</span><span class="nv">$tobase</span><span class="s2">&#34;</span> </code></pre></div><p>But manual searching of files that don&rsquo;t conform to my filename spec is still too much work to do, so I have <em>bulkrename</em> script which fixes my laziness. Well, not exactly fixes laziness itself, but rather its consequences. It takes a note&rsquo;s title and creates a correct file name from it, with spaces replaced by dashes and all.</p> <div class="highlight"><pre class="chroma"><code class="language-bash" data-lang="bash"><span class="cp">#!/bin/bash </span><span class="cp"></span> <span class="nv">skip</span><span class="o">=(</span><span class="s2">&#34;content/;</span> <span class="s2">&#34;content/0/;</span><span class="o">)</span> <span class="k">for</span> dir in content/*<span class="p">;</span> <span class="k">do</span> <span class="k">if</span> <span class="o">[[</span> ! -d <span class="s2">&#34;</span><span class="nv">$dir</span><span class="s2">&#34;</span> <span class="o">]]</span><span class="p">;</span> <span class="k">then</span> <span class="k">continue</span> <span class="k">fi</span> <span class="nv">files</span><span class="o">=</span><span class="k">$(</span>git -C <span class="s2">&#34;</span><span class="nv">$dir</span><span class="s2">&#34;</span> ls-files <span class="s2">&#34;*.md&#34;</span><span class="k">)</span> <span class="k">for</span> f in <span class="nv">$files</span><span class="p">;</span> <span class="k">do</span> <span class="nv">oldf</span><span class="o">=</span><span class="s2">&#34;</span><span class="nv">$dir</span><span class="s2">/</span><span class="nv">$f</span><span class="s2">&#34;</span> <span class="c1"># skip explicitly listed files</span> <span class="k">if</span> <span class="o">[[</span> <span class="s2">&#34; </span><span class="si">${</span><span class="nv">skip</span><span class="p">[@]</span><span class="si">}</span><span class="s2"> &#34;</span> <span class="o">=</span>~ <span class="s2">&#34; </span><span class="nv">$oldf</span><span class="s2"> &#34;</span> <span class="o">]]</span><span class="p">;</span> <span class="k">then</span> <span class="c1"># check if array contains a value</span> <span class="nb">echo</span> <span class="s2">&#34;Skipping </span><span class="nv">$oldf</span><span class="s2"> (excluded)&#34;</span> <span class="k">continue</span> <span class="k">fi</span> <span class="c1"># skip files which already match a filename pattern (</span> <span class="c1"># where shortuuid are first 8 hexadecimal numbers od full UUID</span> <span class="k">if</span> <span class="o">[[</span> <span class="s2">&#34;</span><span class="nv">$f</span><span class="s2">&#34;</span> <span class="o">=</span>~ ^.+-<span class="o">[</span>a-f0-9<span class="o">]{</span>8<span class="o">}</span>.md$ <span class="o">]]</span><span class="p">;</span> <span class="k">then</span> <span class="nb">echo</span> <span class="s2">&#34;Skipping </span><span class="nv">$oldf</span><span class="s2"> (name correct)&#34;</span> <span class="k">continue</span> <span class="k">fi</span> <span class="nv">shortuuid</span><span class="o">=</span><span class="k">$(</span>uuidgen <span class="p">|</span> sed <span class="s1">&#39;s/-.*//&#39;</span><span class="k">)</span> <span class="nv">title</span><span class="o">=</span><span class="s2">&#34;</span><span class="k">$(</span>awk -F <span class="s1">&#39;[ ().?!,/-]+&#39;</span> <span class="s1">&#39;/^title:/ {for(i=2;i&lt;NF;i++) printf &#34;%s-&#34;, tolower($i); print tolower($NF)}&#39;</span> <span class="s2">&#34;</span><span class="nv">$oldf</span><span class="s2">&#34;</span><span class="k">)</span><span class="s2">&#34;</span> <span class="nv">newf</span><span class="o">=</span><span class="s2">&#34;</span><span class="nv">$dir</span><span class="s2">/</span><span class="nv">$title</span><span class="s2">-</span><span class="nv">$shortuuid</span><span class="s2">.md&#34;</span> <span class="nb">echo</span> <span class="s2">&#34;Renaming </span><span class="nv">$oldf</span><span class="s2"> -&gt; </span><span class="nv">$newf</span><span class="s2">&#34;</span> ./rename <span class="s2">&#34;</span><span class="nv">$oldf</span><span class="s2">&#34;</span> <span class="s2">&#34;</span><span class="nv">$newf</span><span class="s2">&#34;</span> <span class="k">done</span> <span class="k">done</span> </code></pre></div><p>One thing it lacks is a progress bar. HAHA. HAHA. HA. And now I&rsquo;m thinking about implementing one (which shouldn&rsquo;t be that hard). Thank you very much, brain, I love you too, that&rsquo;s exactly what my notes setup lacks: a progress bar in a script which will be used maybe once a year or less. Oh boy, have you seen these nice progress bars that recent pytest versions have? I think they even have CONCURRENT progress bars in pytest-xdist! Maybe I can get one too? Because I certainly can run this script concurrently!</p> <section class="footnotes" role="doc-endnotes"> <hr> <ol> <li id="fn:1" role="doc-endnote"> <p>Andrzej Sapkowski - The Lady of the Lake <a href="#fnref:1" class="footnote-backref" role="doc-backlink">&#x21a9;&#xfe0e;</a></p> </li> <li id="fn:2" role="doc-endnote"> <p><code>python3 -m http.server 9999</code> <a href="#fnref:2" class="footnote-backref" role="doc-backlink">&#x21a9;&#xfe0e;</a></p> </li> <li id="fn:3" role="doc-endnote"> <p>Trivia: it&rsquo;d take over 136 years to create that many notes if a note was created every second. <a href="#fnref:3" class="footnote-backref" role="doc-backlink">&#x21a9;&#xfe0e;</a></p> </li> </ol> </section> Don't change alternate file in Vim Wed, 08 Jan 2020 00:00:00 +0000 <p>It seems that in Vim <code>:read &lt;file&gt;</code> command (used for inserting other file&rsquo;s contents below the cursor in current buffer) changes alternate file to the inserted one (you can jump between current and alternate file with <code>C-^</code> or <code>C-6</code>, which is super handy). It&rsquo;s not obvious and in my opinion it breaks the rule of least astonishment, because I totally didn&rsquo;t ask Vim to do anything outside of a current buffer (and buffer list shown by <code>:ls</code> isn&rsquo;t even changed and might not contain that file at all). This behaviour complicated my workflow for quickly adding new notes from templates.</p> <p>Fortunately (TIL) there&rsquo;s easy fix &ndash; or rather workaround &ndash; for this. We can prefix any Vim command with <code>:keepalt</code> and this command won&rsquo;t change alternate file in any way. So <code>:r &lt;file&gt;</code> becomes <code>:keepalt r &lt;file&gt;</code>. Super handy and good to know, especially for making custom functions, plugins etc.</p> Fixed RSS Sat, 04 Jan 2020 00:00:00 +0000 <p>For a moment I broke <a href="">RSS feed</a> and then I broke it again by updating it with all previous posts. I updated Hugo to a new version due Markdown rendering issues in the <a href="">last</a> post and it seems that Hugo team introduced quite a lot backward incompatible changes in recent versions of Hugo. Anyway, RSS feed should be fixed now and it should display 15 last articles. Sorry for inconvenience!</p> Ctags For Markdown Notes Fri, 03 Jan 2020 00:00:00 +0000 <p>I keep my notes in plaintext Markdown files. Previously I used vimwiki for managing them but I felt that it was somehow bloated and its Markdown engine slightly differed from the more typical engine used in Markor (which is an Android app which I sometimes use to view and edit notes on the phone). Since I switched away from vimwiki to plain Markdown, there was one thing I missed: tags.</p> <p>Generally I tag my notes and since recently I use 2 types of tags: inline tags which resemble org-mode style of tagging (<em>:tag1:tag2:tag3:</em>), and document-wide tags which are placed in YAML Front Matter of my Markdown files. Typical note looks like this:</p> <div class="highlight"><pre class="chroma"><code class="language-markdown" data-lang="markdown">--- title: Some title tags: [foo, bar] --- <span class="k">-</span> Lorem ipsum :baz:blah: <span class="k">-</span> dolor sit amet :consecteur: </code></pre></div><p>Vimwiki impements its custom way of hand-generating tags file for org-mode style tags. Thanks to it I can jump in vim directly to the place where selected tag is found. For example with <code>:tjump /&lt;tab&gt;</code> I can easily search through all tags in all of my notes.</p> <p>Today I managed to replicate the way of generating tags for my notes, thanks to <a href="">extensibility</a> of ctags implementation that I use: <a href="">Universal Ctags</a>.</p> <p>Long story short, we have to create a file in <em>~/.ctags.d</em> with the following content:</p> <div class="highlight"><pre class="chroma"><code class="language-txt" data-lang="txt">--langdef=notes --languages=notes --kinddef-notes=t,tag,tags --kinddef-notes=d,doctag,doctags --_tabledef-notes=main --_tabledef-notes=frontmatter --_tabledef-notes=fmtags --_mtable-regex-notes=main/---//{tenter=frontmatter} --_mtable-regex-notes=main/:([a-zA-Z][a-zA-Z0-9]*):/\1/t/{mgroup=1}{_advanceTo=1end} --_mtable-regex-notes=main/.// --_mtable-regex-notes=frontmatter/^tags: *//{tenter=fmtags} --_mtable-regex-notes=frontmatter/---//{tleave} --_mtable-regex-notes=frontmatter/.// --_mtable-regex-notes=fmtags/([a-zA-Z][a-zA-Z0-9]*)/\1/d/ --_mtable-regex-notes=fmtags/\]//{tleave} --_mtable-regex-notes=fmtags/://{tleave} --_mtable-regex-notes=fmtags/---//{tleave}{_advanceTo=0start} --_mtable-regex-notes=fmtags/.// </code></pre></div><p>This defines a new regex parser for files with <em>.md</em> extension. This parser is aware of its context (i.e. whether it is inside Front Matter or not) and works like this:</p> <figure class="center"> <img src="" alt="Ctags table stack"/> <figcaption> <p>Ctags table stack</p> </figcaption> </figure> <ol> <li> <p>Start in <em>main</em> table (context). This is generic context able to capture org-mode style tags. Thanks to <code>{_advanceTo=1end}</code> directive, a single colon between tags acts as a separator (i.e. will be the start character for the next iteration of ctags regex engine).</p> </li> <li> <p>When we&rsquo;re inside main context, look out for 3 dashes. When they&rsquo;re found, push <em>frontmatter</em> context on top of the stack. This context itself acts as intermediate context: it will pop itself when another 3 dashes occur or it will push <em>fmtags</em> context when <em>tags:</em> text is found.</p> </li> <li> <p>When in <em>fmtags</em> context, any alphanumeric string will be matched as a tag. This automatically handles two forms of YAML&rsquo;s lists: one-line inside brackets, and multiline:</p> <div class="highlight"><pre class="chroma"><code class="language-yaml" data-lang="yaml"><span class="k">tags</span><span class="p">:</span><span class="w"> </span><span class="p">[</span>foo<span class="p">,</span><span class="w"> </span>bar<span class="p">]</span><span class="w"> </span><span class="w"></span><span class="k">tags</span><span class="p">:</span><span class="w"> </span><span class="w"> </span>- foo<span class="w"> </span><span class="w"> </span>- bar<span class="w"> </span></code></pre></div></li> <li> <p><em>fmtags</em> context pops when 3 dashes or colon (some other metadata) or list closing bracket is found. 3 dashes mean end of Front Matter and thanks to <code>{_advanceTo=0start}</code> after the pop they&rsquo;re found again by <em>frontmatter</em> table, which can in turn pop again and bring us back to <em>main</em> context.</p> </li> <li> <p>All tags must start with a letter (and not a number) to avoid interpreting time as a tag (e.g. 12:00:13 AM).</p> </li> </ol> <p>Ctags regex parser uses <a href="">Extended Regular Expression</a> syntax for regular expressions as defined by POSIZ and implemented by glibc. Unfortunately it has many limitations. For example, my tag-matching regular expression (<code>([a-zA-Z][a-zA-Z0-9]*)</code>) fails to match non-ASCII characters, which is kinda braindead in the age of Unicode. For now I changed some of my Polish tags to their English names, but I&rsquo;ll keep on searching for more elegant solution.</p> Migadu Thu, 21 Nov 2019 00:00:00 +0000 <p>I just read on Mastodon Drew DeVault&rsquo;s <a href="">recommendation</a> of <a href="">Migadu</a>, which is a small, independent e-mail provider from Switzerland that looks too good to be real. Never heard of them before but I got curious and created a free account. I haven&rsquo;t switched yet (I must re-configure DNS), but I already had a pleasant experience. I checked their website and haven&rsquo;t seen anything about server-side mail filtering so I reached their support about that. About 15 minutes later I was replied by Miki from Migadu that they already implemented Sieve backend, but need few more weeks to finish their management app.</p> <p>Currently I&rsquo;m happy FastMail customer, but I&rsquo;d like to migrate to a service that doesn&rsquo;t charge me per-mailbox. I&rsquo;d love my wife to have mail address in our home domain, but double paying for that sucks. Especially that our son will probably need a mail in future as well. Migadu has 3 paid plans based on number of mails you send every day. Unless I&rsquo;m a part of heavy discussion on some issue tracker, I usually send 2-3 mails a month so today&rsquo;s mail to Migadu almost filled my monthly quota. :)</p> <p>So I&rsquo;ll probably migrate myself in the following days as I don&rsquo;t use server-side filters THAT hard (with Mutt mail management is easy), but it&rsquo;s a no-go for my wife. Well, good luck Migadu, I&rsquo;ll be checking your changelog daily!</p> Configuring msmtp on Debian Buster Fri, 30 Aug 2019 00:00:00 +0000 <p>Recently I decided to switch all my SMTP needs to msmtp, which is a command-line SMTP client. To configure it most tutorials on the Internet suggest to simply create <em>~/.msmtprc</em> configuration file. However, <code>man 1 smtp</code> says that it&rsquo;s also possible to keep configuration in <em>$XDG_CONFIG_HOME/msmtp/config</em> (so in practice in <em>~/.config/msmtp/config</em>), which I&rsquo;d very much prefer, because that&rsquo;s where I store configuration files for other programs.</p> <p>So as always I created a config file in <em>~/config/mail/.config/msmtp/config</em> and symlinked it with GNU Stow to ~/.config. Of course that didn&rsquo;t work (otherwise I wouldn&rsquo;t write this article) and msmtp repeatedly reported <em>no configuration file available</em>, even if it was there all the time.</p> <p>Only blind luck directed me to Debian Bug Tracker which mentions AppArmor - a name which I heard before but I&rsquo;m not very familiar with. In short, AppArmor offers per-application, profile-based Mandatory Access Control system. AppArmor&rsquo;s <a href="">profile for msmtp</a> allows accessing only certain, specific files. For example msmtp can access only files under <em>$HOME/.config/msmtp</em>.</p> <p>There&rsquo;s one problem though. AppArmor resolves symbolic links for directories to their original paths. This is what GNU Stow creates by default - topmost symlink it can afford, in this case a symlink to <em>msmtp</em> config directory. So even if msmtp itself tries to read <em>~/.config/msmtp/config</em>, AppArmor registers that as a request for access to <em>~/config/mail/.config/msmtp/config</em>. And it leaves a syslog trace of denying such request:</p> <pre><code class="language-syslog" data-lang="syslog">kernel: [249329.035844] audit: type=1400 audit(1567152137.648:32): apparmor=&quot;DENIED&quot; operation=&quot;open&quot; profile=&quot;/usr/bin/msmtp&quot; name=&quot;/home/mgoral/config/mail/.config/msmtp/config&quot; pid=14000 comm=&quot;msmtp&quot; requested_mask=&quot;r&quot; denied_mask=&quot;r&quot; fsuid=1000 ouid=1000 </code></pre><p>There are several ways to resolve this issue:</p> <ol> <li>disable AppArmor for msmtp (e.g. <code>sudo aa-disable msmtp</code>);</li> <li>add local rules for msmtp in <em>/etc/apparmor.d/local/usr.bin.msmtp</em>;</li> <li>symlink individual files;</li> <li>use <em>~/.msmtprc</em>;</li> <li>create <a href="">alias for directory symlink</a>.</li> </ol> <p>I chose to simply use <em>~/.msmtprc</em>, because that&rsquo;s the simplest thing which works. All the other require root access, which is yet another password which blocks automatic deployment of my dotfiles.</p> <p>AppArmor also caused a second problem.</p> <p>Instead of storing password in plain text in configuration file, it can read it from the output of arbitrary command. As I use <a href="">pass</a> for all my password management needs and I already have a wrapper for querying certain fields from encrypted files (designed specifically for cases like that) it was no-brainer to use this script for msmtp as well. Didn&rsquo;t work at all, of course. :)</p> <p>The script is called <em>pass-get</em>. Thanks to AppArmor I can&rsquo;t use it, because msmtp&rsquo;s profile restricts access only to few external executables. Fortunately <em>pass</em> command is on the list, as well as <em>head</em>, so I just had to partially duplicate my <em>pass-get</em> command. Which is counter-productive, because if I ever have to change <em>pass-get</em>, I&rsquo;m almost sure I&rsquo;ll forget about that one small configuration file for a program which was installed, configured and forgotten.</p> <p>Anyway, here&rsquo;s the line:</p> <div class="highlight"><pre class="chroma"><code class="language-cfg" data-lang="cfg"><span class="na">passwordeval &#34;pass my-email | head -1&#34;</span> </code></pre></div> Using systemd for X sessions Wed, 29 May 2019 00:00:00 +0000 <p>Recently I wrote an <a href="">article about starting X sessions</a> in Debian. One of stages of Xsession is running <em>~/.xsessionrc</em> script, which, according to <em>xsession(5)</em> manual should be a source of global environment variables. It can be, however, a convenient place for keeping all programs which should start together with X session - because, generally speaking, it is executed for every session type, no matter what. I used this approach for several years and, for reasons highlighted later, I find it suboptimal. In this article I&rsquo;d like to explore another possibility - using systemd user mode for managing user X sessions.</p> <h2 id="xsessionrc-approach">xsessionrc approach</h2> <p>Our xsessionrc can look like this:</p> <div class="highlight"><pre class="chroma"><code class="language-bash" data-lang="bash"><span class="cp">#!/bin/bash </span><span class="cp"></span>xrdb - merge <span class="s2">&#34;</span><span class="nv">$HOME</span><span class="s2">/.Xresources </span><span class="s2"> </span><span class="s2">mpd &amp; </span><span class="s2">xterm -e ncmpcpp &amp; </span><span class="s2"> </span><span class="s2">firefox &amp; </span><span class="s2">thunderbird &amp; </span></code></pre></div><p>There are several problems with such approach (and, to be fair, with any simple script used to run a session):</p> <ol> <li>Programs often depend on each other. For example ncmpcpp (client of <abbr title="Music Player Daemon">MPD</abbr>) doesn&rsquo;t make sense to start before MPD itself, but MPD might delay its start for any reason or even not start at all e.g. due to configuration error.</li> <li>Some programs (like MPD) won&rsquo;t automatically exit with user session. If session is restarted, it will cause problems, because they might use resources (like sockets) which are already used their other instances.</li> <li>If programs break you have to manually restart it - which usually means grepping through xsessionrc in search of exact set of switches used for its invocation. I&rsquo;m sorry, I don&rsquo;t remember my longitude and latitude passed to redshift.</li> <li>Programs start simultaneously and they all write their output to <em>~/.xsession-errors</em> at the same time. If that&rsquo;s not enough, GTK programs are typically VERY verbose and soon finding anything in xsession-errors becomes a horror story. Sure, you can do something like <code>mpd &gt;mpd.out 2&gt;mpd.err</code>, but in practice I&rsquo;ve never heard about anyone doing something like that.</li> <li>Probably it&rsquo;s yet another mechanism which manages session. For example, i3 can also start programs from inside its configuration file and most desktop environments autostarts programs depending on their own sets of rules. I like having a choice, but I hate using several mechanisms at once, each with its own set of quirks and design choices.</li> </ol> <p>That&rsquo;s why in the back of my had I had this idea of creating a uniform system for managing my login sessions. I already used systemd for some programs (especially for some in-house scripts at work, which tend to break every now and then so systemd auto-restarts them) and was quite happy with it, so I thougt to give it a try and manage all of my session startup this way.</p> <h2 id="systemd-approach">systemd approach</h2> <p>First of all, I don&rsquo;t want to reconfigure my whole system and right now <em>Xsession</em> script is rather tightly coupled with the rest of Debian. If I ever have to configure X session on a new computer, I want to simply symlink some files with GNU Stow (you can read about how I approach my configuration in <a href="">dotfiles series</a>). Besides, I&rsquo;d be rewriting big parts of Xsession anyway, because it out-of-the-box contains some very handy bits (like <strong>injecting environment to systemd via DBus</strong>, which usually is a separate topic and in Debian we have it for free).</p> <p>Long story short: I modified <em>~/.xsession</em> to contain exactly 2 lines:</p> <div class="highlight"><pre class="chroma"><code class="language-sh" data-lang="sh"><span class="cp">#!/bin/sh </span><span class="cp"></span>systemctl --user start --wait </code></pre></div><p>Above invocation of systemctl can be of course placed in a <em>*.desktop</em> file. Just keep in mind that it&rsquo;s important to use <code>--wait</code> flag, which disallows immediate exit, which in turn would stop X session right after it.</p> <p><em></em> looks like this:</p> <div class="highlight"><pre class="chroma"><code class="language-ini" data-lang="ini"><span class="k">[Unit]</span> <span class="na">Description</span><span class="o">=</span><span class="s">X session managed by systemd</span> <span class="na">BindsTo</span><span class="o">=</span><span class="s"></span> </code></pre></div><p><em></em> is a synchronization point. User units (started with <code>systemctl --user</code>) cannot wait for targets from system-wide systemctl calls, like <em></em>, <em></em> etc. That&rsquo;s because user units are spawned in a separate daemon, spawned for each user, which has no knowledge of other systemd instances. By hooking into built-in Xsession mechanism we ensured that when <em></em> becomes active, all resources needed by graphical programs (<code>$DISPLAY</code> and <code>$XAUTHORITY</code>) are available.</p> <p><em></em> binds to a special target: <strong></strong>, which is active as long as any other active unit requires it. It also acts as an alias for any graphical session (such as GNOME, KDE, i3, awesome, &hellip;): other units, which are part of X session should contain <code></code>. This way they&rsquo;ll be stopped when <em></em> stops. They also don&rsquo;t need to be changed if i3wm were to be replaced with e.g. some other window manager.</p> <p>Example unit started in session looks like this:</p> <div class="highlight"><pre class="chroma"><code class="language-ini" data-lang="ini"><span class="k">[Unit]</span> <span class="na">Description</span><span class="o">=</span><span class="s">Compton compositor for X11</span> <span class="na">PartOf</span><span class="o">=</span><span class="s"></span> <span class="k">[Service]</span> <span class="na">ExecStart</span><span class="o">=</span><span class="s">/usr/bin/compton --config &#34;%h/.config/compton/compton.conf&#34;</span> <span class="k">[Install]</span> <span class="na">WantedBy</span><span class="o">=</span><span class="s"></span> </code></pre></div><p><em></em> is explicitly required by only one other unit: <em>i3wm.service</em>, which handles starting and stopping window manager:</p> <div class="highlight"><pre class="chroma"><code class="language-ini" data-lang="ini"><span class="k">[Unit]</span> <span class="na">Description</span><span class="o">=</span><span class="s">i3 Window Manager</span> <span class="na">PartOf</span><span class="o">=</span><span class="s"></span> <span class="k">[Service]</span> <span class="na">ExecStart</span><span class="o">=</span><span class="s">/usr/local/bin/i3</span> <span class="na">ExecStopPost</span><span class="o">=</span><span class="s">/bin/systemctl --user stop</span> <span class="na">Restart</span><span class="o">=</span><span class="s">on-failure</span> <span class="k">[Install]</span> <span class="na">RequiredBy</span><span class="o">=</span><span class="s"></span> </code></pre></div><p>Note that <em></em> itself doesn&rsquo;t require anything by itself (via <code>Requires=</code>). That&rsquo;s because I prefer to add and remove programs to autostart via <code>systemctl enable</code> and <code>systemctl disable</code> instead of editing systemd unit files.</p> <p>Another interesting part is <code>ExecStopPost=</code>, which stops graphical-session (and all of its parts) whenever i3 quits. To quit a session, <em></em> must be stopped one way or another and I decided to keep a behaviour that I&rsquo;m familiar with: window manager acts as a session&rsquo;s master and whenever it quits, the whole session is killed as well - it didn&rsquo;t always work for me before, but it&rsquo;s one of the features of systemd.</p> <h3 id="tmux">tmux</h3> <p>Killing the whole session has some quirks though. When I log out, systemd kills tmux server.</p> <p>I like the way systemd works because it&rsquo;s easy enough to mark specific programs which should be kept after session ends and by default I don&rsquo;t want most of daemons to survive logging out. For instance: if I run HTTP server, then it&rsquo;s purposefully run to display some personal pages and it has no purpose to exist after I log out. If I run syncthing, I want it to operate only when I&rsquo;m logged to my system. If I wanted a daemon to run all the time, then it&rsquo;s a system service and should be run via systemctl without <code>--user</code> switch (syncthing for example ships with systemd units which support exactly that use case - useful for infrastructure with a single &ldquo;master&rdquo; syncthing server).</p> <p>However, programs like tmux and screen are special, because they&rsquo;re specifically designed to daemonize to survive user sessions. This can be achieved by running them through <code>systemd-run --remain-after-exit</code> or even <code>systemd-run --scope --user</code> (which should <a href="">finally work</a> in recent systemd versions).</p> <p>To do this automatically, tmux can be aliased:</p> <div class="highlight"><pre class="chroma"><code class="language-sh" data-lang="sh">tmux<span class="o">()</span> <span class="o">{</span> systemd-run --scope --user tmux <span class="s2">&#34;</span><span class="nv">$@</span><span class="s2">&#34;</span> <span class="o">}</span> </code></pre></div><p>And basically this is everything. Just put all <em>*.service</em> files in <em>~/.config/systemd/user</em> and enable/disable the ones which you want to autostart by <code>systemctl --user enable/disable</code>. It&rsquo;s super convienient, because you don&rsquo;t have to remember exact commands, just run <code>systemctl --user enable redshift.service</code>.</p> <p>Source code presented in this article is also available in <a href="">git repository</a>, so take a look if it makes more sense to you.</p> Blog changes Fri, 29 Mar 2019 21:00:00 +0000 <p>I finally did it. I refreshed the theme of my blog, which is something I&rsquo;ve wanted to do for the last year or so. The new one is fabulous <a href="">tale-hugo</a>, although I added a lot of very personal customizations. It&rsquo;s quite minimalistic, but I loved it from the first sight - and I don&rsquo;t use that phrase very often.</p> <p>Additionally I moved my blog from the subdomain to the main domain, so Hugo now handles my whole website and all permalinks stop working riiiight now. Because who cares about backward compatibility, right?</p> Xsession in Debian Fri, 22 Mar 2019 12:00:00 +0000 <p>Xsession is a default way of starting, well, nothing less but X sessions in Debian. You can read about it in detail by running <code>man xsession</code>, or read source code of the script itself: <em>/etc/X11/Xsession</em>. It&rsquo;s not very complicated, but because it&rsquo;s a shell script, it can be a little intimidating for newcomers. That&rsquo;s why I&rsquo;m publishing this small introduction.</p> <p>Keep in mind that it&rsquo;s based on Debian Stretch (9.0, stable as of writing this article), so things might be anywhere from &ldquo;a little different&rdquo; to &ldquo;totally irrelevant&rdquo; for other Debian releases.</p> <h2 id="xsessiond">Xsession.d</h2> <p>Xsession is itself quite straightforward. It first sets some environment variables, does a little bookkeeping and then it sources all scripts from <em>/etc/X11/Xsession.d</em>, one by one, with help of <code>run-parts</code>. This is the place where a lot of recommendations for Debian X session management is implemented, like that a good place to auto start X applications is <em>~/.xsessionrc</em>.</p> <p>It is important to remember that all scripts in <em>Xsession.d</em> are <strong>sourced</strong>, not executed. This way all variables, functions and flags (global <code>set +e</code>) defined before are available for all consecutive scripts.</p> <p>Some third-party packages may install their own scripts into <em>Xsession.d</em> (for example flatpak does that). They&rsquo;re usually transparent to end-user, but sometimes they can be controlled one way or the other. Common way of adding simple, user-controllable flags is <em>Xsession.options</em> file. Keep in mind however that <code>has_option</code> function is defined by <a href="#20x11-common-process-args"><code>20x11-common_process-args</code></a> so if you want to process options with help of this function, you have to ensure that your script is sourced after <code>common_process_args</code>.</p> <p>Other scripts in <em>Xsession.d</em> are shipped by default for all Debian installations with X capabilities. These are the important ones, because they define common environment variables and functions which are re-used in other scripts.</p> <h3 id="20x11-common_process-args"><code>20x11-common_process-args</code></h3> <p>This is where <code>$STARTUP</code> variable is initialized. It&rsquo;s quite important, because it names a script or program which will be executed later as a window/session manager.</p> <p>Users can directly affect executed command by passing a single optional argument to Xsession and <code>common_process-args</code> will process it and assign to <code>$STARTUP</code>. This argument accepts also some special names, which are handled differently:</p> <ul> <li>if it is set to <strong>failsafe</strong>, x-terminal-emulator will be executed (usually xterm, but it can be changed by update-alternatives mechanism) and processing of further scripts will be terminated;</li> <li>if it is set to <strong>default</strong>, the script won&rsquo;t set <code>$STARTUP</code>, which will result in a deduction of default session manager by <a href="#50x11-common-determine-startup"><code>50x11-common-determine-startup</code></a>.</li> </ul> <p>This script also defines <code>has_option</code> function, used for parsing of <em>Xsession.options</em> file.</p> <h3 id="30x11-common_xresources-40x11-common_xsessionrc"><code>30x11-common_xresources</code>, <code>40x11-common_xsessionrc</code></h3> <p>These 2 scripts are sourced right after Xsession arguments are processed.</p> <p>First one merges all system-wide X resources files found in <em>/etc/X11/Xresources</em> and user-specific one from <em>~/.Xresources</em>.</p> <p>Second simply sources <em>~/.xsessionrc</em> file, so all user-defined environment variables found there will be globally available for the whole X session from now on and commands will be executed (note that commands should typically be run in a background to not block execution of Xsession).</p> <h3 id="35x11-common_xhost-local"><code>35x11-common_xhost-local</code></h3> <p>This script simply uses <em>xhost</em> to give access to X server to the local user.</p> <h3 id="50x11-common_determine-startup"><code>50x11-common_determine-startup</code></h3> <p>If Xsession wasn&rsquo;t passed any argument, or if passed argument is not executable, <code>$STARTUP</code> won&rsquo;t be set and <code>common_determine-setup</code> will run the default discovery mechanism of what should be run instead:</p> <ol> <li><em>~/.xsession</em> and <em>~/.Xsession</em> are tried (because some users apparently like to start their file names with capital letters).</li> <li>default system-wide session manager: <em>x-session-manager</em></li> <li>default system-wide window manager: <em>x-window-manager</em></li> <li>default system-wide terminal emulator: <em>x-terminal-emulator</em></li> </ol> <h3 id="90x11-common_ssh-agent"><code>90x11-common_ssh-agent</code></h3> <p>This one often confuses a lot of people. Historically, there always was a problem with finding a good and convienient way to run SSH Agent and even today a lot of people run it from their <em>.bashrc</em> every time new shell starts. Debian solved it by binding its start with start of X session.</p> <p>Anyway, if no previous ssh-agent appears to be running (e.g. gpg-agent with enabled support of OpenSSH Agent Protocol), <code>$STARTUP</code> will be wrapped with ssh-agent call and executed as agent&rsquo;s subprocess.</p> <h3 id="99x11-common_start"><code>99x11-common_start</code></h3> <p>Finally we execute our session manager, as <code>common_start</code> is typically the last file sourced by Xsession. It&rsquo;s quite straightforward:</p> <div class="highlight"><pre class="chroma"><code class="language-sh" data-lang="sh"><span class="nb">exec</span> <span class="nv">$STARTUP</span> </code></pre></div><p>It replaces (via <code>exec</code>) the current shell with command stored in <code>$STARTUP</code> variable. This way executed command inherits the whole environment set up by all previous scripts.</p> <h2 id="when-xsession-will-be-run">When Xsession will be run</h2> <p>Typically Xsession is sourced (not executed) whenever <em>/etc/X11/xinit/xinitrc</em> is executed. This is handled by startx script (but not xinit, which is wrapped by startx and by itself will only load <em>.xinitrc</em> from user&rsquo;s home directory).</p> <p>Xsession is also sourced by display managers, but there are no clear rules when Xsession is used, when it isn&rsquo;t, and what arguments are passed to it. Documentation of most Display Managers also lacks this knowledge so often all you&rsquo;re left is the good old trial and error.</p> <p>This is the source of a lot of headaches about files not being read (or being read unintentionally) during startup. The rule of thumb is: configure your system to ensure Xsession is executed and you&rsquo;re back on a well-known waters.</p> <h3 id="lightdm">Lightdm</h3> <p>Lightdm allows configuring it to use a <em>session-wrapper</em>, which is a script to run session with. This is pre-configured to be Xsession in <em>/usr/share/lightdm/lightdm.conf.d/01_debian.conf</em>.</p> <p>Typically all available types of X sessions are read in a form of .desktop files from <em>/usr/share/xsessions</em> and some other directories. On login screen Lightdm only allows selecting one of those and passes the value of its <em>Exec</em> field to the session-wrapper as the first argument, which will cause executing correct program.</p> <p>There is a special .desktop file there, used for so-called &ldquo;default sessions&rdquo;, which has <em>Exec=default</em>. It will be detected by <a href="#20x11-common-process-args"><code>20x11-common_process-args</code></a> and will result in running a fallback mechanism described in <a href="#50x11-common-determine-startup"><code>50x11-common-determine-startup</code></a>.</p> <h3 id="sddm">SDDM</h3> <p>SDDM is a little more sane than Lightdm. It doesn&rsquo;t have hidden settings or use any obscure features. For custom sessions it executes <em>/etc/sddm/Xsession</em>, which ultimately sources <em>/etx/X11/Xsession</em> itself.</p> <p>SDDM tries to source .profile, .bash_profile, .zprofile etc. It&rsquo;s a little unusual, because these files are usually boud to terminal sessions and graphical sessions usually source .xprofile.</p> <h3 id="gdm3">GDM3</h3> <p>GDM3 runs a bunch of scripts stored in different directories in <em>/etc/gdm3</em>, but ultimately one of them is <em>/etc/gdm3/Xsession</em>. Yep, it has a custom Xsession script, which is similar in some places to the default one, but also adds a bunch of its own logic. For example, if no argument is passed, it defaults to &ldquo;failsafe&rdquo; mode.</p> <p>It doesn&rsquo;t source the default Xsession, but sources scripts from <em>/etc/X11/Xsession.d</em> by itself. Before that it also tries to source profile and xprofile files.</p> <p>It also aliases &ldquo;default&rdquo; session with a word &ldquo;custom&rdquo;.</p> <!-- vim: set tw=80 : --> End of Dotfiles series Tue, 04 Dec 2018 10:00:00 +0000 <p>This series will be discontinued. I stopped writing at all because I couldn&rsquo;t motivate myself to write a next part of <em>Dotfiles</em> series and I felt that it should be finished before anything else. A year passed and exactly nothing happened with <em>Dotfiles</em>. I hope that existing 3 parts of will be useful for someone, but I won&rsquo;t go into this topic any further. Not that there is a lot more to say.</p> <p>Hopefully this way I&rsquo;ll be able to force myself to write more.</p> PGP key refreshed Mon, 03 Dec 2018 17:40:00 +0000 <p>I&rsquo;m in a process of moving this site and my mail addresses to the new domain: Due to this I added my new e-mails to PGP key. Old domain ( will continue to work work at least until mid-May 2019. After that I&rsquo;ll decide whether I need to continue renewing that domain or will it be abandoned.</p> <p>You can still <a href="">download it</a> from the same location. Fingerprint: <strong>0423 DE59 98D1 2C33 E599 CDCF E3DD DA4D C45F 58CB</strong> (not changed)</p> New Year's resolution Mon, 05 Feb 2018 10:58:32 +0100 <p>It&rsquo;s a little late to make a New Year&rsquo;s resolution, but I&rsquo;ll make one nevertheless. From now I&rsquo;m writing only in English - at least on this blog.</p> <p>This move is not about the audience, thought I admit - it&rsquo;d be nice to show some of my previous posts to non-Polish speakers (and as I&rsquo;m getting more and more confident about my work, I&rsquo;ll probably have something more to share in the future). It&rsquo;s a known fact that programming world is dominated by English so I often had to translate some phrases to Polish, at least if I didn&rsquo;t want my articles to turn into a weird mix of Polish and English words randomly thrown here and there (and I didn&rsquo;t, but often it was <a href="">unavoidable</a>). It&rsquo;s a burden because usually Polish lacks good equivalents for some particular words so I either had to skip these or come up with rough and unnatural neologisms. So I figured that using English all over the place will better fit the purpose of these posts.</p> <p>The only problem is my recent &ldquo;Dotfiles&rdquo; series. I don&rsquo;t want to leave it partially Polish and partially English and because I&rsquo;m not particularly interested in running multilingual blog, I&rsquo;ll probably start translating first three parts to English English in the first place.</p> <p>Stay tuned!</p>