atomicules

Push propelled program tinkerer and picture maker.

Escaping quotes in pwman2op

As a separate post to the initial one because it’s interesting.

Despite this pretty much being a single-use bit of code that has already done it’s job it was bugging me that it didn’t handle certain things, namely quotation marks, and then when I started looking it bugged me even more that I couldn’t figure out a fix for it! It’s definitely made more complicated by the fact that it’s lisp code that is shelling out and so escaping certain characters gets tricky. The single quotes especially would cause issues because when the template is echoed it’s wrapped in single quotes:

echo '{json}'

And so if it’s not escaped we end up with:

echo '{js'on}'

Which obviously breaks. I initially tried to do all the escaping in lisp so that what was sent to the command line was correct. Although I could do this for ":

(cl-ppcre:regex-replace-all "\"" text "\\\\\"" :preserve-case t))

I couldn’t figure out any magical incantation that worked on single quotes. So instead opted for a two step approach: One, change all characters to something we can easily regex for later and then, two, use sed on the command line to replace at that point.

So in lisp I had a simple function to escape single and double quotes (and replace with the text SINGLEQUOTE, etc):

; "Escape" certain characters. Will replace back on cli call
(defun escape-quotes
	(text)
	(cl-ppcre:regex-replace-all "\"" (cl-ppcre:regex-replace-all "'" text "SINGLEQUOTE" :preserve-case t) "DOUBLEQUOTE" :preserve-case t))

Since that would be safe in the echo. That function is then used in the templates:

; Just assume two template types
; Obtained from `op get template <category>`
(defun template-secure-note
	(content)
	(concatenate 'string "{\"notesPlain\":\"" (escape-quotes content) "\",\"sections\":[]}"))

And later on when it shells out it replaced using sed:

(defparameter extproc (sb-ext:run-program "sh" (list "-c" (concatenate 'string "echo '" template "' | sed 's#DOUBLEQUOTE#\\\\\"#g' | sed s#SINGLEQUOTE#\\\'#g | op encode")) :search :environment :output :stream))

Which is utterly confusing from an escaping point of view, but worked beautifully.

But then I thought on it some more because it must be possible to handle in lisp. And it turned out it was:

(defun escape-quotes
	(text)
	(cl-ppcre:regex-replace-all "\"" (cl-ppcre:regex-replace-all "'" text "\\u0027" :preserve-case t) "\\u0022" :preserve-case t))

I’d previously tried this Unicode hex approach, but had no luck because of getting the escaping wrong. I.e. I think I just used a single backslash which then got lost. This is definitely the best way to handle quotes in json.

For reference this is the sample XML I tested with:

<?xml version="1.0"?>
<PWMan_PasswordList version="3"><PwList name="Main"><PwList name="SECURE-NOTES"><PwItem><name>test 1</name><host>http://sn</host><user>me</user><passwd/><launch>Stuff '</launch></PwItem><PwItem><name>test 2</name><host>http://sn</host><user>me</user><passwd/><launch>Stuff "</launch></PwItem><PwItem><name>test 3</name><host>http://sn</host><user>me</user><passwd/><launch>Stuff ;</launch></PwItem></PwList></PwList></PWMan_PasswordList>

Moving from PWman to 1Password using Lisp

Some time ago I used to use Lastpass (because I do believe in randomly generated passwords and password managers), but ended up switching to PWman when I went all in on the command line / terminal; There was no command line tool for Lastpass - at the time, anyway. Considering how archaic PWman is it’s actually faired me really well - GPG encryption is still good, security through obscurity is actually a thing and it worked “good enough” when mobile via prompt. Fast forward to now and my life is more financially sane (I think; for now) and for the past three years I’ve been using 1Password for work and it’s really good.

So, getting fed up with iMessaging, etc the family Netflix password around and in general my family using insecure passwords (and still forgetting them; iCloud keychain sync is really good… until you forget that one) I thought it would be a good idea for us to try 1Password Families.

There are command line tools - for three(!!!) different NetBSD “ports” (platforms). There is the 1PasswordX browser plugin that works on my NetBSD desktop. And of course a great mobile app with iOS integration.

For migrating my family’s OSX Keychain / Safari passwords the MrC Converter Suite tools were great.

I had all my passwords in PWman though. I decided it would be a good excuse to re-visit Lisp and hopefully re-use some of what I’d written before. Probably I should have just used some tool to go from XML to CSV and had it imported sooner, but never mind. Once I actually found time I cracked out most of it in a day (after I’d remembered how the hell Lisp works). I’ve changed the name of existing lastpass2pwman repository and added the new script. Now available here: pwman-tools

  • I had parsing errors using xmls so used Cxml instead. The docs really suck though. Fortunately I stumbled across this which was just enough to allow me to figure out things.
  • I could not figure out decrypting the PWman file in the script, well, not in a nice way because SBCL doesn’t have a read-passwd. This thread looked promising, but I couldn’t find or figure out anything that worked from that so it’s easier to just decrypt the file up front.
  • Rather than convert PWman’s XML to CSV this does a direct conversion and import to 1Password via the op command line tool; which is pretty neat.
  • Most things map nicely to 1Password login items, but I had a SECURE-NOTES category in PWman from the time I switched from Lastpass so I decided to keep that mapping to 1Password as well, merging all the PWman fields into the notesPlain field in 1Password
  • That’s the only bit were my script is still lacking - it doesn’t properly escape quotes and apostrophes and so I had some errors as a result of trying to create secure notes; However, only on about five out of almost three hundred items so not a big deal; I may or may not fix this at some point.

None of the Lisp I’ve written is particularly good or clever. I decided to use defparameters within loops, etc as opposed to lets for (my) readability; I gather this isn’t a very lispy approach, but it works:

(dom:do-node-list (category *categories*)
	(progn
			(defparameter category-name (dom:get-attribute category "name"))
			(defparameter pwlists (dom:child-nodes category))))
[...]

Probably I’ve used defun where it should be a macro?

(defun template-secure-note
	(content)
		(concatenate 'string "{\"notesPlain\":\" content "\",\"sections\":[]}"))

Who knows? Not me. It does what I want it to do though.

To encode stuff I had to do it via sh -c:

(defparameter extproc (sb-ext:run-program "sh" (list "-c" (concatenate 'string "echo '" template "' | op encode")) :search :environment :output :stream))

as pipes (|) don’t count as arguments; neither do other commands. I.e. you can’t do: (sb-ext:run-program "echo" (list template "|" "op" "encode").... It would probably be best to base64 encode directly within Lisp, but I didn’t because I noticed subtle differences compared with op encode that I didn’t have time/willpower to investigate.

Fortunately I only created items as “Login” items or “Secure Notes” which meant I could get away with an if statement like this:

(defparameter template
	(if (string= op-category "Login")
		(template-login username passwd launch)
		(template-secure-note (concatenate 'string "host: " host "; user: " username "; password: " passwd "; launch: " launch))))

I couldn’t really figure out how to do what I wanted with case and realised I couldn’t use multiple ifs in progns:

(defparameter template
		(if (string= op-category "Login")
			(template-login username passwd launch)
			(progn
				(if (string= op-category "Secure Note")
					(template-secure-note (concatenate 'string "host: " host "; user: " username "; password: " passwd "; launch: " launch))
					(... something else...)))))

because otherwise template picks up a NIL from progn.

Overall the Lisp is actually pretty succinct, just my comments that make it look worse than it is.

When it comes to using 1Password on the command line this is enough to get started:

op list items | jq -r '.[] | [.overview.title, .uuid] | @tsv' | grep Github

To search for items matching “Github” and return UUIDs and then:

op get item [UUID] | jq -r '.details.fields[] | select(.designation=="password").value'

to get the password field for a UUID. Will probably write some little shell aliases/functions for those.

The only uncertainity I have here (asides from whether my family will actually take advantage of this; Youngest seems keen, don’t know about others) is for command line usage: The whole “Own your own data” thing… what happens if 1Password is down?


[EDIT: 2019-07-01] (yes, that’s the same date as the publish date) - Forgot to mention the op and jq stuff.

LINK: Pkgsrc WIP package for Spotifyd updated to 0.2.5

I’m not going to post every time I update this, but this time I will because:

  • This is harder than it should be to update (for me at least)
  • I’ve been on holiday this week and I’ve still struggled to find the time to do this
  • I thought I was stuck using an older rust version * (which meant patching stuff unnecessarily)
  • After updating it I immediately had to update it again when I realised I’d missed something

I.e: It’s an “achievement”.

Annoyingly it looks like 0.2.8 is out… need to do this all again.

* I installed Rust via pkgsrc/pkgin because (of course I would and…) I did not want to build from source: Building spotifyd is slow and jet-enginey enough as it is. However, I then thought I was stuck on 1.29 because the binary disappeared from pkgsrc. I only just realised today that NetBSD is a supported tier and binaries are available via rustup.

LINK: Caching authorisation token in Simplenote.vim

A recent, small, but useful fix for simplenote.vim to help alleviate this issue. I.e. this is one of those issues with Open Source software and using API access tokens. Oysttyer sees the exact same problem (I just have no time to work on that any more). There is just no way not to make the token visible which leads to abuse (fucking human beings) and being rate limited or blocked.

I honestly did not realise though that Simplenote/Simperium authorisation tokens were basically permanent so it makes sense to cache and re-use. That way if the API token runs into issues again it doesn’t really matter.

LINK: Pkgsrc WIP package for Spotifyd

I am stupid and slow. Struggled translating my previous post into a package for pkgsrc, but eventually got pretty far with it. Tobias Nygren finished it off for me and made it actually build under pkgsrc (I still find a lot of the pkgsrc makefile stuff esoteric).

Building Spotifyd on NetBSD

Irrelevant preamble: When weighing up Spotify vs Apple Music I went with Spotify owing to it being cross platform. I have a NetBSD desktop (I may have mentioned) and it meant I could use the (now defunct) Flash based web player and all was good. Then the Flash player got discontinued for one that uses widevine which doesn’t work on BSD and for awhile I was out of luck on NetBSD (I tried a few things like running the Linux client using compatibility). I wasn’t even aware of Spotify Connect clients, but stumbled across raspotify which lead me librespot which led me to spotifyd.

These are the steps I went through to build and run Spotifyd (this commit at the time of writing) on NetBSD AMD64. It’s a Spotify Connect client so it means I still need to control Spotify from another device (typically my phone), but the audio is played through my desktop… which is where my speakers and headphones are plugged in - it means I don’t have to unplug stuff and re-plug into my phone, work laptop, etc. This is 100% a “good enough for now solution” for me; I have had a quick play with the Go based microcontroller from spotcontrol and that allows a completely NetBSD only experience (although it is just an example application so doesn’t provide many features - great as a basis to build on though).

  1. Install Rust via pkgsrc/pkgin. I used version 1.29 for this
  2. Clone the spotifyd repo
  3. Try to build with cargo build --release --no-default-features --features pulseaudio_backend; Is there a way to do something like cargo fetch-dependencies and avoid having to try an initial build?
  4. You machine will act like a bit of a jet engine… and fail on these which need patching (links to the commit in my forks):
    • termios-0.2.2
    • daemonize-0.2.3
    • [get\_if\_addrs-0.5.3](https://github.com/atomicules/get_if_addrs/commit/601a0cb22e852bf9a88fb522010daf2dbee9f525)

      Fixed upstream

    • tokio-uds-0.2.2 ](https://github.com/atomicules/tokio-uds/commit/27558941953202df50d80fe124cc3275bb3b77c7)

      See EDIT at bottom

    The files are all in $HOME/.cargo/registry/src/github.com-xxxxxxxxxxxx/....

  5. Build properly (so everything links) with:

     RUSTFLAGS='-C link-args=-Wl,-rpath,/usr/lib,-rpath,/usr/pkg/lib' cargo build --release --no-default-features --features pulseaudio_backend --verbose
    

    It takes some minutes to build on my machine: There are lots of dependencies.

  6. (optional) Make sure your firewalls are ok so you can discover the client. I use npf and so ended up with something like this:

     # Simple npf.conf for Desktop
     #
     #set bpf.jit off;
    
     $ext4_if = inet4(bge0)
     $ext6_if = inet6(bge0)
    
     $services_tcp = { domain, ssh }
     $services_udp = { domain, ntp }
     $services_spotify = { 5353 }
     $services_spotify_local = { 57621, 57622 }
     $localnet4 = { 192.168.1.0/24 }
     $localnet6 = { fe80::/10 }
    
     alg "icmp"
    
     procedure "log" {
             log: npflog0
     }
    
     group "external4" on $ext4_if {
             # Stateful passing of all outgoing traffic
             pass stateful out final all
    
             # Stateful passing of inbound services
             pass stateful in final proto tcp to $ext4_if port $services_tcp
             pass stateful in final proto udp to $ext4_if port $services_udp
    
             # Spotify
             pass stateful in final proto tcp from any port $services_spotify to any port $services_spotify
             pass stateful in final proto udp from any port $services_spotify to any port $services_spotify
             pass stateful in final proto tcp from $localnet4 port $services_spotify_local to $localnet4 port $services_spotify_local
             pass stateful in final proto udp from $localnet4 port $services_spotify_local to $localnet4 port $services_spotify_local
             # Spotify uses random unprivileged port. No longer seems like can restrict source to 1900 or 5353 per ArchLinux wiki
             pass stateful in final proto tcp from $localnet4 port 1025-65535 to $localnet4 port 1025-65535
    

    Similarly for ipv6 as well (although I’ve only seen it use ipv4).

  7. (optional) Create a config. I’ve only been able to get it to recognise the config when in /etc/spotifyd.conf. I only use the config to set the name.
  8. (optional) Launch at startup. I haven’t yet created a /etc/rc.d/spotifyd, I just launch when required.

That’s it. Right now under my github I also forked [nix with a tiny change to build 0.9.0](https://github.com/atomicules/nix/commit/30ddcdc4bf79711f5ecb8ced34482ec596897140)~, but this isn't needed with the "pulse audio no default features thing"; In case you are wondering if I'd missed that above (fixed upstream and SpotifyD moved to a more recent version with that fix). Owing to the fact I just don’t have enough time for stuff like this at the time of writing I haven’t even PR’d the ones that are needed; tokio-uds is archived anyway, so not really possible to PR that, get_if_addrs needs a nightly cargo for me to test so might not PR that either and termios-rs I haven’t actually checked the OpenBSD stuff I’ve copied is applicable (it builds and seems to work).

I started with the plan of doing a pkgsrc-wip package for it, but owning to all the dependencies I’m not sure if that’s possible… maybe in a hacky way that probably makes it not worth it.

It’s super cool to have Spotify on BSD, kind of surprised it’s not more known about.

If you look at the tweaks I had to make you’ll see there are all very simple and pretty much copying whatever change someone made for OpenBSD, etc. That means it really must be trivial to get this to build on OpenBSD and probably not difficult on FreeBSD; As a rough guide looking through Rust stuff there are nearly always FreeBSD fixes, then quite often OpenBSD fixes and rarely NetBSD fixes.


[EDIT: 2019-01-17]

[EDIT: 2019-01-17]

Adding a reloadcert command to /etc/rc.d/haproxy

As follow-up to this previous post on using HAProxy and Let’s Encrypt I’d had on my todo list for awhile to perform the certificate reload a little bit better than in a /usr/local/bin/reload-cert.sh script:

#! /bin/sh
# TODO: Really, this should be a /etc/rc.d reload script
cat /usr/pkg/etc/letsencrypt/live/atomicules.co.uk/fullchain.pem /usr/pkg/etc/letsencrypt/live/atomicules.co.uk/privkey.pem > /usr/pkg/etc/haproxy.crt
export conf_file=/usr/pkg/etc/haproxy.cfg
export pid_file=/usr/pkg/etc/haproxy.pid
haproxy -f $conf_file -sf $(cat $pid_file) -p $pid_file -D

Look! It even says in it what I wanted to do. This is one of those things that was actually pretty straight-forward to do, but perceived difficultly put me off doing it until now.

It was just a matter of adding the extra commands to extra_commands:

extra_commands="configtest reload reloadcert"

(configtest was already there). And then adding a mapping for those just afterwards:

reload_cmd="haproxy_reload"
reloadcert_cmd="haproxy_reloadcert"

Then a bit further down in the file adding the commands themselves:

haproxy_reload()
{
	if [ ! -f ${conf_file} ]; then
		warn "${conf_file} does not exist."
		return 1;
	fi
	echo "Reloading config for haproxy"
	${command} -f ${conf_file} -sf $(cat $pid_file) -p ${pid_file} -D
}

haproxy_reloadcert()
{
	if [ ! -f ${conf_file} ]; then
		warn "${conf_file} does not exist."
		return 1;
	fi
	echo "Reloading cert for haproxy"
	cat /usr/pkg/etc/letsencrypt/live/atomicules.co.uk/fullchain.pem /usr/pkg/etc/letsencrypt/live/atomicules.co.uk/privkey.pem > /usr/pkg/etc/haproxy.crt
	${command} -f ${conf_file} -sf $(cat $pid_file) -p ${pid_file} -D
}

The haproxy_reloadcert should really take arguments from rc.conf rather than me hardcoding my certificate paths in, but since this is “just for me” I can get away with being lazy. It’s ultimately doing the exact same thing as the reload-cert.sh script was doing, but just in the right place now. Then the certbot --renew-hook becomes:

-renew-hook "/etc/rc.d/haproxy reloadcert"

Easy when you can be bothered.

Please, please never go away rc.d

(Finally) Using https for my Fossil repos

Since I started using HAProxy there has been nothing stopping me from using TLS for my fossil repos apart from finding the time to do it; I suppose it’s not been that long since I migrated the bulk from github, even though it has been ages since I started hosting fossil.

I just needed to update my cert to include the fossil domain, tweak my haproxy.cfg to add a new backend:

backend fossil
	mode http
	option httpchk
	# This ones gives a 501
	http-check expect status 501
	server fossil 127.0.0.1:18080 check

(I am being lazy with my http checks)

and tweak the frontend section to route to this backend:

frontend https
	bind :::443 v4v6 ssl crt /usr/pkg/etc/haproxy.crt no-sslv3
	http-request redirect prefix https://%[hdr(host),regsub(^www\.,,i)] code 301 if { hdr_beg(host) -i www }
	reqadd X-Forwarded-Proto:\ https
	acl fossil-acl hdr_beg(host) -i fossil
	use_backend fossil if fossil-acl
	default_backend bozohttpd

And lastly, coming up with a crappy rc.d file so I can start fossil as a server:

#!/bin/sh
#
# $NetBSD: fossil
#

# PROVIDE: fossil
# REQUIRES: network

$_rc_subr_loaded . /etc/rc.subr

name="fossil"
rcvar=$name
command="/usr/bin/su -m fossil -c '/usr/pkg/bin/fossil server --port 18080 --localhost --https --repolist /home/fossil/repos &'"

load_rc_config $name

run_rc_command "$1"

Previously I was using fossil in http mode via inetd.

Could do with writing that a bit better, but it does the job for now.

Oh, actually that wasn’t “lastly”. The last thing I needed to do was update all the headers of the skins for each Fossil repo to use secureurl instead of the default baseurl:

<base href="$secureurl/$current_page" />

which was a little bit tedious (like how you can have login-groups with Fossil it would be nice to have a “skin-group” to set one skin across all repos).

Fossil is super nice for self-hosted stuff and personal projects, you really should try it if you haven’t; It would also be nice to for group projects, but it’s hard to argue against the Github ecosystem.

(Finally) Switched my Desktop NetBSD machine to SSD

SSDs were the future about 10 years ago so it makes sense that only now have I switched my desktop NetBSD machine from a spinning hard drive; Just over two years ago my main machine was still a Pentium III laptop; I’m all for progression - just slowly.

Some notes on how I switched over:

  • I bought a Crucial BX500 128GB SSD for about £22. It’s not their latest model, but it’s £22! The 160GB spinning (and failing) hard drive this replaces was £12 for a refurb; I probably only need about 20GB.
  • I bought a Startech SATA power splitter for about £3. I had no real intention of copying data across and was just going to install from scratch, but then realised I’d lost my NetBSD install DVD so needed an alternative plan.
  • Plugged in the SSD to the optical drive connector. The drive is that small it fits between the top of the optical drive and the casing (Optiplex 745 SFF).
  • Booted up my existing NetBSD install.
  • sudo sysinst.
  • Did a fresh install to the wd1 device.
  • Shutdown (should have edited fstab first, but did it messily on the next boot).
  • Switched cables around so the ssd became wd0 and the spinning hard drive became wd1.
  • It failed to boot because I’d forgotten about fstab, but managed to mount manually and fix fstab during boot.
  • Mounted the old drives as: mount /dev/wd1a /oldroot and mount /dev/wd1e /oldhome, etc.
  • Copied my home directory across, fixed permissions, etc.
  • Pretty much it. Of course there are the things I always forget about:
  • And this time round I decided to enable xdm as I’ve gotten bored of typing startx on my Desktop which meant a brief confusing moment as it dropped me into twm and not dwm, until I’d re-installed dwm and symlinked .xsession to my existing .xinitrc.
  • I haven’t yet unplugged and removed the spinning hard drive. Will do soon (just remembered I need to copy across my npf settings first). Don’t ever bother with a case/adaptor, etc for the ssd. Just tuck it in somewhere.

With the £60 21.5” monitor I got last month I now have a pretty awesome desktop NetBSD machine: 5GB RAM; 2x 3.4GHz Pentium D cpus. Nice.

Faster List Index for Simplenote.vim

One of those things I should have done years ago, but never had the need to because date limiting (:SimplenoteList 2018-11-01 was good enough for me). With the switch to Simperium the since parameter became a cursor instead of a date which meant I had to list all 1000+ of my notes. That got frustrating fast (or slow… depending on how you want to word that).

So first of all I added a in memory cache so that although the first call would be slow, subsequent updates of the list index would be quicker.

All this did, really, was change the NoteFetcher from storing things in an array:

    def run(self):
        key = self.queue.get()
        note, status = self.simplenote.get_note(key)
        if status != -1:
          self.note_list.append(note)

self.queue.task_done()

to storing a dict/hash with a lightweight version of the note (i.e. includes the “title” and everything we want to sort the list index on):

    def run(self):
        key = self.queue.get()
        note, status = self.simplenote.get_note(key)
        # Strip down and store a lightweight version
        # Storing key "twice" as makes easier to convert to list later
        note_lines = note["content"].split("\n")
        note_title = note_lines[0] if len(note_lines) > 0 else note["key"]
        notelight = {
            "key": note["key"],
            "modifydate": note["modifydate"],
            "createdate": note["createdate"],
            "tags": note["tags"],
            "systemtags": note["systemtags"],
            "deleted": note["deleted"],
            "title": note_title
        }
        if status != -1:
            self.note_list[note["key"]] = notelight

self.queue.task_done()

Nothing fancy and there are probably neater ways of creating the lightweight object than that; Such as copying note and deleting what we don’t need; But it works.

Since we still need it as a list at certain points we can just:

note_list = list(self.note_cache.values())

When needed. But having it stored as a dict means that when it comes to updating the index we can do:

if self.simplenote.current:
    note_keys, status = self.simplenote.get_note_list(data=False, since=self.simplenote.current)
    note_cache = self.get_notes_from_keys([n['key'] for n in note_keys])
    # Merge with existing
    self.note_cache.update(note_cache)

Which is the bit that makes subsequent updates fast as we only have to pull changes and update the changed objects.

Once that was in place it was just a matter of storing a known state of the index dict on disk so that start-up from scratch could also be fast. For this we are just writing to json:

def write_index_cache(self):
    try:
        with open(INDEX_CACHE_FILE, 'w') as f:
            json.dump({ "current": self.simplenote.current, "cache": self.note_cache}, f, indent=2)
    except IOError as e:
        print("Error: Unable to write index cache to file - %s" % e)

And then we can read that at start-up:

if os.path.isfile(INDEX_CACHE_FILE):
    try:
        with open(INDEX_CACHE_FILE, 'r') as f:
            cache_file = json.load(f)
            self.note_cache = cache_file["cache"]
            self.simplenote.current = cache_file["current"]

Again, there might be more efficient ways of doing this - such as using individual files instead of one large file, but seeing as how this is a lightweight cache (only has note titles, not note content) the one “big” file approach is probably ok for now.

It’s so much better now. I really should have done this ages ago.

These are the ten most recent posts, for older posts see the Archive.