atomicules

Push propelled program tinkerer and picture maker.

(Finally) Using https for my Fossil repos

Since I started using HAProxy there has been nothing stopping me from using TLS for my fossil repos apart from finding the time to do it; I suppose it’s not been that long since I migrated the bulk from github, even though it has been ages since I started hosting fossil.

I just needed to update my cert to include the fossil domain, tweak my haproxy.cfg to add a new backend:

backend fossil
	mode http
	option httpchk
	# This ones gives a 501
	http-check expect status 501
	server fossil 127.0.0.1:18080 check

(I am being lazy with my http checks)

and tweak the frontend section to route to this backend:

frontend https
	bind :::443 v4v6 ssl crt /usr/pkg/etc/haproxy.crt no-sslv3
	http-request redirect prefix https://%[hdr(host),regsub(^www\.,,i)] code 301 if { hdr_beg(host) -i www }
	reqadd X-Forwarded-Proto:\ https
	acl fossil-acl hdr_beg(host) -i fossil
	use_backend fossil if fossil-acl
	default_backend bozohttpd

And lastly, coming up with a crappy rc.d file so I can start fossil as a server:

#!/bin/sh
#
# $NetBSD: fossil
#

# PROVIDE: fossil
# REQUIRES: network

$_rc_subr_loaded . /etc/rc.subr

name="fossil"
rcvar=$name
command="/usr/bin/su -m fossil -c '/usr/pkg/bin/fossil server --port 18080 --localhost --https --repolist /home/fossil/repos &'"

load_rc_config $name

run_rc_command "$1"

Previously I was using fossil in http mode via inetd.

Could do with writing that a bit better, but it does the job for now.

Oh, actually that wasn’t “lastly”. The last thing I needed to do was update all the headers of the skins for each Fossil repo to use secureurl instead of the default baseurl:

<base href="$secureurl/$current_page" />

which was a little bit tedious (like how you can have login-groups with Fossil it would be nice to have a “skin-group” to set one skin across all repos).

Fossil is super nice for self-hosted stuff and personal projects, you really should try it if you haven’t; It would also be nice to for group projects, but it’s hard to argue against the Github ecosystem.

Yet Another HAProxy and Let's Encrypt post

It’s what the world needs.

I caved in and decided to use HAProxy in front of Bozohttpd so I could:

  1. Redirect http to https
  2. Redirect www to just the domain (only just fully finished this bit)

The fiddly bit with Let’s Encrypt and HAProxy is handling the renewal of the cert. All the posts I’ve found either do the simple, but reliable, approach of stopping a web-server, running a renewal using --standalone and then re-starting a web-server, or the slightly more advanced approach of using --standalone on a non-standard port with a HAProxy rule that passes through to it as needed. But why not use --webroot instead?

Since I used --webroot originally I just have a couple of cron entries that run:

/usr/pkg/bin/certbot renew --renew-hook /usr/local/bin/reload-cert.sh

The --renew-hook only gets called if the certificate is actually renewed. Where that script is:

#! /bin/sh
# TODO: Really, this should be a /etc/rc.d reload script
cat /usr/pkg/etc/letsencrypt/live/atomicules.co.uk/fullchain.pem /usr/pkg/etc/letsencrypt/live/atomicules.co.uk/privkey.pem > /usr/pkg/etc/haproxy.crt
export conf_file=/usr/pkg/etc/haproxy.cfg
export pid_file=/usr/pkg/etc/haproxy.pid
haproxy -f $conf_file -sf $(cat $pid_file) -p $pid_file -D

Which does the necessary bits of combining the two parts of the cert (I do like that bozohttpd doesn’t require this) and then doing a hot reload of the HAProxy configuration so that the HAProxy is serving the new cert.

Then in my haproxy.cfg I have a http frontend with these rules:

frontend http
	bind :::80 v4v6
	acl letsencrypt path_beg /.well-known/acme-challenge/
	acl http      ssl_fc,not
	http-request redirect scheme https if http !letsencrypt
	reqadd X-Forwarded-Proto:\ http
	use_backend bozohttpd if letsencrypt

Which re-directs all http requests to a https frontend unless they match the Let’s Encrypt path, in that case they pass through to the backend as http - this is important as the webroot plugin can’t work over https (which seems a bit counter-intuitive for Let’s Encrypt).

Then I have a frontend for the https stuff as follows:

frontend https
	bind :::443 v4v6 ssl crt /usr/pkg/etc/haproxy.crt no-sslv3
	http-request redirect prefix https://%[hdr(host),regsub(^www\.,,i)] code 301 if { hdr_beg(host) -i www }
	reqadd X-Forwarded-Proto:\ https
	default_backend bozohttpd

Which serves the actual Let’s Encrypt cert and also redirects the www prefix to the main domain. The backend is nothing fancy at all:

backend bozohttpd
	mode http
	# Since the check doesn't pass a domain it will 404
	option httpchk
	http-check expect status 404
	server bozo 127.0.0.1:10080 check

And this works lovely.

However, one VERY IMPORTANT thing to be aware of if you are using IPv6 and you starting seeing timeouts reported during renewals or dry-runs of renewals it is GUARANTEED to be due to your site not resolving over IPv6. There are many posts about this on the Let’s Encrypt forums and all of them start with that “definitely not being the problem” and end with “Oh, actually it was”. I too ran into this issue, but hadn’t realised as my IPv6 access had broken at home (and I hadn’t realised) and had also been broken on my server for months (and I hadn’t realised) even though I appeared to have an IPv6 address.


I’d originally only generated a certificate for atomicules.co.uk, but of course if I want to redirect www to my plain domain I also actually need the certificate to be valid for www as well. It took me a little bit to figure out how to do this. I basically relied on Bozohttpd’s virtual host support and created a /var/www/vroot/www.atomicules.co.uk directory (rather than to try to do further clever redirection in the HAProxy) for the sole purpose of serving up the acme-challenge stuff. Then with the above HAProxy setup this worked:

sudo certbot certonly --webroot -w /var/www/vroot/atomicules.co.uk/ -d atomicules.co.uk -w /var/www/vroot/www.atomicules.co.uk -d www.atomicules.co.uk --cert-name atomicules.co.uk

[EDIT: 2018-11-23] See: Adding a reloadcert command to /etc/rc.d/haproxy.

Redirecting From Http To Https With Bozohttpd

Following on from the previous post: I’m so slow/dumb sometimes. Of course it’s possible to redirect http requests to https with Bozohttpd, the very fact I’m running two instances of this makes this possible.

For the httpd (non-https) instance configure it with a different virtual root directory, e.g:

httpd_flags="-S bozohttpd -v /var/www/vredirectroot -M .html 'text/html; charset=utf-8' '' '' -M .xml 'text/xml; charset=utf-8' '' ''"

Then within that directory create the virtual host directory so you have a path like so:

/var/www/vredirectroot/atomicules.co.uk

And then within that directory just place a .bzabsredirect file:

sudo ln -s https://atomicules.co.uk .bzabsredirect

Restart that bozohttpd instance:

sudo /etc/rc.d/httpd restart

And “hey presto!” it works.


[EDIT: 2017-07-31] Spoke too soon. It’s too simplistic. It redirects just the path with the .bzabsredirect file is. So although http://atomicules.co.uk/ merrily redirected to https://atomicules.co.uk an existing blog post like http://atomicules.co.uk/2017/13/32/somepost.html just 404s. Poop. Ok, I think I’ll have to go back to duplicating http and https again for the time being otherwise I’ll break a load of links - well a handful. One thing .bzabsredirect will work for is redirecting www.atomicules.co.uk on it’s own, I’d just left that broken for now. I might take a look at HAProxy as I’m not moving off Bozohttpd.

Now Serving Https As Well

Since it’s 2017 and that; Didn’t want to rush into this. Thought I should finally enable TLS/SSL since it’s free. I’m not sure I entirely agree with the arguments for a site like mine (wouldn’t metadata be the biggest problem?), but it’s pointless trying to argue against the tide; One thing though: Zscaler, anyone who has had to browse through that realises that TLS/SSL isn’t bulletproof. I understand why that exists as a product, but, gah, as an end user it’s just horrible.

The EFF site will guide you down the certbot-auto route for NetBSD, which is silly as there is a py27-certbot package - just use that.

Bozohttpd works fine with Let’s Encrypt, the only issue is that it either serves https OR http, unfortunately not both at the same time. I haven’t yet figured out a way to redirect traffic between ports so that’s meant I’m effectively running two webservers at the moment as per this rc.conf approach. I.e:

  1. Duplicate /etc/rc.d/httpd to /etc/rc.d/httpsd.
  2. Edit and make sure to change name to httpsd and command so command is explicitly calling /usr/libexec/httpd
  3. Add a $procname=$name line (otherwise it’ll get confused between httpd and httpsd and think they are the same).
  4. Change required_dirs to $httpsd_wwwdir
  5. In rc.conf have both a httpsd=YES and a httpd=YES

Then I have the following entries in rc.conf for http:

httpd_flags="-S bozohttpd -v /var/www/vroot -M .html 'text/html; charset=utf-8' '' '' -M .xml 'text/xml; charset=utf-8' '' ''"

Whilst httpsd has these extras:

-Z /usr/pkg/etc/letsencrypt/live/atomicules.co.uk/fullchain.pem /usr/pkg/etc/letsencrypt/live/atomicules.co.uk/privkey.pem -z 'EECDH+CHACHA20:EECDH+AES128:RSA+AES128:EECDH+AES256:RSA+AES256:EECDH+3DES:RSA+3DES:!MD5;'

The ciphers as advised here.

Since running two webservers isn’t ideal I think I’ll ultimately have to redirect all traffic with the firewall (or run a proxy I suppose?), but that is going to have to wait until I perform some server maintenance and finally switch from IPFilter to NPF (which I should be able to do now I’m running on KVM).


[EDIT: 2017-07-30] Note: It’s advisable to set procname in both /etc/rc.d files. I think otherwise the start up order matters (god knows how it gets confused, but I found setting procname in both worked).

Site Performance Improvements

I just couldn’t resist the temptation of trying to get my Speed Index below the magic 1000 number*.

The first efforts I made were eliminating any external requests where possible; in fact, I’d already started making a few tweaks as a result of Jacques Mattheij’s Fastest Blog post. The rest were made after testing on WebPageTest.

Changes made:

  • Hosted the Creative Commons image, used in the footer, locally.
  • Used my own search form, still passing off to DuckDuckGo, as opposed to using their iframe.
  • Provide gzipped versions of all files; And that is as far as I can go with my webserver, Keep-alive and Caching are not an option.
  • Moved styles inline; much to my chagrin it does improve things. Fortunately it is still just as easy to manage from my point of view as I’ve just moved the file from /styles to /_includes.
  • Moved the Google Web Fonts stylesheet to a <link> instead of using an @import; I’m not getting rid of Web Fonts as I don’t want to be ugly, plus it’d be a crime not to use Vernon Adams’ fonts.

Of course, I’m not really sure there is a lot of point in making a site that no one reads really fast to load, but nevermind.

* - This is with the default US server test though, so I imagine faster still from the UK, but I’m still not very mobile friendly; I care far more about Elinks than mobile browsers.

Indieweb - Automatically sending webmentions

In crowbar-ing in webmentions I realised that the syndication code I’d written need overhauling if I was ever going to support more than u-in-reply-to. In fact, so much so that the code in my note syndication post is now obsolete; rather than looping through the posts in the syndication method of each syndication class instance I’m doing the looping just once in the Rakefile and calling the correct syndication class as required.

That overhaul was time consuming and explains the lack of posts here this month - I’ve spent most of my time doing behind-the-scenes work - but things are much cleaner now; although as ever, there is always room for improvement: I am still only webmentioning u-in-reply-to, but it will be easier to add it in for other uses of webmention.

In the Rakefile, when looping through posts, I do this when I come across a new link post since last deploy date:

case yaml["type"]
	when "link"
		post_data["link"] = yaml["link"]
		@posse_pinboard.syndicate(post_data)
		#For the time being webmentions are only sent for link type posts that have this key
		if yaml.has_key?("u-in-reply-to")
			@webmention.send(post_data)
		end
	when "photo"
		#Different things
	#And so on
end

Simple. The actual sending of the webmention is just as easy. I’m using Nokogiri to find the webmention endpoint (As far as I know it will only ever be on link element):

def find_webmention_endpoint(post_url)
	#Should cache these? Probably not worth it.
	page = Nokogiri::HTML(open(post_url))
	webmention_link = page.xpath("//link[@rel='webmention']")
	if webmention_link.empty?
		fail NoWebmentionEndpoint
	else 
		begin
			#Should be only one. I guess it's always on a link
			webmention_endpoint = webmention_link[0][:href]
		rescue
			#Something bizarre going on
			fail NoWebmentionEndpoint
		end
	end
end

And lastly, sending the webmention is as easy as:

res = Net::HTTP.post_form(webmention_endpoint, 'source' => post_data["my_post_url"], 'target' => post_data["link"])

Just in case you’ve forgotten: my Jekyll Indieweb repository

RE: Testing Receiving Webmentions

Testing my cobbled together code for automatically sending webmentions. For the time being it is only for things I’ve marked as being u-in-reply-to. I need to overhaul my code to cover other uses for webmentions.

Testing Receiving Webmentions

This is pretty much a test post so I have something to target as I work on implementing support for automatically sending webmentions on deploy and (obviously) receiving webmentions via webmention.herokuapp.com; I’m far more bothered about being able to automate sending webmentions than I am about receiving them, but from a practical standpoint I can’t do one without the other.

I’ve decided to go with Voxpelli’s service in the first instance to make my life easier. I may ultimately write my own and properly host my own webmentions receiver; I have horribly tempting thoughts to see what I can achieve with Lua and Bozohttpd.

##Some very brief thoughts on webmention.herokuapp.com

to try to flesh out this post a bit

  • It is graciously provided for free and takes all of a second to implement so I have no rights to complain at all, but…
  • It requires Javascript to display mentions. I do use Javascript on my Archive page, but in general I’m trying to be Javascript free (purely because I’m the biggest user of my website and I use Elinks a lot).
  • There’s no administration functionality, such as being able to review webmentions before allowing them to be posted or deleting spam webmentions.

Yet more Indieweb syndication - Notes

Since adding syndication of link posts to Pinboard I’ve added syndication of photos to Flickr (in theory, since I’m still waiting to get last years’ films developed I can’t use it yet) and notes to Twitter. I wasn’t sure about doing notes to Twitter as I don’t really want the clutter on my site, but I’ve decided to take the same approach as I have with Pinboard: I can tweet via my website, but I don’t have to.

I’ll not talk about the Photo syndication as it’s not so interesting; it was all made pretty easy with the Flickraw gem. The Twitter stuff is a bit more interesting because I wanted to make the experience as frictionless as possible; still pretty easy thanks to the twitter gem.

In the Pinboard syndication post I cheated and just linked to the repository, but I’ll try and talk through this one a bit more. So not in the order that I wrote the code, but in an order that kind of makes sense:

I have a rake task that handles POSSEing to Twitter.

task :posse_twitter do
	date = File.open("_deploy_date", &:readline) 
	netrc = Netrc.read
	token, secret = netrc["twitter.com"]
	puts "--> Looking for note posts to syndicate to Twitter"
	posse_twitter = Jekyll_syndicate_twitter.new(token, secret, date)
	posse_twitter.syndicate()
	#Above should fail if no posts found, hence re-build below will only occur if changes made
	Rake::Task["build"].invoke
	Rake::Task["deploy"].invoke
end

This reads a file called _date_deploy from the top of my Jekyll site directory. It’s a text file that contains one line only: the date and time that I last deployed my site. I store my user/access token and secret in a .netrc file (the actual application/consumer token and secret in a _flickr_app file). This rake task then creates a new instance of a Jekyll_syndicate_twitter class and calls the syndicate method on it:

class Jekyll_syndicate_twitter

	def initialize(token, secret, date)
		@date = date
		@twitter = Twitter::REST::Client.new
		#The actual key and secret are stored with the app in a file called...
		twitter_app = YAML.load_file("_twitter_app")
		@twitter.consumer_key = twitter_app["consumer_key"]
		@twitter.consumer_secret = twitter_app["consumer_secret"]
		#Assumes authentication has already taken place
		@twitter.access_token = token 
		@twitter.access_token_secret = secret
	end
	

	def syndicate
		note_posts = find_posts_since("note", @date)
		note_posts.each do |filename, title|
			#Does stuff, we'll get to this later
		end
	end
end

This is fairly simple because the gem does all the hard work. The initialize method just handles creating a new Twitter client given all the correct tokens and secrets. The syndicate method finds all new note type posts since the last deploy date using find_posts_since:

(I’ll cut some of this out and just discuss what changed since the Pinboard syndication).

def find_posts_since(post_type, last_deploy_date)
	#This is looking in _posts not _site
	posts_to_syndicate = []
	#Need to do this differently for note posts and normal posts
	post_directory = "_posts/"
	if post_type == "note"
		post_directory += "notes/"
	end
	#Skip sub-directories
	all_posts = Dir[post_directory+'/*'].reject do |post|
		File.directory?(post)
	end
	all_posts.each do |post|
		filename = File.basename(post)
		#Need to check date in file if available (for notes) if not use filename
		yaml = YAML.load_file(post)
		if yaml.has_key?("date")
			#Seems to come as Time
			publish_date = yaml["date"].to_datetime
		else
			publish_date = DateTime.parse(filename[0..9]) 
		end
		if publish_date > DateTime.parse(last_deploy_date)

			#...Gets the actual bits and bobs required from the files

		end
	end
	if posts_to_syndicate.empty?
		fail NoPostsFoundError, "No #{post_type} posts found"
	end
	posts_to_syndicate
end

I’m storing my notes in a sub-directory of _posts just to keep everything a bit cleaner so the method needs to look in the right place. Since notes have the potential to be posted more than once a day (I haven’t had that problem yet though) I can’t rely on the file name when comparing the deploy date, I need to check the actual date in the file if it is there. If no new posts are found then I raise a custom error so I can capture this in my Rakefile and still allow other things to carry on since it is an ok error.

Now I’ve got a list of note posts I want to syndicate and relevant data from those posts (which for notes is just the text of the note), going back to the syndicate class…

class Jekyll_syndicate_twitter

	def initialize(token, secret, date)
		#The stuff that was up above
	end
	

	def syndicate
		note_posts = find_posts_since("note", @date)
		note_posts.each do |filename, title|
			#For twitter posts can't add any kind of POSSE backlink
			#But should be possible to get the link of the twitter post
			tweet = @twitter.update(title)
			#Can build link from tweet.id, weird that doesn't seem to be a way to get full url
			link_to_tweet = "https://twitter.com/#{tweet.user.name}/statuses/#{tweet.id}"
			#Need to then update the post, as per flickr
			add_link_to_blog_post("_posts/notes/"+filename, link_to_tweet)
		end
	end
end

…it then tweets the relevant bit (I’m using the title field of the yaml front matter to store the tweet/note text) and then edits the note post to add the link to the tweet in:

def add_link_to_blog_post(post, link)
	#Manually do or use some aspect of to_yaml. I think safer to do manually can then quote the url
	lines = File.readlines(post)
	#insert before second ---
	yaml_start_idx = lines.index{ |line| line.strip == "---" }
	yaml_end_idx = lines[yaml_start_idx+1..-1].index{ |line| line.strip == "---" } + yaml_start_idx+1
	lines.insert(yaml_end_idx, "u-syndication: \"#{link}\"\n")
	#Write back out
	f = File.open(post, "w")
	lines.each do |line|
		f << line
	end
	f.close
end

This simply opens the relevant note file for editing and adds a u-syndication property to the yaml front matter which I can then make use of in my Jekyll templates using some Liquid magic to add a syndication link to the note. I’ve also included webactions, but only on the page permalink, not everywhere the note is displayed. I don’t have my Jekyll templates in a public repository anymore so I might go over how I’ve set this up in another post. Might.

Last of all then, and getting to the main point of this post, so I can make tweeting from the command line usable, I have a rake task set up for it:

task :note, [:text] do |t, args|
	#Get argument from command line
	#Can't use commas: http://stackoverflow.com/questions/7258148/how-can-i-use-a-comma-in-a-string-argument-to-a-rake-task
	#Fix idea from: http://blog.stevenocchipinti.com/2013/10/18/rake-task-with-an-arbitrary-number-of-arguments/
	text = args[:text]
	if args.extras.length > 0
		#rake strips white space as well. Will just have to assume I have used spaces after commas
		text += ", "+args.extras.join(", ")
	end
	#Check length
	if text.length > 140
		fail NoteTooLongError, "Note is too long by #{text.length-140}"
	end
	#replace anything that's not a letter or number with a hyphen to make it pretty ish
	filename = text.gsub(/[^0-9a-zA-Z]/, "-")
	#Just to be on safe side in case I've missed something
	filename = CGI.escape(filename)
	#get date
	date = Date.today.iso8601
	datetime = DateTime.now.iso8601
	#write out to file
	File.open("_posts/notes/#{date}-#{filename}.markdown", "w") do |file|
		file << "---\n"
		file << "layout: page\n"
		file << "type: note\n"
		file << "title: \"#{text}\"\n"
		file << "categories:\n"
		file << "- note\n"
		file << "date: #{datetime}\n"
		file << "---\n"
	end
	Rake::Task["build"].invoke
	#deploy, which will call posse_twitter
	Rake::Task["deploy"].invoke
end

With this I can call rake note["Here's a tweet"] on the command line and it’ll build my note post, build my site, syndicate the tweet out and deploy my site. Ok, there is a bit of a delay waiting for the site to build, but actually not too long.

The task looks complex because Rake has an issue with commas in arguments for tasks and will split any string on commas and consider these as separate arguments. So if I tried to tweet something like “All things being well, I’m now syndicating notes” all that I’d end up with would be “All things being well”. I’ve added a work-around to recombine any split string and add the commas back in. I might have to rethink this if I ever want to extend the work I’ve done and syndicate out twitter replies as I’ll need to include an intentional additional argument of the id of the tweet being replied to. But for the time being it’s unlikely I’m going to want to do this as I’ll just jump to TTYtter. I’m also checking that my note isn’t longer than 140 characters because I have no interest in syndicating out truncated notes.

For the most recent code refer, once again, to the repository

Indieweb: Now Syndicating Link posts to Pinboard

Well, I said by the end of the year, that was quicker than expected! As of this post I’ve started syndicating out link posts to Pinboard with this result.

In further Indieweb efforts the code for this is available via Fossil on this very site. Not on Github! I couldn’t get Fossil CGI hosting to work with the web server I use so it’s slightly annoyingly on a different port. Only slightly annoyingly, because I really admire just how easy it is to get Fossil working as a server. For pet personal projects I will continue to use it over Git.

My old deploy script, since my personal computer is also my webserver, used to be a very simple sudo cp .... It’s a bit more complex now. The basic principles are:

  1. Use a Rakefile for deploying.
  2. When deploying (Still basically a sudo cp) write date of deploy to a file.
  3. Find posts in _posts that are link type posts and newer than the last deploy date in file.
  4. Syndicate each of those out to Pinboard via the API. If the link already exists on Pinboard, but doesn’t have a tag representing it’s been syndicated out, the extended description is replaced with content of post and a syndication tag added. Otherwise a new bookmark is added.

This approach suits me fine. Any link post here will appear on Pinboard, but I do not care for having every bookmark I make on Pinboard show up here. This also nets me another 0.1 pt on the Indiemark score.

The Pinboard code I adapted from my Stackexchange favs to Pinboard code. The whole code is very rough and ready at the moment and needs some polishing, but works well (for me). The main limitations at the moment are:

  • Assumes my permalink style.
  • The date of deploy granularity is only to the nearest day.
  • It is seemingly not possible to get the bookmark ID from the Pinboard API. So I can’t update the blog post with a link to where it is syndicated.

Next up will either be photo posts (although that is a continuing story of heartache; still to send off the only four roll of films I got to shoot last year) or more likely figuring out a way to do notes/tweets in a none annoying (to me) way.

See the Fossil repository here.

These are the ten most recent posts, for older posts see the Archive.