As a separate post to the initial one because it’s interesting.

Despite this pretty much being a single-use bit of code that has already done it’s job it was bugging me that it didn’t handle certain things, namely quotation marks, and then when I started looking it bugged me even more that I couldn’t figure out a fix for it! It’s definitely made more complicated by the fact that it’s lisp code that is shelling out and so escaping certain characters gets tricky. The single quotes especially would cause issues because when the template is echoed it’s wrapped in single quotes:

echo '{json}'

And so if it’s not escaped we end up with:

echo '{js'on}'

Which obviously breaks. I initially tried to do all the escaping in lisp so that what was sent to the command line was correct. Although I could do this for ":

(cl-ppcre:regex-replace-all "\"" text "\\\\\"" :preserve-case t))

I couldn’t figure out any magical incantation that worked on single quotes. So instead opted for a two step approach: One, change all characters to something we can easily regex for later and then, two, use sed on the command line to replace at that point.

So in lisp I had a simple function to escape single and double quotes (and replace with the text SINGLEQUOTE, etc):

; "Escape" certain characters. Will replace back on cli call
(defun escape-quotes
	(text)
	(cl-ppcre:regex-replace-all "\"" (cl-ppcre:regex-replace-all "'" text "SINGLEQUOTE" :preserve-case t) "DOUBLEQUOTE" :preserve-case t))

Since that would be safe in the echo. That function is then used in the templates:

; Just assume two template types
; Obtained from `op get template <category>`
(defun template-secure-note
	(content)
	(concatenate 'string "{\"notesPlain\":\"" (escape-quotes content) "\",\"sections\":[]}"))

And later on when it shells out it replaced using sed:

(defparameter extproc (sb-ext:run-program "sh" (list "-c" (concatenate 'string "echo '" template "' | sed 's#DOUBLEQUOTE#\\\\\"#g' | sed s#SINGLEQUOTE#\\\'#g | op encode")) :search :environment :output :stream))

Which is utterly confusing from an escaping point of view, but worked beautifully.

But then I thought on it some more because it must be possible to handle in lisp. And it turned out it was:

(defun escape-quotes
	(text)
	(cl-ppcre:regex-replace-all "\"" (cl-ppcre:regex-replace-all "'" text "\\u0027" :preserve-case t) "\\u0022" :preserve-case t))

I’d previously tried this Unicode hex approach, but had no luck because of getting the escaping wrong. I.e. I think I just used a single backslash which then got lost. This is definitely the best way to handle quotes in json.

For reference this is the sample XML I tested with:

<?xml version="1.0"?>
<PWMan_PasswordList version="3"><PwList name="Main"><PwList name="SECURE-NOTES"><PwItem><name>test 1</name><host>http://sn</host><user>me</user><passwd/><launch>Stuff '</launch></PwItem><PwItem><name>test 2</name><host>http://sn</host><user>me</user><passwd/><launch>Stuff "</launch></PwItem><PwItem><name>test 3</name><host>http://sn</host><user>me</user><passwd/><launch>Stuff ;</launch></PwItem></PwList></PwList></PWMan_PasswordList>