One of those things I should have done years ago, but never had the need to because date limiting (:SimplenoteList 2018-11-01 was good enough for me). With the switch to Simperium the since parameter became a cursor instead of a date which meant I had to list all 1000+ of my notes. That got frustrating fast (or slow… depending on how you want to word that).

So first of all I added a in memory cache so that although the first call would be slow, subsequent updates of the list index would be quicker.

All this did, really, was change the NoteFetcher from storing things in an array:

    def run(self):
        key = self.queue.get()
        note, status = self.simplenote.get_note(key)
        if status != -1:
          self.note_list.append(note)

self.queue.task_done()

to storing a dict/hash with a lightweight version of the note (i.e. includes the “title” and everything we want to sort the list index on):

    def run(self):
        key = self.queue.get()
        note, status = self.simplenote.get_note(key)
        # Strip down and store a lightweight version
        # Storing key "twice" as makes easier to convert to list later
        note_lines = note["content"].split("\n")
        note_title = note_lines[0] if len(note_lines) > 0 else note["key"]
        notelight = {
            "key": note["key"],
            "modifydate": note["modifydate"],
            "createdate": note["createdate"],
            "tags": note["tags"],
            "systemtags": note["systemtags"],
            "deleted": note["deleted"],
            "title": note_title
        }
        if status != -1:
            self.note_list[note["key"]] = notelight

self.queue.task_done()

Nothing fancy and there are probably neater ways of creating the lightweight object than that; Such as copying note and deleting what we don’t need; But it works.

Since we still need it as a list at certain points we can just:

note_list = list(self.note_cache.values())

When needed. But having it stored as a dict means that when it comes to updating the index we can do:

if self.simplenote.current:
    note_keys, status = self.simplenote.get_note_list(data=False, since=self.simplenote.current)
    note_cache = self.get_notes_from_keys([n['key'] for n in note_keys])
    # Merge with existing
    self.note_cache.update(note_cache)

Which is the bit that makes subsequent updates fast as we only have to pull changes and update the changed objects.

Once that was in place it was just a matter of storing a known state of the index dict on disk so that start-up from scratch could also be fast. For this we are just writing to json:

def write_index_cache(self):
    try:
        with open(INDEX_CACHE_FILE, 'w') as f:
            json.dump({ "current": self.simplenote.current, "cache": self.note_cache}, f, indent=2)
    except IOError as e:
        print("Error: Unable to write index cache to file - %s" % e)

And then we can read that at start-up:

if os.path.isfile(INDEX_CACHE_FILE):
    try:
        with open(INDEX_CACHE_FILE, 'r') as f:
            cache_file = json.load(f)
            self.note_cache = cache_file["cache"]
            self.simplenote.current = cache_file["current"]

Again, there might be more efficient ways of doing this - such as using individual files instead of one large file, but seeing as how this is a lightweight cache (only has note titles, not note content) the one “big” file approach is probably ok for now.

It’s so much better now. I really should have done this ages ago.