Heya party people! This is a tricky bit of news to share, but it needs to be done.

We had some down-time

If you're not in the UK you probably didn't know but, two weeks ago, we had some down-time. 😬 This was our first significant period own down-time in the history of Finger-Ink. The entire platform was unusable for a number of hours.

Even worse, a couple of days later there were further issues for the same group of customers. Web forms were still available, but a large part of Finger-Ink's functionality was offline for a number of hours.

I've heard from many of you as to how frustrating this was. To everyone who experienced this outage — I am truly sorry.

It's been sorted

As of writing this, all major issues having a hand in this have been addressed, and Finger-Ink is in the best shape since our introduction of the portal.

If you're interested in understanding what went wrong, and what's been done to address this, please read on.

The first outage

Occasionally, the servers that run Finger-Ink need patching. Security and stability updates are released, and these need to be applied in order to ensure our servers are not vulnerable.

Our hosting provider ensures that our servers are up-to-date. They do this by applying these patches for us, as required. At the time, however, it also meant that our servers would be restarted automatically after these patches were applied.

Prior to this event, all patching had been done during Finger-Ink's office hours (or close enough to them). This was important because, when our application came online, there were some manual steps that needed to be performed in order to make sure that our encryption module comes online. This encryption module is used throughout Finger-Ink — nothing can operate without it.

Unfortunately, on this day, the patching and restarting was done well outside of our regular hours. When the application came back online, and sent a notification that the manual steps were required, no one was around to receive it. This meant the encryption module remained offline, until I checked in on things at 5:30am the following morning, and brought the encryption module online.

The second outage

Following the first outage, on that same day, we deployed a change to ensure that the encryption module could come online all by itself — without requiring any manual steps.

This ran well until another patch was applied a day or two later. The server was restarted again, after hours. The encryption module came back online, as intended, but the key server (our new fancy tech that ensures API keys aren't removed immediately after being told they're invalid by the Cliniko API) did not, due to it running into database connection limits.

While the portal remained usable, web forms couldn't match to patients, and workflow couldn't run.

Getting it sorted

Since then, our top priority has been making sure that this does not happen again. In addition to ensuring that the encryption module always comes back online, we've also taken further steps to improve reliability.

Regarding unplanned patches & server restarts:

1. Patches are now only applied once per week, rather than daily.
2. Server restarts after patching need to be initiated by Finger-Ink, they will no longer happen automatically. This ensures we will be there to make sure everything comes back online, just in case.

Regarding database connectivity & capacity:

• Our database connection limits have been increased 3-fold.
• We've optimised our database connection configuration to ensure that waiting for the database a little longer than usual doesn't cause errors.

Other things we've done:

• Optimised our start-up procedures within the app to reduce initial load on the server during boot.
• Reduced unnecessary load on the server
• Fixed an issue which, due to server load, could have caused appointment syncing (from the iPad) to pause until the next launch of the app.
• Implemented a more thorough monitoring system for our platform.

Going forward

All is going well thus far. The new connection limits have increased the overall snappiness of the portal interface, and we haven't had any further issues.

There are some additional changes coming the platform, which will reduce memory usage even further.

Thank you all for your continued support. 🙏

Cover image by Ian Taylor.

Hey awesome humans! This update will go unnoticed for most of you, but it actually took a surprisingly large amount of engineering resource to get done properly. 😅

What's the deal?

All existing Finger-Ink accounts were created by entering in an API key, which came out of Cliniko. Some accounts have multiple API keys connected: maybe one for each practitioner, or perhaps one for every connected portal user. These API keys are used as passwords so that we can talk to the Cliniko API on your behalf (to do things like send PDFs to your patient records).

Sometimes, these API keys become invalid:

1. They can be removed by the owner via the Cliniko UI.
2. The user can be made inactive in Cliniko.
3. The user can be removed from the original Cliniko account entirely.

If we ever tried to use an invalid API key with the Cliniko API, then Cliniko would tell us that the key is invalid. Before this change, if we ever received a reply from the API that a key was invalid, we'd remove it instantly and go on about our business.

Unfortunately, what we didn't know before, is that there are certain times where is is possible that the API will tell us that a given key is invalid, when it actually isn't. This is not very common, but it does happen occasionally.

During a recent Cliniko API outage, that only lasted a minute or so, our system managed to remove valid API keys from over 30 accounts — which was pretty annoying for everyone involved!

So what have we done?

We fixed it! 😁

There's a lot of technical details I could go into but, for fear of putting you to sleep, I shall not.

The gist is we will no longer remove a connected API key from Finger-Ink, until it has received quite a few "invalid" responses from the Cliniko API, in the space of a week or so.

You may occasionally get an email about a key being suspended. If this happens, and you don't think the key is invalid, please do let us know.

Otherwise, keep using Finger-Ink without having to worry about any of this. 🤩

Cover image by Yuyeung Lau

Hey all! This has actually live for a few weeks now, but here's the low down:

Our web forms have a new date picker! 🥳

Date picking on the web has been somewhat of a contentious issue for quite a while. We originally released our date fields with keyboard-only entry. This was great for some, but others wanted to be able to use a calendar to pick a date — so we changed it. For most, that was great. But for some using Finger-Ink forms internally, this really slowed down entry.

Well, now we've got the best of both worlds: keyboard entry by default, with a date picker when you want one.

Enjoy ❤️