To Our Cloud Customers:

On behalf of TechTime Initiative Group I want to apologise if this incident has affected performance of our apps for you.

What Happened?

Our Cloud app hosting is clustered. During a routine update of the hosting platform, one of the nodes didn’t quite come back to life – though the hosting platform was a alive and well, the actual apps (Google Maps Embedding Macro for Confluence Cloud and EasyTime for Jira Cloud) on that node didn’t re-enable successfully. Unfortunately due to the clustered nature of the platform – the problem was masked as the load balancer is configured to perform round-robin distribution of requests and so was happily sending the rest of the requests to the healthy nodes.

Once we identified what was going on – we restarted the software on the faulty node and everything came back working again.

What Are We Going to Do Now?

We have identified several areas where we can improve.

1. The smoke tests we perform after updates will now explicitly test individual nodes and the state of apps, not just the platform
2. Our regular health-checks will be enhanced in the same way
3. The load balancer will be re-configured to test the health of nodes in a more involved manner i.e. take into account apps, not just the platform
4. There will be more alerts – so we can be notified faster and deal with outages like this faster

More Information

Please feel free to contact our support if you require more information: https://techtime.co.nz/support

Sincerely,
Ed Letifov
CTO