Summary of work
The database change was successful and our systems are now connected to the new database within AWS.
Unforeseen issues
An issue related to the sequencing of the database caused the alarm processing to be stopped for 45mins instead of the expected 30mins.
Processing the queued events took longer than expected and this had an impact an processing new event being received. This caused delays in alarm processing for some sites overnight
In response to this we increased some resources this morning in order to regain normal operation
We fully tested the change process on our staging system prior to doing live. However due to the difference in size and number of alarm connections we have on live, these impacts were not foreseen on staging.
Conclusion
We are continuing to actively monitor the platform today. To report any issues, please use the blue chat icon in the bottom left to submit a support ticket or email support@conxtd.com.
Timeline
Tuesday 29th April
19:00 - Scheduled maintenance window starts
19:15 - Preparation
19:45 - Stopping events and queuing
19:55 - Access to CONXTD platform may be affected during the change
20:15 - Updating table sequences is taking longer than anticipated
20:30 - Change complete, alarms processing restarted & processing the queue.
20:45 - Queue of alarms still being processed. Taking longer than expected.
21:00 - Scheduled maintenance window ends
22:00 - A number of alarm connections have caught up, the remaining are working through a queue of events. There may be a delay in new events appearing in the platform.
23:00 - All alarm connections have caught up. Other internal queues continue.
30th April
06:30 - Queues were still high - increased the number of processes
07:00 - System has returned to normal operation. We will continue to closely monitor throughout the day
Maintenance Notification
Tuesday 29th April - 19:00 to 21:00 BST
During this window it will be necessary for us to pause event processing for up to 30mins while we switch over to our new database.
Alarm delivery to ARC via signalling platform will be unaffected by this work.
Details
Over the past 12 months, as part of our cloud strategy, we have been migrating our services to AWS. We’ve successfully migrated all our development and QA platforms and are now ready to move our live platform.
We have developed a process to ensure our migration keeps user disruption to a minimum.
Our database is currently being replicated across to AWS seamlessly in the background. Once the replication has been completed we will switch our services to connect to the new database. At that time it will be necessary to stop event processing while we switch over to the new database. This should be completed within 30 minutes. Any events received during this time will be queued and then processed once the switch over is complete. We have scheduled this out of hours to ensure minimum disruption.
We will keep a website banner updated with progress to ensure customers are aware of the status.
Why is the migration necessary
As part of our long term Cloud Strategy we have decided it is time to migrate our platform to AWS. As we continue to grow it will provide the resilience and performance our customers demand.
Future work
Once the database migration is complete, we will start to build out the rest of the services within AWS with no impact to customers. We will be building a new distributed architecture with resilience at every layer. Once ready, we will migrate inbound alarm connections one by one over to the new platform. The last step is switching our DNS records (i.e. app.conxtd.com) from the existing platform and point to the new platform. There will be no impact to users or alarm processing when we do this, however we will issue warning of this closer to the time.
CONXTD Cloud Strategy
Migrating cloud providers will allow us as a platform to continue focusing on developing new features and functionality for a customers while confident that our systems have the highest availability possible.
Benefits include:
Geo redundancy and resilience
Improved system monitoring
Support long term business growth
Improved deployment processes including CI/CD
Distributed architecture
Conclusion
The work is being completed by our senior developer and dev-ops engineer and overseen by our CTO.
Further Information
For more information please contact support@conxtd.com