Thunderbirds are go
How we rescued a database driven SaaS system in the Azure cloud
Thunderbirds are go
How we rescued a database driven SaaS system in the Azure cloud
- 3 minute read
Every so often we are tasked with rescuing a project. The rationale behind why a project finds itself in a position where it needs rescuing tends to vary widely, it could be financial, political, ownership, shifting market forces or a change of direction.
Inflight projects can arrive at our door in any state, from part built all the way to production. Sometimes progress has stalled but in the worse cases the deliverable is no longer functional.
A recent case was a database driven SaaS system we were asked to look at a that was built in the Azure cloud. The system was commissioned by a US based company operating in the domestic healthcare sector. This SaaS system is designed to ingest patient demographic and prescription data from a number of subscribing entities, then standardise and store the data to a level that will stand up to external auditing. The resulting data can then be analysed and modelled to create market intelligence.
After several unsuccessful attempts to revive the project it had stagnated and the customer no longer had confidence the system was working. The original developer was not reachable, one thing led to another and it was now our time to us to see what we could do to help.
So we rolled up our sleeves and dived in. Having read though the supplied documentation we grabbed the first set of test data and put the wheels in motion. All went well for a good portion of the process but then things suddenly ground to a halt.
Unfortunately there was little by way of feedback or error messages for us to go on, so this meant it was time to dive in to the code. We fired up our editors of choice and were greeted with something we'd not seen for a long time, Perl. It's true what they say about picking up inflight projects, you really do have to expect the unexpected.
We hadn’t worked with Perl in quite a while up to this point, but that wasn't going to deter us. After some trial and error we identified where we thought the issue could be. We suspected the issue was a communication breakdown between the system itself, which is in a VM, and the master data store which is a database node external to the VM. To test our theory we repointed the relevant section of the program at a database local to the VM and tried again. This time the routine ran without a hitch and the data transfer process completed. Switching back to the original target we tried one last long shot, but that failed to complete, as we suspected it would. It was time to update the code to make it more chatty so we could see more of what was happening during this phase of the process.
It turned out that almost as soon as the routine initiated the data transfer it fell over, so we thought perhaps it might be credentials, particularly as we’d needed to modify those used when testing locally. Adding some feedback at that point revealed that a positive connection was being established so we moved on. After some more digging and testing we discovered that the underlying function used to transfer the data was not able to cope with the quantity of data being provided and was bailing out. Digging through the Azure knowledge base revealed that this particular function had capacity limitations and needed to be replaced.
So we set about updating the code for the preferred function, added in some extra feedback so we could ensure it was behaving as expected, and fired it up again. This time data flowed flawlessly out of the VM into the database node.
The first real test of our update came a few weeks later with the arrival of the first of several batches of subscriber data. We headed back to Tracy Island happy that day having successfully ingested a little over 18 million rows into the database, giving the customer a nice chunk of data to analyse.