Ever bought a piece of flatpack furniture and put it together without incident? Then you get that nagging doubt that it was too easy? Then two weeks later it collapses?
You’ve just learned what not to do!!!
(and that can often be the first step towards understanding what you need to do differently next time)
When things don't go smoothly straight away but you persevere, sometimes you'll end up with something that lasts for years, and remains the best thing you own
(My old Sega was a total nightmare to set up, tune in and get working but lasted 28 Sonic-fuelled years… They don’t make them like they used to).
No HDMI connectors in my day. Worth it.
What does any of this have to do with data?
Recently, we started looking at new ways to present and distribute data using R and Shiny powered web apps. The presentation part was no issue. Fortunately, we have a very talented team who can code R and make these apps.
The issue was the distribution. We have an Open Data platform but that wouldn’t support Shiny apps out of the box, we couldn’t even store them on there. Shiny (the company) supplies its own hosting solution but it’s a one-trick pony (and can get pretty costly).
If you’ve read any of the ecda blogs, you would possibly recall that we have a partner called Cloudwick who supply an AWS solution. One of the things that they offer are virtual servers in the cloud.
At this point, I should mention that I am a card-carrying tech nerd and have built all sorts of devices at home, including reconditioned laptops, games consoles out of old phones and a working web server out of a Raspberry Pi. So, I reckoned we could build a virtual server.
You remember the flatpack furniture that went together flawlessly, only to collapse two weeks later?
That’s pretty much what happened. Our first build went tragically and catastrophically wrong. We ended up rebooting 3-4 times a day just to make the apps work. It constantly flashed up system errors. It ran out of storage. Everything that could go wrong did.
(It really couldn’t)
At this point, someone with more common sense would have just got the off the shelf solution from Shiny. We are not those people.
This was not a failure. This was a learning curve.
Fortunately, the lovely folks at Cloudwick are more than happy to support our intrepidness/pig-headedness and were willing and ready to work with us to figure out where we went wrong. As one of their people said to me, ‘it happens. Servers go up, servers go down’.
The one of many joys of AWS is that you can just delete a server and re-build it quickly and easily without worrying about hardware specs, software installations and the like. You can fail freely without repercussion.
The second version of the R/Shiny server was deployed in less than two days, with a new lightweight OS, more efficient user environments and greater stability for users.
I’d love to say that’s the end of the story, yay we did it, miller time.
But I can’t.
A few weeks ago, one of our nice Shiny apps, started to gain real traction with an audience and we started getting massive amounts of users accessing it, often at the same time.
Our little server that could, couldn’t...
The demand was too high, and we had outages at the worst times. But instead of admitting defeat (and because we are ever so slightly masochistic), we decided to make tweaks, fixes and optimisations on the server.
So far, so good. You can go and see if the server is still working here or here and then tell me it’s not at open.data@essex.gov.uk. (Please note that the tools will not open in Internet explorer and are best viewed in Chrome).
I was approaching an overall point with this ramble and here it is;
This was not a quick win nor was it an easy thing to do nor are we even done yet. This was a long, hard slog with more than a few failures on the way. Progress is rarely the result of a single moment of inspiration. It usually requires graft, a certain level of stubbornness and, paradoxically, failure.
We have now gone live with a stable server and have launched it as an official Open Data Team Product™ and offered it out as part of Open Data’s publication offering to the rest of ECC. We hope that everyone will want to come with us and find new and innovative ways to share and present data.
So, don’t worry about failure. Use it. It’s probably the best teacher you’ll ever have.
Also, we have a new way of publishing data we’d like you to know about.
Leave a comment