In what feels like a long time ago now I wrote a blog around how excited I was about the planned migration of our on-premise data environment to the cloud. I was full of hope and dreams, and then we started…..
ECC was in a similar situation to many other organisations out there, we were looking to get off our existing on-premise setup in order to take full advantage of all the resources and benefits the cloud offered:
- Cost effectiveness
- Scalability
- Flexibility
- Accessibility
- Security
- Reliability.
Microsoft Azure, our chosen cloud solution, delivers on all these points and lots more, but the question I didn’t really ask myself back then was, is everyone in ECC ready for the cloud? We thought we were, yet we were still caught in the project’s wake, and we had our fair share of challenges to overcome.
Our old reporting setup had grown from a solution that was always intended to be temporary to (eight years later) a fully integrated product with limited technical support, apart from a few people that must have been making up for some passed transgressions in a previous life (myself included). Apart from being secure it delivered none of the benefits listed above, again it was supposed to be temporary, but that’s how a lot of organisations’ solutions start out. As far as end users and customers were concerned it wasn’t “too bad,” they had access to data via SQL server and reports could be deployed via an internal Power BI server that “wasn’t too unreliable”. But speaking on behalf of all the developers and database administrators we knew better existed!
So, the move to the cloud was the way forward. The environments were created, all the resources deployed, security, networking and permissions were in place, and it was over to us to start the migration.
We had reviewed the reporting requirements as well as the core data sources that underpinned those reports and we were in a good position to start the move and initially it all went well. We started off building the pipelines required to ingest the data from our source systems and redesigning the long overdue data structures to support data professionals more effectively.
For those of you that have experience with Azure or Synapse we soon realised there were multiple ways to go about most tasks. This is great and gives you flexibility, but when you don’t know the best approach it’s also incredible confusing. I personally take pride in my research methods – my googling abilities and more recently my in-depth conversations with various AIs - these skills were really sharpened during the migration! Every problem I investigated had almost what I was looking for, and lots that I wasn’t, but through hard work and a lot of patience the developers overcame the steep learning curve.
The developers have now also added a new language to their repertoire “Azure error codes”, it took us a while, but we are now fluent. My personal favourite, “user error” appeared more than once, and while I knew I had done something wrong in those early days a little more description of how I had broken a process would have made me question my life choices less often.
Learning a completely new system and processes is difficult, doing so while also trying to deliver a project that requires a good understanding of the new technology is even harder. In hindsight we could all have done with more time and training but there’s nothing like a deadline to focus the mind.
With everyone having gone through the experience above during the ingestion of data from source system, we moved onto the next phase of system processes, or as I like to refer to it Excel files. Now I’m not naïve, and as I like to tell people “I have been in this game a long time”, so I know spreadsheets exist and they’re not going away anytime soon, but why are there so many? Is it just to upset me?
Our old solution only dealt with a small percentage of these, but we were now looking at building Power BI models so needed them all ingested. We knew it was going to be a problem, so the only action was to tackle it head on, and we set about talking to the various data owners.
There was one question that people struggled with “Where does your data come from?” This really highlighted the need for clear data ownership for all data assets across the organisation. The developers, instead of building pipelines and ingesting data, ended up following data trails and it took a real effort to understand the requirements and deliver what our customers needed.
The migration gave us an opportunity to reconsider our data governance, how we engage with data owners and to introduce roles and procedures to resolve any issues. Data Governance boards, change control forums and DevOps has allowed us to move forward but we are still at an early stage.
Azure is fantastic, the migration to Azure was painful, but I truly believe that we had to go through the pain to understand and resolve the problems we faced across the organisation and ensure we have everything in place for to realise the ambitions of our data strategy and Digital Essex. Yes, we could have waited until we were all fully trained before we started. Yes, we could have spent longer understanding the data and yes, we could have been stricter with our data governance, but we managed to complete the migration and deliver immediate benefits for internal customers whose feedback has been overwhelmingly positive. There’s still a lot to work on and Azure has a huge number of resources that I’m yet to explore but we have taken a massive step forward and hopefully the next steps are smaller as we need some time to recover.
We made it to the cloud, just about!
Leave a comment