I lead the Data Engineering Practice within Scott Logic. I have a strong interest and expertise in low latency Front Office trading systems, software managing very large networks and the technologies involved in processing large volumes of data.
Data Engineering
The UK Government has an ambitious data strategy that aims to encourage and facilitate data sharing between departments and businesses. Elements of the strategy appear relatively straightforward, but how will the government fully realise the potential, and align citizens with this bold new approach?
Organisations across the globe have been on a journey to find the optimal approach for managing and leveraging analytics data. In this post, I’ll set out each of the key milestones on the journey, to arrive at the latest milestone – the Data Mesh paradigm – and ask whether it is really a thing.
·
Video
Data Mesh: What's different about it, what it means to DWP, and how you might approach it yourselves
Data Mesh is an emerging architectural paradigm for big data that addresses the challenges inherent in the data lake and data warehouse approaches. In this talk, I define what it is, and Charlie Boundy, DWP Deputy Director and Head of Data Science, brings it to life through DWP's technology and data vision.
What started as one faulty gas reading in the summer of 2017, ended up as a series of wasted calls where my bill kept getting higher and higher until it reached £11k. How could this have been handled faster and left me without considering moving energy provider.
Distributed stream processing engines have been on the rise in the last few years, first Hadoop became popular as a batch processing engine, then focus shifted towards stream processing engines. Stream processing engines can make the job of processing data that comes in via a stream easier than ever before.
Distributed log technologies have matured in the last few years. In this article, I review the attributes of distributed log technologies, compare four of the most popular and suggest how to choose the one that's right for you.
Data lakes? Most Financial Service organisations either have one or are considering one. This white paper explains key considerations and a warning to help guide their creation or optimisation to fulfil its requirement
Yesterday the Financial Times boldly declared that BP saved $7bn since 2014 by investing in Big Data technologies. I spent a couple of hours researching Big Data technologies associated with BP members of staff to try and build up a picture of exactly which technologies they are using.
Apache Spark has quickly become the largest open source project in big data, but why has it suddenly got so much momentum behind it?
Welcome to my blog, Andrew's thoughts on Big Data. This page gives a little background on myself and the blog.