I lead the Data Engineering Practice within Scott Logic. I have a strong interest and expertise in low latency Front Office trading systems, software managing very large networks and the technologies involved in processing large volumes of data.
Podcast
In this episode, I’m joined by Technology Lead Andrew Carr and CTO Colin Eberhardt to delve into the evolving nature of technology strategies within organisations.
Cloud
The impending Digital Operational Resilience Act (DORA) aims to strengthen the IT security of financial entities such as banks, insurance companies and investment firms across Europe. The Bank of England launched an equivalent 'Operational Resilience' regime in March 2022. In this blog post, I share Monzo's smart, modern resilience strategy and look at why traditional resilience strategies put organisations at significant risk.
Podcast
In this episode, Oliver Cronk, Andrew Carr and David Hope talk about the ever-changing world of data, with conversations moving from data warehouse to data lake, and data mesh to data fabric. They discuss the importance of data ownership and common tooling, and their view that data mesh is an approach rather than an architecture.
Tech
I wrote recently about what the future holds for Open Banking in the US. But what about the UK? We’re now over five years into the Open Banking era in the UK. How is it reshaping the banking sector, and what’s coming next?
Tech
If you work in banking in the US, you should care about Open Banking. To help argue the case, I look at the legal changes in the US that will likely push towards an Open Banking standard and argue that implementing Open Banking can reduce your regular operational costs as a bank.
Artificial Intelligence
AI tools are being trialled in many areas including software development, with huge potential to transform ways of working. What might the impact of AI be on software architecture? In this post, I make some predictions and argue that software architecture may become more data-driven, with prototype architectures and architectural diagrams being generated from the actual code itself.
Data Engineering
The UK Government has an ambitious data strategy that aims to encourage and facilitate data sharing between departments and businesses. Elements of the strategy appear relatively straightforward, but how will the government fully realise the potential, and align citizens with this bold new approach?
Data Engineering
Organisations across the globe have been on a journey to find the optimal approach for managing and leveraging analytics data. In this post, I’ll set out each of the key milestones on the journey, to arrive at the latest milestone – the Data Mesh paradigm – and ask whether it is really a thing.
Tech
·
Video
Data Mesh: What's different about it, what it means to DWP, and how you might approach it yourselves
Data Mesh is an emerging architectural paradigm for big data that addresses the challenges inherent in the data lake and data warehouse approaches. In this talk, I define what it is, and Charlie Boundy, DWP Deputy Director and Head of Data Science, brings it to life through DWP's technology and data vision.
Data Engineering
What started as one faulty gas reading in the summer of 2017, ended up as a series of wasted calls where my bill kept getting higher and higher until it reached £11k. How could this have been handled faster and left me without considering moving energy provider.
Data Engineering
Distributed stream processing engines have been on the rise in the last few years, first Hadoop became popular as a batch processing engine, then focus shifted towards stream processing engines. Stream processing engines can make the job of processing data that comes in via a stream easier than ever before.
Data Engineering
Distributed log technologies have matured in the last few years. In this article, I review the attributes of distributed log technologies, compare four of the most popular and suggest how to choose the one that's right for you.
Resources
Data lakes? Most Financial Service organisations either have one or are considering one. This white paper explains key considerations and a warning to help guide their creation or optimisation to fulfil its requirement
Data Engineering
Yesterday the Financial Times boldly declared that BP saved $7bn since 2014 by investing in Big Data technologies. I spent a couple of hours researching Big Data technologies associated with BP members of staff to try and build up a picture of exactly which technologies they are using.
Data Engineering
Apache Spark has quickly become the largest open source project in big data, but why has it suddenly got so much momentum behind it?
Data Engineering
Welcome to my blog, Andrew's thoughts on Big Data. This page gives a little background on myself and the blog.