Member-only story

Using Google BigQuery OMNI to Query Data In-place From AWS S3

The most modern mechanism for shifting your data from one cloud to another for analytics and insights might be to not move it at all

James Matson
Better Programming
10 min readAug 29, 2023

--

From AWS to GCP via the route of magic. No, not really. It’s BigQuery OMNI.

The facts of data analytics

If you’ve been following the trends around data and analytics for a few years now, you’ve probably come across one or more BI or data maturity curve graphs. You know the ones — they start with the least mature organisation dealing with static BI reports and being ‘reactive,’ and gradually ‘maturing’ towards proactive insights and a data-driven culture.

One moment, let’s see if I can find one.

(A few seconds of Google later…)

Ah, here we go!

Image source

There are plenty of variations on this theme of maturity, but they all share a common thread of moving from bits of data siloed all over the place towards a fully accessible and well-formed data warehouse or lake. Here, the data from all over your landscape is brought together and surfaced in a way that makes gathering insights simple and effective.

--

--

James Matson
James Matson

Written by James Matson

DevOps Lead, C# and Python enthusiast, Writer, AWS Community Builder, Microsoft PowerApps Champion, AI/ML Tinkerer

Responses (1)

What are your thoughts?