A Simple Demo To Analyze 32GB AML Data With DuckDB

Weekend build and learn

Photo by Carlos Muza on Unsplash

Please note this article is solely for personal learning and development purposes during the weekend. It is not intended to promote or compare any Big Data vendors currently on the market. :)

Background

I recently reconnected with an old friend who currently holds a high-level role in data management.

During our conversation, we covered various topics, such as recent advancements in data technology and marketplace growth. Here are a few noteworthy points we discussed:

1. It's common for individuals and organizations to have a relatively small amount of data. You will unlikely require petabytes (PB) of data for your latest project. Instead, you have gigabytes (GB) or even megabytes (MB) of data.
2. It's important to remember that the core basis for utilizing data is determining the problem, solution, or story we want to extract for the benefit of the business. The focus should be on more than just the tools being used.
3. One of the most crucial steps for the success of any data project is data modeling, NOT Tooling.

That's interesting! You may have noticed that I "bolded" the NOT Tooling. The reason for this is that I believe the tooling is only…