AI tools write your ETL pipelines faster than ever – functional, correct code that gets the job done. But they make design decisions without understanding your data distribution, query patterns, or how the compute engine will actually execute the work and that’s where the real cost hides.
Your Spark job runs for 40 minutes, so you start guessing: more executors, different API calls, code rewrites, but nothing moves the needle because the bottleneck was never in the code.
This seminar teaches you to stop guessing – understanding how distributed compute and storage actually work from the inside out, so you can design a data stack that’s fast and cost-efficient at every layer. Every concept is demonstrated hands-on in Spark and Athena against the same data, with real measurements.
