🚀 Serverless ETL made easy – My introduction to AWS Glue

As part of our regular Tech Talk round, this time it was Michi's turn to present on the topic of AWS Glue:

More and more companies are facing the challenge of collecting data from a wide variety of sources, cleaning it up, and making it available for analytics or machine learning. Manual ETL processes are often error-prone and expensive – this is where AWS Glue comes in.

In the latest presentation, it was shown what serverless data integration can look like today and why AWS Glue is one of the most flexible ETL services on the market.

🔍 What makes AWS Glue special?

• Fully serverless ETL service – no infrastructure, no cluster management
• Supports PySpark, Python, and SQL
• Perfect for data lakes, analytics, and ML
• Automatic schema detection via Glue Crawler
• Glue Data Catalog as a central metadata hub
• Glue Studio & DataBrew for visual and no-code data preparation

🔧 This is what a typical AWS Glue pipeline looks like:

1. Raw data in S3
2. Glue Crawler recognizes structure & creates metadata
3. ETL job transforms data (e.g., PySpark)
4. Result ends up back in S3
5. Analysis via Athena or QuickSight

💡 My additional key takeaways:

• Glue integrates seamlessly with services such as Lake Formation, Lambda, Step Functions, and EventBridge
• Glue Streaming makes ETL possible in near real time
• Glue Studio Notebooks offer a modern development environment directly in the browser
• Job bookmarking makes it easy to implement incremental loads
• Ideal for teams that want a standardized ETL stack without administrative overhead

⚠️ Limitations to be aware of:

• Cold starts can delay jobs
• Less control than with EMR or your own Spark clusters

🎯 Conclusion: AWS Glue is a powerful, scalable foundation for modern data pipelines—especially when flexibility, speed, and low operational overhead are the focus.