Building a data pipeline from Batch and Streaming data

This presentation delves into the design and optimization of data pipelines, emphasizing the distinctions between batch and streaming data and their applications. It explores Medallion Architecture with Bronze, Silver, and Gold layers, leveraging AWS, Kinesis, and Snowflake for ingestion, and dbt for real-time updates. Focused attention is given to compute resource management, performance enhancements, and visualization strategies using tools like Tableau and Power BI. Practical examples,...