Data engineering is the practice of designing and building systems for collecting, storing, and analyzing data at scale. It is a broad field with applications in just about every industry. Organizations have the ability to collect massive amounts of data, and they need the right people and technology to ensure it is in a highly usable state by the time it reaches data scientists and analysts. This seminar introduces the main concepts and different tools used on premise and cloud based.

• DBAs
• BI experts
• Data architects
• System Architects
• Data Analyst
• Data Scientists
• Software Developers
• DevOps

• The Data World Today
• Big Data Challenges
• What is Data engineer?
  • What are the roles and responsibilities?
  • Data Engineer vs Data Analyst vs Data Scientist
• AI Era and data engineering
• Data sources and file formats
• Data Warehouse vs Data Lake vs Delta Lake
• Big Data databases – Self Managed + Cloud Managed
  • Hadoop
  • Google Big Query
  • Amazon Athena
• NoSQL Databases, their types and use cases
  • Elasticsearch + ELK
  • MongoDB,
  • Redis
  • Cassandra
• ETL Methodologies and tools
• Batch Processing vs Stream Processing
• Streaming Tools – Kafka
• Spark Overview
• Example scenarios