Skip to content

Latest commit

 

History

History
30 lines (20 loc) · 901 Bytes

README.md

File metadata and controls

30 lines (20 loc) · 901 Bytes
Raw

Stock Market Kafka Real Time Data Engineering Project

Introduction

In this project, you will execute an End-To-End Data Engineering Project on Real-Time Stock Market Data using Kafka.

We are going to use different technologies such as Python, Amazon Web Services (AWS), Apache Kafka, Glue, Athena, and SQL.

Architecture

Technology Used

  • Programming Language - Python
  • Amazon Web Service (AWS)
  1. S3 (Simple Storage Service)
  2. Athena
  3. Glue Crawler
  4. Glue Catalog
  5. EC2
  • Apache Kafka

Dataset Used

You can use any dataset, we are mainly interested in operation side of Data Engineering (building data pipeline)

Here is the dataset used in the video - https://github.com/darshilparmar/stock-market-kafka-data-engineering-project/blob/main/indexProcessed.csv

Complete Video Tutorial

Video Link - https://www.youtube.com/embed/KerNf0NANMo