Skip to content

Experiments with Spark, Kafka, Avro, Schema Registry, Hudi and others

Notifications You must be signed in to change notification settings

ahappypie/spotter

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

6 Commits
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 

Repository files navigation

Spotter

Experiments with Spark, Kafka, Avro, Schema Registry, Hudi and others

Setup Hudi (Optional once Hudi releases Scala 2.12 artifacts)

  • Clone apache/incubator-hudi
  • Build Hudi: dev/change-scala-version.sh 2.12 && mvn -Pscala-2.12 -DskipITs clean install
  • Point Hudi dependency at local file

Development Environment

  • See ahappypie/aws-spot for data generation and Kafka/Zookeeper/Schema Registry compose file (may need to docker create network spotter)
  • Bring up minio: docker-compose up -d

About

Experiments with Spark, Kafka, Avro, Schema Registry, Hudi and others

Resources

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published