Code repository for DEBS 2022 Grand Challenge submission for Group 11.
- DEBS 2022 Grand Challenge information
- Evaluation platform and Leaderboard
- Project Code Walkthrough
- Project Presentation Video
You can find our technical paper here.
The following Flink operators have been designed to handle the stream of input batches. (Please click on the image for an enlarged view of the dataflow and dashboard)
- Source API: Gets new batches from the Challenge Client
- Event Generator: Consume each batch and generates StockMeasurement Events
- Query 1: EMA Calculation: A custom Window operator which emits new EMA values for every 5 minute window for each symbol
- Query 1: Benchmark Sideoutput: A sink for collecting benchmarks for every batch and submitting Query 1 results
- Query 2: Crossover check: An operator to check Query 2 requirements of bullish or bearish patterns and find crossover events
- Query 2: Benchmark Sideoutput: A sink for collecting benchmarks for every batch and publishing Query 2 results
- Close Benchmark: An operator which waits for Query 1 and Query 2 to complete before ending the benchmark
- Java OpenJDK 11
- Apache Maven
- Apache Flink 1.14.4
- Run the maven command:
clean package
to generate the required jar files: GrpcClient-.jar and GrpcClient-jar-with-dependencies.jar in the target folder. - Navigate to the
flink-1.14.3/bin
installation directory and start the Flink cluster by running./start-cluster.sh
(if running thru the provisioned VMs this will launch a distrbuted flink cluster with 3 worker nodes, and 12 task slots distributed over 12 CPUs) - Once the Flink cluster is up and running, access the Flink dashboard via
localhost:8081
in a browser window. - Upload the GrpcClient-.jar to the Flink dashboard (via the Submit Job tab) and submit the job to run.
- The progress of the job can be tracked in the Running Jobs tab on the Flink dashboard.
Vivek Unnikrishnan, Emmanouil Kritharakis, Karan Vombatkere, Shengyao Luo
Boston University, April 2022