Test project for storm-kafka-0.8-plus based on information provided in:
- [https://github.com/nathanmarz/storm-contrib/blob/master/storm-kafka/src/jvm/storm/kafka/TestTopology.java] (https://github.com/nathanmarz/storm-contrib/blob/master/storm-kafka/src/jvm/storm/kafka/TestTopology.java)
- [https://github.com/nathanmarz/storm/wiki/Trident-tutorial] (https://github.com/nathanmarz/storm/wiki/Trident-tutorial)
- [https://github.com/nathanmarz/storm/wiki/Trident-state] (https://github.com/nathanmarz/storm/wiki/Trident-state)
- [https://cwiki.apache.org/confluence/display/KAFKA/0.8.0+Producer+Example] (https://cwiki.apache.org/confluence/display/KAFKA/0.8.0+Producer+Example)
Also contains an attempt at a sample implementation of trident state based on [Hazelcast] (http://www.hazelcast.com/)
##Environment setup with Docker
If you are using a Mac follow the instructions here to setup a docker environment.
-
Install fig
-
Install storm (so you can upload your topology to the test cluster)
-
Start the test environment
fig up
-
Start a kafka shell
start-kafka-shell.sh <Docker Ip> <Zookeeper>
-
From within the shell, create a topic
$KAFKA_HOME/bin/kafka-topics.sh --create --topic storm-sentence --partitions 2 --zookeeper $ZK --replication-factor 1
-
For more details and troubleshooting see https://github.com/wurstmeister/kafka-docker and https://github.com/wurstmeister/storm-docker
##Build for running locally:
mvn clean package
##Build for running on a Storm cluster:
mvn clean package -P cluster
##Running the test topologies locally
java -cp target/storm-kafka-0.8-plus-test-0.2.0-SNAPSHOT-jar-with-dependencies.jar storm.kafka.trident.SentenceAggregationTopology <kafkaZookeeper>java -cp target/storm-kafka-0.8-plus-test-0.2.0-SNAPSHOT-jar-with-dependencies.jar storm.kafka.KafkaSpoutTestTopology <kafkaZookeeper>java -cp target/storm-kafka-0.8-plus-test-0.2.0-SNAPSHOT-jar-with-dependencies.jar storm.kafka.TestTopologyStaticHosts
##Running the test topologies on a storm cluster
storm jar target/storm-kafka-0.8-plus-test-0.2.0-SNAPSHOT-jar-with-dependencies.jar storm.kafka.trident.SentenceAggregationTopology <kafkaZookeeper> sentences <dockerIp>storm jar target/storm-kafka-0.8-plus-test-0.2.0-SNAPSHOT-jar-with-dependencies.jar storm.kafka.KafkaSpoutTestTopology <kafkaZookeeper> sentences <dockerIp>
The Storm UI will be available under: http://<dockerIp>:8080/
The Logviewer will be available under: http://<dockerIp>:8000/ e.g. http://<dockerIp>:8000/log?file=supervisor.log
##Producing data
To feed the topologies with data, start the StormProducer (built in local mode)
java -cp target/storm-kafka-0.8-plus-test-0.2.0-SNAPSHOT-jar-with-dependencies.jar storm.kafka.tools.StormProducer <dockerIp>:<kafkaPort>
Alternatively use the kafka console producer from within the kafka shell (see above)
$KAFKA_HOME/bin/kafka-console-producer.sh --topic=storm-sentence --broker-list=<dockerIp>:<kafkaPort>
##Consuming data
To run a DRPC query, start the DrpcClient (built in local mode)
java -cp target/storm-kafka-0.8-plus-test-0.2.0-SNAPSHOT-jar-with-dependencies.jar storm.kafka.tools.DrpcClient <dockerIp> 3772