PageRank causes java.util.NoSuchElementException #52
Comments
ghost
assigned 
|
Interestingly, #51 seems to fix the issue. |
|
simple code is less buggy :) |
|
Hmm, is that error in the master branch? Those line numbers don't seem to "line-up." I guess for now maybe it makes sense to merge #51 since it is simpler. |
|
I'm running into this issue in the latest version built off of #132 The code I'm running is pretty simple: val input = sc.sequenceFile[VectorWritable, VectorWritable](inputPath, classOf[VectorWritable], classOf[VectorWritable])
// not even parsing the vectors, just making some big graph
val edges = input.flatMap {
case (vec1, vec2) =>
Seq(Edge(Random.nextLong, Random.nextLong, 1), Edge(Random.nextLong(),Random.nextLong(), 1))
}
val g = Graph.fromEdges(edges, 1)
val cc = ConnectedComponents.run(g)
cc.vertices.count()
The error I see is this: This is running on a Hadoop 2.2.0 cluster with YARN 2.2.0. I've previously run against the same data set with Bagel and it works great. |
|
Just to make it simpler you can change the input to something like: edit: Also, oddly I can run it with say 11 splits on 10 works and it goes through. I see the same End of Stream error in the logs but it just fails that tasks and keeps going. |
|
Thanks for the report. I haven't yet been able to reproduce this. How many cores does Spark have in your configuration? |
|
@ankurdave I've tried it with 4-8 cores. I can try with 1 if you think it would help pin down what's going on. edit: Just to be clear, I mean 10-20 worker nodes with 4-8 cores each. (the machines have 24 cores each though). |
Empty edge partitions sometimes appear in the output of zipPartitions for unknown reasons, causing calls to Iterator#next to fail. This commit checks these cases, handles them by returning an empty iterator, and logs an error if this would cause GraphX to drop a corresponding non-empty partition. Resolves amplab/graphx#52.
Empty edge partitions sometimes appear in the output of zipPartitions for unknown reasons, causing calls to Iterator#next to fail. This commit checks these cases, handles them by returning an empty iterator, and logs an error if this would cause GraphX to drop a corresponding non-empty partition. Resolves amplab/graphx#52. (cherry picked from commit 2265c87c387979c94275e673a16527f582b2f38a)
Empty edge partitions sometimes appear in the output of zipPartitions for unknown reasons, causing calls to Iterator#next to fail. This commit checks these cases, handles them by returning an empty iterator, and logs an error if this would cause GraphX to drop a corresponding non-empty partition. Resolves amplab/graphx#52. (cherry picked from 7402177)
When running PageRank on a cluster, sometimes I hit a NoSuchElementException that's caused somewhere in VertexSetRDD. Full stack trace and command below. The line numbers may be slightly off due to debugging printlns.
Command:
Stack Trace:
The text was updated successfully, but these errors were encountered: