You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
I wondered why the sparkbench/graph resource nweight-user-features is not loaded as a resource. The resource is packaged together with the .jar anyway, so I'd propose something like this:
NWeightDataGenerator.scala
[...]
defloadModel(modelPath: String, sc: SparkContext, partitions: Int):MatrixFactorizationModel= {
getClass.getResourceAsStream(modelPath) match {
casenull=>thrownewFileNotFoundException(modelPath)
case stream =>valin=newDataInputStream(stream)
valweights=newArray[(Int, Double)](MAX_ID)
for (i <- weights.indices) {
valw= in.readFloat()
weights(i) = (i, w)
}
in.close()
valuserFeatures= sc.parallelize(weights, math.max(400, partitions)).map { case (i, w) =>
(i, Array(w))
}
valuserFeaturesPartitioned= userFeatures.partitionBy(newHashPartitioner(partitions))
userFeaturesPartitioned.cache()
// Model matrix is symmetric, so productFeatures is the same with userFeaturesnewMatrixFactorizationModel(1, userFeaturesPartitioned, userFeaturesPartitioned)
}
}
[...]
Hi,
I wondered why the sparkbench/graph resource
nweight-user-features
is not loaded as a resource. The resource is packaged together with the .jar anyway, so I'd propose something like this:NWeightDataGenerator.scala
and the workload config:
nweight.conf
This could be beneficial in cases where the execution environment does not contain the hibench directory itself.
The text was updated successfully, but these errors were encountered: