We read every piece of feedback, and take your input very seriously.
To see all available qualifiers, see our documentation.
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
spark-tensorflow-connector seems not working with Spark 2.0.X (The prerequisite in readme is Spark 2.0 or later)
spark-tensorflow-connector
I got the following error when compiling with Spark 2.0.2
$ mvn install [INFO] Scanning for projects... [INFO] [INFO] -----------< org.tensorflow:spark-tensorflow-connector_2.11 >----------- [INFO] Building spark-tensorflow-connector 1.10.0 [INFO] --------------------------------[ jar ]--------------------------------- [INFO] [INFO] --- scala-maven-plugin:3.2.2:add-source (compile) @ spark-tensorflow-connector_2.11 --- [INFO] Add Source directory: /Users/edison/IdeaProjects/ecosystem/spark/spark-tensorflow-connector/src/main/scala [INFO] Add Test Source directory: /Users/edison/IdeaProjects/ecosystem/spark/spark-tensorflow-connector/src/test/scala [INFO] [INFO] --- scala-maven-plugin:3.2.2:add-source (test) @ spark-tensorflow-connector_2.11 --- [INFO] [INFO] --- maven-resources-plugin:2.6:resources (default-resources) @ spark-tensorflow-connector_2.11 --- [INFO] Using 'UTF-8' encoding to copy filtered resources. [INFO] Copying 1 resource [INFO] [INFO] --- maven-compiler-plugin:3.0:compile (default-compile) @ spark-tensorflow-connector_2.11 --- [INFO] Changes detected - recompiling the module! [INFO] [INFO] --- scala-maven-plugin:3.2.2:compile (compile) @ spark-tensorflow-connector_2.11 --- [INFO] Using incremental compilation [INFO] Compiling 10 Scala sources to /Users/edison/IdeaProjects/ecosystem/spark/spark-tensorflow-connector/target/classes... [ERROR] /Users/edison/IdeaProjects/ecosystem/spark/spark-tensorflow-connector/src/main/scala/org/tensorflow/spark/datasources/tfrecords/serde/DefaultTfRecordRowEncoder.scala:121: not found: value ArrayData [ERROR] Int64ListFeatureEncoder.encode(ArrayData.toArrayData(row.get(index)).toIntArray().map(_.toLong)) [ERROR] ^ [ERROR] /Users/edison/IdeaProjects/ecosystem/spark/spark-tensorflow-connector/src/main/scala/org/tensorflow/spark/datasources/tfrecords/serde/DefaultTfRecordRowEncoder.scala:123: not found: value ArrayData [ERROR] Int64ListFeatureEncoder.encode(ArrayData.toArrayData(row.get(index)).toLongArray()) [ERROR] ^ [ERROR] /Users/edison/IdeaProjects/ecosystem/spark/spark-tensorflow-connector/src/main/scala/org/tensorflow/spark/datasources/tfrecords/serde/DefaultTfRecordRowEncoder.scala:125: not found: value ArrayData [ERROR] FloatListFeatureEncoder.encode(ArrayData.toArrayData(row.get(index)).toFloatArray()) [ERROR] ^ [ERROR] /Users/edison/IdeaProjects/ecosystem/spark/spark-tensorflow-connector/src/main/scala/org/tensorflow/spark/datasources/tfrecords/serde/DefaultTfRecordRowEncoder.scala:127: not found: value ArrayData [ERROR] FloatListFeatureEncoder.encode(ArrayData.toArrayData(row.get(index)).toDoubleArray().map(_.toFloat)) [ERROR] ^ [ERROR] /Users/edison/IdeaProjects/ecosystem/spark/spark-tensorflow-connector/src/main/scala/org/tensorflow/spark/datasources/tfrecords/serde/DefaultTfRecordRowEncoder.scala:129: not found: value ArrayData [ERROR] val decimalArray = ArrayData.toArrayData(row.get(index)).toArray[Decimal](DataTypes.createDecimalType()) [ERROR] ^ [ERROR] /Users/edison/IdeaProjects/ecosystem/spark/spark-tensorflow-connector/src/main/scala/org/tensorflow/spark/datasources/tfrecords/serde/DefaultTfRecordRowEncoder.scala:132: not found: value ArrayData [ERROR] BytesListFeatureEncoder.encode(ArrayData.toArrayData(row.get(index)) [ERROR] ^ [ERROR] /Users/edison/IdeaProjects/ecosystem/spark/spark-tensorflow-connector/src/main/scala/org/tensorflow/spark/datasources/tfrecords/serde/DefaultTfRecordRowEncoder.scala:133: object ObjectType in package types cannot be accessed in package org.apache.spark.sql.types [ERROR] .toArray[String](ObjectType(classOf[String])).map(_.getBytes)) [ERROR] ^ [ERROR] /Users/edison/IdeaProjects/ecosystem/spark/spark-tensorflow-connector/src/main/scala/org/tensorflow/spark/datasources/tfrecords/serde/DefaultTfRecordRowEncoder.scala:135: not found: value ArrayData [ERROR] BytesListFeatureEncoder.encode(ArrayData.toArrayData(row.get(index)).toArray[Array[Byte]](BinaryType)) [ERROR] ^ [ERROR] /Users/edison/IdeaProjects/ecosystem/spark/spark-tensorflow-connector/src/main/scala/org/tensorflow/spark/datasources/tfrecords/serde/DefaultTfRecordRowEncoder.scala:153: not found: value ArrayData [ERROR] val longArrays = ArrayData.toArrayData(row.get(index)).array.map {arr => [ERROR] ^ [ERROR] /Users/edison/IdeaProjects/ecosystem/spark/spark-tensorflow-connector/src/main/scala/org/tensorflow/spark/datasources/tfrecords/serde/DefaultTfRecordRowEncoder.scala:154: not found: value ArrayData [ERROR] ArrayData.toArrayData(arr).toIntArray().map(_.toLong).toSeq [ERROR] ^ [ERROR] /Users/edison/IdeaProjects/ecosystem/spark/spark-tensorflow-connector/src/main/scala/org/tensorflow/spark/datasources/tfrecords/serde/DefaultTfRecordRowEncoder.scala:159: not found: value ArrayData [ERROR] val longArrays = ArrayData.toArrayData(row.get(index)).array.map {arr => [ERROR] ^ [ERROR] /Users/edison/IdeaProjects/ecosystem/spark/spark-tensorflow-connector/src/main/scala/org/tensorflow/spark/datasources/tfrecords/serde/DefaultTfRecordRowEncoder.scala:160: not found: value ArrayData [ERROR] ArrayData.toArrayData(arr).toLongArray().toSeq [ERROR] ^ [ERROR] /Users/edison/IdeaProjects/ecosystem/spark/spark-tensorflow-connector/src/main/scala/org/tensorflow/spark/datasources/tfrecords/serde/DefaultTfRecordRowEncoder.scala:165: not found: value ArrayData [ERROR] val floatArrays = ArrayData.toArrayData(row.get(index)).array.map {arr => [ERROR] ^ [ERROR] /Users/edison/IdeaProjects/ecosystem/spark/spark-tensorflow-connector/src/main/scala/org/tensorflow/spark/datasources/tfrecords/serde/DefaultTfRecordRowEncoder.scala:166: not found: value ArrayData [ERROR] ArrayData.toArrayData(arr).toFloatArray().toSeq [ERROR] ^ [ERROR] /Users/edison/IdeaProjects/ecosystem/spark/spark-tensorflow-connector/src/main/scala/org/tensorflow/spark/datasources/tfrecords/serde/DefaultTfRecordRowEncoder.scala:171: not found: value ArrayData [ERROR] val floatArrays = ArrayData.toArrayData(row.get(index)).array.map {arr => [ERROR] ^ [ERROR] /Users/edison/IdeaProjects/ecosystem/spark/spark-tensorflow-connector/src/main/scala/org/tensorflow/spark/datasources/tfrecords/serde/DefaultTfRecordRowEncoder.scala:172: not found: value ArrayData [ERROR] ArrayData.toArrayData(arr).toDoubleArray().map(_.toFloat).toSeq [ERROR] ^ [ERROR] /Users/edison/IdeaProjects/ecosystem/spark/spark-tensorflow-connector/src/main/scala/org/tensorflow/spark/datasources/tfrecords/serde/DefaultTfRecordRowEncoder.scala:177: not found: value ArrayData [ERROR] val floatArrays = ArrayData.toArrayData(row.get(index)).array.map {arr => [ERROR] ^ [ERROR] /Users/edison/IdeaProjects/ecosystem/spark/spark-tensorflow-connector/src/main/scala/org/tensorflow/spark/datasources/tfrecords/serde/DefaultTfRecordRowEncoder.scala:178: not found: value ArrayData [ERROR] ArrayData.toArrayData(arr).toArray[Decimal](DataTypes.createDecimalType()).map(_.toFloat).toSeq [ERROR] ^ [ERROR] /Users/edison/IdeaProjects/ecosystem/spark/spark-tensorflow-connector/src/main/scala/org/tensorflow/spark/datasources/tfrecords/serde/DefaultTfRecordRowEncoder.scala:183: not found: value ArrayData [ERROR] val arrayData = ArrayData.toArrayData(row.get(index)).array.map {arr => [ERROR] ^ [ERROR] /Users/edison/IdeaProjects/ecosystem/spark/spark-tensorflow-connector/src/main/scala/org/tensorflow/spark/datasources/tfrecords/serde/DefaultTfRecordRowEncoder.scala:184: not found: value ArrayData [ERROR] ArrayData.toArrayData(arr).toArray[String](ObjectType(classOf[String])).toSeq.map(_.getBytes) [ERROR] ^ [ERROR] /Users/edison/IdeaProjects/ecosystem/spark/spark-tensorflow-connector/src/main/scala/org/tensorflow/spark/datasources/tfrecords/serde/DefaultTfRecordRowEncoder.scala:184: not found: value ObjectType [ERROR] ArrayData.toArrayData(arr).toArray[String](ObjectType(classOf[String])).toSeq.map(_.getBytes) [ERROR] ^ [ERROR] /Users/edison/IdeaProjects/ecosystem/spark/spark-tensorflow-connector/src/main/scala/org/tensorflow/spark/datasources/tfrecords/serde/DefaultTfRecordRowEncoder.scala:189: not found: value ArrayData [ERROR] val arrayData = ArrayData.toArrayData(row.get(index)).array.map {arr => [ERROR] ^ [ERROR] /Users/edison/IdeaProjects/ecosystem/spark/spark-tensorflow-connector/src/main/scala/org/tensorflow/spark/datasources/tfrecords/serde/DefaultTfRecordRowEncoder.scala:190: not found: value ArrayData [ERROR] ArrayData.toArrayData(arr).toArray[Array[Byte]](BinaryType).toSeq [ERROR] ^ [ERROR] 23 errors found [INFO] ------------------------------------------------------------------------ [INFO] BUILD FAILURE [INFO] ------------------------------------------------------------------------ [INFO] Total time: 10.272 s [INFO] Finished at: 2020-04-28T17:10:02+09:00 [INFO] ------------------------------------------------------------------------ [ERROR] Failed to execute goal net.alchim31.maven:scala-maven-plugin:3.2.2:compile (compile) on project spark-tensorflow-connector_2.11: Execution compile of goal net.alchim31.maven:scala-maven-plugin:3.2.2:compile failed.: CompileFailed -> [Help 1] [ERROR] [ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch. [ERROR] Re-run Maven using the -X switch to enable full debug logging. [ERROR] [ERROR] For more information about the errors and possible solutions, please read the following articles: [ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/PluginExecutionException
The problem solves when I change Spark version to 2.1.X (or above)
The text was updated successfully, but these errors were encountered:
No branches or pull requests
spark-tensorflow-connectorseems not working with Spark 2.0.X(The prerequisite in readme is Spark 2.0 or later)
I got the following error when compiling with Spark 2.0.2
The problem solves when I change Spark version to 2.1.X (or above)
The text was updated successfully, but these errors were encountered: