Is it Work with spark 3?

[rubenssoto]

Hello,

Thank you for this great connector.

Is it work with spark 3?
This connector is only supported by this repository? I found in apache bahir repository too.

[abhishekd0907]

This should work with Spark 3 because DataSourceV1 works with Spark 3. I haven't tried it myself though.

[DipeshV]

Hello @abhishekd0907 ,

I am running this connector on EMR 6.1.0 which has Spark 3.0. While running the stream I am getting the below error.

java.util.ServiceConfigurationError: org.apache.spark.sql.sources.DataSourceRegister: Provider org.apache.spark.sql.streaming.sqs.SqsSourceProvider could not be instantiated
at java.util.ServiceLoader.fail(ServiceLoader.java:232)
at java.util.ServiceLoader.access$100(ServiceLoader.java:185)
at java.util.ServiceLoader$LazyIterator.nextService(ServiceLoader.java:384)
at java.util.ServiceLoader$LazyIterator.next(ServiceLoader.java:404)
at java.util.ServiceLoader$1.next(ServiceLoader.java:480)
at scala.collection.convert.Wrappers$JIteratorWrapper.next(Wrappers.scala:44)
at scala.collection.Iterator.foreach(Iterator.scala:941)
at scala.collection.Iterator.foreach$(Iterator.scala:941)
at scala.collection.AbstractIterator.foreach(Iterator.scala:1429)
at scala.collection.IterableLike.foreach(IterableLike.scala:74)
at scala.collection.IterableLike.foreach$(IterableLike.scala:73)
at scala.collection.AbstractIterable.foreach(Iterable.scala:56)
at scala.collection.TraversableLike.filterImpl(TraversableLike.scala:255)
at scala.collection.TraversableLike.filterImpl$(TraversableLike.scala:249)
at scala.collection.AbstractTraversable.filterImpl(Traversable.scala:108)
at scala.collection.TraversableLike.filter(TraversableLike.scala:347)
at scala.collection.TraversableLike.filter$(TraversableLike.scala:347)
at scala.collection.AbstractTraversable.filter(Traversable.scala:108)
at org.apache.spark.sql.execution.datasources.DataSource$.lookupDataSource(DataSource.scala:644)
at org.apache.spark.sql.streaming.DataStreamReader.load(DataStreamReader.scala:194)
... 56 elided
Caused by: java.lang.NoClassDefFoundError: org/apache/spark/internal/Logging$class
at org.apache.spark.sql.streaming.sqs.SqsSourceProvider.(SqsSourceProvider.scala:26)
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at java.lang.Class.newInstance(Class.java:442)
at java.util.ServiceLoader$LazyIterator.nextService(ServiceLoader.java:380)
... 73 more
Caused by: java.lang.ClassNotFoundException: org.apache.spark.internal.Logging$class
at java.net.URLClassLoader.findClass(URLClassLoader.java:382)
at java.lang.ClassLoader.loadClass(ClassLoader.java:418)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:352)
at java.lang.ClassLoader.loadClass(ClassLoader.java:351)
... 80 `more```

Seems like the connector is not compatible with Spark 3.0 on EMR. Also, Spark 3.0 is based on scala 2.12 and the connector is build on scala 2.11.

Can you provide any highlights on this?

Thanks and regards,
Dipesh Vora

[DipeshV]

Hey,
I did some troubleshooting and found that the class org/apache/spark/internal/Logging$class is not the part of aws's version of spark-core and hence the error.
I have downgarded to spark 2.4 and awaiting the fix on EMR
Thanks,
Dipesh

[abhishekd0907]

@DipeshV
Thanks for the update. It will help in saving time for other folks

[FortBlox]

@DipeshV
Do you know if there is an open ticket about this issue?
I can't find anything regarding the internal logging class.