This pull request makes newer Spark versions with yarn use Hadoop 2.7.

[lagerspetz]

This pull request makes newer Spark versions with yarn use Hadoop 2.7. The purpose is to make it possible to use the s3a filesystem scheme with spark-ec2 Amazon EC2 deployments.

[shivaram]

But doesn't this require the s3a jars to also be downloaded ? Also maybe we can add a new option for this rather than editing the existing one ?

[lagerspetz]

I'm not sure if s3a jars need to be in Spark's jars or if it is enough to have them as dependencies of a project wishing to use those URLs. In Carat, we use S3 directly so we have those jars already as dependencies.

In current Spark, s3n urls work; what would we need to make also s3a urls work out of the box?
According to this it should already be in Hadoop 2.7+
https://wiki.apache.org/hadoop/AmazonS3

[nchammas]

To use S3A you do need to pull in additional packages. For reference, this is how Flintrock does it:

• Enable easy S3 access nchammas/flintrock#180
• https://github.com/nchammas/flintrock/blob/4423a8a614e2d732e9f7928d2a04bf9193cce6ab/flintrock/templates/spark/conf/spark-defaults.conf

[lagerspetz]

Actually I think pull #56 does this better.