Downgrade hadoop to 2.8 for docker setup #195
Conversation
| @@ -51,10 +51,10 @@ index d2e535a..2a3718f 100644 | |||
| <spark.home>${execution.root}/dev/spark</spark.home> | |||
| <spark.bin.download.url> | |||
| - https://archive.apache.org/dist/spark/spark-2.4.5/spark-2.4.5-bin-hadoop2.7.tgz | |||
| + https://archive.apache.org/dist/spark/spark-3.1.2/spark-3.1.2-bin-hadoop3.2.tgz | |||
| + https://archive.apache.org/dist/spark/spark-3.1.1/spark-3.1.1-bin-hadoop2.7.tgz | |||
There was a problem hiding this comment.
only 2.7 is available
teamurko
requested review from
sumedhsakdeo,
autumnust,
maluchari and
abhisheknath2011
|
It looks like we might be hiding an issue with downgrade being an workaround. Can we try to see why this stopped working? |
My understanding is it was not supposed to work but it turned out to be working for the first app we had, stats collector app and DLO app started hitting code path that surfaced incompatibility. From SO posts that surface similar problem, when downgrading helped, e.g. https://stackoverflow.com/questions/62880009/error-through-remote-spark-job-java-lang-illegalaccesserror-class-org-apache-h |
|
I see, why not go to the documented supported version 2.10 |
StatsCollector was working before. For eg: #72 . Post this there were no significant changes in statscollector. So I am not sure this is actually a day 0 issue. This appears to be something recent. |
Thank you for sharing @maluchari |
Summary
Maintenance apps happened to work on hadoop3 docker cluster, DLO apps didn't work due to incompatibility between source that depends on 2.10 and runtime jars on the cluster. DLO app example error:
Works after downgrade.
Changes
For all the boxes checked, please include additional details of the changes made in this pull request.
Testing Done
For all the boxes checked, include a detailed description of the testing done for the changes made in this pull request.
Additional Information
For all the boxes checked, include additional details of the changes made in this pull request.