Step 1) Add the following lines to spark-submit to check which jars are being loaded.
--conf "spark.driver.extraJavaOptions=-verbose:class"
Step 2) Then we can pass the path to jars that should be picked up first by. We can validate this using #1
--conf spark.driver.extraClassPath=<Pass the jar to be picked up by driver first>
--conf "spark.driver.extraJavaOptions=-verbose:class"
Step 2) Then we can pass the path to jars that should be picked up first by. We can validate this using #1
--conf spark.driver.extraClassPath=<Pass the jar to be picked up by driver first>
No comments:
Post a Comment