Labels

Tuesday, December 4, 2018

Driver jars to be loaded first

Step 1)   Add the following lines to spark-submit to check which jars are being loaded.
 --conf "spark.driver.extraJavaOptions=-verbose:class"

Step 2) Then we can pass the path to jars that should be picked up first by. We can validate this using #1
  --conf spark.driver.extraClassPath=<Pass the jar to be picked up by driver first>

No comments:

Post a Comment