How to debug spark applications using Intellij IDE

1) Download IntelliJ for Linux

https://www.jetbrains.com/idea/download/#section=linux

2) Unpack the ideaIC.gz or ideaIU.gz file you have downloaded

tar xfz ideaIC.tar.gz or ideaIU.tar.gz. <new_archive_folder>

3) Extract the downloaded directory to /opt. For example, it's possible to enter the following command:

sudo tar xf -*.tar.gz -C /opt/

4) Start IntelliJ using command

/opt/idea-IC-173.4548.28/bin/idea.sh

5) Go to run->edit configurations-> remote and specify the hostname and port where you will run the spark submit job

6) Login to the node as specified in #5

export SPARK_JAVA_OPTS=-agentlib:jdwp=transport=dt_socket,server=y,suspend=n,address=5005

7) Start the PI job 

/opt/mapr/spark/spark-2.1.0/bin/spark-submit --class org.apache.spark.examples.SparkPi     --driver-java-options -agentlib:jdwp=transport=dt_socket,server=y,suspend=y,address=5005 --master yarn     --deploy-mode client     --driver-memory 1g     --executor-memory 2g     --executor-cores 1   /opt/mapr/spark/spark-2.1.0/examples/jars/spark-examples_2.11-2.1.0-mapr-1703.jar     10;

8) Now go to IntelliJ - > Run -> Debug and select the class to be enabled for debugger and add checkpoints.