1) Add this property to the spark job
--conf "spark.yarn.appMasterEnv.LD_PRELOAD=/opt/mapr/lib/libmemtracker.so /usr/lib64/libstdc++.so.6"
2) Make sure /opt/mapr/lib/libmemtracker.so & /usr/lib64/libstdc++.so.6 is avaliable on all the nodes
3) To verify the library are being loaded
[mapr@pn1 jitendra]$ jps -ml | grep ApplicationMaster
29887 org.apache.spark.deploy.yarn.ApplicationMaster --class streaming.SubscribePattern --jar file:/home/mapr/YuCodes3X/target/YuCodes3X-0.0.1-SNAPSHOT.jar --arg /tmp/kafka2 --arg mapr2 --arg earliest --properties-file /tmp/hadoop-mapr/nm-local-dir/usercache/mapr/appcache/application_1541789620039_0057/container_e26_1541789620039_0057_01_000001/__spark_conf__/__spark_conf__.properties
pmap -x 29887 | grep -i libmemtracker
4) Collect the gcore when you see memory usage of spark driver(AM) being high
e.g) Collect during 6G,8G,9G considering driver memory is 10G.
gcore 29887
[mapr@pn1 ~]$ file core.29887
core.29887: ELF 64-bit LSB core file x86-64, version 1 (SYSV), SVR4-style, from '/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.181-3.b13.el7_5.x86_64/jre/bin/java'
[mapr@pn1 ~]$ gdb /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.181-3.b13.el7_5.x86_64/jre/bin/java core.29887
GNU gdb (GDB) 7.8.2
Copyright (C) 2014 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law. Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-unknown-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
<http://www.gnu.org/software/gdb/documentation/>.
For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.181-3.b13.el7_5.x86_64/jre/bin/java...
warning: Cannot parse .gnu_debugdata section; LZMA support was disabled at compile time
(no debugging symbols found)...done.
[New LWP 4583]
[New LWP 29887]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib64/libthread_db.so.1".
warning: Cannot parse .gnu_debugdata section; LZMA support was disabled at compile time
warning: Cannot parse .gnu_debugdata section; LZMA support was disabled at compile time
Core was generated by `/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.181-3.b13.el7_5.x86_64/jre/bin/java'.
#0 0x00007f7d6d487995 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
(gdb) info shared lib
From To Syms Read Shared Object Library
0x00007f7d6d9a0fe0 0x00007f7d6d9a17ae Yes /opt/mapr/lib/libmemtracker.so
0x00007f7d6d6f3510 0x00007f7d6d75a5ba Yes (*) /usr/lib64/libstdc++.so.6
0x00007f7d6d481900 0x00007f7d6d48ce51 Yes (*) /lib64/libpthread.so.0
Ref : using-ld-preload-with-apache-spark-or-yarn
--conf "spark.yarn.appMasterEnv.LD_PRELOAD=/opt/mapr/lib/libmemtracker.so /usr/lib64/libstdc++.so.6"
2) Make sure /opt/mapr/lib/libmemtracker.so & /usr/lib64/libstdc++.so.6 is avaliable on all the nodes
3) To verify the library are being loaded
[mapr@pn1 jitendra]$ jps -ml | grep ApplicationMaster
29887 org.apache.spark.deploy.yarn.ApplicationMaster --class streaming.SubscribePattern --jar file:/home/mapr/YuCodes3X/target/YuCodes3X-0.0.1-SNAPSHOT.jar --arg /tmp/kafka2 --arg mapr2 --arg earliest --properties-file /tmp/hadoop-mapr/nm-local-dir/usercache/mapr/appcache/application_1541789620039_0057/container_e26_1541789620039_0057_01_000001/__spark_conf__/__spark_conf__.properties
pmap -x 29887 | grep -i libmemtracker
4) Collect the gcore when you see memory usage of spark driver(AM) being high
e.g) Collect during 6G,8G,9G considering driver memory is 10G.
gcore 29887
[mapr@pn1 ~]$ file core.29887
core.29887: ELF 64-bit LSB core file x86-64, version 1 (SYSV), SVR4-style, from '/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.181-3.b13.el7_5.x86_64/jre/bin/java'
[mapr@pn1 ~]$ gdb /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.181-3.b13.el7_5.x86_64/jre/bin/java core.29887
GNU gdb (GDB) 7.8.2
Copyright (C) 2014 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law. Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-unknown-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
<http://www.gnu.org/software/gdb/documentation/>.
For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.181-3.b13.el7_5.x86_64/jre/bin/java...
warning: Cannot parse .gnu_debugdata section; LZMA support was disabled at compile time
(no debugging symbols found)...done.
[New LWP 4583]
[New LWP 29887]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib64/libthread_db.so.1".
warning: Cannot parse .gnu_debugdata section; LZMA support was disabled at compile time
warning: Cannot parse .gnu_debugdata section; LZMA support was disabled at compile time
Core was generated by `/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.181-3.b13.el7_5.x86_64/jre/bin/java'.
#0 0x00007f7d6d487995 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
(gdb) info shared lib
From To Syms Read Shared Object Library
0x00007f7d6d9a0fe0 0x00007f7d6d9a17ae Yes /opt/mapr/lib/libmemtracker.so
0x00007f7d6d6f3510 0x00007f7d6d75a5ba Yes (*) /usr/lib64/libstdc++.so.6
0x00007f7d6d481900 0x00007f7d6d48ce51 Yes (*) /lib64/libpthread.so.0
Ref : using-ld-preload-with-apache-spark-or-yarn
No comments:
Post a Comment