Labels

Monday, February 25, 2019

File encoding option for spark YARN container

1) Configuration to pass file ecoding to YARN containers,

--conf 'spark.executor.extraJavaOptions=-Dfile.encoding=UTF-8'

2) File encoding option to read the file using spark local mode    
option("encoding", "UTF-8") .csv(inputPath)

No comments:

Post a Comment