Dynamic width file in Spark
Download as DOCX, PDF0 likes41 views
The document describes an efficient method for handling dynamic width files in Spark using Scala, RDD, and DataFrame. It includes code examples for defining case classes and executing transformations to convert and analyze data. The process involves registering a DataFrame as a temporary table and executing analytical queries on the data.
![How to handle Dynamic Width File in Spark
Dynamic WidthFile is a common type of source fromMainframe sources;The Belowdemonstrationis one of the efficient
ways to handle dynamic widthFile usingScala, Spark RDDandDataframe. Check thiscode, Execute in your REPL.
Source File
Schema of the File
Code to be Executed
case classSubjectwisemarks(subject:String,marks:Int)
case classScoreRecord(id:Int,fname: String,lname:String,numberofsubject:Int,subjectwisemarks:
Seq[Subjectwisemarks])
val dataRDD = data.map(line
=>ScoreRecord(line.substring(0,2).toInt,line.substring(2,12).trim,line.substring(12,22).trim,line.subst
ring(22,24).toInt,convert(line,line.substring(22,24).toInt)));
val df = dataRDD.toDF
Convertisan User define functiontoconvertseriesof Subject-marksintoList
Dataframe Schema
Registeringas Temp Table and Show the Data](https://image.slidesharecdn.com/dynamicwidthfileinspark-2016-160711171248/85/Dynamic-width-file-in-Spark-1-320.jpg)
Dynamic width file in Spark
- 1. How to handle Dynamic Width File in Spark Dynamic WidthFile is a common type of source fromMainframe sources;The Belowdemonstrationis one of the efficient ways to handle dynamic widthFile usingScala, Spark RDDandDataframe. Check thiscode, Execute in your REPL. Source File Schema of the File Code to be Executed case classSubjectwisemarks(subject:String,marks:Int) case classScoreRecord(id:Int,fname: String,lname:String,numberofsubject:Int,subjectwisemarks: Seq[Subjectwisemarks]) val dataRDD = data.map(line =>ScoreRecord(line.substring(0,2).toInt,line.substring(2,12).trim,line.substring(12,22).trim,line.subst ring(22,24).toInt,convert(line,line.substring(22,24).toInt))); val df = dataRDD.toDF Convertisan User define functiontoconvertseriesof Subject-marksintoList Dataframe Schema Registeringas Temp Table and Show the Data
- 2. ImplementingAnalytical Queryinto the temptable SELECT id,fname,lname,CAST(sum(subject_wise_marks.marks)/numberofsubjectasDouble) FROM score LATERAL VIEW explode(subjectwisemarks) marks_table assubject_wise_marksgroupby id,fname,lname,numberofsubject; Result