I have a text file and data is in string format like below
{"a":1,"b":2678,"c":40,"d":2}
{"a":4,"b":267,"c":26,"d":2}
{"a":1,"b":2678,"c":90,"d":17}
{"a":4,"b":2,"c":7,"d":15}
I read the file and converted into Array[String] using below code
val data = Source.fromFile(key+".txt").mkString.split("\n")
and o/p is like
[{"a":1,"b":2678,"c":40,"d":2},{"a":4,"b":267,"c":26,"d":2},{"a":1,"b":2678,"c":90,"d":17},{"a":4,"b":2,"c":7,"d":15}]
Till now, everything is fine and very fast,
then I wrote a function to convert every element into map
def convertToMap(msg:String): Map[String,Double] ={
var parseMsg = JSON.parseFull(msg)
var parseMsgToMap:Map[String,Any] = parseMsg match {
case Some(data: Map[String, Any]) => data
}
return parseMsgToMap.mapValues(_.toString.toDouble)
}
Then I called above function and converted result to list
var accumulatedData = data.map(x => convertToMap(x)).toList
But because I am using map and calling convertToMap function for each element, and converting the result to list.. It is consuming hell lot of time... approx 0.5ms for each element, I have around 200k records and it is taking 45 secs to convert to map
I am figuring out the most efficient way of converting it into List[Map[String,Double]] It should be very very fast
I need to convert this in list of Map
Comments
Post a Comment