i am working on a POC where i need to do the following
i am using java 8 for below processing
read data from a text file(comma separated data) and push it to ActiveMQ one by one. each row is of ~500 characters in length and like that there are 100K records.
same time i need to consume those records and push it to a file in LOCAL file System (LFS).
i am able to achieve it using java but i do not see the performance is satisfactory. kindly help me with your suggestions to boost its code quality and performance
i have also tried to use Java Stream from java 8, but i dint get how to get each row out of stream object and push it to producer.
how should i handle logic if there is no data to be fetch from queue, i need to trigger a job. how should i handle this scenario.
code for point 1:
String fileString = new String(Files.readAllBytes(Paths.get(INPUTFILE)), StandardCharsets.UTF_8);
String[] Line = fileString.split("\n");
for (int i = 0; i < Line.length; i++) {
TextMessage textMessage = session.createTextMessage(Line[i].toString().trim());
producer.send(textMessage);
//System.out.println(textMessage.getText());
}
code for Point 2:
File fileName = new File(sHdfsLandOutPath);
FileWriter fr = null;
BufferedWriter br = null;
fr = new FileWriter(fileName);
br = new BufferedWriter(fr);
try {
while (true) {
Message message = consumer.receive();
if (message instanceof TextMessage) {
TextMessage textMessage = (TextMessage) message;
String str = textMessage.getText();
sMsgCounter++;
sTotalRecCount++;
try {
if (sMsgCounter == sHdfsPushCount) {
System.out.println("Pushing Data in bulk to local file");
System.out.println(sTotalRecCount);
br.write(str );
br.newLine();
br.flush();
sMsgCounter = 0;
} else {
br.write(str + System.getProperty("line.separator"));
}
} catch (Exception e) {
System.out.println("ERROR WHILE UPDATING DATA INTO File");
e.printStackTrace();
}
}
}
} catch (Exception e) {
} finally {
try {
br.close();
br = new BufferedWriter(fr);
} catch (IOException e) {
e.printStackTrace();
}
}
Comments
Post a Comment