Donate. I desperately need donations to survive due to my health

Get paid by answering surveys Click here

Click here to donate

Remote/Work from Home jobs

Efficient way of processing data using Java

i am working on a POC where i need to do the following

i am using java 8 for below processing

  1. read data from a text file(comma separated data) and push it to ActiveMQ one by one. each row is of ~500 characters in length and like that there are 100K records.

  2. same time i need to consume those records and push it to a file in LOCAL file System (LFS).

i am able to achieve it using java but i do not see the performance is satisfactory. kindly help me with your suggestions to boost its code quality and performance

  1. i have also tried to use Java Stream from java 8, but i dint get how to get each row out of stream object and push it to producer.

  2. how should i handle logic if there is no data to be fetch from queue, i need to trigger a job. how should i handle this scenario.

code for point 1:

String fileString = new String(Files.readAllBytes(Paths.get(INPUTFILE)), StandardCharsets.UTF_8);
        String[] Line = fileString.split("\n");
        for (int i = 0; i < Line.length; i++) {
            TextMessage textMessage = session.createTextMessage(Line[i].toString().trim());
            producer.send(textMessage);
        //System.out.println(textMessage.getText());
    }

code for Point 2:

   File fileName = new File(sHdfsLandOutPath);
        FileWriter fr = null;
        BufferedWriter br = null;
        fr = new FileWriter(fileName);
        br = new BufferedWriter(fr);
        try {
            while (true) {
                Message message = consumer.receive();
                if (message instanceof TextMessage) {
                    TextMessage textMessage = (TextMessage) message;
                    String str = textMessage.getText();
                    sMsgCounter++;
                    sTotalRecCount++;
                try {
                        if (sMsgCounter == sHdfsPushCount) {
                            System.out.println("Pushing Data in bulk to local file");
                            System.out.println(sTotalRecCount);
                            br.write(str );
                            br.newLine();
                            br.flush();
                            sMsgCounter = 0;
                        } else {
                            br.write(str + System.getProperty("line.separator"));
                        }

                    } catch (Exception e) {
                        System.out.println("ERROR WHILE UPDATING DATA INTO File");
                        e.printStackTrace();
                    } 
                }
            }
        } catch (Exception e) {

        } finally {
            try {
                br.close();
                br = new BufferedWriter(fr);
            } catch (IOException e) {
                e.printStackTrace();
            }
        }

Comments