I have a machine with proprietary program creating CSV files each of the file have 1200-1300 rows, unfortunately, I have no control over this program, I have an SPC program reading the CSV file but the only data I need to extract is one row that can be anywhere in several rows of the file. Right now I read them row by row looking for data in column number 10 if there is data then I process the whole row.
Some notes: There are 13 columns in CSV file only one row I need is using all of them besides the header row. the other rows use columns 1,2,3,4,13 I chose column 10 do to it contains the important data of the whole file.
Some files don't have any data in that case the file is moved and archived.
if there was a way to extract just the one row of data I need and put in another CSV file that would be ok, as long as the file name of the original CSV file is used or appended to the last item of the CSV file as it contains the date code.
My thought is there must be a faster way to process these files rather than line by line. It takes 60 to 70 sec to process one file the way I am now, that include other operations the program is doing like open FTP to the machine and getting the file or files, processing the files, archiving files and moving to the next operation.
The server is running a virtual Windows Server 2003, basically being a data stuffer for the SPC data that calls Windows batch files to move files and create txt file of each file name so my SPC software can read the file name and open the CSV file.
How else would someone solve this problem to trim all the unneeded rows from the CSV file?
Comments
Post a Comment