Processing CSV files with Processing

I needed to proces some big (>1000000 rows) .csv files and change some data in one column based on time stored in another column. The first thought was to do this with MySQL/phpMyAdmin however importing large files like this takes time and you have to modify some PHP time-out settings.

Since Processing 2.0 working with .csv files is largely improved with the Table command.

Below the sketch that I’ve used to alter the table with a if/else condition. Pretty straightforward and fast (few seconds). Since it only runs one time there is no need to use setup() and draw() functions.

// Example to process CSV data
// Table syntax in Processing reference: <a href="http://www.processing.org/reference/Table.html" target="_blank">http://www.processing.org/reference/Table.html</a>
// copyleft: kasperkamperman.com 17-05-2014

String filename = "13-42-14_rawEEGData";

// variables used in if-function to modify a column based on time
int startWaitingTime = 32000;
int stopWaitingTime  = 610000;

Table table;

table = loadTable(filename+".csv");
println("loaded " + filename + ".csv");
println(table.getRowCount() + " total rows in table"); 

int isWaiting = 1;
int isNotWaiting = 0;

int timeInMillis; // timeInMillis from column
int rowId;        // rowId from column

for (TableRow row : table.rows()) 
{
    //set the isWaiting flag (0,1) b
    //.getInt(row, column)
    timeInMillis = row.getInt(5);
    rowId        = row.getInt(0);
    
    //set the waiting flag based on timeInMillis
    if(timeInMillis>startWaitingTime && timeInMillis<stopWaitingTime) 
    { //.setInt(row, column, value)
      table.setInt(rowId , 8, isWaiting);
    }
    else
    { table.setInt(rowId , 8, isNotWaiting);
    }
}


saveTable(table, "data/"+filename+"_new.csv");
println("saved "+filename+"_new.csv");
println(table.getRowCount() + " total rows in table"); 
  
println("done");