New Flat File Streaming APIs Support Files over 10MB

With NetSuite 2017.1, you can use new file streaming APIs to more efficiently process and stream large CSV and plain text files.

You can load and edit each line into memory using File.lines.iterator(). Call File.appendLine(options) to append the lines back together. Note that you cannot iterate on and append lines at the same time. The File.resetStream() method resets the reading and writing streams that may have been opened. To help you monitor the size of a file, the File.size property now holds a dynamic value. The value is a sum of the saved file size plus any updated or appended lines currently in memory.

For an example demonstrating use of these new APIs, see the following code sample:


require(['N/file', '
N/error', '
N/log'],-
function (file, error, log)
{
-
//
In this sample we will compute the total for the
//
second column value in a csv file.-
//
//
date,amount
//
10/21/14,200.0
//
10/21/15,210.2
//
10/21/16,250.3
-
//
Create the CSV file
var csvFile = file.create({
name: '
data.csv', -
contents: '
date,amount\n', -
folder: 39, -
fileType: '
CSV'
});
csvFile.appendLine({
value: '
10/21/14,200.0'
});
csvFile.appendLine({
value: '
10/21/15,210.2'
});
csvFile.appendLine({
value: '
10/21/16,250.3'
});
var csvFileId = csvFile.save();
-
//
This variable will store the total.-
var total = 0.0;
-
//
Load the file and
//
process all the lines

var invoiceFile = file.load({
id: csvFileId
});
var iterator = invoiceFile.lines.iterator();
–
//
Skip the first line (CSV header)
iterator.each(function () {return false;});
iterator.each(function (line)
{
//
This function updates the total by
//
adding the amount on each line to it
var lineValues = line.value.split(‘,’);
var lineAmount = parseFloat(lineValues[1]);
if (!lineAmount)
throw error.create({
name: ‘
INVALID_INVOICE_FILE’,-
message: ‘
Invoice file contained non-numeric value for total: ‘
+ lineValue
s[1]
});
–
total += lineAmount;
return true;
});
–
//
By the time you are here, the total variable is
//
set to 660.5
log.debug({
title: ‘
total’, –
details; total
});
}
)

Although the File.getContents() method continues to support only files under 10MB, the file streaming APIs enforce the 10MB limit only on individual lines of content. You no longer need to partition your files into smaller, separate files to read, write, and append file contents in memory.

Note: File streaming APIs are designed for CSV and plain text files. They are not suitable for binary files or structured, hierarchal data (such as JSON or XML files).