com.northconcepts.datapipeline.csv

Class CSVReader

java.lang.Object

com.northconcepts.datapipeline.core.DataObject

com.northconcepts.datapipeline.core.Endpoint

com.northconcepts.datapipeline.core.DataEndpoint

com.northconcepts.datapipeline.core.DataReader

com.northconcepts.datapipeline.core.AbstractReader

com.northconcepts.datapipeline.core.TextReader

com.northconcepts.datapipeline.csv.CSVReader

public class CSVReader
extends TextReader

Obtains records from a Comma Separated Value (CSV) or delimited stream.

Nested Class Summary

Nested classes/interfaces inherited from class com.northconcepts.datapipeline.core.DataEndpoint

DataEndpoint.State

Field Summary

Fields inherited from class com.northconcepts.datapipeline.core.TextReader

EOF, reader

Fields inherited from class com.northconcepts.datapipeline.core.AbstractReader

currentRecord, fieldNames, lastRow, startingRow

Fields inherited from class com.northconcepts.datapipeline.core.DataReader

fieldLineage, recordLineage

Fields inherited from class com.northconcepts.datapipeline.core.DataEndpoint

lastRecord, PRODUCT, PRODUCT_VERSION, VENDOR, XML_INPUT_FACTORY_KEY

Fields inherited from class com.northconcepts.datapipeline.core.Endpoint

BUFFER_SIZE, captureElapsedTime, DEFAULT_READ_BUFFER_SIZE

Fields inherited from class com.northconcepts.datapipeline.core.DataObject

id, log, name, TIMESTAMP_FORMAT

Constructor Summary

Constructors

Constructor and Description
CSVReader(File file)
CSVReader(Reader reader)

Method Summary

Modifier and Type	Method and Description
DataException	addExceptionProperties(DataException exception) Adds this endpoint's current state to a DataException.
protected Record	addLineage(Record record)
protected boolean	fillRecord(Record record)
long	getColumnNumber()
String	getEndingQuote() Returns the string used to mark the ending of text values that may contain commas and other special characters (defaults to double quote - ").
String	getFieldSeparator()
String[]	getLineSeparators() Returns the strings used to separate records or null if none were explicitly assigned.
LineParser	getParser()
String	getQuoteChar() Returns the starting quote - same as calling getStartingQuote() (defaults to double quote - ").
String	getStartingQuote() Returns the string used to mark the beginning of text values that may contain commas and other special characters (defaults to double quote - ").
boolean	isAllowMultiLineText() Indicates if line separators are allowed inside quoted fields.
boolean	isAllowQuoteInField() Indicates if unescaped quotes are allowed in fields (default to false).
boolean	isLineageSupported()
boolean	isTrimFields() Indicates if space and tab characters at the beginning and ending of non-quoted-values should be removed.
static RecordList	parse(String text, String fieldSeparator, String quoteChar, Boolean allowMultiLineText) Utility method to quickly parse lines of CSV text.
static RecordList	parse(String text, String fieldSeparator, String quoteChar, Boolean allowMultiLineText, Boolean trimFields) Utility method to quickly parse lines of CSV text.
static Record	parseLine(String line, String fieldSeparator, String quoteChar, Boolean allowMultiLineText) Utility method to quickly parse a line of CSV text.
static Record	parseLine(String line, String fieldSeparator, String quoteChar, Boolean allowMultiLineText, Boolean trimFields) Utility method to quickly parse a line of CSV text.
CSVReader	setAllowMultiLineText(boolean allowMultiLineText) Indicates if line separators are allowed inside quoted fields.
CSVReader	setAllowQuoteInField(boolean allowQuoteInField) Indicates if unescaped quotes are allowed in fields (default to false).
CSVReader	setEndingQuote(String endingQuote) Assigns the string used to mark the ending of text values that may contain commas and other special characters (defaults to double quote - ").
CSVReader	setFieldNames(Collection<String> fieldNames)
CSVReader	setFieldNames(String... fieldNames)
CSVReader	setFieldNamesInFirstRow(boolean fieldNamesInFirstRow)
CSVReader	setFieldSeparator(char columnSeparator)
CSVReader	setFieldSeparator(String columnSeparator)
CSVReader	setLastRow(int lastRow)
CSVReader	setLineSeparator(String lineSeparator) Indicates the string used to separate records.
CSVReader	setLineSeparators(String... lineSeparators) Indicates the string or strings used to separate records.
CSVReader	setQuoteChar(char quoteChar)
CSVReader	setQuoteChar(String quoteChar) Assigns startingQuote and endingQuote to the same value (defaults to double quote - ").
CSVReader	setSaveLineage(boolean saveLineage)
CSVReader	setSkipEmptyRows(boolean skipEmptyRows) Indicates that rows with only null values should not be returned by the reader (default is false).
CSVReader	setStartingQuote(String startingQuote) Assigns the string used to mark the beginning of text values that may contain commas and other special characters (defaults to double quote - ").
CSVReader	setStartingRow(int startingRow)
CSVReader	setTrimFields(boolean trimFields) Indicates if space and tab characters at the beginning and ending of non-quoted-values should be removed.

Methods inherited from class com.northconcepts.datapipeline.core.TextReader

available, close, getFile, getLineNumber, open, readImpl

Methods inherited from class com.northconcepts.datapipeline.core.AbstractReader

getFieldNames, getLastRow, getStartingRow, isFieldNamesInFirstRow, isSkipEmptyRows, read

Methods inherited from class com.northconcepts.datapipeline.core.DataReader

getBufferSize, getNestedEndpoint, getNestedReader, getRootEndpoint, getRootReader, isExhausted, isSaveLineage, peek, pop, push, skip

Methods inherited from class com.northconcepts.datapipeline.core.DataEndpoint

decrementRecordCount, enableJmx, getLastRecord, getRecordCount, getRecordCountAsBigInteger, getRecordCountAsString, incrementRecordCount, isRecordCountBigInteger, resetRecordCount, toString

Methods inherited from class com.northconcepts.datapipeline.core.Endpoint

addElapsedtime, assertClosed, assertNotOpened, assertOpened, finalize, getClosedOn, getDescription, getElapsedTime, getElapsedTimeAsString, getOpenedOn, getOpenElapsedTime, getOpenElapsedTimeAsString, getSelfTime, getSelfTimeAsString, getState, isCaptureElapsedTime, isClosed, isOpen, setCaptureElapsedTime, setDescription

Methods inherited from class com.northconcepts.datapipeline.core.DataObject

exception, exception, exception, getId, getName, resetID

Methods inherited from class java.lang.Object

clone, equals, getClass, hashCode, notify, notifyAll, wait, wait, wait

Constructor Detail

CSVReader

public CSVReader(File file)

CSVReader

public CSVReader(Reader reader)

Method Detail

parseLine

public static Record parseLine(String line,
                               String fieldSeparator,
                               String quoteChar,
                               Boolean allowMultiLineText)

Utility method to quickly parse a line of CSV text.

Parameters:

line - the delimited line of text to parse

fieldSeparator -

quoteChar -

allowMultiLineText -

Returns:

parseLine

public static Record parseLine(String line,
                               String fieldSeparator,
                               String quoteChar,
                               Boolean allowMultiLineText,
                               Boolean trimFields)

Utility method to quickly parse a line of CSV text.

Parameters:

line -

fieldSeparator -

quoteChar -

allowMultiLineText -

trimFields -

Returns:

parse

public static RecordList parse(String text,
                               String fieldSeparator,
                               String quoteChar,
                               Boolean allowMultiLineText)

Utility method to quickly parse lines of CSV text.

Parameters:

text - the delimited lines of text to parse

fieldSeparator -

quoteChar -

allowMultiLineText -

Returns:

parse

public static RecordList parse(String text,
                               String fieldSeparator,
                               String quoteChar,
                               Boolean allowMultiLineText,
                               Boolean trimFields)

Utility method to quickly parse lines of CSV text.

Parameters:

text -

fieldSeparator -

quoteChar -

allowMultiLineText -

trimFields -

Returns:

getFieldSeparator

public String getFieldSeparator()

setFieldSeparator

public CSVReader setFieldSeparator(char columnSeparator)

setFieldSeparator

public CSVReader setFieldSeparator(String columnSeparator)

getQuoteChar

public String getQuoteChar()

Returns the starting quote - same as calling getStartingQuote() (defaults to double quote - "). getStartingQuote()

setQuoteChar

public CSVReader setQuoteChar(char quoteChar)

setQuoteChar

public CSVReader setQuoteChar(String quoteChar)

Assigns startingQuote and endingQuote to the same value (defaults to double quote - "). setStartingQuote(String) setEndingQuote(String)

getStartingQuote

public String getStartingQuote()

Returns the string used to mark the beginning of text values that may contain commas and other special characters (defaults to double quote - ").

setStartingQuote

public CSVReader setStartingQuote(String startingQuote)

Assigns the string used to mark the beginning of text values that may contain commas and other special characters (defaults to double quote - ").

getEndingQuote

public String getEndingQuote()

Returns the string used to mark the ending of text values that may contain commas and other special characters (defaults to double quote - ").

setEndingQuote

public CSVReader setEndingQuote(String endingQuote)

Assigns the string used to mark the ending of text values that may contain commas and other special characters (defaults to double quote - ").

getLineSeparators

public String[] getLineSeparators()

Returns the strings used to separate records or null if none were explicitly assigned.

setLineSeparators

public CSVReader setLineSeparators(String... lineSeparators)

Indicates the string or strings used to separate records.

setLineSeparator

public CSVReader setLineSeparator(String lineSeparator)

Indicates the string used to separate records.

isAllowMultiLineText

public boolean isAllowMultiLineText()

Indicates if line separators are allowed inside quoted fields.

setAllowMultiLineText

public CSVReader setAllowMultiLineText(boolean allowMultiLineText)

Indicates if line separators are allowed inside quoted fields.

isAllowQuoteInField

public boolean isAllowQuoteInField()

Indicates if unescaped quotes are allowed in fields (default to false).

setAllowQuoteInField

public CSVReader setAllowQuoteInField(boolean allowQuoteInField)

Indicates if unescaped quotes are allowed in fields (default to false).

isTrimFields

public boolean isTrimFields()

Indicates if space and tab characters at the beginning and ending of non-quoted-values should be removed. If line separators are explicitly set on this reader then any non-line-separator char matching Character.isWhitespace(int) will be removed (not only spaces and tabs). This is helpful when CSV files may contain spaces between the commas (field separators) and values.

Whitespace characters used as field or line separators will not be removed.

The default is true.

setTrimFields

public CSVReader setTrimFields(boolean trimFields)

Indicates if space and tab characters at the beginning and ending of non-quoted-values should be removed. If line separators are explicitly set on this reader then any non-line-separator char matching Character.isWhitespace(int) will be removed (not only spaces and tabs). This is helpful when CSV files may contain spaces between the commas (field separators) and values.

Whitespace characters used as field or line separators will not be removed.

The default is true.

getParser

public LineParser getParser()

getColumnNumber

public long getColumnNumber()

isLineageSupported

public boolean isLineageSupported()

Overrides:

isLineageSupported in class DataReader

addLineage

protected Record addLineage(Record record)

Overrides:

addLineage in class TextReader

addExceptionProperties

public DataException addExceptionProperties(DataException exception)

Description copied from class: Endpoint

Adds this endpoint's current state to a DataException. Since this method is called whenever an exception is thrown, subclasses should override it to add their specific information.

Overrides:

addExceptionProperties in class TextReader

fillRecord

protected boolean fillRecord(Record record)
                      throws Throwable

Specified by:

fillRecord in class AbstractReader

Throws:

Throwable

setSkipEmptyRows

public CSVReader setSkipEmptyRows(boolean skipEmptyRows)

Description copied from class: AbstractReader

Indicates that rows with only null values should not be returned by the reader (default is false).

Overrides:

setSkipEmptyRows in class AbstractReader

setFieldNames

public CSVReader setFieldNames(String... fieldNames)

Overrides:

setFieldNames in class AbstractReader

setFieldNames

public CSVReader setFieldNames(Collection<String> fieldNames)

Overrides:

setFieldNames in class AbstractReader

setFieldNamesInFirstRow

public CSVReader setFieldNamesInFirstRow(boolean fieldNamesInFirstRow)

Overrides:

setFieldNamesInFirstRow in class AbstractReader

setStartingRow

public CSVReader setStartingRow(int startingRow)

Overrides:

setStartingRow in class AbstractReader

setLastRow

public CSVReader setLastRow(int lastRow)

Overrides:

setLastRow in class AbstractReader

setSaveLineage

public CSVReader setSaveLineage(boolean saveLineage)

Overrides:

setSaveLineage in class DataReader