Frequently Asked Questions

Data Pipeline makes it easy to add data conversion, processing, and transformation to Java applications.

The toolkit has readers and writers for physical files (CSV, Excel, Fixed-width), plus chainable secondary readers for processing and transforming data (filter, remove duplicates, lookups, validation).

Data Pipeline is actively being updated with new features to reflect our client's needs. If you have an idea for a new feature, please post a message in the discussion forums and we'll attempt to include it in a future release.

If your request is of an urgent nature, please email us, we would be happy to provide you a quote to enhance or customize Data Pipeline to your needs.

We host discussion forums where you can post questions for the communuity.  We will read and answer forum questions as time permits.

We also provide commercial support through email for an additional fee on an annual basis. Commercial support provides you with the most responsive support from our experts, it features:

• Priority treatment
• Dedicated Email Address from our Technical Support Team
• Less than 24 hour response time during business days

If you've purchased a Commercial Support Package from us, please send your questions to support@northconcepts.com. Please include the version # and as much detail about your issue as possible.

We provide a Commercial Support Package, which can be purchased on an annual basis, or you may raise questions for the community through our discussion forums.

Yes. We are actively updating Data Pipeline with new features to reflect our client's needs.

Data Pipeline has two licensing options:

Open Source (GPL)

Please see the Open Source License Agreement for complete details.

Commercial

Please see the Purchase page and the Commercial License Agreement for complete details.

If you are interested in an alternative licensing option (such as for OEM or educational purposes), please email us your needs.

The Open Source License should suffice in most such cases.  You can always purchase a Commercial Licence or email us your specific requirements.

Our Getting Started Guide and Examples will have you up and running quickly.

Data Pipeline may stall if DataReader.read() is called before DataReader.open().

Java's default, maximum memory size of 64 MB may not be adequate for your application.  Try using the -Xmx Java option to set a higher maximum.

Data Pipeline supports Sun Java Standard Edition SDK 1.5 or greater.

Yes. DebugReader can be layered onto another DataReader to log everything it reads.

Data Pipeline is distributed with all the necessary libraries (except for JDBC drivers) under the distributions lib folder.

Data can be cached in a RecordList in one of two ways:
1. Automatically, by writing to a MemoryWriter then calling getRecordList(). See the write to memory example.
2. Manually, by creating a RecordList and adding Records to it. See the read from memory example.

Yes. RecordLists can be searched by calling findAll(), findFirst(), or findLast() . Searches can also be performed by filtering incoming data.

Yes. FilteringReader provides a means of selecting records either programmatically or using the run-time expression language. See the filter records example.

Simply reading an MS Excel file with the evaluateExpressions property set to true (by default) will cause VLookups to be evaluated. See the read from an Excel file example.