WARNING: THIS SITE IS A MIRROR OF GITHUB.COM / IT CANNOT LOGIN OR REGISTER ACCOUNTS / THE CONTENTS ARE PROVIDED AS-IS / THIS SITE ASSUMES NO RESPONSIBILITY FOR ANY DISPLAYED CONTENT OR LINKS / IF YOU FOUND SOMETHING MAY NOT GOOD FOR EVERYONE, CONTACT ADMIN AT ilovescratch@foxmail.com
Skip to content
Open
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
96 changes: 49 additions & 47 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -33,56 +33,58 @@ Nextract is a [Extract Transform Load (ETL)](https://en.wikipedia.org/wiki/Extra

## Example Transform

/**
* Example: JSON input and sort...
*/
const path = require('path'),
Nextract = require(path.resolve(__dirname, '../nextract'));

//Define our input and output files
const sampleEmployeesInputFilePath = path.resolve(process.cwd(), 'data/employees.json'),
sampleEmployeesOutputFilePath = path.resolve(process.cwd(), 'data/employees_output.json');

//Tranforms always start with instance of the Nextract base class and a tranform name
const transform = new Nextract('jsonAndSort');
//We load the core plugin and then an additional plugins our transform requires
transform.loadPlugins('Core', ['Input', 'Output', 'Sort', 'Logger'])
.then(() => {
return new Promise((resolve) => {
//STEP 1: Read data in from a JSON file (we specify the object path we care about)
transform.Plugins.Core.Input.readJsonFile(sampleEmployeesInputFilePath, 'data.employees.*')
//STEP 2: Pass data in to be sorted (1 element is pushed back and it is the expected input
//for a new stream read call to sortOut)
.pipe(transform.Plugins.Core.Sort.sortIn(['last_name'], ['asc']))
.on('data', (sortInDbInfo) => {
if (sortInDbInfo !== undefined) {
resolve(sortInDbInfo);
}
});
```javascript
/**
* Example: JSON input and sort...
*/

const path = require('path'),
Nextract = require(path.resolve(__dirname, '../nextract'));

//Define our input and output files
const sampleEmployeesInputFilePath = path.resolve(process.cwd(), 'data/employees.json'),
sampleEmployeesOutputFilePath = path.resolve(process.cwd(), 'data/employees_output.json');

//Tranforms always start with instance of the Nextract base class and a tranform name
const transform = new Nextract('jsonAndSort');

//We load the core plugin and then an additional plugins our transform requires
transform.loadPlugins('Core', ['Input', 'Output', 'Sort', 'Logger'])
.then(() => {
return new Promise((resolve) => {
//STEP 1: Read data in from a JSON file (we specify the object path we care about)
transform.Plugins.Core.Input.readJsonFile(sampleEmployeesInputFilePath, 'data.employees.*')
//STEP 2: Pass data in to be sorted (1 element is pushed back and it is the expected input
//for a new stream read call to sortOut)
.pipe(transform.Plugins.Core.Sort.sortIn(['last_name'], ['asc']))
.on('data', (sortInDbInfo) => {
if (sortInDbInfo !== undefined) {
resolve(sortInDbInfo);
}
});
});
})
.then((sortInDbInfo) => {
transform.Plugins.Core.Sort.sortOut(sortInDbInfo)
//STEP 3: We want to write the sorted data back out to a new JSON file so first we use
//toJsonString to stringify the stream.
.pipe(transform.Plugins.Core.Output.toJsonString(true))
//STEP 4: Write out the new file
.pipe(transform.Plugins.Core.Output.toFile(sampleEmployeesOutputFilePath))
.on('finish', () => {
//Just logging some information back to the console
transform.Plugins.Core.Logger.info('Transform finished!');
transform.Plugins.Core.Logger.info(sampleEmployeesOutputFilePath, 'has been written');
})
.then((sortInDbInfo) => {
transform.Plugins.Core.Sort.sortOut(sortInDbInfo)
//STEP 3: We want to write the sorted data back out to a new JSON file so first we use
//toJsonString to stringify the stream.
.pipe(transform.Plugins.Core.Output.toJsonString(true))
//STEP 4: Write out the new file
.pipe(transform.Plugins.Core.Output.toFile(sampleEmployeesOutputFilePath))
.on('finish', () => {
//Just logging some information back to the console
transform.Plugins.Core.Logger.info('Transform finished!');
transform.Plugins.Core.Logger.info(sampleEmployeesOutputFilePath, 'has been written');
})
.on('end', () => {
transform.Plugins.Core.Logger.info('Transform ended!');
process.exit();
});
})
.catch((err) => {
transform.Plugins.Core.Logger.error('Transform failed: ', err);
.on('end', () => {
transform.Plugins.Core.Logger.info('Transform ended!');
process.exit();
});
})
.catch((err) => {
transform.Plugins.Core.Logger.error('Transform failed: ', err);
});
```

## Development
The source code for this project lives under the `src` directory. Grunt is used to build the project and generate API docs when the source is updated. When developing, simply run `grunt watch` from the project's root directory. As the source is updated Grunt will automatically generate new builds in the `build` directory.
Expand Down