Skip to content

Instantly share code, notes, and snippets.

@JosePedroDias
Created September 15, 2023 17:12
Show Gist options
  • Save JosePedroDias/dee96e28f04597ea576591b772a6cfd3 to your computer and use it in GitHub Desktop.
Save JosePedroDias/dee96e28f04597ea576591b772a6cfd3 to your computer and use it in GitHub Desktop.
streams a non-binary file avoiding loading it all to memory
import { createReadStream } from 'fs';
export function streamSplitBy(file, onMatch, splitBy='\n') {
return new Promise((resolve) => {
const stream = createReadStream(file);
let ongoing = '';
stream.on('data', buff => {
const chunk = buff.toString();
let chunkStartIndex = 0;
let chunkEndIndex = 0;
while (true) {
chunkEndIndex = chunk.indexOf(splitBy, chunkStartIndex);
if (chunkEndIndex !== -1) {
ongoing += chunk.substring(chunkStartIndex, chunkEndIndex);
onMatch(ongoing);
ongoing = '';
chunkStartIndex = chunkEndIndex + splitBy.length;
} else {
ongoing += chunk.substring(chunkStartIndex);
break;
}
}
});
stream.on('end', () => {
if (ongoing !== '') onMatch(ongoing);
resolve();
});
});
}
export function streamLines(file, onMatch) {
return streamSplitBy(file, onMatch, '\n');
}
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment