You can not select more than 25 topics Topics must start with a letter or number, can include dashes ('-') and can be up to 35 characters long.

README.md 2.0KB

1234567891011121314151617181920212223242526272829303132333435363738394041424344454647484950515253545556575859606162636465666768697071
  1. through2-concurrent
  2. ===================
  3. [![NPM](https://nodei.co/npm/through2-concurrent.png?downloads&downloadRank)](https://nodei.co/npm/through2-concurrent/)
  4. A simple way to create a Node.JS Transform stream which processes in
  5. parallel. You can limit the concurrency (default is 16) and order is
  6. *not* preserved (so chunks/objects can end up in a different order to
  7. the order they started in if the transform functions take different
  8. amounts of time).
  9. Built using [through2](https://github.com/rvagg/through2) and has the
  10. same API with the addition of a `maxConcurrency` option.
  11. Non-`objectMode` streams are supported for completeness but I'm not
  12. sure they'd be useful for anything.
  13. Written by Thomas Parslow
  14. ([almostobsolete.net](http://almostobsolete.net) and
  15. [tomparslow.co.uk](http://tomparslow.co.uk)) as part of Active Inbox
  16. ([activeinboxhq.com](http://activeinboxhq.com/)).
  17. [![Build Status](https://travis-ci.org/almost/through2-concurrent.svg)](https://travis-ci.org/almost/through2-concurrent)
  18. Install
  19. -------
  20. ```bash
  21. npm install --save through2-concurrent
  22. ```
  23. Examples
  24. --------
  25. Process lines from a CSV in paralel. The order the results end up in
  26. the `all` variable is not deterministic.
  27. ```javascript
  28. var through2Concurrent = require('through2-concurrent');
  29. var all = [];
  30. fs.createReadStream('data.csv')
  31. .pipe(csv2())
  32. .pipe(through2Concurrent.obj(
  33. {maxConcurrency: 10},
  34. function (chunk, enc, callback) {
  35. var self = this;
  36. someThingAsync(chunk, function (newChunk) {
  37. self.push(newChunk);
  38. callback();
  39. });
  40. }))
  41. .on('data', function (data) {
  42. all.push(data)
  43. })
  44. .on('end', function () {
  45. doSomethingSpecial(all)
  46. })
  47. ```
  48. Contributing
  49. ------------
  50. Fixed or improved stuff? Great! Send me a pull request [through GitHub](http://github.com/almost/through2-concurrent)
  51. or get in touch on Twitter [@almostobsolete][#tom-twitter] or email at tom@almostobsolete.net
  52. [#tom]: http://www.almostobsolete.net
  53. [#tom-twitter]: https://twitter.com/almostobsolete