|
123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960616263646566676869707172737475767778798081828384858687888990919293949596979899100101102103104105106107108109110111112113114115116117118119120121122123124125126127128129130131132133134135136137138139140141142143144145146147148149150151152153154155156157158159160161162163164165166167168169170171172173174175176177178179180181182183184185186187188189190191192193194195196197198199 |
- # fd-slicer
-
- [![Build Status](https://travis-ci.org/andrewrk/node-fd-slicer.svg?branch=master)](https://travis-ci.org/andrewrk/node-fd-slicer)
-
- Safe `fs.ReadStream` and `fs.WriteStream` using the same fd.
-
- Let's say that you want to perform a parallel upload of a file to a remote
- server. To do this, we want to create multiple read streams. The first thing
- you might think of is to use the `{start: 0, end: 0}` API of
- `fs.createReadStream`. This gives you two choices:
-
- 0. Use the same file descriptor for all `fs.ReadStream` objects.
- 0. Open the file multiple times, resulting in a separate file descriptor
- for each read stream.
-
- Neither of these are acceptable options. The first one is a severe bug,
- because the API docs for `fs.write` state:
-
- > Note that it is unsafe to use `fs.write` multiple times on the same file
- > without waiting for the callback. For this scenario, `fs.createWriteStream`
- > is strongly recommended.
-
- `fs.createWriteStream` will solve the problem if you only create one of them
- for the file descriptor, but it will exhibit this unsafety if you create
- multiple write streams per file descriptor.
-
- The second option suffers from a race condition. For each additional time the
- file is opened after the first, it is possible that the file is modified. So
- in our parallel uploading example, we might upload a corrupt file that never
- existed on the client's computer.
-
- This module solves this problem by providing `createReadStream` and
- `createWriteStream` that operate on a shared file descriptor and provides
- the convenient stream API while still allowing slicing and dicing.
-
- This module also gives you some additional power that the builtin
- `fs.createWriteStream` do not give you. These features are:
-
- * Emitting a 'progress' event on write.
- * Ability to set a maximum size and emit an error if this size is exceeded.
- * Ability to create an `FdSlicer` instance from a `Buffer`. This enables you
- to provide API for handling files as well as buffers using the same API.
-
- ## Usage
-
- ```js
- var fdSlicer = require('fd-slicer');
- var fs = require('fs');
-
- fs.open("file.txt", 'r', function(err, fd) {
- if (err) throw err;
- var slicer = fdSlicer.createFromFd(fd);
- var firstPart = slicer.createReadStream({start: 0, end: 100});
- var secondPart = slicer.createReadStream({start: 100});
- var firstOut = fs.createWriteStream("first.txt");
- var secondOut = fs.createWriteStream("second.txt");
- firstPart.pipe(firstOut);
- secondPart.pipe(secondOut);
- });
- ```
-
- You can also create from a buffer:
-
- ```js
- var fdSlicer = require('fd-slicer');
- var slicer = FdSlicer.createFromBuffer(someBuffer);
- var firstPart = slicer.createReadStream({start: 0, end: 100});
- var secondPart = slicer.createReadStream({start: 100});
- var firstOut = fs.createWriteStream("first.txt");
- var secondOut = fs.createWriteStream("second.txt");
- firstPart.pipe(firstOut);
- secondPart.pipe(secondOut);
- ```
-
- ## API Documentation
-
- ### fdSlicer.createFromFd(fd, [options])
-
- ```js
- var fdSlicer = require('fd-slicer');
- fs.open("file.txt", 'r', function(err, fd) {
- if (err) throw err;
- var slicer = fdSlicer.createFromFd(fd);
- // ...
- });
- ```
-
- Make sure `fd` is a properly initialized file descriptor. If you want to
- use `createReadStream` make sure you open it for reading and if you want
- to use `createWriteStream` make sure you open it for writing.
-
- `options` is an optional object which can contain:
-
- * `autoClose` - if set to `true`, the file descriptor will be automatically
- closed once the last stream that references it is closed. Defaults to
- `false`. `ref()` and `unref()` can be used to increase or decrease the
- reference count, respectively.
-
- ### fdSlicer.createFromBuffer(buffer, [options])
-
- ```js
- var fdSlicer = require('fd-slicer');
- var slicer = fdSlicer.createFromBuffer(someBuffer);
- // ...
- ```
-
- `options` is an optional object which can contain:
-
- * `maxChunkSize` - A `Number` of bytes. see `createReadStream()`.
- If falsey, defaults to unlimited.
-
- #### Properties
-
- ##### fd
-
- The file descriptor passed in. `undefined` if created from a buffer.
-
- #### Methods
-
- ##### createReadStream(options)
-
- Available `options`:
-
- * `start` - Number. The offset into the file to start reading from. Defaults
- to 0.
- * `end` - Number. Exclusive upper bound offset into the file to stop reading
- from.
- * `highWaterMark` - Number. The maximum number of bytes to store in the
- internal buffer before ceasing to read from the underlying resource.
- Defaults to 16 KB.
- * `encoding` - String. If specified, then buffers will be decoded to strings
- using the specified encoding. Defaults to `null`.
-
- The ReadableStream that this returns has these additional methods:
-
- * `destroy(err)` - stop streaming. `err` is optional and is the error that
- will be emitted in order to cause the streaming to stop. Defaults to
- `new Error("stream destroyed")`.
-
- If `maxChunkSize` was specified (see `createFromBuffer()`), the read stream
- will provide chunks of at most that size. Normally, the read stream provides
- the entire range requested in a single chunk, but this can cause performance
- problems in some circumstances.
- See [thejoshwolfe/yauzl#87](https://github.com/thejoshwolfe/yauzl/issues/87).
-
- ##### createWriteStream(options)
-
- Available `options`:
-
- * `start` - Number. The offset into the file to start writing to. Defaults to
- 0.
- * `end` - Number. Exclusive upper bound offset into the file. If this offset
- is reached, the write stream will emit an 'error' event and stop functioning.
- In this situation, `err.code === 'ETOOBIG'`. Defaults to `Infinity`.
- * `highWaterMark` - Number. Buffer level when `write()` starts returning
- false. Defaults to 16KB.
- * `decodeStrings` - Boolean. Whether or not to decode strings into Buffers
- before passing them to` _write()`. Defaults to `true`.
-
- The WritableStream that this returns has these additional methods:
-
- * `destroy()` - stop streaming
-
- And these additional properties:
-
- * `bytesWritten` - number of bytes written to the stream
-
- And these additional events:
-
- * 'progress' - emitted when `bytesWritten` changes.
-
- ##### read(buffer, offset, length, position, callback)
-
- Equivalent to `fs.read`, but with concurrency protection.
- `callback` must be defined.
-
- ##### write(buffer, offset, length, position, callback)
-
- Equivalent to `fs.write`, but with concurrency protection.
- `callback` must be defined.
-
- ##### ref()
-
- Increase the `autoClose` reference count by 1.
-
- ##### unref()
-
- Decrease the `autoClose` reference count by 1.
-
- #### Events
-
- ##### 'error'
-
- Emitted if `fs.close` returns an error when auto closing.
-
- ##### 'close'
-
- Emitted when fd-slicer closes the file descriptor due to `autoClose`. Never
- emitted if created from a buffer.
|