Create a new IterableQueueMapper
Function which is called for every item in input.
Expected to return a Promise or value.
The mapper should handle all errors and not allow an error to be thrown
out of the mapper function as this enables the best handling of errors
closest to the time that they occur.
If the mapper function does allow an error to be thrown then the
stopOnMapperError option controls the behavior:
- stopOnMapperError: true - will throw the error
out of next or the AsyncIterator returned from [Symbol.asyncIterator]
and stop processing.
- stopOnMapperError: false - will continue processing
and accumulate the errors to be thrown from next or the AsyncIterator
returned from [Symbol.asyncIterator] when all items have been processed.
IterableQueueMapper options
Private _iterablePrivate _sourceGenerated using TypeDoc
Accepts queue items via
enqueueand calls themapperon them with specified concurrency, storing themapperresult in a queue of specified max size, before being iterated / read by the caller. Theenqueuemethod will block if the queue is full, until an item is read.Remarks
This allows performing a concurrent mapping with back pressure for items added after queue creation via a method call.
Because items are added via a method call it is possible to chain an
IterableMapperthat prefetches files and processes them, with anIterableQueueMapperthat processes the results of themapperfunction of theIterableMapper.Typical use case is for a
background uploaderthat prevents the producer from racing ahead of the upload process, consuming too much memory or disk space. As items are ready for upload they are added to the queue with theenqueuemethod, which isawaited by the caller. If the queue has room thenenqueuewill return immediately, otherwise it will block until there is room.