Interface IterableMapperOptions

Maximum number of concurrent invocations of mapper to run at once.

The number of concurrent invocations is dynamically adjusted based on the maxUnread limit:

• If there are no unread items and maxUnread is 10 with concurrency of 4, all 4 mappers can run.
• If there are already 8 unread items in the queue, only 2 mappers will run to avoid exceeding the maxUnread limit of 10.
• If there are 10 unread items, no mappers will run until an item is consumed from the queue.

This ensures efficient processing while maintaining backpressure through the maxUnread limit.

Setting concurrency to 1 enables serial processing, preserving the order of items while still benefiting from the backpressure mechanism.

Must be an integer from 1 and up or Infinity, and must be <= maxUnread.

Default

4
Copy

Maximum number of unread items allowed to accumulate before applying backpressure.

This parameter is crucial for controlling memory usage and system load by:

1. Limiting the number of processed but unread items in the queue
2. Automatically pausing mapper execution when the limit is reached
3. Resuming processing when items are consumed, maintaining optimal throughput

For example, when reading from a slow database:

• With maxUnread=10, only 10 items will be fetched before the consumer reads them
• Additional items won't be fetched until the consumer reads existing items
• This prevents runaway memory usage for items that cannot be processed quickly enough

Must be an integer from 1 and up or Infinity, and must be >= concurrency. It is not typical to set this value to Infinity, but rather to a value such as 1 to 10.

Default

8
Copy

When set to false, instead of stopping when a promise rejects, it will wait for all the promises to settle and then reject with an aggregated error containing all the errors from the rejected promises.

Default

true
Copy