Overview

Helper classes for the AWS v3 SDK for JS Kinesis Client (@aws-sdk/client-kinesis).

KinesisRetrier will retry failed records with a "successful" PutRecordsCommand that has items marked as failed within the response, which then need to be retried.

KinesisBackgroundWriter will accept writes immediately if there is a concurrent writer slot available, and will block the caller until a slot becomes available if all slots are in use.

Getting Started

Installation

The package is available on npm as @shutterstock/kinesis-helpers

npm i @shutterstock/kinesis-helpers

Importing

import { KinesisBackgroundWriter, KinesisRetrier } from '@shutterstock/kinesis-helpers';
Copy

API Documentation

After installing the package, you might want to look at our API Documentation to learn about all the features available.

Features

• KinesisRetrier
  • Kinesis batch put retrier
  • Batch puts return a list of items that were throttled
  • The batch is marked as succeeded (200 status code) even though items failed
  • As a result, the built-in AWS SDK retry logic will not retry the items in the batch that were throttle
• KinesisBackgroundWriter
  • Accepts writes immediately if there is a concurrent writer slot available
  • Blocks the caller until a slot becomes available if all slots are in use
  • Collects and exposes errors so the caller can detect if errors have been happening

Contributing

Setting up Build Environment

• nvm use
• npm i
• npm run build
• npm run lint
• npm run test

Running Examples

kinesis-retrier

1. Create Kinesis Data Stream using AWS Console or any other method
   1. Example: aws kinesis create-stream --stream-name kinesis-helpers-test-stream --shard-count 1
   2. Default name is kinesis-helpers-test-stream
   3. 1 shard is sufficient
   4. 1 day retention is sufficient
   5. No encryption is sufficient
   6. On-demand throughput is sufficient
2. npm run example:kinesis-retrier
   1. If the stream name was changed: KINESIS_STREAM_NAME=my-stream-name npm run example:kinesis-retrier

kinesis-background-writer

1. Create Kinesis Data Stream using AWS Console or any other method
   1. Example: aws kinesis create-stream --stream-name kinesis-helpers-test-stream --shard-count 1
   2. Default name is kinesis-helpers-test-stream
   3. 1 shard is sufficient
   4. 1 day retention is sufficient
   5. No encryption is sufficient
   6. On-demand throughput is sufficient
2. npm run example:kinesis-background-writer
   1. If the stream name was changed: KINESIS_STREAM_NAME=my-stream-name npm run example:kinesis-background-writer