@shutterstock/kinesis-helpers

npm (scoped) License: MIT API Docs Build - CI Package and Publish Publish Docs

Overview

Helper classes for the AWS v3 SDK for JS Kinesis Client (@aws-sdk/client-kinesis).

KinesisRetrier will retry failed records with a "successful" PutRecordsCommand that has items marked as failed within the response, which then need to be retried.

KinesisBackgroundWriter will accept writes immediately if there is a concurrent writer slot available, and will block the caller until a slot becomes available if all slots are in use.

Getting Started

Installation

The package is available on npm as @shutterstock/kinesis-helpers

npm i @shutterstock/kinesis-helpers

Importing

import { KinesisBackgroundWriter, KinesisRetrier } from '@shutterstock/kinesis-helpers';

API Documentation

After installing the package, you might want to look at our API Documentation to learn about all the features available.

Features

  • KinesisRetrier
    • Kinesis batch put retrier
    • Batch puts return a list of items that were throttled
    • The batch is marked as succeeded (200 status code) even though items failed
    • As a result, the built-in AWS SDK retry logic will not retry the items in the batch that were throttle
  • KinesisBackgroundWriter
    • Accepts writes immediately if there is a concurrent writer slot available
    • Blocks the caller until a slot becomes available if all slots are in use
    • Collects and exposes errors so the caller can detect if errors have been happening

Contributing

Setting up Build Environment

  • nvm use
  • npm i
  • npm run build
  • npm run lint
  • npm run test

Running Examples

kinesis-retrier

  1. Create Kinesis Data Stream using AWS Console or any other method
    1. Example: aws kinesis create-stream --stream-name kinesis-helpers-test-stream --shard-count 1
    2. Default name is kinesis-helpers-test-stream
    3. 1 shard is sufficient
    4. 1 day retention is sufficient
    5. No encryption is sufficient
    6. On-demand throughput is sufficient
  2. npm run example:kinesis-retrier
    1. If the stream name was changed: KINESIS_STREAM_NAME=my-stream-name npm run example:kinesis-retrier

kinesis-background-writer

  1. Create Kinesis Data Stream using AWS Console or any other method
    1. Example: aws kinesis create-stream --stream-name kinesis-helpers-test-stream --shard-count 1
    2. Default name is kinesis-helpers-test-stream
    3. 1 shard is sufficient
    4. 1 day retention is sufficient
    5. No encryption is sufficient
    6. On-demand throughput is sufficient
  2. npm run example:kinesis-background-writer
    1. If the stream name was changed: KINESIS_STREAM_NAME=my-stream-name npm run example:kinesis-background-writer

Generated using TypeDoc