Kinesis Source

This recipe uses the LocalStack AWS environment to run an Amazon Kinesis surrogate. The walkthrough follows the Get started with Kinesis on LocalStack tutorial.

If you intend to invoke the commands on a real AWS environment, just use aws instead of awslocal.

Tip

LocalStack is a cloud service emulator that runs in a single container on your laptop or in your CI environment. With LocalStack, you can run your AWS applications or Lambdas entirely on your local machine without connecting to a remote cloud provider.

Setup

Start the LocalStack service using Docker.

docker run \
  --rm -it \
  -p 127.0.0.1:4566:4566 \
  -p 127.0.0.1:4510-4559:4510-4559 \
  -v /var/run/docker.sock:/var/run/docker.sock \
  localstack/localstack:3.6

Install LorryStream including LocalStack CLI programs.

pip install --upgrade 'lorrystream[carabas]'

Configure

Create a Kinesis Data Stream called testdrive.

awslocal kinesis create-stream \
  --stream-name testdrive \
  --shard-count 1

Check the status of your streams.

awslocal kinesis list-streams

awslocal kinesis describe-stream \
  --stream-name testdrive

Display Stream ARN.

awslocal kinesis describe-stream --stream-name testdrive | jq -r .StreamDescription.StreamARN

Usage

Submit an item to the data stream, using awslocal.

awslocal kinesis put-record \
    --stream-name testdrive \
    --partition-key 1 \
    --data '{"device": "foo", "temperature": 42.42, "humidity": 84.84}'

Submit an item to the data stream, using Python.

export AWS_ENDPOINT_URL="http://localhost:4566"
python examples/aws/kinesis_publish.py testdrive

Consume data stream, printing received payloads to STDOUT. This is suitable for debugging purposes.

export AWS_ENDPOINT_URL="http://localhost:4566"
python examples/aws/kinesis_subscribe.py testdrive

Todo

Demonstrate how to add a processor pipeline element using both either AWS Lambda, or a dedicated processor instance.