# Kinesis Stream to Lambda Listener

## Overview

This script is designed to listen to an Amazon Kinesis data stream and invoke an AWS Lambda function with the events from the stream. It processes records from all shards in the Kinesis stream concurrently and sends the data to the Lambda function as an HTTP POST request.

## Features

- Automatically detects and listens to all shards in the Kinesis stream.
- Processes records in real-time and encodes the data in a format compatible with AWS Lambda.
- Implements retry logic with exponential backoff for Lambda invocation to handle transient errors.
- Utilizes multithreading for concurrent shard processing and efficient event handling.

## Use Case

This script is suitable for scenarios where you need to:

1. Stream data from an Amazon Kinesis data stream.
2. Process the data in near real-time.
3. Forward the processed data to an AWS Lambda function for further processing or integration.

## Environment Variables

The script relies on the following environment variables:

| Variable           | Description                                      |
|--------------------|--------------------------------------------------|
| `KINESIS_STREAM_NAME` | Name of the Kinesis data stream to listen to.     |
| `LAMBDA_ENDPOINT`  | HTTP endpoint of the AWS Lambda function.       |
| `AWS_REGION`       | AWS region of the Kinesis stream (default: `us-east-1`). |

## Prerequisites

- An active Amazon Kinesis data stream.
- A deployed AWS Lambda function with an HTTP endpoint to receive the events.
- AWS credentials configured on the system running the script.

## How It Works

1. **Shard Detection**:
   - The script fetches all shards from the specified Kinesis stream.
2. **Shard Processing**:
   - Each shard is processed in a separate thread, continuously polling for new records.
3. **Event Decoding**:
   - Records from the shard are decoded and formatted into a Kinesis event structure.
4. **Event Forwarding**:
   - Events are added to a shared queue and sent to the Lambda function sequentially.
5. **Error Handling**:
   - Retry logic ensures that transient errors during Lambda invocation do not cause data loss.

## Usage

1. Set the required environment variables:

   ```bash
   export KINESIS_STREAM_NAME="your_kinesis_stream_name"
   export LAMBDA_ENDPOINT="https://your_lambda_endpoint"
   export AWS_REGION="us-east-1"
   ```

2. Run the script:

   ```bash
   python kinesis_lambda_listener.py
   ```

## Logging

The script uses Python's logging module to log:

- Information about shard detection and processing.
- Events received from the Kinesis stream.
- Status and responses from the Lambda function.
- Errors and retries during Lambda invocation.

## Limitations

- The script uses a default shard iterator type of `LATEST`, meaning only new records will be processed.
- Records are sent to Lambda sequentially, which may introduce delays under high loads.
- Ensure the Lambda endpoint can handle the event structure and scale to match the stream's throughput.

## Extending the Script

- Modify the `invoke_lambda_sequential` function to customize the way Lambda is invoked.
- Change the `ShardIteratorType` in the `process_shard` function to read older records if required.

## Dependencies

- `boto3`: AWS SDK for Python.
- `requests`: For making HTTP requests to the Lambda endpoint.

Install the dependencies using:

```bash
pip install boto3 requests