# AWS S3 Source Connector

## Description

Receive data from an Amazon S3 Bucket prefix and writes to Kafka topic. Once you upload a file to S3 bucket it will be picked up by the connector, copy the body to kafka message and (optionally) delete the file from s3 bucket. More information about the connector can be found [here](https://camel.apache.org/camel-kafka-connector/4.0.x/reference/connectors/camel-aws-s3-source-kafka-source-connector.html).


## Connector: camel-aws-s3-source-kafka-connector 4.4.3

## Local testing

### Running the connector

The easiest way to test the connector locally is to use dev Kafka cluster and dev AWS S3 account. So before running the connector make sure that you have access to those services. Dev Kafka cluster is available under VPN connection. For accessing Dev AWS you should have a dedicated AWS account.

After that you need to copy `.env.shadow` to `.env` and fill in the values.

Then run the following commands to start the connector.
```
awsume dev
aws ecr get-login-password --region us-east-1 | docker login --username AWS --password-stdin 086679231553.dkr.ecr.us-east-1.amazonaws.com
docker-compose up --build -d
```

If all is configured correctly you should see the following logs:

```
s3_source connector successfully updated
s3_source connector started
```

and at this point you can see the connector status and its config by the following links:

- http://localhost:8083/connectors/s3_source/tasks/0/status
- http://localhost:8083/connectors/s3_source/config


### Configuration considerations
By default, the connector has `KAFKA_S3_DELETE_AFTER_READ` set to True.
If you set the property to false you’ll consume the same set of objects multiple times and you’ll have to deal with managing the duplicates downstream.


### Kafka messages

#### Headers:
Custom headers on S3 files are not sent to kafka topic. Only these headers are sent with the message.
```
CamelHeader.CamelAwsS3BucketName
CamelHeader.CamelAwsS3Key
CamelHeader.CamelAwsS3ContentLength
CamelHeader.CamelAwsS3ContentType
CamelHeader.CamelAwsS3ETag
CamelHeader.CamelAwsS3ExpirationTime
CamelHeader.CamelAwsS3VersionId
CamelHeader.CamelMessageTimestamp
```

#### Message:
Raw content of the s3 file.