# S3 File Watcher Daemon A lightweight Go daemon that monitors a directory for file changes and automatically uploads matching files to an S3 bucket. ## Quick Start The easiest way to get started is with the all-in-one setup script: ```bash cd s3-file-watcher # Run the setup and start script ./start.sh ``` The script will: 1. Check for required prerequisites (awsume, Go compiler) 2. Verify your AWS configuration has the required role 3. Build the executable if needed 4. Guide you through AWS authentication with awsume 5. Start the file watcher ## Project Structure ```text s3-file-watcher/ ├── bin/ # Compiled binaries ├── build/ │ └── build.sh # Build script for all platforms ├── code/ │ ├── go.mod # Go module definition │ ├── go.sum # Go dependencies lock file │ └── s3-file-watcher.go # Main source code ├── config.json # Your configuration (gitignored) ├── config.example.json # Example configuration ├── start.sh # All-in-one setup and start script └── README.md # This file ``` ## Features - **File System Monitoring**: Watches a directory for file creation and modification events - **Pattern Matching**: Supports both glob patterns (e.g., `*.json`) and regex patterns - **Recursive Watching**: Optionally monitors subdirectories - **Debouncing**: Prevents duplicate uploads from rapid file changes - **Retry Logic**: Configurable retry attempts for failed uploads - **Flexible Configuration**: Configure via command-line flags or JSON config file - **AWS Credential Support**: Works with AWS profiles, environment variables, or explicit credentials - **Automatic Content-Type Detection**: Sets proper MIME types based on file extensions (e.g., `.html` → `text/html`) for correct browser rendering - **Graceful Shutdown**: Handles SIGINT/SIGTERM for clean shutdown ## Installation ### Prerequisites - Go 1.21 or later (for building) - awsume (for AWS authentication with MFA) - AWS credentials configured with the required role ### Using the Start Script (Recommended) The `start.sh` script handles everything automatically: ```bash # First run - will check prerequisites and guide setup ./start.sh # Check setup only without starting the watcher ./start.sh --setup-only # Force rebuild of the binary ./start.sh --rebuild # Pass options to the watcher ./start.sh -- --watch-dir /tmp --s3-bucket mybucket --patterns '*.json' ``` ### Manual Build ```bash cd s3-file-watcher # Build for current platform (auto-detected) ./build/build.sh --current # Or build for all platforms ./build/build.sh --all ``` ### Build Script Options ```bash ./build/build.sh [OPTIONS] Options: --all Build for all platforms (default) --current Build for current platform only (auto-detected) --linux Build for Linux (amd64) --darwin Build for macOS (Intel and Apple Silicon) --darwin-amd64 Build for macOS Intel only --darwin-arm64 Build for macOS Apple Silicon only --windows Build for Windows (amd64) --clean Remove all binaries before building --help Show help message Examples: ./build/build.sh # Build for all platforms ./build/build.sh --current # Build for current platform only ./build/build.sh --darwin --linux # Build for macOS and Linux ./build/build.sh --clean --all # Clean and rebuild all ``` ### Available Binaries | Platform | Architecture | Binary Name | | -------- | ------------ | ----------- | | Linux | x86_64 (amd64) | `s3-file-watcher-linux-amd64` | | macOS | Intel (amd64) | `s3-file-watcher-darwin-amd64` | | macOS | Apple Silicon (arm64) | `s3-file-watcher-darwin-arm64` | | Windows | x86_64 (amd64) | `s3-file-watcher-windows-amd64.exe` | All binaries are output to the `bin/` directory. A convenience binary named `s3-file-watcher` (no suffix) is also created for your current platform. ## Usage ### Using Command-Line Flags ```bash ./bin/s3-file-watcher \ --watch-dir /path/to/watch \ --s3-bucket my-bucket \ --s3-prefix uploads/ \ --patterns "*.json,*.xml,report_*.csv" \ --recursive ``` ### Using Configuration File ```bash ./bin/s3-file-watcher --config config.json ``` ### Mixed (Config File + Flag Overrides) Command-line flags override values from the config file: ```bash ./bin/s3-file-watcher --config config.json --s3-bucket different-bucket ``` ## Configuration Options | Option | CLI Flag | JSON Key | Default | Description | |--------|----------|----------|---------|-------------| | Watch Directory | `--watch-dir` | `watch_dir` | *required* | Directory to monitor for file changes | | S3 Bucket | `--s3-bucket` | `s3_bucket` | *required* | Target S3 bucket name | | S3 Prefix | `--s3-prefix` | `s3_prefix` | `""` | Prefix (folder path) for S3 keys | | S3 Region | `--s3-region` | `s3_region` | `us-east-1` | AWS region for the S3 bucket | | Patterns | `--patterns` | `patterns` | *required* | File patterns to match (comma-separated for CLI) | | AWS Role ARN | `--aws-role-arn` | `aws_role_arn` | `""` | IAM role ARN to assume (STS AssumeRole) | | AWS Role Session | `--aws-role-session-name` | `aws_role_session_name` | `s3-file-watcher` | Session name for assumed role | | AWS Profile | `--aws-profile` | `aws_profile` | `""` | Named AWS profile to use | | AWS Access Key | `--aws-access-key` | `aws_access_key` | `""` | AWS access key ID | | AWS Secret Key | `--aws-secret-key` | `aws_secret_key` | `""` | AWS secret access key | | Recursive | `--recursive` | `recursive` | `false` | Watch subdirectories recursively | | Debounce (ms) | `--debounce-ms` | `debounce_ms` | `500` | Delay before uploading after file change | | Retry Attempts | `--retry-attempts` | `retry_attempts` | `3` | Number of upload retry attempts | | Retry Delay (ms) | `--retry-delay-ms` | `retry_delay_ms` | `1000` | Delay between retry attempts | ## Pattern Matching The daemon supports two types of patterns: ### Glob Patterns Standard glob patterns are automatically converted to regex: - `*.json` - Matches any file ending in `.json` - `report_*.csv` - Matches files like `report_2024.csv`, `report_sales.csv` - `data?.txt` - Matches `data1.txt`, `dataA.txt`, etc. - `**/*.xml` - Matches `.xml` files in any subdirectory ### Regex Patterns For more complex matching, use regex patterns. Patterns starting with `^` or ending with `$` are treated as regex: - `^data_\d{4}-\d{2}-\d{2}\.txt$` - Matches `data_2024-01-15.txt` - `^[A-Z]{3}_report\.csv$` - Matches `ABC_report.csv` ## Example Configuration File ```json { "watch_dir": "/home/user/data/exports", "s3_bucket": "my-data-bucket", "s3_prefix": "imports/raw/", "s3_region": "us-west-2", "patterns": [ "*.json", "*.xml", "export_*.csv", "^backup_\\d{8}\\.tar\\.gz$" ], "aws_profile": "production", "recursive": true, "debounce_ms": 1000, "retry_attempts": 5, "retry_delay_ms": 2000 } ``` ## AWS Credentials The daemon supports multiple methods for AWS authentication (in order of precedence): 1. **Environment variables**: `AWS_ACCESS_KEY_ID`, `AWS_SECRET_ACCESS_KEY`, `AWS_SESSION_TOKEN` 2. **Explicit credentials in config**: Set `aws_access_key` and `aws_secret_key` 3. **Named profile**: Set `aws_profile` to use a specific profile from `~/.aws/credentials` 4. **Default credential chain**: EC2 instance profile, ECS task role, etc. ### Using awsume for MFA-Protected Cross-Account Access (Recommended) If your organization uses MFA-protected cross-account role assumption (common in enterprise AWS setups), you should use [awsume](https://awsume.io/) to handle authentication **before** running the daemon. #### Why awsume? Many AWS configurations require MFA tokens to assume cross-account roles. The daemon cannot prompt for MFA interactively, so you must authenticate first using a tool like `awsume`. #### Setup 1. **Install awsume** (if not already installed): ```bash pip install awsume awsume-configure ``` 2. **Configure your AWS profiles** in `~/.aws/config`: ```ini [profile base_profile] region = us-east-1 mfa_serial = arn:aws:iam::YOUR_ACCOUNT:mfa/your-username [profile target_profile] source_profile = base_profile role_session_name = your-username role_arn = arn:aws:iam::TARGET_ACCOUNT:role/target-role ``` 3. **Authenticate with awsume** (enter MFA when prompted): ```bash awsume target_profile ``` 4. **Run the daemon** in the same terminal session: ```bash ./bin/s3-file-watcher --watch-dir /path/to/watch --s3-bucket my-bucket --patterns "*.json" ``` **Important**: Do NOT specify `--aws-access-key`, `--aws-secret-key`, or `--aws-role-arn` flags. The daemon will automatically use the environment variables set by awsume. #### Example Workflow ```bash # Step 1: Authenticate (will prompt for MFA token) awsume orch_dev_awsume # Step 2: Verify credentials are set echo $AWS_ACCESS_KEY_ID # Should show temporary access key # Step 3: Run the daemon (no credential flags needed) ./bin/s3-file-watcher \ --watch-dir /tmp/s3-watcher-test \ --s3-bucket metamulate.dev.theorchard.io \ --s3-prefix uploads/ \ --patterns "*.json,*.txt" ``` #### Credential Expiration Temporary credentials from awsume typically expire after 1 hour. For long-running daemons: - **Option 1**: Use `awsume --auto-refresh` to keep credentials fresh - **Option 2**: Run the daemon in a script that re-authenticates periodically - **Option 3**: For production, use EC2 instance roles or ECS task roles instead ### IAM Role Assumption (Without MFA) If your role doesn't require MFA, you can use the built-in role assumption: ```json { "aws_role_arn": "arn:aws:iam::123456789012:role/my-role", "aws_role_session_name": "s3-file-watcher" } ``` Or via CLI: ```bash ./bin/s3-file-watcher --aws-role-arn arn:aws:iam::123456789012:role/my-role ... ``` The daemon will use STS `AssumeRole` to obtain temporary credentials. Your base credentials (profile, environment, or instance role) must have permission to assume the target role. ### IAM Permissions Required The daemon requires the following S3 permissions on the target bucket: ```json { "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Action": [ "s3:PutObject" ], "Resource": "arn:aws:s3:::my-bucket/*" } ] } ``` ## Running as a System Service ### systemd (Linux) Create `/etc/systemd/system/s3-file-watcher.service`: ```ini [Unit] Description=S3 File Watcher Daemon After=network.target [Service] Type=simple User=s3watcher ExecStart=/usr/local/bin/s3-file-watcher --config /etc/s3-file-watcher/config.json Restart=always RestartSec=10 [Install] WantedBy=multi-user.target ``` Then enable and start: ```bash sudo systemctl daemon-reload sudo systemctl enable s3-file-watcher sudo systemctl start s3-file-watcher ``` ### launchd (macOS) Create `~/Library/LaunchAgents/com.s3-file-watcher.plist`: ```xml Label com.s3-file-watcher ProgramArguments /usr/local/bin/s3-file-watcher --config /Users/youruser/.config/s3-file-watcher/config.json RunAtLoad KeepAlive StandardOutPath /tmp/s3-file-watcher.log StandardErrorPath /tmp/s3-file-watcher.err ``` Load the service: ```bash launchctl load ~/Library/LaunchAgents/com.s3-file-watcher.plist ``` ## Logging The daemon logs all activity to stdout/stderr: - File detection events - Upload successes and failures - Retry attempts - Errors and warnings For production use, redirect output to a log file or use a logging aggregator. ## Troubleshooting ### Common Issues **1. "watch directory does not exist"** - Ensure the specified directory exists and is accessible **2. "too many open files"** - Increase the file descriptor limit: `ulimit -n 4096` - Reduce the number of watched directories **3. S3 upload failures** - Verify AWS credentials are configured correctly - Check network connectivity to AWS - Ensure the S3 bucket exists and you have write permissions **4. Files not being detected** - Check that file patterns match the filenames - Verify the watch directory is correct - For regex patterns, ensure proper escaping in JSON **5. "User is not authorized to perform: sts:AssumeRole"** This error occurs when trying to assume a cross-account role that requires MFA. The solution is to use `awsume` to authenticate first: ```bash # Authenticate with MFA using awsume awsume your_profile_name # Then run the daemon WITHOUT --aws-role-arn flag ./bin/s3-file-watcher --watch-dir /path --s3-bucket bucket --patterns "*.json" ``` See the [Using awsume for MFA-Protected Cross-Account Access](#using-awsume-for-mfa-protected-cross-account-access-recommended) section for details. **6. Credentials expired after 1 hour** Temporary credentials from `awsume` expire. Options: - Re-run `awsume` and restart the daemon - Use `awsume --auto-refresh` for auto-renewal - For production, use instance roles or long-term credentials with appropriate permissions ### Debug Mode For verbose output, run with environment variable: ```bash AWS_SDK_LOAD_CONFIG=1 ./bin/s3-file-watcher --config config.json 2>&1 | tee watcher.log ``` ## License MIT License