# AWS Connectivity Issues - Troubleshooting Guide

This document tracks AWS connectivity issues and their solutions for the PDP Backfill Upload Application.

## Quick Navigation

- [Issue #1: AWS Credentials Not Available in Browser](#issue-1-aws-credentials-not-available-in-browser) - ✅ **Resolved** → ⚠️ **Being Rolled Back**
- [Issue #2: CORS Error - S3 Bucket Blocks Browser Requests](#issue-2-cors-error---s3-bucket-blocks-browser-requests) - 🔴 **Active Issue** → ✅ **Resolved by Architecture Change**

---

## 🔄 Decision Change (2026-01-07)

**We are switching from Option 1 to Option 2.**

### Decision
After implementing Option 1 (VITE credentials for direct browser-to-S3 uploads), we encountered Issue #2 (CORS errors). While CORS can be fixed by configuring the S3 buckets, we have decided to **switch to Option 2 (Backend Proxy)** instead to avoid needing CORS configuration on QA and Production S3 buckets.

### Rationale
- **CORS Configuration Challenges**: Requires administrative access to modify S3 bucket policies in QA and Production
- **Security Concerns**: Having AWS credentials visible in browser DevTools (even temporary ones) is not ideal
- **Better Architecture**: Backend proxy provides cleaner separation of concerns and better security model
- **Future Flexibility**: Backend can add validation, logging, rate limiting, etc.

### Impact
- Option 1 implementation will be rolled back
- Frontend will call backend API instead of S3 directly
- Backend will handle all S3 uploads server-side (no CORS issues)
- AWS credentials stay server-side (never exposed to browser)

### Next Steps
See `/Users/tcalhoun/work/collab/technicalelvis/pdp-backfill-app/_tasks/04-Create-Backend-Upload-Service.md` for the detailed implementation plan.

---

## Issue #1: AWS Credentials Not Available in Browser

### Status

✅ **Resolved** - Credentials now pass via VITE_ environment variables

### Problem Diagnosis

The application is experiencing a **fundamental architectural issue**: it's trying to use AWS SDK v3 directly in the
browser, but the browser cannot access the AWS credentials set by `awsume` in your shell.

### Why It's Failing

1. **Browser Environment Limitation**: When you run `awsume permissions-platform-qa`, temporary credentials are set as
   environment variables in your shell (`AWS_ACCESS_KEY_ID`, `AWS_SECRET_ACCESS_KEY`, `AWS_SESSION_TOKEN`)

2. **Isolation**: The React application runs in the browser, which is completely isolated from:
    - Shell environment variables (even with `pnpm start` running in that shell)
    - The `~/.aws/credentials` file on your filesystem
    - Environment variables set in the Node.js dev server process

3. **Current Code Assumption**: The code in `src/services/awsConfig.ts` (lines 32-37) has misleading comments suggesting
   the AWS SDK will automatically load credentials from environment variables or `~/.aws/credentials`. This is true for
   **Node.js environments**, but not for **browser environments**.

4. **The Docker Setup Has Same Issue**: Even though docker-compose.yml mounts `~/.aws` and sets `AWS_PROFILE`, those are
   only available to the Node.js dev server, not the React app running in the browser.

### The Missing Link

Vite's environment variable system requires the **`VITE_` prefix** for variables to be exposed to the browser. Only
variables starting with `VITE_` get bundled into the client-side code. Your `awsume` credentials are not prefixed with
`VITE_`, so they never make it to the browser.

## Solution Options

There are three main approaches to fix this issue. Each has trade-offs:

### Option 1: Pass Credentials via VITE_ Environment Variables (RECOMMENDED)

**Pros:**

- Simple to implement (modify 2 files)
- Works seamlessly with existing awsume workflow
- No backend required
- Credentials automatically expire with awsume session

**Cons:**

- Credentials are visible in browser DevTools/source (acceptable for temporary credentials)
- Requires running awsume before starting dev server

**Implementation:**

1. **File: `src/services/awsConfig.ts`**
    - Create custom credential provider that reads from `import.meta.env`:
      ```typescript
      export function createS3Client(): S3Client {
          return new S3Client({
              region: AWS_REGION,
              credentials: {
                  accessKeyId: import.meta.env.VITE_AWS_ACCESS_KEY_ID || '',
                  secretAccessKey: import.meta.env.VITE_AWS_SECRET_ACCESS_KEY || '',
                  sessionToken: import.meta.env.VITE_AWS_SESSION_TOKEN,
              },
          });
      }
      ```
    - Update JSDoc comments to reflect browser limitations

2. **File: `package.json`**
    - Update `start` script to pass credentials with VITE_ prefix:
      ```json
      "start": "bash -c 'export VITE_AWS_ACCESS_KEY_ID=$AWS_ACCESS_KEY_ID VITE_AWS_SECRET_ACCESS_KEY=$AWS_SECRET_ACCESS_KEY VITE_AWS_SESSION_TOKEN=$AWS_SESSION_TOKEN && npx @theorchard/frontend-cli dev'"
      ```

3. **File: `CLAUDE.md`**
    - Document the awsume → pnpm start workflow

**Workflow:**

1. Run `awsume permissions-platform-qa`
2. Run `pnpm start` (automatically passes credentials to browser)
3. Upload files successfully

---

### Option 2: Create Backend Proxy for S3 Uploads

**Pros:**

- Most secure (credentials never exposed to browser)
- Works naturally with awsume (backend runs in Node.js)
- Can add additional validation/logging on backend

**Cons:**

- Requires creating Express/Fastify backend
- More complex architecture
- Need to deploy/manage backend service

**Implementation Overview:**

- Create backend service that accepts file uploads
- Backend uses AWS SDK in Node.js (naturally reads awsume credentials)
- Frontend sends files to backend, backend uploads to S3
- Requires 5-10 files of new code

---

### Option 3: Manual Credential Entry in UI

**Pros:**

- No script changes needed
- Explicit credential visibility

**Cons:**

- Requires manually copying credentials from awsume output
- Poor user experience
- Still exposes credentials in browser

**Implementation Overview:**

- Add form fields for Access Key ID, Secret Access Key, Session Token
- User copies values from `awsume` output or `~/.aws/credentials`
- Store in React state, pass to S3Client
- Requires UI changes and state management

---

## Recommended Approach: Option 1

**Option 1 is recommended** because it:

- Maintains current browser-based architecture
- Works seamlessly with existing awsume workflow
- Minimal code changes (2 files)
- Credentials are temporary and expire automatically
- Acceptable security model for temporary credentials in development environment

## Files to Modify (Option 1 Implementation)

1. **`src/services/awsConfig.ts`** (lines 39-45)
    - Modify `createS3Client()` to read credentials from `import.meta.env`
    - Update JSDoc comments (lines 31-38) to accurately describe browser environment
    - Add error handling for missing credentials

2. **`package.json`** (line 8)
    - Update `start` script to export awsume credentials with VITE_ prefix

3. **`CLAUDE.md`** (lines 117-130)
    - Update "AWS Credentials Setup" section with correct awsume workflow
    - Add troubleshooting section for common credential issues

4. **`docker-compose.yml`** (optional, line 25-28)
    - Add VITE_ prefixed environment variables for Docker environment

## Detailed Implementation for Option 1

### Step 1: Update awsConfig.ts

Replace the `createS3Client()` function (lines 39-45) with:

```typescript
/**
 * Create S3 client instance for browser environment
 *
 * IMPORTANT: This code runs in the browser, not Node.js. Credentials must be
 * provided via Vite environment variables (VITE_AWS_ACCESS_KEY_ID, etc.) which
 * are set at build/dev server start time.
 *
 * For local development with awsume:
 * 1. Run `awsume permissions-platform-qa` to set AWS_* env vars in your shell
 * 2. Run `pnpm start` which automatically exports them with VITE_ prefix
 * 3. Vite bundles these into the browser code as import.meta.env.VITE_AWS_*
 *
 * Note: Credentials are temporary and expire with your awsume session.
 */
export function createS3Client(): S3Client {
    const accessKeyId = import.meta.env.VITE_AWS_ACCESS_KEY_ID;
    const secretAccessKey = import.meta.env.VITE_AWS_SECRET_ACCESS_KEY;
    const sessionToken = import.meta.env.VITE_AWS_SESSION_TOKEN;

    if (!accessKeyId || !secretAccessKey) {
        throw new Error(
            'AWS credentials not found. Please run `awsume permissions-platform-qa` before starting the dev server.',
        );
    }

    return new S3Client({
        region: AWS_REGION,
        credentials: {
            accessKeyId,
            secretAccessKey,
            sessionToken, // Optional, but required for temporary credentials from awsume
        },
    });
}
```

### Step 2: Update package.json

Replace line 8:

```json
"start": "npx @theorchard/frontend-cli dev",
```

With:

```json
"start": "bash -c 'export VITE_AWS_ACCESS_KEY_ID=$AWS_ACCESS_KEY_ID VITE_AWS_SECRET_ACCESS_KEY=$AWS_SECRET_ACCESS_KEY VITE_AWS_SESSION_TOKEN=$AWS_SESSION_TOKEN && npx @theorchard/frontend-cli dev'",
```

### Step 3: Update CLAUDE.md

Update the "AWS Credentials Setup" section to document:

- The awsume → pnpm start workflow
- Why VITE_ prefix is needed
- Troubleshooting steps for "Credential is missing" error

## Testing Plan

### Verification Steps:

1. **Clean start:**
   ```bash
   # Kill any running dev servers
   pkill -f "frontend-cli"
   ```

2. **Set up credentials:**
   ```bash
   awsume permissions-platform-qa
   # Verify credentials are set:
   echo $AWS_ACCESS_KEY_ID
   ```

3. **Start dev server:**
   ```bash
   pnpm start
   # Server should start on http://localhost:3000
   ```

4. **Test in browser:**
    - Navigate to http://localhost:3000
    - Select QA environment
    - Choose CSV files and manifest.json
    - Enter folder path (e.g., "test-upload")
    - Click through to upload step
    - **Expected:** Validation succeeds, upload proceeds
    - **Previous error:** "Failed to validate AWS credentials: Credential is missing"

5. **Verify in browser DevTools:**
    - Open Console
    - Type: `import.meta.env.VITE_AWS_ACCESS_KEY_ID`
    - Should see your access key (confirms credentials are passed to browser)

6. **Test S3 upload:**
    - Complete upload workflow
    - Check S3 bucket for uploaded files
    - Verify manifest triggers backfill (if applicable)

## Notes

- Credentials are visible in browser source (acceptable for temporary awsume credentials)
- Credentials expire when awsume session expires (typically 1-12 hours)
- If credentials expire during use, user must refresh browser after running awsume again
- The `@aws-sdk/credential-providers` package can be removed from package.json if desired (not used in this solution)
- Works for both QA and Production environments (same awsume workflow)
- Docker setup will need similar updates if used (add VITE_ vars to docker-compose.yml)

---

## Issue #2: CORS Error - S3 Bucket Blocks Browser Requests

### Status

🔴 **Active Issue** - Blocking uploads

### Problem Diagnosis

After fixing the credentials issue (Issue #1), uploads are now failing with a CORS (Cross-Origin Resource Sharing)error:

```
Access to fetch at 'https://qa-pdp-backfill.s3.us-east-1.amazonaws.com/' from origin 'http://localhost:3000'
has been blocked by CORS policy: Response to preflight request doesn't pass access control check:
No 'Access-Control-Allow-Origin' header is present on the requested resource.
```

#### Why This Happens

1. **Browser Security Model**: When JavaScript running in a browser (at `http://localhost:3000`) tries to make requests
   to a different origin (S3 bucket at `https://qa-pdp-backfill.s3.us-east-1.amazonaws.com`), the browser performs a
   CORS check.

2. **Preflight Requests**: For certain HTTP methods (like HEAD and PUT, which we use), the browser sends a preflight
   OPTIONS request to check if the cross-origin request is allowed.

3. **S3 Default Behavior**: By default, S3 buckets do NOT have CORS enabled, so they don't return the required
   `Access-Control-Allow-Origin` header.

4. **Where It Fails**: The error occurs on line 80 of `s3Service.ts` when `validateCredentials()` attempts to send a
   `HeadBucketCommand` to verify AWS credentials before upload.

#### Impact

- Credential validation fails immediately
- Files cannot be uploaded
- User sees error in browser console
- Upload workflow is completely blocked

### Solution: Configure CORS on S3 Buckets

The proper solution is to add CORS configuration to the S3 buckets to allow requests from the development origin.

#### Step 1: Create CORS Configuration File

Create a file named `cors-config.json` with the following content:

```json
[
  {
    "AllowedHeaders": [
      "*"
    ],
    "AllowedMethods": [
      "HEAD",
      "GET",
      "PUT"
    ],
    "AllowedOrigins": [
      "http://localhost:3000",
      "http://localhost:5173"
    ],
    "ExposeHeaders": [
      "ETag"
    ],
    "MaxAgeSeconds": 3000
  }
]
```

**Configuration Explanation:**

- **AllowedHeaders**: `["*"]` - Allow all headers in requests
- **AllowedMethods**: `["HEAD", "GET", "PUT"]` - Required for credential validation (HEAD) and file uploads (PUT)
- **AllowedOrigins**: Development URLs that need access
    - `http://localhost:3000` - Default React dev server port
    - `http://localhost:5173` - Default Vite dev server port
- **ExposeHeaders**: `["ETag"]` - Allow JavaScript to read ETag header from responses
- **MaxAgeSeconds**: `3000` - Browser can cache preflight response for 50 minutes

#### Step 2: Apply CORS Configuration to QA Bucket

Assume the QA role and apply the configuration:

```bash
# Assume QA role
awsume permissions-platform-qa-generic

# Apply CORS configuration
aws s3api put-bucket-cors \
    --bucket qa-pdp-backfill \
    --cors-configuration file://cors-config.json
```

#### Step 3: Apply CORS Configuration to Production Bucket

For production, you may want to restrict origins more tightly (e.g., only allow your deployed production domain):

```json
[
  {
    "AllowedHeaders": [
      "*"
    ],
    "AllowedMethods": [
      "HEAD",
      "GET",
      "PUT"
    ],
    "AllowedOrigins": [
      "https://your-production-domain.com",
      "http://localhost:3000"
    ],
    "ExposeHeaders": [
      "ETag"
    ],
    "MaxAgeSeconds": 3000
  }
]
```

Apply to production bucket:

```bash
# Assume Production role
awsume permissions-platform-prod-generic

# Apply CORS configuration
aws s3api put-bucket-cors \
    --bucket prod-pdp-backfill \
    --cors-configuration file://cors-config-prod.json
```

#### Step 4: Verify CORS Configuration

Verify the CORS configuration was applied correctly:

```bash
# For QA bucket
awsume permissions-platform-qa-generic
aws s3api get-bucket-cors --bucket qa-pdp-backfill

# For Production bucket
awsume permissions-platform-prod-generic
aws s3api get-bucket-cors --bucket prod-pdp-backfill
```

Expected output should match your `cors-config.json` file.

### Testing After CORS Configuration

1. **Clear browser cache** (CORS preflight responses may be cached)
    - In Chrome: DevTools → Network tab → Right-click → Clear browser cache
    - Or use Incognito/Private mode

2. **Reload the application**:
   ```bash
   # No need to restart dev server, just refresh browser
   open http://localhost:3000
   ```

3. **Attempt upload workflow**:
    - Select QA environment
    - Choose CSV files and manifest.json
    - Enter folder path
    - Proceed to upload
    - **Expected**: Credential validation succeeds, upload proceeds
    - **Previous error**: CORS error in console

4. **Verify in browser DevTools**:
    - Open Network tab
    - Filter by "qa-pdp-backfill"
    - Look for HEAD request
    - Check Response Headers for `access-control-allow-origin: http://localhost:3000`

### Alternative: Temporary Workaround (Not Recommended)

If you cannot configure CORS on the S3 buckets right now, there is a code-based workaround:

**Option A**: Skip credential validation entirely

Modify `s3Service.ts` line 252 to comment out credential validation:

```typescript
// await validateCredentials(bucket);  // Skip validation due to CORS
```

**Pros**: Allows uploads to proceed
**Cons**:

- Credential errors only appear during upload (not upfront)
- Poor user experience
- Doesn't fix the underlying issue

**Option B**: Use AWS SDK error handling

Let the first upload attempt serve as credential validation:

```typescript
// Remove validateCredentials() call entirely
// Let uploadCsvFiles() handle credential errors naturally
```

This is less user-friendly but works without CORS configuration.

### Security Considerations

**Development CORS Configuration**:

- Allowing `http://localhost:*` is safe for development
- These origins are only accessible from your local machine
- Temporary credentials from awsume have limited permissions and expiration

**Production CORS Configuration**:

- Should only allow your deployed production domain
- Remove localhost origins for production bucket
- Consider using more restrictive CORS policies if possible
- Review with security team before deploying

### Related Files

- `src/services/s3Service.ts:75-85` - `validateCredentials()` function that triggers CORS preflight
- `src/services/s3Service.ts:106-147` - `uploadFile()` function that performs PUT requests
- `src/services/awsConfig.ts:39-45` - S3 client creation with credentials

### References

- [AWS S3 CORS Documentation](https://docs.aws.amazon.com/AmazonS3/latest/userguide/cors.html)
- [MDN CORS Documentation](https://developer.mozilla.org/en-US/docs/Web/HTTP/CORS)
- [AWS CLI put-bucket-cors](https://docs.aws.amazon.com/cli/latest/reference/s3api/put-bucket-cors.html)

### Resolution Checklist

- [ ] Create `cors-config.json` file with proper configuration
- [ ] Apply CORS to `qa-pdp-backfill` bucket
- [ ] Apply CORS to `prod-pdp-backfill` bucket
- [ ] Verify CORS configuration with `get-bucket-cors`
- [ ] Test upload workflow in browser
- [ ] Verify no CORS errors in browser console
- [ ] Document CORS configuration in deployment guide (if applicable)

---