# Plan: Configure Fluent Bit Memory Settings in Terraform-Fargate

## Problem Statement

**Context**: The Cerbos service (QA environment:
`/Users/tcalhoun/work/terraform-infra/permissions-platform/qa/cerbos/main.tf`) is experiencing missing decision logs in
Datadog. Testing shows that "complicated user" decision logs can be **1.9MB (2,024,648 bytes)** in size.

**Issue 1 - Configuration Error**: When attempting to use `log_configuration_options_override` to set Fluent Bit memory
configuration parameters (e.g., `buffer_chunk_size`), the logging container fails with an error:

```text
[2026/01/06 20:39:34] [error] [config] datadog: unknown configuration property 'buffer_chunk_size'. The following properties are allowed: compress, apikey, dd_service, dd_source, dd_tags, proxy, include_tag_key, tag_key, dd_message_key, provider, and json_date_key.
```

This occurs because memory/buffer configuration parameters are service-level (INPUT) settings in Fluent Bit, not output
plugin settings. The `log_configuration_options_override` variable only affects the `logConfiguration.options` passed to
the output plugin (Datadog in this case).

**Issue 2 - Silent Log Drops**: The Forward input plugin has a **default `Buffer_Max_Size` of 32KB**. When logs exceed
this limit (like Cerbos's 1.9MB decision logs), Fluent Bit silently drops them with errors like:

```text
[in_fw] fd=25 incoming data exceed limit (32768 bytes)
```

**Issue 3 - Datadog Truncation**: Even after fixing Fluent Bit, Datadog has a **1MB limit per individual log**. Logs
larger than 1MB are truncated, meaning the 1.9MB Cerbos logs will be cut off.

## Root Cause Analysis

### Terraform-Fargate Configuration Issue

The current terraform-fargate implementation:

1. **Location**: `/Users/tcalhoun/work/terraform-fargate/variables.tf:818-822` and `variables.tf:989-1002`
2. The `log_configuration_options_override` variable merges options into `log_configuration_options`
3. These options are passed to the FireLens `logConfiguration` (see
   `web_service_task_definition_template_file.json:38-45`)
4. FireLens only passes these options to the OUTPUT plugin (Datadog), not to the Fluent Bit service configuration
5. Memory settings like `Mem_Buf_Limit`, `storage.type`, and `Buffer_Chunk_Size` belong to the SERVICE or INPUT sections
   of Fluent Bit config, not OUTPUT

### Fluent Bit Forward Plugin Buffer Limits

The Forward input plugin has critical buffer size limits for receiving messages:

- **`Buffer_Chunk_Size`** (default: 32KB): Initial memory allocation for receiving data. The plugin allocates memory
  incrementally in rounds of this size.
- **`Buffer_Max_Size`** (default: 32KB): **Maximum buffer size to receive a single Forward message**. Messages exceeding
  this are dropped.

**These settings are NOT specific to the Tail plugin** - they apply to the Forward plugin used by FireLens to receive
logs from application containers.

For Cerbos's 1.9MB logs, the default 32KB limit causes immediate drops.

### Cerbos-Specific Issues

Current Cerbos configuration (`permissions-platform/qa/cerbos/main.tf:91-92`):

```hcl
fluentbit_task_cpu    = 128
fluentbit_task_memory = 128  # Only 128MB!
```

With 1.9MB individual logs and 128MB container memory:

- Cannot buffer many large logs simultaneously
- Risk of OOM kills under burst traffic
- Insufficient headroom for filesystem buffering

## Solution Approach

According to AWS
documentation ([How to set Fluentd and Fluent Bit input parameters in FireLens](https://aws.amazon.com/blogs/containers/how-to-set-fluentd-and-fluent-bit-input-parameters-in-firelens/)),
memory and service-level configurations must be set by:

1. Creating a custom Fluent Bit configuration file
2. Baking this config file into a custom Docker image (orchard-fluent-bit)
3. **Using FireLens `config-file-value` option** to specify the custom config

### Key Decision: config-file-value vs. Custom Entrypoint

**Our Approach**: Use FireLens `config-file-value` option

```json
"firelensConfiguration": {
"type": "fluentbit",
"options": {
"config-file-type": "file",
"config-file-value": "/fluent-bit/configs/large-logs.conf"
}
}
```

**Why this approach?**

- ✅ **Keeps FireLens active** - automatic ECS metadata injection continues
- ✅ **Less invasive** - FireLens handles INPUT/OUTPUT routing automatically
- ✅ **Better backwards compatibility** - just adding a file doesn't affect existing services
- ✅ **Simpler maintenance** - no need to manually configure all plugins

**Alternative (NOT used)**: Custom entrypoint

- ❌ Disables FireLens completely
- ❌ Must manually configure all INPUT/OUTPUT plugins
- ❌ Lose automatic ECS metadata injection
- ❌ More maintenance burden

## Implementation Plan

### Phase 1: Update orchard-fluent-bit Docker Image

**Location**: `/Users/tcalhoun/work/orchard-fluent-bit/`

1. **Create Custom Fluent Bit Configuration File**
    - Create `/fluent-bit/configs/large-logs.conf` (optimized for 1.9MB+ Cerbos logs)
    - Include service-level storage settings sized for large logs:
      ```ini
      [SERVICE]
          Flush                     5
          Daemon                    Off
          Log_Level                 ${LOG_LEVEL}
          Parsers_File              /fluent-bit/parsers/parsers.conf
          # Storage settings for large log buffering
          storage.path              /var/log/flb-storage/
          storage.sync              normal
          storage.checksum          off
          storage.max_chunks_up     128
          storage.backlog.mem_limit 10M  # Increased for large logs
      ```

2. **Update INPUT Configuration for Log Forwarding**
    - Add INPUT section with buffer limits sized for large logs (1.9MB+):
      ```ini
      [INPUT]
          Name              forward
          Listen            0.0.0.0
          Port              24224
          Buffer_Chunk_Size 512KB      # Incremental allocation size
          Buffer_Max_Size   4MB        # CRITICAL: Must be > 1.9MB (add headroom)
          Mem_Buf_Limit     20MB       # Allow ~10 large logs in memory
          storage.type      filesystem # Overflow to disk when memory full
      ```
    - **Note**: `Buffer_Max_Size` is the maximum size for a **single message**, not total memory
    - For Cerbos: Set to 4MB to accommodate 1.9MB logs with safety margin

3. **Update Dockerfile**
    - Ensure new config file is copied to the image:
      ```dockerfile
      COPY fluent-bit/configs/large-logs.conf /fluent-bit/configs/large-logs.conf
      ```

4. **Build and Push New Image via Jenkins**
    - **Important**: Direct ECR push requires DevOps access
    - **Standard workflow**: Use Jenkins pipeline (Jenkinsfile)

   Steps:
    1. Create config file `/fluent-bit/configs/large-logs.conf`
    2. Update Dockerfile to COPY the new config
    3. Commit changes to a branch (e.g., `feature/large-logs-config`)
    4. Push branch → Jenkins builds and tags with **git commit SHA**
    5. Note the commit SHA from Jenkins build (e.g., `abc123def...`)
    6. Use that SHA as the image tag in Terraform (see Phase 3)

   **Do NOT merge to master yet** - this prevents the config from becoming `latest` and affecting other services

### Phase 2: Update Terraform-Fargate Module

**Location**: `/Users/tcalhoun/work/terraform-fargate/`

**Approach**: Use FireLens `config-file-value` to specify custom config while **keeping FireLens active**. This
maintains ECS metadata injection and is less invasive.

1. **Add Variable** (`variables.tf`)
   ```hcl
   variable "fluentbit_custom_config_file" {
     description = "Path to custom Fluent Bit config file within the container. Uses FireLens config-file-value option. Maintains FireLens functionality including ECS metadata injection."
     default     = ""
   }
   ```

2. **Update firelens_configuration_options Logic**

   Modify where `firelens_configuration_options` is constructed to conditionally add `config-file-type` and
   `config-file-value`:

   ```hcl
   locals {
     base_firelens_options = merge(
       {
         "enable-ecs-log-metadata" = "true"
       },
       var.additional_firelens_options  # existing user options
     )

     # Add config-file options if custom config specified
     firelens_configuration_options = var.fluentbit_custom_config_file != "" ? merge(
       local.base_firelens_options,
       {
         "config-file-type"  = "file"
         "config-file-value" = var.fluentbit_custom_config_file
       }
     ) : local.base_firelens_options
   }
   ```

3. **No Task Definition Template Changes Needed**

   The existing `firelensConfiguration` block remains unchanged:
   ```json
   "firelensConfiguration": {
     "type": "fluentbit",
     "options": ${jsonencode("${firelens_configuration_options}")}
   }
   ```

   The options map will automatically include `config-file-value` when set.

#### Option B: Simpler Approach - Bake Config into Custom Image

If we bake the memory config directly into the custom image and set it as the default:

1. **Update orchard-fluent-bit Image**
    - Modify the default config file that FireLens uses (`/fluent-bit/configs/parse-json.conf`)
    - Add SERVICE section with memory settings at the top
    - Keep existing OUTPUT configuration

2. **Use Custom Image in Terraform**
    - Services that need memory limits set `fluentbit_custom_image` to the memory-optimized image
    - No changes needed to terraform-fargate module itself

### Phase 3: Service Implementation

#### For Cerbos (Immediate Fix):

Update `/Users/tcalhoun/work/terraform-infra/permissions-platform/qa/cerbos/main.tf`:

```hcl
module "ows_service_fargate_environment" {
  source = "git@github.com:theorchard/terraform-fargate.git//?ref=6.4.1"

  # ... existing config ...

  # CRITICAL: Increase Fluent Bit memory to handle large logs
  fluentbit_task_cpu        = 256  # Increased from 128
  fluentbit_task_memory     = 512  # Increased from 128 (REQUIRED for 1.9MB logs)

  # Use large-logs optimized image (from Jenkins build)
  # Replace <COMMIT_SHA> with actual SHA from Jenkins (e.g., abc123def...)
  fluentbit_custom_image       = "086679231553.dkr.ecr.us-east-1.amazonaws.com/orchard-fluent-bit:<COMMIT_SHA>"

  # Specify custom config via FireLens (keeps FireLens active!)
  fluentbit_custom_config_file = "/fluent-bit/configs/large-logs.conf"

  # NOTE: No custom entrypoint needed!
  # FireLens remains active and handles:
  # - Automatic ECS metadata injection
  # - INPUT from application containers
  # - OUTPUT to Datadog (from existing config)
}
```

**Benefits of this approach:**

- ✅ FireLens stays active (ECS metadata, automatic routing)
- ✅ Custom INPUT buffer settings take effect
- ✅ Less invasive than custom entrypoint
- ✅ Maintains compatibility with existing FireLens features

**To get the commit SHA**:

1. Push orchard-fluent-bit branch
2. Check Jenkins build output
3. Look for "Create a Release" stage → shows ECR image tag
4. Use that SHA in the terraform config above

#### For Other Services with Large Logs:

```hcl
module "my_service" {
  source = "../terraform-fargate"

  # ... existing config ...

  fluentbit_task_memory        = 512  # Increase from default for large log buffering
  fluentbit_custom_image       = "086679231553.dkr.ecr.us-east-1.amazonaws.com/orchard-fluent-bit:<COMMIT_SHA>"
  fluentbit_custom_config_file = "/fluent-bit/configs/large-logs.conf"  # Uses FireLens config-file-value
}
```

#### For Services with Normal Log Sizes (Option B - Baked Config):

```hcl
module "my_service" {
  source = "../terraform-fargate"

  # ... existing config ...

  # Use default image and FireLens configuration
  # No changes needed
}
```

## Configuration Reference

### Fluent Bit Buffer and Memory Settings

| Setting                     | Location        | Purpose                                                   | Cerbos Value |
|-----------------------------|-----------------|-----------------------------------------------------------|--------------|
| `Buffer_Chunk_Size`         | INPUT section   | Incremental memory allocation size for receiving messages | 512KB        |
| `Buffer_Max_Size`           | INPUT section   | **Maximum size for a SINGLE message** (Forward plugin)    | **4MB**      |
| `Mem_Buf_Limit`             | INPUT section   | Total memory buffer limit per input plugin                | 20MB         |
| `storage.type`              | INPUT section   | Enable filesystem buffering (filesystem/memory)           | filesystem   |
| `storage.path`              | SERVICE section | Directory for filesystem buffer                           | /var/log/... |
| `storage.max_chunks_up`     | SERVICE section | Max chunks in memory for retries                          | 128          |
| `storage.backlog.mem_limit` | SERVICE section | Memory limit for backlog chunks                           | 10M          |

**Critical for Large Logs**: `Buffer_Max_Size` must be larger than your largest log message, or the message will be
silently dropped.

### Example Large-Logs Config for Cerbos

```ini
[SERVICE]
    Flush                     5
    Daemon                    Off
    Log_Level                 ${LOG_LEVEL}
    Parsers_File              /fluent-bit/parsers/parsers.conf
    storage.path              /var/log/flb-storage/
    storage.sync              normal
    storage.checksum          off
    storage.max_chunks_up     128
    storage.backlog.mem_limit 10M

[INPUT]
    Name              forward
    Listen            0.0.0.0
    Port              24224
    Buffer_Chunk_Size 512KB      # Incremental allocation
    Buffer_Max_Size   4MB        # MUST be > 1.9MB for Cerbos logs
    Mem_Buf_Limit     20MB       # Allow ~10 large logs in memory
    storage.type      filesystem # Overflow to disk

@INCLUDE /fluent-bit/configs/parse-json.conf

[OUTPUT]
    Name              datadog
    Match             *
    Host              ${DATADOG_HOST}
    TLS               on
    apikey            ${DATADOG_API_KEY}
    dd_service        ${DD_SERVICE}
    dd_source         ${DD_SOURCE}
    dd_tags           ${DD_TAGS}
    dd_message_key    log
    provider          ecs
    compress          gzip  # CRITICAL: Reduce payload size
```

## Datadog Limitations and Long-Term Solutions

### Datadog Log Size Limits

**CRITICAL**: Even after fixing Fluent Bit buffer limits, Datadog has strict size restrictions:

1. **Single Log Limit**: 1MB per individual log
    - Logs larger than 1MB are **truncated**
    - Cerbos's 1.9MB decision logs will be cut off
    - No error returned to Fluent Bit

2. **Payload Limit**: 5MB per HTTP request (uncompressed)
    - Returns HTTP 413 (Payload Too Large) if exceeded
    - `compress gzip` helps but doesn't solve single log truncation

### Impact on Cerbos

The immediate fix (Phase 1-3) will:

- ✅ Stop Fluent Bit from dropping logs
- ✅ Get logs to reach Datadog
- ❌ **But logs will still be truncated at 1MB** (missing ~900KB of data)

### Long-Term Solution Options

#### Option A: Reduce Cerbos Log Size (Recommended)

Configure Cerbos to produce smaller decision logs:

```yaml
# Cerbos audit config
audit:
  backend: local
  decision:
    enabled: true
    logIncludeMedatdata: false  # Reduce verbosity
    logIncludeInputs: false      # Don't log full input payloads
```

**Pros**:

- Logs arrive complete in Datadog
- Lower bandwidth and storage costs
- Better query performance

**Cons**:

- May lose debugging information
- Requires Cerbos configuration changes

#### Option B: Dual Logging Strategy

Send different data to different destinations:

1. **Full logs → S3** (via Firehose or custom OUTPUT)
    - Complete 1.9MB decision logs
    - Long-term archival
    - Query with Athena/CloudWatch Insights

2. **Metadata/summary → Datadog** (< 1MB)
    - User ID, decision outcome, timestamp
    - Key metrics and alerts
    - Fast querying in Datadog

Implementation via Fluent Bit FILTER plugin:

```ini
# Extract summary for Datadog
[FILTER]
    Name         lua
    Match        *
    script       extract_summary.lua
    call         extract_cerbos_summary

# Route full logs to S3
[OUTPUT]
    Name         s3
    Match        *
    bucket       cerbos-decision-logs
    region       us-east-1

# Route summaries to Datadog
[OUTPUT]
    Name         datadog
    Match        cerbos.summary
    compress     gzip
```

**Pros**:

- Best of both worlds
- Complete logs available when needed
- Efficient Datadog usage

**Cons**:

- More complex infrastructure
- Additional S3 costs
- Requires custom Lua filter

#### Option C: Fluent Bit Pre-Processing

Truncate or summarize logs before sending to Datadog:

```ini
[FILTER]
    Name         modify
    Match        *
    Condition    Key_Value_Matches log_size >1000000
    Remove       log
    Add          log [TRUNCATED: Log exceeded 1MB]
    Add          truncated true
```

**Pros**:

- Simple to implement
- Prevents silent truncation
- Logs arrival indicator in Datadog

**Cons**:

- Still lose data
- May not be acceptable for compliance

### Recommendation for Cerbos

**Short-term** (immediately):

1. Deploy Fluent Bit fixes to stop dropping logs
2. Monitor Datadog for truncated logs
3. Verify which data is being cut off

**Medium-term** (1-2 weeks):

1. Investigate reducing Cerbos decision log verbosity
2. If reduction not acceptable, implement Option B (dual logging)

**Monitoring**:

- Check Fluent Bit logs for `[in_fw] incoming data exceed limit` (should disappear)
- Check Datadog for incomplete decision logs (JSON parsing errors, missing fields)
- Monitor S3 if implementing dual logging

## Testing Plan

1. **Test Configuration Locally (Optional)**
   ```bash
   cd /Users/tcalhoun/work/orchard-fluent-bit
   docker build -t orchard-fluent-bit:local .
   docker run --rm orchard-fluent-bit:local /fluent-bit/bin/fluent-bit -c /fluent-bit/configs/large-logs.conf --dry-run
   ```
    - Verify no syntax errors
    - Check Buffer_Max_Size setting appears in output

2. **Build via Jenkins**
    - Push branch to GitHub (e.g., `feature/large-logs-config`)
    - Jenkins automatically builds and pushes to ECR
    - Note the commit SHA from Jenkins output
    - Verify image appears in ECR:
      `aws ecr describe-images --repository-name orchard-fluent-bit --image-ids imageTag=<SHA>`

3. **Deploy to Cerbos QA**
    - Update Cerbos terraform with commit SHA
    - Apply terraform changes
    - Monitor Fluent Bit CloudWatch logs for:
        - `[in_fw] incoming data exceed limit` (should NOT appear)
        - Successful startup without config errors
        - Memory usage patterns
    - Check Datadog for Cerbos decision logs appearing
    - **Verify truncation**: Check if 1.9MB logs are complete or truncated at 1MB

4. **Load Testing with Large Logs**
    - Generate burst of "complicated user" decisions
    - Monitor Fluent Bit container memory (should stay < 512MB)
    - Verify backpressure handling (check filesystem buffer usage)
    - Check for log loss or gaps in sequence
    - Monitor Datadog ingestion rates and errors

## Rollout Strategy

1. **Phase 1**: Create and build large-logs configuration
    - Create `/fluent-bit/configs/large-logs.conf` in orchard-fluent-bit repo
    - Update Dockerfile to COPY the config file
    - Commit to branch (e.g., `feature/large-logs-config`)
    - Push to GitHub → Jenkins builds automatically
    - Capture commit SHA from Jenkins build output
    - **Do NOT merge to master yet** (keeps it off `latest` tag)

2. **Phase 2**: Deploy to Cerbos QA (immediate need)
    - Update Cerbos terraform with:
        - Commit SHA-based image reference
        - Increased Fluent Bit memory (512MB)
        - Custom entrypoint and config path
    - Apply terraform changes
    - Monitor for 24-48 hours:
        - No more "incoming data exceed limit" errors
        - Logs appearing in Datadog (even if truncated)
        - Container memory stable < 512MB

3. **Phase 3**: Assess Datadog truncation impact
    - Analyze what data is being cut off at 1MB
    - Coordinate with Cerbos team to review truncated logs
    - Decide on long-term solution (Option A, B, or C)
    - Implement Cerbos log reduction or dual logging

4. **Phase 4**: Decide on master merge strategy
    - **Option A**: Keep on branch indefinitely
        - Services explicitly opt-in via commit SHA
        - Maintains backwards compatibility
    - **Option B**: Merge to master after validation
        - Becomes `latest` - affects all services using default image
        - Requires coordination and testing across services
    - **Recommendation**: Option A (branch-based, opt-in)

## Risks and Considerations

1. **Datadog 1MB Truncation (HIGH RISK)**
    - Even after Fluent Bit fixes, Datadog truncates logs at 1MB
    - Cerbos's 1.9MB logs will lose ~900KB of data
    - Requires long-term solution (reduce log size or dual logging)
    - **Impact**: Incomplete audit logs may affect compliance or debugging

2. **Container Memory Requirements**
    - Increasing Fluent Bit from 128MB to 512MB per task
    - Additional cost: ~$0.05/hour per task in Fargate
    - Must ensure ECS cluster has sufficient memory capacity

3. **Branch-Based Image Management** (NEW)
    - Using commit SHA instead of semantic tags (like `large-logs`)
    - Requires tracking which SHA has the desired config
    - Cannot use `latest` without affecting all services
    - Alternative: Ask DevOps to manually retag in ECR, but breaks GitOps principles

4. **FireLens Compatibility** (MITIGATED)
    - ✅ Using `config-file-value` keeps FireLens active
    - ✅ ECS metadata injection continues to work
    - ✅ Automatic INPUT/OUTPUT routing preserved
    - ⚠️ Custom config must not conflict with FireLens-generated config
    - ⚠️ Must use config path OTHER than `/fluent-bit/etc/fluent-bit.conf` (reserved for FireLens)

5. **Breaking Change**: Services using `log_configuration_options_override` for buffer settings will see errors

6. **Resource Limits**: Filesystem buffering requires ephemeral storage
    - Fargate provides 20GB ephemeral storage (usually sufficient)
    - Monitor `/var/log/flb-storage/` disk usage
    - Under extreme backpressure, may fill disk

7. **Silent Failures**: If Buffer_Max_Size is still too small
    - Logs will be dropped without errors sent to Datadog
    - Must monitor Fluent Bit CloudWatch logs for drop messages
    - Consider alerting on `incoming data exceed limit` log pattern

8. **Performance Impact**: Large logs increase:
    - Network bandwidth usage
    - Fluent Bit CPU usage (compression)
    - Datadog ingestion costs
    - Query performance in Datadog

## Alternative Approaches Considered

### 1. Use log_configuration_options_override (Current Approach - FAILED)

- **Status**: Doesn't work - memory settings aren't valid for OUTPUT plugins
- **Reason**: FireLens only passes these options to the output plugin

### 2. Modify FireLens Default Config in AWS

- **Status**: Not possible - FireLens uses hardcoded default configs
- **Reason**: AWS doesn't allow modifying FireLens built-in configurations

### 3. Use Sidecar with Volume Mount

- **Status**: Overly complex
- **Reason**: Would require separate config management and volume coordination

### 4. Direct ECR Push with Custom Tag

- **Status**: Requires DevOps "god mode" access
- **Reason**: Standard developers cannot push directly to ECR
- **Workaround**: Use Jenkins CI/CD with commit SHA tags (implemented in this plan)

## Documentation Updates Needed

1. **terraform-fargate README**
    - Document `fluentbit_custom_config_file` variable
    - Explain how it uses FireLens `config-file-value` option
    - Clarify that FireLens remains active (unlike custom entrypoint approach)
    - Provide memory-tuning guidelines for services with large logs
    - Document backwards compatibility (existing services unaffected)

2. **orchard-fluent-bit README**
    - Document available config files (`large-logs.conf` for 1MB+ logs)
    - Explain buffer settings and their impact
    - Provide troubleshooting guide for "incoming data exceed limit" errors
    - Document how to determine if custom config is needed

## Success Criteria

### Phase 1-2 (Immediate Fix)

- [ ] Created `/fluent-bit/configs/large-logs.conf` with 4MB Buffer_Max_Size
- [ ] Updated Dockerfile to COPY large-logs.conf
- [ ] Pushed branch to GitHub (feature/large-logs-config)
- [ ] Jenkins build completed successfully
- [ ] Noted commit SHA from Jenkins build
- [ ] Image available in ECR with commit SHA tag
- [ ] Cerbos QA terraform updated with commit SHA and 512MB memory
- [ ] Terraform apply successful
- [ ] Fluent Bit container starts without configuration errors
- [ ] No more `incoming data exceed limit` errors in Fluent Bit CloudWatch logs
- [ ] Cerbos decision logs appear in Datadog (even if truncated)

### Phase 3 (Long-term)

- [ ] Analyzed Datadog logs to determine truncation impact
- [ ] Decided on long-term solution (reduce size, dual logging, or accept truncation)
- [ ] If applicable: Implemented Cerbos configuration changes to reduce log size
- [ ] If applicable: Implemented dual logging to S3 + Datadog
- [ ] Complete decision logs available (either in Datadog or S3)
- [ ] Compliance requirements met for audit logging

### Documentation

- [ ] terraform-fargate README updated (if module changes needed)
- [ ] orchard-fluent-bit README documents large-logs configuration
- [ ] Runbook created for troubleshooting large log issues
- [ ] Cerbos team informed of truncation behavior

## References

### AWS and Fluent Bit Configuration

- [AWS Blog: How to set Fluentd and Fluent Bit input parameters in FireLens](https://aws.amazon.com/blogs/containers/how-to-set-fluentd-and-fluent-bit-input-parameters-in-firelens/)
- [Fluent Bit Documentation: Forward Input Plugin](https://docs.fluentbit.io/manual/data-pipeline/inputs/forward)
- [Fluent Bit Documentation: Buffering and Storage](https://docs.fluentbit.io/manual/administration/buffering-and-storage)
- [Fluent Bit Documentation: Memory Management](https://docs.fluentbit.io/manual/administration/memory-management)
- [ECS Task Definition Parameters](https://docs.aws.amazon.com/AmazonECS/latest/developerguide/task_definition_parameters.html)

### FireLens Configuration and Integration

- [Using FireLens with Fluent Bit (AWS Documentation)](https://docs.aws.amazon.com/AmazonECS/latest/developerguide/using_firelens.html)
- [FirelensConfiguration API Reference](https://docs.aws.amazon.com/AmazonECS/latest/APIReference/API_FirelensConfiguration.html)
- [AWS GitHub: aws-for-fluent-bit](https://github.com/aws/aws-for-fluent-bit)
- [AWS Samples: FireLens Examples](https://github.com/aws-samples/amazon-ecs-firelens-examples)
- [AWS Samples: FireLens Under the Hood](https://github.com/aws-samples/amazon-ecs-firelens-under-the-hood)

### Buffer Configuration

- [buffer params explanation · fluent/fluent-bit · Discussion #5719](https://github.com/fluent/fluent-bit/discussions/5719)
- [Allow to Increase Buffer_Chunk_Size and Buffer_Max_Size · Issue #5232](https://github.com/fluent/fluent-bit/issues/5232)

### Datadog Integration and Limits

- [Fluent Bit Datadog Output Plugin](https://docs.fluentbit.io/manual/data-pipeline/outputs/datadog)
- [Datadog Logs API Documentation](https://docs.datadoghq.com/api/latest/logs/)
- [[datadog] http output POST size (gzip) · Issue #1187](https://github.com/fluent/fluent-bit/issues/1187)
- [Send Fluent Bit Logs to Datadog](https://docs.datadoghq.com/logs/guide/fluentbit/)

### Related Issues

- [Optimizing Kubernetes Log Aggregation (Medium)](https://arteraai.medium.com/optimizing-kubernetes-log-aggregation-tackling-fluent-bit-buffering-and-backpressure-challenges-fb3129dc5031)
- [Avoiding data loss and backpressure problems with Fluent Bit](https://chronosphere.io/learn/avoiding-data-loss-and-backpressure-problems-with-fluent-bit/)