# terraform-datadog

## Overview
This module provides functionality to create Datadog timeboards, screenboards, and monitors for a given service or resource. It has defaults tailored for Orchard-specific infrastructure configurations. Each type of integration has its own module folder, so take note when configuring the source url.

## Usage

### Provider Settings

The IAM user running this terraform will need read access to the AWS secretsmanager secrets that specify the Datadog API and App keys.

### Variables

At a minimum, set the following variables for the module to function properly:
```
* environment
```

Additional variables with need to be set based on the type of integration, e.g.
```
* lambda_name
* cluster_name
* service_name

# Additional required variables for setting up monitors
*  - notification_endpoints
*  - escalation_notification_endpoints
```

### Monitor-Specific Notification Settings

This is currently only supported by the `service` module. Note that if the notification settings are overridden for a specific monitor, the settings for that monitor will be determined solely by the overrides (i.e. no settings will be inherited from the defaults).

Example:

```
module "service" {
  source = "git@github.com:theorchard/terraform-datadog.git//modules/service?ref=X.X.X"

  environment                       = var.environment
  service_name                      = var.service_name
  application_family                = var.application_family
  notification_endpoints            = "@slack-foo"
  escalation_notification_endpoints = "@slack-foo"

  # Trigger pagerduty alert for 5xx monitor only
  notification_overrides_5xx = {
    endpoints            = "@slack-foo @pagerduty"
    escalation_endpoints = "@slack-foo @pagerduty"
  }
}
```

### Examples

#### Microservice integration
```
module "ows_service_datadog_integration" {
  source       = "git@github.com:theorchard/terraform-datadog.git//modules/service?ref=X.X.X"
  environment  = "${var.environment}"
  service_name = "${var.service_name}"
}
```

#### Lambda integration
```
module "lambda_datadog_integration" {
  source       = "git@github.com:theorchard/terraform-datadog.git//modules/lambda?ref=X.X.X"
  environment  = "${var.environment}"
  service_name = "${var.lambda_name}"
}
```

#### RDS integration

```HCL
module "rds_datadog_integration" {
  source = "git@github.com:theorchard/terraform-datadog.git//modules/rds?ref=X.X.X"

  environment        = var.environment
  service_name       = var.service_name
  application_family = var.application_family

  notification_endpoints            = "@slack-my-team-channel"
  escalation_notification_endpoints = "@slack-my-team-channel"

  # > Optional monitor settings
  #
  # evaluation_period                 = "last_1h"
  # evaluation_delay                  = 240
  # no_data_timeframe                 = 2
}
```

#### Opensearch/Elasticsearch integration

```HCL
module "es_datadog_integration" {
  source = "git@github.com:theorchard/terraform-datadog.git//modules/open_search?ref=X.X.X"

  environment        = var.environment
  service_name       = var.service_name
  application_family = var.application_family

  notification_endpoints            = "@slack-my-team-channel"
  escalation_notification_endpoints = "@slack-my-team-channel"

  # > Optional monitor settings
  #
  # critical_number_5XX = 1
}
```

### StepFunctions StateMachine with Lambdas integration
```HCL
module "statemachine_datadog_integration" {
  source = "git@github.com:theorchard/terraform-datadog.git//modules/step-function?ref=X.X.X"

  statemachine_arn   = var.statemachine_arn

  environment        = var.environment
  service_name       = var.service_name
  application_family = var.application_family

  notification_endpoints            = "@slack-my-team-channel"
  escalation_notification_endpoints = "@slack-my-team-channel"

  # > Optional monitor settings
  #
  # > Log monitor with error message
  # failed_execution_message_log_monitor = {
  #   enabled = true
  #   log_group_name = var.sfn_log_group_name
  # }
  #
  # > Simple failed metric monitor
  # failed_execution_metric_monitor_enabled = true
}
```


### Kafka Connector integration
```HCL
module "connector_service_dashboard" {
  source = "git@github.com:theorchard/terraform-datadog.git//modules/kafka_connector?ref=X.X.X"

  environment        = var.environment
  service_name       = var.service_name
  application_family = var.application_family
  additional_tags    = var.additional_tags

  notification_endpoints            = "@slack-my-team-channel"
  escalation_notification_endpoints = "@slack-my-team-channel"

  monitor_kafka_topics = ["topic1", "topic2"]
  monitor_kafka_topics_cluster_name = "qa-managed-kafka-cdc-destination"

  # > Optional monitor settings

  healthy_tasks_monitor_enabled = true
  healthy_tasks_evaluation_window = "last_15m" # Optional. Default is 5 min
  healthy_tasks_critical_number = 0.1 # Optional. Default is 0.1

  service_cpu_monitor_enabled = true
  service_cpu_time_window = "last_15m" # Optional. Default is 5 min
  service_cpu_critical_number = 80 # Optional. Default is 90 percentage.

  failed_tasks_monitor_enabled = true
  critical_number_failed_tasks = 2 # Optional. Default is 2, so it will raise alert if there are 2 or more failed tasks.

  successful_tasks_monitor_enabled = true
  critical_number_successful_tasks = 1 # Optional. Default is 1.
  successful_tasks_group_by = "account_id_to_backup" # Optional. Default is null
  successful_tasks_query_interval = "last_25h" # Optional. Default is last_1d

  error_monitor_enabled = true
  error_monitor_critical_number_errors = 1 # Optional. Default is 2 error in logs.
  error_monitor_query_interval = "15m" # Optional. Default is 5m. Note this does not have last_ prefix.

  dlq_topic_name = "dlq.musicGraph.mysql.nr" # Optional. Default is null. If you specify this it will notify if there is any message in DLQ topic.

}
```

### Team integration

```
module "datadog_team" {
  source       = "git@github.com:theorchard/terraform-datadog.git//modules/team?ref=X.X.X"

  team_handle        = var.application_family
  team_name          = "Your Descriptive Team Name"
  team_members       = var.team_members
  team_slack_channel = "Your Team Slack channel for notifications" 
}
```

### Elasticache Monitoring and Dashboards

Redis is supported by default. Memcached is not yet supported.

To create for Elasticache Redis:
```
module "datadog_elasticache" {
  source       = "git@github.com:theorchard/terraform-datadog.git//modules/elasticache?ref=X.X.X"

  environment        = var.environment
  service_name       = var.service_name
  application_family = var.application_family
  teams              = var.teams

  notification_endpoints            = "@slack-my-team-channel"
  escalation_notification_endpoints = "@slack-my-team-channel"
}
```

(In the future) To create for Elasticache Memcached:
```
module "datadog_elasticache" {
  source       = "git@github.com:theorchard/terraform-datadog.git//modules/elasticache?ref=X.X.X"

  environment        = var.environment
  service_name       = var.service_name
  application_family = var.application_family
  teams              = var.teams

  engine                            = "memcached"
  notification_endpoints            = "@slack-my-team-channel"
  escalation_notification_endpoints = "@slack-my-team-channel"
}
```

### S3 Monitoring and Dashboards

By default, the S3 module will create a dashboard with some summary metrics.
```
module "datadog_s3" {
  source       = "git@github.com:theorchard/terraform-datadog.git//modules/s3?ref=X.X.X"

  environment        = var.environment
  service_name       = var.service_name
  application_family = var.application_family
  teams              = var.teams

  notification_endpoints            = "@slack-my-team-channel"
  escalation_notification_endpoints = "@slack-my-team-channel"
}
```

A number of optional dashboard sections and monitors can be enabled if the bucket is configured accordingly. See `variables.tf` for more information on optional settings and the prerequisites for enabling them.
```
module "datadog_s3" {
  source       = "git@github.com:theorchard/terraform-datadog.git//modules/s3?ref=X.X.X"

  environment        = var.environment
  service_name       = var.service_name
  application_family = var.application_family
  teams              = var.teams

  notification_endpoints            = "@slack-my-team-channel"
  escalation_notification_endpoints = "@slack-my-team-channel"
  
  dashboard_configuration = {
    enabled                       = true
    show_advanced_storage_metrics = true
    show_replication_metrics      = true
    show_activity_metrics         = true
  }

  replication_failures_monitor_configuration = {
    enabled = true
  }

  replication_latency_monitor_configuration = {
    enabled = true
  }
}
```