# terraform-airflow ## Overview > [!WARNING] > Breaking changes! Starting with version 3.0.0, the module has been refactored to use a DNS provider for Route53 record creation. The following changes have been made: > > 1. The previous `aws_route53_record.airflow_record` resource has been removed, and only the DNS provider-based record is now used. > 2. The variable `networking_route53_zone_id` has been renamed to `override_route53_zone_id`. > 3. Route53 records are now conditionally created based on the `route53_record_creation_enabled` variable. > 4. By default, Route53 records are created for all environments. > > The value for the Route53 zone ID is now determined automatically based on the environment: > - For the development environment, it defaults to `Z21XEY26C989RH`, which is the zone ID for `dev.theorchard.io` in the Orchard development account. > - For all other environments, the default is `Z0183645HDT0XCWHLW7S`, corresponding to the `theorchard.io` zone in the networking account. > - If you need to create records in a different account or zone, please specify the appropriate zone ID using the `override_route53_zone_id` parameter. > > In the development and shared account, you may need to use `moved` blocks to migrate existing resources. See the [migration guide](#migration-guide) for more information. > > See [Providers](#providers) for more information. > [!WARNING] > Breaking changes! Starting with version 2.0.0, the module has been refactored to use a second provider for DNS record creation. This provider should be defined in the same directory as the module and should use the `networking` profile. See [Providers](#providers) for more information. This module provisions an AWS managed airflow with empty DAG's and no custom plugins. In order to add custom DAG's and new plugins you should use special jenkins job created for your particular airflow usage. Example of existing airflow deployment jenkins job : - Deploy: https://pipeline.theorchard.io/job/ows-royalties-workflows-airflow-deploy/ - Healthcheck: https://pipeline.theorchard.io/job/ows-royalties-workflows-airflow-healthcheck/ ## Usage ##### Environment name This module creates an airflow environment with the name `${var.environment}-${var.service_name}`, meaning that `${service_name}` should be set to an environment-neutral name, ideally matching the name of the service or application. ##### VPC settings Airflow runs in AWS VPC, so make sure to specify correct VPC id and subnets id's. VPC setting should be chosen accordingly to environment. Managed airflow should have two subnets in two different availability zones. - `${var.vpc_id}` - `${var.vpc_subnet_ids}` ##### S3 settings - `${var.bucket_name}` contains name of s3 bucket that will contain DAG's, requirements.txt and plugins.zip. By default, all path variables will be place files to root folder of the bucket. - `${var.bucket_dags_path}` path to folder that contain DAG's - `${var.bucket_plugins_path}` path to folder that contain plugins.zip - `${var.bucket_requirements_path}` path to folder that contain requirements.txt ##### MWAA configuration options Managed Airflow has configuration options that are described here: https://docs.aws.amazon.com/mwaa/latest/userguide/configuring-env-variables.html#configuring-env-variables-reference. Inside the module they should be put into `${var.airflow_configuration_options}` as a map of strings. Example: ``` { "core.default_task_retries" = 0 "core.parallelism" = 4 "webserver.default_ui_timezone" = "America/New_York" "celery.worker_autoscale" = "8,2" "custom.ows_env" = var.environment "custom.snowflake_connection_name" = var.snowflake_connection_name "custom.s3_sales_bucket_name" = var.s3_sales_bucket_name "custom.s3_static_data_bucket_name" = var.s3_static_data_bucket_name "secrets.backend" = var.secrets_backend "secrets.backend_kwargs" = "{\"connections_prefix\" : ${local.secrets_prefix}}" } ``` ### Providers The module requires an explicitly defined AWS provider to create DNS records in the shared networking account. This provider must be defined in the same directory as the module and should use the `networking` profile. The provider should be defined as follows: ```hcl provider "aws" { region = var.aws_region } provider "aws" { region = var.aws_region alias = "networking" profile = "networking" } # Optional: If you need to use a specific Route53 zone data "aws_route53_zone" "networking_route53_zone" { provider = aws.networking name = "dev.theorchard.io" } module "airflow" { source = "git@github.com:theorchard/terraform-airflow.git//?ref=3.0.0" providers = { aws.dns = aws.networking } # Optional: Override the default Route53 zone # override_route53_zone_id = data.aws_route53_zone.networking_route53_zone.id # Optional: Control Route53 record creation # route53_record_creation_enabled = true # Set to false to disable Route53 record creation entirely // other settings } ``` ##### DNS settings The DNS zone for your Airflow webconsole is determined automatically based on the environment: - For the development environment, it defaults to `dev.theorchard.io` in the Orchard development account. - For all other environments, it defaults to `theorchard.io` in the networking account. If you need to use a different zone, you can override this by setting the `override_route53_zone_id` variable. By default, Route53 records are created for all environments. You can control this behavior with the `route53_record_creation_enabled` variable - set it to `false` to disable Route53 record creation entirely. In the end, when Route53 records are created, you will get a DNS address for your webconsole in the format: `${var.environment}-${var.service_name}.domain.com` ### Testing To test your changes: 1. Go in to the `test` directory and add your changes in to the respective files 2. Use the aws creds generator (a.k.a. jerry script) and generate creds for aws dev 3. Run `terraform plan` and ensure nothing is out of place and then run `terraform apply` 4. Once the apply is successful, be sure to run `terraform destroy` to tear all the test infraustructure. If you went through all the steps with no errors then your changes are good to go! ### Migration Guide #### Upgrading from version 2.x to 3.x In the development environment and in the shared environment when you upgrade from 2.x to 3.x, you may need to use the following `moved` block to migrate existing resources: ```hcl moved { from = module.airflow.aws_route53_record.airflow_record to = module.airflow.aws_route53_record.networking_airflow_record[0] } ``` Note that the `networking_airflow_record` resource now uses a count parameter, so you need to reference it as `networking_airflow_record[0]` in the moved block. In order to determine if you need to use this `moved` block, check terraform plan output. If you see something like this: ``` # module.airflow.aws_route53_record.airflow_record will be destroyed # (because aws_route53_record.airflow_record is not in configuration) - resource "aws_route53_record" "airflow_record" { - fqdn = "dev-airflow.dev.theorchard.io" -> null - id = "Z21XEY26C989RH_dev-airflow_CNAME" -> null - name = "dev-airflow" -> null - records = [ - "https://12345abcde.us-east-1.airflow.amazonaws.com", ] -> null - ttl = 300 -> null - type = "CNAME" -> null - zone_id = "Z21XEY26C989RH" -> null } # module.airflow.aws_route53_record.networking_airflow_record[0] will be created + resource "aws_route53_record" "networking_airflow_record" { + count = 1 + fqdn = (known after apply) + id = (known after apply) + name = "dev-airflow" + records = [ + "https://12345abcde.us-east-1.airflow.amazonaws.com", ] + ttl = 300 + type = "CNAME" + zone_id = "Z21XEY26C989RH" } ``` Then you need to use the `moved` block above. If you don't see this in the plan output, then you don't need to use the `moved` block. You need to look for the resource name `aws_route53_record.airflow_record` in the plan output being destroyed and `aws_route53_record.networking_airflow_record[0]` being created. Note the index `[0]` which is due to the count parameter that was added to conditionally create the record.