# Python Repo Standardization Guide

## Repo Type Classification

| Prefix | Type | Examples |
|---|---|---|
| `ows-` | Microservice | `ows-vectororder`, `ows-assets`, `ows-delivery-metadata` |
| `python-` | Library | `python-ows-assets-uploader`, `python-vector-utils` |
| `daemon-` | Daemon | `daemon-transcoding`, `daemon-video` |
| `lambda-` | Lambda monorepo | `lambda-vector`, `lambda-assets`, `lambda-metadata-consistency` |
| `scripts-` | Scripts | `scripts-vector`, `scripts-assets` |

---

## Universal Standards

These apply to every Python repo regardless of type.

### README

- Exists at project root, or per component in lambda and scripts repos.
- Must explain how a developer can set up and run the application by doing only two things:
  1. Set up any secrets needed
  2. Run a `docker compose` command
- Org-specific instructions that may change (e.g., how to get QA access, how to authenticate to AWS) should be documented as links to the relevant internal guides, not duplicated inline.

### Package Manager: Poetry 2.x

- Poetry is the sole package manager. No `requirements.txt`, `requirements-dev.txt`, `setup.py`, or `setup.cfg`.
- `poetry.lock` is always committed.
- Lock file updates, formatting, and all checks run inside Docker — never directly on the host.
- Poetry version (2.x.x) must be explicitly pinned in the Dockerfile.
- Private PyPI source is always declared.

### Dependency Versioning

Declare only top-level dependencies in `pyproject.toml`. Unless a specific version is required for a known compatibility reason, use compatible-version ranges rather than exact pins. Exact transitive dependency versions are tracked automatically in `poetry.lock`.

```toml
# preferred — allows any compatible minor/patch update
"boto3~=1.35"

# acceptable when a specific version is required
"some-package == 1.4.4"
```

Note that using the `~=` operator to specify a minor version is preferred. This allows `~=1.0` to resolve to all `1.x.x` versions.

### pyproject.toml Structure

Use the PEP 517 `[project]` table style throughout. The `[tool.poetry]` section is only for package-mode and source declarations.

```toml
[project]
name = "repo-name"
version = "0.1.0"
description = "Short description."
authors = [
    {name = "Sony Music PDE"},
]
requires-python = ">=3.12,<4.0"
dependencies = [
    "some-package~=1.0",
]

[project.optional-dependencies]
dev = [
    "mypy~=1.9",
    "ruff~=0.6",
    "pytest~=8.3",
    "pytest-cov~=4.1",
    "pytest-mock~=3.13",
    "yamllint~=1.37",
]

[tool.poetry]
package-mode = false  # omit for libraries; set packages = [...] instead

[[tool.poetry.source]]
name = "PyPI"
priority = "primary"

[[tool.poetry.source]]
name = "pde"
url = "https://pypi.theorchard.io/pypi/"
priority = "supplemental"

[build-system]
requires = ["poetry-core"]
build-backend = "poetry.core.masonry.api"

[tool.ruff]
line-length = 120

[tool.ruff.lint]
select = ["C", "E", "F", "W", "B", "I", "T20", "TID252", "G"]
ignore = ["C901", "E501", "B007", "B008", "B009"]

[tool.ruff.lint.isort]
combine-as-imports = true
forced-separate = ["tests"]

[tool.mypy]
strict = true
show_error_codes = true
disallow_untyped_defs = true

[[tool.mypy.overrides]]
module = [
    "some_package_without_stubs.*",
]
ignore_missing_imports = true

[tool.pytest.ini_options]
addopts = [
    "--strict-config",
    "--strict-markers",
]
xfail_strict = true
junit_family = "xunit2"

[tool.coverage.report]
exclude_also = [
    "@abstractmethod",
    "@abc.abstractmethod",
    "if TYPE_CHECKING",
    "raise NotImplementedError",
]
```

### Tooling

| Tool | Role | Config location |
|---|---|---|
| `ruff` | Linting + formatting | `[tool.ruff]` in `pyproject.toml` |
| `mypy` | Static type checking | `[tool.mypy]` in `pyproject.toml` |
| `pytest` | Test runner | `[tool.pytest.ini_options]` in `pyproject.toml` |
| `yamllint` | YAML linting (docker-compose files) | inline via `poetry run yamllint` |

No `flake8`, `black`, `isort`, `setup.cfg`, `.flake8`, or `pytest.ini` files.

### Makefile

No `Makefile` exists in any repo. No `make` command is used in Jenkinsfiles. CI and local workflows invoke `docker compose` directly.

### Line Endings (`.gitattributes`)

Every repo must include a `.gitattributes` file at the root:

```
* text=auto eol=lf
```

This normalizes all text files to LF on commit and checkout regardless of OS or editor settings. Without it, Windows developers (whose git defaults to `core.autocrlf=true`) will check out shell scripts with CRLF line endings, causing `bad interpreter` errors when those scripts run inside Linux containers.

### `.gitignore`

Every repo must include a `.gitignore` at the root (or per component in lambda and scripts repos) with at minimum:

```
.env
__pycache__
.pytest_cache
.ruff_cache
build/
.claude/
```

- `build/` — test artifacts (`pyunit.xml`, `coverage.xml`) written by the `unit-lint` container; should not be committed
- `.claude/` — Claude Code project settings; local to each developer's machine

### Scripts and Local Dev (`scripts/` and `localdev/`)

Two directories serve distinct purposes:

- **`scripts/`** — tooling scripts invoked by Docker (entrypoints for `unit-lint`, `format`, `update-lockfile` stages). These are the shell scripts that run inside containers.
- **`localdev/`** — developer convenience scripts for local use (e.g., seeding a local DB, generating test fixtures, running ad-hoc queries). These never run in CI.

All scripts must be executable. Run `chmod +x scripts/*.sh` to set the bit on disk, then track the mode in version control with `git add scripts/*.sh` (git records the executable bit on new files automatically when it is set before staging).

> Note on the <source_dir> placeholder - in microservices this is often the name of the repo without the `ows-` prefix. In lambdas, scripts, and libraries it is often `src`. If it is unclear, prompt the user to provide this value.

**`scripts/unit-lint.sh`**
```bash
#!/usr/bin/env bash

set -e

: "${SKIP_LINT:=0}"
: "${SKIP_TYPE_CHECKS:=0}"
: "${SKIP_TESTS:=0}"

if [ "$SKIP_LINT" -ne 1 ]; then
    echo 'Running linting...'
    poetry run yamllint -s docker-compose.yml
    poetry run ruff check .
    echo 'Running formatter check...'
    poetry run ruff format . --diff --no-cache
fi

if [ "$SKIP_TYPE_CHECKS" -ne 1 ]; then
    echo 'Running mypy...'
    poetry run mypy --cache-dir=/dev/null <source_dir> tests
fi

if [ "$SKIP_TESTS" -ne 1 ]; then
    echo 'Running tests...'
    mkdir -p build
    COVERAGE_FILE=build/.coverage \
        poetry run pytest tests \
            $TEST_ARGS \
            --ignore=tests/integration/ \
            -p no:cacheprovider \
            --cov <source_dir> \
            --cov-report xml:build/coverage.xml \
            --cov-report term \
            --junitxml=build/pyunit.xml
fi

echo 'Done!'
```

**`scripts/format.sh`**
```bash
#!/usr/bin/env bash

set -e

poetry run ruff check --fix --no-cache .
poetry run ruff format --no-cache .
```

**`scripts/update-lockfile.sh`**
```bash
#!/usr/bin/env bash

set -e

if [ ! -s poetry.lock ]; then
    # Lock file is missing or empty; remove it so poetry generates one from scratch
    rm -f poetry.lock
    poetry lock
else
    poetry update --lock $1
fi
```

**`scripts/integration-test.sh`** *(only if `tests/integration/` exists)*
```bash
#!/usr/bin/env bash

set -e

echo 'Running integration tests...'
poetry run pytest tests/integration/ \
        $TEST_ARGS \
        -p no:cacheprovider

echo 'Done!'
```

### Dockerfile

- One `Dockerfile` per repo, or one per function directory under `lambda/` in lambda monorepos.
- Source code is always copied in **after** all packages are installed.
- Base image always comes from `086679231553.dkr.ecr.us-east-1.amazonaws.com/docker-parent-images`.
- `PYTHONUNBUFFERED=1` and `PYTHONDONTWRITEBYTECODE=1` are always set.
- A virtual environment is used inside the container. Its `bin/` directory is added to `$PATH`.
- Use the `worker` user wherever possible. Switch to `root` only for package installation, then switch back to `worker` before copying application files so file ownership is correct.
- Do not expose ports below 1024.

**Stage hierarchy:**
```
base
├── deploy
├── dev                  (not applicable to lambda functions)
├── update-lockfile
└── utility
    ├── unit-lint
    ├── format
    └── integration-test (only if the repo has tests/integration/)
```

Packages needed for testing and linting are never installed outside the `utility` layer. The `deploy` stage installs only main dependencies. Do not add an `integration-test` stage if the repo has no `tests/integration/` directory.

### Docker Compose

- One `docker-compose.yml` per repo, or one per function directory under `lambda/` in lambda monorepos.
- Do not expose ports below 1024.
- `platform` is set on every service.

**Required services:**

| Service | Applicable to | Port | Volume |
|---|---|---|---|
| `dev` | All except lambda functions | Yes (≥1024) | Source code mounted; AWS creds env vars passed |
| `deploy` | All except lambda functions | No | None |
| `unit-lint` | All | No | `./build:/var/app/build` (so `pyunit.xml` and `coverage.xml` are available after the container exits) |
| `format` | All | No | Source code mounted (so formatted files are written back to host) |
| `update-lockfile` | All | No | `./:/var/task` mounted for lambdas / `./:/var/app` for non-lambdas (directory mount required — see note below on `poetry.lock` bootstrapping) |
| `integration-test` | Only if `tests/integration/` exists | No | None |

**Additional services (repo-specific):**

Repos whose unit tests require databases or other backing services may add those as additional services in `docker-compose.yml`. Use `tmpfs` for ephemeral storage and `healthcheck` + `condition: service_healthy` so the `unit-lint` service waits for them to be ready before running tests.

```yaml
  test-db:
    platform: linux/amd64
    image: 086679231553.dkr.ecr.us-east-1.amazonaws.com/docker-database-images:<image-name>
    pull_policy: always
    environment:
      DB_USER: ${TEST_DB_USER}
      DB_PASSWORD: ${TEST_DB_PASS}
    tmpfs:
      - /var/lib/mysql
    healthcheck:
      interval: 1s
      timeout: 5s
      retries: 180
```

**App-specific environment variables:**

The env var lists shown for `dev` and `unit-lint` are minimal examples. Add app-specific variables (DB credentials, feature flags, etc.) as needed.

**`.env.shadow` pattern for repos with test databases:**

Repos that run dockerized databases for unit tests commit a `.env.shadow` file containing the test DB credentials. This file is safe to commit because the credentials are only valid inside the ephemeral test containers (not real QA/prod values).

docker-compose picks up `${VAR}` passthrough values from `.env`. The test container credentials live in `.env.shadow`, so CI and local dev copy it before running tests:

```groovy
// Jenkinsfile — Unit Tests and Style Checks stage
sh 'cp .env.shadow .env'
sh 'mkdir -p build && chmod a+w build'
sh 'docker compose run --rm --build unit-lint'
```

`.env` is gitignored; `.env.shadow` is committed. Developers run `cp .env.shadow .env` once after cloning, then fill in any real secrets (API keys, etc.) that are intentionally left blank in `.env.shadow`.

### Environment Variables

- Configuration is set via environment variables.
- Libraries that load `.env` files (e.g., `python-dotenv`) must not be part of production code; confine them to dev-only dependencies if used at all.
- Environment variable names use `CAPITAL_CASE`. The sole exception is `Environment` (lowercase `e` — legacy convention, retained for backward compatibility).

### Mypy Migration

When adding mypy to an existing repo that has no type annotations, set `SKIP_TYPE_CHECKS=1` in the `unit-lint` docker-compose service and commit it. Add the goal of enabling mypy to your backlog and flip to `SKIP_TYPE_CHECKS=0` once `poetry run mypy` passes clean.

Packages that ship without type stubs need `[[tool.mypy.overrides]]` entries with `ignore_missing_imports = true`. Add them for every internal package and any third-party package that triggers a `Cannot find implementation or library stub` error.

---

## Per-Type Patterns

### Microservices (`ows-`)

Flask or FastAPI applications deployed as ECS Fargate services. One application module per repo.

**`pyproject.toml` notes:**
- `package-mode = false`
- No `packages` declaration
- Include `uWSGI` or `uvicorn` in main dependencies as appropriate
- Include `sentry-sdk[flask]` or `sentry-sdk[fastapi]` as appropriate
- If `tests/integration/` has a `requirements-*.txt` with deps not already in `dev`, migrate them to an `integration` extras group:
  ```toml
  [project.optional-dependencies]
  dev = [...]
  integration = [
      "requests~=2.32",
      # only deps required for integration tests that are not in dev
  ]
  ```
  Then in the `integration-test` Dockerfile stage, add `RUN poetry install --extras integration --no-root --no-cache` after inheriting from `dev`. If no extra deps are needed, no additional install step is required.

**Extra asset directories (e.g., `spec/`):**

If the repo ships a directory that is needed at runtime (e.g., a `spec/` directory containing OpenAPI schemas loaded by the application), copy it in the `deploy` and `dev` Dockerfile stages. Only add it to `unit-lint` if tests actually reference files in it.

**Dockerfile**

Multi-stage. The `base` stage installs Poetry into an isolated venv; all subsequent stages inherit from it.

```dockerfile
FROM 086679231553.dkr.ecr.us-east-1.amazonaws.com/docker-parent-images:python312 AS base

ENV PYTHONUNBUFFERED=1 \
    PYTHONDONTWRITEBYTECODE=1 \
    POETRY_HOME="/opt/poetry" \
    PYSETUP_PATH="/opt/pysetup" \
    POETRY_VIRTUALENVS_IN_PROJECT=1 \
    POETRY_VIRTUALENVS_CREATE=1 \
    VIRTUAL_ENV="/opt/pysetup/.venv"
ENV PATH="$POETRY_HOME/bin:$VIRTUAL_ENV/bin:$PATH"

USER worker

WORKDIR $VIRTUAL_ENV

RUN python -m venv $VIRTUAL_ENV && \
    $VIRTUAL_ENV/bin/pip install --no-cache-dir --upgrade pip setuptools && \
    $VIRTUAL_ENV/bin/pip install --no-cache-dir poetry==2.3.2

WORKDIR /var/app
COPY pyproject.toml ./

################################
### Generate poetry lockfile ###
################################
FROM base AS update-lockfile

COPY scripts/update-lockfile.sh ./scripts/
ENTRYPOINT ["scripts/update-lockfile.sh"]

###########################
### Build image for dev ###
###########################
FROM base AS dev

WORKDIR $PYSETUP_PATH
COPY pyproject.toml poetry.lock ./

# If any dependency requires compilation (e.g., uWSGI), wrap the install in a
# gcc block. Include apt-get update in the same RUN to avoid stale package lists:
#
# USER root
# RUN apt-get update && \
#     apt-get -y install gcc && \
#     poetry install --extras dev --no-root --no-cache && \
#     apt-get purge -y gcc && \
#     apt-get -y autoremove
# USER worker
RUN poetry install --extras dev --no-root --no-cache

USER worker

WORKDIR /var/app
COPY <source_dir> ./<source_dir>
COPY application.py dev.py docker-compose.yml ./
EXPOSE 5000
CMD ["python", "dev.py"]

##############################
### Build image for deploy ###
##############################
FROM base AS deploy

WORKDIR $PYSETUP_PATH
COPY pyproject.toml poetry.lock ./

# Same gcc rule as dev: if any main dependency requires compilation (e.g., uWSGI),
# wrap the install in a gcc block:
#
# USER root
# RUN apt-get update && \
#     apt-get -y install gcc && \
#     poetry install --only main --no-root --no-cache && \
#     apt-get purge -y gcc && \
#     apt-get -y autoremove
# USER worker
RUN poetry install --only main --no-root --no-cache

USER worker

WORKDIR /var/app
COPY <source_dir> ./<source_dir>
COPY application.py uwsgi-start.sh uwsgi.ini ./

EXPOSE 8080
ENTRYPOINT ["./uwsgi-start.sh"]

#################################
### Build image for unit-lint
#################################
FROM dev AS unit-lint
ENV RUFF_CACHE_DIR=/tmp/.ruff_cache

COPY tests ./tests
COPY scripts/unit-lint.sh ./scripts/

USER worker
ENTRYPOINT ["scripts/unit-lint.sh"]

##########################
### Format source code ###
##########################
FROM dev AS format

COPY tests ./tests
COPY scripts/format.sh ./scripts/

USER worker
ENTRYPOINT ["scripts/format.sh"]

########################################
### Build image for integration-test ###
########################################
# Only include this stage if tests/integration/ exists in the repo.
# If tests/integration/ has a requirements-*.txt with deps beyond what is in
# [project.optional-dependencies].dev, add an `integration` extras group to
# pyproject.toml and install it here. Otherwise, no additional install is needed.
FROM dev AS integration-test

COPY tests/integration ./tests/integration
COPY scripts/integration-test.sh ./scripts/

USER worker
ENTRYPOINT ["scripts/integration-test.sh"]
```

**`docker-compose.yml`**

```yaml
---
services:
  dev:
    platform: linux/amd64
    build:
      context: .
      target: dev
    volumes:
      - ./<source_dir>:/var/app/<source_dir>
    ports:
      - "5000:5000"
    environment:
      - AWS_ACCESS_KEY_ID
      - AWS_SECRET_ACCESS_KEY
      - AWS_SESSION_TOKEN
      - AWS_REGION
    env_file:
      - path: ./.env
        required: false

  deploy:
    platform: linux/amd64
    build:
      context: .
      target: deploy
    ports:
      - "5000:5000"
    environment:
      - AWS_ACCESS_KEY_ID
      - AWS_SECRET_ACCESS_KEY
      - AWS_SESSION_TOKEN
      - AWS_REGION

  unit-lint:
    platform: linux/amd64
    build:
      context: .
      target: unit-lint
    environment:
      - SKIP_LINT=0
      - SKIP_TYPE_CHECKS=0
      - SKIP_TESTS=0
      - TEST_ARGS=
    volumes:
      - ./build:/var/app/build

  format:
    platform: linux/amd64
    build:
      context: .
      target: format
    volumes:
      - ./:/var/app

  update-lockfile:
    platform: linux/amd64
    build:
      context: .
      target: update-lockfile
    volumes:
      - ./:/var/app

  # Only include if tests/integration/ exists in the repo.
  # Add QA-specific environment variables as needed (URLs, DB credentials, AWS creds).
  integration-test:
    platform: linux/amd64
    build:
      context: .
      target: integration-test
    environment:
      - Environment=qa
      - QA_BASE_URL
      - AWS_ACCESS_KEY_ID
      - AWS_SECRET_ACCESS_KEY
      - AWS_SESSION_TOKEN
```

**Jenkinsfile CI stages:**
1. `Load Shared Libraries`
2. `Compliance Checks` — `complianceChecks()`
3. `Validate Software Catalog Definition` — `datadogSoftwareCatalogValidate()`
4. On PR: `Static Application Security Tests`, `Sonar Scan and Analysis`, `Unit Tests and Style Checks` (parallel)
5. On PR: `Create and Docker Scan a Release`
6. On merge: `Retag and Scan`, `Deploy to QA`, `Integration Tests` *(if applicable)*, `E2E Tests`, `Load Tests`, `Deploy to Prod`
7. `Publish Software Catalog Definition`

In CI, invoke docker compose directly — no `make`. The `build/` directory must be created and made world-writable before the container runs so the Jenkins agent can read the resulting artifact files:
```groovy
sh 'mkdir -p build && chmod a+w build'
sh 'docker compose run --rm --build unit-lint'
```

The integration test stage runs on merge to master only, after `Deploy to QA`. Repos that test against QA resources will need AWS credentials and/or secrets injected via `withAWS` or `withSecrets`:
```groovy
stage('Integration Tests') {
    when {
        expression { !isPullRequest() }
    }
    steps {
        withEcr {
            sh 'docker compose run --rm --build integration-test'
        }
    }
    post {
        cleanup {
            sh 'docker compose down --remove-orphans'
        }
    }
}
```

---

### Libraries (`python-`)

Importable packages published to the internal PyPI. No application entrypoint.

**`pyproject.toml` notes:**
- Remove `package-mode = false`; instead declare `packages`:
  ```toml
  [tool.poetry]
  packages = [
      {include = "<package_dir>"},
      {include = "<package_dir>/py.typed"},
  ]
  ```
- Include `bump2version` in dev dependencies for release management
- Add `py.typed` marker file to the package directory for PEP 561 compliance

**Dockerfile**

Same multi-stage pattern as microservices. No `dev` or `deploy` stage — only `unit-lint`, `format`, and `update-lockfile`.

```dockerfile
FROM 086679231553.dkr.ecr.us-east-1.amazonaws.com/docker-parent-images:python312 AS base

# ... (same base ENV, Poetry install as microservice) ...

################################
### Generate poetry lockfile ###
################################
FROM base AS update-lockfile
# ... same as microservice ...

#################################
### Build image for unit-lint
#################################
FROM base AS unit-lint

WORKDIR $PYSETUP_PATH
COPY pyproject.toml poetry.lock ./

RUN poetry install --no-root --no-cache

WORKDIR /var/app
COPY <package_dir> ./<package_dir>
COPY tests ./tests
COPY scripts/unit-lint.sh ./scripts/

USER worker
ENTRYPOINT ["scripts/unit-lint.sh"]

##########################
### Format source code ###
##########################
FROM base AS format

# ... same install as unit-lint ...
COPY scripts/format.sh ./scripts/

USER worker
ENTRYPOINT ["scripts/format.sh"]
```

**Jenkinsfile CI stages:**
1. `Load Shared Libraries`
2. `Compliance Checks`
3. `Validate Software Catalog Definitions`
4. `Unit Tests and Style Checks`
5. `Static Application Security Tests`
6. `Sonar Scan and Analysis` (master only)
7. `Create a release` (master only, when `VERSION` param set) — triggers `publish-pypi-package-v2`
8. `Publish Software Catalog Definition` (master only)

---

### Daemons (`daemon-`)

Long-running processes that consume from queues. Structurally identical to microservices but with a daemon entrypoint instead of uWSGI/uvicorn.

**Key differences from microservices:**
- `deploy` stage entrypoint is `./entrypoint.sh` (a shell wrapper around `python main.py`)
- May require system packages (e.g., `ffmpeg`, `mediainfo`) installed in `base` via `apt-get`
- No `uwsgi.ini`, no `uwsgi-start.sh`, no `spec/` directory
- Jenkinsfile is identical to the microservice pattern

**Dockerfile**

```dockerfile
FROM 086679231553.dkr.ecr.us-east-1.amazonaws.com/docker-parent-images:python312 AS base

ENV PYTHONUNBUFFERED=1 \
    PYTHONDONTWRITEBYTECODE=1 \
    POETRY_HOME="/opt/poetry" \
    PYSETUP_PATH="/opt/pysetup" \
    POETRY_VIRTUALENVS_IN_PROJECT=1 \
    POETRY_VIRTUALENVS_CREATE=1 \
    VIRTUAL_ENV="/opt/pysetup/.venv"
ENV PATH="$POETRY_HOME/bin:$VIRTUAL_ENV/bin:$PATH"

# Install system dependencies required by the daemon
RUN apt-get -y update && \
    apt-get -y upgrade && \
    apt-get -y install <system-packages> && \
    apt-get clean && \
    rm -rf /var/lib/apt/lists/*

RUN python -m venv $VIRTUAL_ENV && \
    $VIRTUAL_ENV/bin/pip install --no-cache-dir --upgrade pip setuptools && \
    $VIRTUAL_ENV/bin/pip install --no-cache-dir poetry==2.3.2

USER worker

WORKDIR /var/app
COPY pyproject.toml ./

# update-lockfile, dev, unit-lint, format stages are identical to microservice

##############################
### Build image for deploy ###
##############################
FROM base AS deploy

WORKDIR $PYSETUP_PATH
COPY pyproject.toml poetry.lock ./

RUN poetry install --only main --no-root --no-cache

USER worker

WORKDIR /var/app
COPY <source_dir> ./<source_dir>
COPY entrypoint.sh main.py ./

ENTRYPOINT ["./entrypoint.sh"]
```

---

### Lambdas (`lambda-`)

Monorepos containing multiple Lambda functions (and sometimes ECS tasks) under `lambda/<function-name>/`. Each function is independently deployable.

**Two function sub-types exist within lambda repos:**

| Sub-type | Base image | Poetry install strategy |
|---|---|---|
| Lambda function | `docker-parent-images:lambda-python3XX` | Builder pattern (see below) |
| ECS Task (`type: task` in Jenkinsfile) | `docker-parent-images:python3XX` | Same as daemon |

#### Per-function directory structure

```
lambda/
  <function-name>/
    pyproject.toml
    poetry.lock
    src/
    tests/
    scripts/
      unit-lint.sh
      format.sh
      update-lockfile.sh
    Dockerfile
    docker-compose.yml
    software-catalog.yaml
    README.md
```

No `Dockerfile.tests`. Tests and linting are stages within the single `Dockerfile`.

#### pyproject.toml

Same as universal standard. Name convention: `lambda-<repo-name>-<function-name>`.

```toml
[project]
name = "lambda-vector-throttler"
# ...
```

#### Dockerfile for Lambda functions (builder pattern)

Because the lambda runtime image does not have Poetry, a non-lambda builder stage installs dependencies, then the deploy stage copies the resulting site-packages into the lambda runtime image.

> **Action required:** Verify the Python site-packages path in your lambda ECR base image before adopting this pattern. Run:
> `docker run 086679231553.dkr.ecr.us-east-1.amazonaws.com/docker-parent-images:lambda-python3XX python -c "import site; print(site.getsitepackages())"`
> Replace `<LAMBDA_SITE_PACKAGES>` in the template below with the result. For `lambda-python313`, the path is `/var/lang/lib/python3.13/site-packages`.
>
> **Python version:** The `lambda-python312` image in the template is an example. Use the image matching your lambda runtime (e.g., `lambda-python313` for Python 3.13).
>
> **Root-level `config.py`:** If the lambda has a `config.py` at the project root (not inside `src/`), add `COPY config.py ./` to both the `deploy` and `unit-lint` stages.
>
> **Datadog Lambda extension:** If the lambda uses the Datadog extension binary, add `COPY --from=public.ecr.aws/datadog/lambda-extension:<version> /opt/. /opt/` to the `deploy` stage.
>
> **`DD_LAMBDA_HANDLER`:** The value `src.index.handler` in the template is an example. Use the actual handler path (e.g., `src.app.handler`).
>
> **Lambda task root:** All stages in the lambda Dockerfile use `WORKDIR /var/task`. This is the standard WORKDIR for Python lambda repos; non-lambda repos use `/var/app`.
>
> **Datadog source code integration args:** Every lambda Dockerfile must include the following block immediately after the `FROM ... AS base` line. These args are passed by the CI build step and wired into Datadog environment variables for source code integration and version tracking.
> ```dockerfile
> ARG GIT_COMMIT
> ARG REPOSITORY_URL
> ARG VERSION
> ENV DD_GIT_COMMIT_SHA=$GIT_COMMIT
> ENV DD_GIT_REPOSITORY_URL=$REPOSITORY_URL
> ENV DD_VERSION=$VERSION
> ```

```dockerfile
FROM 086679231553.dkr.ecr.us-east-1.amazonaws.com/docker-parent-images:lambda-python312 AS base

ARG GIT_COMMIT
ARG REPOSITORY_URL
ARG VERSION
ENV DD_GIT_COMMIT_SHA=$GIT_COMMIT
ENV DD_GIT_REPOSITORY_URL=$REPOSITORY_URL
ENV DD_VERSION=$VERSION

ENV PYTHONUNBUFFERED=1 \
    PYTHONDONTWRITEBYTECODE=1 \
    POETRY_HOME="/opt/poetry" \
    PYSETUP_PATH="/opt/pysetup" \
    POETRY_VIRTUALENVS_IN_PROJECT=1 \
    POETRY_VIRTUALENVS_CREATE=1 \
    VIRTUAL_ENV="/opt/pysetup/.venv"
ENV PATH="$POETRY_HOME/bin:$VIRTUAL_ENV/bin:$PATH"

USER worker
WORKDIR $PYSETUP_PATH

RUN python -m venv $VIRTUAL_ENV && \
    $VIRTUAL_ENV/bin/pip install --no-cache-dir --upgrade pip setuptools && \
    $VIRTUAL_ENV/bin/pip install --no-cache-dir poetry==2.3.2

WORKDIR /var/task
COPY pyproject.toml ./

################################
### Generate poetry lockfile ###
################################
FROM base AS update-lockfile

COPY scripts/update-lockfile.sh ./scripts/
ENTRYPOINT ["scripts/update-lockfile.sh"]

#################################
### Build image for unit-lint
#################################
FROM base AS unit-lint

ENV RUFF_CACHE_DIR=/tmp/.ruff_cache

COPY poetry.lock ./
RUN poetry install --no-root --no-cache --extras dev

COPY src ./src
COPY tests ./tests
COPY docker-compose.yml ./
COPY scripts/unit-lint.sh ./scripts/

ENTRYPOINT ["scripts/unit-lint.sh"]

##########################
### Format source code ###
##########################
FROM base AS format

COPY poetry.lock ./
RUN poetry install --no-root --no-cache --extras dev

COPY src ./src
COPY tests ./tests
COPY scripts/format.sh ./scripts/

ENTRYPOINT ["scripts/format.sh"]

##############################
### Build image for deploy ###
##############################
FROM base AS deploy

COPY poetry.lock ./
RUN poetry install --only main --no-root --no-cache

COPY src ./src

ENV DD_LAMBDA_HANDLER=src.index.handler
CMD ["datadog_lambda.handler.handler"]
```

> **`common_parts` in integration tests:** While `common_parts` is being phased out, lambda functions that still depend on it for integration test utilities may temporarily use `additional_contexts` in the `integration-test` build context to copy shared test helpers. Remove this once `common_parts` has been replaced by a proper library.
>
> **`poetry.lock` and the `update-lockfile` service:** Commit an empty `poetry.lock` file alongside `pyproject.toml`. This ensures Docker always binds a file at the mount point — if the file doesn't exist, Docker creates a directory there instead, which breaks Poetry with `[Errno 21] Is a directory`. The `update-lockfile` service mounts `./:/var/task` (the whole project directory, not just the lock file) because a file-level bind mount blocks `unlink` inside the container with "Device or resource busy", preventing `update-lockfile.sh` from removing the empty file before regenerating. `update-lockfile.sh` detects the empty file, removes it, and calls `poetry lock` to generate from scratch; for subsequent updates it calls `poetry update --lock`. All other services require a valid `poetry.lock` to build.

#### Dockerfile for ECS Tasks within lambda repos

Use the exact same pattern as the **Daemon** type. The entrypoint differs but the Docker and Poetry setup is identical.

#### docker-compose.yml (per function directory)

The `COMPOSE_PROJECT_NAME` must be unique per function to avoid container name collisions when multiple functions are tested in parallel in CI. Set it as a build argument or environment variable at the repo level.

```yaml
---
services:
  dev:
    platform: linux/amd64
    build:
      context: .
      target: deploy   # lambda functions reuse deploy image for local invocation
    ports:
      - "9000:8080"
    environment:
      - AWS_ACCESS_KEY_ID
      - AWS_SECRET_ACCESS_KEY
      - AWS_SESSION_TOKEN
      - AWS_DEFAULT_REGION=us-east-1

  unit-lint:
    platform: linux/amd64
    build:
      context: .
      target: unit-lint
    environment:
      - SKIP_LINT=0
      - SKIP_TYPE_CHECKS=0
      - SKIP_TESTS=0
    volumes:
      - ./build:/var/task/build

  format:
    platform: linux/amd64
    build:
      context: .
      target: format
    volumes:
      - ./src:/var/task/src
      - ./tests:/var/task/tests

  update-lockfile:
    platform: linux/amd64
    build:
      context: .
      target: update-lockfile
    volumes:
      - ./:/var/task
```

#### Shared code (`common_parts/`)

The `common_parts/` pattern (present in `lambda-metadata-consistency`) should be removed. Shared code is not included in static code analysis in its current form, and its inclusion complicates the build, test, and deploy processes. Shared code should instead be extracted into a proper `python-` library repo and published to the internal PyPI.

#### Jenkinsfile

The lambda Jenkinsfile uses `withModifiedFunctions` / `MonorepoUtils` to only build functions whose files changed. Each function runs its own `docker compose run --rm --build unit-lint` in CI. The `FUNCTION_CONFIG` map at the top of the Jenkinsfile declares deploy targets per function and must be kept in sync as functions are added or removed.

---

### Scripts (`scripts-`)

Non-production scripts grouped by domain. Each subdirectory is a fully independent, standalone unit with its own dependencies and Docker image.

**Per-script directory structure:**

```
scripts/
  <script-name>/
    pyproject.toml
    poetry.lock
    src/
    tests/           # if applicable
    scripts/
      unit-lint.sh
      format.sh
      update-lockfile.sh
    localdev/        # if applicable
    Dockerfile
    docker-compose.yml
    README.md
```

**`pyproject.toml` notes:**
- `package-mode = false`
- Dev dependencies are optional if the script has no tests; include `ruff` and `mypy` at minimum

**Dockerfile**

Simpler than microservices — no `dev` stage, no server entrypoint. Two variants:

*With tests:*
```dockerfile
FROM 086679231553.dkr.ecr.us-east-1.amazonaws.com/docker-parent-images:python312 AS base

ENV PYTHONUNBUFFERED=1 \
    PYTHONDONTWRITEBYTECODE=1 \
    POETRY_HOME="/opt/poetry" \
    PYSETUP_PATH="/opt/pysetup" \
    POETRY_VIRTUALENVS_IN_PROJECT=1 \
    POETRY_VIRTUALENVS_CREATE=1 \
    VIRTUAL_ENV="/opt/pysetup/.venv"
ENV PATH="$POETRY_HOME/bin:$VIRTUAL_ENV/bin:$PATH"

RUN python -m venv $VIRTUAL_ENV && \
    $VIRTUAL_ENV/bin/pip install --no-cache-dir --upgrade pip setuptools && \
    $VIRTUAL_ENV/bin/pip install --no-cache-dir poetry==2.3.2

WORKDIR $PYSETUP_PATH
COPY pyproject.toml poetry.lock ./

################################
### Generate poetry lockfile ###
################################
FROM base AS update-lockfile
COPY scripts/update-lockfile.sh ./scripts/
ENTRYPOINT ["scripts/update-lockfile.sh"]

#################################
### Build image for unit-lint
#################################
FROM base AS unit-lint

RUN poetry install --no-root --no-cache --extras dev

WORKDIR /var/app
COPY src ./src
COPY tests ./tests
COPY scripts/unit-lint.sh ./scripts/

USER worker
ENTRYPOINT ["scripts/unit-lint.sh"]

##########################
### Format source code ###
##########################
FROM base AS format

RUN poetry install --no-root --no-cache --extras dev

WORKDIR /var/app
COPY src ./src
COPY scripts/format.sh ./scripts/

USER worker
ENTRYPOINT ["scripts/format.sh"]

##############################
### Build image for run     ###
##############################
FROM base AS run

RUN poetry install --only main --no-root --no-cache

WORKDIR /var/app
COPY src ./src

USER worker
ENTRYPOINT ["python", "-m", "src.main"]
```

*Without tests (lint/format only):* Omit the `tests` `COPY` and remove pytest from dev deps. The `unit-lint.sh` should set `SKIP_TESTS=1` by default.

---

## Migration Checklist

### From `requirements.txt` + pip

**Step 1 — Identify top-level dependencies**

Many existing `requirements.txt` files already separate top-level from transitive deps using comments:
```
# top level dependencies
boto3==1.35.57
owslogger==4.0.0

# sub dependencies
certifi==2026.1.4
...
```
Use the top-level section directly. For files without comments, scan all non-test Python source files for import statements, remove Python built-ins and local module imports, and cross-reference the remaining package names against `requirements.txt` to find the versions to use as the basis for `~=` ranges.

**Step 2 — Update Docker infrastructure first**

Docker targets must exist before the lockfile can be generated in a container.

1. Update `Dockerfile` to the Poetry pattern for this repo type; rename/add stages: `base`, `deploy`, `dev`, `update-lockfile`, `unit-lint`, `format`
2. Update `docker-compose.yml`: rename services (`update-poetry-lockfile` → `update-lockfile`), add any missing services, add `./build:/var/app/build` volume to `unit-lint`
3. Create `scripts/unit-lint.sh`, `scripts/format.sh`, `scripts/update-lockfile.sh` from the templates above; delete the old root-level `unit-lint.sh` or `scripts/unit-lint.sh` if present at a non-standard path
4. Ensure all scripts are executable: `git update-index --chmod=+x scripts/*.sh`

**Step 3 — Migrate package management**

5. Create `pyproject.toml` using the universal template above
6. Populate `[project].dependencies` with top-level deps only (from step 1); use `~=` version ranges
7. Move dev/test deps to `[project.optional-dependencies].dev`
8. Run `docker compose run --build update-lockfile` to generate `poetry.lock`
9. Delete `requirements.txt`, `requirements-dev.txt`, `requirements-test.txt`

**Step 4 — Migrate integration tests (if applicable)**

If the repo has no `tests/integration/` directory, skip this step entirely — do not wire up integration test infrastructure.

10. If `tests/integration/requirements-*.txt` exists, migrate its top-level contents to `[project.optional-dependencies].integration` in `pyproject.toml`; delete the file
11. Add the `integration-test` Dockerfile stage (extending `dev`; add `poetry install --extras integration` only if step 10 added an `integration` group)
12. Add the `integration-test` docker-compose service with QA environment variables
13. Create `scripts/integration-test.sh` from the template above; delete the old `scripts/test-integration.sh` or `scripts/integration.sh` if present
14. Add the `Integration Tests` Jenkinsfile stage after `Deploy to QA`

**Step 5 — Clean up**

15. Update Jenkinsfile to call `docker compose` directly (remove any `make` calls); add `sh 'mkdir -p build && chmod a+w build'` immediately before the `unit-lint` step
16. Remove `Makefile`
17. Delete `.flake8`, `pytest.ini`, `setup.cfg`, `pylint.rc`, `.python-version`, `.elasticbeanstalk`, `.editorconfig`, `datadog.repo` if present
18. Move any developer convenience scripts from `scripts/` to `localdev/`
19. Add `.gitattributes` with `* text=auto eol=lf` if not already present

### From legacy `[tool.poetry]` style (libraries)

1. Move `[tool.poetry.dependencies]` to `[project].dependencies`
2. Move `[tool.poetry.group.dev.dependencies]` to `[project.optional-dependencies].dev`
3. Keep `[tool.poetry]` only for `packages`; move `name`/`version`/`description`/`authors` to `[project]`
4. Add `[build-system]` if missing
5. Run `docker compose run --build update-lockfile` to regenerate the lock file for the new format

### From `setup.py`

1. Follow the library template above
2. Migrate `install_requires` → `[project].dependencies`
3. Migrate package metadata → `[project]`
4. Delete `setup.py`, `MANIFEST.in`

---

## Verification

After any migration or change, run the following to confirm the repo is correctly set up. All commands run from the repo root, or from the function/script subdirectory for lambda and scripts repos.

### Prerequisites

**AWS / ECR authentication**

All base images are pulled from ECR. Docker is configured to use `docker-credential-ecr-login` as its credential store (`"credsStore": "ecr-login"` in `~/.docker/config.json`), so no explicit `docker login` is needed — but valid AWS credentials must be present in the environment when any `docker build` or `docker pull` runs.

Authenticate before running any docker compose command:

```sh
eval "$(awsume prod -s 2>/dev/null)"
```

The `-s` flag prints export statements to stdout; `eval` injects them into the current shell. If the cached MFA session is still valid, this runs without prompting. The session is cached by awsume under `~/.awsume/cache/` and is typically valid for 12–36 hours.

When chaining commands (as an agent would), include the auth inline so credentials are present in the same shell invocation:

```sh
eval "$(awsume prod -s 2>/dev/null)" && docker compose run --rm --build unit-lint
```

**Build output directory**

The `build/` directory must exist and be writable before running `unit-lint`, since Docker creates host-side volume mount targets as root if they don't exist:

```sh
mkdir -p build && chmod a+w build
```

**Windows (WSL) requirements**

Most engineers use macOS. For those on Windows, all `docker compose`, `awsume`, and `chmod` commands must be run from a **WSL terminal** (bash), not PowerShell or CMD. Additionally:

- Docker Desktop must be configured to use the **WSL2 backend** (Settings → General → "Use the WSL2 based engine"). The Hyper-V backend has known issues with volume mounts and networking.
- `awsume` must be pip-installed **inside WSL** (`pip install awsume`). The Windows-native awsume cannot set environment variables in a WSL bash shell.
- The repo's `.gitattributes` file (see Universal Standards) prevents git from converting scripts to CRLF on Windows checkout. If a script was already checked out with CRLF before `.gitattributes` was added, re-normalize with `git add --renormalize .`.

### Format first, then lint and test

On a fresh migration — or any time ruff is being adopted for the first time in a repo — always run `format` before `unit-lint`. Existing code is unlikely to match ruff's style out of the box, and the formatter check inside `unit-lint` will fail until it does.

```sh
docker compose run --rm --build format
docker compose run --rm --build unit-lint
```

For subsequent runs on already-formatted code, `unit-lint` alone is sufficient.

### Format

```sh
docker compose run --rm --build format
```

Expected outcome:
- Exit code 0
- Any files that were reformatted are written back to the host via the mounted volume

### Lint and test

```sh
docker compose run --rm --build unit-lint
```

Expected outcome:
- Exit code 0
- `build/pyunit.xml` exists and contains test results
- `build/coverage.xml` exists and contains coverage data

### Lock file update

```sh
docker compose run --rm --build update-lockfile
```

Expected outcome:
- Exit code 0
- `poetry.lock` is either unchanged or updated with newly resolved dependencies

### Dev instance

**Microservices (`ows-`)**

```sh
docker compose up dev
```

Expected outcome: container starts and logs show the uWSGI or uvicorn server listening (typically on port 5000). No crash on startup.

**Daemons (`daemon-`)**

```sh
docker compose up dev
```

Expected outcome: container starts and logs show the daemon initializing. It may pause waiting for queue messages — that is expected. An immediate exit or Python traceback indicates a problem.

**Lambda functions**

Lambda function containers expose a local invocation endpoint:

```sh
docker compose up dev
```

Then in a second terminal:

```sh
curl -XPOST "http://localhost:9000/2015-03-31/functions/function/invocations" -d '{}'
```

Expected outcome: container starts on port 9000 without crashing. The curl returns a JSON response — even a function-level error response is acceptable. A connection refused or HTTP 500 from the Lambda runtime itself indicates a startup problem.

**ECS Tasks within lambda repos**

```sh
docker compose up dev
```

Same expectation as daemons — the task should start and begin its work loop without an immediate exit.

**Scripts**

Scripts have a `run` stage rather than a persistent dev server:

```sh
docker compose run --rm --build run
```

Expected outcome: exit code 0, or an expected non-zero if the script requires input not provided. A Python traceback on import or startup is a failure.

**Libraries**

Libraries have no runnable application. Lint-and-test is the only verification step.

### Integration tests (if applicable)

Integration tests run against a live QA environment and require valid AWS credentials and any service-specific environment variables. They are not expected to pass locally without that access.

To verify the wiring is correct without needing QA access, confirm the image builds cleanly:

```sh
docker compose build integration-test
```

To run against QA (requires credentials):

```sh
docker compose run --rm integration-test
```

Expected outcome: exit code 0. A non-zero exit means one or more integration tests failed against the QA environment.

---

## Gaps & Clarifications

Items identified during standardization of `daemon-video` (2026-03-24) that were not covered by earlier versions of this guide.

### Python version in base image tag

The templates in this document use `python312`. The `python3XX` suffix in the base image tag should always match the repo's actual Python version. Repos already running a newer interpreter (e.g., 3.13) should use `python313`. Only use `python312` when starting a new repo or when a newer image is not yet available.

### `python-dotenv` migration in existing repos

The universal standard prohibits `python-dotenv` (or any `.env`-loading library) in production code. For repos that currently call `load_dotenv()` at module level in `config.py`, the migration path is:

1. Move `python-dotenv` from main dependencies to the `dev` extras group in `pyproject.toml`.
2. Guard the `load_dotenv()` call so it silently no-ops when the package is not installed:
   ```python
   try:
       from dotenv import load_dotenv
       load_dotenv(dotenv_path)
   except ImportError:
       pass
   ```
3. The deployed container never installs dev extras, so the `.env` load is skipped automatically. Local dev environments get the `.env` load as before.

### Dockerfile: no ARG interpolation in FROM

Write the full base image URL literally in every `FROM` statement. Do not use `ARG` to construct the image path:

```dockerfile
# wrong
ARG AWS_ACCOUNT=086679231553
FROM ${AWS_ACCOUNT}.dkr.ecr.us-east-1.amazonaws.com/docker-parent-images:python313 AS base

# correct
FROM 086679231553.dkr.ecr.us-east-1.amazonaws.com/docker-parent-images:python313 AS base
```

ARG interpolation obscures the actual image being pulled, complicates tooling that scans `FROM` lines (e.g., Renovate, Docker Scout), and adds indirection without benefit since the account ID is fixed.

### Daemon Dockerfile: venv ownership

The daemon template shows the venv being created under `USER root` (because apt-get runs first). This causes permission errors when subsequent stages run `poetry install` as `worker`. The fix: create `$PYSETUP_PATH` as root, chown it to worker, then switch to worker *before* creating the venv:

```dockerfile
RUN mkdir /var/log/worker && chown -R worker:worker /var/log/worker && \
    mkdir -p $PYSETUP_PATH && chown -R worker:worker $PYSETUP_PATH

USER worker

RUN python -m venv $VIRTUAL_ENV && \
    $VIRTUAL_ENV/bin/pip install --no-cache-dir --upgrade pip setuptools && \
    $VIRTUAL_ENV/bin/pip install --no-cache-dir poetry==2.3.2
```

This matches the microservice template where `USER worker` is set before venv creation.

### `supervisord` in daemon repos

The daemon template shows `entrypoint.sh` as a thin wrapper around `python main.py`. Some daemon repos use `supervisord` for process management (restart policies, log routing, health endpoints). This is acceptable — the `entrypoint.sh` should then invoke supervisord instead:

```bash
#!/usr/bin/env bash
set -e
exec /usr/bin/supervisord -c /var/app/conf/supervisord.conf
```

Install `supervisor` via `apt-get` in the `base` stage. The `supervisord.conf` should run the application as the `worker` user.

### Additional dev/test dependencies

The `dev` extras group in the template lists the minimum required tools. Additional testing helpers are acceptable dev dependencies. Common additions:

- `flexmock` — mock/stub library; prefer `pytest-mock` for new code but `flexmock` is fine where it is already in use
- `freezegun` — time-travel for tests

Add them to `[project.optional-dependencies].dev` alongside the standard tools.

### Mypy adoption for untyped codebases

Repos that have no existing type annotations will have mypy errors out of the box when `[tool.mypy] strict = true` is set. To avoid blocking CI immediately:

1. Set `SKIP_TYPE_CHECKS=1` in the `unit-lint` service in `docker-compose.yml`.
2. Add type annotations incrementally, module by module.
3. Remove `SKIP_TYPE_CHECKS=1` once mypy passes cleanly.

Do not set `strict = false` permanently — the goal is to reach full coverage.

### `docker-compose.yml` extension

Items identified during standardization of `lambda-releases` / `new_release_json_blaster` (2026-03-24).

The canonical extension is `.yml`, not `.yaml`. The `unit-lint.sh` script hard-codes `docker-compose.yml` in the yamllint step — using `.yaml` causes that check to fail with a file-not-found error. Always use `.yml`.