---
name: new-microservice
description: Use when creating, scaffolding, reviewing, or auditing a Python microservice for the Insights team at The Orchard / Sony Music PDE. Triggers on phrases like "create a new microservice", "scaffold a new service", "review this PR against our standards", "does this service follow our patterns", "port this service to Python". SKIP if the service uses FastAPI/Celery (not our stack), GitHub Actions (not Jenkins), or is not an Insights/analytics service.
---

# Insights Python Microservice

## When to Use This Skill

**TRIGGER — create mode:**
- "Create a new microservice for [domain]"
- "Scaffold a new Insights service"
- "Port this .NET / Node service to Python"
- "Start a new Python Flask service"

**TRIGGER — review/audit mode:**
- "Review this PR against our standards"
- "Does this service follow Insights patterns?"
- "Audit this code against ows-charts / ows-analytics"
- "What's wrong with this service structure?"

**SKIP if:**
- The service uses FastAPI, Celery, or Django (not the Insights stack)
- The repo uses GitHub Actions (`.github/workflows/` exists)
- The user explicitly says it's a non-Insights service

---

## When to Load Each Reference

Load references exactly when needed — no speculative preloading.

### Create mode

| When in the workflow | Load |
|---|---|
| About to generate project structure | [project-structure.md](./references/project-structure.md) |
| About to write `application.py`, `api.py`, `config.py` | [app-factory-logging-config.md](./references/app-factory-logging-config.md) |
| About to write handlers and schemas | [handlers-schemas-errors.md](./references/handlers-schemas-errors.md) |
| About to write Snowflake/Redis connectors | [snowflake-redis.md](./references/snowflake-redis.md) |
| About to write tests | [testing.md](./references/testing.md) |
| About to write Dockerfile or uwsgi-start.sh | [dockerfile-uwsgi.md](./references/dockerfile-uwsgi.md) |
| About to write Jenkinsfile | [jenkinsfile.md](./references/jenkinsfile.md) |
| Running final verification | [quality-checklist.md](./references/quality-checklist.md) |

### Review/audit mode

Load all references before analysis:
- [project-structure.md](./references/project-structure.md)
- [app-factory-logging-config.md](./references/app-factory-logging-config.md)
- [handlers-schemas-errors.md](./references/handlers-schemas-errors.md)
- [snowflake-redis.md](./references/snowflake-redis.md)
- [testing.md](./references/testing.md)
- [dockerfile-uwsgi.md](./references/dockerfile-uwsgi.md)
- [jenkinsfile.md](./references/jenkinsfile.md)
- [quality-checklist.md](./references/quality-checklist.md)

---

## Stack Overview

All Insights microservices share this stack — do not deviate without explicit approval:

| Layer | Technology |
|---|---|
| Web framework | Flask 2.x + uWSGI |
| Language | Python 3.11 (target) |
| Serialization | marshmallow 3.x |
| Primary DB | Snowflake (raw SQL, no ORM for Snowflake) |
| Secondary DB | PostgreSQL via SQLAlchemy (auxiliary only) |
| Cache | Redis + `fakeredis` fallback |
| APM | ddtrace (Datadog) |
| Error reporting | Sentry SDK |
| Logging | `owslogger` (internal — wraps python-json-logger) |
| Service calls | `owsrequest` (internal HTTP client) |
| Response shaping | `oto` (internal — `response.create_fatal_response`, `flaskify`) |
| Auth | Header-based via `owsrequest` context — no API keys |
| Feature flags | `pythonfeatures` (Split.io wrapper) |
| Package index | `https://pypi.theorchard.io/pypi` (internal PyPI) |

Internal packages (`owslogger`, `owsrequest`, `oto`, `snowflake_connector`, `pythonfeatures`) are available only via the internal PyPI. Always prefix `requirements.txt` / `Pipfile` with `-i https://pypi.theorchard.io/pypi`.

---

## Questions to Ask Before Creating a New Service

Use `AskUserQuestion` for each round. Maximum 4 questions per call.

**Round 1 — Service identity (3 questions):**
- Service name (provide format `ows-<domain>`, e.g. `ows-analytics`, `ows-charts`)
- What does this service do? (one-sentence description for `software-catalog.yaml`)
- Primary data source: Snowflake only / PostgreSQL + Snowflake / Snowflake + DynamoDB

**Round 2 — Infrastructure (3 questions):**
- ECR account ID (default: `086679231553`)
- QA account ID and deployment role
- Prod account ID and deployment role

**Round 3 — Features (3 questions, multi-select):**
- Does the service need Redis caching? (Yes / No)
- Does the service call other microservices? (Yes — list them / No)
- Slack notifications channel (default: `#insights-engineering`)

**Variable mapping:**

| Answer | Used in |
|---|---|
| Service name | `SERVICE_NAME` in config, Dockerfile WORKDIR label, `software-catalog.yaml` |
| Description | `software-catalog.yaml` metadata.description |
| ECR account ID | `ECR_ACCOUNT_ID` in Jenkinsfile |
| QA/Prod account IDs | `QA_ACCOUNT_ID`, `PROD_ACCOUNT_ID` in Jenkinsfile |
| QA/Prod roles | `QA_DEPLOYMENT_ROLE`, `PROD_DEPLOYMENT_ROLE` in Jenkinsfile |
| Slack channel | `SLACK_NOTIFICATIONS_CHANNEL` in Jenkinsfile |

---

## Generation Workflow (Create Mode)

Work through these phases in order. Complete each phase fully before moving on.

**Phase 1 — Gather answers** (Rounds 1–3 above)

**Phase 2 — Generate project skeleton**
Load [project-structure.md](./references/project-structure.md). Create the directory tree and all empty `__init__.py` files.

**Phase 3 — Core files** (generate in this order)
Load [app-factory-logging-config.md](./references/app-factory-logging-config.md):
1. `<package>/config.py`
2. `<package>/api.py`
3. `application.py`
4. `dev.py`

**Phase 4 — Handlers, schemas, errors**
Load [handlers-schemas-errors.md](./references/handlers-schemas-errors.md):
5. `<package>/handlers.py` (including global error handler)
6. `<package>/schemas/base.py`
7. `<package>/schemas/` — one file per endpoint group

**Phase 5 — Data layer**
Load [snowflake-redis.md](./references/snowflake-redis.md):
8. `<package>/connectors/snowflake.py`
9. `<package>/connectors/redis.py` (if Redis enabled)
10. `<package>/utils/cache.py` (if Redis enabled)
11. `<package>/logic/` — one module per domain

**Phase 6 — Infrastructure**
Load [dockerfile-uwsgi.md](./references/dockerfile-uwsgi.md) and [jenkinsfile.md](./references/jenkinsfile.md):
12. `Dockerfile`
13. `uwsgi-start.sh`
14. `Jenkinsfile`
15. `software-catalog.yaml`
16. `requirements.txt` + `requirements-dev.txt`
17. `.flake8` / `setup.cfg` / `mypy.ini`
18. `Makefile`
19. `.env.shadow`

**Phase 7 — Tests**
Load [testing.md](./references/testing.md):
20. `tests/conftest.py`
21. `tests/unit/conftest.py`
22. `tests/unit/test_handlers.py`
23. `tests/integration/conftest.py`
24. `tests/integration/test_api_endpoints.py`

**Phase 8 — Verification**
Load [quality-checklist.md](./references/quality-checklist.md). Run every item. Report failures before declaring done.

---

## Audit Workflow (Review/Audit Mode)

**Step 1** — Load all reference files (see table above).

**Step 2** — Read the service being audited. Use `find` to get the full file tree, then read key files: `application.py`, `config.py`, `api.py`, handlers, `Dockerfile`, `Jenkinsfile`, `software-catalog.yaml`, `requirements.txt` / `Pipfile`, and any test files.

**Step 3** — Run the full verification checklist from [quality-checklist.md](./references/quality-checklist.md) against the service. For each failure, record:
- The specific item that failed
- Current state (what the code does)
- Required state (what it should do)
- Severity: **Blocker** (deploy will fail or data will be wrong) / **Important** (should fix before merge) / **Minor** (style/cleanup)

**Step 4** — Present findings grouped by severity. Include file paths and line numbers. Be specific and actionable.

**Step 5** — Do not edit files unless the user asks. Present the full audit first.

---

## Non-Negotiable Rules

These apply to both create and review modes. Never compromise on them:

1. **`ddtrace.patch_all()` must be the first code executed** in `application.py`, before any other imports. A comment `# noqa` must follow the line.
2. **Sentry must be validated at startup** — if `SENTRY_DSN` is not configured in QA or prod, raise `Exception` and refuse to start.
3. **`software-catalog.yaml` must exist** — `datadogSoftwareCatalogValidate()` will fail in CI without it.
4. **`uwsgi-start.sh` must exist** — the Dockerfile `ENTRYPOINT` references it.
5. **All versions in `requirements.txt` must be pinned** — no `>=` ranges, no `~=`. Exact pins only.
6. **Test files must be present** — no PR merges without at least unit tests for every handler and every logic function.
7. **No marshmallow validation errors exposed as 400** — they are mapped to 500 via the base `RequestSchema` error handler (Orchard standard).
8. **Redis connector must fall back to `fakeredis`** when `REDIS_HOST` is not set — do not fail at import time.
9. **Handler functions must be thin** — extract params, call logic layer, return. No business logic in handlers.
10. **Route path strings live in `config.py`** as module-level constants — no inline strings in `@app.route(...)`.
11. **`Dockerfile` must not copy test files into the deploy stage** — only copy `<package>/`, `spec/`, `application.py`, `requirements.txt`.
12. **Jenkinsfile `VULNERABILITIES_TO_IGNORE`** must be an empty list `[]` in initial commits. Only add CVE IDs after investigation with a comment explaining the decision and remediation plan.
13. **The deploy stage user must be `uwsgi`** (non-root, no shell: `-s /bin/false`).