# songwhip-api The backend interface for `songwhip-web` hosted at `api.songwhip.com`. Where all state lives and the glue that ties the Songwhip micro-services together. ## Prerequisites - Node.js configured repo\ https://www.notion.so/Node-js-project-setup-guide-86feadffeb304962b06e6b1bb8f09213 - Docker\ https://docs.docker.com/get-docker/ ## Installing - Install the dependencies: `pnpm` - Initialize translations: `pnpm i18n:init` - Copy the `.env.shadow` to `.env` and fill in the values: `cp .env.shadow .env` ## Running 1. Make sure the Docker daemon is running 2. Start the server: `pnpm dev` 3. Hit the API with the HTTP client of your choice ## Running against staging database 1. Connect to the VPN 2. Ensure you have a `.env.staging` filled with all values from `.env.staging.shadow` 3. Start the server: `pnpm dev:staging` 4. Hit `https://localhost:5002/` with the HTTP client of your choice ## Running against production database 1. Connect to the VPN 2. Ensure you have a `.env.production` filled with all values from `.env.production.shadow` 3. Start the server: `pnpm dev:production` 4. Hit `https://localhost:5002/` with the HTTP client of your choice ## Running against a local database in a test environment 1. Ensure you have a `.env.test` filled with all values from `.env.test.shadow` 2. Start the server: `pnpm dev:test` 3. Hit `https://localhost:5003/` with the HTTP client of your choice Note that this is mostly useful for pointing your songwhip-web e2e tests to your local songwhip-api instance. Great when you're working on a feature that requires changes to the API that might break songwhip-web tests. ## Logging and monitoring ### Logging We use Pino for logging. In development (`NODE_ENV=development`), logs are pretty-printed for readability. In all other environments, logs are output in JSON format with all fields and trace IDs included. #### Example Logs - **Development (Pretty-Printed)** ```shell [16:07:48]: [songwhip/api] : request took: 797 ms ``` - **Production (JSON Format)** ```json { "level": "30", "time": 1723556299380, "pid": 3333, "hostname": "my-hostname", "namespace": "songwhip/api", "x-amzn-trace-id": "Root=1-5f6b0f3a-3d8e0b605b9b9b3b", "msg": "request took: 797 ms" } ``` ### Monitoring 1. **Sentry** is used to capture errors, and its context includes trace IDs for better traceability. (Also, some logs are intentionally sent to Sentry to describe error cases) 2. **Datadog (DD)** is used for logging informational messages. The context also includes trace IDs. ### Tracing Trace IDs Each log entry in production environments includes a traceId, which is used to track requests across distributed systems. This helps in correlating logs from different services and components, providing better visibility and easier debugging of issues that span multiple parts of the application. | # | Header Name | Example | Description | | --- | --------------------- | ---------------------- | ---------------------------------------------------------------------- | | 1 | `x-amzn-trace-id` | `Root=1-678901234....` | Amazon Trace ID used for tracing requests from AWS Load Balancer. | | 2 | `x-datadog-trace-id` | `1234567890127....` | Datadog Trace ID used for tracking a request in Datadog APM. | | 3 | `x-datadog-parent-id` | `98765432765......` | Datadog Parent ID representing the immediate parent span in the trace. | ## Testing - Run the linter and the unit tests: `pnpm test` - Run the E2E tests locally: `pnpm test:e2e` or `pnpm test:e2e:watch` - The E2E tests will be run automatically by a hook when pushing changes - In order to update automatically generated snapshots run `pnpm test:e2e -- -u` ## Deployment - Deployments are made using Jenkins. - All commits to `master` branch will trigger a `staging` deployment and a `production` deployment to `api-staging.songwhip.com` and `api.songwhip.com` - All commits to `staging` branch will trigger a `staging` deployment to `api-staging.songwhip.com`. - We recommend you push your pull-requests to `staging` (`git push origin my-feature-branch:staging`) to test before deploying to `production`. ### songwhip-api npm package We use a private npm package for sharing code between our services. The package is manually published to GitHub Packages by triggering the `publishPackage` workflow in this repo. Make sure you bump the version in `packages/songwhip-api/package.json` before triggering the workflow. ## Database This service connects to a single AWS RDS Postgres instance located in the `us-east-1` region. ### Backups Snapshots of the database are taken once a day in the morning (~9am GMT). To restore a database from a snapshot you must create an new instance from the chosen snapshot. ### Schema migration We use Knex.js for database migrations. https://knexjs.org/guide/migrations.html When you need to change the database schema, e.g adding a new table or modifying an existing one, you should create a **schema** migration. #### Step 1: Create a new db schema migration script To create a new migration, run: ```bash pnpm db:migrate:schema:create ``` This will create a new schema migration file here `app/database/migrations/schema/.ts`. Add your migration code in the `up` and `down` functions of the migration file. **Note that the `down` function should reverse all the changes made in the `up` function.** #### Step 2: Generate Zod schemas Once your migration is ready, you need to regenerate the Zod schemas to reflect the changes in the database schema. Run: ```bash pnpm db:migrate:schema:zod ``` This will regenerate the Zod schemas for your pending database changes and update the `app/database/schema/generated` folder with the new zod schemas. It does that by starting a local Postgres instance, running your migration against it, and then generating the Zod schemas from that instance. #### Step 3: Create your PR and merge it Once your migration is ready and the Zod schemas are updated, create a pull request with your changes and merge it once it's approved. #### Step 4: Let the CI run the migration The Jenkins CI will automatically run the migration in the staging and production environments when the pull request is merged. https://pipeline.theorchard.io/job/theorchard/job/songwhip-api/job/master/ ### Data migrations When you need to change the data in the database, e.g. updating existing records or inserting new ones, you should create a **data** migration. To create a new data migration, run: ```bash pnpm db:migrate:data:create ``` This will create a new data migration file here `app/database/migrations/data/.ts`. Add your migration code in the `up` and `down` functions of the migration file. **Note that you should not change the database schema in data migrations, only the data itself.** The migration file will be automatically executed in staging and production environments in the same way as schema migrations. No zod schema updates are required for data migrations. ## GraphQL ### TypeScript Code Generation We use `graphql-codegen` to generate TypeScript types for our GraphQL queries. The configuration file for this tool is located at `graphql.codegen.ts`. Types should be re-generated whenever a GraphQL query or the remote schema changes. To regenerate types, run: ```bash pnpm graphql:codegen ``` ### Query File Structure The codegen utility looks for GraphQL (GQL) queries stored in the `app/lib/orchard/api/queries` and `app/lib/orchard/api/fragments` folders. It generates a `generatedTypes.ts` file in the same folder as each query file. Note that each `generatedTypes.ts` file references `schemaTypes.ts`, which holds types for the remote schema. ``` -- app -- lib -- orchard -- api -- schemaTypes.ts -- fragments -- [fragmentName] -- index.ts -- generatedTypes.ts (generated) -- queries -- [queryName] -- index.ts -- generatedTypes.ts (generated) ``` ### Naming Queries - **Folder Names**: Use camelCase for query folder names, avoiding verbs like `get` or `query`. - **Query Names**: Define query names in PascalCase, excluding words like `Query` or `Get`. For instance, an `artist` query should be named: ```ts export const getOrchardArtist = `/* GraphQL */ query OrchardArtist { ... } `; ``` Following this naming convention will generate types like `OrchardArtistQuery` and `OrchardArtistQueryVariables`. #### Magic GraphQL Comment To ensure the codegen tool detects your GQL queries, include a `/* GraphQL */` comment before each query string. This also enhances syntax highlighting and formatting in your editor. ### Using GQL Fragments Fragments are useful for reusing parts of a query across multiple queries. The codegen tool will automatically pick up fragments and generate types for them. Place fragments in the `app/lib/orchard/api/fragments` folder, following the same naming conventions as queries (PascalCase without verbs and prefixed with `Orchard`). Example fragment: ```ts export const TRACK_FRAGMENT = `/* GraphQL */ fragment OrchardTrack on GlobalSoundRecording { id name } `; ``` To use fragments in a query: ```ts import { TRACK_FRAGMENT } from '../fragments/track'; export const getOrchardArtist = `/* GraphQL */ ${TRACK_FRAGMENT} # include the fragment query OrchardArtist { artist { id name tracks { ...OrchardTrack # use the fragment } } } `; ``` When multiple queries use the same fragment, it enables consistent data formatting, allowing you to create reusable formatters for fragments and avoid duplicating formatting logic across files. ### Mocking GQL Queries When writing tests, you may want to mock GQL queries to avoid making real network requests. To do this, use the `fetchMock` utility. This utility can replace the GQL query with a mock function that returns a predefined response. Example: ```ts import { fetchMock } from 'test/e2e/lib/mocks'; beforeEach(() => { fetchMock.mockGraphqlRequest('OrchardLabel', { data: { orchardLabel: { id: { vendorId: 1, subaccountId: 0 }, name: 'My Label', }, }, }); }); ``` The `mockGraphqlRequest` function is type safe, using the generated types from our GQL queries. **Note that when adding a new GQL query, you must also add types for it in the `test/e2e/lib/utils/fetchMock/graphqlTypes.ts` file.** ## Localization We are managing localization using the [@theorchard/frontend-cli-i18n](https://github.com/theorchard/orchard-suite/tree/master/packages/frontend-cli-i18n) cli tooling. The `frontend-cli-i18n` init command scans through the repo for `*.i18n.json|md` files and constructs type safe translation files in the `./locales/` folder. The `i18n:init` script is run after `pnpm install` so that the `./locales/` folder is always generated and ready to be referenced in code. Whenever you change a `i18n.json` or `*.i18n.md` file you need to update the locale definitions by running `pnpm i18n:init`. Alternatively you can run the `pnpm i18n:watch` script to watch for changes and automatically re-run init. ### Defining localized emails Each email that should be localized should be defined in its own folder under `lib/email`. ``` lib/email/myEmail - index.ts // code for constructing email - i18n.json // terms for subject and other fields - body.i18n.md // email body in markdown - textBody.i18n.md // email text body in markdown ``` Format localized strings by using the formatter util defined in `./locale.ts`. ```ts import { getFormatter } from '~/app/locale'; import locales from '~/locales/fragments/myFeature'; // create a formatter for French using fragment-specific locales const t = getFormatter(locales, 'fr'); const frenchString = t('myKey', { someArg }); ``` ### Syncing remote translations Talking to POEDitor requires a `POEDITOR_API_TOKEN`. Get it from your team mates or ask about it in the #frontend slack channel. Make sure your `.env` file contains the following: ```shell POEDITOR_PROJECT_ID=523757 POEDITOR_API_TOKEN=[your token] ``` We are syncing translations with POEditor using the `frontend-cli-i18n` tooling. The following command is run as part of the CI: ```shell pnpm i18n:sync ``` Running the i18n:sync command will upload new / updated terms and English translations to POEditor. Then download all translations in the supported locales. All locale files are stored in the `./locales/` folder.