OWS Podcast microservice ===================================== The boilerplate defines the basic structure of flask applications that will live in AWS Fargate. This boilerplate, in addition to the classic python stack, includes support for Datadog, Sentry, and Loggly. # Quick Links | Repository | Links |---|---| | frontend-podcast | [GitHub: [code](https://github.com/theorchard/frontend-podcast) / [PRs](https://github.com/theorchard/frontend-podcast/pulls)] [Jenkins: [Pipeline](https://pipeline.theorchard.io/job/frontend-podcast-pipeline/) / [PRs](https://pipeline.theorchard.io/job/frontend-podcast-pull-request/) / [E2E](https://pipeline.theorchard.io/job/frontend-podcast-e2e-cypress-tests/)] | podcast-gateway | [GitHub: [code](https://github.com/theorchard/podcast-gateway) / [PRs](https://github.com/theorchard/podcast-gateway/pulls)] [Jenkins: [Pipeline](https://pipeline.theorchard.io/job/podcast-gateway-pipeline/) / [PRs](https://pipeline.theorchard.io/job/podcast-gateway-pull-request/) / [E2E](https://pipeline.theorchard.io/job/podcast-gateway-e2e-cucumber-tests/)] | ows-podcast | [GitHub: [code](https://github.com/theorchard/ows-podcast) / [PRs](https://github.com/theorchard/ows-podcast/pulls)] [Jenkins: [Pipeline](https://pipeline.theorchard.io/job/ows-podcast-pipeline/) / [PRs](https://pipeline.theorchard.io/job/ows-podcast-pull-request/)] | ows-asset-transcoder | [GitHub: [ows-asset-transcoder](https://github.com/theorchard/ows-asset-transcoder) ] | lambda-asset-transcoder | [GitHub: [code](https://github.com/theorchard/lambda-asset-transcoder) / [PRs](https://github.com/theorchard/lambda-asset-transcoder/pulls)] [Jenkins: [Pipeline](https://pipeline.theorchard.io/job/lambda-asset-transcoder-pipeline/) / [PRs](https://pipeline.theorchard.io/job/lambda-asset-transcoder-pull-request/)] | terraform-ows-podcast | [terraform-ows-podcast](https://github.com/theorchard/terraform-ows-podcast) (Archived) ## Environments ### DEV | Object | Links | |---|---| | podcast-gateway | [[WWW](https://podcast.dev.theorchard.io)] [[AWS](https://console.aws.amazon.com/ecs/home?region=us-east-1#/clusters/dev-podcast-gateway/services)] [[Logs](https://console.aws.amazon.com/cloudwatch/home?region=us-east-1#logStream:group=dev-podcast-gateway)] [No Sentry] [[LB](https://console.aws.amazon.com/ec2/home?region=us-east-1#LoadBalancers:search=podcast;sort=loadBalancerName)] | ows-podcast | [[API](https://dev-ows-podcast.dev.theorchard.io/podcasts)] [[AWS](https://console.aws.amazon.com/ecs/home?region=us-east-1#/clusters/dev-ows-podcast/services)] [[Logs](https://console.aws.amazon.com/cloudwatch/home?region=us-east-1#logStream:group=dev-ows-podcast)] [[Sentry](https://sentry.io/organizations/the-orchard/issues/?project=1807378)] [[LB](https://console.aws.amazon.com/ec2/home?region=us-east-1#LoadBalancers:search=podcast;sort=loadBalancerName)] | podcast-assets | [[S3 output](https://s3.console.aws.amazon.com/s3/buckets/dev-orcd-podcast-output-assets/)] ### QA | Object | Links | |---|---| | frontend | [[WWW](https://qa-frontend-podcast-standalone.theorchard.io/)] [[Sentry](https://sentry.io/organizations/the-orchard/issues/?project=1552977)] | podcast-gateway | [[WWW](https://podcast.qaorch.com/)] [[AWS](https://console.aws.amazon.com/ecs/home?region=us-east-1#/clusters/dev-podcast-gateway/services)] [[Logs](https://console.aws.amazon.com/cloudwatch/home?region=us-east-1#logStream:group=dev-podcast-gateway)] [No Sentry] [[LB](https://console.aws.amazon.com/ec2/home?region=us-east-1#LoadBalancers:search=podcast;sort=loadBalancerName)] | ows-podcast | [[API](https://dev-ows-podcast.dev.theorchard.io/podcasts)] [[AWS](https://console.aws.amazon.com/ecs/home?region=us-east-1#/clusters/qa-ows-podcast/services)] [[Logs](https://console.aws.amazon.com/cloudwatch/home?region=us-east-1#logStream:group=qa-ows-podcast)] [[Sentry](https://sentry.io/organizations/the-orchard/issues/?project=1552981)] | podcast-assets | [[S3 output](https://s3.console.aws.amazon.com/s3/buckets/qa-orcd-podcast-output-assets/)] ### PROD | Object | Links | |---|---| | frontend | WWW [[Sentry](https://sentry.io/organizations/the-orchard/issues/?project=1552979)] | ows-podcast | AWS API etc [[Sentry](https://sentry.io/organizations/the-orchard/issues/?project=1552982)] | podcast-assets | [[S3 output](https://s3.console.aws.amazon.com/s3/buckets/prod-orcd-podcast-output-assets/)] ### Installation Before starting make sure you have python3.4 installed. An easy way to manage python versions is with pyenv. ``` $ brew update $ brew install pyenv $ pyenv install 3.6 $ pyenv local 3.6 ``` For your convenience there is already a .python-version file set to 3.4.8 so all you have to do is `pyenv install 3.6`. If you use Windows, just ask for a remote VM that has it installed for you. ```bash $ cd ows-podcast make pip_dev ``` ### Permissions In PROD and QA environments, to accept and authorize incoming and outgoing requests, make sure you have the right DynamoDB permissions, as highlighted in the corresponding [tech design](https://docs.google.com/document/d/1eHoI_BddTFMi15yCaHS6KvhSSoTrMEd3WwJINIpgNpM/edit). ### Running #### Locally When all dependencies have been installed, you can run the flask application on your local instance by running: ```bash (env) $ python dev.py ``` or ```bash make dev ``` All make commands will automatically install all dependencies in an virtual environment folder called env. You can activate a virtual environment by running `. env/bin/activate` or `source env/bin/activate`. By using the development server, you will have access to specific features that are not necessarily available in production, such as the exception tracer. #### Locally with docker-compose You can also build and run microservice in Docker container. It uses MySQL database schema from ./dev/ows_podcast.sql Run the command: ```bash dicker-compose up --build ``` OR ```bash make dc ``` Open url: http://localhost:8080/podcasts You can also update SQL schema from AWS DEV account by ```bash cd dev make dump-all ``` #### Deploy to AWS container repository You can create new docker image and push it to DEV AWS account. Make sure you have fresh AWS temporally creds in your .aws/credentials. Run `dev/aws-creds-generator/generate.sh dev` to update them. ```bash $ make docker-login # run it once per session $ make docker-pipeline ``` After deploy check: https://dev-ows-podcast.dev.theorchard.io/podcasts ### Testing To run the tests, all you have to do is to run: ```bash (env) $ py.test tests/unit/ (env) $ py.test tests/integration/ (env) $ flake8 podcast/ tests/ ``` or To run unit tests: `make test` To run unit tests with coverage reporting: `make test_unit` To run integration tests: `make test_integration` To lint: `make lint` ### Updating To install new dependencies. ```bash (env) $ pip install -r requirements.txt (env) $ pip install -r requirements-dev.txt ``` or ```bash make pip_dev ``` ### Notes Windows users will not be able to use the `make` commands as Make is a Unix util. Windows users can attempt to install [GNUWin](http://gnuwin32.sourceforge.net/packages/make.htm) to get this functionality. ## Features of the Boilerplate ### Linting This boilerplate comes with customization on flake8 plugins. Please make sure you keep any derivative flask microservices in-sync with these standards, as they are added. ### Third-Party Integrations for Reporting This boilerplate comes with support for Loggly, Sentry, and Datadog. ### Grass Access Validation The boilerplate includes module `validation.access` which contains functionality for validating Grass headers in the context of a Flask request. This is valuable for making microservice endpoints compatible for both Grass and non-Grass (i.e. microservice-to-microservice) requests. It makes it easier to facilitate endpoint reuse. For example, suppose you need to validate that a request where Grass headers are required. Additionally, the `vendor_id` passed in the route must match - thus the Grass Account Type must be 'vendor' and the Grass Account Id must match `vendor_id`. You would use as follows: ```python @app.route('/vendor//something') def do_something_only_for_vendors(vendor_id): """Do something only for vendors. Args: vendor_id (int): unique identifier for vendor. """ validation = access.verify_grass_access( request, required=True, vendor=vendor_id) if not validation: return validation // do something ``` Another example - suppose Grass headers are not required. Additionally, the `subaccount_id` passed in the route must match. The only acceptable Account Type is subaccount: ```python @app.route('/subaccount//something') def do_something_only_for_subaccounts(subaccount_id): """Do something only for subaccounts. Args: subaccount_id (int): unique identifier for subaccount. """ account_type, account_id = access.get_grass_headers(request) validation = access.verify_grass_access( request, required=False, subaccount=subaccount_id) if not validation: return validation // do something ``` Now, building on the last example, we can accept Account Type of vendor AND subaccount: ```python @app.route('/subaccount//something') def do_something(subaccount_id): """Do something. Args: subaccount_id (int): unique identifier for subaccount. """ account_type, account_id = access.get_grass_headers(request) validation = access.verify_grass_access( request, required=False, vendor=account_id, subaccount=subaccount_id) if not validation: return validation // do something ```