Requests ======== This library is designed to help construct requests that are sent from one microservice to another one. This library is fully compliant with: * [Security Requirements]( https://docs.google.com/document/d/1eHoI_BddTFMi15yCaHS6KvhSSoTrMEd3WwJINIpgNpM/) * [Logging Requirements]( https://docs.google.com/document/d/1rafa6dzWNcrrUengXvAEbC210i85ItzyUBoiw5vw7PY/) Installation ------------ ```bash make pip_dev ``` Usage ----- #### Testing Linting can be run using the command `make lint`. Pytest tests can be run locally using a dockerized dynamodb instance. You can automatically bring it up and down for tests by running `make test`. #### Service Discovery This is an older document that reflected some initial thoughts on Service Discovery: https://docs.google.com/a/theorchard.com/document/d/1gkrtBTOf8CvGp_czbquK8kdftbzw-c-FbYwd8TWAg1Q/) python-owsrequest supports basic service discovery, given: - `environment`: `prod`, `qa`, `uat`, and `dev`. - `service_name` - `OWSREQUEST_SERVICE_MAP` environment variable, a stringified json dictionary, where the key is the service_name, and the value is the full url, including protocol. For example: ``` OWSREQUEST_SERVICE_MAP="{\"ows-permissions\": \"https://qa-ows-permissions.theorchard.io\", \"ows-account\": \"http://localhost:8888\", \"ows-participant\":\"https://qa-ows-participant.theorchard.io\"}" ``` When the environment is `qa` or `prod`, python-owsrequest will always "discover" the service_name `ows-account` as: - For `qa`: https://qa-ows-account.theorchard.io - For `prod`: https://prod-ows-account.theorchard.io When the environment is `uat` or `dev`, python-owsrequest will try to find the service_name and use `OWSREQUEST_SERVICE_MAP`. Thus, if the `OWSREQUEST_SERVICE_MAP` is expressed as above, python-owsrequest would "discover" the service_name `ows-account` as http://localhost:8888. When the service_name is not in `OWS_REQUEST_SERVICE_MAP`, python-owsrequest will "discover" the service_name `ows-account` as: - For `uat`: https://uat-ows-account.theorchard.io - For `dev`: https://qa-ows-account.theorchard.io If the `environment` is something other than `prod`, `qa`, `uat`, or `dev`, python-ows-request will "discover" the service_name `ows-account` as https://qa-ows-account.theorchard.io. #### Making a Request This method uses python-requests to create and serve an HTTP Request to one of our microservice. First, it resolves the service name to the corresponding domain, then it creates the HMAC for the request, and finally it sends the request. In order to make a request, you need the following information: * Service information (discovery): * Name of the service. * Environment name. * Signature (security): * New Correlation Id (chained from the parent). * HMAC Signature Example: ```python from owsrequest import request # Example of calling a service request.process( # Information about your application 'your-application-name', 'application-environment', # Request information 'GET', 'service-name', '/resource', 'correlation-id', # Additional options, those are provided to python-requests. headers={ 'Grass-Account-Type': 'vendor' }) ``` > Note: owsrequest will retry requests that result in a 408, 502, 503, or 504 > status. The number of retries defaults to 5 and can be configured using the > `OWSREQUEST_MAX_RETRIES` environment variable. If you plan on doing more than one request, you should create partials of the requests, so you don't have to consistently provide all those information. For instance: ```python from functools import partial from owsrequest import request as owsrequest # You should have one per method, this is just an example request = partial( owsrequest.process, 'your-application-name', 'application-environment') # And consuming this method: request('GET', 'service-name', '/resource', 'correlation-id', headers={ 'Grass-Account-Type': 'vendor'}) ``` #### Authorizing requests Incoming requests can be authorized by using the method `confirm_authorization`. This method returns a boolean: `true` if the user is authorized, `false` otherwise. Example: ```python is_authorized = confirm_authorization( 'environment', 'authorization header', 'Incoming correlation id', 'HTTP Request method', 'Request Body', 'Request Path'): ``` Grass Headers ------------- There are 2 set of grass headers that are/will be supported by ows-grass. They are: 1. `Grass-Account-Type` + `Grass+Account+Id` + `Orchard-User-Id` => this is currently in use by all systems. 2. `Orchard-Profile-Type` + `Orchard-Profile-Id` + `Orchard-Identity-Id` => New headers as we move towards using profile based system instead of implicit label based. There are helper functions to validate each set of headers. See section below. Validating Grass Requests (helper functions) -------------------------------------------- `flask_request` library includes helper functions that can be used in microservice to validate the Grass headers from an incoming request. Different validations that can be performed for different Grass headers. These functions are: 1. request_context_from_headers (route decorator) 2. verify_grass_access 3. verify_grass_ownership 4. verify_profile_headers 5. verify_profile_headers_match_route 6. verify_rules_access_standalone This section summarizes what each function does, when each should be used, and how each should be used. ### `request_context_from_headers` This method is a flask route decorator which inspects the current request headers and determines the context class. The resulting RequestContext() class will be available on `flask.g` as `g.request_context` to be accessed in any layer of your microservice. #### Usage Import `flask.g` and the decorator function. ```python from flask import g from owsrequest.flask_request import request_context_from_headers ... ``` Decorate your route. ```python @app.route('/features', methods=['GET']) @request_context_from_headers() def get_features(): ... ``` The resulting `request_context` class is in `flask.g` for relevant context details. It will have the following attributes: - `context_type` (str): `'profile'` or `'account'` based on the request headers. - `profile_type` (str): `Orchard-Profile-Type` or `Grass-Account-Type`respective of context. - e.g.: `'ArtistProfile'` vs `'vendor'` - `profile_id` (str): `Orchard-Profile-Id` or `Grass-Account-Id` respective of context. - `identity_id` (str): `Orchard-Identity-Id` a.k.a. Auth0 user id. - `orchard_user_id` (str): `Orchard-User-Id` ONLY present in `'account'` context. - e.g.: `'alw:777'` or `'oa:1043'` - `roles` (list): `Orchard-Roles` present in BOTH `'account'` & `'profile'` context. - e.g.: `['catalog', 'analytics']` or `['admin']` ```python if g.request_context.context_type == context.PROFILE_CONTEXT_TYPE: result = features_logic.get_all_features_with_profile_context( g.request_context.profile_type, g.request_context.profile_id, g.request_context.identity_id) headers = {'Cache-Control': 'max-age=10'} return flaskify(result, headers) ``` You could also now access `flask.g` from any layer of the app. ### `verify_profile_headers(flask_request)` [verify_profile_headers](https://github.com/theorchard/python-owsrequest/blob/master/owsrequest/flask_request.py#L276) is a validator method to ensure that both the profile headers are present or none at all. Grass will always send both Profile type and profile id headers but microservice-to-microservice can skip sending any headers at all. This function allows both type of calls, but return 403 error if request has partial headers (from spoofing). #### Usage Example 1 > handler.py ```python from owsrequest import flask_request @app.route('/resource_name', methods=['GET']) def get_resource(): access = flask_request.verify_profile_headers(request) if not access: return access # Do other stuff ``` ### `verify_profile_headers_match_route(flask_request, profile_type_in_route, profile_id_in_route)` [verify_profile_headers_match_route](https://github.com/theorchard/python-owsrequest/blob/master/owsrequest/flask_request.py#L294) is a validator method to ensure profile headers match profile values in route url. This function is for endpoints that take profile_type and profile_id in url params. This function: * Grass calls + url profile params => Grass calls will always send profile headers (**for new profile users). This fn will enforce that url prams match header values so a user cannot access other user's profile information. 403 error response if they don't match. * Microservice to microservice call => They wont have headers as this is backend call (not via grass) so allow it to access any profile. This assumes that the caller microservice if it was called via grass did the header match to verify that the profile and the caller are the same. So no need of any dupe checks here. * Grass calls + no url profile param => Then you should not use this function. You should use `verify_profile_headers` #### Usage Example 1 > handler.py ```python from owsrequest import flask_request @app.route('/users/profile-type//profile//resources/all', methods=['GET']) def get_resource(type, id): access = flask_request.verify_profile_headers_match_route(request, type, id) if not access: return access # Do other stuff ``` ### `get_ows_headers` This method can only be used in a route that has used request_context_from_headers. It will give you a dictionary ready to pass to a request header that will pass along the current headers despite whether you have the old or new style headers. #### Usage ```python from owsrequest.flask_request import get_ows_headers user_response = request.get( OWS_USERS, '/users/auth0/email/{}'.format(email), headers=get_ows_headers()) ... ``` ### `verify_grass_access` [verify_grass_access in owsrequest](https://github.com/theorchard/python-owsrequest/blob/master/owsrequest/flask_request.py#L177) is a validator method to ensure a specific dataset matches grass headers. Here are some use cases: * Requiring Grass headers to be present * Enabling only one Grass-Account-Type to be able to access an API route * Ensuring an API route which contains vendor id or subaccount id matches Grass headers Please follow through the examples below, which should all be implemented in the handler layer of the microservice. #### Example 1 If you have to require that Grass headers are present, but do not need to ensure the API route matches grass headers: ```python from owsrequest import flask_request @app.route('/resource_name', methods=['GET']) def get_resource(): access = flask_request.verify_grass_access(request, required=True, vendor=request.headers.get('Grass-Account-Id'), subaccount=request.headers.get('Grass-Account-Id')) if not access: return access # Do other stuff ``` > handler.py Note that you should always specify what Account types (vendor or subaccount) are allowed for the given route. Thus, *DON'T* because it will always return a `403`: ``` access = flask_request.verify_grass_access(request, required=True) ``` #### Example 2 If you require that Grass headers are present, and need to ensure the API route is only accessible to vendors, but not subaccounts: ```python from owsrequest import flask_request @app.route('/resource_name', methods=['GET']) def get_resource(): access = flask_request.verify_grass_access(request, required=True, vendor=request.headers.get('Grass-Account-Id')) if not access: return access # Do other stuff ``` > handler.py #### Example 3 If you do not require that Grass headers are present, and you want to make sure your API route (`///resource_name`) matches Grass headers. ```python from owsrequest import flask_request @app.route('/account_type/account_id/resource_name', methods=['GET']) def get_resource(account_type, account_id): access = flask_request.verify_grass_access(request, required=False, account_type=account_id) if not access: return access # Do other stuff ``` > handler.py #### Example 4 If you do not require that Grass headers are present, do not need to limit access per Grass Account Type, and don't have any account-specific information in the API route to match against: ```python @app.route('/resource_name', methods=['GET']) def get_resource(account_type, account_id): # Do not call flask_request.verify_grass_access # Do other stuff ``` > handler.py ### `verify_grass_ownership` [verify_grass_ownership in owsrequest](https://github.com/theorchard/python-owsrequest/blob/master/owsrequest/flask_request.py#L202) is not a validator but rather a method that defers the decision to another method. This method captures the Grass headers from the incoming Flask request, and provides the information to the callable method. The callable method must take two parameters: `account_type` and `account_id`. #### The Example If you require that the Grass headers belong to a user who owns a product, which is information available via another microservice: ```python from owsrequest import flask_request from app_name.logic import product @app.route('/product/', methods=['GET']) def get_resource(product_id): ownership = flask_request.verify_grass_ownership(request, product.is_owner, product_id) if not ownership: return ownership # Do other stuff ``` > handler.py ```python from app_name.models import product def is_owner(upc, account_type, account_id): return product.is_owner(product_id, account_type, account_id) ``` > logic/product.py ```python from oto import response from owsrequest import request PRODUCT_SERVICE = 'ows-product' PRODUCT_OWNER_RESOURCE = '/{account_type}/{account_id}/product/{product_id}' def is_owner(product_id, account_type, account_id): resource = PRODUCT_OWNER_RESOURCE.format( account_type=account_type, account_id=account_id, product_id=product_id) ownership = request.head(PRODUCT_SERVICE, resource) return response.Response(status=ownership.status_code) ``` > model/product.py Keep in mind: * A model layer is implemented to handle interactions between this microservice and the product microservice. * A logic layer separates the model from the handler (even though it looks like it is entirely a pass-through, this will enable you to keep your handler from directly calling your model layer). * The handler layer is responsible for calling `verify_grass_ownership` and supplying the callable method (`product.is_owner`). ### Using both `verify_grass_access` and `verify_grass_ownership` You should absolutely be empowered to use both `verify_grass_access` and `verify_grass_ownership` within the same handler method. ### `verify_rules_access` and `verify_rules_access_standalone` #### `verify_rules_access` [pending deprecation] `verify_rules_access` is pending deprecation, in favor of `verify_rules_access_standalone`. The approach hooks into `before_request` and makes it difficult to chain authorization checks to Permissions Platform. #### access_rules configuration file You can name it however - most services have landed on `access_rules.yml`. For each route + method, a list of Profile Types and Roles can be defined to authorize an incoming request. * `Orchard-Profile-Type` header is used to get the incoming request's profile type * `Orchard-Profile-Id` header must be present too, although no validation is performed to assert the profile id is associated with the profile type * `Orchard-Roles` header is used to get the incoming request's role * `Orchard-Requestor-Service` header is required to be present for any rules to be checked -- it must be graphql-* or ows-grass ``` rules: - path: /hi methods: [GET] profiles: LabelProfile: ['catalog'] OrchAdminProfile: ['*'] - path: /something/ methods: ['*'] profiles: LabelProfile: ['catalog'] ``` #### How to use `verify_rules_access_standalone` `verify_rules_access_standalone` was created after `verify_rules_access` in order to more easily port each handler from using profile-based headers to additionally use Permissions Platform authorization checks. Because many teams already have an existing `access_rules.yml`, this approach is to maintain as much compatibility with the logic in `verify_rules_access`, but to break out of using the Flask `before_request` middleware. You should not use both `verify_rules_access` and `verify_rules_access_standalone` at the same time. The downside of this approach is that if you do not call `verify_rules_access_standalone` from the handler, the default behavior is to Authorize access. ```python import flask from flask import request from owsrequest import flask_request from owsresponse import response from yourapp import config application = flask.Flask(config.service_name) flask_request.set_rules_validator( application, 'access_rules.yml', ) flask_request.setup( application, config.environment, add_request_context=True, verify_access=False, # The default is False rules_file=None, # The default is None ) ``` ```python # handlers.py @application.route('/hi', methods=['GET']) auth = flask_request.verify_rules_access_standalone(request) if not auth: # call PP Authorization check auth = hypothetical_pp_auth_check(something, something) if not auth: return response.create_error_response( code=error.ERROR_CODE_AUTHORIZATION, message='Unauthorized', status=401) ``` #### How to use `verify_rules_access` `verify_rules_access` was created to support checking per-endpoint/method access given the headers for `Orchard-Profile-Type` and `Orchard-Roles`. It existed before `verify_rules_access_standalone`. You should not use both `verify_rules_access` and `verify_rules_access_standalone` at the same time. ```python import flask from owsrequest import flask_request from yourapp import config application = flask.Flask(config.service_name) flask_request.setup( application, config.environment, add_request_context=True, verify_access=True, rules_file='access_rules.yml', access_log_only=False, ) ``` ```python # handlers.py @application.route('/hi', methods=['GET']) def handle_hi(): # Only allowed for LabelProfile and catalog role # or OrchAdminProfile with any role @application.route('/hi', methods=['POST']) def post_hi(): # Only allowed for OrchAdminProfile with any role # via the final wild card rule ``` Error Responses ------------ Some common header-based validation responses are available for use in `access.py`. Especially useful when used in conjunction with [request_context_from_headers](#request_context_from_headers). Example (assuming you have decorated your route with [request_context_from_headers](#request_context_from_headers)): ```python from owsrequest.access import create_error_incomplete_account_headers from owsrequest.constants import CONTEXT_TYPE_ERROR @app.route('/features', methods=['GET']) @request_context_from_headers() def get_features(): if g.request_context.context_type == CONTEXT_TYPE_ERROR: return flaskify( create_error_incomplete_account_headers()) ``` Flask Plugin ------------ The Flask plugin adds an additional layer which automatically takes care of: * Authenticating incoming requests. * Signing outgoing requests to other microservices (HMAC and Correlation Id) Sample: ```python import flask from owslogger import flask_logger from owsrequest import flask_request from owsrequest import request application = flask.Flask('service-name') flask_logger.setup( application, 'loggly http/s url', '{environment}', 'logger_name', logging.INFO, 'service_name', '1.0.0') flask_request.setup(application, '{environment}') @application.route('/hi') def handle_hi(): # Calling a Microservice: response = request.get( 'service-name', '/resource', headers={'...': '...'}) # ... ``` #### Celery and Flask Workaround While not an ideal solution, in scenarios where Celery workers are implemented in the same repository as Flask applications you can do something like the following: ```python from celery import Celery import flask from owslogger import flask_logger from owsrequest import flask_request from owsrequest import request application = flask.Flask('service-name') flask_logger.setup( application, 'loggly http/s url', '{environment}', 'logger_name', logging.INFO, 'service_name', '1.0.0') flask_request.setup(application, '{environment}') application.config['CELERY_RESULT_BACKEND'] = 'RESULT_BACKEND' application.config['BROKER_URL'] = 'BROKER_URL' # and so forth... celery = Celery(app.name) celery.conf.update(app.config) TaskBase = celery.Task # ... ``` This will enable your Celery worker to use the Flask plugin for making requests. For a concrete example, check out [ows-masters-registry](https://github.com/theorchard/ows-masters-registry/blob/master/application.py#L35) Mocking OWS-Request in Flask Applications ----------------------------------------- This package provides pytest plugin that automatically mock all owsrequest calls. All unexpected calls will raise `OwsrequestMockException`. You can register call specifications to identify what call are expected in your test: ```python from owsrequest import request from owsrequest.utils import mock_request def test_direct_path_match(): mock_request.post('ows-awesome', '/awesome-thing/123', {'foo': 'bar'}) mock_request.get('ows-awesome', '/awesome-thing/123', status=403) response = request.get('ows-awesome', '/awesome-thing/123') assert response.message == {'foo': 'bar'} ``` It also supports regular expressions: ```python from owsrequest.utils import mock_request mock_request.post('ows-awesome', r'/awesome-thing/\w+') ``` If response not provided, it will be empty dictionary `{}`. Mocking method returns mocks for provided specification, so it can be checked: ```python from owsrequest.utils import mock_request mock = mock_request.post('service', '/path/smth', {'foo': 'bar'}) ... assert mock.called assert len(mock.calls) == 2 assert mock.calls[0]['method'] == 'GET' assert mock.calls[1]['method'] == 'POST' ``` If you don't want plugin to patch ows-request during your test you can mark it to skip this: ```python import pytest @pytest.mark.no_owsrequest_patch def test_processing(): pass ``` Running with DynamoDbLocal -------------------------- Our solution for owsrequest-authorization requires a DynamoDB database table. When working in dev, you can use DynamoDbLocal, by setting four environment variables, describing the DynamoDbLocal endpoint, access key, secret access key and region. Please see the documentation on AWS for [DynamoDbLocal](http://docs.aws.amazon.com/amazondynamodb/latest/developerguide/DynamoDBLocal.html) to set up DynamoDbLocal. You will need to set these environment variables: ``` DEV_DYNAMO_ENDPOINT_URL DEV_DYNAMO_ACCESS_KEY_ID DEV_DYNAMO_SECRET_ACCESS_KEY DEV_DYNAMO_REGION (defaults to us-east-1) ``` Running Dockerized DynamoDb --------------------------- To bring up a local dynamoDB instance, run `docker compose up -d`. This is required to run tests locally. Setting Protocol ---------------- Individual requests may receive a protocol parameter of either 'http' or 'https'. If no protocol is specified, requests will be made using https. This default can be overridden by setting the `SERVICE_DEFAULT_PROTOCOL` environment variable. Adding jwt to m2m requests ---------------- To add m2m jwt in requests between 2 services and forward an incoming jwt if available, - The services should have permissions to access jwt secrets. - The services need to have a terraformed fargate module version of >=1.3.0 in both QA and PROD, for these permissions to apply to the services. - https://github.com/theorchard/terraform-fargate/releases/tag/1.3.0 - Reference: https://github.com/theorchard/terraform-infra/pull/866/files - The sender and receiver services that communicate should be running on uwsgi. - Add uwsgi caching configurations in uwsgi-start.sh file of the service that configures and runs uwsgi in both services. - ```--cache2 name=uwsgi_cache,items=10``` - Upgrade ```owsrequest``` version to ```>=0.31.8``` in the sender and receiver service. - Require ```flask``` version to be ```>=1.1.1``` in both sender and receiver service. - To enable uwsgi caching for m2m jwt and for forwarding jwt need add below code in microservice, ```python ... try: import uwsgi # noqa uwsgiRunning = True except ImportError: uwsgiRunning = False ... ``` and require to pass the below parameters in ```flask_request.setup``` function. ```python uwsgi_cache_enabled=uwsgiRunning, add_request_context=True ``` - Reference: https://github.com/theorchard/ows-analytics/pull/620/files JWT Service List ---------------- Although this package will automatically update the list of jwt enabled services, we have migrated the management of it to be through terraform. Thus, please be sure to make PRs against [`terraform-infra`](https://github.com/theorchard/terraform-infra) to add your service to `qa/lambda-jwt-refresh/jwt_enabled_services.tf` and `prod/lambda-jwt-refresh/jwt_enabled_services.tf`. While we continue to make sure all parts of our application support JWT Bearer tokens, we need to maintain this list.