Design ====== .. toctree:: :maxdepth: 4 Overview -------- With our increasing usage of decoupled systems (clients, etl jobs), we need to scale our API layer to manage better CRUD operations, and advanced requests (bundling, multi-processing), while providing granular access to endpoints and advanced features such as threading. Please note: this tech design focuses on the API, questions regarding the *caching strategies*, *databases* will be addressed in different tech design as they belong more in the logic layer rather than the handler and response mechanisms. Requirements ------------ Our API system has to support all the following features to be in a good state and allow us to seamlessly scale: * Allow threading to facilitate parallel computing, and create non-blocking operations. * Provide a built-in extensible ACL system: ``token``, ``user`` and ``client`` will be required by default. – which will allow us to always track client consumptions of our endpoints. The ACL system will also handle session (for user browser requests.) * Consistent API responses: all API responses will have the exact same format (header, json body) which will help API clients to consume immediately the data returned without the need of performing additional transformations. * Caching capabilities: provide a lightweight and simple way to store the data into a cache (local to the machine, or on a caching service / database such as ElasticCache or redis.) * Allow the use of multiple databases seamlessly by providing an abstract layer of our models. * Easy to write and unit testable handlers: handlers should be simple methods that consume our logic layer * Allow versioning. * Enable logs on handlers: to understand better what our API clients are consuming and the performance, an abstraction of the log layer should be available by default. Analysis -------- Today, our applications rely on the Zend architecture. There are mechanisms in place to allow a little of DRY and SoC, but little is done when it comes to build a scalable architecture, due to the very nature of php. Our `current api `_ follows some basic API aspects: it provides different controllers, and the API is capable of managing the data on its own. However, our existing system has some limitations (either due by language or by the frameworks we’re using): * Single threaded, initiated on every request: it’s impossible to parallelise small tasks, everything runs in a procedural fashion. * No easy to use ACL system. * Some of our `API responses `_ are currently cached: if any of the model has been updated, we will need to wait until the cache expires to be able to reflect this change. It means that any system that depend on high consistency will not be able to use our API. To perform high consistency: no full api response should be cached, instead parts of the logic (whenever needed and appropriate) should be cached. * The API response can be inconsistent (`code shows instability `_ and cache dependency): it sometimes returns the json of the requested item, but at other times, it provides an “envelope”. * The API does not `use consistently `_ the HTTP status to provides more details about the response. * Implementation of global behaviors (such as logging) will add significant latency. Scaling VAPI to handle all the above requirements and to fix the underlying issues of Zend/Php will require us more work than progressively moving to a new scalable framework. Identified Solution ------------------- Moving to Python3 ^^^^^^^^^^^^^^^^^ Python3 is a well-supported language with hundreds of high performance libraries, including AWS Boto. The language handles basic structures (list, set, dictionary), and provides more advanced structures (queues.) Writing documentation and tests is easy, and has been done on the accounting run. In addition to those lightweight data structures, python offers more advanced features such as threading. It means that we can easily write in python non blocking operations. For instance: if you have 3 fetches that don’t require each others, you can easily spawn 3 threads, get the answer from each and regroup. Example: .. code:: python threads = [] for query in queries: t = Thread(query) t.start() # Launch the process threads.append(t) for t in threads: t.join() # block until all threads have completed. **Questions** *How long does it take to learn python?* For experienced engineers, learning the basics of python takes a few hours. The syntax is very similar to php/javascript and the documentation provided explains the language and the different features available. The python community is large, so it is rare to not find something that has not been already done by someone else. What about coding styles? Python has only one styleguide. *What are going to be the main differences?* While python allows you to create classes and objects, the language itself focuses more on modules and packages. It means that you will only create objects when it’s absolutely necessary. For instance, today, all our handlers are inheriting the zend controller class. In the new system: our API is an object, and we will be attaching handlers. Overall Application Structure ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ The application structure is simple and encapsulates the notion of separation of concern. It uses the following architecture: API client ^^^^^^^^^^ The api client is responsible for performing actions on the behalf of the user. For every request, the API client provides a ``token``, ``user`` and ``client``. Those 3 parameters provide enough information for the ACL system to ask “Can this specific client, for this specific user access this specific endpoint.” ACL ^^^ The access control list (ACL) system provides a lightweight mechanism to perform small atomic checks on whether or not an endpoint can be accessed. The ACL check goes through all the provided methods, and returns on the first one that grant an access. If none of the provided methods grant access, the ACL checks have failed and the api returns an Unauthorized status (401) code. API Handler (managed by flask) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ API handlers are small, discrete methods that are in charge of performing calls to the logic layer, consuming those responses (including possible transformation) and sending back a json response (with valid http status.) Our handlers will be managed by Flask, which is an easy-to-extend micro framework. The base of flask framework allows us to create and customize our api application by writing our own extensions. For instance, the default flask route behavior does not support ACL. We were able to extend flask to add this Orchard-specific feature. In addition, whenever we need, we can always take a look at the available extensions and use them. The main advantage of flask is its simplicity. It only takes a few lines to write new api handlers. Two API Handler Examples:: @route('/releases/', access=[OA_CLIENT], methods=['GET']) def releases(user): """Return all the releases for a specific user. """ return logic.releases.get_all_for_user(user.id) @route('/release/', access=[OA_CLIENT], methods['GET']) def release(user, id): """Return a release information that belongs to a user. """ release, status = logic.release.get_release_for_id(id) if not release.labelid == user.id: abort(401) return release, 200 Logic Layer ^^^^^^^^^^^ The logic layer is responsible for all the small tasks that require data fetching, data transformation, and data bundling. By having these discrete tasks, it means that this layer is an intermediary between the API Handler and the models. To standardize the implementation and usage of Logic layer modules, the Logic layer will return only a tuple, containing a data structure (list, dictionary, boolean) and a http status code. **Questions** *Why do we need this layer?* The best way to answer this question is take the counter example: what if we don’t have this layer and let the handler do the operations. It means that the API handlers will be responsible for not only decoding the request, but they will also be responsible for fetching different pieces of information. Let’s take an example: you have 3 api handlers that need to get a release by its id and get the tracks. Overall, you will repeat this code 3 times. Ideally for this case: you will need to extract this duplicated code into a separate method – which is what the logic layer is. *Why not just do it on the model layer?* In modern applications, models can live in many different databases and cache layers. Sometimes models need to be bundled together. All those transformations and fetches will be easier to maintain on a logic layer rather than a model layer (the model is just a connection between our application and the model representation within our database). An additional point to consider: models should not be aware of other models. If this ever happens, it means you can run into a cyclic dependency, which can be difficult to resolve. *Why is an HTTP status needed on the logic layer?* First: HTTP statuses are awesome, they provide essential information on how your data has been handled, highlighting any errors that may have occurred (permissions as well as failures.) Now consider: whenever we use external services (AWS for instance, or plain API systems), we make requests to those. Some of them will return http statuses (e.g. 403 – access forbidden, or 404, not found, 500 - internal errors). We need to be able to take actions on them on the logic layer, and those might also impact on how the handler is creating its response. Example: YouTube API raises a 500. If the logic layer cannot bubble those, it means that we will return an empty object to the handler. This handler will then not be able to tell the API client that something went wrong with the YouTube API. *Is an HTTP status required on all methods that are in the logic layer?* No. If you have an helper method that is only consumed within the logic layer, this method does not need to return an HTTP status. It means the scope of the helper method must be such that only the logic layer can consume it - the API Handler would not be able to call it directly. For instance: if you have a method that cleans an object to remove a field (let’s say password) - this method does not need an HTTP Status. Model layer ^^^^^^^^^^^ The model layer is a connector between databases, external 3rd party apis and our application. It means whenever you want to get data from a database, you will call the corresponding model. In addition to this responsibility, the models are also in charge of validating the data – we want to make sure the data that comes in (for any type of operation) is always in a clean state. To handle our models, we’ll start with two small frameworks: SqlAlchemy (SQL) and Schema (Dynamodb). Life of a Request ----------------- In the designed system, this will be the life of a request. Please note: * Logic object in the diagram is shown as abstracted because it can consume more than one model. The API focuses on sending information to the logic layer, and this is what this diagram represents. * Caching and logs are not represented in the diagram below. Access logs will be part of the API Handler layer, caching will be part of the Logic layer. **Questions** *How can we use this api in javascript?* First, to use this API in javascript, we need to allow an API endpoint to be publicly accessible. For a browser to perform a request, a session id will be required. This session id is generated by the api client (oa, alw), it makes a request on /auth/session/ which returns a session id, and after this request completes, the client generates a cookie with the returned value in it. After this step, the javascript can fetch directly the api. One thing to remember: those cookies have a short life (several minutes). The api client will be responsible for refreshing it on a regular basis. *What’s the performance like?* Myron has a very lightweight project which requests data from a dynamodb table. In his performance test, it was taking 1.5s to process the request. With Grass, benchmark (on a local machine) shows performance around 50ms for the same request. That’s 30x times faster .. code:: $ ab -n 500 localhost:5000/stats/label/16998/?date=5550,5570 Benchmarking localhost (be patient) [...] Server Software: Werkzeug/0.9.6 Server Hostname: localhost Server Port: 5000 Document Path: /stats/label/16998/?date=5550,5570 Document Length: 47710 bytes Concurrency Level: 1 Time taken for tests: 26.805 seconds Complete requests: 500 Failed requests: 0 Write errors: 0 Total transferred: 23929000 bytes HTML transferred: 23855000 bytes Requests per second: 18.65 [#/sec] (mean) Time per request: 53.610 [ms] (mean) Time per request: 53.610 [ms] (mean, across all concurrent requests) Transfer rate: 871.78 [Kbytes/sec] received Percentage of the requests served within a certain time (ms) 50% 49 66% 51 75% 53 80% 55 90% 61 95% 67 98% 86