# ML-Engine [insert diagram] ### Adding new model templates 1. Add new modules to `service/tasks` (see examples in `service/tasks/data_model.py`) 2. Add their endpoints and behavior to `service/tasks_controller.py` 3. Rebuild (see Deployment section of this README)! ## Setting up locally Requires you to install Docker (instructions vary depending on your OS) `apt-get install docker` Also install [docker-compose](https://docs.docker.com/compose/install/#install-compose) 1. While in data_processor dir, build container with `docker-compose build` 2. cd to data_processor dir and execute `bash run_container.bash` (this will pull environment variables from your local aws config. NB! change `--profile fansifter` to `--profile default` if you have default profile configured). Also notice the DATABASE_NAME env variable there. You should be able to see it from `localhost:12345/health` (and other endpoints configured in tasks_controller) *other useful commands:* - `docker ps` - get the docker ids - `docker logs {docker_id}` - see container logs.. substitute the {docker_id} block with container id from previous command You will need to rebuild container every time you do changes (unless you mount a shared volume) ## Setting up in production NB! This task has been automated by CodePipeline. You can skip this whole section and go tkae a look at CodePipeline (which follows the same steps - building container, pushing it to ECR, and then deploying a new fargate). We host container image in AWS Elastic Container Registry), and spin up the container service in ECS (Elastic Container Service). To push images to ECR (and set up the service), you need to configure your AWS access on your local machine. Either go edit `~/.aws/config` and `~/.aws/credentials` or run `aws configure` if setting up AWS credentials the first time. You can retrieve AWS key/secret from AWS console, under IAM (Identity & Access Management). ### Create and tag image 1. While in data_processor dir, build container with `docker-compose build` 2. Find the image name with `docker images` and tag the image (it should be `ml_engine_ml-engine`) ``` docker tag ml_engine_ml-engine 776891437216.dkr.ecr.eu-west-1.amazonaws.com/fansifter/ml_engine ``` #### Push image to ECR (Elastic Container Registry) 1. Authenticate with ECR (make sure you have latest version of aws cli). It should say "Login succeeded" ``` aws ecr get-login-password | docker login --username AWS --password-stdin 776891437216.dkr.ecr.eu-west-1.amazonaws.com ``` If this worked for you, skip to step 2. If you have multiple AWS profiles configured locally, and have created on for fansifter, execute this instead: ``` aws ecr get-login-password --profile fansifter | \ sudo docker login --username AWS --password-stdin 776891437216.dkr.ecr.eu-west-1.amazonaws.com ``` (notice that you can avoid using sudo with docker if create docker group and add your user there: ``` sudo groupadd docker sudo gpasswd -a $USER docker ``` 2. Push the container to ECR (`fansifter/ml_engine` is the name of the pre-created container repository) ``` docker push 776891437216.dkr.ecr.eu-west-1.amazonaws.com/fansifter/ml_engine ``` [AWS ECR CLI Official Docs](https://docs.aws.amazon.com/AmazonECR/latest/userguide/getting-started-cli.html) 3. Build a new service, defining 3 environment variables for the container. TODO! this should be using CodePipeline to automatically deploy service when merged # big_ml_engine Fargate containers have hardware limits. We need quite beefy nodes for ML processes (unsupervised learning can eat up a lot of memory). So for this reason, we need to provision plain EC2 nodes (or an auto-scaling group of those). An alternative is distributed training, but this takes quite a bit of more effort to properly set up. See http://big_ml_engine.fansifter.cloud/health (the DNS A records point to the private IP of the respective EC2 node). As of writing, it is r5.4xlarge (16 cpu, 128gb ram, $1.128 per hour, or ~$800 per month). ### Setting up Create and prepare a memory-optimized EC2 (ubuntu ami) node and then: first login: `ssh -i ~/.ssh/fansifter_ec2.pem ubuntu@172.31.65.158` 1. install docker (`apt-get install docker` or `snap install docker`) 2. install aws cli (https://docs.aws.amazon.com/cli/latest/userguide/install-cliv2-linux.html). Might need to also `apt-get install unzip` (part of awscli v2 installation) run `aws configure` (key and secret in Secrets manager (ml_engine_env_vars), region is eu-west-1, output format json) 3. Login to ECR: `aws ecr get-login-password | \sudo docker login --username AWS --password-stdin 776891437216.dkr.ecr.eu-west-1.amazonaws.com` 4. Pull image: `sudo docker pull 776891437216.dkr.ecr.eu-west-1.amazonaws.com/fansifter/ml_engine:latest` 5. Start up container (see the --restart option for more controls) ``` sudo docker run --restart=always -d -p 80:12345 \ -e AWS_ACCESS_KEY_ID=$(aws configure get aws_access_key_id) \ -e AWS_SECRET_ACCESS_KEY=$(aws configure get aws_secret_access_key) \ -e DATABASE_NAME=fansifter \ 776891437216.dkr.ecr.eu-west-1.amazonaws.com/fansifter/ml_engine:latest ``` To see latest 100 lines from logs, use `sudo docker logs -f --tail 100 31956eeffe64` To see if it's working properly, go to http://172.31.65.158/health (it should say "healthy"). NB! Perhaps you can merge steps 3-5 with docker-compose (see run_big_ml_engine.sh and docker-compose-big-ml.yml) ### #TODO - Auto-deployments (e.g. invoking a lambda function from CodePipeline alongside the fargate deployment), which will pull the latest image, stop the current container (gracefully) and start up the latest image. This could also be a scheduled cron tasks (but CodePipeline would be nicer). - Auto-scaling groups - currently it's a single node, but would be nice to have a load-balancer, the DNS record pointing to the load-balancer, and having an ASG (auto-scaling group) for a bunch of EC2 nodes. This is something that we'd need to sort out in context of App Autoscaling objective w Toomas.