## Introduction To schedule your (python) scripts, you'll need access to what we call the *api server* (bus-all-api.smeanalyticsapps.com). ???+ warning While the api server is actually not a server from a techincal point of view, but rather an AWS EC2 instance, we'll use the term "server" in this documentation to refer to the instance. Getting set up to use the api server can be tedious. The purpose of this documentation is to list the steps needed, and the possible issues that might arise along your way. ## Getting Started The first step in getting access to the api server is to reach out to **SME Cloudops**. To request access, please send an email to **Tom Koshy** (cc'ing Matti and Julien) saying that you would like access to *bus-all-api.smeanalyticsapps.com*. If you're not sure what to say, feel free to reach out to us. The following chapters might also be helpful. ### Personal user vs. Superuser Everyone actually has **two** users on the server. For logging in, you'll need your *personal user*, which is equal to your network ID. For running scripts, you'll use a *superuser*, which is team-specific. ???+ note "Is someone in your team already using the api server?" If someone on your team is already using the api server, you'll only need to request for a personal user, which will then be assigned to your team and thus have priviledges to use the superuser. ???+ note "Are you the first in your team to use the api server?" If you are the first in your team to use the api server, you'll also need to request cloudops to set up the superuser. ### Command Line Interface The server is used through a command line interface over ssh. If you're not familiar with working through the command line, this can be intimidating at first. But luckily, the number of commands needed for most things is actually pretty limited, so you get the hang of fast! The server is running Amazon Linux 2, based on the Fedora distribution, so the commands are unix-like. If you are new to working with a command line interface and unix commands, see [here](#command-line-interface-and-unix-commands) On mac, use the terminal. On Windows, use PowerShell or git bash. ### SSH Keys After requesting access, Cloudops will send you your *ssh private key*. You'll save this key on your computer in a .ssh folder. The default location of the .ssh directory is in your user directory: `C:/Users//.ssh` on Windows, `/Users//.ssh` on MacOS. The typical convention is to name the private key file `id_rsa.pem`. Note that Cloudops might also send you a PuTTY key, but this is useless. TOSS IT! ???+ warning "Known issues: File formatting" Make sure your private key ends with a newline character! ## Loggin In The api server can only be accessed via the so-called *bastion server*. This is just an additional security layer, but it makes logging in a two-step process. This also means, that twice as many things can go wrong, and it is unfortunately not unusual that your user is not immediately set up correctly on both servers. ### SSH Config To help you log easily onto the api server, you can create a config -file in the .ssh folder (note that this file has no file extension!). After adding the following commands into the file, you can log in simply by typing `ssh api`. ???+ note "Example config" ``` Host bastion HostName cloudops-all-bastion.smeanalyticsapps.com User IdentityFile ~/.ssh/id_rsa.pem MACs hmac-sha2-512 Host api HostName bus-all-api.smeanalyticsapps.com User IdentityFile ~/.ssh/id_rsa.pem ProxyCommand ssh bastion -q -W %h:%p MACs hmac-sha2-512 ``` ???+ note "Pro tip" By default, you'll get a message upon each login how *"this is a private computer system yada yada yada..."*. This might be cool and make you feel like a hacker at first, but if you want to suppress this message, type `ssh -q api` for logging in *quietly*. If logging in does not work, and you get an error message along the lines of `permission denied`, you can see if you can still log onto the bastion server by typing `ssh bastion`. Typically this access will be taken care of first. ## Running Scripts on the API Server After successfully logging onto the api server, the first thing you should do is switch over to your super user. ??? abstract "Switch to the superuser" `sudo -u -i bash` - `sudo`: *superuser do* - `-u`: *user* - `-i`: *interactive* - `bash`: *the type of shell you want* Congratulations! *You're in!* # ![image](img/hacker.png){: style="height:150px;width:150px"} If you're the first to use the superuser, there isn't gonna be much there. The first things you're probably going to want to do are: 1. [Creating the .env file and adding content to it](#1-creating-the-env-file-on-the-server) 2. [Cloning the infamous repository](#2-cloning-the-infamous-repository) ### 1. Creating the .env file on the server In the majority of (if not all) cases, your .env will be accessed by all of your projects and scripts. Therefore, we suggest that you place it in your superuser home directory. This is the directory that you will be in after switching over to the superuser, but you can always get back here with `cd ~`. Start by creating the file with `touch .env`. You can check that the file was created by command `ls -la`. Next, start editing your .env by opening the file in vim, by the command `vim .env`. You should now see a blank file in vim. #### *What Makes Vim Different from Typical Text Editors?* **Vim** is different from typical text editors, mainly for two reasons: 1. Vim is a **modal editor**—meaning it has different modes for editing, navigating, and issuing commands. 2. Unlike most editors that rely heavily on the mouse, Vim is designed for fast navigation and editing using the **keyboard only**. These features make it useful when working with a command line interface, but also more challenging to learn initially. ???+ note "Vim modes" - **Normal mode**: For navigation and commands. This is the mode you are in when opening a file with vim. Navigate the text using arrow keys or the h, j, k, l keys (left, down, up, right). - **Insert mode**: For typing and editing text (like a typical editor). Enter insert mode by typing `i` in normal mode, and exit by hitting Esc. - **Command mode**: For running commands like saving or quitting. Enter command mode by typing `:` in normal mode. ???+ note "Basic Vim Commands" - **Save and quit**: In Normal mode, type :wq and press Enter to save changes and quit. - **Quit without saving**: Type :q! to quit without saving. - **Save without quitting**: Type :w to save changes and stay in Vim. ???+ note "Example Workflow" 1. Open Vim: vim file.txt. 2. Press i to enter Insert mode and type your text. 3. Press Esc to return to Normal mode. 4. Type :wq to save and quit. ???+ note "Copying and pasting contents with Vim" Initially you are likely going to want to copy and paste the contents of your local .env file into the server side .env. Down the line, when your keys, passwords etc. get updated, you'll also likely want to simply copy the new credentials in stead of typing them out. Here's how to paste contents with vim. Start by copying the contents of the local .env to the clipboard. Then do the following steps: 1. Open the server-side .env `vim .env` 2. Enable paste mode with `:set paste` and hit Enter 3. Enter insert mode by hitting `i`. The bottom of the screen should now display `-- INSERT (paste) --`. 4. Right click to paste your contents. 5. Press esc to exit insert mode. 6. Save and exit by typing `:wq`, and hit enter. ### 2. Cloning the infamous repository The first thing you'll need to do is generate a GitHub PAT for you to use on the server. !!! tip "Steps to Generate PAT. *Click on steps for images on where to click and what to do.*"" ??? abstract "1. Go to your GitHub *Settings*:" # ![image](img/pat1.png){: style="height:325px;width:150px"} ??? abstract "2. Go down to *Developer settings*:" # ![image](img/pat2.png){: style="height:325px;width:450px"} ??? abstract "3. Click on *Personal access tokens* and select *Tokens (classic)*" # ![image](img/pat3.png){: style="height:150px;width:300px"} ??? abstract "4. Click on *Generate new token* and select *Generate new token (classic)*" # ![image](img/pat4.png){: style="height:150px;width:350px"} ??? abstract "5. Give a name for your token, set an expiration (Matti uses 90 days) and select a scope (*repo* is usually a good choice)" # ![image](img/pat5.png){: style="height:200px;width:325px"} ??? abstract "6. Gopy the token (!) and paste it in a file. This is the last time GitHub will display it for you." # ![image](img/pat6.png){: style="height:200px;width:400px"} ??? abstract "7. Click on *Configure SSO* and *Authorize* the token for SME-BUS." # ![image](img/pat7.png){: style="height:200px;width:250px"} Great! Now you can clone the repo! !!! tip "Steps to Clone the Repo *Click on steps for details.*" ??? abstract "1. Go to the infamous repo" ??? abstract "2. Copy the web URL (HTTPS)" `https://github.com/SME-BUS/cea-infamous.git` ??? abstract "3. Log onto the api server" `ssh -q api` ??? abstract "4. Switch over to your superuser" `sudo -u -i bash` ??? abstract "5. cd to wherever you want to clone cea-infamous on the server" `cd ~/WORKSPACE` ??? abstract "6. Clone the repo" `git clone https://github.com/SME-BUS/cea-infamous.git` ??? abstract "7. Enter your GitHub username when prompted" ??? abstract "8. Copy the PAT you created" **Pro tip**: You can simplify your life by creating functions to copy the PAT. Assuming your PAT is in a txt file called `pat.txt` in your home directory. For PowerShell users, put this in your Powershell `$profile`: ``` function pat { cat "$HOME\pat.txt" | clip } ``` For bash users, put this in your `.bashrc`: ``` function pat { cat "$HOME/pat2.txt" | pbcopy } ``` Now, you can copy the PAT by simply typing the comman `pat`. Note that you will need tp restart your terminal/PowerShell/gitbash for this to work. ??? abstract "9. Right click on your terminal to paste the PAT when prompted" - Note that you will not see the PAT getting pasted. Regardless, only paste it once! - You might have to hit enter after the right click to submit the PAT. And voilà! You have now cloned cea-infamous on the api server! ### Scheduling Scripts with CRON To schedule scripts, we use a utility called cron. Note that cron can only run files that are executable. This means that to run python scripts, you need create a shell script that contains the paths to the .py files. When you initialize a new project with `init_new_project.sh`, a boilerplate `run.sh` will automatically be created. ???+ warning "Known issues: Shell file is not executable" It often happens that a shell file does not have execute permissions. You can test this by simply typing the name of the file in bash. If you get an error saying that the file is not executable, you can change the permissions by the command: `chmod +x /path/to/file`. You can also run `check_crontab_permissions.sh` from the tools-section to check if all shell scripts in your crontab are executable. ???+ warning "Known issues: User not allowed to use cron" For some reason, the custom at SME is to not allow freshly created superusers to use cron. If you're setting up cron for the first time and get a permission error, reach out to Cloudops to get access. To interact with cron, use the `crontab` command. To edit your cron jobs, type `crontab -e`, which opens your crontab file in vim. Each line in the crontab file represents a cron job and follows this format: `minute hour day-of-month month day-of-week command`. For example, `0 2 * * * /path/to/script.sh` will run the script at 2 AM every day (see more examples below). To list all current cron jobs, use `crontab -l`. ???+ note "Cron examples" ??? abstract "Run every hour on the hour" `0 * * * * /path/to/command` ??? abstract "Run every day at midnight, logging output" `0 0 * * * /path/to/command >> /path/to/logfile.log 2>&1` The output (stdout) and errors (stderr) are both appended to `/path/to/logfile.log`. `>>` appends the output, while `2>&1` redirects errors (stderr) to the same file. ??? abstract "Run at 8:30 AM every weekday (Monday to Friday)" `30 8 * * 1-5 /path/to/command` ??? abstract "Run at 12:15 PM on the 1st and 15th of every month" `15 12 1,15 * * /path/to/command` ??? abstract "Run every 15 minutes between 9:00 AM and 5:00 PM, Monday through Friday" `*/15 9-17 * * 1-5 /path/to/command` ??? abstract "Run at 10:00 PM on the last day of every month" `0 22 28-31 * * [ "$(date +\%d -d tomorrow)" == "01" ] && /path/to/command` ??? abstract "Run at 6:00 AM on the first Monday of every month" `0 6 * * 1 [ "$(date +\%d)" -le 7 ] && /path/to/command` ??? abstract "Run every 3 hours, only on Monday and Thursday" `0 */3 * * 1,4 /path/to/command` ??? abstract "Run at 11:45 PM on the last Friday of every month" `45 23 * * 5 [ "$(date +\%d -d next-saturday)" -le 07 ] && /path/to/command` ### Command Line Interface and Unix Commands In essence, you can think of a command line interface (CLI) like this: You are in a specific location inside the computer. You have two options: 1. You can either move to another place on the computer 2. Or, you can do *something* in your current location Doing *something* can mean a whole bunch of things. For example, you can create or remove a file, display the contents of a file, edit a file, execute a script etc. While in a graphical user interface (GUI) you click on things to either create, remove or edit files, in a CLI you do this by typing commands, often followed by arguments that specify exactly what you want to do. The below lists some of the most used unix commands. Another command (which is not a unix command) is obviously the "python" or "py" command, which executes a python script whose path is passed an argument. ???+ note "Pro tip" Your computer is smart! When you type in a command and start typing arguments, hitting tab will help you with autocompleting! In stead of typing `cd cea-infamous`, you can type in `cd c`, hit tab, and voilà, you will likely have the repository name completed. !!! tip "Most Commonly Used Unix Commands" ??? abstract "cd" Change Directory. Examples: ``` cd ./cea-infamous # go to the cea-infamous directory under your current location cd ./cea-infamous/projects-cea # go to the projects-cea subdirectory in cea-infamous cd .. # go one level up (parent directory of current location) cd ../cea-infamous # go to the cea-infamous directory under your parent directory cd ../.. # go two levels up from you current location cd ~ # go to the home directory cd ~/WORKSPACE/cea-infamous # go to the infamous directory ``` ??? abstract "ls" List contents. Examples: ``` ls # list files and folders in your current location ls -a # list files and folders (including hidden ones) in your current location ls -la # list files and folders (including hidden ones), and their metadata in your current location ls -lat # list files and folders (including hidden ones), and their metadata in your current location, sorted by time last modified ls ~/WORKSPACE/cea-infamous # list the direct subfolders and files under cea-infamous ``` ??? abstract "touch" Create file Examples: ``` touch .env # Create the dot env file in your current location ``` ??? abstract "mkdir" Create directory Examples: ``` mkdir ./toto # Create a directory called toto in your current location ``` ??? abstract "rm" Remove a Directory or File. Examples: ``` rm ./foo.txt # remove the foo.txt file found in your current location rm -r ./unwanted_folder # remove the unwanted_folder and it's contents (-r is for recursive) ``` ??? abstract "cp" Copy a Directory or File. Examples: ``` cp ./foo.txt foo_copy.txt # make a copy of the file foo.txt, name the copy foo_copy.txt cp -R ./my_folder my_folder_copy # copy the folder and it's contents (-R is for recursive) ``` ??? abstract "mv" Move/Rename a Directory or File. Examples: ``` mv ./foo.txt ./subdir # move the file foo.txt into a subdirectory called subdir mv ./foo.txt bar.txt # rename the file foo.txt as bar.txt ``` ??? abstract "cat" Displays the contents of a file Examples: ``` cat _data/my_project/processed/output.csv ``` ??? abstract "head" Displays the beginning of a file. Default is 10 lines. Examples: ``` head _data/my_project/processed/output.csv head -50 _data/my_project/processed/output.csv # Displays the first 50 lines ``` ??? abstract "tail" Displays the end of a file. Default is 10 last lines. Examples: ``` tail _logs/my_project/cron.log tail -20 _logs/my_project/cron.log # Displays the last 20 lines of the log ``` ??? abstract "source" Execute a file in the current shell Examples: ``` source _envs/my-env/bin/activate # Activates the virtual environment "my-env" source run.sh # Executes run.sh found in the current location in the current shell ``` ??? abstract "chmod" Modify the permissions of a file/directory Examples: ``` chmod +x ./run.sh # Make the file run.sh executable ``` ??? abstract "vim" Edit a file in vim Examples: ``` vim .env # Edit the dotenv file with vim ```