In this part or the repo you'll find some automated scripts to help with your development workflow.  

!!! warning "Pre-requisites"
    The **.env file** should exists and the following variables should be defined:   
    - **Mandatory**: ```INFAMOUSDIR```, ```ENVDIR```  
    - Optional if you want to use data and logs directories: ```DATADIR```, ```LOGDIR```   
    - Optional if you want to work with the api-server: ```APIDIR```, ```pem_path```, ```api_user```, ```api_suuser```, ```bastion_server```, ```api_server```  

???+ warning "Windows users"
    If your laptop run under windows, you will need a solution to run sh scripts.   
    The simplest one we found is by using git-bash.   
    > If you don't have git-bash installed on your laptop already, you can get it from the Software Center by searching for Git.   

    Then you can simply use git-bash as your terminal to run the sh scripts.   
    or if you already use the windows terminal, run ``` & 'C:\Program Files\Git\git-bash.exe'``` which will open a git-bash terminal

???+ note
    All examples below are written to run from the cea-infamous directory, but you can run the scripts from anywhere.

### initialize/deploy a new project
This script has two typical use cases:

1. You can **initialize** a new project directory with the ```init_new_project.sh``` script. In the vast majority of cases (actually all we can think of right now), this should be done locally on your computer 
2. You can also use the script to set up the log and data directories and the python virtual environment when you are **deploying** a project on the api server.

???+ success "Syntax"    
    ```
    ./tools/init_new_project.sh project_name team
    ```

#### initializing a new project

???+ example "Examples"
    To start a completely new project:
    ```
    ./tools/init_new_project.sh awesome-project cea
    ```   
    This will create a project ```awesome-project-cea``` in the ```projects-cea``` directory
    ```
    ./tools/init_new_project.sh shook-ones fr
    ```   
    This will create a project ```shook-ones-fr``` in a ```projects-fr``` directory    

???+ warning "Project teams and branches"
    Projects with team other than CEA (such as ```shook-ones-fr``` above) will not be merged to main. These projects are only to be merged to your country-specific branches

!!! info "Execution"
    During the execution you will be prompted to figure if you want to :   

    - Create a log directory for the project (this will be created under the path specified in LOGDIR in .env)   
    - Create a data directory for the project (this will be created under the path specified in DATADIR in .env)   
    - Create a python virtual environment for the project
!!! abstract "What you'll get"  
    The project will be initialized with the following structure:   
    ![image](./img/projectdir.png)

    + a ```README.md``` file: to document your project   
    + a ```requirements.txt``` file: the requirements file to store the needed python packages (initialized with the latest version of djagitit)
    + a ```notebooks``` folder: to store your notebooks 
    + a ```sh``` folder: to store you bash scripts
    + a ```src``` folder: to store you source .py files   
    + a ```sql``` folder: to store your .sql files + a _create subfolder to store the sql scriots to create your tables/views   
    ```

#### deploying a project on the api server

???+ example "Examples"
    To deploy a project on the api server:
    ```
    [ca-europesys@busd-api01 cea-infamous]$ ./tools/init_new_project.sh awesome-project cea
    ```   
    The script will look for ```awesome-project-cea``` in ```projects-cea```. If the project is found, you will be prompted to confirm whether you are trying to deploy the project on the api server:
    ```
    Are you deploying the project and want to set up the external directories? (Y/n): Y
    ``` 
    A blank answer (simply hitting enter), y or yes (case insensitive) will be interpreted as a yes.

!!! info "Execution"
    Just like when using the script to initialize a new project, you will be prompted individually to confirm the creation of each directory, as well as the python virtual environment. The path to these directories will be read from the server-side ```.env``` file.

???+ warning "Existing directories"
    If any of the directories or the python virtual environment already exists on the api server, they will not be created/overwritten, but rather skipped.

!!! abstract "What you'll get"  
    After running the srcipt for deployment purposes, you will end up with the original project directory, as well as all the required external directories needed to successfully deploy the script on the api server.

### create a virtual environment
You can create a new project directory with the ```create_env.sh``` script.  

???+ success "Syntax"
    ```
    ./tools/create_env.sh project_name
    ```
???+ example "Examples"
    ```
    ./tools/create_env.sh awesome-project
    ```   
    This will create a virtual environment ```awesome-project-env``` in the ```ENVDIR``` directory specified in the ```.env``` file
!!! info "Execution"
    During the execution you will be prompted to figure if you want to :   

    - Use one of the requirements files if any was found (corresponding to your project_name) to install the needed packages.
    - Or install the latest version of the djagitit package.
    - Or don't install any package


### log on the api-server
You can log onto the api server with the ```logon_apiserver.sh``` script.  

???+ success "Syntax"
    ```
    ./tools/logon_apiserver.sh
    ```
!!! warning "Once connected"
    You're logged in as your personal user (```api_user```) not the super user.


### check your crontab for issues 
You can make sure that all the scripts in your crontab do exist and are executable with the ```check_crontab_permissions.sh``` script.

???+ success "Syntax"
    ```
    ./tools/check_crontab_permissions.sh
    ```
!!! info "Info"
    
    The script will parse your crontab and look for all shell scripts on uncommented lines. For each shell script, the script will check and print out whether:

    1. the file exists
    2. the file is executable

    If any non-executable files are files are found, you will be prompted on whether you want to change the permissions (which you probably will).

???+ example "Examples"
    ```sh
    [ca-europesys@busd-api01 cea-infamous]$ ./tools/check_crontab_permissions.sh
    /home/ca-europesys/WORKSPACE/cea-infamous/projects-cea/give-up-the-goods-cea/sh/run.sh                        is executable.
    /home/ca-europesys/WORKSPACE/cea-infamous/projects-cea/survival-of-the-fittest-cea/sh/run.sh    is found but not executable.
    Do you want to change permissions of the non-executable files? (yes/no): yes
    Changed permissions of /home/ca-europesys/WORKSPACE/cea-infamous/projects-cea/survival-of-the-fittest-cea/sh/run.sh to executable.
    ```

!!! warning "Crontab permissions"
    Make sure your superuser is allowed to access crontab. If not, you need to contact cloudops (Shastry).


### run a script on the api server

In stead of scheduling, you can run scripts on a one-off -basis on the api server with the ```run_on_apiserver.sh``` script. The scripts will continue running even after closing the connection, so this is handy when you want something to continue running after your working hours, possibly even overnight.

???+ success "Syntax"
    ```sh
    ./tools/run_on_apiserver.sh remote_path [log_subdir] [env_name] [-- script_args...]
    ```

!!! info "Info"
    
    You can run python or shell scripts with this tool.

    **Arguments:**

    - `remote_path`: Path to the script on the api server, relative to `INFAMOUSDIR`.
    - `log_subdir` (optional): Subdirectory for logs. Default is `oneoff`.
    - `env_name` (optional): Environment name for Python scripts. Default is `oneoff-env`.
    - `-- script_args...` (optional): Command line arguments passed to the script.

???+ example "Examples"
    ```sh
    ./tools/run_on_apiserver.sh path/to/script.py
    ./tools/run_on_apiserver.sh path/to/script.py myfancylog
    ./tools/run_on_apiserver.sh path/to/script.py myfancylog myfancy-env -- arg1 arg2
    ./tools/run_on_apiserver.sh path/to/script.sh
    ./tools/run_on_apiserver.sh path/to/script.sh myfancylog
    ```

!!! warning "Environment and Log Directory"
    Ensure your `.env` file is properly set up and accessible both locally and on the api server. Make sure the file you wish to run exists on the api server. (If not, use ```upload_to_apiserver.sh```) Also make sure that the necessary log directories and environments exist on the server, and contain all the subdirectories and libraries required by the script.


### download api-server files
You can download a directory from the api-server with the ```download_from_apiserver.sh``` script.
???+ success "Syntax"
    ```
    ./tools/download_from_apiserver.sh path_to_download
    ```
???+ example "Examples"
    With ```api_suuser``` set as ```francesys``` and ```APIDIR``` set as ```${INFAMOUSDIR}/_apiserver``` in the .env file
    ```
    ./tools/download_from_apiserver.sh LOGS/cradle-to-the-grave
    ```
    Will result in downloading from the api-server every file located in ```/home/francesys/LOGS/cradle-to-the-grave```   
    To the following time-stamped directory on my local machine : ```./cea-infamous/_apiserver/LOGS/cradle-to-the-grave/20240109-103726```


### upload files to api-server
You can upload a local file/directory to the api-server with the ```upload_to_apiserver.sh``` script.
???+ success "Syntax"
    ```
    ./tools/upload_to_apiserver.sh local_path server_path
    ```
???+ example "Examples"
    With ```api_suuser``` set as ```francesys```
    ```
    ./tools/upload_to_apiserver.sh ./up-north-trip whateveryouwant
    ```
    Will result in uploading all files in ```./up-north-trip```
    To the api-server in ```/home/francesys/whateveryouwant```   


### release new versions
Once your repository is installed on the api-server, you can get the latest version of any branch/version of your repo by running the ```release_on_apiserver.sh``` script.
!!! warning
    **This script can only be used on the api-server**
???+ success "Syntax"
    ```
    ./tools/release_on_apiserver.sh branch_name
    ```
    OR
    ```
    ./tools/release_on_apiserver.sh version
    ```
???+ example "Examples"
    ```
    ./tools/release_on_apiserver.sh v1.1.0
    ```
    ```
    ./tools/release_on_apiserver.sh main
    ```