# youtube-statistics

youtube-statistics is a flow lets us scrape artists data and metrics using YouTube Analytics API.
The flow provides us with the next metrics:
*  current **subscribers** count, and it's growth compared to the previous week 
* **videos** count
* current **likes** count,  and it's growth compared to the previous week
* current **dislikes** count,  and it's growth compared to the previous week
* current **views** count,  and it's growth compared to the previous week=stats.get('percentage_change_in_views'),
* current **comments** count,  and it's growth compared to the previous week=stats.get('percentage_change_in_views'),
* **likes** and **comments engagement**
* the latest artist's release

YouTube Analytics API default quotas amount is not enough to srape all the artists's data, so, on this purpose we use several Applications having it's own keys and quotas. ```helpers.py``` contains ```get_youtube_api_credentials()``` method getting all the Applications' API keys from AWS Secrets Manager.
When the flow starts it connects to YouTube API using the above mentioned keys with ```pyyoutube``` library. It's a side library so you need to install it separately.
3d party libraries are listed at ```Pipfile``` which lock is using at ```Dockerfile``` while installing it.

###Technical features

Comments, likes, dislikes and views are the total amount scrapped for every single video at artist's channel.
Comments are implemented at ```pyyoutube``` as linked lists that you could access to having ```nextPageToken```, so ```get_all_the_content_with_next_page_token(self, id_, page_token, items: list, type_='playlist')``` method is done recursive.
The method could be used for scrapping both comments and playlists units. Playlists are made in the same way.

For getting artist's channel ```pyyoutube``` need to be provided with their *channelId*, that could be got from channel link.

After all the data is scrapped, it is put at a dictionary with artist's name as a key.
And then, using the name, it is loaded to SnowFlake table.

As the initial table we could get the links from is *ARTISTS_TO_TRACK*, having all the links to Spotify, Instagram and SoundCloud also.

#Running the flow

To run the flow the repository contains ```Dockerfile```.
There we have OS installation (Centos8), location set up. 
Then we have *ARG* and *ENV* declarations. To provide them while building the ```Dockerfile``` **--build-arg {var_name}={var_value}** notation for is required.
Required environment variables* are:
* Environment
* SNOWFLAKE_ROLE
* SNOWFLAKE_WAREHOUSE
* SNOWFLAKE_PASSWORD
* SNOWFLAKE_USER
* SNOWFLAKE_ACCOUNT
* SNOWFLAKE_DATABASE
* SNOWFLAKE_SCHEMA

**Environment variable is needed for getting the correct keys. Locally it's set as **dev**, but on the prod it is to be set as **prod**.*
For local run at least 3 secrets need to be stored having *YOUTUBE_CLIENT_ID_{number}* naming and valid *AWS_ACCESS_KEY_ID*, *AWS_SECRET_ACCESS_KEY*, *AWS_SECURITY_TOKEN*, *AWS_REGION* at .env file connected to the run configuration (it lets to get secrets from Secrets Manager).

To build the Docker container the following command from the repository directory:
```docker build -f Dockerfile --build-arg SNOWFLAKE_ROLE=${SNOWFLAKE_ROLE} --build-arg SNOWFLAKE_WAREHOUSE=${SNOWFLAKE_WAREHOUSE} --build-arg SNOWFLAKE_PASSWORD=${SNOWFLAKE_PASSWORD} --build-arg SNOWFLAKE_USER=${SNOWFLAKE_USER} --build-arg SNOWFLAKE_ACCOUNT=${SNOWFLAKE_ACCOUNT} --build-arg SNOWFLAKE_DATABASE=${SNOWFLAKE_DATABASE} --build-arg SNOWFLAKE_SCHEMA=${SNOWFLAKE_SCHEMA} -t myimage .```
*${VAR_NAME}* need to be replaced with *required environment variables.
After this the container starts to build, and the last step written down is ```RUN python3 /youtube-statistics/youtube_with_pyyoutube.py``` that starts the flow.