# Tiktok daily_trends

The data source for theorchard is s3://prod-orcd-ftp/ftp/data_feed/tiktok/.
The data source for sme is s3://sme-ca-prod-partners/tiktok/in/sme/daily_trends/.

* The staging raw table name is staging_raw_tiktok

# SME backfill

The first available date is 2021-07-27. During that period of time the file 
format has been changed several times. The main columns are the same, but 
country_region, gender, searches, metasong_URI has been removed. The new columns 
lfv_creations, lfv_video_views, lfv_avg_watchtime has been appeared. 

Sometimes this columns appeared in the middle of period (like for 2022-09-26) 
and then disappeared again. 

For some days the file format for Tiktok and Douyin is different for same day.
For these cases one of the file has been changed (e.g. remove redundant 
columns). The original file has been renamed as filename_original.txt.

For example,
s3://cucumbers/TikTok/archives_sme/2022-10-10/
   * PLY3_D_CN_20221010_original.txt 
        with new columns, which appears after 2022-10-30  
   * PLY3_D_CN_20221010.txt 
        with file format for that time period 
   * PLY3_D_ROW_20221010.txt 
        with file format for that time period.
This file PLY3_D_CN_20221010_original.txt has not been loaded, but kept 
for history. It contains the same numbers as PLY3_D_CN_20221010.txt.