# go-apollo ## Builds Build artifacts for services: - `appledbingest` - `appleconsumeranalytics` - `spotifydbingest` - `spotifyconsumeranalytics` For listed platforms: - `darwin` - `linux` Results will be placed into `./build` folder Build all at once: make all You can also build each binary independently. E.g.: make appledbingest ## Debug All services listed above are instrumented with `pprof` package. Host can be customized with a `-host` parameter. --- # pkg/appleconsumeranalytics Does in-memory ISRC level aggregations for Apple Music reports from specified source S3 bucket. The aggregation results are stored in specified target S3 bucket. Expects the source bucket to have Delphi SLZ's directory structure. Database operations are done through a `leveldbgrpc` GRPC client. Run aggregations: ``` go run cmd/appleconsumeranalytics/appleconsumeranalytics.go \ -startdate=2020-04-02 \ -enddate=2020-04-02 \ -report= \ -licensor=smej,smejintl,sony,theorchard \ -source-region=us-east-1 \ -source-bucket=dev-sme-data-archive \ -target-region=us-east-1 \ -target-bucket=dev-sme-aggregated-stream-reports ``` Load data into database: ``` go run cmd/appledbingest/appledbingest.go \ -startdate=2020-04-01 \ -enddate=2020-04-02 \ -db-address=127.0.0.1:50020 \ -region=us-east-1 \ -bucket=dev-sme-aggregated-stream-reports \ -licensor=smej,smejintl,sony,theorchard ``` --- # pkg/leveldbgrpc Go client for `js/leveldbgrpc`. Used by `appleconsumeranalytics` and `spotifyconsumeranalytics` to communicate with LevelDB through GRPC. --- # pkg/spotifyconsumeranalytics Does in-memory ISRC level aggregations for Spotify reports from specified source S3 bucket. The aggregation results are stored in specified target S3 bucket. Expects the source bucket to have Delphi SLZ's directory structure. Database operations are done through a `leveldbgrpc` GRPC client. Run aggregations: ``` go run cmd/spotifyconsumeranalytics/spotifyconsumeranalytics.go \ -startdate=2020-04-01 \ -enddate=2020-04-02 \ -report= \ -licensor=smej,smejintl,sony,theorchard \ -countrycode= \ -source-region=us-east-1 \ -source-bucket=dev-sme-data-archive \ -target-region=us-east-1 \ -target-bucket=dev-sme-aggregated-stream-reports ``` Load data into database: ``` go run cmd/spotifydbingest/spotifydbingest.go \ -startdate=2020-04-01 \ -enddate=2020-04-02 \ -licensor=smej,smejintl,sony,theorchard \ -db-address=127.0.0.1:50020 \ -region=us-east-1 \ -bucket=dev-sme-aggregated-stream-reports ``` --- # js/leveldbgrpc GRPC server with an embedded LevelDB instance with Get/Put/Delete/Iterator services. LevelDB database operations can be done through a GRPC client. ### Setup: Use Node.js version 8.15.1 Install dependencies: ``` (cd js/leveldbgrpc && yarn) ``` ### Run: ``` node js/leveldbgrpc/server.js \ --dir=./test.db \ --address=127.0.0.1:50020 ``` --- # js/consumeranalytics Node.js server that exposes data from a `leveldbgrpc` server through a REST API. Used by Apollo for track streams and playlist track streams for Spotify and track streams, track demographics, and playlist track streams for Apple Music. ### Setup Use Node.js version >= 8.15.1 Install dependencies: ``` (cd js/consumeranalytics && yarn) ``` ### Run ``` node js/consumeranalytics/server.js --port=8000 ``` --- # js/cleanup-leveldb Deletes records written to the database for specific dsps, dates, and licensors. In the event that we need to reprocess data from Spotify/Apple Music this process can be used. The aggregation result of the `spotifyconsumeranalytics`/`appleconsumeranalytics` processes writes files into S3 buckets and then the `spotifydbingest`/`appledbingest` process uses those files to load data into the database. This process scans through all aggregated files in the specified S3 bucket and figures out which records have been written by going through each line in the files. It then deletes these records from the database. Specify a `dsp`, `startdate`, `enddate`, list of `licensor`s, LevelDB GRPC `db-address`, `s3-bucket` and `s3-region`. ### Setup: Use Node.js version >= 8.15.1 Install dependencies: ``` (cd js/cleanup-leveldb && yarn) ``` ### Run: For Spotify: ``` node js/cleanup-leveldb/src/main.js \ --dsp=spotify \ --startdate=2020-03-29 \ --enddate=2020-03-29 \ --db-address=127.0.0.1:50020 \ --s3-bucket=dev-sme-aggregated-stream-reports \ --s3-region=us-east-1 \ --licensor=smej,smejintl,sony,theorchard ``` For Apple: ``` node js/cleanup-leveldb/src/main.js \ --dsp=apple \ --startdate=2020-03-29 \ --enddate=2020-03-29 \ --db-address=127.0.0.1:50020 \ --s3-bucket=dev-sme-aggregated-stream-reports \ --s3-region=us-east-1 \ --licensor=smej,sony,theorchard ``` --- ### Manual testing Use Node.js version >= 8.15.1, Go version >= 1.14.1 Install dependencies: ``` (cd js/cleanup-leveldb && yarn) && (cd js/leveldbgrpc && yarn) ``` Start a Node.js LevelDB GRPC server: ``` node js/leveldbgrpc/server.js \ --dir=./test.db \ --address=127.0.0.1:50020 ``` Load in data for a specific day (use licensor=smej because the dataset is small): For Spotify: ``` go run cmd/spotifydbingest/spotifydbingest.go \ -startdate=2020-03-29 \ -enddate=2020-03-29 \ -licensor=smej \ -db-address=127.0.0.1:50020 \ -region=us-east-1 \ -bucket=dev-sme-aggregated-stream-reports ``` For Apple: ``` go run cmd/appledbingest/appledbingest.go \ -startdate=2020-03-29 \ -enddate=2020-03-29 \ -db-address=127.0.0.1:50020 \ -region=us-east-1 \ -bucket=dev-sme-aggregated-stream-reports \ -licensor=smej ``` Check number of records in the database: ``` NODE_NO_WARNINGS=1 node -e '"use strict";let GRPC = require("./js/cleanup-leveldb/node_modules/grpc");let proto = GRPC.load("js/leveldbgrpc/src/leveldb.proto");let db = new proto.LevelDB("127.0.0.1:50020", GRPC.credentials.createInsecure());let count = 0;db.iteratorKeys({limit: -1}).on("data", () => {count++;}).on("end", () => console.log("records:", count));' ``` Run cleanup job: For Spotify: ``` node js/cleanup-leveldb/src/main.js \ --dsp=spotify \ --startdate=2020-03-29 \ --enddate=2020-03-29 \ --db-address=127.0.0.1:50020 \ --s3-bucket=dev-sme-aggregated-stream-reports \ --s3-region=us-east-1 \ --licensor=smej ``` For Apple: ``` node js/cleanup-leveldb/src/main.js \ --dsp=apple \ --startdate=2020-03-29 \ --enddate=2020-03-29 \ --db-address=127.0.0.1:50020 \ --s3-bucket=dev-sme-aggregated-stream-reports \ --s3-region=us-east-1 \ --licensor=smej ``` Check number of records in the database and ensure it prints `records: 0`. ``` NODE_NO_WARNINGS=1 node -e '"use strict";let GRPC = require("./js/cleanup-leveldb/node_modules/grpc");let proto = GRPC.load("js/leveldbgrpc/src/leveldb.proto");let db = new proto.LevelDB("127.0.0.1:50020", GRPC.credentials.createInsecure());let count = 0;db.iteratorKeys({limit: -1}).on("data", () => {count++;}).on("end", () => console.log("records:", count));' ``` --- # Reprocessing In case of a need to reprocess data, here is what needs to be done. ### Step 1 Spotify: - Stop and disable `spotifyconsumeranalytics` and `spotifydbingest` systemd services on the EC2 instance. - Stop and disable .NET task `SpotifyPlaylistAnalyticsImport` and `SpotifyGeneralAnalyticsImport` on the EC2 instance (prod-apollo-utility-streams-01). Apple: - Stop and disable systemd services `appleconsumeranalytics` and `appledbingest` on the EC2 instance. - Stop and disable .NET task `UpdateAppleMusicAnalytic-ContainerStreams` on the EC2 instance (prod-apollo-utility-streams-01). ### Step 2 Run the `js/cleanup-leveldb` script for the specific day to remove data from the LevelDB database. ### Step 3 Cleanup data in MySQL. Spotify Playlist Streams: ```sql DELETE FROM dbSony_dbo.tblSpotifyPlaylistStream WHERE Date = '2020-04-09'; DELETE FROM dbSony_dbo.tblSpotifyAnalyticsAccountPlaylistCategoryStreamSummary WHERE Date = '2020-04-09'; DELETE FROM dbSony_dbo.tblSpotifyAnalyticsDate WHERE Date = '2020-04-09' AND FileType = 2; DELETE FROM dbSony_dbo.tblSpotifyAnalyticsDatePlaylistProcessed WHERE Date = '2020-04-09'; ``` Spotify Track Demographics: ```sql DELETE FROM dbSony_dbo.tblSpotifyTrackStreamDemographics WHERE Date = '2020-04-09'; ``` Apple Music Playlist Streams: ```sql DELETE FROM dbSony_dbo.tblAppleMusicAnalyticsDate WHERE Date = '2020-04-09' AND FileType = 2; DELETE FROM dbSony_dbo.tblAppleMusicContainerStreams WHERE Date = '2020-04-09'; DELETE FROM dbSony_dbo.tblAppleMusicContainerStreamSummary WHERE LatestDate = '2020-04-09' ``` ### Step 4 Delete aggregation result files from the `sme-aggregated-streams` bucket for the specific day. ### Step 5 Make sure Delphi SLZ has ingested new files for the specific day. ### Step 6 Spotify: - Enable `spotifyconsumeranalytics` and `spotifydbingest` systemd services on the EC2 instance. - Enable .NET task `SpotifyPlaylistAnalyticsImport` and `SpotifyGeneralAnalyticsImport` on the EC2 instance (prod-apollo-utility-streams-01). Apple: - Enable `appleconsumeranalytics` and `appledbingest` systemd services on the EC2 instance. - Enable .NET task `UpdateAppleMusicAnalytic-ContainerStreams` on the EC2 instance (prod-apollo-utility-streams-01). The aggregation job (`spotifyconsumeranalytics` or `appleconsumeranalytics`) will notice that the aggregation result bucket `sme-aggregated-streams` is missing data. It will start running aggregations from Delphi SLZ for the missing data. The database load job (`spotifydbingest` or `appledbingest`) will notice that there is data in the aggregation result bucket `sme-aggregated-streams` and will load that into the database. .NET jobs will notice there is data available in `sme-aggregated-streams-reports` and will start loading.