"""Logic for getting metadata about playlists.""" import collections import hashlib from itertools import chain import json import re from flask import g from oto import response from analytics import config from analytics.connectors import redis from analytics.consts import error from analytics.models import amazon_music from analytics.models import apple_music_kit from analytics.models import spotify from analytics.utils import parallel store_handlers = { 'amazon_music': { 'rule': re.compile( r'https?://music\.amazon\.com/.*|' r'nohref://amazon_music/.*'), 'handler': amazon_music.get_playlists_metadata }, 'apple_playlist': { 'rule': re.compile( r'https?://itunes\.apple\.com/(?P[a-z]{2})/' r'(?Pplaylist)/id(?P.*)'), 'handler': apple_music_kit.get_playlists_metadata }, 'apple_station': { 'rule': re.compile( r'https?://itunes\.apple\.com/(?P[a-z]{2})/' r'(?Pstation)/id(?P.*)'), 'handler': apple_music_kit.get_playlists_metadata }, 'spotify': { 'rule': re.compile( r'https?://open\.spotify\.com/' r'(user/[^/]+/)?' # pre-GDPR format compatibility r'playlist/(?P.*)' ), 'handler': spotify.get_playlists_metadata }, 'spotify_personalized_playlist': { 'rule': re.compile( r'spotify-personalized-playlist:' r'(?P.*)' ), 'handler': spotify.get_personalized_playlists_metadata } } def _get_playlist_metadata_cache_key(playlist_url): """Get cache key for playlist. Args: playlist_url (str): Playlist url. Returns: str: Redis cache key. """ return 'playlistmetadata:{}'.format( hashlib.md5(playlist_url.lower().encode()).hexdigest()) def _xstr(s): return '' if s is None else str(s) def _cache_playlists_metadata(metadata_dict): """Cache metadata using redis. Respects config.PLAYLIST_METADATA_CACHE_CODES Args: metadata_dict (dict of str: dict): Metadata for each playlist. """ pipe = redis.client.pipeline() for playlist_url, playlist_metadata in metadata_dict.items(): if ( playlist_metadata.get('status_code') in config.PLAYLIST_METADATA_CACHE_CODES ): pipe.setex( name=_get_playlist_metadata_cache_key( playlist_url + _xstr( _match_playlist_with_store(playlist_url)[0] ) ), time=config.PLAYLIST_METADATA_CACHE_TTL, value=json.dumps(playlist_metadata)) pipe.execute() def _load_json_bytes_or_none(data): """Decode JSON value from bytes data if data is not None. Args: data (bytes|None): Encoded data dict. Returns: dict: Decoded data. """ if data is not None: return json.loads(data.decode()) def _get_cached_playlist_metadata(playlist_urls): """Get cached metadata for listed playlists. Args: playlist_urls (list[str]): Playlist urls. Returns: dict: Cached metadata for each playlist (None if not found). """ encoded_data_list = redis.client.mget([ _get_playlist_metadata_cache_key( playlist_url + _xstr( _match_playlist_with_store(playlist_url)[0] ) ) for playlist_url in playlist_urls ]) return { playlist_url: _load_json_bytes_or_none(encoded_data) for playlist_url, encoded_data in zip(playlist_urls, encoded_data_list)} def obtain_playlist_metadata(playlist_links): """Get metadata for each playlist. Args: playlist_links (list): list of playlists links. Returns: Response: list of dict of playlist_link and related metadata. """ spotify_links = [link for link in playlist_links if 'spotify' in link] apple_music_links = [link for link in playlist_links if 'apple' in link] if len(spotify_links) + len(apple_music_links) != len(playlist_links): return response.Response( status=400, message='Some links have unsupported format.') spotify_response = spotify.obtain_playlist_metadata(spotify_links) if not spotify_response: return spotify_response apple_music_response = apple_music_kit.obtain_playlist_metadata( apple_music_links) if not apple_music_response: return apple_music_response return response.Response( message=spotify_response.message + apple_music_response.message) def _match_playlist_with_store(playlist_link): """Try to match playlist link with Store link pattern. Args: playlist_link (str): Store playlist URL. Returns: tuple(str, dict): If playlist link matched with any store then returns store name and dict with math details, otherwise tuple of None values. """ for store in store_handlers: match = store_handlers[store]['rule'].match(playlist_link) if match: data = {'playlist_link': playlist_link} data.update(match.groupdict()) return store, data return None, None def _split_playlist_links_by_stores(playlist_links): """Split playlists by stores. Args: playlist_links (list[str]): List of playlist URLs. Returns: dict: Key is store name, value is list of store playlist links. """ result = collections.defaultdict(list) for playlist_link in playlist_links: store, data = _match_playlist_with_store(playlist_link) if data: result[store].append(data) return result def _prepare_result(results): """Transform results from all stores. Transform responses from all stores to the dictionary where key is the playlist link and the value is playlist metadata. Args: results: List of oto.Response. Return: dict: Playlists metadata. """ return { playlist['playlist_link']: playlist for playlist in chain(*[result.message for result in results])} def _playlist_links_valid(playlists_links, matched_playlists): """Check if all passed playlist_links have valid format. If number of the matched_playlists is not equal to number of original links then it means some of them have invalid format. Args: playlists_links (list): List of playlists from request. matched_playlists (dict): Only playlists that match to allowed URL patterns. Return: bool: True if number of matched_playlists is equal to number of original links, otherwise False. """ return len(playlists_links) == sum( len(links) for links in matched_playlists.values()) def get_playlists_metadata(playlist_links, feed_ids, distributors): """Get metadata for each playlist. Check cache first. Args: playlist_links (list[str]): list of playlists links. feed_ids (list): List of available feed_ids. distributors (list): List of distributors names. Returns: Response: list of dict of playlist_link and related metadata. """ metadata_dict = _get_cached_playlist_metadata(playlist_links) missed_links = [link for link, data in metadata_dict.items() if not data] if missed_links: fetched_data = fetch_playlists_metadata( missed_links, feed_ids, distributors) if fetched_data.errors: return fetched_data _cache_playlists_metadata(fetched_data.message) metadata_dict.update(fetched_data.message) metadata_list = list(metadata_dict.values()) return response.Response(metadata_list) def fetch_playlists_metadata(playlist_links, feed_ids, distributors): """Get metadata for each playlist from remote sources. Args: playlist_links (list): list of playlists links. feed_ids (list): list of available feeds. distributors (list): list of distributors names. Returns: Response: dict[str] playlist_links and related metadata. """ store_playlists = _split_playlist_links_by_stores(playlist_links) if not _playlist_links_valid(playlist_links, store_playlists): return response.Response( errors=error.INVALID_LINK_FORMAT_MESAGE) requests = {} for store, links in store_playlists.items(): requests.update({ store: { 'func': store_handlers[store]['handler'], 'args': (links,), 'kwargs': { 'logger': g.log, 'feed_ids': feed_ids, 'distributors': distributors} } }) results = parallel.execute_in_parallel(requests).values() items = _prepare_result(results) return response.Response(items)