# orchardLabelSearch and accountsxPP Query Comparison Script This directory contains a script for comparing the results of different account query methods: - `orchardLabelSearch` and `orchardLabels` with `accountsxPP`. _Claude Code has been used for both the data preparation and script development 🤖_ ## Purpose This script compares the results returned by these queries to identify differences for [PP-1260](https://theorchard.atlassian.net/browse/PP-1260), in preparation for replacing the query used by Bulk Uploads Account Search. Performance comparison will be covered separately in [PP-1397](https://theorchard.atlassian.net/browse/PP-1397). ## Data Used ### Sources To compare these queries, I've compiled a list of actual and potentially common search terms from the following sources (see the `data/` folder): - **`DD_orchardLabelSearch_terms.md`** - Results from [this DataDog query](https://sonymusic-pde.datadoghq.com/apm/traces?query=%40graphql.variables.term%3A%2A%20resource_name%3A%28%22query%20orchardLabelsSearch__graphql_knowledge__0%28%24first%3AInt%2C%24labelType%3ALabelType%2C%24offset%3AInt%2C%24term%3AString%29%7BorchardLabelSearch%28first%3A%24first%20labelType%3A%24labelType%20offset%3A%24offset%20term%3A%24term%29%7B__typename%20id%7B...a%7D...on%20Subaccount%7B__typename%20subaccountId%20uuid%20vendor%7B__typename%20id%7B...a%7DvendorId%7DvendorId%7D...on%20Vendor%7B__typename%20subaccountId%20type%20vendorId%7D%7D%7Dfragment%20a%20on%20LabelId%7B__typename%20subaccountId%20vendorId%7D%22%20OR%20%22query%20OrchardLabelSearch__graphql_knowledge__0%28%24limit%3AInt%3D0%2C%24term%3AString%21%29%7BorchardLabelSearch%28limit%3A%24limit%2Cterm%3A%24term%29%7B__typename%20id%7B...a%7D...on%20Subaccount%7B__typename%20uuid%20vendor%7B__typename%20id%7B...a%7DvendorId%7D%7D...on%20Vendor%7B__typename%20type%20vendorId%7D%7D%7Dfragment%20a%20on%20LabelId%7B__typename%20subaccountId%20vendorId%7D%22%20OR%20%22query%20OrchardLabelSearch__graphql_knowledge__0%28%24labelType%3ALabelType%3DVENDOR%2C%24limit%3AInt%3D0%2C%24term%3AString%21%29%7BorchardLabelSearch%28labelType%3A%24labelType%20limit%3A%24limit%20term%3A%24term%20vendorStatuses%3A%5B%5D%29%7B__typename%20id%7B__typename%20subaccountId%20vendorId%7Duuid...a...on%20Subaccount%7B__typename%20uuid%20vendor%7B...a%7D%7D%7D%7Dfragment%20a%20on%20Vendor%7B__typename%20vendorId%7D%22%20OR%20%22query%20OrchardLabelSearch__graphql_knowledge__0%28%24labelType%3ALabelType%3DVENDOR%2C%24limit%3AInt%3D0%2C%24term%3AString%21%29%7BorchardLabelSearch%28labelType%3A%24labelType%20limit%3A%24limit%20term%3A%24term%20vendorStatuses%3A%5B%5D%29%7B__typename%20id%7B__typename%20subaccountId%20vendorId%7Duuid...on%20Subaccount%7B__typename%20uuid%20vendor%7B__typename%20vendorId%7D%7D...on%20Vendor%7B__typename%20vendorId%7D%7D%7D%22%20OR%20%22query%20OrchardLabelSearch__graphql_knowledge__0%28%24term%3AString%21%29%7BorchardLabelSearch%28labelType%3AVENDOR%2Climit%3A0%2Cterm%3A%24term%29%7B__typename%20id%7B__typename%20subaccountId%20vendorId%7Duuid...on%20Subaccount%7B__typename%20uuid%7D...on%20Vendor%7B__typename%20vendorId%7D%7D%7D%22%29%20env%3Aprod&agg_m=count&agg_m_source=base&agg_q=%40graphql.variables.term&agg_q_source=base&agg_t=count&cols=core_service%2Ccore_resource_name%2Clog_duration%2Clog_http.method%2Clog_http.status_code&fromUser=false&graphType=flamegraph&historicalData=true&messageDisplay=inline&query_translation_version=v0&shouldShowLegend=true&sort=desc&spanType=all&storage=hot&top_n=1000&top_o=top&trace_group_by_from=a&traceQuery=&view=spans&viz=query_table&x_missing=true&start=1768608000000&end=1771326138352&paused=true) showing the most common search terms for `orchardLabelSearch` over the past month. Note that this query is used in multiple places, not just Bulk Uploads Account Search. Data has been sanitized to remove any one- or two-character inputs. - **`DD_rum_account_clicks_totals.csv`** - Results from [this DataDog query](https://sonymusic-pde.datadoghq.com/rum/sessions?query=%40type%3Aaction%20service%3Acontent%20env%3Aprod%20%40view.name%3A%2Fcreate%2Fdigital-audio%2Faccount-selection&agg_m=count&agg_m_source=base&agg_t=count&cols=&fromUser=true&sort_by=time&sort_order=desc&viz=stream&from_ts=1768694400000&to_ts=1771420240893&live=false) showing clicks on accounts in the Bulk Uploads Account Selection page. Data has been sanitized to remove non-account clicks. The most-clicked account is `HANDS OFF TEST ACCOUNT (Jacob Fowler) • 25824`, likely from product demos. - **`insights_top_accounts_by_streams.csv`** - Top accounts on Insights by streams over the last 28 days. While not the most accurate proxy (streams ≠ products), it provides another data point. ### Search Terms From these sources, 103 unique search terms were extracted (see `search_terms_to_test.py`): - **51 terms** from `DD_orchardLabelSearch_terms.md` (appeared 3+ times) - **14 accounts** from `DD_rum_account_clicks_totals.csv` (clicked 2+ times), with account names and numbers separated to reflect how users typically search (based on patterns in `DD_orchardLabelSearch_terms.md`). This includes demo accounts as an additional comparison point. - **7 accounts** from `insights_top_accounts_by_streams.csv` (top by streams) - **First words** extracted from each account name to reflect common search patterns (based on `DD_orchardLabelSearch_terms.md`) Total: **103 unique terms** (107 before removing duplicates) ## Setup 1. **Install Poetry** (if not already installed): ```bash curl -sSL https://install.python-poetry.org | python3 - ``` 2. **Install dependencies:** ```bash poetry install ``` 3. **Set up environment variables:** ```bash cp env.shadow .env ``` 4. **Fill in the required values** in `.env`: - `ORCHARD_PROFILE_TYPE` - `ORCHARD_PROFILE_ID` - `ORCHARD_IDENTITY_ID` - `APOLLOGRAPHQL_CLIENT_NAME` - `ORCHARD_PROFILE_UUID` - `AUTHORIZATION` (JWT token - keep this secure!) - `ORCHARD_ROLES` ## Usage ### First Run (Query + Compare) Run the script to query both endpoints and compare results: ```bash poetry run python main.py ``` This will: 1. Query accounts with no search term (using `orchardLabels` and `accountsxPP` with empty string) 2. Query both `orchardLabelSearch` and `accountsxPP` for all 103 search terms 3. Save raw query results to `results/query.json` 4. Compare the results and save to `results/comparison.json` 5. Print a summary to the console **Alternative:** Activate the Poetry shell first: ```bash poetry shell python main.py ``` ### Re-running Comparison Only To re-run just the comparison logic without re-querying (useful for development): 1. Edit `main.py` and set `SKIP_QUERIES = True` 2. Run: `poetry run python main.py` Note: This requires existing results in `results/query.json` from a previous run. ### Configuration Options In `main.py`, you can configure: - **`SKIP_QUERIES`** - Set to `True` to skip querying and use existing `results/query.json` - **`LIMIT_TERMS`** - Set to a number to limit total queries/results, or `None` for all ## Output Files ### `results/query.json` Raw query responses for each search term. The first entry queries accounts with no search term: ```json [ { "term": "", "orchardLabels": { /* GraphQL response */ }, "accountsxPP": { /* GraphQL response */ } }, { "term": "rimas", "orchardLabelSearch": { /* GraphQL response */ }, "accountsxPP": { /* GraphQL response */ } }, ... ] ``` ### `results/comparison.json` Comparison results for each search term: ```json [ { "searchTerm": "", "firstResultSame": false, "matchingAccountsCount": 0, "orchardLabelsResultsCount": 10, "accountsxPPResultsCount": 10 }, { "searchTerm": "rimas", "firstResultSame": true, "matchingAccountsCount": 5, "orchardLabelSearchResultsCount": 10, "accountsxPPResultsCount": 8 }, ... ] ``` **Fields:** - **`firstResultSame`** - `true` if the first result from both queries has the same `vendorId` and `name` - **`matchingAccountsCount`** - Number of accounts that appear in both result sets (matching on `vendorId` and `name`) - **`orchardLabelsResultsCount`** - Total results from `orchardLabels` (only present for no-term query) - **`orchardLabelSearchResultsCount`** - Total results from `orchardLabelSearch` (only present for search term queries) - **`accountsxPPResultsCount`** - Total results returned by `accountsxPP` ## Understanding the Results The script outputs a summary showing: - **First result matches** - How many search terms have the same first result (most important for UX) - **Perfect matches** - Search terms where all results are identical in both queries - **Matching thresholds** - How many search terms have 3+, 5+, or 7+ matching results ## Analysis See **`ANALYSIS.md`** for a comprehensive analysis of the comparison results. This document analyzes whether `accountsxPP` can provide enough consistency to replace the orchard queries, with particular focus on: - Match quality and overlap between queries - Exact match precision (where orchard returns 1 result, does xPP include it?) - Patterns in search behavior (number searches vs text searches) - User experience implications ### Regenerating the Analysis After running new queries or updating the comparison results, you can ask Claude to regenerate the analysis: ``` Look at the results/query.json and results/comparison.json files and update ANALYSIS.md with your analysis. Focus on whether accountsxPP can provide enough consistency compared to the orchard queries. Look for patterns like: how much overlap there is, cases where orchard returns 1 exact match but xPP returns many (and whether the exact match is included), and any other patterns that affect migration viability. ``` The analysis is based on Claude's interpretation of the data and may highlight different patterns or insights each time based on what stands out in the results.