# Development Workflow Instructions for Claude

## Overview
This repo is for individual contributors to create their own folders and work on POCs, experimentation, etc. This folder (`mymac80`) is the workspace for development activities.

## Standard Development Workflow

### 1. Starting a New Initiative
When beginning a new project or proof of concept:

1. **Rebase with upstream**: Always start by rebasing the local master branch with upstream to ensure we have the latest changes
   ```bash
   git checkout master
   git fetch upstream
   git rebase upstream/master
   ```

2. **Create project branch**: After getting the initiative name, create a new project-specific branch
   ```bash
   git checkout -b <initiative-name>
   ```

### 2. Development Practices
- **Atomic commits**: Use atomic commits as much as possible - each commit should represent a single logical change
- **Meaningful commit messages**: Write clear, descriptive commit messages that explain the "why" of the change

### 3. Pull Request Process
1. **Create PR**: When ready, create a Pull Request against upstream's master branch
2. **Review on GitHub**: Review the PR thoroughly on GitHub to ensure quality
3. **Squash and merge**: If everything looks good, use squash and merge to maintain clean history

### 4. Post-Merge Cleanup
After the PR is merged:
1. **Switch back to master**: `git checkout master`
2. **Pull upstream changes**: Rebase master with upstream to get the merged changes
   ```bash
   git fetch upstream
   git rebase upstream/master
   ```

## Security Guidelines
**CRITICAL**: Never commit sensitive information to the repository, including:
- Client IDs and client secrets
- API keys and tokens
- Database credentials and connection strings
- Private keys and certificates
- Passwords and authentication secrets
- Environment-specific configuration with sensitive data

**Best Practices**:
- Use environment variables or `.env` files for sensitive configuration (ensure `.env` files are in `.gitignore`)
- Use placeholder values in example configurations
- Review all code changes before committing to ensure no secrets are included
- When in doubt, always err on the side of caution and exclude potentially sensitive information

## Key Principles
- Always start with an up-to-date master branch
- Use atomic commits for better history and easier debugging
- Review PRs carefully before merging
- Maintain a clean, linear history through rebasing
- Keep the development cycle tight: branch → develop → PR → review → merge → cleanup

## Claude Workflow Integration
When Claude is asked to start a new initiative:
1. Claude should ask for the initiative name
2. Execute the rebase workflow with upstream
3. Create the appropriately named project branch
4. Proceed with development following atomic commit practices

## Streamlit Container Runtime on Snowflake

### Overview
Streamlit Container Runtime is a Snowflake private preview feature that enables running Streamlit applications directly within Snowflake's environment. This provides seamless access to Snowflake data without external connections.

### Key Architecture Components

#### 1. Infrastructure Setup
- **Compute Pool**: Required for Container Runtime execution (e.g., `CPU_X64_XS` instance family)
- **External Access Integration**: Controls network access to external APIs and package repositories
- **Network Rules**: Define specific endpoints allowed for external communication
- **Stage**: File storage location for application code and assets

#### 2. Database Configuration
```sql
-- Essential setup pattern
CREATE COMPUTE POOL <pool_name>
    MIN_NODES = 1
    MAX_NODES = 3
    INSTANCE_FAMILY = CPU_X64_XS;

CREATE EXTERNAL ACCESS INTEGRATION <integration_name>
    ALLOWED_NETWORK_RULES = (<network_rules>)
    ENABLED = TRUE;

CREATE STREAMLIT <app_name>
    ROOT_LOCATION = '@<stage_name>'
    MAIN_FILE = '<main_file>.py'
    QUERY_WAREHOUSE = <warehouse_name>
    EXTERNAL_ACCESS_INTEGRATIONS = (<integration_name>);
```

### Development Best Practices

#### 1. Code Structure and Patterns
**Snowpark Integration**:
```python
from snowflake.snowpark.context import get_active_session

@st.cache_resource
def get_snowpark_session():
    return get_active_session()

@st.cache_data(ttl=300)
def query_data():
    session = get_snowpark_session()
    return session.sql("SELECT * FROM table").to_pandas()
```

**Key Pattern**: Always use `session.sql().to_pandas()` or `session.sql().collect()` for data retrieval, NOT `st.connection("snowflake").query()`.

#### 2. Compatibility Considerations
- **Streamlit Version**: Container Runtime may use older Streamlit versions
- **Working Features**: `st.rerun()`, session state, UI components all work correctly
- **Package Dependencies**: Standard data science packages (pandas, numpy) are available
- **External Packages**: Require proper network rules and external access integration

#### 3. Security and Network Access
**Network Rules Best Practices**:
- Be specific with endpoints - avoid wildcards like `*:443` or `*:80`
- Use exact domain and port combinations
- Example for PyPI access:
```sql
CREATE NETWORK RULE pypi_rule
    MODE = EGRESS
    TYPE = HOST_PORT
    VALUE_LIST = (
        'pypi.org:443',
        'files.pythonhosted.org:443'
    );
```

#### 4. File Management and Deployment
**Correct Upload Pattern for Container Runtime**:
```sql
-- Upload all files to stage (CRITICAL: Use AUTO_COMPRESS=FALSE)
PUT file:///path/to/app.py @stage_name AUTO_COMPRESS=FALSE;
PUT file:///path/to/utils.py @stage_name AUTO_COMPRESS=FALSE;

-- Create or update Streamlit app (CRITICAL: Use FROM syntax with trailing slash)
CREATE OR REPLACE STREAMLIT app_name
FROM '@schema_name.stage_name/'
    MAIN_FILE = 'app.py'
    QUERY_WAREHOUSE = warehouse_name
    COMPUTE_POOL = pool_name
    EXTERNAL_ACCESS_INTEGRATIONS = (integration_name);
```

**Key Requirements for Container Runtime**:
- `AUTO_COMPRESS=FALSE` - File compression breaks Container Runtime apps
- `FROM '@schema.stage/'` - Use FROM syntax, not ROOT_LOCATION
- `CREATE OR REPLACE` - Use REPLACE for updates
- Stage path must end with `/`
- `COMPUTE_POOL` is required for Container Runtime apps
- **Multiple files supported** - Upload all Python modules to same stage

### Troubleshooting Common Issues

#### 1. Debugging with Logs and Traces
**View Application Logs** (using Snowflake's default event table):
```sql
-- Check recent application logs
SELECT timestamp, record:severity_text::STRING as severity_text, 
       record:body::STRING as body, record:logger_name::STRING as logger_name
FROM SNOWFLAKE.TELEMETRY.EVENTS
WHERE record:logger_name::STRING LIKE 'streamlit%'
ORDER BY timestamp DESC
LIMIT 50;

-- Filter by error severity
SELECT timestamp, record:body::STRING as body, record:logger_name::STRING as logger_name
FROM SNOWFLAKE.TELEMETRY.EVENTS
WHERE record:severity_text::STRING = 'ERROR'
  AND timestamp >= CURRENT_TIMESTAMP - INTERVAL '1 hour'
ORDER BY timestamp DESC;
```

**Analyze Trace Events**:
```sql
-- View performance traces
SELECT timestamp, record:name::STRING as event_name, record:attributes as attributes
FROM SNOWFLAKE.TELEMETRY.EVENTS
WHERE record:name::STRING = 'function_completed'
ORDER BY timestamp DESC
LIMIT 20;

-- Check execution times
SELECT 
    record:name::STRING as event_name,
    AVG(record:attributes:execution_time::FLOAT) as avg_execution_time,
    MAX(record:attributes:execution_time::FLOAT) as max_execution_time
FROM SNOWFLAKE.TELEMETRY.EVENTS
WHERE record:name::STRING = 'function_completed'
GROUP BY record:name::STRING;
```

**Reference**: [Event Table Setup Documentation](https://docs.snowflake.com/en/developer-guide/logging-tracing/event-table-setting-up)

#### 2. Application Errors
**Symptom**: "Python Interpreter Error: TypeError: bad argument type for built-in operation"
**Solution**: Usually caused by over-engineered compatibility fixes. Revert to simpler, original code.

**Symptom**: Module import errors
**Solution**: Verify package availability or add to external access rules for installation.

#### 3. Performance Monitoring and Optimization
**Monitor Query Performance**:
```python
import streamlit as st
import logging
import time
from snowflake import telemetry

logger = logging.getLogger("performance_monitor")

@st.cache_data(ttl=300)
def monitored_query(sql_query):
    session = get_snowpark_session()
    start_time = time.time()
    
    try:
        # Execute query with performance tracking
        telemetry.set_span_attribute("query_type", "data_fetch")
        result = session.sql(sql_query).to_pandas()
        
        execution_time = time.time() - start_time
        
        # Log performance metrics
        logger.info(f"Query executed successfully in {execution_time:.2f}s")
        telemetry.add_event(
            "query_completed",
            {
                "execution_time": execution_time,
                "row_count": len(result),
                "query_hash": hash(sql_query)
            }
        )
        
        return result
        
    except Exception as e:
        logger.error(f"Query failed: {str(e)}")
        telemetry.add_event("query_failed", {"error": str(e)})
        raise
```

#### 4. Database Connectivity
**Working Pattern**:
```python
# ✅ Correct - Works in Container Runtime
session = get_active_session()
data = session.sql("SELECT * FROM table").to_pandas()

# ❌ Incorrect - May not work
data = st.connection("snowflake").query("SELECT * FROM table")
```

#### 5. Performance Optimization
- Use `@st.cache_data` for expensive queries with appropriate TTL
- Use `@st.cache_resource` for session management
- Limit query results with `LIMIT` clauses for testing
- Consider data freshness requirements when setting cache TTL
- Monitor performance through telemetry and logging

### Phase 0 Validation Approach
Before building full applications, validate Container Runtime capabilities with progressive tests:

1. **Test 1**: Basic UI components and Streamlit functionality ✅
2. **Test 2**: Database connectivity and Snowpark integration ✅
3. **Test 3**: External API connectivity (if needed) ✅
4. **Test 4**: Package management and dependencies ✅
5. **Test 5**: Multi-file application structure ✅ (Updated: Works with correct deployment)
6. **Test 6-7**: Logging/tracing validation (Skip until GA)
7. **Test 8**: Simple module imports ✅ **WORKS**

### Lessons Learned

#### What Works Well
- Standard Streamlit components and session state
- Snowpark integration with `get_active_session()`
- Direct SQL execution with `.sql().to_pandas()`
- File uploads with `AUTO_COMPRESS=FALSE` parameter
- Basic caching with `@st.cache_data` and `@st.cache_resource`
- `CREATE OR REPLACE STREAMLIT` with `FROM '@stage/'` syntax
- **Module imports work** - Can use modular Python architecture with multiple files

#### What Requires Attention  
- Network access must be explicitly configured
- Package installations need proper external access rules
- Performance tuning for large datasets
- Cache management for frequently updated data

#### Common Pitfalls
- Over-complicating simple applications with unnecessary fixes
- Using `st.connection()` instead of Snowpark session
- Broad network access rules (security risk)
- Not testing incremental complexity (jumping to complex apps)
- Assuming latest Streamlit features without validation
- **Using file compression** - Always use `AUTO_COMPRESS=FALSE` for uploads
- **Wrong CREATE syntax** - Use `FROM '@stage/'` not `ROOT_LOCATION`
- **Missing COMPUTE_POOL** - Required for all Container Runtime apps

### Development Workflow
1. **Phase 0**: Validate Container Runtime capabilities with simple tests
2. **Infrastructure**: Set up compute pool, network rules, and external access
3. **Iterative Development**: Start simple, add complexity gradually
4. **Testing**: Test each component (UI, database, external APIs) separately
5. **Optimization**: Add caching, performance tuning after functionality works
6. **Security Review**: Ensure network rules follow least-privilege principle

### Known Limitations (Private Preview)

#### Logging and Tracing Functionality
**Status**: Not fully functional in private preview  
**Issue**: Event table logging and telemetry features are not reliably working  
**Recommendation**: Skip logging/tracing functionality until Container Runtime goes GA  
**Workaround**: Use basic Streamlit debugging (st.write, st.error) for development  
**Note**: Once GA, refer to logging patterns in this document for implementation