Multi-Market Implementation: Phase 8 - Remove CoffeeBean.available Field¶

Status: Planning (Awaiting Data Migration)¶

Dependencies¶

Requires completed: - ✅ Phase 1 (Entities) - New entities exist - ✅ Phase 2 (Migration) - CrawlUrl.available field added and data migrated - ✅ Phase 3 (Services) - Services use CrawlUrl.available - ✅ Phase 4 (Cache/Repository) - Repositories use CrawlUrl.available - ✅ Phase 5 (API Integration) - API uses market-based availability - ✅ Phase 6 (EasyAdmin) - Admin interfaces ready - ⏳ CRITICAL: All existing CrawlUrls must be migrated to have markets assigned (via their RoasterCrawlConfigs)

Blocks: - None (final phase)

📋 Todo Checklist¶

[ ] Verify all RoasterCrawlConfigs have markets assigned
[ ] Verify all code references to CoffeeBean.available removed
[ ] Create database migration to drop available column
[ ] Test migration up/down (reversibility)
[ ] Update entity definition
[ ] Update fixtures and test data
[ ] Run full test suite
[ ] Document breaking changes
[ ] Plan deployment strategy

🔍 Analysis & Investigation¶

Problem Statement¶

After full transition to the multi-market system, the global CoffeeBean.available field is obsolete: - Availability is now per-CrawlUrl, not per-bean - Market-based availability provides more granular control - Keeping the field causes confusion and data inconsistency

However, this field cannot be removed during transition period. It must stay until: 1. All existing CrawlUrls have been assigned to RoasterCrawlConfigs with markets 2. All code uses CrawlUrl.available instead 3. All availability checks use the market-based logic

Architecture Clarification¶

Based on implementation discussions (2025-01-24), the multi-market architecture follows these principles:

Availability vs Monetization (Union Logic)¶

RoasterCrawlConfig.shipsTo = "Can the user buy this?" (Availability) - Determines which countries can physically purchase from this source - Controls whether a bean is shown to visitors from specific countries

Market.countries = "Which affiliate link do we use?" (Monetization) - Determines which affiliate program to apply - Controls URL transformation for commission tracking

The Flow: 1. User from France visits 2. Check: Does any RCC ship to France? (RCC.shipsTo) - NO → Don't show bean - YES → Show bean, continue to step 3 3. Check: Do we have a Market that includes France? - YES → Use that Market's AffiliateProgram to transform URL - NO → Show original roaster URL (no affiliate tracking)

Key Insight: Availability is determined by RCC.shipsTo (union across all RCCs), while URL transformation is determined by Market.countries. This allows showing beans that users can buy even if we haven't set up monetization for that region yet.

Multiple Sources Per Roaster¶

The architecture supports multiple RoasterCrawlConfigs per roaster for different sales channels:

Example: Blue Bottle Coffee - RCC #1: "Blue Bottle Direct US" → Ships to [US, CA] → Market: North America - RCC #2: "Blue Bottle on Amazon DE" → Ships to [DE, AT, FR...] → Market: Amazon Europe

Same bean, different CrawlUrls, different shipping regions, different affiliate programs.

Current State (Before Cleanup)¶

CoffeeBean entity: - Has available boolean field - No longer used by services/API - Still exists in database schema - Data is stale (not updated)

System behavior: - Services use CrawlUrl.available and market logic - API returns market-based availability - Admin can still see old available field (confusing)

Target State (After Cleanup)¶

CoffeeBean entity: - available field removed from entity - Database column dropped - No legacy availability logic remains

System behavior: - All availability checks use CrawlUrl.available - Market-based logic is the only availability system - No confusion about which field to use

📝 Implementation Plan¶

Prerequisites¶

CRITICAL Prerequisites (verify before proceeding):

All active RoasterCrawlConfigs have markets (optional but recommended):

SELECT COUNT(*) FROM roaster_crawl_config WHERE market_id IS NULL AND active = true;
-- Ideally should be 0, but not strictly required
-- Configs without markets will show original URLs (no affiliate)

Note: The system supports RCCs without markets (legacy mode). They will work but won't apply affiliate transformations.

All code uses CrawlUrl.available:
Search codebase for ->getAvailable() on CoffeeBean
Search codebase for ->setAvailable() on CoffeeBean
Verify no results (except in EntityToDtoMapper fallback)
API integration tested:
All endpoints use market-based availability
Endpoint availability calculation uses union logic (shows bean if ANY RCC ships to country)
Affiliate transformation works when Market exists
Original URLs shown when no Market configured
Migration reversibility tested:
Can rollback to previous schema
Data migration can be reversed

Step-by-Step Implementation¶

Step 1: Verify Prerequisites¶

Script: bin/verify-multi-market-migration.sh

#!/bin/bash
# Verify system is ready for CoffeeBean.available removal

set -e

echo "Verifying multi-market migration prerequisites..."

# Check for active configs without markets
echo -n "Checking for RoasterCrawlConfigs without markets... "
COUNT=$(bin/console dbal:run-sql "SELECT COUNT(*) FROM roaster_crawl_config WHERE market_id IS NULL AND active = true" --quiet)
if [ "$COUNT" -eq "0" ]; then
    echo "✓ All active configs have markets"
else
    echo "✗ Found $COUNT active configs without markets"
    echo "Run: SELECT * FROM roaster_crawl_config WHERE market_id IS NULL AND active = true"
    exit 1
fi

# Check for code references to CoffeeBean.available
echo -n "Checking for code references to CoffeeBean->getAvailable()... "
REFS=$(grep -r "->getAvailable()" src/ --include="*.php" | grep -v "CrawlUrl" | grep -v "RoasterCrawlConfig" | wc -l)
if [ "$REFS" -eq "0" ]; then
    echo "✓ No references found"
else
    echo "✗ Found $REFS references"
    echo "Search for: ->getAvailable() in CoffeeBean context"
    exit 1
fi

echo -n "Checking for code references to CoffeeBean->setAvailable()... "
REFS=$(grep -r "->setAvailable()" src/ --include="*.php" | grep -v "CrawlUrl" | wc -l)
if [ "$REFS" -eq "0" ]; then
    echo "✓ No references found"
else
    echo "✗ Found $REFS references"
    exit 1
fi

# Verify API tests pass
echo "Running API tests..."
bin/phpunit tests/Controller/Api/ --stop-on-failure

echo ""
echo "✓ All prerequisites verified! Safe to proceed with cleanup."

Run this script before proceeding:

chmod +x bin/verify-multi-market-migration.sh
./bin/verify-multi-market-migration.sh

Step 2: Update CoffeeBean Entity¶

File: src/Entity/CoffeeBean.php (modify existing)

Remove the available field:

<?php

namespace App\Entity;

// Remove or comment out:
// #[ORM\Column(type: Types::BOOLEAN, options: ['default' => true])]
// private bool $available = true;

// Remove getters/setters:
// public function isAvailable(): bool { ... }
// public function setAvailable(bool $available): self { ... }

// Rest of entity remains unchanged

Important: Document this as a breaking change for any custom code or integrations.

Step 3: Create Database Migration¶

File: migrations/VersionXXX_remove_coffee_bean_available.php

<?php

declare(strict_types=1);

namespace DoctrineMigrations;

use Doctrine\DBAL\Schema\Schema;
use Doctrine\Migrations\AbstractMigration;

final class VersionXXX_remove_coffee_bean_available extends AbstractMigration
{
    public function getDescription(): string
    {
        return 'Remove CoffeeBean.available field (replaced by CrawlUrl.available with market-based availability)';
    }

    public function up(Schema $schema): void
    {
        // Drop the available column
        $this->addSql('ALTER TABLE coffee_bean DROP COLUMN available');

        $this->write('CoffeeBean.available field removed. System now uses CrawlUrl.available with market-based logic.');
    }

    public function down(Schema $schema): void
    {
        // Restore the available column
        $this->addSql('ALTER TABLE coffee_bean ADD COLUMN available BOOLEAN DEFAULT true NOT NULL');

        // Restore data from CrawlUrl.available
        // Set bean available = false if ANY of its CrawlUrls are unavailable
        $this->addSql('
            UPDATE coffee_bean cb
            SET available = false
            WHERE EXISTS (
                SELECT 1 FROM crawl_url cu
                WHERE cu.coffee_bean_id = cb.id
                AND cu.available = false
            )
        ');

        // Set bean available = true if ALL CrawlUrls are available (or no CrawlUrls)
        $this->addSql('
            UPDATE coffee_bean cb
            SET available = true
            WHERE NOT EXISTS (
                SELECT 1 FROM crawl_url cu
                WHERE cu.coffee_bean_id = cb.id
                AND cu.available = false
            )
        ');

        $this->write('CoffeeBean.available field restored from CrawlUrl.available data.');
    }

    public function isTransactional(): bool
    {
        return true;
    }
}

Notes: - Migration is reversible (down restores field and data) - Uses same logic as Phase 2 data migration - Transactional for safety

Step 4: Update Fixtures and Test Data¶

Files: src/DataFixtures/*.php

Remove any references to $coffeeBean->setAvailable():

// REMOVE:
$coffeeBean->setAvailable(true);

// Availability is now determined by CrawlUrl.available

Files: tests/Fixtures/*.php

Update test fixtures:

// Instead of:
$bean->setAvailable(false);

// Use:
$crawlUrl->setAvailable(false);

Step 5: Update Documentation¶

File: docs/multi-market-migration-complete.md

# Multi-Market Migration Complete

## Breaking Changes

### CoffeeBean.available Field Removed

**Date:** [Migration date]

**Change:** The global `CoffeeBean.available` boolean field has been removed.

**Replacement:** Availability is now determined per-market using `CrawlUrl.available`.

**Impact:**
- API responses: `available` field now calculated from CrawlUrl.available + market logic
- Database schema: `coffee_bean.available` column dropped
- Entity: `CoffeeBean::isAvailable()` method removed

**Migration Guide:**

If you have custom code that references `CoffeeBean.available`:

```php
// OLD (no longer works):
if ($coffeeBean->isAvailable()) {
    // ...
}

// NEW (use service):
$available = $buyingOptionService->isAvailableInAnyMarket($coffeeBean);
// or for specific country:
$available = $buyingOptionService->getUrlForVisitor($coffeeBean, $country) !== null;

// NEW (use repository):
$available = $coffeeBeanRepository->isAvailableForCountry($coffeeBean, $countryId);

API Behavior:

No changes to API endpoints or responses. The available field in DTOs continues to work, but is now calculated from CrawlUrl.available.

Database:

To rollback this change, run the down migration:

bin/console doctrine:migrations:migrate prev

This will restore the available column and populate it from CrawlUrl data.

#### Step 6: Run Migration

**Steps:**

1. **Create database backup:**
   ```bash
   make db-backup
   ```

2. **Run migration:**
   ```bash
   make migrate
   ```

3. **Verify schema:**
   ```bash
   bin/console doctrine:schema:validate
   ```

4. **Test API:**
   ```bash
   curl http://localhost/api/coffee-beans?available=true
   ```

5. **Run test suite:**
   ```bash
   make test
   ```

#### Step 7: Verify Cleanup Complete

**Verification checklist:**

- [ ] CoffeeBean entity has no `available` field
- [ ] Database schema has no `coffee_bean.available` column
- [ ] All tests pass
- [ ] API endpoints return correct availability
- [ ] No grep results for `CoffeeBean.*available` in src/
- [ ] EasyAdmin forms don't show available field for beans
- [ ] Documentation updated

**Command to verify no stale references:**

```bash
# Should return 0 results:
grep -r "coffeeBean.*available\|->setAvailable\|->getAvailable\|->isAvailable" src/ \
  --include="*.php" \
  | grep -v "CrawlUrl" \
  | grep -v "RoasterCrawlConfig"

Testing Strategy¶

Pre-Migration Testing¶

Prerequisite Verification:
Run verification script
Verify all configs have markets
Verify no code references
Backup:
Create database backup
Test restoration process

Migration Testing¶

Up Migration:
Run migration
Verify column dropped
Verify no errors
API Testing:
Test all endpoints
Verify availability filtering works
Test with different countries
Down Migration (Rollback):
Run migration down
Verify column restored
Verify data restored correctly
Run migration up again

Post-Migration Testing¶

Full Test Suite:
Unit tests
Integration tests
API tests
Repository tests
Manual Testing:
Test bean availability in admin
Test API with various countries
Test filtering by availability

Deployment Strategy¶

Staging Deployment¶

Deploy to staging:
Deploy code without running migration
Verify app works (field still exists)
Run verification script:
Ensure prerequisites met
Run migration:
Test up migration
Test rollback (down migration)
Test up migration again
Smoke test:
Test critical flows
Verify availability works

Production Deployment¶

Recommended approach: Blue-Green deployment

Prepare:
Database backup
Rollback plan ready
Deploy:
Deploy new code
Run migration
Verify immediately
Monitor:
Check error logs
Monitor API responses
Watch availability filtering
Rollback if needed:
Run migration down
Deploy previous code
Verify system stable

Downtime: Minimal (seconds for column drop)

Risk: Low (migration is reversible)

🎯 Success Criteria¶

CoffeeBean.available field removed from entity
Database column dropped
No code references to CoffeeBean.available
All tests pass
API behavior unchanged (availability still works)
Migration is reversible
Documentation updated
Deployment successful with no issues
No errors in production logs

Breaking Changes¶

Entity-level: - CoffeeBean::isAvailable() method removed - CoffeeBean::setAvailable() method removed - CoffeeBean::available property removed

Database: - coffee_bean.available column dropped

API: - No breaking changes (DTOs unchanged) - Availability calculation changed internally

Custom Code: Any code that directly accesses CoffeeBean.available will break and needs to be updated to use: - BuyingOptionService::isAvailableInAnyMarket() - CoffeeBeanRepository::isAvailableForCountry() - CrawlUrl::isAvailable() (for specific URLs)

Files to Modify:¶

src/Entity/CoffeeBean.php (remove available field)
src/DataFixtures/*.php (remove setAvailable() calls)
tests/Fixtures/*.php (update test fixtures)
docs/multi-market-migration-complete.md (document changes)

Files to Create:¶

migrations/VersionXXX_remove_coffee_bean_available.php
bin/verify-multi-market-migration.sh
docs/multi-market-migration-complete.md

Files to Verify (should NOT reference CoffeeBean.available):¶

src/Service/Api/EntityToDtoMapper.php
src/Repository/CoffeeBeanRepository.php
src/Controller/Api/CoffeeBeanController.php
All test files

Rollback Plan¶

If issues occur after deployment:

Immediate rollback:

# Rollback migration
bin/console doctrine:migrations:migrate prev --no-interaction

# Deploy previous code version
git checkout [previous-tag]
make deploy

Verify rollback:
Check coffee_bean.available column exists
Run test suite
Verify API works
Investigation:
Review error logs
Identify root cause
Fix issues before re-attempting

Timeline¶

Assuming Phases 1-7 are complete:

Prerequisite verification: 1 hour
Code updates: 1 hour
Migration creation: 30 minutes
Testing: 2 hours
Documentation: 1 hour
Staging deployment: 1 hour
Production deployment: 30 minutes

Total Estimated Time: 7 hours

Final Notes¶

This is the last phase of the multi-market implementation. After completion:

✅ Multi-market system fully operational ✅ Legacy fields removed ✅ System simplified and consistent ✅ Market-based availability is the standard

Current System Capabilities (as of 2025-01-24)¶

The implemented system supports: - ✅ Multiple RoasterCrawlConfigs per roaster (e.g., direct store + Amazon) - ✅ Different shipping regions per config - ✅ Optional markets per config (legacy mode without market works) - ✅ Affiliate URL transformation when market has active program - ✅ Fallback to original URL when no market configured - ✅ Priority-based URL selection when multiple configs serve same country - ✅ Union logic: Show bean if ANY RCC ships to visitor country - ✅ BuyingOption manual overrides per market

Post-cleanup tasks: - Monitor production for 1 week - Update developer documentation - Train admin users on market management - Migrate existing RoasterCrawlConfigs to have markets assigned - Plan future enhancements (additional providers, new markets, etc.)

Next Steps¶

After completing this phase: - Monitor production for any issues - Gather feedback from admins using the system - Plan future enhancements: - Additional affiliate providers - More granular market segmentation - Enhanced analytics on buying option performance - A/B testing different affiliate programs