Feature Implementation Plan: API Cache Warming

Overview

Implement event-driven cache warming that automatically re-populates the cache for popular API endpoints after cache invalidation. This ensures the frontend almost always receives cached responses, even immediately after data changes.

Dependencies: - api-request-logging.md (must be implemented first - provides pattern data) - Existing CacheInvalidationSubscriber infrastructure

Related Plans: - api-request-admin-ui.md (uses same pattern data)

Todo Checklist

• [ ] Add relatedTags field to ApiRequestPattern entity
• [ ] Update logging to capture cache tags for each request
• [ ] Create WarmCacheMessage message class
• [ ] Create WarmCacheMessageHandler handler
• [ ] Create CacheWarmingService for core warming logic
• [ ] Configure new Messenger transport for cache warming
• [ ] Integrate with CacheInvalidationSubscriber
• [ ] Add freshness tracking (lastWarmed field)
• [ ] Implement throttling and lock-based deduplication
• [ ] Write tests

Analysis & Investigation

Existing Infrastructure

Cache Invalidation Flow:

Entity Change (Doctrine event)
    ↓
CacheInvalidationSubscriber::invalidate()
    ↓
├── $apiCache->invalidateTags($tags)  ← Backend cache cleared
└── $frontendRevalidation->revalidate($tags)  ← Frontend notified

Tag Mapping: - CacheInvalidationSubscriber maps entities to cache tags (e.g., CoffeeBean → coffee_beans_list) - FrontendRevalidationService maps backend tags to frontend tags - Tags are granular: coffee_beans_list, roasters_detail, filter_metadata, etc.

Async Processing: - Messenger with Redis transport - Multiple transports for different workloads - Retry strategies configured per transport

Architecture Decisions

Decision	Choice	Rationale
Timing	Async via Messenger	Don't block Doctrine transactions
Scope	Tag-specific patterns only	Don't warm unrelated endpoints
Request method	Call service layer directly	Avoid HTTP overhead
Throttling	Lock + rate limit	Prevent overwhelming during bulk updates
Freshness	Track lastWarmed timestamp	Skip recently warmed patterns

Flow Diagram

┌─────────────────────────────────────────────────────────────────┐
│  Cache Warming Flow                                             │
├─────────────────────────────────────────────────────────────────┤
│                                                                 │
│  Entity Update (e.g., CoffeeBean saved)                         │
│       │                                                         │
│       ▼                                                         │
│  ┌────────────────────────────┐                                 │
│  │ CacheInvalidationSubscriber│                                 │
│  │   - Invalidate backend tags│                                 │
│  │   - Notify frontend        │                                 │
│  │   - Dispatch WarmCacheMsg  │ ← NEW                           │
│  └────────────────────────────┘                                 │
│       │                                                         │
│       ▼ (async via Messenger)                                   │
│  ┌────────────────────────────┐                                 │
│  │ WarmCacheMessageHandler    │                                 │
│  │   - Acquire lock           │                                 │
│  │   - Query top patterns     │                                 │
│  │   - Filter by tags         │                                 │
│  │   - Check freshness        │                                 │
│  └────────────────────────────┘                                 │
│       │                                                         │
│       ▼                                                         │
│  ┌────────────────────────────┐                                 │
│  │ CacheWarmingService        │                                 │
│  │   - Reconstruct Request    │                                 │
│  │   - Call service layer     │                                 │
│  │   - Update lastWarmed      │                                 │
│  └────────────────────────────┘                                 │
│       │                                                         │
│       ▼                                                         │
│  ┌────────────────────────────┐                                 │
│  │ Service Layer              │                                 │
│  │ (FilterAggregationService, │                                 │
│  │  CoffeeBeanRepository)     │                                 │
│  │   - Execute query          │                                 │
│  │   - Store in Redis cache   │ ← Cache now warm!               │
│  └────────────────────────────┘                                 │
│                                                                 │
└─────────────────────────────────────────────────────────────────┘

Implementation Plan

Prerequisites

• ApiRequestPattern entity exists (from api-request-logging.md)
• Logging pipeline is operational with pattern data
• Some traffic has been logged to have patterns to warm

Step 1: Add relatedTags and lastWarmed to ApiRequestPattern

File: src/Entity/ApiRequestPattern.php (modify)

#[ORM\Column(type: Types::JSON)]
private array $relatedTags = [];

#[ORM\Column(nullable: true)]
private ?DateTimeImmutable $lastWarmed = null;

public function getRelatedTags(): array
{
    return $this->relatedTags;
}

public function setRelatedTags(array $relatedTags): self
{
    $this->relatedTags = $relatedTags;
    return $this;
}

public function getLastWarmed(): ?DateTimeImmutable
{
    return $this->lastWarmed;
}

public function markAsWarmed(): self
{
    $this->lastWarmed = new DateTimeImmutable();
    return $this;
}

Generate migration:

make sf c="doctrine:migrations:diff"

Step 2: Create Tag Extractor Service

File: src/Service/Api/RequestTagExtractor.php

<?php

namespace App\Service\Api;

use Symfony\Component\HttpFoundation\Request;

/**
 * Extracts cache tags that would be relevant for a given API request.
 * Used to associate logged patterns with cache tags for warming.
 */
final class RequestTagExtractor
{
    private const PATH_TO_TAGS_MAP = [
        '/api/coffee-beans' => ['coffee_beans_list', 'coffee_beans'],
        '/api/roasters' => ['roasters_list', 'roasters'],
        '/api/filter-metadata' => ['filter_metadata'],
        '/api/varieties' => ['varieties_list'],
        '/api/regions' => ['regions_list'],
        '/api/countries' => ['countries_list'],
        '/api/processing-methods' => ['processing_methods_list'],
        '/api/roast-levels' => ['roast_levels_list'],
        '/api/flavor-wheel' => ['flavor_wheel'],
    ];

    private const PARAM_TO_TAGS_MAP = [
        'roaster' => ['roasters_detail'],
        'roasters' => ['roasters_detail'],
        'region' => ['regions_detail'],
        'regions' => ['regions_detail'],
        'variety' => ['varieties_detail'],
        'varieties' => ['varieties_detail'],
        'processingMethod' => ['processing_methods_detail'],
        'processingMethods' => ['processing_methods_detail'],
        'roastLevel' => ['roast_levels_detail'],
        'roastLevels' => ['roast_levels_detail'],
    ];

    /**
     * Extract cache tags relevant to this request.
     *
     * @return string[]
     */
    public function extractTags(Request $request): array
    {
        $path = $request->getPathInfo();
        $params = $request->query->all();

        $tags = [];

        // Add tags based on path
        foreach (self::PATH_TO_TAGS_MAP as $pathPrefix => $pathTags) {
            if (str_starts_with($path, $pathPrefix)) {
                $tags = array_merge($tags, $pathTags);
                break;
            }
        }

        // Add tags based on query params used
        foreach ($params as $key => $value) {
            $normalizedKey = rtrim($key, '[]');
            if (isset(self::PARAM_TO_TAGS_MAP[$normalizedKey]) && !empty($value)) {
                $tags = array_merge($tags, self::PARAM_TO_TAGS_MAP[$normalizedKey]);
            }
        }

        return array_values(array_unique($tags));
    }
}

Step 3: Update Logging Handler to Capture Tags

File: src/MessageHandler/LogApiRequestHandler.php (modify)

public function __construct(
    private ApiRequestPatternRepository $patternRepository,
    private EntityManagerInterface $entityManager,
    private RequestTagExtractor $tagExtractor,  // NEW
    private LoggerInterface $logger,
) {}

public function __invoke(LogApiRequestMessage $message): void
{
    // ... existing code ...

    // Extract tags for the pattern
    $request = Request::create($message->path, $message->method, $message->queryParams);
    $tags = $this->tagExtractor->extractTags($request);

    // Update pattern with tags (upsert should handle this)
    $patternId = $this->patternRepository->upsertPattern(
        $message->method,
        $message->path,
        $queryHash,
        $message->queryParams,
        $message->responseTimeMs,
        $tags,  // NEW parameter
    );

    // ... rest of code ...
}

Step 4: Update Repository Upsert to Handle Tags

File: src/Repository/ApiRequestPatternRepository.php (modify)

public function upsertPattern(
    string $method,
    string $path,
    string $queryHash,
    array $queryParams,
    float $responseTimeMs,
    array $relatedTags = [],  // NEW
): string {
    $conn = $this->getEntityManager()->getConnection();

    $sql = <<<'SQL'
        INSERT INTO api_request_pattern
            (id, method, path, query_hash, query_params, related_tags, hit_count, avg_response_time_ms, last_seen_at, created_at)
        VALUES
            (gen_random_uuid(), :method, :path, :hash, :params, :tags, 1, :responseTime, NOW(), NOW())
        ON CONFLICT (query_hash) DO UPDATE SET
            hit_count = api_request_pattern.hit_count + 1,
            avg_response_time_ms = (
                (api_request_pattern.avg_response_time_ms * api_request_pattern.hit_count) + :responseTime
            ) / (api_request_pattern.hit_count + 1),
            last_seen_at = NOW(),
            related_tags = CASE
                WHEN api_request_pattern.related_tags = '[]'::jsonb THEN :tags::jsonb
                ELSE api_request_pattern.related_tags
            END
        RETURNING id
        SQL;

    $result = $conn->executeQuery($sql, [
        'method' => $method,
        'path' => $path,
        'hash' => $queryHash,
        'params' => json_encode($queryParams),
        'tags' => json_encode($relatedTags),
        'responseTime' => $responseTimeMs,
    ]);

    return $result->fetchOne();
}

/**
 * Find patterns that need warming based on invalidated tags.
 *
 * @param string[] $tags Invalidated cache tags
 * @return ApiRequestPattern[]
 */
public function findPatternsToWarm(
    array $tags,
    int $limit = 10,
    int $minSecondsSinceLastWarm = 60,
): array {
    $conn = $this->getEntityManager()->getConnection();

    // Use native query for JSON array overlap check
    $sql = <<<'SQL'
        SELECT p.id
        FROM api_request_pattern p
        WHERE p.related_tags ?| :tags
          AND (p.last_warmed IS NULL OR p.last_warmed < :threshold)
        ORDER BY p.hit_count DESC
        LIMIT :limit
        SQL;

    $threshold = new \DateTimeImmutable("-{$minSecondsSinceLastWarm} seconds");

    $result = $conn->executeQuery($sql, [
        'tags' => '{' . implode(',', $tags) . '}',
        'threshold' => $threshold->format('Y-m-d H:i:s'),
        'limit' => $limit,
    ]);

    $ids = $result->fetchFirstColumn();

    if (empty($ids)) {
        return [];
    }

    return $this->createQueryBuilder('p')
        ->where('p.id IN (:ids)')
        ->setParameter('ids', $ids)
        ->getQuery()
        ->getResult();
}

Step 5: Create WarmCacheMessage

File: src/Message/WarmCacheMessage.php

<?php

namespace App\Message;

final readonly class WarmCacheMessage
{
    /**
     * @param string[] $invalidatedTags Cache tags that were invalidated
     */
    public function __construct(
        public array $invalidatedTags,
    ) {}
}

Step 6: Create CacheWarmingService

File: src/Service/Api/CacheWarmingService.php

<?php

namespace App\Service\Api;

use App\Entity\ApiRequestPattern;
use App\Repository\ApiRequestPatternRepository;
use Doctrine\ORM\EntityManagerInterface;
use Psr\Log\LoggerInterface;
use Symfony\Component\HttpFoundation\Request;
use Symfony\Component\Lock\LockFactory;

final class CacheWarmingService
{
    private const MAX_PATTERNS_PER_RUN = 10;
    private const MIN_SECONDS_SINCE_LAST_WARM = 60;
    private const DELAY_BETWEEN_PATTERNS_MS = 50;
    private const LOCK_TTL_SECONDS = 120;

    public function __construct(
        private readonly ApiRequestPatternRepository $patternRepository,
        private readonly EntityManagerInterface $entityManager,
        private readonly FilterAggregationService $filterAggregationService,
        private readonly CoffeeBeanListService $coffeeBeanListService,
        private readonly LockFactory $lockFactory,
        private readonly LoggerInterface $logger,
    ) {}

    /**
     * Warm cache for patterns matching the invalidated tags.
     *
     * @param string[] $invalidatedTags
     */
    public function warmForTags(array $invalidatedTags): void
    {
        if (empty($invalidatedTags)) {
            return;
        }

        // Acquire lock to prevent concurrent warming
        $lock = $this->lockFactory->createLock(
            'cache_warming',
            self::LOCK_TTL_SECONDS
        );

        if (!$lock->acquire()) {
            $this->logger->debug('Cache warming skipped - another job is running');
            return;
        }

        try {
            $patterns = $this->patternRepository->findPatternsToWarm(
                $invalidatedTags,
                self::MAX_PATTERNS_PER_RUN,
                self::MIN_SECONDS_SINCE_LAST_WARM,
            );

            if (empty($patterns)) {
                $this->logger->debug('No patterns to warm for tags', [
                    'tags' => $invalidatedTags,
                ]);
                return;
            }

            $warmed = 0;
            foreach ($patterns as $pattern) {
                if ($this->warmPattern($pattern)) {
                    $warmed++;
                }

                // Small delay to avoid overwhelming the system
                usleep(self::DELAY_BETWEEN_PATTERNS_MS * 1000);
            }

            $this->logger->info('Cache warming completed', [
                'tags' => $invalidatedTags,
                'patterns_warmed' => $warmed,
                'patterns_found' => count($patterns),
            ]);
        } finally {
            $lock->release();
        }
    }

    private function warmPattern(ApiRequestPattern $pattern): bool
    {
        try {
            // Reconstruct request from pattern
            $request = Request::create(
                $pattern->getPath(),
                $pattern->getMethod(),
                $pattern->getQueryParams(),
            );

            // Route to appropriate service based on path
            $this->executeForPath($pattern->getPath(), $request);

            // Mark as warmed
            $pattern->markAsWarmed();
            $this->entityManager->flush();

            return true;
        } catch (\Throwable $e) {
            $this->logger->warning('Failed to warm pattern', [
                'pattern_id' => $pattern->getId(),
                'path' => $pattern->getPath(),
                'error' => $e->getMessage(),
            ]);
            return false;
        }
    }

    private function executeForPath(string $path, Request $request): void
    {
        // Match path to service - this naturally populates cache
        // via the existing cache logic in these services
        match (true) {
            str_starts_with($path, '/api/coffee-beans') => $this->coffeeBeanListService->getList($request),
            str_starts_with($path, '/api/filter-metadata') => $this->filterAggregationService->computeFacets($request),
            // Add more endpoints as needed
            default => throw new \RuntimeException("Unknown path for warming: {$path}"),
        };
    }
}

Step 7: Create WarmCacheMessageHandler

File: src/MessageHandler/WarmCacheMessageHandler.php

<?php

namespace App\MessageHandler;

use App\Message\WarmCacheMessage;
use App\Service\Api\CacheWarmingService;
use Psr\Log\LoggerInterface;
use Symfony\Component\Messenger\Attribute\AsMessageHandler;

#[AsMessageHandler]
final readonly class WarmCacheMessageHandler
{
    public function __construct(
        private CacheWarmingService $cacheWarmingService,
        private LoggerInterface $logger,
    ) {}

    public function __invoke(WarmCacheMessage $message): void
    {
        $this->logger->debug('Processing cache warming message', [
            'tags' => $message->invalidatedTags,
        ]);

        $this->cacheWarmingService->warmForTags($message->invalidatedTags);
    }
}

Step 8: Configure Messenger Transport

File: config/packages/messenger.php (add to existing config)

// Add new transport for cache warming
'cache_warming' => [
    'dsn' => '%env(MESSENGER_TRANSPORT_DSN)%',
    'options' => [
        'queue_name' => 'cache_warming',
    ],
    'retry_strategy' => [
        'max_retries' => 1,  // Best-effort, don't retry much
        'delay' => 5000,
    ],
],

// Add routing
'routing' => [
    // ... existing routes ...
    WarmCacheMessage::class => 'cache_warming',
],

Step 9: Integrate with CacheInvalidationSubscriber

File: src/EventSubscriber/CacheInvalidationSubscriber.php (modify)

use App\Message\WarmCacheMessage;
use Symfony\Component\Messenger\MessageBusInterface;

final class CacheInvalidationSubscriber
{
    public function __construct(
        #[Autowire(service: 'api.cache')]
        private readonly TagAwareCacheInterface $apiCache,
        private readonly FrontendRevalidationService $frontendRevalidation,
        private readonly MessageBusInterface $messageBus,  // NEW
    ) {}

    private function invalidate(LifecycleEventArgs $args): void
    {
        // ... existing tag determination code ...

        if ($tagsToInvalidate === []) {
            return;
        }

        $this->apiCache->invalidateTags($tagsToInvalidate);
        $this->frontendRevalidation->revalidate($tagsToInvalidate);

        // NEW: Dispatch cache warming message
        $this->messageBus->dispatch(new WarmCacheMessage($tagsToInvalidate));
    }
}

Step 10: Create CoffeeBeanListService (if not exists)

If there's no dedicated service for coffee bean list queries, create one that wraps the repository with caching:

File: src/Service/Api/CoffeeBeanListService.php

<?php

namespace App\Service\Api;

use App\Repository\CoffeeBeanRepository;
use Symfony\Component\DependencyInjection\Attribute\Autowire;
use Symfony\Component\HttpFoundation\Request;
use Symfony\Contracts\Cache\ItemInterface;
use Symfony\Contracts\Cache\TagAwareCacheInterface;

final readonly class CoffeeBeanListService
{
    public function __construct(
        private CoffeeBeanRepository $repository,
        private CacheKeyGenerator $cacheKeyGenerator,
        #[Autowire(service: 'api.cache')]
        private TagAwareCacheInterface $cache,
    ) {}

    public function getList(Request $request): array
    {
        $cacheKey = $this->cacheKeyGenerator->createKey($request, 'coffee_beans_');

        return $this->cache->get($cacheKey, function (ItemInterface $item) use ($request) {
            $item->tag(['coffee_beans_list', 'coffee_beans']);

            $page = max(1, $request->query->getInt('page', 1));
            $limit = min(100, max(1, $request->query->getInt('limit', 20)));

            return $this->repository->findByRequest($request, $page, $limit);
        });
    }
}

Testing Strategy

Unit Tests

1. RequestTagExtractor
2. Test path-to-tags mapping
3. Test param-to-tags mapping

4. Test combined extraction

5. CacheWarmingService

6. Test lock acquisition
7. Test pattern filtering by tags

8. Test freshness check (skip recently warmed)

9. WarmCacheMessageHandler

10. Test message processing

Integration Tests

1. Full warming flow:
2. Create pattern with specific tags
3. Dispatch WarmCacheMessage with matching tags

4. Verify cache is populated

5. Freshness skip test:

6. Warm a pattern
7. Immediately try to warm again

8. Verify second warming is skipped

9. Lock test:

10. Start warming job
11. Try concurrent warming
12. Verify second job is skipped

Manual Testing

# Start cache warming worker
make sf c="messenger:consume cache_warming -vv"

# Trigger cache invalidation (e.g., update a coffee bean in admin)
# Watch worker logs for warming activity

# Verify cache is populated
make api path="/api/coffee-beans" # Should show X-Cache: HIT

Success Criteria

• [ ] Cache warming triggers automatically after invalidation
• [ ] Only patterns matching invalidated tags are warmed
• [ ] Recently warmed patterns are skipped (freshness check)
• [ ] Concurrent warming jobs are deduplicated (lock)
• [ ] System is not overwhelmed during bulk updates (rate limiting)
• [ ] Cache hit rate improves after implementation

Metrics to Monitor

Add logging/metrics for: - Patterns warmed per invalidation event - Time spent warming - Cache hit rate before/after - Patterns frequently warmed but rarely hit (false positives)

Configuration Options

Consider making these configurable via environment variables:

# .env
CACHE_WARMING_MAX_PATTERNS=10
CACHE_WARMING_MIN_SECONDS_SINCE_LAST=60
CACHE_WARMING_DELAY_MS=50

Future Enhancements (Out of Scope)

1. Priority-based warming - Warm highest-hit patterns first even if tags don't match
2. Scheduled warming - Proactive warming on a schedule, not just reactive
3. Warming analytics - Dashboard showing warming effectiveness
4. Per-endpoint configuration - Different warming strategies per endpoint
5. Warm on deploy - Trigger warming after deployment to pre-populate cache