Laravel queue optimization for bulk processing: the 80,000-folio pipeline, translated into jobs

Laravel queue optimization starts with an uncomfortable question: is your bulk process already a queue, even if you don't call it one? If a cron job picks up pending records every few minutes and processes them one by one, the answer is yes. Just without retries, without visibility and without control.

I know because I built a system like that. It managed a pool of 80,000 folios in production and verified receipts with OCR + LLM, with no manual review.

One caveat before going further: that system did not run on Laravel. It ran on WordPress, with custom PHP plugins and a cron job every 2 minutes. This article is how I would translate that same pipeline into Laravel queues, and what the change buys you. The code is illustrative.

The starting point: a cron job that was already a queue

The system is Panini Fan Zone FWC 2026. Each receipt went through Tesseract to extract its text, and then a LLaMA 3.3 70B model, served by Groq, validated it against the promotion rules.

The 2-minute cron job worked. But a design like that forces you to solve by hand things a queue solves out of the box:

• Overlapping runs. If one execution takes longer than the interval, the next one starts on top of it. You need a lock.
• Retries. If the LLM fails because of the network, the receipt stays pending until the next run, with no control over how many times it was attempted.
• Visibility. How many are in flight? Which ones failed, and why? Without a queue, you build that with your own logging.
• A single lane. Everything shares the same line: an urgent task waits behind thousands of verifications.

In the article on Python optimization I solved those problems with file locks and capped batches. Laravel offers the same guarantees as framework building blocks.

Laravel queue optimization: one job per receipt

The first decision is granularity. A job that processes 5,000 receipts is a cron job in disguise: if it fails at number 3,000, the retry repeats everything.

The right unit is one receipt. Each job is small, retries on its own, and its failure drags nobody else down.

class VerifyReceipt implements ShouldQueue, ShouldBeUnique
{
    use Queueable;

    public int $tries = 3;
    public array $backoff = [10, 60, 300]; // increasing waits: the provider recovers on its own
    public int $timeout = 90;              // always below the connection's retry_after

    public function __construct(public int $receiptId) {}

    public function uniqueId(): string { return (string) $this->receiptId; }
}

Every line answers one of the cron job's problems.

ShouldBeUnique prevents queuing the same receipt twice while the first one is still pending. If the dispatcher runs again too early, there is no duplicate and no double LLM charge.

$backoff as an array gives a different wait per attempt. A network error at 10 seconds usually clears up; one that persists gets spaced out so you don't hammer the provider.

$timeout stops a stuck OCR from holding a worker forever. Its relationship with retry_after deserves its own section, further down.

Passing the ID instead of the full model is also a decision. The job is serialized into the queue; an ID weighs nothing and forces a fresh read of the database state at execution time.

Protecting the LLM provider

The expensive, slow step is the model call. If you scale to 20 workers, you make 20 times more requests per minute, and the provider answers with rate-limit errors.

The fix is to declare the limit once and have every worker share it, on whatever server it runs:

// AppServiceProvider::boot(): one limit shared by every worker.
RateLimiter::for('llm', fn (object $job) => Limit::perMinute(60));

// In the job: protect the provider and avoid duplicate work.
public function middleware(): array
{
    return [
        (new WithoutOverlapping($this->receiptId))->expireAfter(180),
        new RateLimited('llm'),
    ];
}

The number is an example; yours comes from your contract with the provider. What matters is that it lives in a single place.

WithoutOverlapping is the second layer of protection. Even if something dispatches the same receipt through another path, two workers won't process it at once. expireAfter releases the lock if a worker dies mid-process.

A detail that bites: when RateLimited holds a job back, it releases it back onto the queue, and that counts as an attempt. With a tight limit, three attempts run out fast. Raise $tries or use retryUntil() to set a time window instead of an attempt count.

Dispatching in capped batches

The cron job doesn't disappear. It changes roles: it no longer processes, it only dispatches. And it does so in named batches, so you can track their progress.

// ⚠️ Production: back up the table before changing the status flow.
$ids = Receipt::where('status', 'pending')
    ->orderBy('id')->limit(5000)->pluck('id');   // capped run

Bus::batch($ids->map(fn ($id) => new VerifyReceipt($id)))
    ->name('verification-'.now()->format('YmdHi'))
    ->onQueue('verification')
    ->allowFailures()            // one bad receipt doesn't cancel the batch
    ->finally(fn (Batch $b) => Log::info('batch', ['failed' => $b->failedJobs]))
    ->dispatch();

To use batches, the job also needs the Batchable trait and the batches table Laravel creates with its migration.

allowFailures() is key in bulk processing. Without it, the first unreadable receipt cancels the rest of the batch. With it, failures are recorded and everything else keeps going.

The cap of 5,000 serves the same purpose as in the original cron job: every run has a known size. Whatever doesn't fit goes to the next one.

Best practices for bulk processing

The timeout always below retry_after

This is the setting that causes the most incidents. In config/queue.php, every connection has a retry_after: how long Laravel waits before treating an in-flight job as lost and handing it to another worker.

If a job can run for 90 seconds and retry_after is 60, at 60 seconds another worker picks up the same job. Now it runs twice in parallel, and nobody notices until duplicate results show up.

The rule is simple: retry_after must be greater than the longest timeout of any job on that connection, with margin. If one kind of job is much slower than the rest, it deserves its own connection.

Idempotency, even with uniqueness

ShouldBeUnique and WithoutOverlapping reduce duplicates. They don't eliminate them. A worker can die after saving the result and before acknowledging the job, and the queue will hand it out again.

That's why every job starts by checking the state: if the receipt is no longer pending, it exits without doing anything. And the result is saved in a single operation that leaves the same state even if it runs twice.

That discipline comes from the original system. In its registration module, a suite with 50 simultaneous workers hitting the same route confirmed that exactly one registration won each slot. Concurrency is proven, not assumed.

Separate queues by priority

A single queue is a single lane. If 5,000 verifications are queued and a confirmation email arrives, it waits behind all of them.

• urgent for what a user is waiting on: confirmations, notifications, responses to an action.
• verification for bulk processing, with its own worker limit.
• maintenance for reports and cleanups that can wait until the small hours.

A worker can serve several queues in priority order, but real separation comes from distinct worker pools. That's exactly what Horizon configures.

Failures are data

A job that runs out of attempts ends up in the failed jobs table, with the full exception. That table is a diagnostic board, not a trash can.

• Implement failed() on the job. Mark the receipt as "needs review" so it doesn't sit pending forever.
• Tell errors apart. An unreadable image is not fixed by retrying: fail immediately with fail(). A network timeout, yes.
• Retry with intent. queue:retry after fixing the cause, not as a reflex.
• Prune by policy. queue:prune-failed with a defined retention keeps the table from growing without bound.

Advanced Horizon configuration

Horizon is Laravel's queue supervisor for Redis. Its value isn't the pretty dashboard: it's that worker configuration lives in code, versioned and reviewable.

'production' => [
    'urgent' => [
        'connection' => 'redis', 'queue' => ['urgent'],
        'balance' => 'auto', 'minProcesses' => 1, 'maxProcesses' => 4,
    ],
    'verification' => [
        'connection' => 'redis', 'queue' => ['verification'],
        'balance' => 'auto', 'maxProcesses' => 6,
        'memory' => 256, 'tries' => 3, 'timeout' => 90,   // timeout < retry_after
    ],
],

Two supervisors, two pools. Bulk verification never takes workers away from urgent work, because each one has its own cap.

balance => 'auto' scales processes up and down based on each queue's load, within minProcesses and maxProcesses. In quiet hours it uses little; when a big batch lands, it grows up to the cap.

maxProcesses isn't chosen by available RAM. It's chosen by the LLM provider's limit: more workers than allowed requests only produce waits and retries.

memory restarts the worker when it exceeds that usage in megabytes. Long-lived PHP processes accumulate memory over time; a controlled restart beats a server out of RAM.

Three more settings worth reviewing:

• waits sets how many seconds of waiting on a queue trigger an alert. It's the early signal that you're short on workers.
• trim controls how long Horizon keeps recent and failed jobs. In bulk processing, keeping everything fills Redis.
• silenced hides noisy, trivial jobs from the dashboard, so what matters doesn't get lost.

And one deploy rule: after shipping new code, run php artisan horizon:terminate. Workers are long-lived processes and keep running the old code until they restart. The system supervisor brings them back up with the current version.

What NOT to do

• One giant job that processes thousands of records. A failure halfway repeats everything, the timeout becomes impossible to calibrate, and there's no visibility into progress. One record per job, grouped into batches.
• A timeout greater than retry_after. The same job runs twice in parallel and the problem looks random. Check that pair on every connection.
• Scaling workers to "go faster" against a rate-limited API. More processes don't increase allowed requests; they increase errors, retries and cost. The limit is declared with RateLimited and workers are tuned to it.
• Forgetting to restart workers on deploy. The new code is on disk, but the processes keep running the old one. "Fixed" bugs keep showing up.

The next step

Laravel queue optimization isn't about raising the worker count. It's choosing the right unit of work, protecting what's expensive, calibrating the timings, and separating the urgent from the bulk.

If your team is facing bottlenecks in bulk processing, queues that get stuck, or LLM integrations that never quite become reliable, at Conqr Digital Agency we design architectures that scale. Let's review your case.

The original system is documented in the Panini Fan Zone case study, and the Python version of the pipeline in Python optimization for 80,000 folios.

FAQ

What is the difference between timeout and retry_after in Laravel?

The timeout is how long a job may run before the worker kills it. The retry_after, on the queue connection, is how long Laravel waits before treating an in-flight job as lost and handing it out again. If retry_after is lower than the timeout, the same job can run twice in parallel.

When should I use ShouldBeUnique on a job?

When dispatching the same work twice would be a mistake or a waste: verifying the same receipt, charging the same order, regenerating the same report. The job declares its unique ID and Laravel will not queue a duplicate while the first one is still pending.

Do I need Horizon to use queues in Laravel?

No. Queues work with queue:work and a process supervisor. Horizon is worth it when you use Redis and want worker configuration in code, automatic queue balancing, and metrics and failures in a dashboard.