Optimizing WordPress for OCR + LLM in production: queue, cron and uniqueness with 80,000 folios

Optimizing WordPress for OCR + LLM in production has nothing to do with caching plugins or image compression. It comes down to something more basic: heavy work must never happen inside a web request.

I learned that building the system behind Panini Fan Zone FWC 2026. It was two custom WordPress plugins that managed a fixed pool of 80,000 folios (unique entry codes) and verified purchase receipts with Tesseract and a LLaMA 3.3 70B model served by Groq.

In the Python optimization article I covered the processing side. This is the other side: how WordPress is prepared to receive, queue and record that work without mixing it with the public pages.

The code examples are illustrative. They show the pattern; table names, columns and paths are made up for the example.

The problem: WordPress is not a worker

WordPress is built to answer requests: someone asks for a page, PHP assembles it and returns it. Every request has a time limit and shares resources with all the others.

An OCR + LLM pipeline does not fit that mold. Reading an image with Tesseract burns CPU. Calling a remote model depends on the network and on the provider's limits. Neither should keep a user waiting.

The most common mistake is processing the receipt at the moment it is uploaded. The form keeps spinning, the request approaches its timeout and, if several people upload at once, PHP processes get tied up with work that is not serving pages.

The architecture that works separates three moments:

• Receive. The web request validates the minimum, stores the file and creates a row in the queue. It responds right away.
• Process. A separate process, triggered by the system cron, takes batches from the queue and runs them through OCR and the LLM.
• Record. The result is written to the database under rules that MySQL enforces.

In the real case, that process ran every 2 minutes on an AlmaLinux VPS, with nobody reviewing receipts one by one.

Taking work out of the request: system cron, not WP-Cron

WP-Cron has a misleading name. It is not a cron: it is a check WordPress makes when a page loads. If there are overdue events, it fires a request to wp-cron.php to run them.

That has two consequences for a critical pipeline. With no visits, nothing gets processed. With many visits, execution depends on traffic, not on a schedule. Neither is acceptable when receipts are waiting.

The fix is to turn off visit-triggered execution and let the operating system set the pace:

# wp-config.php: WP-Cron stops firing on page visits.
#   define( 'DISABLE_WP_CRON', true );

# Site user's crontab: the system sets the pace, not the traffic.
# Every 2 minutes, with flock so two runs never overlap.
*/2 * * * * flock -n /tmp/verifier.lock \
  wp --path=/home/site/public_html cron event run --due-now \
  >> /home/site/logs/cron.log 2>&1

# Check which events WordPress has scheduled before changing anything.
wp cron event list --fields=hook,next_run_relative

Three decisions in a few lines. DISABLE_WP_CRON takes the work away from visits. WP-CLI runs the events outside Apache, without a web request's time limit. And flock -n drops the new run if the previous one is still alive.

One detail people forget: disabling WP-Cron also stops other plugins' tasks, such as scheduled posts or cleanups. The system cron must run every overdue event, not just the pipeline's.

A queue in its own table, with states and indexes

The temptation is to store each receipt's status in post meta or in wp_options. That works with dozens of records. With thousands, it becomes a problem.

wp_options is loaded on every request when the option is set to autoload. And post meta is a key-value table: asking "give me the oldest pending items" forces MySQL to scan far more than it needs to.

A dedicated table gives the worker exactly the query it needs:

CREATE TABLE wp_verif_queue (
  id          BIGINT UNSIGNED AUTO_INCREMENT PRIMARY KEY,
  folio       CHAR(12) NOT NULL,
  status      ENUM('pending','processing','approved','rejected','error')
              NOT NULL DEFAULT 'pending',
  attempts    TINYINT UNSIGNED NOT NULL DEFAULT 0,
  claimed_by  CHAR(36) NULL,            -- which run claimed the row
  claimed_at  DATETIME NULL,            -- detects orphaned jobs
  -- The worker always asks for the oldest pending rows: a purpose-built index.
  KEY idx_status_id (status, id),
  UNIQUE KEY uq_folio (folio)           -- one receipt per folio, enforced by MySQL
) ENGINE=InnoDB;

The explicit states are the important part. A receipt is pending, processing, approved, rejected or in error. There are no ambiguous combinations, and anyone on the team can count how many sit in each phase with one query.

The claimed_at column handles a case that always arrives: a run that dies halfway. If a row has been in processing for too long, it goes back to pending and gains an attempt. If it exceeds the maximum attempts, it moves to error for review.

In a plugin, this table is created with dbDelta() on activation, and its version is stored in an option to know when a migration is needed.

Claiming work without two processes taking the same thing

flock prevents two cron runs from overlapping. But if you ever run two workers on purpose, you need another guarantee: they never take the same row.

The pattern is to claim with an atomic UPDATE and then read only what that worker marked:

// Claim a batch atomically: two workers never take the same row.
global $wpdb;
$token = wp_generate_uuid4();
$wpdb->query( $wpdb->prepare(
    "UPDATE {$wpdb->prefix}verif_queue
        SET status = 'processing', claimed_by = %s, claimed_at = NOW()
      WHERE status = 'pending' ORDER BY id LIMIT %d",
    $token, 25
) );
// Process only what this worker marked with its token.
$batch = $wpdb->get_results( $wpdb->prepare( "SELECT id, folio FROM {$wpdb->prefix}verif_queue WHERE claimed_by = %s", $token ) );

MySQL runs the UPDATE as a single operation. If two workers fire it at the same time, each row ends up with one token. There is no window between "read" and "mark" for another process to slip through.

The batch size is deliberately small. A short batch finishes within the cron window, and if something fails, little work is lost.

Uniqueness: the rule lives in MySQL, not in an if

This is the lesson that cost me the most on the project. The registration module for the Guinness World Record attempt needed its own rule: one person, one registration, scoped to that specific activity.

I gave it its own database schema instead of reusing the folio plugin's validation. Even so, the first version of the uniqueness index was not scoped to the activity the way it should have been, and the same email could be counted in a way the rule did not allow.

The fix was a composite unique index scoped to the activity, plus a migration that checks the existing schema before altering it. The plugin went from version 1.1.0 to 1.1.1.

// v1.1.1: a migration that checks the schema before altering it.
// ⚠️ Production: back up the table first (mysqldump --single-transaction).
function registration_migrate_index() {
    global $wpdb;
    $table = $wpdb->prefix . 'record_registration';
    // Idempotent: if the composite index already exists, there is nothing to do.
    if ( $wpdb->get_var( "SHOW INDEX FROM {$table} WHERE Key_name = 'uq_activity_email'" ) ) { return; }
    // A pre-existing duplicate would make the ALTER fail: stop first and log it.
    $dup = $wpdb->get_var( "SELECT COUNT(*) FROM (SELECT 1 FROM {$table} GROUP BY activity_id, email HAVING COUNT(*) > 1) d" );
    if ( $dup ) { error_log( "registration: {$dup} duplicates, migration stopped" ); return; }
    $wpdb->query( "ALTER TABLE {$table} ADD UNIQUE KEY uq_activity_email (activity_id, email)" );
}

That function holds three ideas that apply to any plugin with real data:

• Idempotency. The migration can run twice without harm. If the index already exists, it exits.
• Pre-flight check. If there is already data that violates the new rule, the ALTER would fail. It is better to stop, log it and calmly decide what to do with that data.
• The rule in the database. A PHP if that asks "does this email already exist?" before inserting has a race: two simultaneous requests can both pass the check before either inserts. The unique index has no such race.

Validation in PHP is still useful for a friendly message. But the guarantee comes from MySQL.

Proving concurrency: 50 requests at the same slot

A well-placed index is a hypothesis until you test it under load. For that module I wrote an independent verifier script and a suite of 15 tests.

One of them launched 50 simultaneous workers against the same registration route. The success condition had two parts: exactly one registration wins each slot, and zero server errors.

The simplest way to reproduce that test fits in a few lines of Bash:

# 50 simultaneous requests at the same slot: exactly one must win.
# Staging only: never fire this at production.
URL="https://staging.example.test/wp-json/registration/v1/enroll"
seq 50 | xargs -P 50 -I{} curl -s -o /dev/null -w "%{http_code}\n" \
  -X POST "$URL" -H "Content-Type: application/json" \
  -d "{\"activity\":7,\"email\":\"qa@${QA_DOMAIN}\"}" \
  | sort | uniq -c
# Expected: 1 response 201, 49 responses 409 and no 5xx.
# A 500 means MySQL stopped the race, but the plugin did not turn the error into a 409.

The second condition matters as much as the first. If the index stops duplicates but the plugin lets the exception escape as a 500 error, the rule works and the experience does not.

With the migration in place, the module passed its full verifier, 24 out of 24 checks, and its suite, 15 out of 15 tests. That evidence is what lets you say "done".

Keeping the pipeline away from public pages

All of the above has a benefit that is not obvious: public pages never find out the pipeline exists. And that is exactly what protects the site's performance and SEO.

• Visits do not run heavy work. With WP-Cron disabled, loading a page does not trigger OCR, model calls or cleanups.
• Each plugin loads only where it is used. The form's scripts and styles are enqueued only on the registration page, not across the whole site.
• Internal routes are not indexed. API endpoints and staff pages stay out of the sitemap and require authentication.
• The queue does not share tables with content. The worker's queries do not compete with the ones that build pages.

The result is a site that is still a site, with a processing system next to it, not inside it.

What NOT to do

• Process the file in the same request that receives it. The user waits, the PHP process stays busy and the request may time out. Receive, queue and respond.
• Trust uniqueness to a prior check in PHP. "Check if it exists, then insert" has a race that only shows up with real traffic. A unique index removes it.
• Migrate the schema without a backup or a check. An ALTER TABLE over data that already violates the new rule fails halfway, in production, with the event running. Back up, check for duplicates and make the migration idempotent.

The next step

Optimizing WordPress for an OCR + LLM pipeline is a matter of separation: receive on the web, process outside it, and let MySQL enforce the rules. With that in place, the AI part can be swapped or scaled without touching the site.

If your team is facing WordPress bottlenecks, needs to integrate AI processing without putting the site at risk, or wants to audit plugins already in production, at Conqr Digital Agency we design architectures that scale. Let's review your case.

The full system is in the Panini Fan Zone case study. If this is the kind of work you need, take a look at the custom web development and AI automation services.

FAQ

Why not use WP-Cron for an OCR + LLM pipeline?

Because WP-Cron only fires when someone visits the site, and it runs inside that visit. With no traffic nothing gets processed, and with heavy traffic it can launch overlapping runs. For critical work it is better to disable it and call the events from the system cron.

Can I keep the queue in wp_options or post meta?

For a handful of tasks, yes. For thousands, no: wp_options is loaded on every request and post meta has no indexes designed for querying by status. A dedicated table with its own index answers the worker's query without affecting the rest of the site.

How do I test that my registration endpoint handles simultaneous requests?

In a test environment, fire many requests at the same time against the same slot and count the responses. Exactly one must win, the rest must get a controlled rejection, and no server error should appear.