Python optimization: how to speed up scripts that process 80,000 folios with OCR + LLM

Python optimization rarely starts with rewriting code. It starts with knowing where the time goes. I learned that on a system that managed a pool of 80,000 folios in production, with receipts verified by OCR + LLM and no manual review.

In this article I go through three decisions that define the performance of a bulk-processing script: the data structure, memory handling, and how the work is distributed. Each one comes with code you can adapt today.

One note before we start: the examples are illustrative. They show the pattern, not the client's code, and the table names and folio formats are made up for the example.

The case: 80,000 folios, OCR and an LLM in production

The system is Panini Fan Zone FWC 2026. It issued and controlled a fixed pool of 80,000 folios (unique entry codes), and verified the purchase receipts that participants submitted.

Verification was a two-step pipeline. First, Tesseract extracted the text from the image. Then a LLaMA 3.3 70B model, served by Groq, validated that text against the promotion rules.

Everything ran unattended, on a cron job every 2 minutes, on an AlmaLinux VPS. Nobody was reviewing receipts one by one.

That design imposes three constraints that any script of this kind shares:

• Each run has a short window. If one execution takes longer than 2 minutes, it overlaps with the next.
• Memory is shared. The script lives on the same server as the website; if it eats the RAM, it takes the site down.
• The slow step is external. The LLM call depends on the network and on the provider's limits, not on your CPU.

With that in mind, let's look at where optimizing actually pays off.

Optimizing Python scripts with lists vs sets

The first mistake is the cheapest to fix. Validating a folio is a membership question: does this code exist in the pool?

If the pool lives in a list, Python compares item by item until it finds a match. With 80,000 folios, the worst case is 80,000 comparisons per lookup. That is O(n).

A set stores its elements in a hash table. It computes the hash of the value you are looking for and jumps straight to its slot. On average that is O(1), whether the pool has 800 or 80,000 elements.

The change is one line. The difference grows with the size of the data:

import timeit

folios = [f"FZ-{n:06d}" for n in range(80_000)]
folios_set = set(folios)      # built once, at startup
wanted = "FZ-079999"          # worst case for the list: it is at the end

# The list scans until it finds it: O(n) per lookup.
t_list = timeit.timeit(lambda: wanted in folios, number=1_000)
# The set computes a hash and jumps straight there: O(1) on average.
t_set = timeit.timeit(lambda: wanted in folios_set, number=1_000)

# Don't trust someone else's numbers: measure on your machine, your data.
print(f"list: {t_list:.4f} s   set: {t_set:.4f} s")

Run it and compare. I'm not quoting a number here because it depends on your hardware and Python version, and because the lesson is not the figure: the cost of the list grows with the pool, and the cost of the set does not.

Three details that prevent surprises:

• Build the set once. Calling set(folios) inside the loop costs O(n) on every iteration and wipes out the gain.
• A set keeps neither order nor duplicates. If you need them, keep the list and use the set only as a lookup index.
• To relate data, use a dictionary. If you need the folio's status and not just its existence, a dict from folio to record gives you the same access speed.

And one architecture rule: if the pool already lives in MySQL behind a unique index, the database is your set. Pulling 80,000 rows into memory to look up one is the same mistake in a different costume.

Memory management in Python: process as a stream

Python frees memory through reference counting. When nothing points to an object any more, it is destroyed right then. The garbage collector only steps in for reference cycles.

The practical consequence is simple: memory is not released while you hold a reference. And the most common way to hold references by accident is to accumulate everything in a list.

The pattern that causes it looks innocent: a fetchall() that pulls the whole table, a results list that grows during the run, opened images that nobody closes.

The alternative is to process as a stream. A generator hands over one row at a time and keeps only the current batch in memory:

def pending_receipts(connection, batch=500, cap=5_000):
    """Yield pending rows without loading the whole table."""
    cursor = connection.cursor(dictionary=True)
    # The cap bounds the run: whatever is left goes to the next cron run.
    cursor.execute(
        "SELECT id, image_path FROM receipts "
        "WHERE status = 'pending' ORDER BY id LIMIT %s",
        (cap,),
    )
    while True:
        rows = cursor.fetchmany(batch)   # memory stays bounded by the batch
        if not rows:
            break
        yield from rows                  # the consumer sets the pace
    cursor.close()

Memory use stops depending on the size of the table. It depends on the size of the batch, and you decide that.

With images the rule is the same, only more urgent. A photo of a receipt takes far more memory once decoded than it does on disk. Open it in a with block, extract the text, and let it go before moving on to the next one.

For diagnosis, the standard library already ships the tooling:

• tracemalloc takes memory snapshots and tells you which line allocated each block. Compare a snapshot at the start against one taken after several batches.
• cProfile tells you which functions the time goes to. Use it before touching a single line.
• __slots__ shrinks every instance when you create thousands of small objects of the same class.

Calling gc.collect() by hand is almost never the fix. If memory grows, something is holding a reference. Find it.

Idempotent batches and capped concurrency

A cron job every 2 minutes carries an obvious risk: a slow run overlaps with the next one and both process the same receipt. That means paying for the LLM twice and, worse, writing two results.

The defense has two layers. The first is a file lock: if the previous run is still alive, the new one exits without doing anything. The second is making every operation idempotent: processing the same receipt twice must leave the same result.

On that foundation you can parallelize. OCR runs in a subprocess and the LLM is a network call, so the work is I/O-bound. Threads do help there, even with the GIL.

import fcntl
import sys
from concurrent.futures import ThreadPoolExecutor

# ⚠️ Production: back up the table before changing the status flow.
lock = open("/var/lock/verifier.lock", "w")
try:
    # If the previous run is still alive, this one exits: no double work.
    fcntl.flock(lock, fcntl.LOCK_EX | fcntl.LOCK_NB)
except BlockingIOError:
    sys.exit(0)

# Few workers on purpose: the real limit is set by the LLM provider.
with ThreadPoolExecutor(max_workers=4) as pool:
    pending = pending_receipts(connection)
    for result in pool.map(verify, pending):
        save(result)   # a single thread writes: no races in MySQL

There is a subtle detail: pool.map consumes the whole generator when it queues the tasks. That is why the query has a cap. Without it, parallelism takes you right back to the memory problem you just solved.

The number of workers is not chosen by gut feeling. It is tuned to your provider's rate limit and raised only when measurements justify it.

The same discipline applies to testing. In that project's registration module, the suite included a concurrency test with 50 simultaneous workers hitting the same route, to confirm that exactly one registration won each slot. Concurrency is proven, not assumed.

Best practices for efficient Python scripts

With the data structure, memory and batches sorted out, what remains is discipline. These are the practices that pay off most in an OCR + LLM pipeline:

• Profile before you optimize. If 90% of the time goes to waiting on the LLM, optimizing the Python loop changes nothing.
• Filter cheap before you spend big. If OCR returned empty or unreadable text, reject right there. Don't pay for a model call to confirm the obvious.
• Validate the LLM output with deterministic rules. Ask for a structured response and check every field in code. The model proposes; your rules decide.
• Set timeouts and retries with backoff. A remote API will fail some day. One failure should not stop the whole batch.
• Don't repeat work. Store the hash of every processed image. A resubmitted receipt should not cost another round of OCR and model.
• Log per batch, not per line. How many came in, how many passed, how many failed, and how long it took. That is how you spot degradation before it becomes an outage.

None of these practices is exotic. The hard part is applying all of them at once, and doing it before volume forces you to.

What NOT to do

• Optimize without measuring. Rewriting a loop with unreadable tricks to save milliseconds while the script spends seconds waiting on an API. Measure first; optimize only what the profile points at.
• Load everything into memory "because it fits". It fits today, on your machine, with test data. In production the table grows and the server is shared with the website.
• Parallelize without a limit. Firing hundreds of threads at the LLM speeds nothing up: it trips the rate limit, floods the error log and makes cost unpredictable.
• Trust the LLM as the only validation. A language model can be wrong with total confidence. Business rules live in code and in database constraints.

The next step

The Python optimization that matters in production is not a syntax trick. It is choosing the right structure, bounding memory, and distributing work without runs stepping on each other.

If your team is facing bottlenecks in bulk processing, OCR pipelines, or LLM integrations that never quite become reliable, at Conqr Digital Agency we design architectures that scale. Let's review your case.

If you want to see the full system behind this article, it is documented in the Panini Fan Zone case study. And if automating processes with AI is your goal, take a look at the AI automation service.

FAQ

When should I use a set instead of a list in Python?

When the dominant operation is asking whether an element exists. A list scans item by item; a set resolves by hash in constant time on average. If you need order or duplicates, keep the list and build a separate set for lookups.

Threads or processes for an OCR + LLM pipeline?

If OCR runs in a subprocess and the LLM is a remote API, the work is I/O-bound and threads are enough. If OCR runs inside the same Python process and saturates a core, use processes.

How do I know if my script has a memory leak?

Take two tracemalloc snapshots, one at the start and one after several batches, and compare them. If the same line of code keeps growing batch after batch, that is where objects are piling up.