How to Scale PDF and Excel Report Generation

Report export usually begins as a button. A user clicks “Export to PDF”, the web server queries the data, renders the document and streams it back, all within one request. With a few hundred rows that works well. Then customers accumulate years of data, reports start including photos and attachments, and the same button begins to time out, exhaust memory or return a half-finished file.

The fix is rarely a faster library. It is an architectural change: stop generating heavy documents inside the web request. This article explains why report generation becomes difficult, how a queue-based architecture solves it, and the details that decide whether that architecture is reliable in production.

Why Document Generation Becomes Difficult at Scale

Several pressures arrive together:

• Large datasets. The query returns tens of thousands of rows, and a naive implementation loads them all into memory before rendering.
• Images and attachments. Photos taken on modern phones are large. Downloading them, decoding them and embedding them at full resolution multiplies both processing time and file size.
• File formatting. PDF layout involves pagination, repeated headers, fonts and tables that split across pages. Excel output involves styles, formulas and multiple sheets. The libraries for each format behave very differently.
• Memory usage. Rendering PDFs with a headless browser is memory-hungry, and building a large workbook entirely in memory can exceed what a web server process should use.
• Execution timeouts. Load balancers, API gateways and serverless functions all limit how long a request may run. A user may also give up and click again, starting a second copy of the same work.

None of these problems appears in a demo with ten rows, which is why they tend to surface in production, with the largest customers.

Synchronous vs Asynchronous Generation

Synchronous generation is not wrong. For small, bounded documents that reliably finish in a few seconds, such as a single invoice or a one-page summary, it is the simplest option.

Consideration	Generate in the request	Use a background job
Document size	Small and bounded	Unbounded, or growing with customer data
Duration	Reliably a few seconds	Variable, sometimes minutes
Attachments	None or few, already optimized	Many, or user-uploaded images
Concurrency	Occasional	Many users exporting at the same time
Failure handling	The user retries	Automatic retries, status and alerts
User experience	Immediate download	Progress status, then a notification or download link

Moving to asynchronous generation changes the user experience, so design it deliberately. The request returns immediately with a job reference, often as 202 Accepted. The interface shows the job’s status and offers the download when it is ready, and for longer jobs an email or in-app notification saves the user from watching a progress bar.

A Queue-Based Reporting Architecture

The core pattern has five parts:

1. API. Validates the request, checks that the user may see the data, records a job with the status “queued” and places a message on a queue. The message contains the job ID, not the data.
2. Job queue. Holds the work until a worker is ready, absorbing bursts of requests so they do not overwhelm the database.
3. Worker. Takes a message, loads the job and its parameters, queries the data in chunks, renders the document, uploads it and updates the job status.
4. File storage. Keeps the finished file privately.
5. Download. The application checks permission again and provides the file through a short-lived link.

The job record is the backbone of the design. Store who requested it, the parameters, its status, the number of attempts, timestamps, the location of the output and any error. Status pages, retries, support investigations and clean-up jobs all read from it.

Separating PDF and Excel Processing

PDF and Excel generation look similar from the outside, since both turn data into a file, but they behave differently. PDF rendering, especially through a headless browser, needs more memory and is sensitive to fonts and layout. Excel generation is mostly about writing many rows efficiently and can often stream them to the file.

Running them as separate workers, or at least separate queues, lets each be sized and tuned independently. A memory-heavy PDF job then cannot starve a burst of quick spreadsheet exports, and a change to the Excel library cannot break PDF output. Keep the data-access layer shared, so both formats show the same numbers, and keep the format-specific rendering separate.

Handling Attachments and Large Datasets

Stream instead of loading. Query the data in pages and write rows as they arrive. Many spreadsheet libraries offer streaming writers for exactly this. For large PDFs, render sections independently and merge them.

Size images for the document. A photo shown a few inches wide on a page does not need its original resolution. Resize and compress images before embedding them, cache the processed versions, and decide what happens when an attachment is missing: a clearly marked “image unavailable” note is usually better than a failed report.

Split very large reports. When a single report is too big for one worker, divide it into parts that run in parallel, then assemble the result. This needs coordination, which is where an orchestration service such as AWS Step Functions helps, as described below.

Set limits. A maximum number of rows or attachments per report, with a clear message suggesting filters, protects the system and is usually more helpful to users than a document nobody can open.

Retries, Dead-Letter Queues, and Idempotency

Amazon SQS standard queues use at-least-once delivery (opens in a new tab): a message can occasionally be delivered more than once. Workers therefore have to be idempotent. Before starting, check whether the job is already complete. Write the output to a location derived from the job ID, so a second run overwrites rather than duplicates, and update the status with a condition so two workers cannot both mark the job done.

The visibility timeout controls how long a message stays hidden while a worker processes it. If it is shorter than the processing time, another worker picks up the same job mid-flight. When AWS Lambda consumes the queue, AWS recommends setting the visibility timeout to at least six times the function timeout (opens in a new tab), which leaves room for retries when the function is throttled.

Separate temporary failures from permanent ones. A timeout calling the database is worth retrying; a report whose parameters are invalid will fail every time, so mark it failed and stop. For messages that keep failing, configure a dead-letter queue (opens in a new tab): after a set number of receives, SQS moves the message aside so it stops consuming capacity. AWS recommends a maxReceiveCount of at least 5 for Lambda consumers. Alarm on any message reaching the dead-letter queue, fix the cause, then redrive the messages. When Lambda processes messages in batches, enable partial batch responses so one failing report does not send the whole batch back to the queue.

Secure File Storage and Download Access

Reports often contain a customer’s most sensitive business data, so treat the files that way:

• Store them in a private bucket with public access blocked, and encrypt them at rest.
• Organize object keys by tenant, and use unguessable names, but do not rely on obscurity for security.
• Check authorization in the application every time someone asks for a download.
• Deliver the file through a short-lived link, or stream it through the application.
• Expire old reports automatically with lifecycle rules that match your retention policy.

Amazon S3 presigned URLs (opens in a new tab) grant time-limited access to a private object. Anyone holding the link can use it until it expires, so keep the expiry short and generate the link only after the permission check.

Monitoring Report Jobs

Background work fails quietly unless you watch it. Useful signals include:

• Queue depth and the age of the oldest message. SQS publishes both as CloudWatch metrics; a growing age means workers are not keeping up.
• Job duration by format and report size, so you can see which reports are approaching limits.
• Failure rate and the number of messages in the dead-letter queue.
• Worker memory and output file sizes.

Log every step with the job ID so one report can be traced from request to download. Add a periodic sweep for jobs stuck in “processing” longer than any job should take, and show users an honest status. “Failed, please try again or contact support” is better than a spinner that never stops.

AWS Implementation Considerations

A common AWS implementation combines API Gateway or an application server for the API, Amazon SQS for the queue, AWS Lambda for workers and Amazon S3 for storage. A few service limits shape the design. Lambda functions run for at most 15 minutes per invocation and can be configured with 128 MB to 10,240 MB of memory (opens in a new tab), with CPU allocated in proportion to memory, and temporary /tmp storage of up to 10,240 MB. SQS visibility timeouts can be extended up to 12 hours from when a message is first received.

Some practical points:

• Keep messages small. Put the job ID in the message and load the parameters from the database. Event payload sizes are limited.
• Protect the database. Many concurrent workers can exhaust database connections. Limit worker concurrency on the event source, and use connection pooling.
• Give each worker its own permissions. A PDF worker needs to read the data it renders and write to one storage location, and nothing more.
• Watch cold starts and package size for workers that bundle a headless browser.

When to Move Beyond Serverless Workers

Lambda suits bursty, short-to-medium jobs. Look at other options when a single job can exceed the 15-minute limit, when it needs more memory or disk than a function provides, or when it keeps a heavy process like a browser running for a long time.

The usual next steps are container-based workers on Amazon ECS with AWS Fargate, or AWS Batch for large batch workloads. Both are designed for long-running work and are not bound by Lambda’s 15-minute limit. For multi-step processes, such as splitting a report into parts, rendering them in parallel and assembling the result, AWS Step Functions (opens in a new tab) provides orchestration. Its Standard Workflows can run for up to one year, with exactly-once workflow execution.

The most valuable design choice is keeping the job contract stable. If the API creates jobs, workers update a status record and files land in private storage, the worker runtime can change from functions to containers without changing the API or the user interface.

Example Enterprise Reporting Architecture

The enterprise reporting case study describes this pattern in practice. It draws on anonymized enterprise engineering experience rather than a PathWebSolutions client engagement, and no organization or system is identified.

In that architecture, an API validates each report request, records a job and returns immediately. The job goes onto Amazon SQS, AWS Lambda background processing hands it to a dedicated PDF or Excel worker, attachments and images are processed and assembled into the final report, and the file is written to private Amazon S3 storage. The application tracks the job’s status and provides a controlled download. The trade-offs it records match the ones above: the status and download experience, concurrency limits, image resolution and chunked processing. No performance, volume or cost figures are published for that work.

If your exports are timing out or getting slower as customers grow, our reporting and document automation service and cloud and backend engineering service cover this kind of redesign.