The Comprehensive Guide to Modern Lab Data Management

In a June 2024 FDA Warning Letter, drug applications from various labs were deemed unacceptable due to systemic data integrity failures at Synapse Labs, a contract testing lab they relied on.

Among these failures were irregularities in test results, unexplained anomalies, and, in some instances, indications of falsified data. This is an extreme example, to be sure, but how well does your lab have a read on every single sample, test result, instrument calibration record, or unit of inventory?

Manual methods for data management may have worked in prior decades. But today, labs are in danger of falling behind or shutting their doors if they cannot keep up with demand while still meeting compliance requirements. 

In this guide, we’ll share how modern labs manage their data to support scale.

What is Lab Data Management?

Lab data management is the process by which labs handle, organize, and secure their data throughout its lifecycle – from initial sample intake to final reporting and long-term archival.

The type of information your lab handles will vary by industry, but it generally falls into the following buckets:

• Sample data: This includes unique identifiers, chain of custody, storage locations, test assignments, and status tracking.
• Inventory data: Such as reagents, consumables, reference standards, expiration dates, and reorder thresholds.
• Customer data: Including client information, submission history, reporting preferences, and billing details.
• Test data: This consists of methods, raw instrument output, calculations, results, QC checks, and approval workflows.

For decades, labs managed data in homegrown systems, or even rudimentary systems of notecards and physical notebooks. That worked when sample volumes were lower, regulatory scrutiny was lighter, and a single technician could keep the full picture in their head. Those days are gone.

Why Effective Data Management Matters

As shared above, poor data management led to the shutdown of operations for a contract testing lab and the denial of applications for labs that relied on it. Clearly, data management matters, but here are three more reasons to pay attention:

• It’s required
• It’s efficient
• Your customers deserve it

It’s Required

The obvious reason is that proper data management is required to meet regulatory compliance standards. This is the case across industries. For example, here’s how the following regulations impact data management:

• HIPAA: HIPAA has high standards for patient and clinical data management. These standards are so high that only HIPAA-compliant software is suitable for use in some cases.
• ISO 17025: ISO 17025 requires labs to maintain traceable, accurate records with proper version control to demonstrate technical competence
• 21CFR Part 11: 21 CFR Part 11 mandates electronic records have audit trails, electronic signatures, and system validation to ensure data integrity for FDA-regulated operations

You could argue that you can follow data management best practices by maintaining and organizing your spreadsheets and notebooks, and that’s a fair point. But it’s worth noting that your data must be defensible and accessible, and should the FDA drop by for an audit, they need to be able to review it. 

One cosmetics lab we spoke with managed their data in 4x6 notecards and lab notebooks that they stored in a file drawer. This worked well, until an auditor refused to touch them, claiming they were “disgusting” and demanded digital records to review. 

It’s Efficient

We’ve heard horror stories of labs that double-tested samples, ran out of inventory, or wasted precious hours trying to reconcile test results to isolate what went wrong. Manual data management places the onus on your lab to own every step of every process, while automated systems and software (which we will cover later) can help build guardrails for your team. 

Using our earlier example, it wasn’t just Synapse Labs that was affected. Here’s what the FDA has to say to all the labs that worked with them:

FDA is requiring sponsors of approved, tentatively approved or pending ANDAs and NDAs to repeat the bioavailability/bioequivalence studies, when those studies are essential for approval, using an entity other than Synapse or any other organization with known unresolved data integrity concerns…. FDA has determined that Among these failures were irregularities in test results, unexplained anomalies, and, in some instances, indications of falsified data. This is an extreme example, to be sure, but how well does your lab have a read on every single sample, test result, instrument calibration record, or unit of inventory?, or to show affected generic products are bioequivalent to brand-name products. (Emphasis added)

Cutting costs and corners may save time now, but in the case above, all the labs that relied on Synapse needed to re-run their tests and resubmit their applications. That’s costly hours wasted and thousands more spent.

Your Customers Deserve it

Compliance and efficiency aside, your customers and stakeholders deserve you to take your data seriously.

We’ve worked with labs testing food for health and safety, clean energy labs that tap into carbon sequestered underground, a popular West Coast fast-food chain, and many more labs that directly impact the health, safety, and well-being of people across the globe.

You may be tempted to think that manual methods are good enough with proper care and attention. Next, we’ll reveal why that’s not always the case.

Why Spreadsheets are Not Enough for Data Management

Spreadsheets and paper-based methods may work for a period of time, until they don’t. 

The most common case we see is labs trying to persist with brittle systems until a catastrophic failure occurs, such as data loss or a failed audit, which forces them to take their operations seriously. While spreadsheets might work, they are not enough to shoulder the data load of modern labs for these reasons:

• Not scalable
• Ripe for errors
• Insecure by design
• Hidden costs of manual work

Spreadsheets Are Not Scalable

As your lab grows, spreadsheets and paper systems become unmanageable. This is especially the case for labs that increase sample volume or introduce new test types.

While a small-scale operation can be faithfully and accurately documented in a notebook, the minute you scale your throughput, test volume, or team size, those manual systems will break down. From missing data to testing samples multiple times across your staff, there are numerous ways that information can slip through the cracks.

A 20-sample-per-day lab can manage in Excel. At 200 samples, you're hiring someone just to maintain the spreadsheet. At 2,000, you're crossing your fingers hoping nothing breaks.

Spreadsheets Are Ripe for Errors

Beyond scalability issues, manual systems are prone to data-entry errors.

In 2016, a study on healthcare data accuracy showed that manual compliance processes result in error rates of 79%-87% when complex spreadsheets are used, compared to <2% when processes are automated. 

Paper-based systems create audit trail gaps, making it difficult to prove data integrity during regulatory inspections. As we’ve shared above, physical and manual data capture methods may work at a small scale but quickly become unsustainable and can lead to failed audits, loss of certification, or legal liability. 

Spreadsheets Are Insecure by Design

A paper notebook can be locked in a drawer, but even that is inherently less secure than encrypted cloud-hosted software. 

When you manage data manually, it is up to your team to physically secure your systems to prevent unauthorized access from within and outside your team, and to ensure data is backed up in the event of a disaster. 

Cloud-hosted software, like QBench LIMS, gives you:

• HTTPS encryption
• Regular data backups
• Role-based access control for users

Spreadsheets Carry the Hidden Costs of Manual Work

Remember those earlier labs that relied on Synapse Labs? Months of testing and research were needed again with a new contract testing lab that had proper data management practices.

It’s not always so extreme, though. Sometimes double work and wasted hours can seem benign – until you add up all the time spent. One dietary supplement lab we spoke with struggled to meet demand without a LIMS. They spent hour after tedious hour generating new COAs every single week while sifting through spreadsheets filled with test data. This slowed them down, prevented them from scaling, and led to double-work across their team.

Manual methods may work to a point, but they’re an accident waiting to happen. Whether it’s double-testing samples because data is spread across spreadsheets and notebooks or a compliance failure, it’s not a matter of if but when.

The key is to leverage software and automation to manage your lab’s data. Read on for the best platforms to choose from.

The Best Software for Modern Lab Data Management

Now that you’re convinced software is the answer, where do you start?

There are three categories of software we recommend. You may wish to combine them, so we recommend treating this list as a “both-and” rather than an “either-or.”

• ELN
• QMS
• LIMS

ELN

ELNs (Electronic Lab Notebooks) are a digital version of a traditional paper lab notebook. They provide a structured, yet flexible, environment for documenting experiments, protocols, and observations. 

Beyond that, ELNs are useful for:

• Standardizing data capture and documentation.
• Providing version control and data audits.
• Allowing for real-time collaboration.
• Integrating with more powerful platforms, like a LIMS.

An ELN on its own is a good start, but it's rarely enough if your lab has more complex needs, such as inventory, sample, or test management.

QMS

A QMS (Quality Management System) is a software platform designed to help labs meet customer requirements and regulatory standards. This can ensure that your lab consistently and efficiently produces products and performs services. Modern labs rely on a QMS for the following:

• Monitoring quality control
• Calibrating instruments
• Tracking processes and modifications
• Helping labs meet regulatory requirements (such as ISO 17025)
• Ensuring labs follow Good Manufacturing Practice (GMP)
• Managing CAPA (corrective and preventive actions)

With a QMS, your lab can standardize its processes, demonstrate reliability, and pinpoint issues should they arise.

Still, a QMS cannot address all the tasks we mentioned at the start of this article, such as sample or inventory management. For that, modern labs need a LIMS.

LIMS

A LIMS is a comprehensive software platform designed to manage and track samples, tests, and results throughout the entire lab workflow. 

LIMS are information management and automation powerhouses for modern labs, available in multiple deployment models and configurations:

• They can be built from scratch and self-hosted or licensed
• You can host them yourself (on-prem) or access them via the cloud
• Some require heavy customization to make changes, while others offer in-app configuration
• Some are industry-agnostic, while others are industry-specific
• Some are enterprise platforms with pro services, while others are self-service

A LIMS can be a fantastic platform for automating workflows and strengthening compliance. Some of the top features of a LIMS include:

• Sample management and tracking
• Workflow automation and optimization
• Instrument integration and data capture
• Quality control and assurance
• Reporting and data analysis
• Chain of custody management
• Product lot tracking
• Batch test re-runs
• Equipment scheduling and calibration

Between these three platforms, when it comes to data management there is no substitute for a LIMS. In an interview with the founder of JAF Consulting, a lab compliance and validation firm, a LIMS was the clear winner for labs looking to improve their processes:

“If you configure the workflows and all of your processes correctly, a LIMS will force compliance. You also force doing things the wrong way if it's defined the wrong way. When these systems force compliance, they force the activities. There are gates, approvals, and everything else to move data through. For calibration programs, a LIMS will notify you when the equipment is ready to be calibrated. A LIMS allows you to register a piece of equipment if it's in this calibration period, and won't allow you to do it if it's outside of this calibration period. It'll let you know when equipment's down, up, and under maintenance.” Joe Franchetti, founder of JAF Consulting

A LIMS is a value-add, but it’s only as good as you set it up to be. Next, we’ll share how to select the right software for your lab.

What to Look for When Selecting Software for Your Lab

Once you know what you need, it’s time to choose your software. Based on the above descriptions, first determine whether you need a LIMS, ELN, QMS, or a combination of the three. Some LIMS come with a built-in QMS, so take the time to review each vendor in full, as one may cover multiple use cases. 

Beyond that, it’s also worth considering the following:

• Cloud vs. on-premise: Cloud-based LIMS offer faster implementation, lower upfront costs, automatic updates, and easier remote access, while on-premise systems may be preferred by labs with specific security requirements or limited internet connectivity.
• Configurability: Does the vendor require custom coding from a developer, or is it configurable so anyone on your team can log in and adjust settings to adapt the platform to your needs? Take care to evaluate a vendor’s stance on this, and ask for examples of what can be configured in the platform.
• Implementation timeline and approach: No LIMS comes “out of the box,” but some platforms can be implemented more easily than others. Take care to understand the vendor’s approach to implementation and requirements before you begin. Fast is not always best; some vendors offer a quick implementation with months of work after your go-live date for extended services. 
• Expertise of the team: Anyone can build a software platform and market it to labs, but how many vendors have actually worked in a lab setting before? Look for a vendor who understands your needs because they’ve walked a mile or two in your shoes. For example, at QBench we’re proud to employ many people who have worked in your shoes and understand what lab managers and staff go through each day.

The market for lab software is broad, and you’ll find a wide range in price and complexity. If you’re just starting your search for a LIMS, here’s why modern labs choose QBench.

Unlike many industry-specific, legacy players in the space, QBench is industry-agnostic. Rather than hyper-focusing on a single set of lab workflows, QBench is built to be adaptable for a variety of needs, giving your lab full flexibility to set things up the way you want. And with a built-in QMS, you’ll get a well-rounded platform that can support your compliance needs and workflows.

Make Sure You Invest in the Right LIMS for Your Lab: Download the LIMS Buyer’s Guide

Once you’ve decided to invest in a LIMS, it’s time to make sure you invest in the right LIMS. 

There’s one small problem: which vendor(s) will you review? With so many vendors to pick from and features to consider, we created a LIMS buyer’s guide to help you make the right choice for your lab. In this guide, you will learn the following:

• The different types of LIMS available
• Key features to look out for
• A vendor comparison

And more!

Fill out the form below to get your free guide and take the first step toward automating your lab today.