Reproducibility is a cornerstone of science, but studies show that 50 to 90 per cent of published scientific research cannot be replicated by other scientists (Aarts A, 20151, Baker, M. 20162, Mullard, A. 20113,Freedman, LP. 20174).
Over the past decade, this phenomenon has come to be known as the “reproducibility crisis.” (Figure 1: How Cloud Connectivity Can Combat the Reproducibility Crisis)
Scientists recognize and are working to correct the problem. According to a Nature survey of over 1,500 scientists (Baker, M. 20165), 59 percent of respondents say they had implemented procedures over the past five years to improve reproducibility.
Many promising corrective measures have been proposed. Indeed, when Nature offered 11 suggestions in their survey, ranging from better training to incentivizing reproducibility studies, all 11 were met with at least two-thirds of the community’s approval (Baker, M. 20165), indicating that any and all solutions would be welcome.
But what is the best starting point for life science researchers in their quest for more verifiable science?
An easy place to start might be with the two tools most indispensable in research: the pipette and the lab notebook. What if we could enable these tools to help us achieve full traceability of our work and therefore contribute to more verifiable science? We are working on a solution that is familiar in many everyday applications, yet somewhat new to the life sciences: preserving data in the cloud.
Challenges to reproducibility: the pipette and the lab notebook
Executing experiments using well-defined laboratory protocols is the lifeblood of how research gets done. And as an indispensable laboratory instrument that is a part of almost every experimental protocol, the pipette represents one opportunity for intervention, innovation, and optimization for improved reproducibility. Yet in this vital step, there is also ample room for error: authors at the Global Biological Standards Institute estimated that laboratory protocols accounted for about a tenth of the errors that contribute to irreproducibility (Freedman, LP. 20174)(Figure 1).
A multitude of little things in the pipetting process can impact performance. Repetitive manual pipetting, and the sore thumbs that come with it, can wear down even the most diligent lab personnel, leading to pressing a pipette plunger more slowly than usual, or scraping the pipette tip along the inside of a liquid container and altering the amount of liquid pipetted (Artel6). Or a lack of training might be the culprit. Not pre-wetting a pipette tip has been shown to affect liquid volume (Wenk RE, 19747). Pipetting performance can also be impacted by environmental conditions such as temperature, humidity, and barometric pressure (Artel7). For instance, sample temperature can impact the accuracy of liquid delivered by up to four percent (Millet F, 20078).
A single unnoticed mistake can result in an incorrect volume, which can snowball and tip a study toward inaccurate results. However, errors introduced by pipetting are no one person’s fault. Margins of error exist in all tasks, including pipetting, and battling the reproducibility crisis is a matter of reducing those margins wherever possible.
The paper laboratory notebook is another ubiquitous research tool ripe for innovation, especially as an increasing amount of experimental data is being generated digitally.
According to Gilson, as much as 50 percent of a scientist’s day can be spent collecting and managing data (Figure 1). Despite the enormous effort that goes into this endeavor, research shows that with every passing year after a study is published, 17 percent of data related to that study is estimated to be lost (Vines, TH. 20138), making it more and more difficult to verify a result.
One reason data is lost is that it is dispersed around several places. Most labs use several solutions to manage their workflows, organize their work, keep experimental records, and collaborate with their team. Take, for example, a scientist trying to find an ex-colleague’s data that might be relevant to her current research. The lab notebook is stored in a box when the lab head moved from a different institution. There are digital folders on a server. Some are saved, but some are missing because the hard disk failed a few years ago. Critical observations were exchanged via email and never recorded anywhere. While unfortunate, such a situation is probably not too far from reality.
In a perfect world detailed records of every experiment would be kept, neatly organized and stacked away in a safe place where it can be easily accessed and readily understood. But as we all know, inconsistencies in record keeping are common, and often exacerbated by high workloads and the pace of routine lab work.
Finding a solution in a scientific Internet of Things
The Internet of Things (IoT) refers to a network of internet-enabled physical devices that are connected each other and to the cloud. Modernizing pipettes and lab notebooks by making them part of a ‘scientific’ IoT represents a significant opportunity to enhance reproducibility.
Researchers might be leery about whether their data will be safe, secure, and kept private if it resides in the cloud. Though a legitimate question, it should not be a concern. IoT has enjoyed widespread adoption in manufacturing, retail, transportation and other industries that value security precisely because technologies exist to protect cloud-stored information from prying eyes.
Gilson’s cloud-connected pipette, set for launch later this year, is a powerful example of IoT in the research laboratory. Gilson has developed the technology to retrofit existing pipettes that labs are already using to confer Bluetooth capabilities, thus smoothing the transition to cloud-enabled labs.
To enable the cloud, a specific button is fitted to the plunger rod of the pipette (Figure 2: the cloud-connected pipette). Protocols and data are then automatically uploaded to the cloud, and can be transferred to other cloud-enabled instruments in the laboratory workflow.
Through Gilson’s online application, researchers can view information related to execution of protocols for their corresponding projects. The software tracks, records, and relays to the cloud actions performed with connected lab instruments, such as pipettes. Traceability, an important step in improving reproducibility, will also be improved, as the Bluetooth-enabled PIPETMANÒ will allow researchers to see instrument calibration data, the volume dispensed, environmental conditions, along with other steps in the protocol. In the future, Gilson is planning to have IoT-connected pipettes with the ability to correct pipetting technique in real-time, aiding researchers in conducting accurate and reproducible experiments.
Going beyond instruments, Gilson has also partnered with sciNote, a free, open-source electronic lab notebook (ELN) to improve the other mainstay of the lab – the lab notebook. By replacing paper records with an ELN, details about the researchers’ work on projects, experiments, protocols and results can be automatically recorded, which makes the process of recordkeeping effortless (Figure 3: Creating or updating a protocol to SciNote). Traceability and audit trails are an essential part of verifiable science.
sciNote is intuitive and flexible, which means that every laboratory can use it in their specific way. The ELN enables researchers to organize their projects, invite colleagues to collaborate, set due dates and write messages to each other. Every team member has their own set of permissions. This allows team leaders to coordinate the work processes and monitor the progress in the lab. Within a project, researchers can create different experiments as well as define protocols and entire workflows. Within each protocol, researchers can write descriptions, create checklists or tables, attach numerous file types, and even open and edit Microsoft Office Online files. Sample and protocol repositories are also available.
For example, researchers will be able to use sciNote to select experiments pertaining to a specific paper and trace it back to specific protocols, results, and even the location of stored samples. They can also instantly pull together reports for entire projects and export them as printable PDF files, eliminating the need to wrangle information from traditional lab notebooks, emails, and various handwritten notes.
Integration allows users to sync information from LabHub projects (LabHub is Gilson’s cloud-based solution for managing and tracking protocols) with specific projects or tasks in sciNote.
The entire loop of laboratory information management—from planning an experiment, executing it, documenting experimental results, and generating reports—can thus be closed with cloud connected instruments and one ELN.
Long-Term Solutions for Irreproducibility
Tackling the reproducibility crisis will ultimately require action by the worldwide scientific community, and need to focus on every aspect of science. Changing the way how traditional peer review is conducted, for instance, won’t happen overnight.
Within the laboratory, however, interoperability between hardware, software and secure cloud solutions that work together to automate and optimize traditionally manual tasks will empower scientists to make an impact on reproducibility. It’s not a matter if, but when. For the public, and scientists themselves, to put their full faith in science, reproducibility must not be in doubt. The technology is now available to help them be one step closer to making this a reality.
Nicolas Paris is CEO of Gilson Inc., is a marketing data scientist and an expert in sustainable development of businesses in the lab instrumentation industry.
Klemen Zupancic, is CEO of sciNote. He is a business development professional with a Doctor of Philosophy (PhD) focused in Biomedicine – Genomics.
- Aarts, A. et. al. (2015). Estimating the reproducibility of psychological science. Science, 349(6251). doi:10.1126/science.aac4716
- Baker, M. (2016). Biotech giant publishes failures to confirm high-profile science. Nature, 530(7589), 141-141. doi:10.1038/nature.2016.19269
- Mullard, A. (2011). Reliability of ‘new drug target’ claims called into question. Nat Rev Drug Discov, 10(9), 643-644. doi:10.1038/nrd3545
- Freedman, L. P., Cockburn, I. M., & Simcoe, T. S. (2015). The Economics of Reproducibility in Preclinical Research. PLOS Biology PLoS Biol, 13(6). doi:10.1371/journal.pbio.1002165
- Baker, M. (2016). 1,500 scientists lift the lid on reproducibility, 533(7604). doi: 10.1038/533452a
- Artel blog post: “Does weather affect pipetting? YES!”
- Wenk RE. et. al. (1974). Technology of manually operated sampler pipettes. Clin Chem 20(3):320-3.
- Millet, F. et. al. (2007). Securing accuracy and precision when pipetting hot and cold liquids with Microman®. Nature Methods, 4. doi :10.1038/nmeth1086
- Vines, TH et. al. (2013). The Availability of Research Data Declines Rapidly with Article Age. Current Biology, 24(1). http://dx.doi.org/10.1016/j.cub.2013.11.014