Shaping experiments
From conceptual ideas to concrete data
There is no shortage of new ideas in science.
Humans observe the world around them. As scientists, we investigate the current state of knowledge about these observations. We develop ideas for what might explain an observation. And we design an experiment to test our idea, carry it out, and analyze the results. Finally, we assess the results in the context of the original idea and the cycle repeats. Over time, these explanations become facts (more likely to be true, but still a probability).
Making the jump from an idea to an experiment is one of the hardest parts of science. Ideas are conceptual. Experimental data—numbers, charts, figures—are concrete. How are conceptual ideas translated to concrete data? How are promising ideas distinguished from dead ends? What details are important? Which details are not? Which methods should one use?
Experimental design is a critical junction for all research projects. It is the bridge between abstract ideas and the hard-fought reality of the work. Good ideas are abundant. The availability of time and money to conduct experiments limits scientific insight.
How do we choose which ideas to pursue? How are experiments born from ideas? We developed an experimental design framework inspired by Ryan Singer's Shape-up philosophy. Framing experiments in this manner helps us identify which ideas to chase, and which to set aside. The framework relies on three levels of design with varying granularity: coarse, intermediate, and fine.
The coarse scale contours the idea from the original observations. Ideas are complex multidimensional monsters. Ironing out this complexity is critical to producing interpretable results. Our first step is to simplify these complex concepts into something tangible. We call this process Shaping.
Shaping is a process that combines our laboratory expertise with the tools and resources we have available while taking into account any limitations of time or money. It involves experimenting with different designs to turn our ideas into reality. The process is highly creative and allows us to think big to bring our ideas to life.
When Shaping a project, I sketch the project out. Because the ideas are complex and rough, and I initially want to keep them at a coarse level, I'll use a writing utensil with a wide tip. This imposes a physical limit on the amount of detail I can include in the sketch. There are wide-tipped whiteboard and paper markers. I like the flexibility of the pencil and have used a lead holder with 5.6 mm lead. Recently, I've migrated to cheap carpenter's pencils. I’m sure crayons would work well too. If you're more tech-savvy, you might use an iPad notetaking app with a wide line. These implements allow for the right amount of detail while Shaping at the coarse level.
Shaping involves mapping the experimental design, time scales, controls, and high-level data analysis. I sketch workflows. Temporal dynamics. I'll sketch examples of what the end figures might look like. I'll highlight parts that might be tricky to pull off.
The goal of Shaping is to uncover the essence of the idea. If I run out of room on the paper or the whiteboard, the scale is not coarse enough. I'll look for things to take away while getting at the core of the idea. Shaping should feel a bit unrefined when it’s finished.
An idea that has taken shape provides researchers with ample opportunity to make their mark on the project. They can rely on their experience to fill in the intermediate and fine-scale details. The complex idea is now well-defined, with clear boundaries in place. We have identified the crucial aspects of the idea and either noted or avoided them altogether. The core of the idea flows smoothly. When experimenting, new ideas and interesting avenues inevitably emerge. By laying out the broader vision ahead of time, we confine the experiment to what we’ve mapped out and limit it accordingly. This approach lets our researchers know what is in play and what they should avoid doing.
The intermediate scale adds resolution to the coarse scale. This includes important details like the number of replicates to obtain. The sources of organisms or samples. The statistical tests to use. The order of operations and timeline estimates. Who to involve and when. The essential experimental controls, and those that are nice to have. Often, I’ll get a finer pencil in a different color and overlay these details onto the coarse-scale sketch. This is when the real scope of the experiment begins to take shape.
We’ll often design a pilot experiment from these intermediate-scale experimental plans. Pilot experiments are useful if we're venturing into a new area of research. These small experiments teach us how to do the experiment before setting it up on a larger scale.
The fine-scale is the mortar holding everything together. It fills the remaining gaps and finalizes important decisions. How much substrate to add. Which protocol to use. Materials needed. Statistical packages or code to use for data analysis. The figures to generate. We rarely think of these details ahead of conducting a pilot experiment. Why? Unexpected issues always arise, and you can never cover all the bases ahead of time. Data are noisy; we need more replicates. Cells grow slower than predicted; we need to adjust the timeline. The plots conceal a data distribution you hadn't anticipated; we need different plots.
At this fine-scale resolution, decision fatigue can (and often does) set in. There are so many decisions to make, and they are often interrelated. We all want to nail it on the first try. But that rarely happens. Every decision made results in a splay of downstream what-ifs. Spot this in your thinking with thoughts like “But what if…?” “How can we know that…” “I worry about…”. The fine details are often the most overwhelming part of the experimental process. But remember, you can’t cover all bases. Ever.
Once we have all three levels filled in we’ll assess its feasibility. Often, pilot data will help us to decide what to pursue and what to shelf. Budgetary or time constraints factor in as well. We continually ask “What experimental attributes can we take away?” before the whole thing falls apart. This simplifies everything and is the antithesis of the obsession with new methods. Once we have a shaped idea with all the details filled in that’s within our scope…it’s game on!
Do you have a structured way of thinking through new ideas? How do you develop experiments? What tools or tricks do you use? Please share in the comments!
Intriguing methodology, I've never heard of this.
The shaping sketches look familiar, usually there's a "brainstorming" period I call "Ideation".
You can have all the great ideas in the world, but without purpose and a plan, they're just ideas.
Thanks for sharing this fascinating approach applied to scientific research.