Most people think that Edison invented the first light bulb.

They’re wrong.

In fact, Edison was spectacularly late to the game.

In 1878, when the 36-year-old inventor decided to focus on building a light bulb, 23 others had already invented early versions called arc lamps, some of which were being used commercially to light streets and large buildings.

So how did Edison win in such a crowded field when he was so far behind?

He and his team spent a year working day and night doing thousands of experiments. On October 21, 1879, they succeeded, creating a light bulb for everyday use in the home.

Edison would go on to pioneer five different multibillion-dollar fields with his invention factory: electricity, motion pictures, telecommunications, batteries, and sound recording. In today’s terms, you can think of Edison as Elon Musk, Jeff Bezos, and Mark Zuckerberg all rolled into one.

What was the key to Edison’s incredible success? In two words — deliberate experimentation. For Edison, building a company was synonymous with building an invention factory.

The technique is just as powerful today. “Our success at Amazon is a function of how many experiments we do per year, per month, per week, per day,” Jeff Bezos has claimed. In a recent interview, Mark Zuckerberg explained, “one of the things I’m most proud of that is really key to our success is this testing framework … At any given point in time, there isn’t just one version of Facebook running. There are probably 10,000.”

Bezos and Zuckerberg aren’t saying that experimentation is one of many strategies. They are saying it is THE strategy. In this article, you’ll see how luminaries across many fields use deliberate experimentation and how you can use it to increase your odds of success in your professional and personal life.

Why 10,000 Experiments Beat 10,000 Hours

Perhaps the most popular current success formula is the 10,000-hour rule popularized by Malcolm Gladwell. The idea is that you need 10,000 hours of deliberate practice to become a world-class performer in any field.

Research now tells us, however, that this formula is woefully inadequate to explain success, especially in the professional realm. A 2014 review of 88 previous studies found that “deliberate practice explained 26% of the variance in performance for games, 21% for music, 18% for sports, 4% for education, and less than 1% for professions. We conclude that deliberate practice is important, but not as important as has been argued.”

This chart summarizing the results should cause any ardent believer in the 10,000-hour rule to pause:

This means that deliberate practice may help you in fields that change slowly or not at all, such as music and sports. It helps you succeed when the future looks like the past, but it’s next to useless in areas that change rapidly, such as technology and business.

What Edison and others (see more examples below) teach us is that we should maximize the number of experiments, not hours. Instead of the 10,000-hour rule, we need what I call the 10,000-experiment rule.

Throughout history, the scientific method has arguably produced more human progress than any other philosophy. At the heart of the scientific method is experimentation: develop a hypothesis, perform a test to prove the hypothesis right or wrong, analyze the results, and create a new hypothesis based on what you learned. The 10,000-experiment rule takes this proven power of experimentation out of the lab and into day-to-day life.

Following the 10,000-experiment rule means starting your day with not just a to-do list but a “to-test” list like Leonardo Da Vinci. According to Walter Isaacson, one of Da Vinci’s biographers, “Every morning his life hack was: make a list of what he wants to know. Why do people yawn? What does the tongue of a woodpecker look like?”

As you go through your day, following the 10,000-experiment rule means constantly looking for opportunities to collect data rather than just doing what you need to do. It means adding a deliberate reflection process based on reviewing data before the day ends.

For example, do you want to improve your sales results by asking a new question at the end of sales calls? Now every sales call becomes an opportunity to ask that question and collect data so that you can learn how to make better sales calls in the future. Do you want to sleep better so that you can have more energy during the day? You can research all the best practices for falling asleep, turn the most compelling ones into a routine, use a sleep tracker to get objective data on the quantity and quality of your sleep, and then make adjustments to your routine to improve the results.

To achieve 10,000 hours of deliberate practice requires three hours of deliberate practice per day for 10 years. I argue that the 10,000-experiment rule is just as difficult, yet doable, requiring three experiments per day.

Why 10,000 Experiments Yield Success According To Decades Of Academic Research

If Edison’s approach is universal, you would expect it show up repeatedly among top performers. As it turns out, the academic world has been studying the phenomenon for decades, and that’s exactly what they’ve found.

Researcher Dean Keith Simonton has spent his career studying the world’s preeminent creative achievers and painstakingly dissecting their careers to find patterns. To share his findings, he has published more than 340 academic articles and 13 books, including Greatness: Who Makes History And Why and Origins of Genius: Darwinian Perspectives on Creativity.

Two fascinating insights have emerged from Simonton’s (and others’) research. The first is that most innovative ideas are generated by a small number of superstars. In any given field, the top 10% of performers produce more than 50% of breakthroughs.

Why are these superstars so much more successful? Is it because their ideas are just superior from the get go? Here’s what is really fascinating: The answer is no. The second lesson to learn from Simonton’s research is that superstars produce just as many bad ideas as everyone else — they just produce more ideas overall. Having many more ideas means they have more failures but also more hits.

“What is especially fascinating is that creative individuals are not apparently capable of improving their success rate with experience or enhanced expertise,” Simonton has written. “Creative persons, even the so-called geniuses, cannot ever foresee which of their intellectual or aesthetic creations will win acclaim.”

In other words, the key to maximizing creative success, according to the theory, is producing more experiments.

Editorial Note: For a more nuanced understanding of the Blind Variation And Selective Retention, you can read Simonton’s 2011 Academic Paper, Creativity and Discovery as Blind Variation: Campbell’s (1960) BVSR Model After the Half-Century Mark.

From Health to Stand-up Comedy: The 10,000-Experiment Rule Applies Across Fields

When you consider many of the most important achievements across different fields, you often see this theory at play.

A Fast Company article written by advertising legends Ben Clarke and Jon Bond points out that thanks to a combination of new technologies and lean business approaches, the world’s most innovative businesses are running thousands of experiments more annually:

In academia, Einstein is best known for his paper on relativity, but he published 248 other papers. Paul Erdos coauthored more than 1,500 mathematical research articles during his career. 1,500! As you might expect, Erdos made significant contributions, and although most of his papers have been forgotten, a handful of them made him one of the most influential mathematicians of the 20th century! Now consider that fewer than 1% of scientists publish a paper every year.

In the world of entertainment, SNL, one of the longest-running TV shows in history, has a grueling weekly experimentation process of brainstorming, researching, and rewriting scripts. Only a tiny percentage of sketch ideas ever air. The iconic cartoons published by The New Yorker are the result of a process in which 50+ freelancers submit up to 10 sketches each for consideration per week:

Pixar, one of the most successful movie studios in history, developed 100,000+ storyboards (i.e., step-by-step plot sequences) for the film Wall-E’s ultimate plot. 100,000!

Those who enthusiastically embrace experimentation in their personal lives tend to reap significant benefits as well. Take, for example, Shonda Rhimes, producer and writer of Grey’s Anatomy, Scandal, and other hit shows. She set up an experiment she called The Year of Yes to confront her debilitating social anxiety, limit her workaholism, and accept herself. Instead of continually saying no to social experiences, she committed to saying yes for an entire year. Among the many lessons she learned from the experience was that to know what to focus on you first need to try many things.

Entrepreneur Jia Jang took something most of us fear — rejection — and made it into an experiment with his 100 Days of Rejection project. Every day for 100 days he forced himself to do something socially awkward, where the result was likely to be rejection (i.e., asking to play soccer in someone’s backyard), all while video-recording himself. Journalist Elizabeth Gilbert quit her job and marriage and then spent a year traveling the world to discover herself. She divided the year into three experiments: eat, pray, and love. Her experience turned into a best-selling book and movie. Young entrepreneur Ari Meisel used data and experimentation to cure his Crohn’s disease, which his doctors said could not be cured.

+++++++++++++++++++++++++

Reprinted by permission of Harvard Business Review Press. Excerpted from Experimentation Works: The Surprising Power of Business Experiments. Copyright 2020 Stefan H. Thomke. All rights reserved.

Experimentation Works

Our success at Amazon is a function of how many experiments we do per year, per month, per week, per day. —JEFF BEZOS, CEO, AMAZON

Innovation is important because it drives profitable growth and creates shareholder value. But here is the dilemma: despite being awash in information coming from every direction, today’s managers operate in an uncertain world where they lack the right data to inform strategic and tactical decisions.

Consequently, for better or worse, our actions tend to rely on experience, intuition, and beliefs. But this all too often doesn’t work. And all too often, we discover that ideas that are truly innovative go against our experience and assumptions, or the conventional wisdom. Whether it’s improving customer experiences, trying out new business models, or developing new products and services, even the most experienced managers are often wrong, whether they like it or not.

Business experiments raise the innovation game dramatically.

Today, Microsoft and several other leading companies—including Amazon, Booking.com, Facebook, and Google—each conduct more than ten thousand online controlled experiments annually, which individually engage millions of users. Startups and companies without digital roots, such as Walmart, State Farm Insurance, Nike, FedEx, the New York Times Company, and the BBC, also run them regularly, though on a smaller scale. These organizations have discovered that an “experiment with everything” approach has surprisingly large payoffs.

In 2016, Jeff Bezos gave shareholders a rare insight into Amazon’s innovation engine. In his annual letter, he explained: “One area where I think we are especially distinctive is failure. I believe we are the best place in the world to fail (we have plenty of practice!), and failure and invention are inseparable twins. To invent you have to experiment, and if you know in advance that it’s going to work, it’s not an experiment. Most large organizations embrace the idea of invention, but are not willing to suffer the string of failed experiments necessary to get there.”

Bezos didn’t stop there. For him, the business logic for tolerating, even inviting, failure came from the outsized economic returns of winning. He explained why experiments have been so important to Amazon’s growth model:

“Outsized returns often come from betting against conventional wisdom, and conventional wisdom is usually right. Given a ten percent chance of a 100 times payoff, you should take that bet every time. But you’re still going to be wrong nine times out of ten. We all know that if you swing for the fences, you’re going to strike out a lot, but you’re also going to hit some home runs. The difference between baseball and business, however, is that baseball has a truncated outcome distribution. When you swing, no matter how well you connect with the ball, the most runs you can get is four. In business, every once in a while, when you step up to the plate, you can score 1,000 runs. Long-tailed distribution of returns is why it’s important to be bold.”

Business experimentation has become part and parcel of how Amazon makes decisions, through a process Bezos calls “the unnatural thing of trying to disconfirm our beliefs.” After all, humans strongly prefer evidence that confirms their preexisting beliefs. But such confirmation bias gets in the way of making decisions about innovation, an arena in which most ideas don’t work.

So it should not have come as a surprise when, partially in an effort to kick its experiments up a notch, Amazon acquired Whole Foods about fourteen months after Bezos wrote the letter. Industry observers felt that its physical supermarkets could become laboratories for radical experiments. Correspondingly, share prices of competing grocery chains plummeted after the announcement. Amazon’s reputation for fearless innovation was fueled by such radical business experiments—so-called big swings—and, equally important, the tens of thousands smaller and disciplined experiments that have led to a highly optimized user experience in its web store.

If the case for business experimentation is so compelling, then why don’t more companies conduct rigorous tests of their risky overhauls and expensive innovation proposals in order to make better decisions? Why do executives rely on hierarchy, persuasion, or PowerPoints instead of demanding that teams present experimental evidence before making business decisions?

Clearly, there are cultural obstacles that inhibit experimentation. It’s also true that managers often misuse the term, saying “We experiment” in lieu of “We are trying something new,” but without putting enough thought into the discipline and rigor needed to get useful test results. In the most egregious cases, projects or business initiatives become “experiments” after they are finished, in an effort to excuse poor execution.

Many organizations have considerable difficulty executing good experiments. Although the process of experimentation seems straightforward, it is surprisingly hard in practice, owing to myriad organizational, management, and technical challenges. Moreover, most tests of new business initiatives are too informal. They are not based on proven scientific and statistical methods, and so executives end up misinterpreting statistical noise as causation—and make bad decisions.

Companies can run good experiments by systematically following a clear set of principles.

In an ideal experiment, testers separate an independent variable (the presumed cause) from a dependent variable (the observed effect) while holding all other potential causes constant. They then manipulate the former to study changes in the latter. The manipulation, followed by careful observation and analysis, yields insight into the relationships between cause and effect, which ideally can be applied and tested in other settings.

To obtain that kind of learning—and ensure that each experiment yields better decisions—companies should ask themselves seven important questions:

• Does the experiment have a testable hypothesis?
• Have stakeholders made a commitment to abide by the results?
• Is the experiment doable?
• How can we ensure reliable results?
• Do we understand cause and effect?
• Have we gotten the most value out of the experiment?
• Are experiments really driving our decisions?

Although some of the questions seem obvious, many companies conduct tests without fully addressing them. As a result, they miss out.

As the Amazon example shows, business is of absolute importance to a company’s ability to compete. Experimentation helps us begin to answer the kinds of questions that all organizations confront, from what products to make and what customer experiences to offer, to what information we need to make those decisions.

Reprinted by permission of Harvard Business Review Press. Excerpted from Experimentation Works: The Surprising Power of Business Experiments. Copyright 2020 Stefan H. Thomke. All rights reserved.