Monday, August 10, 2020

Out of the Crisis #18: Sal Khan on institution building, solving the digital divide, and education as the lever for all decisions

"It was delusional for a guy operating in a walk-in closet, but I was like, 'You live once, you might as well swing for the fences.'" That's the impulse that led Sal Khan, founder of Khan Academy, to believe he might someday be able to turn the not-for-profit online learning platform into a world-shaping institution along the lines of the Smithsonian or Oxford. Even before the pandemic, it reached 20 million students all over the world every month, who engaged in 30 million minutes of learning per day. Now, with remote learning as the new norm, those numbers have tripled.

As an entrepreneur, Sal has always believed in thinking long-term. And when it comes to education, he sees the value in that approach even more clearly. As he told me, "The underlying lever for everything is education, and not enough people are thinking on large time horizons. Most people, they're not even thinking in their lifetime. There's probably things you could do in the world if you have a framework of even hundreds of years, much less thousands or tens of thousands of years."

Sal has made it Khan Academy's mission to envision a world where everyone has the ability to access free, personalized mastery learning, and he measures the value of his work purely in those terms. "I think any entrepreneur has this delusional optimism that surely the world will recognize how valuable this is," he says. "This was a social venture, but that the social return on investment is through the roof."

We talked about why he chose to make the organization not-for-profit, the duty to serve, closing the education gap, not being bound by time and space and many other things.

You can listen to our conversation on Apple, Google, or wherever you like to download podcasts.


 



There's also a full transcript beneath the show resources below.


Highlights from the show:

  • Sal introduces himself and talks about his quarantine set-up (2:44)
  • The strange silver linings of staying at home (3:45)
  • When the pandemic came on Sal and Khan Academy's radar screen (5:54)
  • Khan Academy's three pillars (7:42)
  • Coming up with Khan Academy's daily schedule as stay-at-home and work-at-home swept the country (9:02)
  • Operating as a not for profit (11:02)
  • How Sal became the founder of an education movement (13:19)
  • Being a "tiger cousin" to 12 year-old Nadia (15:33)
  • The Khan Academy MVP (17:17)
  • Starting a not-profit while working at a hedge fund, when there was massive potential for profit (20:00)
  • Thinking long-term about education (21:55)
  • The benevolent aliens helping Khan Academy (24:52)
  • What it was like when the pandemic drove up costs and also impact (26:10)
  • The duty to step up (27:11)
  • The hard work of accelerating during a pandemic (28:58)
  • How Sal thinks the education system needs to change (30:38)
  • The challenges and opportunities that have become apparent post-COVID (32:42)
  • Mastery learning and Bloom's 2 Sigma Problem (33:41)
  • Partnering with Duck Duck Moose to close the education gap (37:14)
  • What it will take to launch a new education paradigm post-pandemic (39:40)
  • Khan Academy's "Getting Ready For" programs (41:29)
  • Preventing wide-scale knowledge loss (44:13)
  • The Diamond Age (46:10)
  • Building institutions (47:58)
  • John Rawls's Veil of Ignorance (49:43)
  • Sal's advice for parents (50:59)
  • What Sal has learned during the pandemic (52:52)
  • Solving the digital divide and keeping an open mind (54:48)

Show-related resources:


Transcript for Out of the Crisis #18: Sal Khan


Eric Ries: This is Out of the Crisis. My name is Eric Ries.

Overnight, we became a nation of homeschoolers. Listen, homeschooling has its benefits. Mastery education, for example, has two sigmas better educational outcomes than teaching in a classroom, but schools provide more than just education. Parents and families rely on schools for so many social services. So, this transition has been incredibly hard for working families, especially all over the world.

Like with so many other examples, the crisis is exposing the underlying issues in our society. A quality education for our children in a safe and secure space shouldn't be impossible to achieve. It's not a pipe dream. It should be our everyday reality. We need to use this moment to reimagine our education system. Sal Khan may be the most famous educator in the world. When he first started producing online educational videos, he never intended to turn it into a business. In fact, all he was trying to do was teach his niece and nephew in a way that could supplement what they were learning at school. This initial experiment turned into Khan Academy, a nonprofit organization with the mission of providing a free, world-class education for anyone anywhere.

Khan Academy was successful before, but in the crisis, it has turned into a vital resource. Their usage jumped 250% in a matter of days, and they are now serving over 100 million students. They've launched tons of new resources in this crisis and were one of the founding members of the Schoolclosures.org Coalition. Entire districts had to switch to Khan Academy overnight. They've provided weekly math learning plans, daily schedules, webinars for teachers, webinars for parents. Five thousand parents signed up for the first one, and office hours for teachers who are navigating this new online reality.

But Sal understands that right now, these are stopgap measures. We have to view Khan Academy's experience as a case study in how to bring about systemic change. It is showing us what a new education system could look like, but this crisis is also showing us just how fragile our old systems were. So, let's not waste this opportunity. There are people in the fight right now, who can help us build more resilient, equitable and long-lasting institutions, if we let them. Here's my conversation with Sal Khan.

Sal Khan: My name is Sal Khan. I'm the Founder of Khan Academy. We're not-for-profit with the mission of providing a free, world-class education for anyone anywhere.

Eric Ries: Sal, thanks so much for making time to come on and talk to us. I want to start with just these are dark times, how are you doing? How's your family? What's your quarantine been like?

Sal Khan: Thanks for asking. This is a time where that answer of how you're doing, it's all relative, relative to folks who obviously are dealing with health issues. They're dealing with economic issues. I'm just fine. I'm  actually in the walk-in closet where Khan Academy all started, but it has a window. I'm looking at a nice garden. I have my three kids and I'm actually enjoying the time with them. It gets a little tense in the household every now and then with all of us here, but we have enough space. My wife is able to work part time and do some remote work. My mother in law lives with us. So, I feel reasonably well supported and pretty fortunate in this situation. What about yourself?

Eric Ries: It's definitely been a journey, but we're also very privileged. Our two kids have been thriving in this time. We've really been trying to look at the silver linings of what the pandemic could mean in terms of our family and the educational opportunities that it unlocks, not just what's been taken away. Our kids are young. So, it has been a challenge. I think it's a little bit strange to say that there's positive things that can come out of such a bad thing that's happening in the world. So, we've really tried to focus on that as best we can.

Sal Khan: Yeah, I mean, I actually do feel a little guilt because part of my job normally is, I have to do a reasonable amount of travel, which I really don't like. I enjoy meeting people in other cities, I hate getting on planes.

Eric Ries: I know.

Sal Khan: I've had a little pang of joy every time my trips have been canceled. So, I hate to admit that. I've been taking a lot of my Zoom meetings walking, and I've been sitting under redwood trees in our local park. So, I've been enjoying that as well.

Eric Ries: It is really wild how things that we used to think require travel and they required conferences and they required all this schlep and prep. It's like now incredibly efficient. Just hop on a call, hop on Zoom, and get it done. I remember really vividly maybe in the first week of the Shelter in Place Order, I think even before the schools had officially closed, we had a conversation. It's very vivid in my mind because it was late at night. It was still in the very early days of trying to figure out what is this pandemic going to mean for families and kids and for all of us.

Our tone was very different than where we are now. It felt like a much more of a dire situation. We were still struggling with what was going to come next. But I remember even in that very, very dark moment, you had this very positive vision of you're almost calling it like America's homeroom, a way that we could turn the pandemic into an incredible learning opportunity. Do you want to just talk a little bit about when the pandemic came onto your radar, when you realized it was going to mean a major change for you and not just for your family, but for Khan Academy?

Sal Khan: Sure, it all feels like I'm sure for everybody a lifetime ago now.

Eric Ries: Isn't it? I know.

Sal Khan: If you go back to mid-February, it sounds like the world was very different then, but it was in  mid-February that we started seeing traffic pickup in China and South Korea. Those aren't major geographies for Khan Academy, but we do have users there. There was a South Korean teacher in particular, who started emailing us telling us how he was using Khan Academy to keep his students learning through the school closures there. So, that was the first sign that it kind of dawned on us that, "Oh, we have a role to play in what's going on with this pandemic in Asia at the time." And then you fast forward a couple of weeks, I live here in Santa Clara County in Silicon Valley, which actually I think had one of the first cases or one of the-

Eric Ries: Of community spread, yeah.

Sal Khan:... first cases of community spread. So, it was on our radar pretty fast, like, "Oh, this thing is now in Santa Clara County." People here started talking about things. The first week of March, there was a local private school that had a case of someone's family member coming in touch with that first case of COVID, I believe, so they closed down the school for a couple of days. So, that was the first moment that we said, "This looks like it could be a thing in California." We're in Santa Clara County. We weren't fully thinking about California or the country at that point.

But as you went through that, I think it was either the first or second week in March, the week before you and I had our phone call, every hour, it just became more and more clear that not only Santa Clara County might close as of that Friday or that Monday. I'm on the board of my children's school. We started having a board meeting. Do we close on Thursday? Do we close on Friday? Do we wait until Monday? So, my Khan Academy has said, "It's one of those moments where you look left and you look right, you're like, 'I think this is us.'" None of us could have imagined this circumstance happening. But over the years, Khan Academy, there's been three pillars to our vision.

One is to provide access to anyone who wants to learn, who wants to tap into their potential as early as Pre-K through elementary, middle high school, and college in math and reading and writing and science. We were imagining that this would be for either...We have these stories of kids who are in Taliban-controlled, Afghanistan, and young girls were forbidden from going to school who are using Khan Academy in that way. Obviously, there's a lot of usage in classrooms in the US. So, it's kind of a hybrid learning environment. Half of our usage has always been in classrooms. So, there's always been this notion that we could help learning not be bound by time or space.

Another pillar of our vision was learning could be personalized. All kids can learn at their own time and pace, and what matters is the outcome of do they master the material or not, versus how much time they spent in a chair or where that chair might be. So, all of a sudden, we realized, "Alright, schools might close in the next few days, maybe next week." We had a lot of the materials to keep folks learning, but it's becoming clearer by the hour that most teachers, parents, students, districts were just feeling overwhelmed. They were already worried about the virus. The economic outcome was starting to become a major concern.

And then all of a sudden, you're telling all these parents that you're working from home, if you have a job that you can do from home, your kids are going to be from home. You're expected to somehow, over the next three days, come up with a homeschooling curriculum to keep the learning. So, we said, "What do they need?" First of all, can we structure all of the resources that Khan Academy already has and other things on the internet so that we can help parents and teachers get their legs under them? So, we published these daily schedules. This was really me and a couple of team members on Friday night that weekend.

Eric Ries: I remember.

Sal Khan: I think that might have been the Friday we first chatted, or we were exchanging some emails. We just said, "Okay, if you have students of this age, this is what your day could look like." It includes having your breakfast and getting some outdoor time and doing some yoga. Also, when you do your math and reading and writing. We started running webinars for teachers and parents. It was around that same time that this homeroom idea came out, where it's just like "Everyone is feeling very isolated," especially in that first weekend because it was so jarring, especially with so much uncertainty that was going to happen over the next few weeks.

So, then we said, "Wouldn't it be cool if we just did a daily homeroom where we could just talk about stuff, talk about obviously COVID, talk about education or just talk about random topics?" So, we started that. We've been doing it every day since. But then when the school closures hit... Actually, that week, we're starting to stress test our server saying, "Who knows? We might get two times the usage." But by the end of that first week, when the schools actually did start closing around the world, especially in the US, we saw our traffic at about 300% of normal, has been varying between about 200 and 300%. Ever since, the registrations have been 5 to 10X of normal depending on the day. So, it's been a wild ride.

Eric Ries: It's a total inverse of anyone who's in the travel business or in a business that is really being shuttered by the pandemic and the shutdown.

Sal Khan: Yeah, it's interesting. We've always had this not-for-profit model, which has always been a source of debate. Is this a model by which you can scale? Is this a model by which you can attract talent and capital and all of the standard things? So, you can imagine now, our traffic is 200%, 300% of normal. Our server costs, we're on track to be 200%, 300%. They're almost linear in terms of usage. So, in some ways, for sure our costs have gone up, and we're trying to fast forward all these programs to support people even more. But we love it that our costs are up, because that shows that our impact is that much higher, but it did create pressures for us to-

Eric Ries: Of course.

Sal Khan: ... how do we keep up with the costs of things?

Eric Ries: Well, we'll be sure to include a link to the donations page in our show notes in the hopes that some listeners will be able to help. I want to thank you because I remember, you were one of the very first organizations to sign on to the School Closures Coalition, as we were building the website. I think it was that very weekend when we had those late-night conversations. I remember we had volunteers manning a hotline, as parents were calling in. A lot of them panicked about what to do with their kids. We were rushing to get your sample day by age. I remember we were trying to translate those documents onto the web as fast as we could, and teach our volunteers how to use them.

What I really appreciated was the kind of almost reckless abandon with which you threw yourself into the relief effort at a time when I was still talking to a lot of corporate leaders who were saying, "Well, let's wait and see. I don't know. Maybe it will pass," or "There's not really anything that we can do. We need to wait for the government to fix it." People had a lot of different excuses for why they weren't the ones in particular that were called to do it. First of all, just thank you. But I also wonder if you could kind of go back in time now and talk about your personal journey. How did you wind up running Khan Academy? I remember reading your bio of the many, many things that you did including... Weren't you an intern at PARC? Do I remember that right?

Sal Khan: I was. Very few people know that. At Xerox PARC.

Eric Ries: Yeah, Xerox PARC. If you're in the technology business, that's like sacred ground. So, of course, it caught my eye. I remember reading that in a profile of you once. So, I just wonder if you could talk about your journey. How did you become this unusual founder of this education movement? Maybe then we can kind of connect that to how those kind of... I remember you were talking about how that founder DNA, that founder instinct got reactivated in you as you saw the pandemic crashing all around us.

Sal Khan: Yeah, I'll give the background, but I think there's a general theme, which is when you're doing anything, you have to have obviously a reasonable idea that people need. You have to be able to execute on it well, and then there's timing, but every now and then that door opens. I can't tell you how many times I see people who push against the door when it's closed, but then when the door opens, when the window opens, they're not sprinting through it. No, you’ve got to sprint through it when the world has opened the door, the world has a need that you have a chance of serving.

But if you go back, I kind of fell into this in a fairly random way. Although, if I'm honest, in the back of my mind, I always gravitated into the space of education and human potential, etc. But my original background was in technology. I was a computer science and math major in college. My first job was a Product Manager at Oracle, then I worked at a tech startup called meVC. And then when NASDAQ took a dive, I was two and a half years out of college. I said, "Well, maybe I should apply to business school and take some shelter from the carnage that was going on in the economy." So, I went to business school, and it was interesting. That PARC internship you're talking about, it was actually between my two years of business school.

Eric Ries: Really?

Sal Khan: Yeah. They were trying to figure out how to monetize a lot of the technology that they had. Could they make businesses out of it? So, I kind of spent a summer digging around, looking under rocks, thinking about what their business plan could be written for any of this stuff. But then by my second year, I had a real interest in finance. So, I ended up after business school, working as an analyst at a very small hedge fund. It was really myself and my boss, who was our portfolio manager. We were based in Boston.

And then a year out of school, I was getting married and actually in New Jersey, but my family was visiting me. I showed them around Boston, while they were up. They were coming from New Orleans, which is where I was born and raised. It just came out of conversation that my 12-year-old cousin Nadia was having trouble with math. She had trouble with unit conversion. Because of that, she had bombed a placement exam the previous year, and she was put into kind of a slower math track.

So, I told Nadia, I'm like, "I'm 100% sure you're capable of understanding unit conversion. How about when you go back to New Orleans, I'm happy to get on a phone with you and remotely tutor you if you're up for it?", and she was. So, I started doing that. We just got on the phone. We found Yahoo! Instant Messenger, which was fairly popular instant messaging at the time, had a little thing going-

Eric Ries: Now you're dating yourself a little bit.

Sal Khan: I am dating myself. There was a thing called Yahoo! Doodle where you could actually draw, and other people could see it on the other side. So, we started to draw like math with our mice. And then I got her a little $50 pen tablet, so that we could write a little bit more clearly. But that was our tutoring sessions. Slowly but surely, she got through unit conversion. She got caught up with her class. Frankly, she got a little bit ahead of her class.

At that point, I became what I call a tiger cousin. I called up her school and I said, "I really think Nadia Arman should be able to retake that placement exam from last year." They said, "Who are you?" I said, "I'm her cousin," and they let her. She went from being placed into a remedial math class into an advanced math class. So, I was hooked. It was a fun way to bond with a young cousin across the country. I loved the subject matter. It was a part of my brain that I wasn't using at work. So, I started tutoring her younger brothers. Then word spread around the family that free tutoring was going on.

Eric Ries: Oh, sure, dangerous.

Sal Khan: It was dangerous although I was enjoying it. After about a year, I had about 12, 15 cousins from around the country and a few family friends that I was tutoring. By this point, we had moved out to Northern California. The background in software, I just saw that a lot of my cousins... What was holding them back wasn't some type of an innate ability or that they didn't have good teachers or anything like that. It was more that they just had gaps in their knowledge. They were in an algebra class, but they never really mastered dividing decimals, or they never really mastered negative numbers.

So, I started writing practice software for them, so that it would generate questions for them, and then they could answer them, get immediate feedback. I could track what they were getting wrong or right on a little dashboard that I put together. That was the first Khan Academy. It had nothing to do with videos. It was practice software for my cousins to learn at their own time and pace. It was a friend... By this point, we moved to California out here. ... who said, "Well, why don't you scale your lessons by recording videos and uploading them onto YouTube for your family?" This was in the early days of YouTube. I thought, "No, it's a horrible idea."

At the time, it would be like someone telling you today to use TikTok as an education. I'd be like, "No way, TikTok, it's frivolous." That's how YouTube felt at the time. I said, "This is normally for cats playing piano," but I got over the idea that it wasn't my idea. I gave it a shot. I remember that first video, it took like 15 minutes and then YouTube said, "Nope, you can't upload. It has to be less than 10 minutes." I was like, "How can you make an educational video less than 10?" So, then I did it. I was like, "Oh, that's not so bad." And then, I told my cousins, "Watch this at your own time and pace. It supplements these exercises I've created for you. We can dig deeper on the phone."

And then after about a month, they famously gave me the somewhat backhanded feedback that they like me better on YouTube than in person. I took that as positive feedback, and I kept going. It soon became clear that people who weren't my cousins were watching and were using the software. Actually, so many people were using the software at some point that I have to just shut it down. My $30 a month web hosting was crashing because of it.

You fast forward to about 2009. I frankly had trouble focusing on my day job. There are about 50,000 to 100,000 people depending on the month using it. I was getting letters from around the world, people were saying how they were benefiting from it. In the back of my mind, I was like, "Gee, if it's reaching 50,000 or 100,000 people a day, who knows? Maybe one day, it could reach 500,000 or a million. Do I dare dream 100 million or a billion people?" So, I quit my day job. Khan Academy was set up as a nonprofit with a mission of free, world-class education for anyone anywhere. That's kind of when I tried to make it work. I think any entrepreneur has this delusional optimism that surely the world will recognize how valuable this is. This was a social venture, but that the social return on investment is through the roof.

Eric Ries: Talk about how you decided to make it a social venture. I mean, you've gone to business school. You were hanging out in Silicon Valley, I'm sure many people must have told you, "You’ve got to monetize this. You’ve got to build equity value." I'm not sure what the jargon is. I'm kind of dating myself, trying to remember what the exact jargon would have been. But I got to believe there are a lot of people around you who were excited about the for-profit potential of what you were doing.

Sal Khan: Oh, yeah. I didn't know anyone who had started not-for-profit, I'd never worked in a not-for-profit. All of my friends were either in tech, hedge funds. They were entrepreneurs or they're venture capitalists. A few of them were willing or they introduced me to people who are willing to write a quick check for me to start it as a for-profit. It was intriguing, it was tempting, because then hey, I could work on my passion full time starting tomorrow.

But by meeting two or meeting three, it got a little squirrely. Nothing wrong with for-profit, but especially in something like education where we started saying, "Oh, maybe we do a freemium model. We give this stuff away for free, but the really valuable stuff, we charged for." That just felt a little squirrely to me based on these letters I was getting from people around the world saying how valuable it was for them. For me to imagine if I put a paywall, some of those people would not be able to get that benefit.

I also saw in my hedge fund, my day job, we used to talk to five, six publicly traded companies a day. We saw that many founders had very positive intent of what they want their company to do, sometimes even have a social aspect to it. But as companies grow, as you have more shareholders, especially once they last beyond the founder's influence, you seldom actually see companies that are able to keep that focus on some type of a mission over generation. So, that was my experience in the for-profit world. I just kind of did a thought experiment. I was like, "Let me be a little bit delusional." I've always been a fan of Isaac Asimov's Foundation series where-

Eric Ries: Oh, sure.

Sal Khan:... Hari Seldon collects all of the world's knowledge at the periphery of the Galactic Empire because he saw dark ages coming. He says, "I can shorten the dark ages by having the world's knowledge at the periphery of the galaxy, and that dark ages could shorten from being 10,000 years to 1,000 years." I remember when I read that in middle school, two things really struck me. One was Hari Seldon is right. The underlying lever for everything is education, and not enough people are thinking on large time horizons. Most people, they're not even thinking in their lifetime. They're thinking next quarter, bonus cycle. I was like, "How powerful is that? There's probably things you could do in the world if you have a framework of even hundreds of years, much less thousands or tens of thousands of years."

So, my science fiction nerd hat said, "Maybe Khan Academy could be like the Foundation." Where anyone in the world, wherever you're born, if you just have a low-cost device, you could tap into your potential. It could keep us from being in a dark age or in pockets of the world where there are dark ages, it can still give kids a lifeline. If you imagine the great institutions of the world, the Smithsonian's, the Oxford's, those are all not-for-profits. They all have issues that make them maybe not perfect, but they've done a lot of good for the world.

So, I said, "Well, let me just take that bet. What if Khan Academy could be one of those one day?" It was delusional for a guy operating in a walk-in closet, but I was like, "You live once, you might as well swing for the fences." Everyone who starts a for-profit business, they all dream of being the next Google or Apple.

Eric Ries
: Why not? Why not be delusional?

Sal Khan: Yes, Smithsonian or Oxford and frankly, even at that time, even when I was operating in a walk-in closet, we were serving more kids per year than Oxford serves in their thousand-year history. So, I'm like, "It's not that crazy."

Eric Ries: It's interesting to me how often Hari Seldon comes up talking to entrepreneurs. This is not the first time Foundation and books like that are so powerful. This theme of thinking long term, of course, very near and dear to my heart. But it's precisely because when we talk to people who have that long-term vision... I mean, I was just talking to Brian Chesky. He was talking about how so many of the errors that people make when they do the wrong thing, it's not so much malice, it's wrong time horizon.

When you're trying to optimize for tomorrow or the next quarter, you miss out on these opportunities to think in a bigger and more impactful way. So, I don't think it's crazy at all. I guess maybe I've been around entrepreneurs long enough to find it to be normal. I guess you just have to hope that the mule is not coming-

Sal Khan: Or is somewhat benevolent.

Eric Ries: Yeah, yeah, exactly. So, there's some hiccups that can come.

Sal Khan: I have a narrative that I share with the team and there's whole Slack threads about whether Sal really believes it or not, but that benevolent aliens are helping Khan Academy because we've had a lot of things fall our way that have helped us in this journey. That they're helping Khan Academy because they want us to help prepare humanity for first contact.

Eric Ries: Yeah, I think I saw a movie about that once.

Sal Khan: Is there? Because I've been trying to write a screenplay. I think that's interesting.

Eric Ries: But it is interesting that that sense of serendipity that, again, is common to entrepreneurs. I know I feel this way about LTSE that there's just been these moments when the very right person arrives at the very right time through no fault or credit of my own where something that I thought was the worst thing that ever happened to us, actually secretly turns out to be the best thing that ever happened. It's kind of the curse of entrepreneurship that you can't tell good from bad since you don't know what the long-term consequences of anything are. Therefore, the only rational or psychologically sane way to approach that is through equanimity and not to get too excited or too stressed out when the good or the bad happens.

So, kind of take that mindset, you've been on this crazy journey, you had this long-term vision. It's such an unlikely success that perhaps we should attribute it to aliens. And then now you're faced with this global pandemic, which is immediately driving your costs up, immediately driving your impact up, immediately making new things possible. Just walk us through, when did that dawn on you and what was it like?

Sal Khan: Yeah, as I said, when we started seeing traffic pickup in Asia and we started having these conversations that week before schools actually closed and what do we do? I remember there's two things that our team was thinking about. One it was and I said, "We turned left, turned right. I guess this is us." But then, we were very sensitive that we shouldn't be viewed to take advantage of the situation, but then I reminded the team. I was like, "We're not a business. We have a mission of free, world-class education for anyone anywhere. We don't profit." None of us become any richer. In fact, our jobs become harder.

So, it's interesting because we have a lot of people who come from the for-profit world. You constantly have to remind, and I have to even remind myself like, "No, this is good for the world. If we step up, it's good for the world." That helps. Sometimes, I'm in negotiations with partners around revenue or something. Halfway through the negotiation, I'm like, "Look, guys, I'm not negotiating for me. I don't get a penny if you guys give us an extra dollar. This is for the kids of the world. You really got to negotiate against that?" At first, everyone kind of laughs like, "Oh, funny joke." I'm like, "No, I'm serious. This is real. I really believe this." It helps.

But it's the same thing, I told our team, "No, not only would we not be taking advantage, it's our duty to step up. We're a not-for-profit." Government is... They're caught flat footed and not to blame them. No one could have been able to move fast enough in this world. For-profits, their model almost doesn't work in this world. A lot of them were and are doing the right thing, making their products free and whatever else, but even that, how long can they do that? A lot of people are going to be suspicious of it. Is it going to be a bait and switch if a district moves on to that? Frankly, there's just trust issues. What are they going to do with our data? What's the quality? So, it's our role as a trusted not-for-profit that's highly accessible to step up.

And then the other issue was, how much can we step up? Because we were already running at a deficit before. We knew that it would cost more if we wanted to accelerate. I just told our team and this was a little bit of my mindset when I quit my job in the early days and back in 2009 was, "Look, this is the window where we've got to step up, it's our duty. Let's do all we can and then hope that the universe or the aliens or whoever conspires to make us whole or ensure that we can keep doing the work we're doing." So that's the bet that we decided to take a couple of months ago.

Eric Ries: How's it going so far?

Sal Khan: I don't regret making that decision because it has accelerated our team's ability to do a lot of things that I thought we were going to end up doing in six months or a year. We pushed it forward and there's cost. You can imagine, it's also been hard on the team. The team's been energized and excited to have a role to play in this global pandemic. They're excited that our resources are two to three times as impactful as they already were. They were already serving 20 million students every month. They were already serving about 30 million learning minutes per day, but now we're serving 80, 90 million learning minutes per day.

So, everyone was excited, but at the same time, you could imagine now everyone's working from home. People had small kids at home. They're worried about their health. So, it was stressful. So, we've had to balance that and ensure that everyone knows that this is kind of a marathon with a few sprints in between.

People have come out of the woodwork to help support us. We're continuing to run at a deficit, but we've had corporations step up when they found out. Definitely people are taking our phone call. If anything, a lot of people are asking us to think even bigger about how we can support folks, not just through the end of the school year, not just through the summer, but back to school. How can we even reimagine kind of the architecture of education over the next few years?

Eric Ries: This seems like one of those moments, once in a generation opportunity to do that kind of rethink. I certainly know since having kids myself, I've become very radicalized on the need for reforming education. But talk a little bit about what you see as the changes to the education system that we need to make both right now and as we kind of get back into the new normal. If you could wave a magic wand, what do you think needs to be different about education?

Sal Khan: Yeah, even pre-COVID, I would have said, access is a major issue. Obviously, there's internet and device access. But even when kids have that, access to truly rigorous, high quality courses. It's well documented. Large chunks of the world and even in the US, majority of minority and majority schools don't offer courses that we would assume are table stakes. Sometimes, they don't offer algebra two or physics or biology, much less than calculus or statistics. That's a majority of minority and majority schools.

And then even when they are offered at a lot of schools, they're not at the same standard. It's well documented that in the American school system, kids go through the motions of taking algebra and geometry and trig, and oftentimes calculus. And they get to college or community college, and large chunks of them, 70% of all kids going to community college, these are kids who graduated who are trying to do the right thing, 70% have to take remedial math, which is the equivalent of about seventh grade math.

Which tells you that even though they nominally had access to courses, those courses really weren't serving them; or kind of the second point, because they move forward at a fixed pace, the kids were kind of just promoted, pushed ahead with significant gaps that essentially stunted their learning. So, the second pillar... Access is number one. The second one is personalization--that we want to create a world where every student can learn at their own time and pace. Every teacher will tell you that when kids show up the first year, oftentimes, they have the standardized test scores.

They know that kids are all over the place. They know that some are three years behind. Some are two years behind. Some are one year behind. Some are ready for grade level, and some are ready for one or two years ahead of grade level. How does a teacher address that huge variance and those different gaps that are unique for every student? Well, that's where technology could be helpful to let the teacher personalize, allow students to work at their individual paces. And then the teacher gets data, so they can do more focused interventions to unblock kids wherever they might be blocked. That was pre-COVID.

Post-COVID, there's this extra thing, which actually was always kind of true. It goes to that access point, is that learning fundamentally does need to not be bound by time and space. We know that obviously, it's been weird right now. It's going to be weird through the summer. This coming back to school is going to be weird in terms of you might go back to school, but you might have to shut down pretty quickly. You might have some type of shift-based architecture. You might have situations where you start, but families just don't feel comfortable sending their kids because of health issues. They might have an elderly grandparent at home or someone's immunocompromised.

In all circumstances for the next 12 months, we cannot assume that learning is bound by time or space, which is frankly something that we should never have assumed. Right now, we have the technology to do it. So, the way we're thinking about it is can we create tools that are valuable even in "normal times" that allow for the personalization, allow for the rigorous access to coursework and skills and concepts?

But at the same time, if schools get closed or weird, you can lean much more heavily on them. You can keep learning through summers. You can keep learning through breaks. You can keep learning from home. That everyone in the chain has data to understand what's working and what's not working. That kind of ties into another concept that it has been trendy, but it hasn't gone mainstream in education, which is mastery learning or competency-based learning.

Eric Ries: I was actually just about to ask you about that.

Sal Khan:We're on the same wavelength.

Eric Ries: Oh, my goodness. Yeah. I was just about to ask you about Bloom's 2 Sigma Problem.

Sal Khan: Yeah. So, Bloom in... I think it was 1984. He published this 2 Sigma Problem, which was that he showed that if a student who is operating essentially at the 50th percentile... Although he speaks in terms of standard deviations, just imagining percentiles makes it a little bit easier. If a student was operating at the 50th percentile, if they're able to get a personal tutor and the value of the personal tutors that the personal tutor can understand that student's unique gaps and pace. With that personal tutor, they work in a mastery learning framework, which means if I'm your tutor, we go through, let's say, factoring quadratics.

If you only understand 80% of it, in a traditional model, non-mastery based, I would just give you a C and we would move on to the next concept. That's what happens in traditional classes. But in a mastery framework, I'm your tutor. I'm like, "No, look, it's not okay to just know 80%. Let's keep working on it. We'll keep working until you get to 90, 95%." Bloom, his threshold was 90%. And then we move on to the next.

Eric Ries: You don't move on to the next concept until you're proficient in the previous concept.

Sal Khan: Exactly. You could even loosen a little bit where maybe sometimes it is okay to move on, but you are always motivated or incented or have the supports to go back and up your previous gaps. So, either way, you're always motivated to not just stay in 80% or 70%. You should always be filling those gaps and topping them off to 90%, 95%. He showed in that earlier research, and it was in a narrow domain that was very easy to test and kind of a multiple-choice type of test. He had a two standard deviation improvement.

So that takes a student in the 50th percentile to like the 95th plus percentile, which is kind of a mind-blowing result. Because in most of education research, even a 0.2 standard deviation over one year from an intervention would be a great outcome. He was talking about two standard deviations, so 10 times as much. Since then over the last 30 or so years, there's been many studies that got various outcomes. Bloom also saw with kind of an intelligent tutor automated system and this is back in 1984, that he was able to get a one standard deviation improvement, which is still huge.

Eric Ries: Still massive.

Sal Khan: Over the years, people have done it in different contexts, some with live tutors, some with kind of online systems, but they've been able to show that look, if you allow people to learn at their own time and pace, fill in their gaps, give them immediate feedback, that you're seeing anywhere between a 0.2 and a 2 standard deviation improvement, even 0.2 is actually quite good. That could take a student from the 50 to 58th percentile in the year. So, you can imagine if you do that for 12 years, that student is actually capturing years of work.

Eric Ries: You can't overstate how powerful of an intervention it would be if we actually could get it widely deployed. Yeah, but most of us are not good at reasoning statistically, ironically, because of the math education that we had. Sometimes I think it's easy for people to be like, "Well, what's sigma here, sigma there? What does it matter?" This would be the difference between every child being dramatically smarter overnight.

Sal Khan: Yeah. We just did an efficacy study with our Khan Academy Kids product, which is free, non-commercial. That was fascinating example of the benevolent aliens. This was a for-profit company. Duck Duck Moose I've always said makes the best early learning apps. They showed up at our door one day saying they wanted to donate their company to Khan Academy.

Eric Ries: Wow.

Sal Khan: I was like, "Where does that happen?" And then we said, "We should just create kind of Pre-K, Kindergarten, First Grade in a box that goes with reading, writing, math, social emotional learning." It's essentially like 100 apps in 1 with 100 of books in it, all of it-

Eric Ries: Our son loves it.

Sal Khan: Yeah, it's all free, not commercial. We did an efficacy study. This is a randomized control trial, which is kind of the gold standard. Where the control group, these were low income children, who, their average family income was $25,000 a year. These kids were operating at the 30th percentile on average. While obviously, the average is 50th percentile. With just six weeks of 20 minutes per day, that gap was completely closed between these kids who live in poverty and your and my kids. That's six weeks. They saw, I believe, is a 0.7 standard deviation-

Eric Ries: Unbelievable.

Sal Khan:... in six weeks from 20 minutes a day. So, you can imagine, traditionally, these kids are showing up already at kindergarten a year or two behind your and my kids. So, immediately their confidence gets shot. People's biases get built against them. It's only going to get worse from there because the whole class is going to have to move at a fixed pace. So, yeah, obviously, I'm a big believer in this and we're seeing the-

Eric Ries: Here's the thing I want to ask you about though, because one of the things I find so shocking, I've, like I said, become very radicalized by my own kids' experience of education, but I keep reading books and articles about education reform or mastery learning. So, much of the literature, it's considered a failed result. Not because it's not true, it's obviously been well replicated. This is like a 50 year old finding now, but because no one's been able to scale it to a classroom setting, because the amount of time, effort, and energy for a teacher to provide mastery type education to 30 children at same time is not physically or emotionally possible.

Therefore, people conclude that there must be something wrong with mastery learning or it's only for the few. All I can think every time I read that, it's like, "Maybe the problem is the classroom setting itself, not mastery learning." So, talk about what would it take? If this is going to be a reset moment for many of our institutions, what would it take to really come into the new normal on the other side of this with a new education paradigm that actually could provide this kind of improvement?

Sal Khan: Yeah. For me, it's not a problem with even the physical classroom. I think there's huge value in that. I would want your and my children to have a physical classroom experience, but it's really the incentive structures and the norms that sometimes there's a lot of inertia to them. You're absolutely right. Actually, even well before Benjamin Bloom, 100 years ago in Winnetka, Illinois, they did the Winnetka Plan where they did essentially mastery learning. They saw off the charts results, but they did it with like worksheets, which is logistically complex. So, they decided to discontinue it despite the results, because to your point, it was too logistically complex. It was hard to support, but that's what software is good for.

So, there's a teacher in Hesperia, California. He serves 90% free and reduced lunch to kids, 90% of them are not just one grade level behind, three grade levels behind when they show up in the sixth-grade class. He has all the kids starting from 1 + 1 on Khan Academy. If they know it, they get through it in minutes. But then they go concept by concept and in a mastery learning framework. By the end of the year, 90% of them are at least a grade level ahead, at least a grade level ahead. Two of them are even two or three grade levels ahead. This is a group of kids that most people would say... They kind of give up on. So, it's well documented.

What I've been advocating, especially through this crisis, I actually just got off the phone with a legislator in a state. I was like, "Look, we want to be there for your schools. There's going to be a crisis when you come back to school this year. Some kids would have kept learning on platforms like Khan Academy, other kids would have not. So, whatever was normally a summer slide, which only exacerbates variants and inequity, it's going to be that much larger."

So we're creating these Getting Ready For programs, like Getting Ready For grade level, which essentially allows you to replicate what Tim Vanderburgh does in Hesperia, California, which is like all of the essential skills, if you're a sixth grader to get ready for sixth grade as quickly as possible, accelerate you through them, if you already know them. And then you can work on sixth grade at your own time and pace.

We've been advocating with school districts, like "Look, that first week or two of school, a lot of school districts are even thinking about coming early, focus on that. Get every kid at the start and then work at their own time and pace in those first two weeks, so that they can master every gap that they've ever had. If they can do that, not only will it mitigate some of the damage from COVID, but they'll actually might be more prepared than they would have been even normally. And then continue with that self-paced learning."

There's an open mindedness to it right now is that every state has their credit requirements. It's often driven by the state university system. In California, it's the A-G requirements. In New York, it's the Regents system, where you need four years of math. You need three years of English. You need two years of foreign language, etc. I've been saying, "Look, kids are going to be learning at home, at school. They're going to be moving around. Credit recovery is going to be a huge thing. What if a student could get their credit that they need for graduation through mastery on Khan Academy?" The eeacher just vouches, "Yeah, that 90% that Eric got, that was his work." It's very hard to cheat. Someone would have to cheat for you for 100 hours.

And then if the teacher just proctors like our course challenge on it, which is kind of a little summative test, you can take as many times as you want, because it gives you different items every time, then yeah, that's Eric's performance. That wouldn't be a substitute for the in-person classroom, because kids will need the supports to be able to get through that. We obviously have supports on the platform. But the idea is when the teacher helps, unblock them. The teacher can put them in small groups, you can have peer-to-peer support. That's the value of the classroom.

And then the teachers have a true north. They're not saying, "Oh, I just need to cover the material," which is what usually they're told. You have 180 school days, you have 60 standards. Cover them one every three days, even if half the kids don't understand it. Now, it's just like, "Get them to mastery." If a kid by the end of the year only has 60% mastery or 70% mastery, that's okay. It doesn't mean they're a failure.

Eric Ries: At least they have some mastery.

Sal Khan: They can keep working on it. They can work it on that summer. You shouldn't just promote them to the next class. If they're only 50% mastery in algebra two, there's no point in taking precalculus, and then calculus. And then they go to community college and the community college says, "You're not even ready to take algebra two. You should let them keep working on algebra two."

Eric Ries: I saw a study, and we'll find a link and put it in the notes here, saying that kids are going to lose a year of math, because of the closure if we just extrapolate from the basic trends. That's assuming that the schools do reopen when people think. Whereas it seems to me based on the research I've been looking at, we're still pretty optimistic about what's going to be possible for schools next year. So, how do we prevent just this wide scale knowledge loss?

Sal Khan: Yeah. So, I'm familiar with the same research you're talking about. It's actually our partners, NWEA that published their assessment. Yeah, you'd only see the summer slide and now you're going to have five, six months. It's going to be that much worse than it. So, not only are you not learning for those five, six months for a lot of students, but they're forgetting, so that's why it turns into a year. The ideal obviously, if kids can keep learning on Khan Academy through the end of the school year through the summers, but we know as many kids who are using us, 20, 30, 40 million now, that's still not all of the kids. They're not using us at the same level as they could or should to truly keep learning.

And that level, especially if you're not going to school right now, is 20 to 40 minutes a day. Students able to do that, especially in math, not only will they be able to keep up with where they should be, I think they're actually going to accelerate. So, we're trying to get that word out. But as much as possible for this coming back to school, that's why we're creating these Getting Ready For courses to quickly fill in kids' gaps and get them to a grade level, give teachers a sense of where they are, and that they can keep leveraging that tool to then get into grade level.

If school has to close, if it's shift based, whatever, just lean as heavy as you need to lean on Khan Academy. We're a supplemental resource, but we have all the standards in it because we've always wanted to support the homeschooling students, the girl in Afghanistan who needs to learn everything from scratch. So, in that way, we're a very--a supplement is one way to think about it.

Eric Ries: It's almost like The Diamond Age.

Sal Khan: That's another book. So, in the early days, everyone would read Foundation and The Diamond Age. So, Diamond Age, for those who don't know, it sounds like you do know. It takes place in this like Neo-Victorian dystopian future, near future where-

Eric Ries: We're getting near every day sadly.

Sal Khan: Where this noble person wants his granddaughter to be educated by creating a tablet app that teaches her everything that she needs to be a fully empowered member of nobility. It gets bootlegged in China. It all takes place in China, and these 200,000 orphans who live on barges are able to leverage this app to essentially foment a revolution and takeover. That's another. It talks about the power of education. That one, I stopped recommending, because it has some weird stuff in it like kind of underwater explicit things. There's weirdness in that book, but the idea that "Hey, if you can get this type of thing in the hands of young people, it gives them at least a lifeline."

Eric Ries: Foundation is a little more plain vanilla.

Sal Khan: Foundation is a little bit more plain vanilla. Exactly.

Eric Ries: But I do think this kind of theme of thinking big and really trying to imagine not the world as it is right now, but as it could be, that's been coming up again and again in these conversations. There's something about the pandemic that is making it easier for people to reexamine or question. Yet, a number of folks that I've been talking to, executives, CEOs, I've been urging folks to get involved. They have this kind of sense of hopelessness or futility, like everything we're talking about seems kind of small, even this. It's like okay, some kids are learning some extra math on videos, how does that compare to the scale of this disastrous thing which has befallen us?

I really like something that you said earlier about really trying to see Khan Academy not as a successful startup or even as the next great tech company, but as the next Oxford or the next really lasting institution. I think it's hard for us to remember that those institutions, powerful and indomitable as they seem today, were not like that on their first day. The people who are building them didn't know exactly what it was going to become. They didn't see the future. So, many of the seeds that were planting right now, on the other side may yet become the lasting institutions that our grandchildren thank us for having created.

Sal Khan: Yeah. I think it's all about having that point of view of institutional building, because you're absolutely right. I mean, for sure, 10 years into the Smithsonian or Oxford, they were not how you would imagine today. Even today, you view these as these pillars of our society type of things. I mean, these are all institutions that I love and think very highly of. The world needs more museums and libraries and great universities, but the scale of what we can do even now is kind of the scale of all of those things combined to times 10 or 100.

I feel a lot less delusional talking about creating an institution that could serve a billion or one day five billion, serve everyone, and that--who knows what happens after we die? But if you imagine that you're reincarnated, you're less likely to be reincarnated in our homes, you're more likely to be reincarnated in a village in India or Africa or China with not a lot of opportunity. If future-state you at least just has a low-cost device or something and you discover Khan Academy, you will have a lifeline. You will have a way to improve yourself. You will have a way to hopefully even prove what you know and get degrees and plug into society.

Eric Ries: The veil of ignorance of John Rawls, we'll put a link to that too.

Sal Khan: Exactly. If we can unveil the veil of ignorance, I'm pretty confident we'll be kind of in a brave new world. I mean, we're working on this good, brave new world, not the dystopian brave.

Eric Ries: I was just talking to another CEO about that we're all starting to get a crash course in the power of exponential growth for good or for ill. So, obviously, the pandemic is exponential growth gone awry, like a cancer or a Ponzi scheme. But many of the reforms to create a new institution, to educate a billion children, we're going to have to harness the power of exponential growth.

Sal Khan: Yeah, absolutely.

Eric Ries: Let's switch topics for a second because you were talking before about parents and the stress, even your own team has been under, all of us have been feeling this. Parents are already being taxed to an extreme level with everything that's going on. For a lot of folks, they're being asked to work from home and school from home maybe for the first time. So, what can parents be doing? Let's say there's a parent who's listening now and they buy into your thesis of mastery learning or they're intrigued by that possibility that even 20 minutes a day might make a big difference. What advice would you give them about how to educate their children?

Sal Khan: What I've been telling parents, including myself and my wife is, number one, take care of yourself. It's so overwhelming. It's very easy to imagine that everyone else's family has it all figured out, and everyone else's kids are perfectly well behaved and learning etc, etc. So, what I've been telling everyone including myself is "Look, if your child is able to even do 20, 30 minutes a day of reading, of writing, of math, they're actually going to be just fine." That's actually a great habit, a great muscle that you could build, not just through the COVID closures, but every summer, every holiday. Any habit that you do 20, 30 minutes a day, that's how you really build it into a competency.

So, on top of that, if you can scaffold other activities, that's great, but that should hopefully lower your stress level and hopefully lower the stress level in the house. Because I also tell myself and everyone else, "We all can feel guilty about our kids not getting everything or whether we're doing everything we can for them, but the worst thing we can do for our kids is be stressed out." Because if we're stressed out ourselves, we'll snap at them. They'll feel the stress. I'm sure they're feeling anxiety themselves. So, if we're calm, we're relaxed, we're in a good place.

Obviously, there's a lot that's negative about this, but there are silver linings. This is a time to get to know our family better, spend time with our kids. Kids grow up fast, we're going to, in some ways, reminisce about some of this time when we were all in the home together for many months. So, as well as we can, take care of ourselves. There are supports like Khan Academy to keep your kids learning. It can be as little as 20, 30, 40 minutes a day. Appreciate some of the small silver linings that we are able to get out of this.

Eric Ries: So, what have you learned during the crisis? What's been the most surprising to you?

Sal Khan: I don't know if it's surprising, but I've learned that at times of crisis, for the most part, bring out the best and most of the people I interact with. This goes into government leaders, school districts, everyone that I have talked to, philanthropists, is just trying to do what's right by students to optimize learning. People haven't been trying to take advantage of this. So, that's just been a very heartening notion. I think the open mindedness, the entrepreneurial mentality that you're seeing, even in some institutions that you don't normally associate with that.

We're seeing large school districts, places like Las Vegas, Miami, New York, literally in weeks, closing the digital divide, distributing hundreds of thousands of laptops, just making things work. Things that you normally associate with scrappy Silicon Valley entrepreneurship, people are doing it on the scale of school districts with 300, 400, 500,000, a million students. So, that's been really heartening and inspiring. It gives me more energy to say, "Look, this is a little bit of a marathon. It's like a sprint marathon. It's more of like a mile race type of thing, but there's a window where groups like Khan Academy step up. Because if we do, there's an openness to work with us."

Eric Ries: So finally, how do we get out of the crisis?

Sal Khan: Well, we could talk another hour about how to get out of the crisis. I mean, we could obviously talk about the biology of it. I made a few videos on the biology of COVID and the epidemiology of it.

Eric Ries: I'll be sure to link to that.

Sal Khan: That was early days when we were much more in panic mode, but it is intriguing to see what's going to happen now that things are going to loosen up and with schools and all of that. We've talked about that. This could be a 6- to 18-month saga that we go through just on the buyer side. On the economic side, it could be longer. But I think a lot of what we talked about, which is we create more and better supports, a lot of those are going to be online because they're not bound by time and space.

One of the silver linings is a lot of energy behind solving the digital divide in much of the world, but especially the United States right now, we keep an open mind. We don't throw out the baby with the bathwater. That there is a huge value in the physical experience, but there's going to be new models of hybrid, where you can anchor on the virtual. But then when the physical is available, you can get the benefits of that. But then when you have to shut it down, you can be in between. So, I think that's what's going to keep us 6, 12 months, maybe longer.

Eric Ries: So, thank you so much for taking time and for all of your work on behalf of kids and learners everywhere.

Sal Khan: Great. Thanks for having me.

Eric Ries: This has been Out of the Crisis. I'm Eric Ries. Out of the Crisis is produced by Ben Ehrlich and edited by Jacob Tender and Shawn Maguire. Music composed and performed by Cody Martin. Hosted on Breaker. For more information on the COVID-19 crisis and ways you can help, visit helpwithcovid.com. If you're working on a project related to the pandemic, please reach out to me on Twitter. I'm @E-R-I-C-R-I-E-S. Thanks for listening.









Tuesday, August 4, 2020

Out of the Crisis #17: Max Henderson on Covid Act Now, exponential growth, and how to help

From the very start, the coronavirus has been a fast-moving target. The mechanisms of where and how it's moved through the country have made managing the spread challenging. Max Henderson founded Covid Act Now to help with that process by making data-driven recommendations for sheltering in place (and later, for reopening) combined with public advocacy guidance to get local government to pay attention to the real facts about spread and safety. As he told me, "The cognitive load of just understanding what the heck is going on and where we stand and what's coming next is extremely, extremely difficult." The work of Covid Act Now has made it the definitive resource for getting cities and states to take action against what he thinks of as our common enemy--the coronavirus.

As the son of immigrants from Cuba and Germany, Max has deep faith in the power of American ingenuity. All of his work is designed to help it thrive. "From the very beginning," he explained, "our goal has always been to, one, be extremely clear, two, be extremely action-oriented, and three, be additive."

We talked about contributing solutions as an antidote to despair, being the target of a disinformation campaign as a result of putting real numbers about exponential growth into the public sphere, collaborating with experts, and how not to lose sight of the big picture even when your head is in the details. 


You can listen to my conversation with Max Henderson on Apple, Google, or wherever you like to download podcasts.


 


In addition, there's a full transcript of the show below.


 Highlights from the show:

  • Max introduces himself (2:43)
  • Grieving during COVID-19 (3:59)
  • Feeling powerless to help others and what he did (5:18)
  • On how contributing eases despair (8:05)
  • How he started Covid Act Now (10:52)
  • Description of Covid Act Now (12:00)
  • The Covid Act Now origin story (13:00)
  • How Max modeled Covid in January (17:09)
  • The danger of not understanding exponential growth (17:47)
  • Motivated reasoning and stay at home fatigue (22:01)
  • On being the target of a disinformation campaign (24:21)
  • The power of making a personal connection over the facts (28:44)
  • How we know the models are accurate (29:45)
  • The original model showing the effects of shelter-in-place and New York as example (31:53) 
  • The difficulties of managing micro knowledge versus macro knowledge about the virus (34:30)
  • Why Max made advocacy and action part of his work from the start (37:30)
  • The importance of public opinion and political air cover (39:35)
  • Covid Act Now's Resistbot campaign (41:18)
  • On American ingenuity and being the child of Cuban and German immigrants (43:47)
  • On getting the model right and not crowding out experts (46:37)
  • Transitioning from models to metrics and what the data shows (50:26)
  • Sitting in the eye of the storm (52:40)
  • Management and moving forward (54:01)

 

Show-related resources:



Transcript for Out of the Crisis #16, Max Henderson and Covid Act Now


Eric Ries: This is Out of the Crisis. My name is Eric Ries. Are we really ready to reopen here in the U.S.? I don't think so. It's almost like we've forgotten how scary, dangerous, or impossible this all seemed in the early days.

One of the recurring themes in these conversations is that exponential growth is hard to understand. We're not hardwired for it as human beings. Remember when the numbers were small and people dismissed a threat? Remember how quickly things went from okay to catastrophic? We are making the same mistakes as before, looking at our lower numbers and concluding that the virus is gone and everything will be okay.

But it doesn't have to be this way. We made drastic and radical changes before and we could do it again. Remember when shelter-in-place seemed like an impossible or radical idea? Remember how quickly it went from inconceivable to inevitable?

Max Henderson is a technologist who founded a group called Covid Act Now. They had simple calculators so that anyone could see the humanitarian toll of even a single day of their life. As the name suggests, Covid Act Now is not just a model, it's not just about data. It's about the need for urgent action in an exponential crisis.

They had an incredibly effective pressure campaign that helped ordinary citizens write their Mayor, write their governor with simple metrics like, "If we don't shut down, this many people will die." They had calculators so you could see how every day of delay could cost lives. That pressure campaign was a necessary counterweight to the lobbying that many public officials were receiving about the need to keep the economy open because the economic costs were going to be high. But what Max and his team understood is that there is no economy if the people who power it are sick and dying.

Through public advocacy and an unyielding commitment to data-driven recommendations, Covid Act Now became the definitive resource for getting cities and states to shelter-in-place. Now, they're turning their attention to modeling the reopening. And what have they found? Check the data for yourself. We are not anywhere close to being ready.

Understanding exponential growth is hard, but we're going to all have to get good at it for all our sakes. Here is my conversation with Max Henderson.

Max Henderson: Hey, my name is Max Henderson and I'm the CEO and founder of Covid Act Now. Prior to Covid Act Now, I was at Google and Firebase as a senior data science and Go-To-Market leader.

Eric Ries: Max. I really want to thank you for taking time to come talk to us during what has been a really challenging time for so many people. Before we get to Covid Act Now and, of course, action has been your watchword through this whole crisis, how are you doing? How's your family? What's your quarantine setup like?

Max Henderson: Yeah, of course, thank you so much for having me, Eric. It's been an interesting time. I and my family are safe. Thank God. We are-

Eric Ries: Glad to hear that.

Max Henderson:... distributed around the country and the world. And so, I think what's missing the most is just not being able to be together during a hard time like this, and I know lots of people are going through the same thing. I actually had a death in my family during this period, and it really highlighted-

Eric Ries: Oh, I'm so sorry.

Max Henderson: Thank you. It really highlighted the fact that you can't even get together to mourn. And so, it's certainly been real. I thankfully have... Me and mine are safe, and we are certainly in a more privileged position than most. So, despite all that, I'm very thankful for the opportunity to be able to have what I have during this.

Eric Ries: Do you mind saying a little bit about how you've been managing that grieving process when we can't use the normal tools of grief and the normal rituals that help people get through a hard time?

Max Henderson: Yeah, of course. It feels post-apocalyptic, right? I lost my uncle. He died suddenly in Germany. And my family there had to essentially drive through a border control that technically wasn't letting anybody across state lines, right? So, my other aunt who lives in another place had to literally drive to a checkpoint and explain why she should be let through when the borders were closed, to be able to go and grieve. Most of us who are not within driving distance had no chance of being able to get there.

The process has been extremely challenging and interesting. You have conversations on the phone that you would want to have in person. You can't hug a person. There are no words. I'm so thankful for being able to contribute in some way because it does give me a sense of galvanization and a way of channeling that energy into trying to help others and just move the effort forward. I think if I had less to do and less ways of being additional or additive that it would have made the situation a lot harder.

Eric Ries: I want to dig into that. What's been the hardest thing for you while we've all been in quarantine during the shelter-in-place?

Max Henderson: Hardest thing? That's a great question. I think--I'm very blessed. I was able to bring some friends on quarantine with me and so, I've, at least, had some sense of community during this and-

Eric Ries: That's great.

Max Henderson:... being busy. My family and my significant other have had other folks to interact with so that they're effectively, to some degree, mourning the loss of me because I'm constantly working on other things and they support me a ton in that, so I'm very, very blessed.

I think the hardest thing has just been the desire to want to help others. I realized, this is a bit perhaps of a cop-out answer given the fact that I have found something to channel my energy into, to me being in control in a crisis-response situation has added so much psychological safety that the personal elements of being in quarantine have really fallen into the background compared to just the urgency of being able to execute on all these things we want to do.

And so, the reality is that, for me, and I don't know if this is a particularly satisfying answer to this question, but I've received so much from this opportunity to serve in the sense that it's allowed me to take the feelings that I feel and I'm sure everyone feels around lack of control, lack of clarity of the future, lack of an opportunity to participate and channel those into productive things. It's just helped my psychology so much compared to the way I know I would feel if I was just sitting on the couch wondering what's going on and what I can do and just feeling completely powerless.

Eric Ries: I'm glad you said that because it's actually a pretty consistent theme among the folks that I have talked to, who are in the fight trying to make a difference. And look, I've had my moments where I envy the people who were bored during the crisis and wonder what it would be like if I was running out of Netflix shows to watch. But actually, having now talked to a number of those people and convinced them to volunteer and to help out, it seems like it's paradoxical to me being busy and being exposed to these dark facts, especially with the work that you've been doing, having a real sense of the danger of the epidemic and the scale of it. I think there is a psychological security in it.

It's just one of those really clear moments where the people who are being the most of service, they're not actually... It's not an act of self-sacrifice. It's actually psychologically very healthy to feel like there's something you can do in a situation that for most people, there really isn't.

Max Henderson: Yeah, 100%. They say that depression is fundamentally the feeling of being unable to control your own destiny, right? And that feeling of despair is something that I certainly have had less of because I've been able to participate in this way. And to be clear, I mean, there are days where I wake up and I'm like, "Wow, another 18-hour day. I'm exhausted." The work is extremely galvanizing. And it's rare that there is an opportunity, there is something that is so pressing and emergent and relevant that is also so hard, so operationally hard, so difficult from a data perspective.

This is one of those things that no matter how deep you dig, it is truly like a problem unknowable by one person. It just grows fractally in complexity, like the data is bad, our understanding of the disease is bad, human psychology and our ability to stay in these really difficult interventions is a complex problem. And so, there's so much complexity here that you can really dive into it. And I think that has had an incredibly positive effect paradoxically, on my psyche, totally.

And it doesn't surprise me at all that other people have had the same experience. It's a lot like giving, where you think giving is going to be a thing that only benefits the receiver, but ultimately, it benefits the giver as much as the receiver.

Eric Ries: Or maybe even more.

Max Henderson: I had a very similar experience. Or maybe even more, yeah, that's right.

Eric Ries: Yeah, that's certainly been my experience. I'm incredibly grateful for the opportunities to serve even when they've been hard. And I know there's some people listening who are like, "Oh, come on," but just you'll have to take these... Here's two testimonials that that's been our experience during a time when I honestly think despair is the real enemy even more than the virus. Because we can outsmart the virus, we have the tools and the technology and the science.

But the question is, will we have the will? Will we have the ability, the social cohesion to actually take the actions that are needed? And that's what I really appreciated about Covid Act Now and the work that you've done, is you're fighting that. That's our real enemy.

Max Henderson: Yeah, that's right. I mean, people might say like, "Oh, gosh, get out of here. You're being so holier than now," but I really think it's a much more practical thing than that. I think understanding... I'm a person who has always felt an incredible need to wrap my head around problems and just understand them. And so, it's purely... In some way, it's selfish.

By way of working on this full time, I get to have a better understanding. This whole thing started with me just being like, "Wow, I don't understand this." And there's such an overwhelming amount of information on a context that I can't imagine that many people understand it either. I'm not special. So, if I don't get it, then probably most people don't get it. How can I take all this complexity and collapse it and just make it so easy? If we can't all get on the same page about what's happening, it's impossible for us to make decisions. Like a political decision maker will not be able to say, "Hey, I'm doing this for this reason because this information and mistrust will just overwhelm that person's political capacity to make our political capital to make the decision."

People will argue about what the right solution is instead of just focusing on the solution, and the timelines are too short for that. So, it felt like, for me, the starting point was I need to understand what's going on. And then, it quickly became, "Well, if I've understood what's going on in a way that's simple, then I have a responsibility to share that with other people." And that fundamental thread is ultimately what developed into Covid Act Now.

Eric Ries: Say a little bit about what Covid Act Now is for those that don't know.

Max Henderson: So, Covid Act Now is a tool that was trying to send one very specific message, and that is that immediate action is required in the face of exponential growth in order to prevent catastrophic outcomes, right? So, the disease grows exponentially, and human beings don't think in terms of exponentials. We think in terms of linear change, so one goes to two goes to three goes to four. The disease fundamentally doesn't work that way. The disease grows exponentially, so one goes to two goes to four goes to eight. And those numbers start small, but they eventually become extremely big.

And just being able to help people build the intuition to understand that if we don't act now, catastrophic outcomes might result was the original takeaway for me in my research, and the thing that I felt and the group of us that started this felt a calling to share with the American public.

Eric Ries: Tell us the origin. So, was there a moment for you when the gravity of the pandemic crystallized in your mind?

Max Henderson: Yeah, there was a moment. This was really interesting. There's that commencement speech by Steve Jobs where he says that you can really only connect the dots in retrospect, and that it sounds like a trope, but it's been so real for me in so many places in my life. I've always had an interest in medicine and was in EMS for a while.

Eric Ries: Explain what an EMS is for those that don't know.

Max Henderson: Oh, yeah, of course. So, EMS is emergency medical services. So, think, emergency medical technicians, ambulances. EMS folks deal with disorders and cardiovascular disorders on the regular, right? Most emergencies that people have are either breathing problems or heart problems, major injuries. And so, I've always had an interest in this sort of thing. I thought for a long time that I wanted to be a doctor, spend some time in the pharmaceutical industry as well, and got a degree in System Dynamics. I've always been interested in nonlinear systems. And so, this was a really interesting coming together of two areas of interest for me academically in an extremely morbid, but also extremely practical way.

And so, I was on sabbatical studying to get my EMS certification and came across COVID. And I remember being extremely fascinated by it from the very beginning. In January, I started building a model just to try to understand as it was starting to spread it in China, just understanding the nature of the spread. It was designed just to share with me and mine, right? It was focused on what might happen in the Bay Area where I live if it started to spread here.

And it was after some of the data started to roll in that I realized how quickly it was growing. I had this epiphany, that we were already starting to have a case or two in the Bay Area, just how quickly this thing could possibly grow, right? We're talking from the first reported case to potentially thousands of hospitalizations in only a couple weeks. I had an aha moment that this information needed to not just be kept amongst me and my community but needed to be shared as widely as possible.

Within a few days of me sharing my model out just to my network, I ended up having to create a newsletter because thousands of people had signed up for updates. And eventually, after about a week or so of sending updates, I got approached by some folks that I had never met before who said, "Hey, Max, there are lots of states that don't have an analysis like this." I mean, keep in mind, this is a spreadsheet, right? I mean, it's not some super complex... Certainly, the thing has evolved, but back then it was just one tab in a spreadsheet. "You need to make something like this for every state."

And so, I agreed that I would do that, and I partnered with my other founders, Zack, Igor, and Jonathan. We built a copy of the spreadsheet for every state in the union, and Igor built the website. We hooked it up as quickly as we could. And within 48 hours of launching the thing, nearly 10 million Americans have come to the website and seen our predictions. And from there, as these things go, every time more people saw our predictions or relied on our math, the more galvanized and the stronger of a calling I had to continue and elaborate it and make it more reliable and complex and feature-rich. And so, we've, from that early success, just continue to build.

Eric Ries: How did you make that original model? I think what's really interesting, you're talking about like it's no big deal. But most of us encountered that same set of information earlier this year, and very few of us had the insight that you did to say, "Hey, let's build a model and see what could happen in the Bay Area." But how did you actually do it? What does it mean to model a disease like this?

Max Henderson: Yeah, it's a great question. I mean, the thing is, you can always make models more complicated and all models are always going to be wrong, right? So, I had to start with that, you're never going to have a perfect model. And so, in some ways, simplicity is better because something that's simple is easier to understand.

And the reality is that the most important thing here is the nonlinear behavior of the disease. I mean, I'll start by saying that, like I said, I got a degree in System Dynamics, and so spreadsheets and models are my jam. I'm no world class nonlinear systems expert at this point.

Eric Ries: Well, but you were on sabbatical from Google.

Max Henderson: That's right. But I mean, I hadn't touched a nonlinear model probably since college, right? So, the key insight here is not some complex modeling insight, but rather that the numbers, whether it's hospitalizations, deaths, infections, etc., all double on some fixed period. So, the simplest possible analysis here really is just take one cell in a spreadsheet and then measure how many days all the systems... So, when a nonlinear system like this is in exponential growth, every single category, infections, deaths, hospitalizations, they all grow at the same rate. And that rate is some doubling period that you can measure from the empirical data.

So, if you've got a four-day doubling period, which is what it was in most places before interventions went in place, the most simple analysis here is just one column for deaths, one column for infections, one column for hospitalizations. And every four days, the number in each column doubles. And everything else is really refinements and simplifications.

Obviously, at some point, when everyone is infected, things are no longer growing exponentially so there are some... Obviously, it doesn't continue forever. But that's the fundamental analysis, and it's actually really, really simple. So, it doesn't take the... Oftentimes, I found in this and other things that I've worked on that the answer, and this is something we've tried to keep in Covid Act Now until this day, we're done when there's nothing left to take away and not when there's nothing left to add.

So, we can always make the thing more complex. And the model is now this complicated Python thing that takes a 96-core machine 12 hours to run, but the fundamental dynamics are very much the same. And that is the scary thing about the fall that we face, right? Especially now that we've delayed a potential spike here, I think we're forgetting the fundamental nonlinear dynamics at work. And if we forget that, it's going to be very much to our detriment.

Eric Ries: It's driving me crazy. I was just talking to a very smart person who is saying, "Well, because here in the Bay Area, we've got the disease under control. Therefore, hospitalizations are down. And just looking at the linear data, therefore, it will be safe to open back up and therefore, there won't be danger, exponential-type danger anymore. We're past that phase." And it was a completely reasonable inference. I completely understood where they were coming from, but it's dead wrong. And if people draw that conclusion from the fact that these early interventions worked, aren't we in a lot of danger?

Max Henderson: Absolutely. Yeah. I mean, I actually heard one of the leading epidemiologists in Germany call this the prevention paradox, and I loved it. The idea that because through positive action, you stopped the bad thing from happening. The bad thing either was never going to happen or has gone forever now and can never come back, right? And effectively what we've done is buy ourselves time. That's it.

Every person who hasn't yet been infected is a potential person who can be infected once non-linear growth comes back. And when it comes back, it'll start off slow. At first, it'll seem like everything is fine when we're past the worst of it. Because when that one goes to two, and two goes to four, and four goes to eight, the numbers are small. But when you're at 500 and it goes to 1,000 in four days, and then it goes to 2,000 in four more days, and then 4,000 in four more days like we saw in New York, that's when things really become completely insane. And the scary thing is we all have to check our bias on this all the time because it's just not a way... It's not intuitive. It's never going to be intuitive.

And so, you really do have to look past the intuition of this thing. And even I, looking at the math every day, sometimes I feel it. I feel the, "Oh gosh. We've been doing this for two months, and it doesn't feel like it can continue and nothing bad has happened. So, was this all maybe an overreaction?" I know for certain the answer is no, right? There's no question in my mind whatsoever. But I think feeling that and knowing that that emotion is there is a super powerful thing because we're all feeling it, right? Everyone's tired. Everyone is uncertain. And if we-

Eric Ries: And motivated reasoning starts to kick in.

Max Henderson: Yeah, that's right. Motivated reasoning starts to kick in. And it's so subtle, you don't even notice it's happening.

Eric Ries: I have friends who wanted to have a party, and they were trying to convince themselves that it was safe to have the party and they were going to socially distance at the party. They had a whole plan for why it was going to be safe. But I noticed in the planning of this party that not only were they trying to convince themselves that it was safe, but it started to spill over into them making justifications of the pandemic wasn't actually that dangerous.

I don't think they were consciously aware of the fact... The fact that they were just tired of being stuck at home and missed each other and wanted to have this party was starting to affect their ability to reason objectively about the data that they were seeing from the outside world, never mind the fact that the data is grim and it's hard to face it head on.

It's funny you talked about the paradox of prevention. I actually wrote a blog post called The Curse of Prevention, but not recently, back in 2009. I had to look it up. It's that old. Where I was grappling with this, because in engineering, this comes up all the time, because how do you know when it's worth it to pay the cost of a mitigation of something that may never appear? And there's this political, we should call it what it is, this political impulse to use the curse of prevention to attack something that you don't want to have happen. And you can use it both ways.

You can say, “You're alarmist." The thing wasn't really true that we had to do the lockdown and criticize that way. You can use it the other way. You can make up potential dangerous things that could force someone to do a prevention that they don't really want to do, which has been the criticism on the other side. You've had the distinction of even being called fake news, which is one of the highest honors you can achieve in our current crazy times.

Talk a little bit about what that's been like to be on the receiving end of this idea that you're somehow manipulating the data or trying to serve some... I'm not even sure what your secret nefarious agenda even would be, but whatever it is, how do we know that this data is accurate and what's it been like to be the subject of this disinformation campaign?

Max Henderson: Yeah, it's been surreal, right? Because never having been the subject of something like this before, you'd think, "Oh, well, maybe there's some nuance or confusion." Where is the line between stretching the truth and just making something up? I can confirm there are people out there who are just willing to make things up.

Eric Ries: Just arguing in bad faith.

Max Henderson: Yeah, yeah. Arguing totally in bad faith.

Eric Ries: 100%.

Max Henderson: It's really, really interesting. Gosh, how can I describe it? Yeah, surreal is probably the best way to describe it. I mean, the funny thing is that’s the one major piece that's ever gotten put out on us, and we certainly get a lot of hate mail and things like that and way more positive encouragement thankfully. Everyone gets their share of malicious actors or people acting in bad faith.

The one article that really got written that I think wasn't most in bad faith also contained something in it that was like, "And these guys are predicting 13,000 hospitalizations in New York in two weeks. Can you believe it? And the number ended up being like 19,000, so there's some... It didn't age very well.

Eric Ries: Yeah. And I'm sure they didn't print a retraction and apology.

Max Henderson: No.

Eric Ries: But they're just onto the next conspiracy theory.

Max Henderson: They certainly did not. Yeah, nobody ever went back and, in fact, check that. Honestly, I don't know what our nefarious ends would be. But more than anything, it makes me sad that we are... At least there are some portions of our national discourse that are so divorced from reality that we can't help. Almost my reaction to these things has been like, "Gosh. Well, I wish I could just sit down with you and show you that I'm a human and that we're the same," and just show you what I see. And rather than argue with you, just try to bring you in and show you like, "Well, here's the challenge that I see. Let's just logic through this together."

And how do you see us solving this problem? What is it here that you think is fake? What is it here that you think my motivation is? Or is it just because like I'm some faceless person somewhere on the internet that it feels like this is the right way to handle the situation?

Eric Ries: Believe me, having done that all with such people, I mean, write a book and they'll see this experience and see what it's like. I think I can safely predict that not a single one of those people would be willing to sit down with you precisely because I think, at some level, at some visceral level, they understand that it would work. This bad faith nonsense, it is only effective if you can hide behind a veil of pseudo anonymity and you can really other the person that you're talking to. And if we actually had the connection and trust and love, right, the fundamental human connection that is needed, it's not possible to sustain that kind of ideology. And therefore, those things are actually dangerous to it.

Max Henderson: Yeah, I try. I certainly don't have the time to do this with all of them, but I try as much as I can when people post a troll comment or something or send me like a really, totally, unacceptably vitriolic email like, "Hey, you guys are the worst. You're trying to destroy the country," or even worse things that I won't even mention.

Eric Ries: We will not be linking in the show notes, but people can find them if they would like.

Max Henderson: Yeah. A lot of them are coming privately directly to me, I try in almost every case to react with exactly the opposite. I mean, some of them, I just can't engage with, but I try to respond with exactly the opposite language and say like, "Hey, thank you for bringing this up. You're so right to be skeptical. There's so much bad information out there. Here's what I see. And really, I see this situation as a once in all of our lifetime opportunity to save lives.

I love this country as much as anybody else does, and I'm trying to do the right thing. These are the facts as I see them and I'm willing to have a debate with you about what you see as being problematic here." And 90% of the time, people don't take me up on it. But about 10% of the time people do actually like, "Wow, I never expected an answer to this email. I'm sorry. I see what you're talking about. I have these two other questions." And then, it ends up turning out to be a productive discussion.

Eric Ries: That's outstanding.

Max Henderson: Those things are making me, honestly, they make me happier than almost anything else when I can actually make a connection with a person who is coming from such a place of mistrust and, I mean, I don't want to use the word "ignorance" because it makes it sound like I am somehow the keeper of knowledge, but mistrust and negativity. They can turn around to something as positive as a real personal connection and an alignment on some kind of objective reality.

Eric Ries: So, how do we know the models are accurate?

Max Henderson: It's a good question. We're working. So, I think first of all, we have to really understand what accuracy in the context of modeling means, right? All models are wrong, some models are useful. And the goal of modeling really more than calling an exact number X number of days, weeks, months out is to understand the variety of positive outcomes, find the catastrophic ones and figure out how to prune those, figure out how to make sure that those catastrophic outcomes never happen, and build intuition for how our actions can change the future.

You don't have to be right on every single right layer. There's so many ways a model can be right or wrong. It can get the actual number of hospitalizations or deaths or infections wrong, or the relative ratios of those things to each other, or exactly when they happen or what the shape of the curve is exactly. So, there are just so many different ways that things can be wrong.

And then, there's also the fact that this is a system where our expectations of the future change the future, right? If I expect that a huge spike of infection is coming and we shut down, the huge spike of infections doesn't come or at least it gets delayed like the situation we're in now. And so, in a world like that, right and wrong are a little bit more complicated than just like, "Could you call the exact number?" So, we think about this in a couple different ways.

One is we think about getting the shape of the curve right and understanding how the scenarios differ relative to each other. So, if I do X versus doing Y, what is the difference that I can expect on average between those two different outcomes, even if the exact numbers are going to be slightly different? The second is how do my actions change reality? Our original tool showed you what happened if we went to shelter-in-place and what would happen if we didn't. Even if those curves don't perfectly mirror reality, the relative difference between them is incredibly important for building intuition.

Eric Ries: Actually, if you don't mind, talk about what the model said at that time, because I think many of us have gotten used to the idea that we're going to do shelter-in-place and we forget that it was like five minutes ago, in historical time, that that was an extremely controversial idea.

Max Henderson: Yeah, I mean, when we first started talking about shelter-in-place, it wasn't even clear that this was within the realm of possibility, right? I mean, when we put the tool up, again, we were not telling anyone to do anything necessarily. We were just pointing out the speed at which a decision needed to be made because these were the relative possible outcomes that we were dealing with. But I mean, we were talking, at that time, about hundreds of thousands of hospitalizations in just a few short weeks.

I mean, we saw New York was the last to act. And so, New York really provides one of the few real examples of how bad it could get, and given it's an urban area, but even in the state, how bad it can get, 0.1% of the entire population of New York state has died from COVID, not 0.1% of infected. It's literally 0.1% of the entire population of the state. And last we checked, the antibody testing to see how many people had been infected came out to somewhere between 10% and 20% depending on where you were in the state. So, you're talking... This could have been five times as bad at least.

And so, the models were showing something actually quite similar to that. It was basically somewhere around 1% to 3% fatality rate for anyone who is infected, and probably about 70% or so of people infected before herd immunity would be reached. And so, you're talking about 1% to 2% or so in that range of fatalities in this country. So, at a population of 300 and 30 million, you're talking about somewhere between 3 million and 6 million people.

We've learned a lot more about the disease since then, but the numbers actually have not shifted that much, right, now that more data has come in about infection rates and we actually have some antibody tests. Keep in mind, at this time, we didn't have any. So, we didn't know how many people were actually getting sick. We just knew how many people were getting sick enough to go to the hospital.

The numbers have come down a bit. Our estimates of death rate have been cut roughly by a factor of two. But I mean, even a 1% death rate or a half a percent death rate in the United States is still somewhere between 1.5 million and 3.5 million people.

Eric Ries: Still a catastrophic loss.

Max Henderson: Still a catastrophic outcome, and that hasn't changed. That is our latest thinking. That is our more optimistic thinking.

Eric Ries: It seems like that, in some ways, because we're having such a hard time forming a national consensus about the facts as we learn more about the microstructure of the epidemic and the disease, and the effects of vitamin D, and the antibody tests, and all this micro knowledge, we lose sight of the fact that the macro facts have been established and are pretty well understood and have been pretty stable through the whole thing, namely this is a highly deadly, highly contagious, exponentially growing epidemic. And nothing can or will change that until a vaccine is developed.

Max Henderson: That's right. That's right. This was one of the reasons that we created and continue to work on Covid Act Now is that the reality is that we are doing a really bad job of paying attention to the forest and not the trees, right? Our national discourse is largely occupied by either, "Here's the number of deaths there were yesterday, or here's the number of hospitalizations there were last week. Or hey, someone said that chloroquine is a great treatment for..." We're dominated either by facts out of context or anecdata, and so it becomes incredibly... The cognitive load of just understanding what the heck is going on and where we stand and what's coming next is extremely, extremely difficult.

Using a weather analogy, instead of saying like, "This is what the temperature is going to be. Oh, you should bring an umbrella because there's a 50% chance of rain," well, right now in terms of COVID communication is, "Well, it was 75 degrees three days ago at 5:00. It has rained at least three times this year," which is not at all useful for understanding what you actually have to do.

Eric Ries: And who could really trust those umbrella manufacturers anyway?

Max Henderson: Right. Exactly. Maybe, "I've never seen rain, so maybe rain is not a thing. And I don't even need an umbrella."

Eric Ries: Yeah, distraction, delay, obfuscations act as information.

Max Henderson: Exactly. And when you factor in the fact that there, obviously, massive economic, public health, and mental health concerns even to the extreme measures we're taking, it becomes a massively complicated thing to understand.

Eric Ries: I kept thinking about-- in the early days of the crisis, especially in that wave when shelter-in-place was coming into effect, I tried to imagine what life must be like for the principal policymakers, especially mayors and governors, who must have been hearing from endless business lobbyists. None of whom I'm sure were saying we need to shelter-in-place. I'm sure they were all saying, "I need you to understand the catastrophic impact, the actions you're talking about taking will have on my industry and you better believe will remember who blah, blah, blah." The usual lobbying, self-interested playbook, that must have been so loud in the ears of policymakers. And one of the very few counterweights to that was the campaign that you were running with Covid Act Now.

So, talk about why you decided to take action and advocacy orientation versus just having like a neutral model at the beginning. What did you do and why was it important to you that policymakers have somebody telling them the actions that they could and must take?

Max Henderson: Yeah, great question. So, first of all, let me say I have... I mean, mistakes have been made and will be made, but I have just an immense respect for every policymaker and decision-maker in this situation. I mean, talk about the worst possible situation you could possibly be in as a decision-maker, you are responsible for every life lost and you're responsible for every job lost. And no one is ever going to know how it would be if you had acted differently. And so, you are the risk vessel. You are the person responsible no matter what, and that's a lose-lose.

So, from the very beginning, our goal has always been to, one, be extremely clear, two, be extremely action-oriented, and three, be additive. So, the goal here is not to say you have to do X or you have to do Y, but rather provide information that allows decision-makers not just to... They're suffering from the same information overload all the rest of us are. So, distill the information down to something so clear that the action is obvious, right?

The next thing to do, forget 10 levels down the decision tree and all the detail, just look at the situation in its most basic clear form and make the decision that obviously needs to be made no matter how hard it is. That, in and of itself, is a difficult thing. It's a really, really hard thing to do and something that we've been focused on as being the primary difference between us and all the other models out there, right, because there are now dozens of them, and they all do great work. Some of them are private and some of them are public. But the reality is that we don't necessarily need more complexity. We need less complexity and more clarity.

The other thing that I would add here is for decision-makers specifically, I think we overlook how important public opinion and political air cover is to making the right choices. In this country, we believe in freedom. You mentioned like a shutdown would nearly have been unmentionable or wasn't unmentionable when we started this thing, and if you're willing to convince the American public that something like this is necessary, and I'm convinced it was and continues to be necessary, then you need to clearly explain why that's the case. And you can't just hand over a model that only a PhD expert can interpret and say, "Well, we're just doing this. You got to trust me." That's just not the level of national discourse in this country.

And so, we wanted to create a tool that wasn't just telling decision-makers how to act, but that was simple enough that the general public could see it, understand it, and provide air cover for their leaders to do what was required. And one of the things that worries me the most right now is that as we shift into this prevention paradox, "Oh, well, maybe this was all an overreaction, or even if it wasn't, we're past the worst and it's time to go back to normal."

If we don't have similarly clear graphics to explain why either we're not there yet or what it will take to get there, then we're going to lose control as a nation. We're going to lose control over our decision-making here because ultimately, we're going to go back to something much more reactionary and less first principles-driven because it's just impossible for everyone to agree. And so, what ends up happening is decision-makers are forced to just take the average of public opinion and do that, right?

Eric Ries: Talk about the campaign you did with Resistbot to have people write to policymakers and try to motivate them. I was really proud to see that come together.

Max Henderson: Yeah, of course. So, the campaign we put together is we essentially sent out text messages to several million people who were Resistbot members to essentially let them know, on a push basis, what was going on in their state to actually look at the numbers and to come to their own conclusions about what action needed to be taken, and then make it really easy for them to communicate with their electeds about what their opinion was, right?

So, essentially taking the website and instead of forcing you to find it, sending it to people and just laying the information out and saying, "Look, here's a couple different scenarios. You can interact with them and see what the outcomes would be. And if you're convinced that action is needed, you should let your electeds know." It's that kind of thing, and I'm super proud that we did this. That, I think, made probably a bigger difference than modeling, right?

The model is a tool. It's a tool to an end. And that end is ultimately to get us as a country to make faster and better decisions. We could create the fanciest model in the world and be extremely proud of it on an academic basis, but if it doesn't actually create change, then it's a waste of time, in my opinion.

Eric Ries: Quick shout out to Dustin Moskovitz who underwrote the cost of the Resistbot campaign at a critical time. What role do you think that campaign and Covid Act Now played in convincing policymakers to take that drastic action to save lives?

Max Henderson: Wow, what a question. I'd like to believe, I mean, having communicated directly with probably half the states in the union, either in the public health department or in the governor's office, my sense is I believe I played a non-trivial role. I mean, at the end of the day, I'm much less worried about the role that we individually play and more worried about us getting to the right answer.

And the only reason I even care about the role we play is purely just to make sure that we are adding to the signal and not adding to the noise. We're actually doing something that has any effect whatsoever and we're not making the situation worse in some way. Because it is possible to have negative impact, right? It's possible to-

Eric Ries: Absolutely.

Max Henderson: I like to stay curious about our impact, but the truth is we're all on the same team together, right? I mean, my view here is this is... Depending on how you look at it, this is the first time that we've had a real enemy on our soil, and the American ingenuity can overcome anything. I mean, I'm convinced of that through and through, right? I mean, I'm a first-generation immigrant. My parents came here for a reason. As long as we are all on the same page, we can do anything. And so, if I can help, if our organization can help get us all on the same page, even just a little bit, that is work worth doing.

Eric Ries: Where did your parents come from?

Max Henderson: My father is Cuban. He fled the Castro regime in the late 1950s, early 1960s, and actually fought in the Bay of Pigs invasion. So, he was part of the entire Cuban missile crisis. I wrote a book about it. Mom is German immigrant, and she also came over here around the same time in the '60s. They met in Miami which is, I guess, where a Cuban and a German would be.

Eric Ries: I guess that that makes a lot of sense. And I appreciate you--there's nothing... We often see that immigrants and the children of immigrants have the most optimistic view of what America is capable of. And I appreciate you tying it to that patriotic sensation that we all ought to have, that we have a common enemy. It's really, in some ways, our truly global enemy, and that coordination, cooperation, solidarity, those are going to be essential ingredients to combating.

Max Henderson: That's right. And that's why I see there being a real missed opportunity to come together here, right? And I realized that either out of ignorance or out of malice, there were those that want to divide us. But the reality is that this is a unique opportunity for all of us to come together across the world, but even more so inside this country to find something that all of us have a common stake in, right?

I mean, all of us have parents and grandparents and people we care about that we don't want to lose but more importantly, whether you care about National Defense, or whether you care about protecting this country and what it stands for, or whether you care about the public health outcome, the reality is that all roads lead to Rome here. And so, if you choose to see this thing as the first successful invasion of the United State by some foreign enemy, then feel free to see it that way. But when you... I can't see how anybody looks at this and says like, "Oh, it's not a big deal, or we should just ignore it." That, I really cannot understand.

Eric Ries: You talked about how important it is to make sure that you're adding to the signal and not the noise. Can you talk a little bit about some of the guardrails you've put in place as this has grown to make sure that the data is accurate, obviously, within the parameters you described, but also that you are integrated with the scientific and public health communities? And this is not just tech... I think of the caricature of ignorant tech people trying to metal in scientific matters we don't understand. Just talk a little bit about the seriousness with which you've taken the need to get the model right.

Max Henderson: There are two things here, one is getting the model right and the other is not crowding out. We're not adding so much, just adding so many more voices to the discussion that the true experts cannot be heard, right? And we think about them both separately.

So, getting the model right is actually almost, in some ways, the more straightforward thing because there's a lot of prior art in epidemiological models that we can reach into. So, our model, we ended up moving from a spreadsheet model to a model based on one created by Alison Hill at Harvard that is open source. We ended up taking that thing and modifying it to have a couple more features, but ultimately, fundamentally operate in the same way.

We've partnered with Stanford and Georgetown epidemiological and public health folks there as well as other advisors like Nirav Shah, who is ex-Health Commissioner in New York and also a lecturer at Stanford and ex COO at Kaiser Permanente, to help provide us the guidance to make sure that we're sending the right public health message, the model has the right inputs, the output is believable. And so, our approach there has really leverage existing well-respected prior art and bring advisors into the fold so that we have...

We're not making these decisions on our own, right? The value that we add is the engineering work, being able to scale the thing up, communicate it clearly, essentially taking this great academic work and turning it into a consumer product that is reminiscent of the best in class tech products out there. That is the value that we're adding, but the guts are all public health and epidemiological people. So, that part is pretty straightforward because it's just like, "Hey, let's go find the experts and let's just amplify their thing."

I think that the much harder thing here is figuring out if you’re additional. Are you crowding out the experts by just adding more noise? And are there others who are sending pathological messages that you have a responsibility to answer to or to amplify the right message in the face of? And this is a much more difficult question. I think we've ultimately come to the conclusion because we're working with experts who tell us, we rely on them to tell us whether they believe our message to be additional, and we ask the question, "Are we actually adding to the message here? Are we saying something that others aren't already saying? Or are we crowding them out?" And we've come to the conclusion that, again, where we really add the most value is making the message approachable.

And that is the thing that it seems nobody quite had the ability to do, and it's not because we're special, it's just the multi-disciplinary nature of our organization. We brought together epidemiological experts who really understand the science, public health experts who really understand what the public message for that science needs to be, and tech folks who understand how to take a message and make it super approachable, super easy to interact with, super polished, and just very confident and inspiring. It's that combination of things that has really made us additional. The mixture really is greater than the sum of its parts.

Eric Ries: So, Max, tell us what the models are saying now. Where are we in this crisis?

Max Henderson: It's a great question. So, we've recently transitioned much more from models to metrics. And the model was really, really useful when we were trying to understand how non-linear behavior like explosive exponential growth was going to impact us in the short term because it's so counterintuitive. Right now, we're in a state where we've at least delayed, not avoided, but delayed the worst.

We've transitioned to a much more of a metrics approach. The difference here being rather than trying to understand what the future is going to be like, I'm trying to set goals and I'm trying to manage against goals, right? So, in this case, reopening. When am I ready and what does a good reopening look like versus a bad reopening?

There are a couple things that are clear. We are still maintaining the models because once we get through reopening, there's obviously risk of a second spike. And so, having the models in order as well. So, there's a couple really clear takeaways here. The first is we have not prevented. We have delayed what is potentially the inevitable unless we work on a holistic containment strategy.

The second is that the lockdown containment strategy, although it is extremely effective, and the best thing about it is that you can do it really quickly, is I think we all know is fundamentally unsustainable and we need to transition out of it as soon as possible. Because unlike some of the criticisms that are levied, I think all of us are intimately acquainted with the economic and human cost of being on lockdown. Nobody thinks that this is a free lunch. So, we need to transition away from this as quickly as possible. And the way to do that is through some combination, a slow-controlled reopening where we carefully and conservatively test how much disease growth comes back as we introduce various different parts of the... Or reopen various different parts of the economy, coupled with testing and tracing, right? So, coupled with both the opportunity.

Essentially, we have to know where the disease is spreading and how fast it is spreading at all time. And the way that we do that is we test as many people as possible to understand where the disease is, and then we trace all of their contacts and we test them as well. And anybody who's positive, we isolate them such that they cannot re-infect people. This way, we keep each infection from infecting less than one other person and things start to trend to zero.

I think our biggest risk here is that we believe we're outside the storm, but we're really in the eye of the storm right now. I'm from Florida, so you're going to hear hurricane analogies for me a lot. We're in the eye of the storm. And if we lose, if we take our eye off the ball, we will have as big of a spike as what we were predicting before this whole thing started. But we will be coming at it from a much weaker economic position, a much weaker emotional position because I know we're all tired.

And so, the main thing to pay attention to here is we are not out of the woods. And if we're not careful when we drop the ball halfway, we are in for as bad or potentially even worse of an outcome than before. So, careful management here is key.

Eric Ries: Where do you think we go from here? How do we get out of the crisis?

Max Henderson: That is a great question. Nobody has a crystal ball, but my view here is that... So, containment actually worked a lot better than we originally anticipated. There was a question at the very beginning when we were building these models, "Well, what is social distancing really going to buy us? Is it going to slow the disease, but it'll stay growing exponentially? Is it going to flatten? Or are we actually going to be able to make the disease shrink just by social distancing?" And the reality is that... At least our team was surprised.

We essentially set up two confidence intervals for our model, best case and worst case. And we ended up basically right on the button for the best-case scenario. So, from our perspective, this lockdown, as extreme as it was, actually has been much more effective than it could have been based on the data we had when we started. And that, to me, is really encouraging.

So, I think it's completely achievable for some combination of policies, lightweight social distancing policies, policies like masks for all that are really focused on preventing transmission in close quarters, and a combination of testing and tracing to allow us to bring the economy back online while at the same time slowing or completely stopping disease spread. I think it is possible.

The thing that scares me the most is that rather than approach this the way that we have past crises as an American people to say like, "We can do anything. We're going to figure this out, right? There's no challenge too big," it seems like we're copping out to some degree and saying like, "Oh, well, that seems unachievable. We can't do that, so we're not even going to try," right? And so, that is the thing that gives me the most pause through all this.

The way that we choose to frame these problems are going to have such a huge impact on what we end up doing. So, we can think of this as unemployment and people losing their jobs and the economy collapsing or we can think about this as a national effort where these people are heroes for slowing the spread of a deadly pathogen and we're going to treat them as such, right?

Eric Ries: Like a 21st century WPA?

Max Henderson: Yeah, exactly. This can either be like an aspirational, "We are coming together," or it can be, "Oh, my God, things are falling apart." And the reality is that that framing is super, super important. Similarly, workers on the frontlines, they can either be people who are unfortunate enough to have to return to work or they can be the heroes going to battle for us that we take care of with free health care and free testing and hazard pay. We can do this, right?

Eric Ries: Protective equipment.

Max Henderson: Right, exactly. That framing is super, super important. Similarly, any national challenge is either an opportunity to come together and rise to the occasion or a cause for despair. And I think it is important for us to realize that we have control over these narratives and we have control over these outcomes.

Eric Ries: Max, thanks for taking time to share the story and thank you for your work as an antidote to despair in these really challenging times. It's meant a lot to me, and I know to millions of others.

Max Henderson: My pleasure, Eric. Thank you for having me. It's been great.

Eric Ries: This has been Out of the Crisis. I'm Eric Ries. Out of the Crisis is produced by Ben Ehrlich, edited by Jacob Tender and Shawn Maguire, music composed and performed by Cody Martin, hosted by Breaker. For more information on the COVID-19 crisis and ways you can help, visit helpwithcovid.com. If you are working on a project related to the pandemic, please reach out to me on Twitter. I'm @E-R-I-C-R-I-E-S. Thanks for listening.