Julie Zhuo Former design VP Facebook & Co-Founder Sundial chats with Amit Somani Managing Partner Prime Venture Partners.

Listen to the podcast to learn about:

02:20 - From an intern to VP design at Facebook

07:00 - Building for the user when you are no longer the user

14:00 - To Build or Not to build. How do you decide?

25:20 - Metrics are a proxy for the change that we want to see in the world

32:30 - How to communicate with all the stakeholders

41:45 - Data can’t lead you to what you should build

44:00 - Understanding the most important narrative of data and turning it into actionable insights

Read the complete transcript below:

Amit Somani - 01:21

Welcome to the Prime Venture Partners podcast. Today I have with me a very special guest, Julie Zhuo, who was formerly the design VP at Facebook from its early days. In fact, as I was researching this, I found that you started there as an intern, and is now the co-founder of a new company called Sundial. Welcome to the show, Julie.

Julie Zhou - 01:43

Thanks, Amit. I’m glad to be here.

Amit Somani - 01:47

Julie, one of the things I was struggling with when I was preparing for our podcast is you have such a versatile background and a lot of different interests. You’ve written a book, ‘The making of a manager, you have a lovely journey at Facebook, you’re obviously into design and product thinking, etc. So I’m a little bit spoiled for choice here. But I’m going to focus our podcast here on building great products and great design, and some of the only lessons perhaps you learned at Facebook. So can you tell us about your early journey at Facebook, maybe even how you got started there? And we’ll dive in from there.

Julie Zhou - 02:23

Sure thing. I first became a user of Facebook, and it was a product that I loved, I was obsessed with and all my friends were obsessed with it. That’s because we were all in college when it swept through college as one of those hot social networks.

My earliest memories of it were just writing on my friends’ walls. I would check every day to see who had uploaded a new photo, I would learn about new musicians and artists I should listen to from the profiles that my peers at school shared. So when I graduated, I had just taken a class at Stanford about entrepreneurship. And coming into Stanford, I didn’t know anything about Silicon Valley and I didn’t know its history. I wasn’t really sure exactly what it was. The senior year, I take this class, it goes into a lot about what makes Silicon Valley famous? What is a startup? What is the VC ecosystem and a lot of those very basic questions? It made me really excited about the prospect of just the idea of starting something from nothing. A small group of ragtag individuals forming a company and building something that becomes hugely valuable.

I resolved that I wanted to go and work at a small company. I had done some internships at Microsoft in the summers before which was great and I learnt a lot. But after I took that class, I had my sights set on wanting to join a company. So I looked around, I looked at what I was using, I had some friends who had joined Facebook just very recently and I was thinking about an internship. So I decided to go and try my hand at applying for an internship at Facebook, and they didn’t have an intern programme. So I was like, that’s okay, hire me anyway, I’ll just come out for summer; I’m graduating soon. I was trying to dangle the carrot. If it’s a good experience, I’ll very quickly join full time.

So it took me about one month to decide that I was just going to join full time. I converted actually, halfway through my internship, I decided that I was going to finish up my last semester at school while working full time because I loved what I was doing so much. And when I joined Facebook, it was still a college and high school network. It wasn’t very widely known. Most people had no idea what Facebook was. My mother was like, “what, are you sure you don’t want to work for Microsoft?” I know, Microsoft, it’s a good company. But I think that there was just this. I can’t tell you. People ask me now, “When you joined Facebook, Did you know it’s gonna be what it is today? Did you have a sense that it was gonna be this huge thing and transform social networking?” I don’t think the answer is yes, definitely not when I was 22 years old. I just thought that this was a very good product. I was also certain that it was a better product than Myspace or any of the other competitors in social networks.

I think that’s pretty much what brought us all together. Everybody was pretty much at that time, a college graduate or a college dropout. It felt in a lot of ways, like an extension of what we had just come for in college. We were also the primary users of our product and getting into talking about product thinking and I think that’s one of the reasons why it took off so well, because initially, it was a product for college and high school students, by college students. So our early intuition about what people like us wanted was spot on because we were designing for ourselves and our friends. That was how we did stuff. We would just dream up an idea, we talk it over with some of our friends, they would tell us it’s stupid, or they say, this is a great idea. We hacked all night, and we built it as an extension of school. We were all like night owls who would be listening to Daft Punk while coding until 4 or 5 am. That was the early days of Facebook.

Amit Somani - 06:32

Very, very interesting. Lots of follow on questions, but let me pick on the product one, which is, You went from a very modest beginning only for high school and colleges and so forth, with I think 100 people is what I read when you join them to hundreds of millions and billions of users, right across multiple products and features. You guys then went on an acquisition spree with Instagram and all that. But obviously, you could not continue to empathise with the user all along. When you were a college student and you joined right out of college which is great. How do you build that customer empathy, both from a design and a product point of view, as it scaled?

Julie Zhou - 07:12

We had to learn that lesson the hard way. So early on, in its first few years, there were a couple of things that Facebook invented. We were one of the first companies to do photo tagging. I think Flickr invented the idea of tagging various entities, but we did it with people. So we could say, I could take a picture of the two of us and I could say, this is a picture of me and Amit and you would get that photo on your profile, it would show up in newsfeed and you would be alerted to it. You would see it and it became wildly popular very quickly.

Facebook became one of the top photo-sharing sites in the world, even though we had like a fraction of the features of all of our big photo-sharing competitors. We didn’t offer high res photos, we didn’t really have beautiful navigation. It’s actually hard to really navigate. But we did have this photo tagging feature, which meant you could look up Amit and you could look at all of the photos, anyone had taken of you with your friends and had tagged you and people loved that. So that was one thing we invented that was hugely successful. And we were like, yes, this is working.

Next thing, newsfeed came out from our own observation. Every day we would log on to Facebook and we would ask, we would wonder, ‘hey, who updated their profile?’ ‘What new bands did they add?’ ‘What new photos did they add?’ So we had this pattern that we ourselves were doing and that we recognised all of our friends were doing. You’d log on to Facebook, then you would search for your first best friend, you would go look at their profile and say ‘Did anything change?’ Okay, nothing, great. Next friend, look at their profile, did anything change? Oh, cool, They added some new bands, interesting. Next profile, photos and so forth.

So every day it was the same habit. We thought that’s really repetitive and that’s just a lot of work. Why don’t we just make a feature where we could tell you everything that changed from your friends. So you don’t have to go and do that and we did that. Actually, that one was an interesting story. Because we launched newsfeed, we were sure that everybody would love it, because we’re like, we just solved this problem. We just saved you 10 minutes of your day. We can just show you stuff and you’ll never miss anything interesting that happened with your friends. We were wrong because people were up in arms about this feature. They really hated it. They thought that it was creepy. They thought that suddenly, this thing is following me, tracking me and telling my friends what I’ve been doing. Within a week or so, 10% of the user base joined a group that said, ‘I effin hate the Facebook newsfeed.’ This is the thing that actually put us on the map because nobody was paying attention to Facebook before this. We’re a small college, high school site. That was almost like such a big story that news reports like CNN and everybody was all reporting on it because the story was, ‘there’s this college site and they did this thing that so many of their users hated and are protesting. “Haha, isn’t this funny.” That was how we first made national news.

That was the first time that Facebook became the story but what we noticed is that, even though people were saying they hated it, they were using it. Usage of news feeds shot up and not to mention, the only reason that 10% of the user base were able to join this group is because they saw it on Newsfeed. They wouldn’t even have been aware of the fact that this ‘I hate the newsfeed group’ existed if not for the fact that they were on newsfeed all day long, checking it out and being made aware of things. So, Mark made a tough call then. But it was absolutely the right one where he said, ‘even though people are saying they hate it, their behaviour is telling us that there’s something valuable here.’ Instead of just rolling it all the way back as many people at the company were saying, ‘we should roll it back.’ People hate it, we got to give them what they want. He said, No, let’s just listen to what their concerns are. Their concerns are that they want more control over which stories get published or not, let’s just build that.

So we had a team hack on it overnight, and they introduced a bunch of controls for what appears on Newsfeed. And, so newsfeed stayed and thrive and continued to grow. So that’s two stories now, where we trusted our instinct, and it totally worked out. Eventually, this method started to fail. I think this was about two or three years after I joined. And we had a series of other types of, what we thought were again marquee events that we were going to ship. One of them was called Graph Search and it was an idea that you could instead of just searching by keyword, I could say, Show me photos of Amit from 2017 and just get all the photos. It was a sort of smart graph query oriented search. We thought it was this huge thing. But it totally failed and nobody used it. There were two other projects like that where we thought it was amazing and came up with it intuitively, and they totally failed. And that was when we realised the lesson that we had, at that point, grown our user base. So that it was no longer mostly college and high school students. Our sense of what the user base wanted started to become less and less good at predicting what they actually wanted. Because at that point, we had added lots of people throughout the United States and we had started to expand internationally. We had some people who are less savvy, technically speaking. When they came on to the site, we realised we really had to change our method of product development. We had to become much more rigorous about understanding the pulse of the user base instead of relying so much on our own intuition. That was also when we decided that we needed to start testing stuff much earlier and not just one day, flip the switch and say, Hey, here’s this feature for everyone. But we needed to make sure we go to a small market first, and we see what and how they react. We make sure that it’s good before we turn off the switch.

Amit Somani - 13:36

Very interesting, I want to kind of drill down later into the role of data versus gut that you already alluded to. I want to go back to the photo tagging feature and particularly in large rapidly growing brands, such as Facebook, even back 12 years ago, 13 years ago, which is, ‘How do you pick what to build or not build?’ And I found this interesting framework that I saw you speak about, which is how a Facebook designer things and it has three questions. What people’s problems are we solving? How do we know it’s a real problem? And how will we know if we’ve solved it? Certainly, the first two are pretty relevant. But when you’re being inundated with, let’s do photo tagging, let’s do newsfeed. Let’s do like button. I’m sure there were like a million other things. How do you pick what to do both as a combination of gut and data? If you can talk us through that. And this is, again, more relevant, trying to apply it to more early-stage or early growth companies today.

Julie Zhou - 14:33

Yeah. I go back to that framework all the time because I think it’s a simple way to think often enough and good ideas. Good ideas for features, products come from everywhere. They come from observing someone using their product. They come from internal engineers, PMs, designers, everybody has ideas. And so, we go back to that framework of ultimately how it used to be. What would happen if sometimes someone would say, you know, I have got a great idea for a product, we should do X. Before we get into thinking about x, What is X? How should you implement x? Or what should it look like? That’s where the first question that you just quoted came in? Instead of focusing on a solution, we first need to ask ourselves, what problems are people having and make sure we really understand that problem. We can’t just go into the solution if we don’t quite have alignment on what is the actual problem. The second question helps us understand the priority of that problem. So the second question is, how do we know it’s a real problem? Another way to think about it is how big of a problem is it? Is it just a problem for you? Are you this special niche user? That’s very techie and you really want it to work a certain way, but you’re like one of 10 people who want that. That’s a real problem for you perhaps but just not a real problem in the sense that many people have this problem.

So the second question is meant to try and help us quantify what is the depth or extent or scope of how bad this problem is. And ideally, we can pick on the things that are real problems that we can understand and are pretty big opportunities or big problems. When you give people a solution or give people a better way, they’re just like, yes, that’s absolutely what I want. That really resonates with me. So those two questions are meant to help us with that level of prioritisation.

The third question is how will we know if we’ve solved it. Is to try and keep us honest on what behaviours, what would we see, that would give us an indication that whatever we came up with, was successful. It sometimes fights against this bias that we all naturally have. We think we understand the problem. We build something and we’re looking, we’re biased, so we’re always looking for reasons to believe that we did well. We’re looking for any positive signs. This person said something good and we tend to latch on to the positive feedback, much more so than we might be willing to look at or admit the negative feedback, or the graph that isn’t growing or whatever it is. So, by trying to define the criteria of what success looks like or what would we have seen in the behaviour? What would people have told us? What would have changed about their habits or whatnot? By trying to ask that question way upfront? We try to make it so that we have more objective criteria set up at the beginning. So we’re not looking at a bunch of metrics after the fact and then trying to, you know, piece it together into the best possible interpretations.

Amit Somani - 17:56

Right. So the question is on the first one, and I often get this pushback and hate. I mean, I really hate it. Which is, hey, look, if it’s a latent problem, people don’t even know that they have a problem. Like the classic Steve Jobs thing. I can see that you have this problem, but you don’t even know it yet.

Any thoughts on that? Because even like a profile idea when you saw it, because every day, you would go to different people’s walls and see what had changed and stuff like that. But often people may not realise that I want to know what has changed on my friends’ profile, what’s happening in the world or what’s happening in the news feed. So any thoughts on uncovering sort of more latent problems, or you just experiment?

Julie Zhou - 18:35

You’re absolutely right that if I just go and ask the user, hey, what problems do you have? They might list a couple of things but they may actually have a lot of opportunity in their daily lives that they don’t know that maybe to call it a problem or they can’t imagine what a better solution might look like. So, I do agree that oftentimes, you’re not going to get maybe the best ideas for things just by asking people. You should ask and you’ll get some signal. Especially the stuff that’s obviously broken, but not all of the best product ideas come from what people say explicitly that they have a problem with.

I think there’s another way to look at what are opportunities or problems, which is by observation, for example, if you shadow someone for a day, and you look at what they’re actually doing, you as a third party observer may recognise that something they’re doing is slow. It’s inefficient. Were you aware that technology could help you automate all of this stuff and save you like 30 minutes? And so I do think that’s why user research, especially the ethnographic research where you just actually watch people or you try to understand how they live their day to day life or if you are researching your own product, how it is that they incorporate your product into their daily lives? That’s just oftentimes, quietly watching can give you a tonne of inspiration that you can augment what you ask them and what they explicitly tell you, they wish could be better about your product.

But this is also where we get into things like looking at data which I know we’ll talk about soon. Oftentimes, if you look at the data, it could also present opportunities. If you find that there’s a huge drop off in a funnel between steps B and C, it’s probably something for you to look into. Why are people dropping off there? Is that expected? Is that not? Let’s go into it and you may find opportunities to improve whatever is the user experience between steps B and C. So looking at data is another way of finding problems.

I think the third way that I like to think about it is that a lot of human needs have been quite universal and unchanged in all of history. There are just different formats for us to express that. I think there’s the idea of one person that I heard which I liked. I don’t necessarily like negative framing. It’s like every problem is maybe in some variation of the seven deadly sins. It’s like gluttony or envy or the idea of, let’s say, newsfeed. It just boils down to you don’t want to feel the FOMO. You don’t feel left out so you don’t want to feel like you missed the news or you’re not in on what happened. I think that’s been true in all of history. Humans have always wanted to feel like they were a part of something and that they were clued in on what was happening. Newsfeed is just a manifestation of another way to maybe improve that but it’s not the end of it. I still think that desire continues and we can see it in the way that people respond to things like push notifications or whenever there’s an alert about something that might have happened to you. You can see a lot of retailers also utilise that particular need with things that seem scarcity or a sale that’s happening for today only. So they’re all tapping into a lot of these desires. I think that every product is just another way of doing something that expresses one of those human basic desires.

Amit Somani - 22:22

I want to go back to ethnographic research. We often recommend it, but think of yourself in the current role as a co-founder of a young early-stage company. How do you do that? It seems too rich. To say, let’s hire a graphic researcher or a UX researcher and you could go the other extreme where you could be just saying, I’ll just go talk to 30 users and whatever I learned is what I’m going to go. Any suggestions for how to hack it till you can afford to get a formal researcher or agency to help you with this kind of ethnographic research work?

Julie Zhou - 22:58

Yeah, my perspective on how you start out and let your approach to product development is when you’re at a very early, let’s say, you’re founding something or you’re going from zero to one versus one to n. I think there are actually two very different approaches. So if you’re going from zero to one, oftentimes it does behoove you to work on a problem that you personally understand very well. Or that you yourself are a target user or you know people who are target users. Something that speaks to your personal experience because then you can trust your intuition a little bit more. You are somebody who has experienced this problem. You felt it quite deeply and probably know other people who experienced this problem. A lot of times it gives you that mission and that meaning for why you should tackle this problem because you have that personal connection to it.

Now, it is possible that you can just approach a very blank slate and you can say, look, I just want an interesting problem. It doesn’t have to be personally connected to me, let me go and just talk with a bunch of people and figure out what that opportunity is. That is one way of doing it. But I find that I think Paul Graham gives us advice. It’s like, ‘first build something that you yourself would want. Because it does then make it so that you could cut down on all of those steps of needing to talk to 100 customers because you can’t trust your own intuition, but you can. Therefore you should go from that. That’s what we’re doing with Sundial.

It came from a lot of our experience of building products and relying on data, not to mention, my co-founder, Chandra was a data scientist himself. He’s managed a lot of data scientists, he’s looked at data at a lot of companies. So he has a lot of personal knowledge and domain expertise for this field, which really helps guide our initial assumptions. But also, in our early days, we are trying to talk to customers and our customers in this case, or other high growth companies and really get to hear their stories. Ask them very basic questions like what do you struggle with or what questions you have on a day to day basis when it comes to your data? What would you like to know and sometimes we’ll send out surveys or we’ll just meet one on one with them and try and actually use that to validate our own assumptions about what the biggest problems are that customers have.

When we designed our initial versions, I think quite quickly we started to ask experts in our field. Other people who were executives and leaders of data science at various companies or who were very familiar with it just drop in and critique our work. Look at our idea, tell us what they thought was important, what wasn’t important and that’s not expensive. We’re not going out and hiring ethnographic research or an agency. We’re just going and inviting people and other folks who have a deep understanding of the problem and the domain space and getting other external eyeballs on our thinking and our solution from the very, very early days. Right before we have the semblance of a solution, or before we’ve even written much code. We were trying to have a process like that and I think we will continue to do so even as we develop these first initial versions.

Amit Somani - 26:14

Let’s move towards the data part of this equation and in particular, again, the framework, we cited the two questions. Is it a real problem? Like measuring, have we made progress or made a dent towards solving this problem? Were there any obvious metrics or frameworks that you used other than increasing CTR or NPS goes up that helped you which may be relevant for others as well?

Julie Zhou - 26:39

Yes, the most important thing that I’ve learned about data is that this goes back to that third question, how do we know that we’ve solved it? I think what solving it means is oftentimes not like a metric going up. We always have to keep in mind that metrics are a proxy for the change that we want to see in the world. As an example when we came up with Facebook, our vision was that everybody in the world would feel connected. They would feel like they could easily access the people who were close to them. They would feel like they understood what was going on with the people who were close to them. They had tools to express themselves, however, they wanted, whether it was to their best friend or their parents or to their College Alumni Network, or to the public at large. We felt that maybe the best incarnation of our success would be if we could go out, and could talk to every single person, and we can ask them, well maybe not even ask them, because again, people don’t always tell you what it is they truly feel in the survey. But if we could actually, almost like, have a magic wand so we could read their minds. We could know to what degree Facebook and the tools help them feel more connected to the people in their lives. That’s actually fulfilling that mission would be.

The problem is, we can’t go and read people’s minds and we probably can’t even talk to everyone in the world. So we need to look at what is the best proxy that helps us understand if we’re on the right track or if we’re doing that. So, the first thing we have to do before we end up looking at any kind of data is, map out your vision. What does the change you want to happen? What does the world truly look like? If you can call the wave the magic wand technique, wave a magic wand, what’s different? What would people tell you? What would they feel? What would they do? Then you have to have that in your head first to then be able to say, Okay, now let’s figure out which metrics and I say metrics rather than any one metric. I’m a big believer that no single metric can truly tell you what it is to know. I think, again, we humans have this bias where we want that clarity when we make decisions, it would be great if we knew that we were successful if this one number went off. If we knew we would fail, if that one number, like there’s something very elegant, simple and actionable about that.

But the problem is, the real world is quite complex, there’s no one metric that is the end-all and be all. Every single metric is a proxy for what you really care about. And so the best thing that happens is you then need to look at probably a basket of metrics. This is where we get into kind of the art of, you know, choosing the right goal metrics, like choosing the right number, you don’t want 20 because you can’t even keep 20. I can’t even list off like 20 things in my head. If you had 20 and then some of them went up, and some of them now it’s very hard to get an accurate story in your head about like, how are things actually going? But you also don’t want one? And so the right art of it is like, how do you select the right basket that serves as a good proxy also with counter metrics and other weights that happen in place. So what are the pros you mentioned CTR, as an example? One of the problems with CTR is it’s every metric is probably easy to game. If you set that as a team’s goal. If I said, ‘Hey, we’re going to judge the success of this team on whether they can drive the CTR of this particular button, then they’re gonna make the button Red, they’re gonna make it flashing. They’re going to promise you like $100 if you click on this button. There are a lot of ways to get someone to click on the button. But that may not actually serve what is the purpose that your product is trying to do.

You probably don’t want to look at CTR but you probably want to look at something like retention. Retention is a very good metric.It’s something that I think is often underlooked.It’s usually often times when companies look at what is a good product, they look at growth. They look at how quickly it has grown and how many users does it have? How many did it have a month ago and for new products? I generally think you have to look at retention before growth because retention is more correlated if some of your product is really great and someone used it once, are they coming back? If they come back a lot, then that probably is a better sign that you’ve made something valuable, more so than if you were able to drive the top line growth number, because lots of people can be coming through the door and if they don’t buy anything or they don’t come back, then that’s not really good for you. It’s not really a sustainable business. And so regardless, there is some art to picking the right number of metrics, picking the right set of metrics that really encapsulates what you can, what you feel success is, and what the behavioural or outcome change that you want for people are.

The other art of the metric is that you want it to be operational. So the problem with retention; retention is great again, for knowing if your general product is good enough that people are going to come back. Part of the challenge is that it’s not terribly operational. And what I mean by that is, it’s very hard to run a small experiment, and have it actually impact top line retention. So it’s more of a lagging indicator. If you know you’ve got it or you know you don’t, you don’t have it, but it’s if you just run let’s say 10 or 20, minor experiments, it’s really not going to change top line retention much. Therefore, it’s not a great metric to help you understand how you should make micro level decisions like , ‘should I go with redesign A or redesign B?’ And so a good metric is again both a good proxy for your value, but it’s also something that is sensitive enough that you can use it to help you make decisions about specific launches, specific tests and specific other things. And because you do need to launch a test and you need to know like, ‘should I run them in this launch? Is this a good feature? Or is it not, and you need to look at something that goes up or down to help you make those decisions. So that’s another factor that goes into the art of choosing the right metrics or looking at the right data.

Amit Somani - 32:53

I love this. I don’t think I’ve heard this before that the right metrics are a proxy to measure the change you wish to see in the world, or do you anticipate to see in the world. Very interesting. I just crowdsourced a couple of questions from Twitter, right before we got on the air a couple of hours ago and there were several themes. So we won’t be able to cover all but I want to pick two. One is how do you drive communication between design, product, business, customer service and the whole nine yards? and maybe any kind of best practices, they’re both at the early stage, especially at the shipping velocity that you guys were at. And as you kind of scaled-up. So that let me just start with that particular one?

Julie Zhou - 33:35

That’s a great question. I think about it in sort of two dimensions. The first is, does everyone understand the ‘Why’ for why we’re doing and what we’re doing? It’s easy for us to communicate the ‘What’ but what’s more important is the ‘Why’. Does that mean everyone aligned on the problem that we’re trying to solve? Is everyone aligned on what this is meant to do for people? And if you think about it, designers do have a different language than engineers or PMs. What I mean by that is, designers often talk about things that may be engineers or PMS don’t really understand. This page needs to breathe better or this page needs to go and make cleaner or more minimal. Oftentimes, that doesn’t really resonate with an engineer or a PM. Similarly, a PM might say something like, we need to improve the CTR on this thing and that might not resonate with a designer. But what I think everybody can get behind is the language of users and what we’re trying to do for people. What is the change that we want people to see? How do we want people to feel? What actions do we want people to take? Do we want them to feel confused or not, or whatever it is, right? We want them to feel excited to make a purchase because ‘this is such a good deal’. Everyone can understand the impact on the ultimate customer. So I always find that’s sort of the foundational common ground language.

I think that if we’re talking past each other or we’re taught that we don’t understand the goal. Let’s go back to why are we doing this, we’re doing this because we want to do something for people, for our customers. Let’s talk about what we’re trying to do in the language of what we’re trying to achieve for people. So if we can do that, then it is much easier to actually talk about what the priority should be or if we disagree about this approach versus that approach, like who’s or which one has the best outcome for people? So that’s the first thing.

The second thing that also helps us again, to get to be able to get to a good, operationally rigorous culture. It does mean that things have to be tangible and that’s where you get to, why, it’s so important to set clear proxy metrics and to select a good set of proxy metrics. So again, everyone understands the story we’re trying to do. Everyone understands why these are the proxy metrics that we believe are operational, and that can help us understand whether things are good or not. It also everyone understands why they map to our version of success and that way, you’re most people will then be able to function with the idea of trusting these metrics and what they say about certain tests.

But you also can tell if one day you get a result where the metric went up. But your intuition about this is that maybe it’s not a great experience for some reason. And if you understand why the metric was chosen, and what the potential blind spots are with the metric that doesn’t quite cover it. You will then be able to still dig a little bit deeper to find the truth of whether this thing is good or not, experienced based on your understanding of the ‘Why’ and ‘Why” and how the proxy metrics were selected.

Amit Somani - 36:55

Great, many more questions, Julie. But this is a good segue into what you’re doing now at the Sundial, which is how to make kind of data-informed decisions using obviously data, gut, the why, proxy metrics, etc. So can you talk to us a little bit about melting these two threads of data and intuition? Maybe even an example or two, perhaps, to continue with the Facebook journey on where you made misinformed decisions. The kind of data that led you to the wrong answer? Or does that in your gut lead you to a wrong answer? So let’s talk about bringing these together now; data and gut.

Julie Zhou - 37:32

I have like 1000 examples of times when my gut was wrong. Mostly when it comes to things at scale when we actually launch our product to billions of people and again most of the average users are looking very different from my colleagues’ experience, living in the San Francisco Bay Area in America and working for a tech company. I remember the best example of this was a redesign of the web that we were thrilled about. We called it the Light stand, and the whole idea is that we were going to make the Facebook newsfeed experience a lot more immersive. That meant that we were going to widen the site and operating from like an 806 by 100 type of resolution. We know computer hardware had gotten a lot better and everything was faster for people to work on machines with better performance. So we needed to make everything bigger. We want to make it more immersive and this will be so amazing. We’ll be able to show photos so much richer than before. We spent a lot of time building this beautiful design and I remember, we presented it to the press and it was very well received and everybody was very excited about it.

So that we started to test it out to a particular market. Remember, we had learned our lesson now, we don’t just flip the switch, we go and just test it to like 2%, and every metric tanked. Everything that we cared about went down and I was convinced that this was a bug and something was wrong. It must be performance-related because maybe we just had a bug and made everything like a second slower and that’s why everything is going down. We went and just investigated every possible thing that this could be, and it was actually very difficult because we bundled hundreds of changes into just one giant redesign. That’s a big lesson for us, we never did that. Again, we always afterwards did redesigns in chunks so that we didn’t have to go in and try and claw our way back until which feature is the one responsible for why everything’s going down.

So it took us quite a few months to get to the bottom of it. What we realised was we made a big mistake by assuming that everybody had or wanted larger screens. A lot of the world was still operating off of these very small laptops with pretty limited resolution. On those laptops, if your friend posted a photo, it would not fit on the viewport. It was like the photo was just too big for your viewport which, of course, is a terrible experience for anybody. That’s what you’re trying to see for the news feed. And just so people got fed up and they just used it less because it was a worse experience. It sounds very obvious in retrospect that we should have realised this, but I think, again, the entire company was using very nice computers, big monitors. None of us had ever gotten one of these. None of our relatives or even maybe like our friends were operating in these laptops that were a couple of years old and that had lower resolution. But enough of the world was and still a significant enough chunk. This was frankly a much worse experience for them. We ended up having to basically redesign the whole thing again to fit for that. That was one very hard-learn lesson to do a lot more of this to break up large assumptions into smaller ones that we could understand and test much more piecemeal and to not get too precious about any beautiful redesign because again, it may not be representative of what the user base sees. It can only be appreciated by a very small fraction of that user base. So that’s one thing that we’ve learned.

I mean, there’s a lot of other things where we continue to use our intuition. One big area that I think really matters is whenever you’re coming up with a new product or you’re coming up with something new like a feature that didn’t exist before, data can’t lead you to the answer of what it is you should build. Now data can help you identify problems, it can help you identify where might be opportunities but it isn’t gonna help you invent the next thing. You still have to have some assumptions and still rely on certain observations are things where you were, ‘this is something that is a unique approach to this problem’, or this is something that hasn’t been done before. This is something that is similar to what someone else does but with a unique twist that helps us do this in a better manner. You still have a spark of what makes for a good experience. And for anything that’s zero to one, I’m still a firm believer that you can’t data analyse your way into what is the right thing to build.

So those are examples of how I think of the balance, but if you’re thinking about scaling and you’re thinking about iterating, you’re thinking about growing a thing, or you’re thinking about do people like icons, and labels versus just text labels versus icons in order to represent an idea. That’s the kind of stuff that you should just A-B test and the data will tell you which of these is more impactful. By the way, the answer to that is icons plus labels, which works better than the second-best is text. The third best is icons, as much as I, as a designer, love minimal icons, that is just not as effective or as usable for the vast majority of the world, at least at this point.

Amit Somani - 43:26

Wonderful. Julie, great, maybe just a quick summary of what Sundial is. I’m sure it’ll help lots of high growth companies leverage both data and intuition and figuring out the why as they go building, as we wrap up here.

Julie Zhou - 43:45

So you could tell that I am pretty passionate about this idea of data and design. What Sundial tries to do is that it is a storytelling product that helps product companies understand the most important narrative of their data and how to turn that into actionable insights. So the problem that we’re trying to solve because again, we start with people problems is that today, what happens is, we’re all logging everything. So we have tonnes of data and we have probably tonnes of dashboards but you and I can look at a dashboard of 16 charts, and we could walk away with a completely different interpretation about how our business is doing. You might be looking at one chart, it’s going up to the right, you’re patting yourself on the back, you’re happy. I’m looking at some other chart in our business, and it’s going down, and I’m really stressed out. And so even, though, we’re looking at the exact same raw facts, our interpretation of the data is different. If our interpretation of the data is different then it’s hard for us to align on what is the right product strategy going forward.

This is why companies hire data scientists. Their whole job is to go and look at this and try and contextualise the data and come up with the right narrative. Our product is trying to automate a lot of that work to make it easier for data scientists to not have to run the same automated repetitive manuals and to do that in a way that focuses on telling the story in an easy to understand and digestible manner. So data storytelling is a very interesting area for us. It’s for me to think about, even personally, from a design perspective. I’ve always been somewhat intimidated as a designer, looking at all these like crazy numbers and charts. I’ve had to learn, oftentimes the hard way, but I’m sort of very sensitive to just like the fact that it is hard to grok. If you don’t feel like you’re a quantitative person who understands statistics, it’s hard to understand and I think we should be able to democratise that and make it so that every builder knows what’s going on, knows the right story and then knows how to take action on it.

Amit Somani - 45:48

Thanks a tonne Julie. We wish you all the very best with Sundial. I’m sure you’ll do great and we look forward to using the product as it comes out. Thank you again for being on the Prime Venture Partners podcast.

Julie Zhou - 46:01

Thanks so much. This was so fun.

 

Enjoyed the podcast? Please consider leaving a review on Apple Podcasts and subscribe wherever you are listening to this.

Follow Prime Venture Partners:

Twitter: https://twitter.com/Primevp_in

LinkedIn: https://www.linkedin.com/company/primevp/