You fake it you do a fake door test you do a smoke test wizard of oz tests. We used a lot of those in the tabbed inbox by the way one of the first early versions was actually we showed the tabbed inbox working to people. But it wasn t really. Gmail. It was just a facade of html and behind the scenes and according to the permissions that the users gave us some of us moved just the subject and the sender into the right place. So initially the interviewer kind of distracted them and then showed them their inbox and then the top messages were sorted to the right place more or less if we got it right. And people were like. Wow this is actually very cool. But it gave us some evidence to go and say hey we should try and build this thing welcome to lenny s podcast where i interview world class product leaders and growth experts to learn from their hard won experiences building and growing today s most successful products today my guest is itamar gilad itamar is a product coach author speaker and former longtime product manager at google where you worked on gmail identity and youtube he also just published an awesome new book called evidence guided creating high impact products in the face of uncertainty itamar has an important perspective on why and also how you can push your team and organization from an opinion based decision making process to a more evidence guided approach in our conversation itamar shares a number of very practical and handy frameworks that do just that including the confidence meter metrics trees gist and the gist board plus his take on how people often misuse ice for prioritizing ideas also how you could make your okrs more effective and so much more enjoy this episode with itamar gilad after a short word from our sponsors this episode is brought to you by ezra the leading full body cancer screening company i actually used ezra earlier this year unrelated to this podcast completely on my own dime because my wife did one and loved it. And i was super curious to see if there s anything that i should be paying attention to in my body as i get older the way it works is you book an appointment you come in you put on some very cool silky pajamas that they give you that you get to keep afterwards you go into an mri machine for to minutes and then about a week later you get this detailed report sharing what they found in your body luckily i had what they called an unremarkable screening which means they didn t find anything cancerous. But they did find some issues in my back which i m getting checked out at a physical next month probably because i spend so much time sitting in front of a computer half of all men will have cancer at some point in their lives as will one third of women half of all of them will detect it late according to the american cancer society early cancer detection has an survival rate compared to less than for late stage cancer the ezra team has helped of their customers identify potential cancer early and of them identify other clinically significant issues such as aneurysms disc herniations which may be is what i have or fatty liver disease ezra scans for cancer and other conditions in organs using a full body mri powered by ai and just launched the world s only minute full body scan which is also their most affordable their scans are non invasive and radiation free and ezra is offering listeners off their first scan with code lenny book your scan at ezra com lenny that s. E z r a com lenny. This episode is brought to you by vanta helping you streamline your security compliance to accelerate your growth thousands of fast growing companies like gusto quorum quora and modern treasury trust vanta to help build scale manage and demonstrate their security and compliance programs and get ready for audits in weeks not months by offering the most in demand security and privacy frameworks such as soc iso gdpr hipaa and many more vanta helps companies obtain the reports they need to accelerate growth build efficient compliance processes mitigate risks to their businesses and build trust with external stakeholders over fast growing companies use vanta to automate up to of the work involved with soc and these other frameworks for a limited time lenny s podcast listeners get off vanta go to vanta com lenny that s v a n t.

Itamar thank you so much for being here welcome to the podcast it s a pleasure being here thank you for inviting me. It s my pleasure i thought we d start with the story of your work on google and gmail and how those experiences formed your perspective on how to build a successful product can you share that story google was my first experience at gmail i joined gmail in august and the first thing they asked me is let s connect gmail with google if you re hazy about the story back then facebook was massive. It s still massive. But then it was growing like mushrooms people were spending hours that really freaked out google and the obvious solution was to launch a social network of google called google. And we all believe in this thing it really caught on very well initially we all used it we all believed in it. So our mission was to build this thing and google really cut no costs it created a whole new division within google and it created a whole strategy around google and we had to connect gmail and youtube and search to google to make them more personalized in a sense and more social. So that was the idea and we went on and we launched a series of features in gmail for a couple of years honestly and google itself became this massive project very feature rich and with a lot of redesigns and iterations and none of it worked it turned out people actually didn t need another social network people didn t love it people didn t use it eventually in gmail we rolled back all the google integration a few years later and google itself was shut down in so putting aside all the tremendous waste that went into this all the millions of person hours and personal weeks. In hindsight not only did google bet on the wrong thing it missed much easier opportunities so just not far from google s headquarters there was whatsapp not very famous in the us. But they actually created massive impact hundreds of millions of people were using their stuff and they became a threat to facebook much more than google was so google missed the opportunity of social mobile apps like whatsapp like snapchat etc and for me this story kind of was the epitome of what i call today opinion based development we come up with an idea we believe in it all the indications show it s good. Maybe the early tests show it s good then we just go all in and we try to implement it. And i made this very mistake many times as the product manager i was the guy pushing for the ideas. So for me this was kind of a turning point i felt we need to adopt a different system and just before you move on to the next story how big was the team roughly how many years was spent on this area just to give people a sense of the waste as you said so there was a tremendous earthquake inside google to create the google team teams and the entire divisions were kind of thrown apart and reformatted and i think at its peak it was about people inside. Wow it was a division the size of android and docs and a really sizable thing they re under their own buildings it s taken from the playbook of steve jobs create this whole secretive project inside and just run like hell. Yeah i remember though facebook was really scared i remember they shut everything down it as like a code defcon one situation too so it really scared facebook at the same time. Yeah it s true. But at the end of the day neither google s advertising revenue was affected neither was facebook affected so it turned out this idea was not that necessary after all.

That didn t work because it was opinion based software i think the phrase you used and then there s a different experience with tabs i think with gmail that s. Right. So google is a very successful company it s not for me to criticize it or to in hindsight kind of say you guys need to be better and some of the people that were behind google was some of the smartest leaders and i still think they are despite this story if you look back at the history of google how things started in the first decade or so google was what i call an evidence guided company. So essentially it put a high premium on focusing on customers coming up with a lot of ideas on looking at the data looking at how these ideas actually worked out they weren t shy about launching betas and things that were very rough and incomplete and learning from that and then they expected people to take action based on the results so fail fast is a very famous paradigm and so you had to kill your project or pivot it seriously if it didn t work out. And i think had we kept fail fast it would ve really have helped google if we had this mentality. But for some reason with google put this playbook aside and used a different playbook which i call plan and execute essentially. But i think inside google the dna still existed so inside gmail the next project after google was the tabbed inbox. So it was kind of the reverse of google it started as a very small idea that no one believed in and we started looking what s behind the city what s the goal what s the problem actually we re trying to solve it turned out that a lot of people were receiving social notifications and promotions etc and most of them were very passive they weren t clearing their inbox they were just living in this world of clutter and i came up with an idea how to fix this i was sure it was great i wanted to push it plan and execute but my colleagues were like hold on we actually tried this we have a bunch of ideas to help people organize their inbox they re not using it why is your idea good. So that sent us kind of me and my team into researching these users into establishing a goal that was much more user centric and then thinking of other ideas. And then we started testing them much more rigorously and basically we started testing on our own inboxes. And then we recruited other dog footers other googlers to test the same inbox then we put it outside for external testers we did usability studies we did data we built a whole data mining team and a whole machine learning team to build the right categorization and we ended up with a solution that turned out to be very successful for a lot of these passive users this was a surprise to a lot of people because most of my colleagues and most of the people i talk with actually know how to manage their inbox so for them that solution makes complete nonsense like splitting promotions and social to the side sounds like the stupidest idea. But there s about of the population to that absolutely love it and today gmail has about billion active users according to gmail most of these users are using this feature so it was a pretty high impact feature as well and the feature specifically just in case people aren t totally getting it is the promotions folder and the social i think and then the regular. Yeah there are a couple more that you can enable in settings if you like. Yeah i use it i love it except it puts my newsletter in people s promotions folder who do i talk to about that. Yeah newsletters are a very complicated scenario for the categorization engine. Yeah we just need an exception for my newsletter. And then we re good. Okay. But go on so in hindsight i was asking and saying why was this project so different. And i think the reason is that we didn t have that much confidence in our opinions we had opinions we had ideas. But we didn t just go all in and just let s build it. We actually used an evidence guided system and i think that s not unique just to google i think every successful product company out there that you look at amazon airbnb anyone you will check at least in their best periods they found a way to balance human judgment with evidence. They didn t try to obliterate human judgment and opinion just to supercharge them with evidence and they came up with very different models apple is another example but the principle still holds in all of these companies. Awesome so you took that experience and all the experience you ve had from coaching product leaders working with companies and you wrote this book called evidence guided.

On youtube could see sitting there behind you. So i want to talk through some of these stories and then some of these other lessons and frameworks that emerged. But maybe just to start what s the elevator pitch for this book. So this is a book for people like us product people who want to bring evidence guided thinking or modern product management if you like into their organizations there s a lot of challenges it s not simple we all read the books we all know the theory we all know some parts of the system it tries to give you a system how to do that it s a meta framework that kind of helps you lift your organization in the direction of evidence guidance if that s what you want to do so going back to the story briefly before we get into the frameworks and lessons of the book in the first example of google basically it came top down hey.

We need to build a social network go build it obviously that happens at a lot of companies i don t know if there s an easy answer to this but are there cases where it does make sense to approach it that way obviously apple is a classic example of steve jobs is like we need to build an iphone i don t know if that s exactly how it went but are there instances where it is worth just approaching new product ideas that way based on the experience and creativity and insights of the founder or is your thinking it should always come from this evidence based approach i think the founders are very important especially in the startup and scale ups phase they come up with many of the most important ideas and it s super important that they have the space to express and to push the organization to look at those however it s not about shutting them down it s about looking at them critically you need to create the environment in the organization where the leader comes and says you know what i talked to these three customers i figured it out here s what we need to do in the next five years. And you need to ask where s your evidence and by the way the example you give that s a classic example steve jobs he just brainstorm in his kitchen the iphone and then just told the team to build it that s the story steve jobs told but it s not the real story at all now we know what actually happened and the iphone has actually a story of discovery of trial and error multiple projects to do it multitouch with phones most of them failed steve jobs was the architect he kind of managed to connect the dots and eventually come up with this perfect device. But he wasn t actually the creator it wasn t his brainchild. He was actually against it for a while but over time as he saw the evidence as he saw what this thing can do as he saw the demos he was able to piece together something that was very useful that s really important insight people that are hearing this might feel like i like this idea of pushing back and encouraging the founders to make it more evidence guided in the case of say google was it even possible could you have come to larry and sergey and be like here s all this data i ve gathered that tells us this is not going to work. Do you have any advice for how to push back and encourage the founders and execs to really take the counterpoint seriously or really kind of vet their idea so another nice thing about google is that it s a very open culture and people are not shy to tell even sergey and larry that they are wrong. And they do this all the time in certain forms right you need to know the right channels but there was a very big discussion about google and whether it s the right thing to create a clone of facebook there was a very public internal discussion i think what i would change is not have this discussion based on opinions because when you have the discussion you come with your own opinions usually the most senior person s.

Opinions will win that s just the way it is if we had come with hard data and we said listen things are not actually panning out the way you guys are expecting what can we do should we continue should we pivot this i think the discussion would ve done better. Now i m doing a huge disservice i was not in all the discussions i know probably in google there were very serious discussions happening along these lines but it s just as a general trend i find that evidence is very empowering for us smaller people in the organization or mid level managers to be empowered to challenge the opinions is there anything tactically you found to be useful and effective in giving people say they don t work at google. They work at companies where founders and bosses and execs are not as open to challenge any tactically found about how to present a counter proposal or. Like. Hey i have this data that we should really pay attention to i think if you come with data if you run a secret experiment and you come back. And you show them you usually get one of two results either they get extremely mad at you and they tell you to get back to work and to do what you were told and in that case probably you need to start polishing your resume and look for another place either inside the organization or outside it because that person is not being reasonable to be honest but the more common case is they re pleasantly surprised and that s what happened with steve jobs as well he was against phones but then people showed him all sorts of evidence that apple can make a phone he was against multitouch initially. But then he changed his mind there was a lot of back and forth so even steve jobs given evidence was willing to flip. And i say this in many organizations so evidence is so powerful that s why this is the principle i based the book on you have this concept of being evidence guided people listening may feel like hey we re evidence guided we re in experiments we make decisions using data oftentimes.

Aren t actually. And so what are signs that maybe you re not actually that evidence guided or as evidence guided as you think you are i think there s a few telltale signs that i look for first the goals are very unclear either there are many or they re very kind of obscure and vague or they are about output there s misalignment. So the goals part is not there usually this goes hand in hand with metrics missing metrics or just using revenue and business metric but there s no user facing metrics so that s another telltale sign then there s a lot of time and effort spent on planning especially on road mapping creating the perfect roadmap which really can consume a lot of time of the top management and pms etc then as you go down you see there s not a lot of experimentation and if there is experimentation there s not a lot of learning and finally another telltale sign is that the team is disengaged so the engineers are kind of getting the signal that what they need to do is. Deliver they re focused on output that s what they re measured on so they re kind of disengaged they re disengaged from the users from the business they don t care. That much. It s usually something that you can fix by adopting a more evidence guided system. Okay. So let s dive into your approach to becoming more evidence guided in the book you share this.

Model that you call the gist model which is kind of this overarching approach to building a product that almost forces you to be more evidence guided so let s just start with what s the simplest way to understand this gist model with your permission i can show a few slides. Oh let s do it. And maybe that will help here we go. And then yeah a good excuse to go check this out on youtube all right you re seeing this so this is the gist model goals ideas steps and tasks and essentially it s tries to break the change which is a really big change for a lot of companies into four slightly more manageable parts they re still big but each one you can tackle on its own and that s kind of the reason i kind of split it and goals are about defining what we re trying to achieve ideas are hypothetical ways to achieve the goals steps are ways to implement the idea and validate it at the same time so essentially build measure learn loops and tasks are the things we manage in kanban and jira and all these good tools these are the things that your development team is usually very focused on and just listening to this a lot of this will sound familiar to you because gist is not a brand new invention it s a meta framework that puts in place a lot of existing methodologies it s based on lean startup on design thinking product discovery growth there s a lot of all of these things here it just tries to put them all into one framework or one model. So what s the simplest way to think about what this model is meant for is this how you think about your roadmap is this how you plan what is this trying to tell people to do differently in the way they build product broadly i would say these are four areas that you need to look at and ask are we doing the right thing in each in each you may need to change or even transform and as i go and explain each one of those i l give you basically three things in each chapter in the book i try to touch on three things the principles behind them the frameworks or models that implement the principles and then process and the process honestly is the most brittle part and the one that you would need to change and adapt to your company because not two companies are exactly the same and it s very tempting when you write a book not to give any process but that s the part that people actually want the most so it s included as well but just be aware that you will have to change this process. Awesome. Okay so we re going to talk about each of these four layers before we do that where do vision and strategy fit into this do they bucket into one of these four layers and how.

Do you think about strategy and vision that s a great question so there s this whole strategic context that is outside of gist is not trying to tackle that it assumes it s in place there s another huge blob which is research gist is not about research it s more about discovery and delivery but strategy is extremely important and you can use some of the tools we will talk about to develop your strategy as well in many companies the strategy is just a roadmap on steroids it s small plan and execute just on a grand scale and google again was a strategic choice actually if you think about it. So in the book there is a chapter where i touch on strategy and i explain how the same evidence guided methods are being used by companies to develop their strategy as well awesome maybe one last context question so people might be seeing this and thinking. Okay cool. I have goals i have ideas steps i have tasks i m already doing this what is this kind of a counter or reaction to what are people probably missing when they re seeing this and they re like. Oh i see this is like what we re not doing and this is the most important this is something we should probably change. And we l go through these in detail too. I think talking about each one will help okay let s do it. But we can talk about in each level what s actually being done. So when people say i have goals usually they take the goals layer and use it as a planning session they talk about what shall we build by when what are the resources and that s actually not goals at all that s planning work cool let s talk about goals and i know part of this is okrs related too so i m excited to hear your take on okrs oh that s a whole different discussion you had christina the real expert over there so i doubt i can add more to that. But it s true okr is all part of it but let s start with goals what s our goals supposed to be goals are supposed to paint the end state to define where we want to end up and the evidence will not guide you unless you know where you want to go and in many companies what you have is goals at the top for revenue market share whatever it is. And then a bunch of siloed goals for each department there s engineering goals there s design goals there s marketing goals etc and that actually pushes people into different vectors and it s really hard to decide i would argue that in evidence guided companies and you ve worked for a few so probably you ve seen this they use models in order to construct overarching goals for the entire organization one of the models i show in the chapter about goals is the value exchange loop where basically the organization is trying to deliver as much value as it can to the market and to capture as much value back and. By creating a feedback loop between these two you are actually able to grow very fast now i would argue that you want to measure both of these and to put a metric on each and the metric we usually use to measure value delivered is called the north star metric i know you wrote an article a very good article about it thank you. And in it you listed dozens and dozens of companies like leading companies and what they considered the north star metric is super interesting i would argue that what they told you is what is the most important metrics we measure what is the number one metric for us. But it s not what i call the north star metric the north star metric measures how much value we create for the market for example let s take whatsapp for a very long time measured messages sent because every message sent is a little incremental of value for the sender the receiver it s free it s rich media you can send it for anywhere in the world compared to sms that s huge value. So if in year one we have a billion messages being sent in year two billion probably we doubled the amount of value in airbnb i think one of your key metrics or the real north star metric was nights booked. I don t know if it was still the case while you were there. Yeah absolutely. And there are examples like this in amplitude for example they measure active learning users or weekly active learning users which are users that found in the tool some insight that was so important that they shared it with at least two other users and they consume it. So it s a very powerful thing to point at this metric and say this is the most important metric combined with the value metric that we want to capture revenue market share whatever it is once you have these two you can further break them down into what i call metrics trees. So there s a metric three for the north star metric and there s the metric three for the top kpi the top business metric which you see here on the left side in blue and usually they overlap so.

You might find in the middle some metrics that are super important because moving them actually moves the needle on everything else can you clarify again the difference between what you call this top kpi versus north star metric. So the north star metric is measuring how much value we re creating for the user the core value that they re getting in this case this is some productivity suite so this is number of documents created per month for example because we think that every document created maybe it s a small document i don. T know. Ai is in fashion now is a little incremental value so that s the number we re trying to grow the top kpi is what we expect to get it should be revenue or profit. I see this is the value exchange i see one is what users are getting one is what you re getting back from them exactly basically how the business is benefiting awesome. I think this is a really important concept the metric tree i think a lot of people think they have something like this in mind where they re just like cool here s our north star metric here s the levers and things that we can work on to move that. But i think actually mapping it out the way you have it here where it kind of goes layers and layers deep to all of the different variables that impact this metric not only is it a way to think about impact and goals and things like that but also helps you estimate the impact of the experiment you re potentially thinking about running. So if you re going to work on something at the bottom here like activation rate say you move that how much is that going to impact this global metric. It s probably a very small amount this is a very important one. And we l talk about impact assessment shortly this helps with it also helps with alignment because the entire organization is trying to move these two metrics it s the two sides of our mission essentially we have the mission that s the top objective of the company and these are the two top most key results if you like the top most things so when you go and work with another team and you say hey why don t you work on my project they might say this idea actually might move the north star metric model in your idea and that helps you guys align and i ve seen cases where team b put aside their own ideas to jump on the ideas of team a because of this model it also creates an opportunity to give some sub metrics to teams to own on an ongoing basis so it creates a little sense of ownership as well and mission within the tree it also helps you figure out what teams you should have which teams have the biggest potential to impact the metric another thing that happens in a lot of organizations the team topology reflects the structure of the software or some hierarchical model where we want to organize the organization in a particular way. But if you start with a metrics tree you can try to arrange the topology around goals and sometimes you need to readjust it s not a constant reorg but from time to time you will realize the goals have changed and we need to reorganize so the tree helps visualize that as well i think for people that are listening to this and thinking about this i think the simplest way to even think about this is basically there s a math formula that equals your north star metric or your revenue or whatever you re trying to do and if you don t have some ideally really clear sense of what that math formula is you should work on that because that will inform so much of how you think about where to invest what teams to have where to invest more resources less resources. Right imagine a place where you can find all your potential customers and get your message in front of them in a cost efficient way if you re a b business that place exists and it s called linkedin ads allows you to build the right relationships drive results and reach your customers in a respectful environment two of my portfolio companies webflow and census are linkedin success stories census had a x increase in pipeline with the linkedin startup team for webflow after ramping up on linkedin in q. They had the highest marketing source revenue quarter to date with linkedin ads you l have direct access to and can build relationships with decision makers including million members million senior execs and over million c level executives you l be able to drive results with targeting and measurement tools built specifically for b in tech linkedin generated two to five x higher return on ad spend than any other social media platforms audiences on linkedin have two times the buying power of the average web audience and you l work with a partner who respects the b world you operate in make b marketing everything it can be and get credit on your next campaign just go to linkedin com podlenny to claim your credit that s linkedin com podlenny terms and conditions apply. Okay. So metrics trees what comes next. All right. So next we need to go to the ideas layer and the ideas layer is there to help us sort through the many ideas we might encounter.

And they may come from as you said the founders the managers the stakeholders from the team from research from competitors we re flooded with ideas and what usually happens inside organization is some sort of battle of opinions or some sort of politics sometimes or highest paid person s opinion you had ronny kohavi who invented this term in your show what doesn t happen is very rational logical decisions these are the best ideas because it s really hard to predict honestly there is so much uncertainty in the needs of the users in the changes in the market in our technology in our product in our own organization. It s almost impossible to say this idea is going to be the best but we do say this because we have cognitive biases that kind of convince us that this idea is far superior to anything else and it s definitely the right choice in order to avoid this what we want to do is to evaluate the ideas in a much more objective and consistent and transparent way in the book i suggest using ice impact confidence and ease i think i have a slide coming on this so impact confidence and ease which is basically a way to assign three values to each idea the impact tries to assess how much impact it l have on the goals and that s why it s so important that we have very clear goals and not many how we are measuring the ideas on the north star metric on the top business kpi on a local metric of the team whatever it is let s be clear about it and then let s evaluate the ideas against this thing ease is basically the opposite of effort how easy or hard it s going to be but both of those are guesstimates both of those are things we need to estimate i would argue that just by breaking the question to these two questions we usually have a slightly better discussion than just my idea is better than yours. But then there s the third element which is confidence which tries to assess how sure are we or should we be about our first guesstimates about the impact and the ease it s interesting you use the word ease because i think it s usually effort you kind of make it positive is that an intentional tweak you made i m using the definitions of sean ellis sean invented ice you know. Sean i don t know if you ve had him yet. But he s i haven t had him on. Yet. Yeah. For the people who don t know him sean is amazing. He s like one of the fathers of the growth movement he coined the term growth hacking. And he popularized the concept of product market fit. Yeah he created ice he created a bunch of things that we use in product that we don t even know. Wow i. Didn. T know he came up with ice. Okay. Cool. So the original version of ice is ease instead of effort. Exactly yeah fun fact a lot of your viewers are wondering where s the r because there s another variant of this culture rice where there s rich as well i prefer ice because i prefer to fold the rich into the i for various reasons but both are valid both are equivalent in a sense i m in your boat that s exactly how i think about it. I think people over complicate this stuff and try to get so many math formulas involved with estimating impact and i feel like these are just simple heuristics to kind of bubble the best ideas to the top it doesn t have to be a perfect estimate of impact and confidence and all those things. So i think the simpler is better and it always ends up being a spreadsheet people always have these tools to estimate these things. But it s like a spreadsheet google sheets. Great. So. Yeah you re actually leading me to my next point. So when you come to estimate impact you will realize it s the hardest part. So sometimes it s just a gut feeling and it s a guess and sometimes it s based on some spreadsheet or some analysis and the back of envelope calculation.

You ve done and i think that s legitimate sometimes these things do show you some things you didn t think of and sometimes the best case it s based on tests you actually tested it you interviewed customers you show them the thing and out of those only one actually liked it you should reduce your impact based on that usually or you do other types of tests we l talk about testing in a second what happens is that people tend to just go with gut instinct and then give themselves a high confidence they say it s an eight and i m pretty convinced so it s eight for confidence and i found this a bit disturbing because it kind of subverts the whole system. So i wanted to help people realize when they have strong evidence in support of their guesses and when it s weak evidence how to calculate confidence in a sense for that i created a tool called the confidence meter which you can see here this colorful thing and should i go and explain it. Yeah let s do it. And then again if you re just listening to this you can check this out on youtube and you can see the actual slide. All right awesome. So basically i constructed it a bit like a thermo meter it goes from very low confidence which is the blue area or the upper right all the way to high confidence which is the red area and you can see the numbers going from zero to where zero is very low confidence we don. T know basically anything we re just guessing in the dark and is full confidence you know for sure this thing is a success no doubt about it and across the circle i put various classes of evidence you might find along the way. So for example starting at the top. Right all of these blue areas about opinions it could be your own self confidence in the idea your self conviction you feel it s a great idea guess. What behind every terrible idea that was ever someone thought it was great that gives you out of maybe you created a shiny pitch deck or a six page document that explains in detail why this is a great idea slightly harder to do but still very low confidence maybe you connected it to some theme it s about the blockchain. Well sorry the blockchain is out of fashion what s hot right now ai. Exactly. Ai it s about ai that makes it a good idea absolutely not or the strategy of the company that s another thematic support thousands and thousands of terrible ideas are being implemented right now as we speak based on these themes so all these things combined can give you a maximum out of according to the tool if you follow it then we move into slightly harder tests one is reviewing it with your colleagues your managers your stakeholders the idea they don t know it either they don t have a crystal ball they re usually not the users they cannot predict. But they can evaluate it in a slightly more objective way and maybe find flaws in your idea on the other hand groups tend to have biases too politics group thing so groups can actually arrive sometimes with worse decisions than individuals there s some research to that next our estimates and plans so you may do some sort of back of the envelope calculation or your colleagues might go out and try to evaluate the ease a little bit better that gives you a little bit more confidence but still we re at the level of guesswork at this point next we re moving to data and data could be anecdotal. So you find a few data points dotted across your data or you talk to a handful of customers or maybe one competitor has that same idea. In many companies i meet if the leading competitor has this feature and we think it s a good idea validation is done let s launch it that s it s a great idea we need to do it never works honestly you should not assume that your competitor actually knows what they re doing anymore than you do data could be also what i call market data that comes from surveys from assessing a lot of your data by doing a deep competitive analysis and there are other methods where you create a larger dataset and you contrast your idea against it finally to gain medium and high confidence you really need to build your idea and test it and that s where the red area is so there s various forms of tests we l talk about them if we have time and they give you various levels of confidence awesome this is a very cool visual we l link to a image of this in the show notes too if people want to check it out i think what s awesome about this is you could just use this as a little tool on your team of just like where are we along the spectrum we think the impact of this is very high. But we re probably in this blue area of confidence and so let s just make sure we understand that and it s really clear language to help people understand i see if we had this. It d be a lot more confident so you can also tie your investment into the idea based on the level of confidence you had found essentially so early on you want to do the cheap stuff just to gain more confidence. And then you can go and invest more if it s a really cheap idea you can jump to a high confidence idea you can test you can do an ab experiment early adopter program whatever it is and then launch it some ideas you don t need to test sometimes the expert opinion is enough if you re just changing the order of the settings no one sees this or no one will be impacted the risk is low you can launch it without testing so part of the trick is also knowing when to stop not just trying to force your way all the way up when you don t have to that s a really important point the other important point here is just a big part of a pm s job is to say no and to stop stupid shit from happening and this is an awesome tool to help you do that to be like. Okay here s this idea. You have just like let s just be real how confident are we in this. And okay it s going to take us three months to do this maybe we should think about something different maybe we should work up the confidence meter before we actually commit to this. Yeah. This is a real world usage that i hear about a lot some people use this to kind of do an objective way to say no and gently or to say we l think about it but look at these other ideas we have and how their impacting is and confidence stack up classic pm move just like that was a great idea but what about this better idea coming back to something that we talked a bit about at the beginning say you have a founder who s actually very smart and experienced say even at a startup where you don t really have the time to build tons of evidence for.

Ideas do you have a different perspective on how much time to spend building confidence in ideas versus just like cool they actually have really good ideas let s just see what happens so there s always like a trade off between speed of delivery and speed of discovery and that actually leads to the next layer of how do we combine the two because people tend to think it s an either or either we are building very fast or we are learning and then we re building very slow. But i think we re using the wrong metric the metric is not how fast can we get the bits into production when there s a lot of uncertainty and we all face uncertainty and startup especially it s not about getting the bits to production it s about getting the right bits to production it s about creating the outcomes that you need the impact. And so it s about time to outcomes. And i would argue that the evidence guided method is far more impactful it s far faster it s far more resource efficient than the opinion based method because opinion based methods tend to waste a lot more of your resources building the wrong things or discovering learning too late. Well evidence guided helps you learn earlier plus it is a fallacy that if you learn you don t build good teams know how to do both at the same time and that s actually what the steps layer is meant to teach you or to help you do awesome. So maybe just to close off that loop say someone listening is at a bigger company say netflix versus a series a series b or startup is there something you d recommend about them.

Approaching this differently any kind of guidance there of just how to take what you re sharing differently if you re a different source of companies like that absolutely i think the concept we talked about of the north star metric the value created versus the value captured is very important in every company building your entire metrics trees maybe overkill doing heavy weighted okrs may be overkill for early stage early stage companies even don t know how they create value so they need to iterate and their goals is really to find product market fit beyond that what happens is that you need to start building your business model. So that s your goal and you iterate towards that and you need to put metrics on that. And then when you move into scale you need to try to create order because when you scale up. And all of this is covered in the book there s a special chapter just about these questions when you scale up you get a lot of people and a lot of money and everything is happening at the same time. So there you need a order of evaluating ideas in a very systematic way in a company like netflix by the way i don t know if they need this specific method they re very. Yeah maybe that was a bad example they re probably doing things pretty well. One thing i discovered by the way there s two types of companies that really benefit from this technique one is those companies that are kind of emerging into modern product development they have product teams they have product managers they have okrs they re starting to do agile. But they re starting to do experimentation but they re struggling to put it all together every cpo is building their own little framework and the other type is those companies that used to be evidence guided and they regressed and that happens way too often change of management change of culture and then all of a sudden they need to rediscover to rekindle that spirit that was lost along google. So some of the people that actually respond to the strongest are actually surprisingly in these companies what i love about your frameworks and kind of all these things we re talking about is these are just a you can almost think of them as a grab bag set of tools to make you more evidence guided as a company you could start with thinking about the confidence meter you could start using ice more you could start using the metrics tree and all these things just push you closer and closer to being more evidence guided you don t have to adopt this whole thing all at once absolutely i would recommend that you don t try because if the transformation is way too big you will get fatigued and you will just create a lot of process for a lot of people and you would not see the results and after a quarter you l give up so exactly what you suggested is the right approach what would be the first thing you d suggest if people were trying to move closer to being less opinion oriented and more evidence based which of these frameworks or models would you recommend first i recommend that they discuss internally where is the biggest problem that they re facing if.

The goals are unclear there s misalignment we keep chasing the wrong things start at the goals layer try to establish your north star metric your top business metric your metrics trees start assigning teams with their own area of responsibility if you re spending a lot of time in debates and you re constantly fighting and changing your mind start with the ideas there and establish impact is confidence or whatever prioritization model you like but involve evidence in it i think the confidence meter is a good tool to use irrespective if you re building too much and you re not learning enough start adopting the steps layer which we haven t seen yet and if your team is very disengaged you have one of these teams where the developers are very into agile very into quality very into launching things start working on the tasks there. Awesome okay let s keep going all right so steps are about kind of helping us learn and build at the same time as we said and one of the patterns i see is that organizations don t know.

Learn at a much lower cost they believe they need to build this elaborate mvp which is not minimal in any way and then launch it. And then they will discover it. And basically it s what we used to call beta years ago but just with a different name what i m trying to do here in the steps layer is to help companies realize there s a gamut of ways to validate your ideas or more specifically to validate the assumptions in your idea. And i created a little model for this it s called after assessment fact finding tests experiments and release results but again it s just putting together things that much smarter people invented so in assessment you have very easy things that don t require a lot of work you check if it aligns with the goals this idea that you have in your hand you do maybe some business modeling you do ice analysis you do assumption mapping which is great tool by david j blend or you talk to your stakeholders one on one just to see if there are any risks etc these are usually not expensive things and they can teach you an awful lot about the impact and the ease of your idea the next step is to dig data and usually that goes hand in hand with this. So you can find data in your data analysis through surveys through competitive analysis through user interviews and through field research observing your users obviously these last two are pretty expensive. So it s often good not to wait until you have the idea and then start doing your research it s best to keep doing your research ongoing. And then you have some sort of data to lie on and to compare your idea against but until now we didn t build anything now you re ready to start testing building versions of the product and putting them in front of users and measuring the results. But initially you don t build anything you fake it. You do a fake door test you do a smoke test wizard of oz test a concierge test usability test we used a lot of those in the tabbed inbox by the way one of the first early versions was actually we showed the tabbed inbox working to people but it wasn t really. Gmail. It was just a facade of html and behind the scenes and according to the permissions that the users gave us some of us moved just the subject and the sender into the right place. So initially the interviewer distracted them and then showed them their inbox and in it the top messages were sorted to the right place more or less if we got it right and people were like. Wow this is actually very cool. And that gave us a lot of evidence that s an awesome story so that was in the user research it wasn t rolled out to people it was a manual individual there wasn t a single line of code written this was just cooked up by the researchers and our designers. But it gave us some evidence to go and say we should try and build this thing love that. So initially you fake it mid level tests are about building a rough version of it s not complete it s not polished it s not scalable but it s good enough to give to users to start using so those are early adopter programs alphas longitudinal user studies and fish food fish food is testing on your own team fish food. I haven t heard that term before. So it s dog fooding but more local to your team. I think it s a googly thing but some people told me that they use fish food as well in their company the name. So i m using it. I don t know if there s a better name for it i wonder why it s called fish food because it s like little it s like little gentle little clicks.