DISQUS

Andrew Chen (@andrew_chen): Built to Fail: How companies like Google, IDEO, and 37signals build failure-tolerant systems for anything!

  • rogervalade · 4 months ago
    Nice article, Andrew. If you don't mind, I'll be making a reference to it from my blog, Fail Fast (http://failfast.me) -- it is very relevant to the theme I'm interested in!

    Roger.
  • Vladimir Oane · 4 months ago
    The fail fast idea is quite stupid I think. You should spend more time on it and iterate fast. Don't quit just fast....
  • MicahWedemeyer · 4 months ago
    A little on the harsh, but I agree. Give yourself a little time to see if you're on the right track. Real success takes years, not a couple days.
  • Kevin Shaum · 2 months ago
    I think you are mistaken about what the term "fail fast" means:

    http://en.wikipedia.org/wiki/Fail-fast
  • cedric · 4 months ago
    Interesting point. Perhaps a better example with Google would be their 20% policy, where engineers are encouraged to pursue their own ideas. Most of it is discarded, but a few projects turn into highly successful products (gmail & co..)
  • Kamal Syed · 4 months ago
    This sounds very random.

    There are a lot of analysts that would wholeheartedly disagree with this concept.

    I think its more likely that this just demonstrates that relationships and planned outcomes are a lot more complicated than we can expect or perhaps even understand fully.

    "Planning to fail" is a key part of contingency planning and risk mitigation, but its a very different thing than expecting random outcomes (particularly in personal relationships).

    MKS
  • Vincent Chan · 4 months ago
    As Ram Charan said: Failure is a fact of life for companies that pursue innovation seriously and leaders should know that failures represent opportunities to learn.

    The key should be finding the right metrics to measure failures and learn from them quickly. If you have read "The Game-Changer" by A.G. Lafley and Ram, you will find out even P&G has adopted similar approaches.

    Additional information on how failure breeds success, including the examples of GE, Intuit, Coke...etc:

    http://www.businessweek.com/magazine/content/06...
  • jonziskind · 4 months ago
    Andrew -

    We are scaling up our company right now and pressing forward and this piece is fantastic. Thanks.

    Jon at zhiing
  • Jagtesh · 4 months ago
    Good article, Andrew.
  • samnet · 4 months ago
    I agree and it's sort of a learning process. How many times did you fall down before you learned to walk or run?
    You didn't just one day say -- ok lets start walking today!
  • peter_zaballos · 4 months ago
    Great post Andrew;
    A nuance I think you touch on is that it's not so much that organizations who embrace the reality of failure are motivated by a fear of not succeeding, they know that the knowledge gained from failure only speeds them to success. So, yes, they love the idea of lots of low cost experiments because its the failure as much as the success with these that will provide a durable advantage. IDEO's "structured brainstorming" is a wonderful "operationalization" of this philosophy.

    Organizations who fear failure build in a structural inability to be nimble, to learn, and to make the critical adaptations rapidly that will get them to even be in the same game/ They may succeed at not making a "mistake" but winning that battle will lose them the war.

    I wrote a post on this theme of "Lots of Low Cost Experiments" earlier in the year, which touches on some of the key points in your post today. http://openambition.com/2009/04/22/lots-of-low-...

    Pete
  • Warren Benedetto · 4 months ago
    You might want to watch the first few minutes of Jason Fried (37 Signals) giving the keynote at BigOmaha recently: http://vimeo.com/4717683?pg=embed&sec=

    Based on his comments about not embracing failure, I have a feeling he might disagree with your thesis about his company. Maybe you should consider updating the post to reference Ruby On Rails specifically, rather than 37 Signals. That seems to be a more accurate comparison.
  • Stephane Rodet · 4 months ago
    I just limit myself to the Google argumentation:
    Well, it's a different architecture - different needs too. The mainframe is much more expensive, so it all comes to requirements. If one Google box fails that references 10000 of websites, and these are absent of a search result, that's no big deal. Also if 1000 Gmail users can't log in during 15 minutes, that's no big deal - no one is going to be fired or to sue Google for that.
    Now think about 1000 of transactions on the NYSE that fail. That's another type of consequences. That signifies $$$ of losses. Or would you like your monthly paycheck to be lost randomly? Probably not. These are the case where you need special hardware. It's fault tolerant, complex & expensive, because it would be even more expensive not to have that degree of tolerance-fault. That's why both architectures coexist. There is not a "one fit all" solution.

    BTW, some study has shown that people that have a plan B tend to be less depressive because they can adapt better to the new reality... So failure has to be in the plan, too.
  • ssaikia · 4 months ago
    WHO IS YOUR TARGET AUDIENCE FOR THIS POST?

    The concepts outlined in this post probably apply to bigger organizations that have significant resources. A small resource-constrained startup is not going to plan for failure and build the redundancy that you prescrible.

    NOT a good post Andrew - it does not sound like you know your material! Sorry for this harsh criticism.
  • Happy Cloud Moments · 4 months ago
    Hi Andrew,

    Thanks for stopping by my blog! You have some interesting articles here. If you need to reach me, you can email me at happycloudmoments at gmail.

    Have a great night!
  • Andrew Hedges · 4 months ago
    There is a corollary here with different styles of dog training (believe it or not). Some people practice corrective training where the dog is "punished" (usually verbally) for incorrect behavior and praised/rewarded for correct behavior. Another style, using "markers", uses rewards (sometimes praise, but more often food) for correct behavior and verbal markers for feedback when the dog is doing either the wrong thing or is on the right track.

    The difference is subtle between "no" as feedback and "no" as correction, but the result is dogs that are problem solvers rather than ones afraid to try new things for fear of being punished. The goal of any company should be to "breed" problem solvers. You do this, as Andrew points out, by planning for, accepting, and even celebrating failed attempts to solve problems.
  • mxaddison · 4 months ago
    Andrew -- I can say with experience, at least to the part about marriage, that if you have pure passion for each other you'll figure out the rest. Perhaps the same holds true for entrepreneurship.
  • Martin Thomas · 4 months ago
    Interested in your thoughts.. Does 37signals or E-myth have the right philosophy for business today? www.purlem.com/blog/?p=38
  • woodka · 4 months ago
    A waterfall model with iterative design-build-test rapid prototype built within it can work well, especially if everything is well documented. For instance, design with user interface models to determine what interface a user likes and features needed, then move on to the build cycle to build the actual system, loop back into design if you need to. Keeps you from building things before the user knows what they want, and lets you go back to change things as needed. But still move through the waterfall steps as you go. It also helps if you ALWAYS build in testability and use a very modular design. Have standard code pieces in the library once verified so you're not reinventing wheels. This does take a lot of discipline, and what I've found doing software process consulting is that a lot of those who dis software engineering models like waterfall simply have never worked in a well-structured software environment. Once they're introduced to a good process with design and code reviews, good documentation and configuration management, they love it and find it really helps their creativity and ability to get things done.

    Don't play to the "cowboys" here -- there is always a need for process, and anything has a process, even if it is chaotic. It can be documented and managed well to produce a better result.
  • Phil Simon · 3 months ago
    Interesting post. I have a few thoughts. Failure is probably more tolerant at Google than other organizations because they have so many things going on. A problem with a Okrut or Google Maps might be inconvenient, but it will hardly tarnish Google's brand. Plus, Google's apps are so widely used in many cases that problems will be discovered and fixed relatively quickly.

    Also, as Cedric mentioned, Google's 20% policy is huge. In a way, they bake failure into the culture. Relative to other companies, at Google there doesn't appear to be the blamed placed on those who tried and failed.

    However, for a large organization in the middle of a massive IT project affecting supplies, employees, or vendors, failure may be less acceptable. The consequences can be so severe that it affects employees receiving paychecks, corporate security, the accuracy of financial reports, and the like. This is particularly pronounced in development efforts that follow the Waterfall method.

    If adopted, methodologies like Agile software development should decrease failure rates. Of course, people will ultimately determine whether these projects fail much more than any methodology or technology.