Do Our Brains Really Even Know How To Evaluate A Monte Carlo Analysis?

Posted by Michael Kitces on Thursday, May 17th, 11:03 am, 2012 in Retirement Planning

As a growing body of research shows, our brains are not quite the logical, rational decision-making machines we think they are – or at least, wish they could be. Instead, our brains take shortcuts; we substitute easier questions for difficult ones, often without realizing it, and respond accordingly with our words and our actions. This can be especially problematic in the world of financial planning, where we often ask clients to make difficult decisions with limited information. As a result, questions like “what is an acceptable probability of success/failure for your retirement plan?” often get switched for other questions, like “how intensely bad would you feel if your retirement plan failed?” While the questions are still similar, there is an important difference: if you have not clearly defined both the meaning of success and the meaning of failure, your clients may misjudge the intensity of the consequences, leading to an irrational and inappropriate decision about how much or how little “risk” to take.

The inspiration for today's blog post is a chapter from Daniel Kahneman's "Thinking, Fast and Slow" where he explored how our brains often figure out how to solve difficult questions by – sometimes unwittingly – substituting an easier question, and answering that instead.

Substitution Questions

For instance, if someone asks us “How popular will the
president be six months from now” most of us don’t actually go through the
detailed analysis to project how changing trends will impact the president’s popularity
down the road. Instead, our brains will tend to take a shortcut, like looking
at how popular the president is now, and simply projecting that into the future
(perhaps with some very small adjustments). Similarly, if someone asks you how
happy you are with your overall life right now, you’re more likely to simply
reflect what your current mood is right now – it’s an easier, more readily
available question your brain knows how to answer. If your mood is good, you'll probably say life is pretty good; if your mood is bad, you'll answer accordingly, too.

The research shows that for some types of questions, the
easier question we substitute may simply be one that reflects the current
situation or environment, such as our mood for our overall life or the current
popularity of the president for an estimate in the future. With some kinds of
abstract questions, however, we have to go a step further.

Relating to Intensity

It turns out that one thing our brains are especially good
at is thinking in terms of intensity, and relating things on different scales
to a similar one based on intensity. For example, Kahneman’s book gives the
example of a situation: “Julie read fluently when she was four years old.” If a
researcher were then to ask you a relative question, such as “How tall is a man
who is as tall as Julie was precocious” it turns out that most people give a
fairly consistent response – in this context, we relate it to some height that
is as remarkable as a four-year-old who can read is also remarkable.

In fact, within a consistent cultural environment – where we
all tend to relate to the context similarly – we could also answer questions
like “What level of income in your profession matches Julie’s reading
achievement?” or “Which crime is as severe as Julie is precocious?” or “Which
graduating GPA in an Ivy League college matches Julie’s reading?” The point,
simply put – we’re quite good at relating comparative intensities on different
scales and translating between the two.

Combining Substitution and Intensity Questions

In some situations, our brains will attempt to answer
difficult questions by using a combination of substitution, for an easier
question, along with intensity to answer that question. For instance, Kahneman's book notes that when we are asked a question like “How much would you contribute to save an endangered
species?” what most people really answer is the question “How much emotion do I
feel when I think of dying dolphins?” We grade the answer on our emotional
intensity scale, relate it to an associated dollar amount, and answer the
question accordingly.

In fact, in one experiment (which has since been confirmed repeatedly) first conducted not long after the Exxon Valdez spill, researchers asked various groups of participants about
their willingness to pay for nets to cover oil ponds in which migratory birds
often drowned. The participants were asked to state how much they would pay to
save either 2,000, 20,000, or 200,000 birds. Logically, saving 200,000 birds
should be worth radically more than saving just 2,000 birds, but the
researchers found in fact that the average contributions of the three groups
were $80, $78, and $88 respectively. The number of birds made remarkably little
different at all. In this case, it was because the participants first
substituted the question “how much would you pay to save XXX birds” for the
easier question “how intensely do you feel about the image of a helpless bird
drowning because its feathers are soaked in thick oil” and then related the
intensity of their thoughts about a single bird into a dollar amount. While our
brains are good at relating intensities, the substitution effect meant that the
participants had virtually entirely disregarded the part of the question about
the number of birds (without even
realizing it) and the magnitude of the project and its success or failure.

Substitution and Intensity Questions in Financial Planning

Reading through this discussion on how our brains evaluate
difficult questions was striking to me, because we often ask similarly
challenging questions of clients as a part of the financial planning process.
And given the research on how our brains think, it’s almost certain our clients
engage in a similar process – which is somewhat concerning, as the consequences
can lead to very distorted conclusions, such as the study participants who were
only willing to pay 10% more to save 100 times as many birds.

For instance, imagine the situation where the client is
asked to decide what probability of success is acceptable for his/her
retirement plan. As the conversation often goes, the client is asked which plan
is preferable: one that has an 85% probability of success, a 95% probability,
or if the client would like to save more/spend less/retire later so that the
plan can have a 99% probability.

In practice, our brains have little framework to really
evaluate such probabilities; in the end, we don’t really know how to evaluate a retirement that has a 90% probability
of succeeding. Instead, the research suggests that we probably substitute an
easier question, such as “how intensely bad would you feel about running out of
money in retirement?” Given our ease of converting intensity questions on
different scales, the brain can easily answer this question, evaluating the
intensity of negative feelings about the potential adverse outcome, and then
converting them to a 1% - 100% scale. Clients who have intensely bad feelings
about a potential retirement “failure” will give higher required probabilities
of success, while clients who are less emotionally distressed at the thought
will answer lower probabilities. Thus, notwithstanding the original question,
clients who suggest a preferred probability of success are probably not
actually indicating how much risk (of
failure) they wish to expose themselves to, but instead are indicating how
distressing in their minds that failure would be.

Framing the Consequences

The reason that this substitution effect matters – where
clients answer the question “how much risk would you like to take” with the
easier “how intensely bad would you feel if the adverse risky event happened”
is that as planners, we often do a poor job of effectively defining exactly
what the risky outcome would be.

For instance, think again about the scenario where a client
is asked what probability of success would be preferable for a retirement plan:
85%, 95%, or 99%. In asking this question, we generally leave it up to the client
to imagine what failure would look like. Without any other information, the
logical conclusion – and in fact, the one sometimes implied by the planner – is
that failure means a total loss of assets. Lifestyle and enjoyment ends. The
family home is sold. From now on dinner is dog food, and you can never afford to see the grandchildren again.

Yet when we look at the realities of a retirement plan and
how the financial planning process is executed, this is really a gross mis-statement.
As discussed previously on this blog, the reality is that the probability of
“failure” would be more actually characterized as a probability of adjustment instead
. It represents the
odds that the client would be heading down an adverse path that, through
monitoring, might require a mid-course correction to get back on track. As
research by Guyton in the Journal of Financial Planning has shown, “mere” spending
cuts of 10% in difficult times can be effective to get clients back on track,
and in fact they generally can make up the spending cuts and more in the future
when the good returns come back.

So imagine a world where two clients are asked a similar but
different question:

Client A: “What probability of success would be preferable
for your retirement plan, 85%, 95%, or 99%?”

Client B: “What probability of success would be preferable
for your retirement plan, 85%, 95%, or 99%, where a ‘failure’ means you would
have to engage in a 1-5 year spending where your spending is cut by 10% until
the market recovers?”

In reality, both scenarios describe the same situation, at
least how it would likely play out with a planner engaging in an ongoing
monitoring process with a typical client (who could intervene with a mid-course
correction if the client was heading towards a danger zone). Except the reality
is that because the scenarios have a very different implied outcome – client B
faces “just” a potential 10% spending cut for a few years, while client A is
left to his/her own imagination about how catastrophic the failure must be
(given no other information) – the clients may convey very different comfort
levels and risk preferences, even though it’s actually the same planning
scenario, because they're expressing different intensities around what they think are different outcomes!

Which means the bottom line is that in situations where
clients are invited to make a decision about how much risk to take, it is crucial to define what a risky event
or an adverse outcome really means. Is it a retirement plan that requires some
mid-course corrections with moderate spending cuts, or total destitution? Is it
a portfolio that could experience a 20% pullback, or a 100% total loss? If the
consequences aren’t defined clearly, the client at best will simply infer
whatever consequence he/she thinks would be the result, judge the intensity
accordingly, and make a decision about risk taking. At worst, the client
infers the wrong risky outcome, leading to an
entirely inappropriate conclusion about risk taking. Because like it or not,
the research – as discussed in Kahneman’s book – clearly shows that this is how
our brains operate.

So what do you think? Do you always clearly define the
consequences of a potential risk decision? Do you frame for clients that
probabilities of success are about success versus total failure? Or success
versus moderate mid-course corrections? Do you think it would change their
decisions about which retirement plan they choose?


  • Mike McGinley

    I’ve never felt comfortable with Monte Carlo simulations and do not use them. I think it is a huge mistake to tell a client they have an 85% probability of success. It projects an image that there is more control than reality.

    My opinion is that planners should focus more on helping clients make reasonable adjustments when things change and focus less on false assurances.

  • http://Obrienfp.com Dave O’Brien

    Thanks for a great comparison between using Monte Carlo analysis with that section of Kahneman’s book. I have had concerns with how advisors and clients use and interpret MC, and your comments add yet another issue I had not seen.

    When using Monte Carlo, we need to make sure clients understand that all we simulate is how a fixed spending scenario survives over a certain period under a large number of varied portfolio returns. The percentage of success is a pass/fail for that test, and doesn’t show what percentage of the trials would have permitted more spending, nor what portion could have passed with moderate spending reductions (as you cite from Guyton’s article). Change any of the spending assumptions, and the MC pass rate changes.

    I hope I have educated my clients that MC “results” are helpful educational aids to indicate if our current direction is acceptable, or if a course correction (saving rate, timing and size of spending) should change. Sadly, many people seem to take the mental shortcut of thinking the result is a chance of success, just like your chances of winning the lottery.

    • Eric Bruskin

      Regarding keeping track of “what percentage of the trials would have permitted more spending, nor what portion could have passed with moderate spending reductions” – good things to track, by the way – if you keep track of that information for each scenario, and have a flexible enough spreadsheet, these things and many more can indeed be tracked, arranged into distributions, and used to put the results in better context.

  • http://www.ifs-advisor.com Joe Pitzl, CFP®

    You are spot on about the necessity of proper framing. Monte carlo is not a financial plan. Monte carlo is merely a tool…an often misused and misunderstood tool. And I would argue that it is most effectively used as a research tool rather than something to base decisions on.

    Modern financial modeling is an incredibly powerful example of how our industry takes shortcuts instead of answering the real question (at least when we do not provide the proper context and framing). This problem is extraordinarily pervasive in the investment realm, as James Montier recently pointed out in his keynote presentation at the CFA Institute Annual Meeting: http://bit.ly/Iw7KS2, and it is beginning to creep into planning as well.

    As Montier explains, “mathematics is ordinarily considered as producing precise, dependable results. But in the stock market, the more elaborate and obtuse the mathematics, the more uncertain and speculative the conclusions we draw therefrom. Whenever calculus is brought in, or higher algebra, you can take it as a warning signal that the operator is trying to substitute theory for experience…If you were to give a bunch of monkeys a CAPM pricing model or a VAR risk model, you will end up creating a financial crisis,” he said. “In fact, I am pretty sure that is just what we have done.”

    Human beings feel more confident when we eliminate variables and uncertainty, which is something we attempt to do through this type of analysis. However, unless we give this type of analysis its proper respect and truly recognize its limitations, we are going to do every bit as much harm as good.

  • Mike McGinley

    Michael,

    All very good points. I very much agree with you. I am in no way trying to mischaractarize our past bear markets as black swans. You mention that the bear markets were both probable and expected, and I completely agree.

    I also agree that the term black swan has been abused. I think it was Taleb who coined the term and he meant it to be related to fat tails, which of course could be both bad and good, and which both occur often in the financial markets. The exceptionally good results you mention should also be construed as black swans.

    My point was that from a client’s perspective I feel it tends to send a wrong message.

    • Michael Kitces

      I’m still not sure I’d agree it sends a “wrong” message – that’s a very judgment-laden term – but certainly I’d agree, as was part of the point of the post here, that most clients do NOT have the tools and framework to evaluate what it means. We think in binary terms – the event happens, or it doesn’t. We really don’t have the equipment to think probabilistically, which means at best we have a high burden as advisors to translate it into something more relevant for making a decision.
      – Michael

      • Mike McGinley

        I guess “wrong” is not the right word. I guess my feeling is the tool is often misconstrued even with the advisor’s best intentions and efforts.

        Anyway, thanks for the post. Your blog is a pleasure.

        • Michael Kitces

          Mike,
          Indeed, no question results are often misconstrued. I question whether all advisors even fully understand the implications of the Monte Carlo tools they are using, much less translate them effectively for clients.

          That doesn’t necessarily make the tool bad. It is a huge concern regarding effective advice, though. :/
          – Michael

          • Mike McGinley

            I agree. When I first entered the business I was at a large brokerage firm that utilized them and the inputs ended up being widely optimistic.

            It was kind of Monte Carlo simulation as a sales technique!

            I do feel it can be a very good “behind the scenes” tool if used well and can help advisors in recommendations to clients.

  • http://www.ifs-advisor.com Joe Pitzl

    And that, Michael, ought to be our value proposition. We spend too much time attempting to get things exactly right, more precise, etc., believing our results yield better advice and a higher degree of confidence. However, greater precision in our work often results in less accurate and less realistic results.

    We absolutely have the equipment to think probabilistically, but that requires us to think. Tools can too easily become a crutch that eliminates the need to think. If we focus instead on guiding people toward decisions that yield the highest probability of success given an unquantifiable amount of uncertainty, our clients will be far better off.

  • Trey Jamison

    Good Luck getting a retiree to actually cut spending an any year by 15%. This is why the statistical tools are very important in helping us understand what our clients can successfully spend each month and year starting from the beginning of retirement

    • Michael Kitces

      Trey,
      I think it depends on the client and the nature of their spending.

      We have clients where a 5% spending cut would be traumatic. We have others where a 20% cut would be easy – they’d just trim or downscale the number of lavish cruises and vacations they take for a year or two.

      I do think we could use a better framework for discussing with clients the flexibility of their spending, though. It’s not a conversation we’re typically trained to do.
      – Michael

  • Pingback: quickreadbuzz.com()

  • Pingback: quickreadbuzz.com()

  • Pingback: kitces.com | Nerd's Eye View()

  • J Tillman

    This is precisely why I have incorporated 1.)RiskalyzePro and 2.) ‘trading’ (w/ stop losses) into my practice.

    I, like Khaneman, believe the most important # an advisor needs to know is the client’s “Pain #”. Knowing the point where a client’s emotional state will overrun their rational self is the goal.

  • Pingback: blog.betterfinancialeducation.com()

  • Michael Kitces

    Mike,
    Understood, but at some point we need SOME understanding of what paths are likely to lead to no adjustments, some adjustments, or intolerable adjustments?

    How else would we know whether $20k, $40k, $60k, or $80k/year of spending is prudent at all from a $1,000,000 portfolio without some form of probabilistic analysis?

    Respectfully,
    – Michael

  • Mark Brice

    This is interesting, I’m pretty sure my brain isn’t capable of fully using monte carlo output as it relates to my own life, but I’ve never really thought about why it seems so abstract and theoretical.

    I also agree that some form of probabilistic analysis is needed, what I’m not so sure about is if we should even talk to our clients about the raw numerical output? I don’t know if this is true or not, but it seems we are taking a technical tool and placing it on a pedestal.

    Maybe, and this is a hypothesis, we should instead use monte carlo, along with our understanding of what changes in spending might be needed to get back on track and how that might impact their life (risk capacity of the goal), and account for our understanding of their risk attitude, and derive our recommendation. I guess what I’m saying is, if clients aren’t capable of making these decisions we’re asking them to make then perhaps we should take more responsibility to come to a rational conclusion and recommend it. As the advisor, put a stake in the ground. The alternative, which you gave an example of Michael, is to figure out a different way to both ask and frame the question. I’m not so sure I know a good way to do that.

    Great post

  • Mike McGinley

    You make a great point in your article about asking the client questions about how likely they are to tolerate a spending cut (plan failure). And I think that is a great (and crucial) conversation to have on an ongoing basis.

    But I also feel the conversation should be very broad and should never be about confidence levels, etc.. Simply asking a question “what probability will you tolerate at 85%, 95% or 99%…. is a bad idea because it may give the client a feeling that there is more control than is actually the case.

    Investing is not physics. Over the last twenty years, we have experienced several financial events that were supposed to happen only once in a lifetime. Trying to produce models and actually showing them to clients is a disservice, in my humble opinion.

  • Michael Kitces

    Mike,
    With all due respect, there have NOT been any “once in a lifetime” events in the past twenty years on an ANNUAL or longer basis. These “black swan” events have been DAILY events that evened out over the span of just a few weeks and months, and given that clients don’t need to make material liquidations on a daily or weekly basis, had NO black-swan-style impact. (The consequences for over-leveraged hedge funds, on the other hand, would be quite different.)

    You might see some of the prior posts I’ve made on this, such as http://www.kitces.com/blog/archives/255-Are-Black-Swans-Just-A-Short-Term-Distraction.html – which shows that on an annual basis, the only “black swans” we’ve seen in the past century were impossibly GOOD returns, and not a single impossibly BAD one.

    Our industry has grossly mischaracterized entirely-probable-and-expected big bear markets with black swan events. Individual companies have plenty of black swans; markets in the aggregate have not over any time period longer than minutes, days, or weeks.

    In other words, as long as we’re running Monte Carlo analysis on an annual basis, the primary error has been underestimating very GOOD years, not very bad ones.

    Respectfully,
    – Michael

4 Pingbacks/Trackbacks

Blog Updates by Email

Nerd’s Eye View Praise

@MichaelKitces Twitter


Out and About

Monday, December 8th, 2014

TBD @ Private Event

Tuesday, December 9th, 2014

Cutting Edge Tax Planning Developments & Opportunities Setting a Proper Asset Allocation Glidepath in Retirement Future of Financial Planning in the Digital Age @ FPA Central Pennsylvania

Thursday, December 11th, 2014

TBD @ NAPFA DC Study Group

VIEW FULL SCHEDULE