Skip to main content

Counterfactuals, Dopamine, and Risky Behavior: Deriving a neurobiological theory of decision-making under risk

10 References

1. Daniel Bernoulli. (1954). Exposition of a new theory on the measurement of risk. Econometrica, 22(1), 23–36.
2. Neumann, J. von, & Morgenstern, O. (1947). Theory of games and economic behavior, 2nd edn. Princeton University Press.
3. Allais, M. (1953). Le comportement de l’homme rationnel devant le risque: Critique des postulats et axiomes de l’ecole americaine. Econometrica, 21(4), 503–546. https://doi.org/10.2307/1907921
4. Kahneman, D., & Tversky, A. (1979). Prospect Theory: An Analysis of Decision under Risk. Econometrica, 47(2), 263. https://doi.org/10.2307/1914185
5. Loomes, G., & Sugden, R. (1982). Regret theory: An alternative theory of rational choice under uncertainty. The Economic Journal, 92(368), 805–824. https://doi.org/10.2307/2232669
6. Bell, D. E. (1982). Regret in decision making under uncertainty. Operations Research, 30(5), 961–981. https://www.jstor.org/stable/170353
7. Neal J. Roese. (1997). Counterfactual thinking. Psychological Bulletin, 121.
8. Liu, Z., Li, L., Zheng, L., Hu, Z., Roberts, I. D., Guo, X., & Yang, G. (2016). The neural basis of regret and relief during a sequential risk-taking task. Neuroscience, 327, 136–145. https://doi.org/10.1016/j.neuroscience.2016.04.018
9. Barron, G., & Erev, I. (2003). Small feedback-based decisions and their limited correspondence to description-based decisions. Journal of Behavioral Decision Making, 16(3), 215–233. https://doi.org/10.1002/bdm.443
10. Pavlov, I. P. (1927). Conditioned reflexes: An investigation of the physiological activity of the cerebral cortex. Annals of Neurosciences, 17(3). https://doi.org/10.5214/ans.0972-7531.1017309
11. Bush, R. R., & Mosteller, F. (1951). A mathematical model for simple learning. Psychological Review, 58(5), 313–323. https://doi.org/10.1037/h0054388
12. Bush, R. R., & Mosteller, F. (1951). A model for stimulus generalization and discrimination. Psychological Review, 58(6), 413–423. https://doi.org/10.1037/h0054576
13. Glimcher, P. W. (2011). Understanding dopamine and reinforcement learning: The dopamine reward prediction error hypothesis. Proceedings of the National Academy of Sciences, 108(Supplement_3), 15647–15654. https://doi.org/10.1073/pnas.1014269108
14. Sutton, R. S. (1988). Learning to predict by the methods of temporal differences. Machine Learning, 3(1), 9–44. https://doi.org/10.1007/BF00115009
15. Richard S. Sutton; Andrew G. Barto. (2018). Reinforcement learning: An introduction (2nd ed.). MIT Press.
16. W. Mischel, & E. B. Ebbesen. (1970). Attention in delay of gratification. Journal of Personality and Social Psychology. https://content.apa.org/doiLanding?doi=10.1037%2Fh0029815
17. Shoda, Y., Mischel, W., & Peake, P. K. (1990). Predicting adolescent cognitive and self-regulatory competencies from preschool delay of gratification: Identifying diagnostic conditions. Developmental Psychology, 26(6), 978–986. https://doi.org/10.1037/0012-1649.26.6.978
18. Watts, T. W., Duncan, G. J., & Quan, H. (2018). Revisiting the Marshmallow Test: A Conceptual Replication Investigating Links Between Early Delay of Gratification and Later Outcomes. Psychological Science, 29(7), 1159–1177. https://doi.org/10.1177/0956797618761661
19. Montague, P., Dayan, P., & Sejnowski, T. (1996). A framework for mesencephalic dopamine systems based on predictive Hebbian learning. The Journal of Neuroscience, 16(5), 1936–1947. https://doi.org/10.1523/JNEUROSCI.16-05-01936.1996
20. Schultz, W., Dayan, P., & Montague, P. R. (1997). A neural substrate of prediction and reward. Science (New York, N.Y.), 275(5306), 1593–1599. https://doi.org/10.1126/science.275.5306.1593
21. Hart, A. S., Rutledge, R. B., Glimcher, P. W., & Phillips, P. E. M. (2014). Phasic dopamine release in the rat nucleus accumbens symmetrically encodes a reward prediction error term. The Journal of Neuroscience: The Official Journal of the Society for Neuroscience, 34(3), 698–704. https://doi.org/10.1523/JNEUROSCI.2489-13.2014
22. Tomas Ljunberg, Paul Apicella, & Wolfram Schultz. (1992). Responses of monkey dopamine neurons during learning of behavioral reactions. Journal of Neurophysiology, 67.
23. Schultz, W., Apicella, P., & Ljungberg, T. (1993). Responses of monkey dopamine neurons to reward and conditioned stimuli during successive steps of learning a delayed response task. The Journal of Neuroscience, 13(3), 900–913. https://doi.org/10.1523/JNEUROSCI.13-03-00900.1993
24. Schultz, W., Tremblay, L., & Hollerman, J. R. (1998). Reward prediction in primate basal ganglia and frontal cortex. Neuropharmacology, 37(4-5), 421–429. https://doi.org/10.1016/s0028-3908(98)00071-9
25. Bayer, H. M., & Glimcher, P. W. (2005). Midbrain dopamine neurons encode a quantitative reward prediction error signal. Neuron, 47(1), 129–141. https://doi.org/10.1016/j.neuron.2005.05.020
26. Zaghloul, K. A., Blanco, J. A., Weidemann, C. T., McGill, K., Jaggi, J. L., Baltuch, G. H., & Kahana, M. J. (2009). Human substantia nigra neurons encode unexpected financial rewards. Science, 323(5920), 1496–1499. https://doi.org/10.1126/science.1167342
27. Kishida, K. T., Sandberg, S. G., Lohrenz, T., Comair, Y. G., Sáez, I., Phillips, P. E. M., & Montague, P. R. (2011). Sub-Second Dopamine Detection in Human Striatum. PLOS ONE, 6(8), e23291. https://doi.org/10.1371/journal.pone.0023291
28. Moran, R. J., Kishida, K. T., Lohrenz, T., Saez, I., Laxton, A. W., Witcher, M. R., Tatter, S. B., Ellis, T. L., Phillips, P. E., Dayan, P., & Montague, P. R. (2018). The Protective Action Encoding of Serotonin Transients in the Human Brain. Neuropsychopharmacology, 43(6), 1425–1435. https://doi.org/10.1038/npp.2017.304
29. Bang, D., Kishida, K. T., Lohrenz, T., White, J. P., Laxton, A. W., Tatter, S. B., Fleming, S. M., & Montague, P. R. (2020). Sub-second Dopamine and Serotonin Signaling in Human Striatum during Perceptual Decision-Making. Neuron. https://doi.org/10.1016/j.neuron.2020.09.015
30. Kishida, K. T., Saez, I., Lohrenz, T., Witcher, M. R., Laxton, A. W., Tatter, S. B., White, J. P., Ellis, T. L., Phillips, P. E. M., & Montague, P. R. (2016). Subsecond dopamine fluctuations in human striatum encode superposed error signals about actual and counterfactual reward. Proceedings of the National Academy of Sciences, 113(1), 200–205. https://doi.org/10.1073/pnas.1513619112
31. Montague, P. R., Hyman, S. E., & Cohen, J. D. (2004). Computational roles for dopamine in behavioural control. Nature, 431(7010), 760–767. https://doi.org/10.1038/nature03015
32. Schultz, W., Dayan, P., & Montague, P. R. (1997). A neural substrate of prediction and reward. Science (New York, N.Y.), 275(5306), 1593–1599. https://doi.org/10.1126/science.275.5306.1593
33. Bayer, H. M., & Glimcher, P. W. (2005). Midbrain dopamine neurons encode a quantitative reward prediction error signal. Neuron, 47(1), 129–141. https://doi.org/10.1016/j.neuron.2005.05.020
34. Roesch, M. R., Calu, D. J., & Schoenbaum, G. (2007). Dopamine neurons encode the better option in rats deciding between differently delayed or sized rewards. Nature Neuroscience, 10(12), 1615–1624. https://doi.org/10.1038/nn2013
35. B. Liebenow, A. Jiang, E. DiMarco, L. P. Sands, M. Moya-Mendez, M. S. Siddiqui, I. HAQ, & K. T. Kishida. (2021). The ‘expected value’ (not the surprising outcomes) of risky choices drives subjective pleasure of risky decision making in patients with impulse control disorder. https://www.abstractsonline.com/pp8/#!/10485/presentation/17371
36. Wilson, R. C., & Collins, A. G. (2019). Ten simple rules for the computational modeling of behavioral data. eLife, 8, e49547. https://doi.org/10.7554/eLife.49547
37. Dogucu, A. A. J. M. Q. O. M. (2022). Bayes rules! An introduction to applied bayesian modeling. CRC Press. https://www.bayesrulesbook.com/
38. Schoot, R. van de, Depaoli, S., King, R., Kramer, B., Märtens, K., Tadesse, M. G., Vannucci, M., Gelman, A., Veen, D., Willemsen, J., & Yau, C. (2021). Bayesian statistics and modelling. Nature Reviews Methods Primers, 1(1), 1–26. https://doi.org/10.1038/s43586-020-00001-2
39. Sokol-Hessner, P., Hsu, M., Curley, N. G., Delgado, M. R., Camerer, C. F., & Phelps, E. A. (2009). Thinking like a trader selectively reduces individuals’ loss aversion. Proc Natl Acad Sci U S A, 106(13), 5035–5040. https://doi.org/10.1073/pnas.0806761106
40. Richard McElreath. (2020). Statistical rethinking: A bayesian course with examples in r and stan. CRC Press.
41. Haines, N. (2018). Human choice and reinforcement learning (3). http://haines-lab.com/post/2018-03-24-human-choice-and-reinforcement-learning-3/2018-03-24-human-choice-and-reinforcement-learning-3/
42. Stan Development Team. (2022). Stan modeling language users guide and reference manual. https://mc-stan.org
43. Ahn, W.-Y., Krawitz, A., Kim, W., Busemeyer, J. R., & Brown, J. W. (2011). A model-based fMRI analysis with hierarchical Bayesian parameter estimation. Journal of Neuroscience, Psychology, and Economics, 4(2), 95–110. https://doi.org/10.1037/a0020684
44. Ahn, W.-Y., Haines, N., & Zhang, L. (2017). Revealing neurocomputational mechanisms of reinforcement learning and decision-making with the hBayesDM package. Computational Psychiatry, 1, 24–57. https://doi.org/10.1162/CPSY_a_00002
45. Ahn, W.-Y., Vasilev, G., Lee, S.-H., Busemeyer, J. R., Kruschke, J. K., Bechara, A., & Vassileva, J. (2014). Decision-making in stimulant and opiate addicts in protracted abstinence: Evidence from computational modeling with pure users. Frontiers in Psychology, 5, 849. https://doi.org/10.3389/fpsyg.2014.00849
46. Gelman, A., & Rubin, D. B. (1992). Inference from iterative simulation using multiple sequences. Statistical Science, 7(4), 457–472. https://doi.org/10.1214/ss/1177011136
47. Gronau, Q. F., Singmann, H., & Wagenmakers, E.-J. (2020). bridgesampling: An R Package for Estimating Normalizing Constants. Journal of Statistical Software, 92, 1–29. https://doi.org/10.18637/jss.v092.i10
48. Vehtari, A., Gelman, A., & Gabry, J. (2017). Practical Bayesian model evaluation using leave-one-out cross-validation and WAIC. Statistics and Computing, 27(5), 1413–1432. https://doi.org/10.1007/s11222-016-9696-4
49. Plummer, M. (2008). Penalized loss functions for Bayesian model comparison. Biostatistics, 9(3), 523–539. https://doi.org/10.1093/biostatistics/kxm049
50. Thaler, R. H. (2000). From Homo Economicus to Homo Sapiens. Journal of Economic Perspectives, 14(1), 133–141. https://doi.org/10.1257/jep.14.1.133
51. Agranov, M., & Ortoleva, P. (2017). Stochastic Choice and Preferences for Randomization. Journal of Political Economy, 125(1), 40–68. https://doi.org/10.1086/689774
52. Tversky, A. (1969). Intransitivity of preferences. Psychological Review, 76(1), 31–48. https://doi.org/10.1037/h0026750
53. Lerner, J. S., Li, Y., Valdesolo, P., & Kassam, K. S. (2015). Emotion and Decision Making. Annual Review of Psychology, 66(1), 799–823. https://doi.org/10.1146/annurev-psych-010213-115043
54. Kishida, K. T., & Sands, L. P. (2021). A Dynamic Affective Core to Bind the Contents, Context, and Value of Conscious Experience. In C. E. Waugh & P. Kuppens (Eds.), Affect Dynamics (pp. 293–328). Springer International Publishing. https://doi.org/10.1007/978-3-030-82965-0_12