Current Issue

The Korean Journal of Cognitve & Biological Psychology - Vol. 33 , No. 4

[ Original Article ]
The Korean Journal of Cognitve & Biological Psychology - Vol. 33, No. 4, pp. 265-279
Abbreviation: KCBPA
ISSN: 1226-9654 (Print)
Print publication date 30 Oct 2021
Received 30 Jul 2021 Revised 31 Oct 2021 Accepted 31 Oct 2021
DOI: https://doi.org/10.22172/cogbio.2021.33.4.004

자극의 불확실성이 연합학습에 미치는 영향: 수리모델간 비교연구
정지훈 ; 조양석 ; 최준식
고려대학교 심리학부

Discrepant predictions from computational models of associative learning on the effect of contingency uncertainty
Ji Hoon Jeong ; Yang Seok Cho ; June-Seek Choi
School of Psychology, Korea University
Correspondence to : 최준식, 고려대학교 심리학과, (02841) 서울시 성북구 안암로 145, E-mail: j-schoi@korea.ac.kr


ⓒ The Korean Society for Cognitive and Biological Psychology
Funding Information ▼

초록

보상 정보의 불확실성은 정서 조절부터 의사결정 최적화까지 다양한 영역에 영향을 끼친다. 그러나 불확실성은 정량적인 연구가 진행되기보다는 다양한 심리 현상들의 부수적인 설명으로 사용되어왔다. 이러한 문제를 해결하는 한 가지 방법은 계산 모델을 구현해, 이를 동물과 사람 실험 결과에 적용해서 불확실성이 포함된 조건화 상황에서 각 모델의 예측치를 비교하는 것이다. 본 연구에서는 Matlab 기반 연합학습 모델 시뮬레이션 통합환경(Korea University Conditioning Simulator: KUCS)을 구축하였다. 이 시뮬레이터에서는 GUI 기반으로 다양한 학습 계획을 적용해서 Rescorla-Wagner 모델, Mackintosh 모델, Pearce-Hall 모델, Schmajuk-Pearce-Hall 모델, Esber-Hasselgrove 모델, Temporal Difference 모델의 특성을 비교할 수 있다. 검증을 위하여 KUCS를 사용하여 기초적인 연합 조건화 현상인 acquisition, extinction, blocking, conditioned inhibition, latent inhibition, second-order conditioning을 각 모델에서 구현할 수 있는지 확인했고, 모델의 한계점과 예측에 관한 몇 가지 새로운 사실을 발견하였다. 또한, 보상의 불확실성이 존재하는 학습 계획을 사용해 모델의 연합력(association strength)과 연합가능성(associability) 값이 실제 동물과 사람 실험 데이터와 일치하는지를 확인하였다. 시뮬레이터 프로그램은 https://github.com/knowblesse/KUCS 에 공개하였다.

Abstract

The role of information uncertainty has wide implications ranging from emotional modulation to optimal decision making. Yet the concept has been employed as an ad-hoc explanation for various phenomena. One useful approach to the problem is to use a formal computational model to test different parameters extracted from animal and human studies on stimulus uncertainty.

We developed an integrated simulation environment written in Matlab (Korea University Conditioning Simulator: KUCS) which provides graphical user interface for several influential models of associative learning such as Rescorla-Wagner model, Mackintosh model, Pearce and Hall model, Schmajuk-Pearce-Hall model, Esber-Hasselgrove model, and Temporal Difference model. Using KUCS, We first demonstrated common predictions on basic conditioning phenomena: acquisition, extinction, blocking, conditioned inhibition, latent inhibition, and second-order conditioning to confirm the validity of the simulator and to find some novel limitations and predictions. We then generated a series of data under uncertainty and compared them with animal and human experiments to examine how the models’ predictions on the associative strength and associability concur with the experimental data. The simulator program is available in https://github.com/knowblesse/KUCS


Keywords: modeling, associative learning, learning model, reward uncertainty
키워드: 수리모델, 연합학습, 보상 불확실성

Acknowledgments

본 연구는 한국연구재단의 지원을 받아 수행되었음(2020R1A2C2014830, 2021M3E5D2A01023887, 2017H1A2A1044665).


References
1. Anderson, B. A., Laurent, P. A., & Yantis, S. (2011). Value-driven attentional capture. Proc Natl Acad Sci U S A, 108(25), 10367-10371.
2. Cho, S. A., & Cho, Y. S. (2021). Uncertainty modulates value-driven attentional capture. Atten Percept Psychophys, 83(1), 142-155.
3. Esber, G. R., & Haselgrove, M. (2011). Reconciling the influence of predictiveness and uncertainty on stimulus salience: a model of attention in associative learning. Proc Biol Sci, 278(1718), 2553-2561.
4. Harris, J. A., Kwok, D. W. S., & Gottlieb, D. A. (2019). The partial reinforcement extinction effect depends on learning about nonreinforced trials rather than reinforcement rate. J Exp Psychol Anim Learn Cogn, 45(4), 485-501.
5. Haselgrove, M., Aydin, A., & Pearce, J. M. (2004). A partial reinforcement extinction effect despite equal rates of reinforcement during Pavlovian conditioning. J Exp Psychol Anim Behav Process, 30(3), 240-250.
6. Humphreys, L. G. (1939). The effect of random alternation of reinforcement on the acquisition and extinction of conditioned eyelid reactions. Journal of Experimental Psychology, 25(2), 141-158.
7. Kamin, L. J., & Szakmary, G. A. (1977). Configural-like effects observed in the course of second-order conditioning in rats. Learning and Motivation, 8(1), 126-135.
8. Lipp, O. V., Stephens, J., & Smith, T. A. (1999). RWMODEL II: computer simulation of the Rescorla-Wagner model of Pavlovian conditioning. Behav Res Methods Instrum Comput, 31(4), 735-736.
9. Mackintosh, N. J. (1975). Theory of Attention - Variations in Associability of Stimuli with Reinforcement. Psychological Review, 82(4), 276-298.
10. Miller, R. R., Barnet, R. C., & Grahame, N. J. (1995). Assessment of the Rescorla-Wagner Model. Psychological Bulletin, 117(3), 363-386.
11. Mercier, P. (1996). Computer simulations of the Rescorla-Wagner and Pearce-Hall models in conditioning and contingency judgment. Behavior Research Methods Instruments & Computers, 28(1), 55-60.
12. Moore, J. W., & Choi, J. S. (1997). Conditioned response timing and integration in the cerebellum. Learn Mem, 4(1), 116-129.
13. Mowrer, O. H., & Jones, H. (1945). Habit Strength as a Function of the Pattern of Reinforcement. Journal of Experimental Psychology, 35(4), 293-311.
14. Pavlov, I. P. (1927). Conditioned reflexes: an investigation of the physiological activity of the cerebral cortex: Oxford University Press, London. .
15. Pearce, J. M., & Hall, G. (1980). A model for Pavlovian learning: variations in the effectiveness of conditioned but not of unconditioned stimuli. Psychological Review, 87(6), 532.
16. Rescorla, R. A. (1999a). Partial reinforcement reduces the associative change produced by nonreinforcement. Journal of Experimental Psychology-Animal Behavioral Processes, 25(4), 403-414.
17. Rescorla, R. A. (1999b). Within-subject partial reinforcement extinction effect in autoshaping. Quarterly Journal of Experimental Psychology Section B-Comparative and Physiological Psychology, 52(1), 75-87.
18. Rescorla, R. A., & Wagner, A. R. (1972). A theory of Pavlovian conditioning: Variations in the effectiveness of reinforcement and nonreinforcement. In Classical conditioning: current research and theory. New York: Appleton-Century-Crofts.
19. Schmajuk, N. A., & DiCarlo, J. J. (1991). A neural network approach to hippocampal function in classical conditioning. Behav Neurosci, 105(1), 82-110.
20. Schmajuk, N. A., & Moore, J. W. (1985). Real-Time Attentional Models for Classical-Conditioning and the Hippocampus. Physiological Psychology, 13(4), 278-290. Retrieved from <Go to ISI>://WOS:A1985A611600010
21. Sheffield, V. F. (1949). Extinction as a function of partial reinforcement and distribution of practice. J Exp Psychol, 39(4), 511-526.
22. Sutton, R. S., & Barto, A. G. (1987). A temporal-difference model of classical conditioning. Paper presented at the Proceedings of the ninth annual conference of the cognitive science society.
23. Sutton, R. S., & Barto, A. G. (2018). Reinforcement learning: An introduction (2 ed.): MIT press.
24. Thomas, D. (1970). Stimulus selection, attention, and related matters. Current issues in animal learning, 311-356.
25. Wagner, A. (1981). SOP: A model of automatic processing in animal behavior. Information Processing in Animals: Conditioned Inhibition.
26. Wagner, A. R., & Brandon, S. E. (1989). Evolution of a structured connectionist model of Pavlovian conditioning (AESOP). In Contemporary learning theories: Pavlovian conditioning and the status of traditional learning theory. (pp. 149-189). Hillsdale, NJ, US: Lawrence Erlbaum Associates, Inc.
27. Yin, H., Barnet, R. C., & Miller, R. R. (1994). Second-order conditioning and Pavlovian conditioned inhibition: operational similarities and differences. J Exp Psychol Anim Behav Process, 20(4), 419-428.