在實際決策中，決策者一般無法事先完全掌握每個選項的好壞，卻仍必須在多個選項之間做出選擇，而想要了解這些選項，通常也需要投入時間與成本。多臂拉霸機（Multi-Armed Bandit, MAB）正好可以用來描述這類的決策問題，其核心在於決策者如何在探索未知選項與利用已知較佳選項之間取得平衡。

這類問題可對應到廣告投放、影音推薦、個人化治療與供應商選擇等情境。在這些情境中，每個選項的真實價值一開始並不明確，必須透過後續觀察或測試逐漸學習。選擇某一選項時，決策者不僅會取得該選項本身的資訊，也可能從中推估其他相似選項的價值。因此，各選項之間不一定是完全獨立的，它們可能因為具有共同特徵，而在資訊上彼此相關。

為了描述此類選項間資訊相互影響的情境，本研究建立於 Huh et al.（2025）所提出的雙層學習架構之上。該研究探討具有個體層次與群體層次學習效果的最佳停止問題，但決策者一旦切換至下一個選項後，便無法再回到先前的選項。因此本研究將此概念納入 MAB 架構，使決策者能夠重複選擇先前已觀察過的手臂，並在持續學習的過程中平衡探索與利用。

本研究目的為描述各手臂指標值的結構，並分析雙層學習機制如何影響手臂價值，以及個體資訊與群體資訊在決策中的相對重要性。受到群眾智慧（Wisdom of the Crowd）概念的啟發，決策者可能會先選擇成本較低、能提供群體層資訊的手臂，以掌握整體選項的共同特徵，之後再將觀察資源分配給能獲得個別差異資訊的手臂，以加深對特定選項價值的理解。

Decision makers frequently face multiple alternatives under incomplete information, where exploration requires time and cost. The multi-armed bandit (MAB) framework provides a natural model for such dynamic learning problems. Traditional MAB models usually assume that arms are independent, so observing one arm only updates the belief about that arm. However, in many real-world settings, alternatives are related. For example, advertisements may share target customers, videos may share genres or viewing patterns, treatments may be related through patient characteristics, and suppliers may share cost structures or service quality. Therefore, information obtained from one option may provide partial insights into other related options.

To capture this dependence, this study builds on the two-level learning framework of Huh et al. (2025), which considers individual-level and population-level learning in an optimal stopping problem. In their model, once a candidate is abandoned, it cannot be revisited. In contrast, this study considers a repeated arm-selection setting in which previously selected arms can be revisited and sampled again. This setting allows a richer exploration-exploitation trade-off.

The objective of this study is to characterize the index value associated with each arm and examine how two-level learning affects arm values and selection decisions. Inspired by the “wisdom of the crowd,” the decision maker may first select arms that provide population-level information and then allocate observations to arms that provide more distinctive individual-level information.

[1] Shipra Agrawal and Navin Goyal. Analysis of thompson sampling for the multi-armed bandit problem. In Conference on Learning Theory, pages 39–1, 2012.

[2] Jean-Yves Audibert, Sébastien Bubeck, and Rémi Munos. Best arm identification in multi-armed bandits. In Proceedings of the 23rd Annual Conference on Learning Theory, pages 41–53, 2010.

[3] Peter Auer, Nicolò Cesa-Bianchi, and Paul Fischer. Finite-time analysis of the multi-armed bandit problem. Machine Learning, 47(2–3):235–256, 2002.

[4] Hamsa Bastani, Mohsen Bayati, and Khashayar Khosravi. Mostly exploration-free algorithms for contextual bandits. Management Science, 67(3):1329–1349, 2021.

[5] Hamsa Bastani, David Simchi-Levi, and Ruihao Zhu. Meta dynamic pricing: Transfer learning across experiments. Management Science, 68(3):1865–1881, 2022.

[6] Manel Baucells and Saša Zorc. Search in the dark: The normal case, 2022. SSRN working paper.

[7] Dimitri P. Bertsekas. Dynamic Programming and Optimal Control, Volume II. Athena Scientific, Belmont, MA, 1995.

[8] Dimitris Bertsimas and José Niño-Mora. Conservation laws, extended polymatroids and multiarmed bandit problems: A polyhedral approach to indexable systems. Mathematics of Operations Research, 21(2):257–306, 1996.

[9] Felipe Caro and Jérémie Gallien. Dynamic assortment with demand learning for seasonal consumer goods. Management Science, 53(2):276–292, 2007.

[10] Olivier Chapelle and Lihong Li. An empirical evaluation of thompson sampling. In Advances in Neural Information Processing Systems, pages 2249–2257, 2011.

[11] Stephen E. Chick and Peter I. Frazier. Sequential sampling with economics of selection procedures. Management Science, 58(3):550–569, 2012.

[12] Stephen E. Chick and Noah Gans. Economic analysis of simulation selection problems. Management Science, 55(3):421–437, 2009.

[13] Morris H. DeGroot. Some problems of optimal stopping. Journal of the Royal Statistical Society: Series B, 30(1):108–122, 1968.

[14] Morris H. DeGroot. Optimal Statistical Decisions. Wiley, 2005.

[15] Sanjiv Erat and Stylianos Kavadias. Sequential testing of product designs: Implications for learning. Management Science, 54(5):956–968, 2008.

[16] Kris Johnson Ferreira, David Simchi-Levi, and He Wang. Online network revenue management using thompson sampling. Operations Research, 66(6):1586–1602, 2018.

[17] Andras Fülöp, Junye Li, Hening Liu, and Cheng Yan. Estimating and testing long-run risk models: International evidence. Management Science, 71(4):3517–3536, 2024.

[18] Andrew Gelman, John B. Carlin, Hal S. Stern, David B. Dunson, Aki Vehtari, and Donald B. Rubin. Bayesian Data Analysis. CRC Press, 2013.

[19] John C. Gittins. Bandit processes and dynamic allocation indices. Journal of the Royal Statistical Society: Series B, 41(2):148–164, 1979.

[20] Alexander Goldenshluger and Assaf Zeevi. Optimal stopping of a random sequence with unknown distribution. Mathematics of Operations Research, 47(1):29–49, 2022.

[21] Woonghee Tim Huh, Michael Jong Kim, and Meichun Lin. Uncertain search with knowledge transfer. Management Science, 2025. Published online May 16, 2025.

[22] T. Tony Ke and Song Lin. Informational complementarity. Management Science, 66(8):3699–3716, 2020.

[23] T. Tony Ke, Zuo-Jun Max Shen, and J. Miguel Villas-Boas. Search for information on multiple products. Management Science, 62(12):3576–3603, 2016.

[24] Michael Jong Kim and Andrew E. B. Lim. Robust multiarmed bandit problems. Management Science, 62(1):264–285, 2016.

[25] Tze Leung Lai and Herbert Robbins. Asymptotically efficient adaptive allocation rules. Advances in Applied Mathematics, 6(1):4–22, 1985.

[26] Steven A. Lippman and Kevin F. McCardle. Uncertain search: A model of search among technologies of uncertain values. Management Science, 37(11):1474–1490, 1991.

[27] Qing Liu and Donald A. Pierce. A note on gauss-hermite quadrature. Biometrika, 81(3):624–629, 1994.

[28] Kevin F. McCardle. Information acquisition and the adoption of new technology. Management Science, 31(11):1372–1389, 1985.

[29] Adam J. Mersereau, Paat Rusmevichientong, and John N. Tsitsiklis. A structured multiarmed bandit problem and the greedy policy. IEEE Transactions on Automatic Control, 54(12):2787–2802, 2009.

[30] Sinno Jialin Pan and Qiang Yang. A survey on transfer learning. IEEE Transactions on Knowledge and Data Engineering, 22(10):1345–1359, 2009.

[31] Herbert Robbins. Some aspects of the sequential design of experiments. Bulletin of the American Mathematical Society, 55:527–535, 1952.

[32] Christian P. Robert. The Bayesian Choice: From Decision-Theoretic Foundations to Computational Implementation. Springer, New York, 2007.

[33] Paat Rusmevichientong, Zuo-Jun Max Shen, and David B. Shmoys. Dynamic assortment optimization with a multinomial logit choice model and capacity constraint. Operations Research, 58(6):1666–1680, 2010.

[34] Eric M. Schwartz, Eric T. Bradlow, and Peter S. Fader. Customer acquisition via display advertising using multi-armed bandit experiments. Marketing Science, 36(4):500–522, 2017.

[35] Steven L. Scott. A modern bayesian look at the multi-armed bandit. Applied Stochastic Models in Business and Industry, 26(6):639–658, 2010.

[36] Steven L. Scott. Multi-armed bandit experiments, 2013. Google Analytics documentation.

[37] William R. Thompson. On the likelihood that one unknown probability exceeds another in view of the evidence of two samples. Biometrika, 25(3/4):285–294, 1933.

[38] Sebastian Thrun and Lorien Pratt, editors. Learning to Learn. Springer Science & Business Media, 2012.

[39] Sabina Tomkins, Peng Liao, Predrag Klasnja, and Susan Murphy. Intelligentpooling: Practical thompson sampling for mhealth. Machine Learning, 110(9):2685–2727, 2021.

[40] John N. Tsitsiklis. A lemma on the multi-armed bandit problem. IEEE Transactions on Automatic Control, 31(6):576–577, 1986.

[41] Sofía S. Villar, Jack Bowden, and James Wason. Multi-armed bandit models for the optimal design of clinical trials: Benefits and challenges. Statistical Science, 30(2):199–215, 2015.

[42] Yingfei Wang, Hamed Mamani, David G. Coffey, and Zhijin Zhou. How do tumor cytogenetics inform cancer treatments? dynamic risk stratification and precision medicine using multi-armed bandits, 2019. SSRN working paper.

[43] Yining Wang and Warren B. Powell. An optimal learning method for developing personalized treatment regimes, 2016. arXiv preprint arXiv:1607.01462.

[44] Yining Wang, Chenguang Wang, and Warren B. Powell. The knowledge gradient for sequential decision making with stochastic binary feedbacks. In International Conference on Machine Learning, pages 1138–1147, 2016.

[45] John Myles White. Bandit Algorithms for Website Optimization: Developing, Deploying, Debugging. O’Reilly Media, Sebastopol, CA, 2013.

[46] Peter Whittle. Multi-armed bandits and the gittins index. Journal of the Royal Statistical Society: Series B, 42(2):143–149, 1980.

[47] Peter Whittle. Restless bandits: Activity allocation in a changing world. In A Celebration of Applied Probability, pages 287–298. Applied Probability Trust, 1988.