本研究聚焦於轉移機率分配具有不確定性的環境，建立一個分佈式穩健多臂拉霸機模型。經典多臂拉霸機模型通常假設決策者能夠完整掌握各手臂的動態資訊，但在實際應用中，模型參數可能受到資料噪音、樣本代表性不足及環境變動等因素影響，使根據名目模型所制定的策略產生偏誤。為處理此類不確定性，本研究導入分佈式穩健最佳化架構，透過最壞情況分析提升策略在不同環境下的穩定性。過去研究多採用 Kullback–Leibler 散度衡量轉移機率分配之間的差異。雖然 Kullback–Leibler 散度具有良好的數學結構，有利於封閉形式推導、對偶轉換及演算法設計，但通常要求候選分配與名目分配具有相同支撐集，且主要著重於機率質量差異，未能充分反映事件或狀態閾值之間的排序關係。而本研究採用 Jager–Wellner 差異取代 Kullback–Leibler 散度，並透過由累積分配函數上下界所構成的一階隨機優勢模糊集合，描述轉移機率的不確定性。此方法可放寬相同支撐集的限制，涵蓋離散與連續分配，並同時考量機率質量與閾值排序差異，以更完整地刻畫分配不確定性。根據研究結果顯示，穩健策略能降低策略報酬的波動。雖然在多數情境下需犧牲少量平均報酬作為穩健代價，但其標準差下降幅度明顯，顯示穩健策略能在平均報酬與風險控制之間取得較穩定的平衡。此外，在不確定程度較高的環境中，經典策略較容易受到名目轉移機率分配偏誤影響，而穩健策略不僅能降低報酬波動，部分情境下亦能取得較高的平均報酬。

This study develops a distributionally robust multi-armed bandit model for environments with uncertain transition probability distributions. Classical models assume that decision makers have complete knowledge of each arm’s dynamics information. In practice, such information may be affected by data noise, limited sample representativeness, and environmental changes, causing policies based on the nominal model to become biased. To address this issue, this study introduces a distributionally robust optimization framework and applies worst-case analysis to improve policy stability.

Previous studies commonly use the Kullback--Leibler divergence to measure differences between transition probability distributions. Although it has a convenient mathematical structure. It usually requires the candidate and nominal distributions to share the same support and only penalizes pointwise differences in probability mass, failing to account for the distance or ordering between different support points. This study therefore adopts the Jager--Wellner discrepancy and represents transition uncertainty through a first-order stochastic dominance ambiguity set constructed from lower and upper cumulative distribution function bounds. This approach relaxes the common-support restriction, accommodates both discrete and continuous distributions, and captures differences in both probability mass and threshold ordering.

The results show that the robust policy reduces reward variability. Although it often sacrifices a small amount of average reward as the price of robustness, the reduction in standard deviation is substantial. In highly uncertain environments, the robust policy may also achieve a higher average reward than the classical policy.

Aalto, S., Ayesta, U., & Righter, R. (2009). On the gittins index in the m/g/1 queue.Queueing Systems, 63(1), 437–458.
Abbou, A., & Makis, V. (2019). Group maintenance: A restless bandits approach. INFORMSJournal on Computing, 31(4), 719–731.
Amiri, N. H., Udenio, M., & Boute, R. N. (2023). Adaptive multi-armed bandits for non-stationary inventory control. Available at SSRN 4650653.
Ben-Tal, A., den Hertog, D., Waegenaere, A. D., Melenberg, B., & Rennen, G. (2013).Robust solutions of optimization problems affected by uncertain probabilities. Management Science, 59(2), 341–357.
Ben-Tal, A., & Nemirovski, A. (1998). Robust convex optimization. Mathematics of Operations Research, 23(4), 769–805.
Ben-Tal, A., & Nemirovski, A. (2000). Robust solutions of linear programming problems contaminated with uncertain data. Mathematical Programming, 88, 411–424.
Bertsekas, D. P. (2005). Dynamic Programming and Optimal Control, Volume I. Belmont,Massachusetts: Athena Scientific, third ed.
Bertsimas, D., & Sim, M. (2004). The price of robustness. Operations Research, 52(1),35–53.
Brown, D. B., & Smith, J. E. (2020). Index policies and performance bounds for dynamic selection problems. Management Science, 66(7), 3029–3050.
Caro, F., & Gupta, A. D. (2022). Robust control of the multi-armed bandit problem. Annals of Operations Research, 317, 461–480.
Chen, Y., Guo, Q., Sun, H., Li, Z., Wu, W., & Li, Z. (2018). A distributionally robust optimization model for unit commitment based on kullback–leibler divergence. IEEE Transactions on Power Systems, 33(5), 5147–5160.
Dantzig, G. B. (1955). Linear programming under uncertainty. Management Science,1(3/4), 197–206.
Delage, E., & Ye, Y. (2010). Distributionally robust optimization under moment uncertainty with application to data-driven problems. Operations Research, 58(3), 595–612.
El Ghaoui, L., Lebret, H., & Oustry, F. (1998). Robust solutions to uncertain semidefinite programs. SIAM Journal on Optimization, 9(1), 33–52.
Fu, M., Li, X., & Zhang, L. (2024). Distributionally robust newsvendor under stochastic dominance with a feature-based application. Manufacturing & Service Operations Management, 26(5), 1962–1977.
Ghosal, S., & Wiesemann, W. (2020). The distributionally robust chance-constrained vehicle routing problem. Operations Research, 68(3), 655–964.
Gittins, J. C. (1979). Bandit Processes and Dynamic Allocation Indices, vol. 41. Wiley.
Iyengar, G. N. (2005). Robust dynamic programming. Mathematics of Operations Research, 30(2), 257–280.
Jager, L., & Wellner, J. A. (2007). Goodness-of-fit tests via phi-divergences. Annals of Statistics, 35(5), 2018–2053.
Kim, M. J., & Lim, A. E. (2016). Robust multiarmed bandit problems. Management Science, 62(1), 264–285.
Klabjan, D., Simchi-Levi, D., & Song, M. (2013). Robust stochastic lot-sizing by means of histograms. Production and Operations Management, 22(3), 691–710.
Lim, A. E. B., & Shanthikumar, J. G. (2007). Relative entropy, exponential utility, and robust dynamic pricing. Operations Research, 55(2), 198–214.
Lin, F., Fang, X., & Gao, Z. (2022). Distributionally robust optimization: A review ontheory and applications. Numerical Algebra, Control and Optimization, 12(1), 159–212.
Liu, J., Chen, Z., Lisser, A., & Xu, Z. (2017). Closed-form optimal portfolios of distributionally robust mean-cvar problems with unknown mean and variance. Applied Mathematics and Optimization, 79(3), 663–691.
Mahajan, A., & Teneketzis, D. (2008). Multi-armed bandit problems. In V. Krishnamurthy, & H. V. Poor (Eds.) Foundations and Applications of Sensor Management,(pp. 121–151). New York: Springer.
Mulvey, J. M., Vanderbei, R. J., & Zenios, S. A. (1995). Robust optimization of large scale systems. Operations Research, 43(2), 264–281.
Nilim, A., & El Ghaoui, L. (2005). Robust control of markov decision processes with uncertain transition matrices. Operations Research, 53(5), 780–798.
Ni˜ no-Mora, J. (1996). Conservation laws, extended polymatroids and multiarmed bandit problems: A polyhedral approach to indexable systems. Mathematics of OperationsResearch, 26(2), 257–304.
Petersen, I. R., James, M. R., & Dupuis, P. (2000). Minimax optimal control of stochastic uncertain systems with relative entropy constraints. IEEE Transactions on AutomaticControl, 45(3), 398–412.
Rahimian, H., & Mehrotra, S. (2019). Distributionally robust optimization: A review.arXiv preprint arXiv:1908.05655.
Scarf, H. (1958). A min-max solution of an inventory problem. In S. K. Kenneth J. Arrow,&H.Scarf(Eds.) Studies in the Mathematical Theory of Inventory and Production, (pp.201–209). Redwood City, CA: Stanford University Press.
Scully, Z., Blelloch, G., Harchol-Balter, M., & Scheller-Wolf, A. (2017). Optimally scheduling jobs with multiple tasks. ACM SIGMETRICS Performance Evaluation Review, 45(2), 36–38.
Shapiro, A., & Ahmed, S. (2004). On a class of minimax stochastic programs. SIAM Journal on Optimization, 14(4), 1237–1249.
Sun, L., Xie, W., & Witten, T. (2023). Distributionally robust fair transit resource allocation during a pandemic. Transportation Science, 57(4), 954–978.
Talias, M. A. (2007). Optimal decision indices for r&d project evaluation in the pharma ceutical industry: Pearson index versus gittins index. European Journal of Operational Research, 177(2), 1105–1112.
Thompson, W.R.(1933). Onthelikelihood that one unknownprobability exceeds another in view of the evidence of two samples. Biometrika, 25(3/4), 285–294.
Tsitsiklis, J. N. (1986). A lemma on the multi-armed bandit problem. IEEE Transactions on Automatic Control, 31(6), 576–577.
Villar, S. S., Wason, J., & Bowden, J. (2015). Response-adaptive randomization for multi arm clinical trials using the forward looking gittins index rule. Biometrics, 71(4), 969–978.
Wald, A.(1945). Statistical decision functions which minimize the maximum risk. Annals of Mathematics, 46(2), 265–280.
Wallace, S. W., & Fleten, S.-E. (2003). Stochastic programming models in energy. In Handbooks in Operations Research and Management Science, vol. 10, (pp. 637–677).Elsevier.
Weber, R. R. (1992). On the gittins index for multiarmed bandits. The Annals of Applied Probability, 2(4), 1024–1033.
Whittle, P. (1980). Multi-armed bandits and the gittins index. Journal of the Royal Statistical Society, 42, 143–149.
Xu, H., & Mannor, S. (2012). Distributionally robust markov decision processes. Mathematics of Operations Research, 37(2), 288–300.
Zhang, L., Yang, J., & Gao, R. (2024). Optimal robust policy for feature-based newsvendor. Management Science, 70(4), 2315–2329.