生物為了適應環境從而演化成最適合生存的樣貌。這些生物材料具有相當優異的性質(輕量化，高強度，高韌性)。因為生物結構基本上是由剛性基質和有機質組合組成的複合材料，且同時具備跨越不同尺度的微觀結構，使其具有合成材料所無法達成的非凡機械性能。為了模仿生物材料的卓越材料特性，科學家師法自然，透過探討生物結構材料，並加以學習生物材料其非凡機械性能的原因，進而提出了具有生物啟發性的結構材料的想法。而要效仿其演化過程，我需要在複雜設計空間中將其進行最佳化。然而，複合材料在合理的設計空間尺寸中可能的組合數通常將造成設計上困難。因此，我們提出了一種使用強化學習的新設計方法，以解決生物啟發的複合材料在複雜設計空間中所面臨的設計問題。
在本研究中，我們利用強化學習來為結構設計由脆性和軟性材料組成的微觀結構。我們的模型學習將通過訓練，學習如何設計生物啟發的結構複合材料。我們的設計空間為具預裂縫之2D複合材料結構。並且利用強化學習的方法，經由觀察設計新結構時其性質的變化，學習如何最佳化以獲取更高的破壞韌性，從而獲得更好的機械性質。其設計時的難度會隨著結構的解析度的提升，解析度提升將更有機會能找到更好性質的結構，不過可能組合數將大量的增加，而最佳化所需的時間也會隨之上升。為了加快學習過程，在這裡我們設計了階層式強化學習程序。訓練過程從較小的設計空間開始，在取得當前空間大小之最佳設計後，我們進而擴大了設計空間，並以先前獲得的最佳設計作為初始狀態繼續設計，以利用更高的分辨率尋找最佳組合。
階層式設計的能夠有效的增加搜尋時的收斂性，能夠更快的使其從環境中獲得的回饋收斂，透過由低解析度的結構開始設計，經過多次的迭代並提升解析度（設計空間）後，在各個空間所得出的結果可以觀察到，由較低解析度設計出的最佳結構只在下半部擺放軟材來減少應力集中，在經由解析度的提升後，藉由更為精細的軟硬材料設置，中上部份的軟材對於應力集中的減少，也能有所貢獻。在模型經由迭代後設計出的高分辨率複合結構材料，可以大大降低裂紋尖端的應力集中，使破壞不由預裂縫處開始傳播。在我們的模型設計出了符合我們目標性質的結構後，我們通過3D列印技術的實驗進一步檢查了設計的結果。從實驗結果表明與AI設計有著很好的一致性。本研究所提出之設計框架可作為在具有各種約束的複雜設計空間中進行優化的替代方法。本研究未來將可應用於仿生結構設計，奈米工程和材料設計中。

In this research, we proposed a design paradigm using reinforcement learning to solve the design problem of bioinspired composite materials. Biological structure composites have extraordinary mechanical properties because of their microstructure and material composition. In order to emulate the reasons that make biological materials have extraordinary properties; we need to optimize bioinspired composite materials in a complex design space.
The optimization goal of the reinforcement learning model is the fracture toughness of the material. Our design space is a 2D composite with pre-crack. The model learns how to maximize fracture toughness by changing properties during the design process. Moreover, the difficulty of design will increase when the resolution of the composite increases. Here we have developed a hierarchical design program to speed up the learning process. Hierarchical design is using the result of a smaller design space for larger design space.
In the design result of the 32×32 design space, the design of high-resolution composites greatly reduces the stress concentration at the crack tip. Furthermore, the soft material on the top and middle parts can also contribute to the reduction of stress concentration.
Eventually, we further checked the design results through experiments with 3D printing technology. The experimental results show that there is an excellent consistency with design. In this study, we proposed the design framework. This framework can be used as an alternative method for optimization in complex design space with various constraints.

1. Wegst, U. G. K., Bai, H., Saiz, E., Tomsia, A. P. &Ritchie, R. O. Bioinspired structural materials. 14, 23–36 (2015).
2. Fratzl, P. &Weinkamer, R. Nature’s hierarchical materials. Prog. Mater. Sci. 52, 1263–1334 (2007).
3. Huang, W. et al. Multiscale Toughening Mechanisms in Biological Materials and Bioinspired Designs. 1901561, 1–37 (2019).
4. Zhang, C., Ii, D. A. M. &Grunlan, J. C. Nano / Micro-Manufacturing of Bioinspired Materials : a Review of Methods to Mimic Natural Structures. 6292–6321 (2016) doi:10.1002/adma.201505555.
5. Gao, H. Application of fracture mechanics concepts to hierarchical biomechanics of bone and bone-like materials. Int. J. Fract. 138, 101–137 (2006).
6. Buehler, M. J. Molecular nanomechanics of nascent bone : fibrillar toughening by mineralization. (2007) doi:10.1088/0957-4484/18/29/295102.
7. Nair, A. K., Gautieri, A., Chang, S. &Buehler, M. J. fibrils in bone. Nat. Commun. 1–9 (2013) doi:10.1038/ncomms2720.
8. Fernandes, M. C., Aizenberg, J., Weaver, J. C. &Bertoldi, K. Mechanically robust lattices inspired by deep-sea glass sponges. Nat. Mater. 20, 237–241 (2021).
9. Sundar, V. C., Yablon, A. D., Grazul, J. L., Ilan, M. &Aizenberg, J. Fibre-optical features of a glass sponge. Nature 424, 899–900 (2003).
10. Gu, G. X., Takaffoli, M. &Buehler, M. J. Hierarchically Enhanced Impact Resistance of Bioinspired Composites. Adv. Mater. 29, 1700060 (2017).
11. Magrini, T. et al. Transparent and tough bulk composites inspired by nacre. Nat. Commun. 10, 1–10 (2019).
12. Tang, Z., Kotov, N. A., Magonov, S. &Ozturk, B. Nanostructured artificial nacre. Nat. Mater. 2, 413–418 (2003).
13. Li, A. et al. Nacre-Inspired Composite Electrolytes for Load-Bearing Solid-State Lithium-Metal Batteries. 1905517, 1–9 (2020).
14. Huang, T., Chen, C. &Chang, S. Microcrack patterns control the mechanical strength in the biocomposites. Mater. Des. 140, 505–515 (2018).
15. Lee, S. H. &Wang, S. Biodegradable polymers/bamboo fiber biocomposite with bio-based coupling agent. Compos. Part A Appl. Sci. Manuf. 37, 80–91 (2006).
16. Zhao, H., Yang, Z. &Guo, L. Nacre-inspired composites with different macroscopic dimensions: Strategies for improved mechanical performance and applications. NPG Asia Mater. 10, 1–22 (2018).
17. Guo, K., Yang, Z., Yu, C.-H. &Buehler, M. J. Artificial intelligence and machine learning in design of mechanical materials. Mater. Horizons (2021) doi:10.1039/d0mh01451f.
18. Yang, Z., Yu, C.-H. &Buehler, M. J. Deep Learning Model to Predict Complex Stress and Strain Fields in Hierarchical Composites. Sci. Adv. 1–17.
19. Hsu, Y. C., Yu, C. H. &Buehler, M. J. Using Deep Learning to Predict Fracture Patterns in Crystalline Solids. Matter 3, 197–211 (2020).
20. Yu, C.-H., Qin, Z., Martin-Martinez, F. J. &Buehler, M. J. A Self-Consistent Sonification Method to Translate Amino Acid Sequences into Musical Compositions and Application in Protein Design Using Artificial Intelligence. ACS Nano 13, 7471–7482 (2019).
21. Yu, C. H. &Buehler, M. J. Sonification based de novo protein design using artificial intelligence, structure prediction, and analysis using molecular modeling. APL Bioeng. 4, (2020).
22. Silver, D. et al. Mastering the game of Go with deep neural networks and tree search. Nature 529, 484–489 (2016).
23. Silver, D., Schrittwieser, J., Simonyan, K., Nature, I. A.- &2017, U. Mastering the game of Go without human knowledge. Nature 550, 354 (2016).
24. Yang, Y. &Whinston, A. A Survey on Reinforcement Learning for Combinatorial Optimization. arXiv 4165, 0–2 (2020).
25. Yu, C., Liu, J. &Nemati, S. Reinforcement learning in healthcare: A survey. arXiv (2019).
26. Dennis, M. et al. Emergent complexity and zero-shot transfer via unsupervised environment design. arXiv (2020).
27. Vinitsky, E. et al. using Adversarial Populations.
28. Kalashnikov, D. et al. Qt-opt: Scalable deep reinforcement learning for vision-based robotic manipulation. arXiv 1–23 (2018).
29. Bojarski, M. et al. End to End Learning for Self-Driving Cars. 1–9 (2016).
30. Sallab, A.El, Abdou, M., Perot, E. &Yogamani, S. Deep reinforcement learning framework for autonomous driving. arXiv 70–76 (2017).
31. Bhalla, S., Ganapathi Subramanian, S. &Crowley, M. Deep Multi Agent Reinforcement Learning for Autonomous Driving. Lect. Notes Comput. Sci. (including Subser. Lect. Notes Artif. Intell. Lect. Notes Bioinformatics) 12109 LNAI, 67–78 (2020).
32. Köker, R. A genetic algorithm approach to a neural-network-based inverse kinematics solution of robotic manipulators based on error minimization. Information Sciences vol. 222 528–543 (2013).
33. Duka, A.-V. Neural Network based Inverse Kinematics Solution for Trajectory Tracking of a Robotic Arm. Procedia Technol. 12, 20–27 (2014).
34. Gu, G. X., Dimas, L., Qin, Z. &Buehler, M. J. Optimization of Composite Fracture Properties: Method, Validation, and Applications. J. Appl. Mech. 83, 71006 (2016).
35. Yu, C.-H., Qin, Z. &Buehler, M. J. Artificial intelligence design algorithm for nanocomposites optimized for shear crack resistance. Nano Futur. (2019) doi:10.1088/2399-1984/ab36f0.
36. Gay, G. R., Salomoni, P. &Mirri, S. E-Learning. Encycl. Internet Technol. Appl. 292, 179–184 (2007).
37. Even-Dar, E., Kakade, S. M. &Mansour, Y. Experts in a markov decision process. Adv. Neural Inf. Process. Syst. (2005).
38. Tokic, M. &Palm, G. Value-difference based exploration: Adaptive control between epsilon-greedy and softmax. Lect. Notes Comput. Sci. (including Subser. Lect. Notes Artif. Intell. Lect. Notes Bioinformatics) 7006 LNAI, 335–346 (2011).