Table of Contents

Section I Introduction

Chapter 1 □ Overview of Reinforcement Learning

Chapter 2 □ Overview of CyberPhysical Systems and Cybersecurity

Section II Reinforcement Learning for Cyber-Physical Systems

Chapter 3 □ Reinforcement Learning Problems

Chapter 4 □ Modelbased Reinforcement Learning

Chapter 5 □ Modelfree Reinforcement Learning

Chapter 6 □ Deep Reinforcement Learning

Section III Case Studies

Chapter 7 □ Reinforcement Learning for Cybersecurity

Chapter 8 □ Case Study: Online CyberAttack Detection in Smart Grid

Chapter 9 □ Case Study: Defeat Maninthemiddle Attack

About the Author

Chong Li is co-founder of Nakamoto & Turing Labs Inc. He is Chief Architect and Head of Research at Canonchain Network. He is also an adjunct assistant professor at Columbia University. Dr. Li was a staff research engineer in the department of corporate R&D at Qualcomm Technologies. He received a B.E. in Electronic Engineering and Information Science from Harbin Institute of Technology and a Ph.D in Electrical and Computer Engineering from Iowa State University.

Dr. Li’s research interests include information theory, machine learning, blockchain, networked control and communications, coding theory, PHY/MAC design for 5G technology and beyond. Dr. Li has published many technical papers in top-ranked journals, including Proceedings of the IEEE, IEEE Transactions on Information Theory, IEEE Communications Magazine, Automatica, etc. He has served as session chair and technical program committee for a number of international conferences. He has also served as reviewer for many prestigious journals and international conferences, including IEEE Transactions on Information Theory, IEEE Transactions on Wireless Communication, ISIT, CDC, ICC, WCNC, Globecom, etc. He holds 200+ international and U.S. patents (granted and pending) and received several academic awards including the MediaTek Inc. and Wu Ta You Scholar Award, the Rosenfeld International Scholarship and Iowa State Research Excellent Award. At Qualcomm, Dr. Li significantly contributed to the systems design and the standardization of several emerging key technologies, including LTE-D, LTE-controlled WiFi and 5G. At Columbia University, he has been instructing graduate-level courses, such as reinforcement learning, blockchain technology and convex optimization, and actively conducting research in the related field. Recently, Dr. Li has been driving the research and development of blockchain-based geo-distributed shared computing, and managing the patent-related business at Canonchain.

Meikang Qiu received the BE and ME degrees from Shanghai Jiao Tong University and received Ph.D. degree of Computer Science from University of Texas at Dallas. Currently, he is an Adjunct Professor at Columbia University and Associate Professor of Computer Science at Pace University. He is an IEEE Senior member and ACM Senior member. He is the Chair of IEEE Smart Computing Technical Committee. His research interests include cyber security, cloud computing, big data storage, hybrid memory, heterogeneous systems, embedded systems, operating systems, optimization, intelligent systems, sensor networks, etc.