Collaborative Research: CNS Core: Medium: Foundations and Scalable Algorithms for Personalized and Collaborative Virtual Reality Over Wireless Networks

Personnel

Principal Investigator

  • Bin Li, Associate Professor, The Pennsylvania State University
  • Feng Qian, Associate Professor, University of Minnesota Twin Cities
  • Atilla Eryilmaz, Professor, The Ohio State University
  • R. Srikant, Professor, University of Illinois at Urbana-Champaign

Project Goals

Virtual reality (VR) over wireless networks can provide an interactive and immersive experience for multiple users simultaneously and thus has many applications, especially in VR-based education/training. However, satisfactory personalized user experience in such wireless immersive services demands stringent performance requirements, including: (1) high-speed and high-resolution panoramic image rendering; (2) extremely low delay guarantees; and (3) seamless user experience. Besides the aforementioned requirements, collaborative user experience requires both scalability and fairness of VR service. Existing VR systems heavily rely on various heuristic designs and do not efficiently exploit VR content commonality and its predictability, which impede their large-scale deployment. In this proposal, we aim to develop the theoretical foundations and complete implementation of a system for providing both personalized and scalable collaborative VR experience over wireless networks. This project will integrate machine learning, wireless networking, and mobile computing to enable high-quality and scalable wireless immersive applications on commodity mobile devices. The theory and practical implementations to be developed in this project will be integrated into both undergraduate and graduate curriculum, as well as exposing K-12 students to state-of-the-art wireless and VR technologies.

Our designs are motivated by a number of insights that we have developed from our preliminary work, including (1) viewport-adaptive rendering; (2) commonality among VR content for multiple users to enable multicasting; and (3) predictability of VR content to enable prefetching. The proposed research will contribute to and advance both theoretical and system-oriented research in the fields of wireless networks and virtual reality. We explicitly exploit the unique characteristics of both immersive VR applications and wireless networks, and propose the following four interdependent research thrusts: (I) Dealing with network and prediction uncertainties: This thrust will investigate algorithm designs to optimize personalized user experience given both network and viewport prediction uncertainties. (II) Meeting stringent immersive service requirements: This thrust will develop wireless scheduling algorithms that provide stringent immersive, personalized service guarantees for multiple VR users. (III) Supporting smooth collaborative interaction: This thrust will focus on the algorithm design that leverages the VR content similarities and predictabilities that naturally emerge during collaborative interactions. (IV) Scalable system integration, implementation, evaluation, and deployment: This thrust will integrate Research Thrusts I through III into a holistic system, perform system-level optimizations, and evaluate it through lab experiments and real-world classroom deployment.

Publications

  1. Xiaoyi Wu, Bin Li, Achieving Regular and Fair Learning in Combinatorial Multi-Armed Bandit, In Proc. IEEE International Conference on Computer Communications (INFOCOM), Vancouver, Canada, May, 2024. [ Acceptance rate: 19.6%]
  2. Jiangong Chen, Xudong Qin, Guangyu Zhu, Bo Ji, and Bin Li, Motion-Prediction-based Wireless Scheduling for Interactive Panoramic Scene Delivery, accepted by IEEE Transactions on Network Science and Engineering, 2023.
  3. Jiangong Chen, Tian Lan, Bin Li, GPT-VR Nexus: ChatGPT-Powered Immersive Virtual Reality Experience, In Proc. IEEE Conference on Virtual Reality and 3D User Interfaces (VR), Orlando, FL, USA, March, 2024 (demo paper). [Best Demo Honorable Mention Award]
  4. Xiaoyi Wu, Jiangong Chen, Rui Tang, Kefan Wu, Bin Li, Demo: Immersive Remote Monitoring and Control for Internet of Things, In ACM International Symposium on Mobile Ad Hoc Networking and Computing (MOBIHOC), October, 2023 (demo paper). [Best Poster/Demo Award]
  5. Anlan Zhang, Chendong Wang, Yuming Hu, Ahmad Hassan, Zejun Zhang, Bo Han, Feng Qian, and Shichang Xu, Habitus: Boosting Mobile Immersive Content Delivery through Full-body Pose Tracking and Multipath Networking, To Appear in NSDI 2024.
  6. Qiao Jin, Yu Liu, Ruixuan Sun, Chen Chen, Puqi Zhou, Bo Han, Feng Qian, and Svetlana Yarosh, Collaborative Online Learning with VR Video: Roles of Collaborative Tools and Shared Video Control, In ACM CHI 2023, Hamburg, Germany.
  7. Zixian Yang, R. Srikant, Lei Ying, Learning While Scheduling in Multi-Server Systems With Unknown Statistics: MaxWeight with Discounted UCB, Proc. of The 26th International Conference on Artificial Intelligence and Statistics, 2023.
  8. Yashaswini Murthy, Mehrdad Moharrami, R. Srikant, Modified Policy Iteration for Exponential Cost Risk Sensitive MDPs, Proc. of The 5th Annual Learning for Dynamics and Control Conference, 2023.
  9. Anna Winnicki, R. Srikant, On The Convergence Of Policy Iteration-Based Reinforcement Learning With Monte Carlo Policy Evaluation, Proc. of The 26th International Conference on Artificial Intelligence and Statistics, 2023.
  10. Semih Cayci, Siddhartha Satpathi, Niao He, R. Srikant, Sample Complexity and Overparameterization Bounds for Temporal-Difference Learning With Neural Network Approximation, IEEE Transactions on Automatic Control, 2023.
  11. Ronshee Chawla, Daniel Vial, Sanjay Shakkottai, R. Srikant, Collaborative Multi-Agent Heterogeneous Multi-Armed Bandits, In International Conference on Machine Learning (ICML), 2023.
  12. Xiaoyi Wu, Jing Yang, Huacheng Zeng, Bin Li, Joint User Association and Wireless Scheduling with Smaller Time-Scale Rate Adaptation, In International Symposium on Modeling and Optimization in Mobile, Ad hoc, and Wireless Networks (WiOpt), Singapore, August, 2023.
  13. Zhanchen Dong, Jiangong Chen, Bin Li, Demo: Collaborative Mixed-Reality-Based Firefighter Training, In Proc. IEEE International Conference on Computer Communications (INFOCOM), May, 2023 (demo paper). [Best Demo Award]
  14. Daniel Vial, Sanjay Shakkottai, R Srikant, Robust Multi-Agent Bandits Over Undirected Graphs, Proc. of the ACM on Measurement and Analysis of Computing Systems, 2022.
  15. Daniel Vial, Advait Parulekar, Sanjay Shakkottai, R Srikant, Regret Bounds for Stochastic Shortest Path Problems with Linear Function Approximation, Proc. of the 39th International Conference on Machine Learning, 2022.
  16. Shi, Z. and Eryilmaz, A.,  A Bayesian Approach for Stochastic Continuum-armed Bandit with Long-term Constraints. AISTATS., 2022
  17. Zhou, Xujin and Koprulu, Irem and Eryilmaz, Atilla and Neely, Michael J.,  Efficient Distributed MAC for Dynamic Demands: Congestion and Age Based Designs. IEEE/ACM Transactions on Networking. 1 to 14. Published. 1063-6692, 2022
  18. Quan, G. and Eryilmaz, A. and Shroff, N.,  Regret-Optimal Learning for Minimizing Edge Caching Service Costs. WiOpt, 2022
  19. Abolhassani, B. and Tadrous, J. and Eryilmaz, A. and Yeh, E,  Fresh Caching of Dynamic Content Over the Wireless Edge. 30. (5). IEEEACM transactions on networking, 30. 2315-2327. Published. 1558-2566, 2022
  20. Abolhassani, Bahman and Tadrous, John and Eryilmaz, Atilla,  Single vs Distributed Edge Caching for Dynamic Content. 30. (2). IEEE/ACM Transactions on Networking, 30. 669 to 682. Published. 1063-6692, 2022
  21. Cayci, S. and Zheng, Y. and Eryilmaz, A,  A Lyapunov-Based Methodology for Constrained Optimization with Bandit Feedback. Proceedings of the AAAI Conference on Artificial Intelligence. Published. 2159-5399, 2022
  22. Jiangong Chen, Feng Qian, Bin Li, Enhancing Quality of Experience for Collaborative Virtual Reality with Commodity Mobile Devices, In Proc. IEEE International Conference on Distributed Computing Systems (ICDCS), July, 2022.
  23. Harsh Gupta, Jiangong Chen, Bin Li, R. Srikant, Online Learning-Based Rate Selection for Wireless Interactive Panoramic Scene Delivery, In Proc. IEEE International Conference on Computer Communications (INFOCOM), May, 2022.
  24. Xinyi Yao, Jiangong Chen, Ting He, Jing Yang, Bin Li, A Scalable Mixed Reality Platform for Remote Collaborative LEGO Design, In Proc. IEEE International Conference on Computer Communications (INFOCOM), May, 2022 (demo paper).
  25. Michael Artlip, Jiangong Chen and Bin Li, “Virtual Reality-Based Gymnastics Visualization Using Real-Time Motion Capture Suit”, National Workshop for REU Research in Networking and Systems (REUNS), Denver, CO, October, 2022 (demo paper). [Michael is an undergraduate student who worked on my NSF REU supplemental project]
  26. Owen Sweeny, Zhanchen Dong and Bin Li, “A Collaborative Augmented Reality Platform for Interactive and Immersive Education”, National Workshop for REU Research in Networking and Systems (REUNS), Denver, CO, October, 2022 (demo paper). [Owen is an undergraduate student who worked on my NSF REU supplemental project]
  27. Daniel Vial, Sujay Sanghavi, Sanjay Shakkottai, R. Srikant, Minimax Regret for Cascading Bandits, Advances in Neural Information Processing Systems 35 (NeurIPS 2022).
  28. Daniel Vial, Advait Parulekar, Sanjay Shakkottai, R Srikant, Regret Bounds for Stochastic Shortest Path Problems with Linear Function Approximation, Proceedings of the 39th International Conference on Machine Learning, PMLR 162:22203-22233, 2022.
  29. Qiao Jin, Yu Liu, Svetlana Yarosh, Bo Han, and Feng Qian, How Will VR Enter University Classrooms? Multi-stakeholders Investigation of VR in Higher Education, In ACM CHI 2022, New Orleans, LA.
  30. Yu Liu, Bo Han, Feng Qian, Arvind Narayanan, and Zhi-Li Zhang, Vues: Practical Volumetric Video Streaming through Multiview Transcoding, In ACM MobiCom 2022, Sydney, Australia.
  31. Anna Winnicki, Joseph Lubars, Michael Livesay, R. Srikant, The Role of Lookahead and Approximate Policy Evaluation in Reinforcement Learning with Linear Value Function Approximation, arXiv:2109.13419, 2021.
  32. Abolhassani, B. and Tadrous, J. and Eryilmaz, A, Optimal Load-Splitting and Distributed-Caching for Dynamic Content. Proceedings of the International Symposium on Modeling and Optimization in Mobile Ad Hoc and Wireless Networks. Published. 2690-3334, 2021
  33. Kang, Sunjung and Eryilmaz, Atilla and Shroff, Ness B.,  Remote Tracking of Distributed Dynamic Sources over A Random Access Channel with One-bit Updates. WIOPT. Published, 2021
  34. Zhou, Xujin and Koprulu, Irem and Eryilmaz, Atilla and Neely, Michael J., Low-Overhead Distributed MAC for ServingDynamic Users over Multiple Channels. Proc. 19th international symposium on modeling and optimization in mobile, ad hoc, and wireless networks (WiOpt), 2021.

Educational Activities

  • [Fall 2023] Dr. Bin Li taught EE 360 Communication Systems. I introduced Virtual/Augmented Reality applications and highlighted the need for low-latency communication design.
  • [Spring 2023]: Dr. Atilla Eryilmaz taught a new course on “convex and stochastic optimization”, numbered ECE 6500, which talks about the design and analysis of optimization methods for stochastic problems, as those arising in the context of this project.
  • [Fall 2022]: Dr. R. Srikant taught a course on MDPs and Reinforcement Learning, where the material from the research on this project was useful in providing practical examples of RL applications.
  • [Fall 2022]: Dr. Bin Li integrated research findings from this project into EE 597 Optimization and Learning for Communication Networks.
  • [September 2022]: Dr. Bin Li was invited to give a guest lecture in EE 396 (research topics course for EE honor students) on “Collaborative Virtual/Augmented Reality”.
  • [August 2022]: Dr. R. Srikant gave a tutorial on Foundations of Reinforcement Learning at the Simons Institute.
  • [July 2022 ]: Dr. Bin Li was invited to give a keynote talk “Wireless Networking for Personalized and Collaborative Virtual Reality” at the ICDCS workshop SocialMeta.
  • [April 2022]:  Dr. Bin Li was invited to give a guest lecture “Wireless Networking for Personalized and Collaborative Virtual Reality” in CS 638 at Purdue University.
  • [November 2021]:  Dr. Bin Li was invited to give a talk on networking for virtual/augmented reality at Eta Kappa Nu (HKN) Research Night.

Outreach Activities

  • [June 2024] EECS summer campers visited Dr. Bin Li’s lab and learned various virtual/augmented reality (VR/AR) applications.
  • [November 2023] Dr. Bin Li demonstrated virtual/augmented reality (VR/AR) applications to kids at Park Forest Middle School.
  • [November 2022]: Dr. Bin Li demonstrated virtual/augmented reality (VR/AR) applications to kids at Mount Nittany Elementary School.
  • [July 2022]: EE summer campers visited Dr. Bin Li’s lab and learned virtual reality-based gymnastics.

  • [June 2022]: Dr. Bin Li co-organized a one-week “Design Your Own Reality” Summer Camp for K7-K9 female students. His group introduced basic virtual/augmented reality concepts to campers and developed basic virtual/augmented reality lab tutorials that deploy Nittany Lion in Google Pixels.

  • [May 2022]: Dr. Bin Li introduced virtual/augmented reality (VR/AR) technology at the Gray’s Woods Elementary School and demonstrated various VR/AR applications to kids.
  • [February 2022]: Dr. Bin Li was invited to be a guest on The Peggy Smedley Show to introduce virtual/augmented reality technology and its applications in education, manufacturing and online shopping. Peggy Smedley show is ranked as the No. 1 IoT, digital transformation, and sustainability podcast.