Compact and Energy-Efficient Compute-in-Memory Accelerator for Deep Learning Leveraging Ferroelectric Vertical NAND Memory

vdp5074 on

Overview:

The field of artificial intelligence (AI) has recently made significant strides, with notable advancements such as large language models like ChatGPT taking the world by storm. However, these breakthroughs would not have been possible without the availability of powerful computing hardware, such as graphics processing units (GPUs). Such hardware has benefited from several decades of technology scaling following Moore’s law. As technology approaches its physical limits and AI models require exponentially increasing hardware resources, including computation and storage, alternative computing paradigms with superior energy efficiency and performance are necessary for a sustainable future. Compute-in-memory is one promising approach where computations are directly performed in memory units, eliminating most data movements, a key bottleneck in conventional computers. However, to best exploit the compute-in-memory for acceleration of AI models on the scale of giga-byte to tera-byte levels, it is critical to have high capacity, energy-efficient, and high performance memory technology to fit the models. NAND memory is a form of erasable programmable read-only memory that takes its name from the not-and (NAND) logic gate. The proposed research aims to develop ferroelectric vertical NAND memory to meet these demands and at the same time train students for developing a future workforce for the semiconductor industry.

Vertical NAND memory offers the highest density by increasing the number of stacked layers vertically. However, conventional vertical NAND memory based on floating gate or charge trap flash suffers from poor performance, including high write voltage, low speed, and poor endurance, despite their large capacity. To address these issues, this research proposes the development of a vertical NAND flash alternative: the vertical NAND ferroelectric field-effect transistor (FeFET), which achieves high density and high performance simultaneously. By leveraging the recently discovered ferroelectric HfO2, superior performance can be achieved as ferroelectric programming is driven by an applied electric field, which can be energy-efficient and fast. The project aims to design and evaluate vertical NAND FeFET-based compute-in-memory accelerators from devices to architectures, with innovations such as novel cell designs to achieve multi-level cell and variation suppression, vertical NAND array disturb mitigation with a novel array structure, and mapping and benchmarking of various important information processing tasks to the vertical NAND FeFET array. Additionally, this research includes workforce training activities such as lectures and hands-on experience offered to K-12 students and teachers to promote excitement and attract them to the talent pipeline for the semiconductor industry. The proposed research will recruit graduate and undergraduate students via the Research Experience for Undergraduates (REU) program from underrepresented groups, and the knowledge acquired in this project will be distributed through curriculum development and online sharing repositories.

EFRI BRAID: Neuroscience Inspired Visual Analytics

vdp5074 on

Overview:

Intelligence is determined by how efficiently a machine learns and stores knowledge about the world, enabling it to handle unanticipated tasks and new environments, learn rapidly without supervision, explain decisions, deduce the unobserved, and anticipate the likely outcomes. A key limitation of current machine learning approaches is their specialized training for different tasks, environments and contexts to achieve desirable accuracies. However, such an approach is not scalable for a visual assistant system that faces an increasing variety of tasks, contexts and environments during the lifetime of the assisted individual. In contrast, the envisioned system in this project will effectively leverage key principles embodied in human intelligence such as feedback from the continuously evolving memory, and higher cognitive processes like attention, knowledge models, and decision making. Through coordinated hardware-software codesign inspired by neuroscience principles, the project envisions a new energy-efficient visual analytics paradigm that can provide persons with visual impairments assistance in a variety of tasks comparable to that of human assistants in a portable form-factor. The envisioned AI cognitive assistant will understand the entire visual field, reason about relationships between objects, and deduce how they relate to the person?s goals (e.g., navigating to a specific location, accomplishing a specific task, or summarizing what is going on in the surroundings). These advances will be a significant leap from current assistive systems that are rudimentary in their assistive capabilities, limited primarily to object detection and navigation in controlled settings with no support for personalized adaptation or continuous learning.

Biological brains, under evolutionary and environmental pressure to survive, cannot afford the luxury of lengthy retraining for every new combination of environment, task, goal, and context. Instead, mechanisms such as attention have evolved to dynamically prioritize information based on these combined factors. This project envisions enhancing the efficiency and accuracy of traditional AI systems by embedding top-down attention mechanisms that selectively process spatial and temporal regions of the input that are most relevant. Through support for continuous lifelong learning with selective model interleaving at different scales by exploiting the hierarchical structure of knowledge, the envisioned system will dynamically adapt the effort for learning and inference to enable energy-proportional computing that is commensurate with the task and environmental complexity. Some of these neuroscience principles will be embedded in Field Programmable Gate Array (FPGA) based hardware fabrics and custom hybrid CMOS-ferroelectric-based hardware fabrics to accentuate the efficiency benefits through the co-design of hardware and algorithms. The project will evaluate our innovations in an engineered system that serves as a cognitive visual assistant for persons with visual impairments. This work will enable a new generation of cognitive vision systems that can abstract, learn, adapt and reason akin to sighted human assistants, transforming the landscape of currently available visual assist systems that are rudimentary compared to human assistants. While machine learning approaches are widely used in many applications, they are not easy to adapt to new environments or tasks. The energy consumed by training new machine learning models is placing enormous demands on global power consumption. In contrast, the proposed neuroscience-inspired visual analytics system can adapt to new environments and tasks with a fraction of the cost of traditional systems by leveraging attention and life-long learning principles. The resulting energy-efficient visual analytics will enhance the independence of persons with visual impairments, transforming current smart assistive gadgets to virtual companions, and greatly enhancing the range of activities for assistance. Educational, outreach and broadening participation initiatives are integral to the various components this project.

Funding:

National Science Foundation

Collaborators:

  • Laurent Itti –  University of Southern California
  • Anand Raghunathan – Purdue University
  • Abhronil Sengupta – Penn State University
  • Mehrdad Mahdavi – Penn State University

NRT-HDR: Interdisciplinary Studies in Entomology, Computer Science and Technology NETwork (INSECT NET)

vdp5074 on

Overview:

The NRT-HDR: Interdisciplinary Studies in Entomology, Computer Science and Technology NETwork (INSECT NET) is a transdisciplinary program at Penn State University that aims to empower students to develop solutions to the insect biodiversity crisis. The program is funded by a $3 million grant from the National Science Foundation (NSF).

INSECT NET is a transdisciplinary program integrating the life sciences, computer science, engineering, and data science, while leveraging advances in artificial intelligence. INSECT NET’s unique solutions-driven approach – where collaborative student teams develop Cyber Physical Systems to address stakeholder needs through coursework, internships and research projects – encourages creative and dynamic problem-solving, empowering students to readily adapt to new challenges, concepts, and teams. Trainees will develop a growth mindset, a convergence mindset, and skills in team science.

Trainees will gain competencies in the following:

  • Insect systematics
  • Ecology
  • Sensor design
  • Energy efficient optimizations
  • Continual learning and adaptation of AI systems
  • Robotic systems
  • Data integration and data management, including designing visualization tools accessible to stakeholders.

Trainees will receive formal instruction in communicating science to diverse audiences (including the public and K-12 schools), project management, and best practices for convergence research. Trainees will expand their professional skills and networks through internships, entrepreneurship programs, and engagement with our Stakeholder Advisory Board.

INSECT NET’s training opportunities will be formalized in a new Graduate Certificate in Technology and AI for Agricultural and Ecological Science at Penn State. INSECT NET’s unique recruitment pipeline will connect prospective students with current students and alumni, leveraging new and existing partnerships with Minority Serving Institutions.

Funding:

National Science Foundation Research Traineeship award

Collaborators:

  • Christina Grozinger
  • Anthony Robinson
  • Bo Cheng
  • Huanyu Cheng

 

 

FuSe-TG: FAB: A Heterogeneous Ferroelectronics Platform for Accelerating Big Data Analytics

vdp5074 on

Overview:

Data generated over social media, biomedical monitoring, and the Internet-of-Things are increasing exponentially. The ability to extract useful information from this big data is critical towards advances in varied domains. The enormity of the data is outstripping the ability for electronic systems to process, communicate and store the data that is being generated. While historically transistor scaling helped address the growth in computational needs, with scaling now reaching its limits, new approaches are needed. To this end, this project brings together experts for realizing a new ferroelectric-based platform that can provide unprecedented opportunities to scale chip designs in the third dimension, to blur the gap between memory and logic functionality through seamless integration to accelerate big data analytics. We have gathered together researchers with cross-layer expertise from materials to systems to create a ferroelectronics platform to provide transformative impetus towards accelerating big data analytics. The teaming exercise will help draw in additional expertise and engagement in workshops and pilot projects. Our cross-layer strategies will lead to concurrent advances such as novel multi-gate devices that leverage the polarization for compact implementation of complex functions, new architectures that harness the intrinsic properties of ferroelectrics, such as polarization domains and optical sensitivity, to support novel security and sensor features. and in-storage computing to support in-place analysis of large amounts of data.

Novel ferroelectric systems will have transformative impact on energy efficiency of computing infrastructure from large data centers to the Internet-of-Things. A highly capable workforce will be the foundation of global leadership in this emerging area. Consequently, strategies for workforce development in partnership with community colleges, four-year institutions and industry will be a key focus. Our planning effort will also invest in pilot cross-institutional projects to accelerate engagement of individuals from underrepresented communities. Planned workshops will assess the educational structure and training programs along with process integration and technology ramp-up to rapidly establish the United States as the center of ferroelectric innovation.

Funding:

National Science Foundation

Collaborator:

  • Suman Datta – Georgia Institute of Technology
  • Anand Raghunathan – Purdue University
  • Santosh Kurinec – Rochester Institute of Technology
  • Jon Ihlefeld – University of Virginia

 

PRISM – Processing with Intelligent Storage and Memory.

vdp5074 on

Overview

Data volumes are growing faster than Moore’s Law and outstripping the total silicon production capacity. Future personalized, secure, data-intensive, and interactive applications require massive, distributed memory and storage. Systems have to compute in real-time – they need to digest the data as it is being generated, integrate it, and select carefully what should be stored and where while optimizing for energy efficiency, reliability, and security. Data movement costs bottleneck today’s systems.

PRISM aims to explore revolutionary pathways that will deliver long-term industry impact. We will build adaptive and flexible HW or SW “prisms”, to replace the “blocks” that are a byproduct of today’s rigid abstraction layers. Combinations of such HW and SW components will expose the right capabilities and interfaces at the right time and place, resulting in holistic cross-layer IMS codesign, without affecting programming complexity. Instead of a rigid hierarchy that exists today, we envision an IMS system where the software layer adapts to existing frameworks for compatibility and distributes work throughout the system for maximum performance and efficiency while ensuring security. The architecture layer unifies near-data and distributed computing by leveraging CXL and heterogeneous chipset integration technologies, efficiently and securely integrating traditional and computational memory sub-systems.

Read More

FET: Leveraging Monolithic 3D for Architectural Innovations

vdp5074 on

Overview:

The design of more efficient processors is a major driving force of the computing industry. Over multiple decades, reducing the size of transistors, which are the building blocks of microprocessors, has been the main driver for efficiency improvements. With this trend of transistor shrinking nearing the physical limit, this research explores three-dimensional chips, where transistors are stacked in multiple layers, as an alternate approach to enhance efficiency. A major challenge with three-dimensional chips is the large dimension of the connections compared to the transistor itself. This project leverages a new technology called monolithic three-dimensional design (M3D), where inter-layer connections are of dimensions similar to that of a transistor. This research results in new M3D-based designs for microprocessors that reduce energy and time consumed in interconnections, and enable new functional features for microprocessors.

The research will first develop a simulation infrastructure that enables systematic exploration of the space by varying the architectural and technology parameters. Leveraging the simulation infrastructure, this project will perform the following tasks:

  • Evaluation of layer-wise partitioning of processor core and cache structures
  • Design of primitive compute functions for integration with the cache memories
  • Design of M3D-enabled application-specific accelerators
  • Integration of heterogeneous memory technologies to provide support for power efficiency.

Research outcomes:

1. Caminal, Helena and Yang, Kailin and Srinivasa, Srivatsa and Ramanathan, Akshay Krishna and Al-Hawaj, Khalid and Wu, Tianshu and Narayanan, Vijaykrishnan and Batten, Christopher and Martinez, Jose F. “CAPE: A Content-Addressable Processing Engine” IEEE Int’l Symp. on High-Performance Computer Architecture (HPCA) , 2021 https://doi.org/10.1109/HPCA51647.2021.00054

2. Deng, Shan and Benkhelifa, Mahdi and Thomann, Simon and Faris, Zubair and Zhao, Zijian and Huang, Tzu-Jung and Xu, Yixin and Narayanan, Vijaykrishnan and Ni, Kai and Amrouch, Hussam “Compact Ferroelectric Programmable Majority Gate for Compute-in-Memory Applications” 2022 International Electron Devices Meeting (IEDM) , 2022 https://doi.org/10.1109/IEDM45625.2022.10019400

3. Xiao, Yi and Xu, Yixin and Jiang, Zhouhang and Deng, Shan and Zhao, Zijian and Mallick, Antik and Sun, Limeng and Joshi, Rajiv and Li, Xueqing and Shukla, Nikhil and Narayanan, Vijaykrishnan and Ni, Kai “On the Write Schemes and Efficiency of FeFET 1T NOR Array for Embedded Nonvolatile Memory and Beyond” IEDM , 2022 https://doi.org/10.1109/IEDM45625.2022.10019542

4. Ma, Xiaoyang and Zhong, Hongtao and Xiu, Nuo and Chen, Yiming and Yin, Guodong and Narayanan, Vijaykrishnan and Liu, Yongpan and Ni, Kai and Yang, Huazhong and Li, Xueqing “CapCAM: A Multilevel Capacitive Content Addressable Memory for High-Accuracy and High-Scalability Search and Compute Applications” IEEE Transactions on Very Large Scale Integration (VLSI) Systems , v.30 , 2022 https://doi.org/10.1109/TVLSI.2022.3198492

5. Bashar, Mohammad Khairul and Vaidya, Jaykumar and Surya Kanthi, R. S. and Lee, Chonghan and Shi, Feng and Narayanan, Vijaykrishnan and Shukla, Nikhil “Ferroelectric-based Accelerators for Computationally Hard Problems” GLSVLSI ’21: Proceedings of the 2021 on Great Lakes Symposium on VLSI , 2021 https://doi.org/10.1145/3453688.3461745

6. Ramanathan, Akshay Krishna and Rangachar, Srivatsa Srinivasa and Govindarajan, Hariram Thirucherai and Hung, Je-Min and Lee, Chun-Ying and Xue, Cheng-Xin and Huang, Sheng-Po and Hsueh, Fu-Kuo and Shen, Chang-Hong and Shieh, Jia-Min and Yeh, Wen-Kuan and H “CiM3D: Comparator-in-Memory Designs Using Monolithic 3-D Technology for Accelerating Data-Intensive Applications” IEEE Journal on Exploratory Solid-State Computational Devices and Circuits , v.7 , 2021 https://doi.org/10.1109/JXCDC.2021.3087745

7. Wang, Jianfeng and Xiu, Nuo and Wu, Juejian and Chen, Yiming and Sun, Yanan and Yang, Huazhong and Narayanan, Vijaykrishnan and George, Sumitha and Li, Xueqing “An 8T/Cell FeFET-Based Nonvolatile SRAM with Improved Density and Sub-fJ Backup and Restore Energy” 2022 IEEE International Symposium on Circuits and Systems (ISCAS) , 2022 https://doi.org/10.1109/ISCAS48785.2022.9937438

8. Tang, Wenjun and Lee, Mingyen and Wu, Juejian and Xu, Yixin and Yu, Yao and Liu, Yongpan and Ni, Kai and Wang, Yu and Yang, Huazhong and Narayanan, Vijaykrishnan and Li, Xueqing “FeFET-Based Logic-in-Memory Supporting SA-Free Write-Back and Fully Dynamic Access With Reduced Bitline Charging Activity and Recycled Bitline Charge” IEEE Transactions on Circuits and Systems I: Regular Papers , v.70 , 2023 https://doi.org/10.1109/TCSI.2023.3251961

9. Chen, Yiming and Fu, Yushen and Lee, Mingyen and George, Sumitha and Liu, Yongpan and Narayanan, Vijaykrishnan and Yang, Huazhong and Li, Xueqing “FAST: A Fully-Concurrent Access SRAM Topology for High Row-Wise Parallelism Applications Based on Dynamic Shift Operations” IEEE Transactions on Circuits and Systems II: Express Briefs , v.70 , 2023 https://doi.org/10.1109/TCSII.2022.3231589

10. Ma, Xiaoyang and Deng, Shan and Wu, Juejian and Zhao, Zijian and Lehninger, David and Ali, Tarek and Seidel, Konrad and De, Sourav and He, Xiyu and Chen, Yiming and Yang, Huazhong and Narayanan, Vijaykrishnan and Datta, Suman and Kämpfe, Thomas and Luo, Q “A 2-Transistor-2-Capacitor Ferroelectric Edge Compute-in-Memory Scheme with Disturb-Free Inference and High Endurance” IEEE Electron Device Letters , 2023 https://doi.org/10.1109/LED.2023.3274362

11. Yu, Tongguang and Xu, Yixin and Deng, Shan and Zhao, Zijian and Jao, Nicolas and Kim, You Sung and Duenkel, Stefan and Beyer, Sven and Ni, Kai and George, Sumitha and Narayanan, Vijaykrishnan “Hardware functional obfuscation with ferroelectric active interconnects” Nature Communications , v.13 , 2022 https://doi.org/10.1038/s41467-022-29795-3

12. Ramanathan, Akshay Krishna and Shahri, Sara Mahdizadeh and Xiao, Yi and Narayanan, Vijaykrishnan “Achieving Crash Consistency by Employing Persistent L1 Cache” 2022 Design, Automation & Test in Europe Conference & Exhibition (DATE) , 2022 https://doi.org/10.23919/DATE54114.2022.9774777

Funding Source:

National Science Foundation

Collaborators:

  • Nikhil Shukla – University of Virginia
  • Xueqing Li – Tsinghua University
  • Kai Ni – Notre Dame,
  • Rajiv Joshi – IBM

ASCENT: Ferroelectric-based Compute-in-Memory Dynamical Engine (Ferro-CoDE) to Solve Hard Combinatorial Optimization

vdp5074 on

Overview:

The ASCENT project aims to revolutionize computing efficiency by developing a novel hardware platform called the Ferroelectric-based Compute-in-Memory Dynamical Engine (FerroCoDE). This groundbreaking platform harnesses the inherent efficiency of physical phenomena in nature, specifically entropy maximization, to create unique synergies between physics and computation. The FerroCoDE platform will enable orders-of-magnitude improvements in computational efficiency, unlocking the potential for deploying relevant applications at a scale and in energy-constrained environments that are currently beyond the reach of present-day computers.

The ASCENT project addresses a critical systems challenge by designing, fabricating, and demonstrating a Ferroelectric Hafnium Oxide (HfO2) based Compute-in-Memory (CiM) Dynamical Engine (FerroCoDE). This innovative solution leverages the rich non-linear analog dynamics of oscillators in conjunction with the area and energy efficiency of ferroelectric CiM architecture to accelerate the computationally challenging maximum satisfiability problem. The FerroCoDE employs a novel formulation of the satisfiability problem as the direct maximization of entropy in the compute engine. This formulation is being developed through a vertically integrated materials-to-systems effort that focuses on:

  • Development of phase- and crystallographic-texture-engineered HfO2-based ultra-thin ferroelectric and antiferroelectric films with tunable properties
  • Design and fabrication of a novel non-volatile 1FeFET-1FTJ memory cell and array to enable in-memory programming and evaluation of the satisfiability clauses
  • Energy-efficient AFE oscillator arrays
  • Synergistic convergence between the hardware and algorithm
  • System engineering, with emphasis on developing learning algorithms to optimize dynamical system initialization, development of annealing schedules, and hardware scalability
  • Development and demonstration of a FerroCoDE prototype on PCB

Research outcomes:

1. Mallick, Antik and Zhao, Zijian and Bashar, Mohammad Khairul and Alam, Shamiul and Islam, Md Mazharul and Xiao, Yi and Xu, Yixin and Aziz, Ahmedullah and Narayanan, Vijaykrishnan and Ni, Kai and Shukla, Nikhil “CMOS-compatible ising machines built using bistable latches coupled through ferroelectric transistor arrays” Scientific Reports , v.13 , 2023 https://doi.org/10.1038/s41598-023-28217-8

2. Deng, Shan and Benkhelifa, Mahdi and Thomann, Simon and Faris, Zubair and Zhao, Zijian and Huang, Tzu-Jung and Xu, Yixin and Narayanan, Vijaykrishnan and Ni, Kai and Amrouch, Hussam “Compact Ferroelectric Programmable Majority Gate for Compute-in-Memory Applications” 2022 International Electron Devices Meeting (IEDM) , 2022 https://doi.org/10.1109/IEDM45625.2022.10019400

3. Xiao, Yi and Xu, Yixin and Jiang, Zhouhang and Deng, Shan and Zhao, Zijian and Mallick, Antik and Sun, Limeng and Joshi, Rajiv and Li, Xueqing and Shukla, Nikhil and Narayanan, Vijaykrishnan and Ni, Kai “On the Write Schemes and Efficiency of FeFET 1T NOR Array for Embedded Nonvolatile Memory and Beyond” IEDM , 2022 https://doi.org/10.1109/IEDM45625.2022.10019542

4. Tang, Wenjun and Lee, Mingyen and Wu, Juejian and Xu, Yixin and Yu, Yao and Liu, Yongpan and Ni, Kai and Wang, Yu and Yang, Huazhong and Narayanan, Vijaykrishnan and Li, Xueqing “FeFET-Based Logic-in-Memory Supporting SA-Free Write-Back and Fully Dynamic Access With Reduced Bitline Charging Activity and Recycled Bitline Charge” IEEE Transactions on Circuits and Systems I: Regular Papers , v.70 , 2023 https://doi.org/10.1109/TCSI.2023.3251961

5. Ma, Xiaoyang and Deng, Shan and Wu, Juejian and Zhao, Zijian and Lehninger, David and Ali, Tarek and Seidel, Konrad and De, Sourav and He, Xiyu and Chen, Yiming and Yang, Huazhong and Narayanan, Vijaykrishnan and Datta, Suman and Kämpfe, Thomas and Luo, Q “A 2-Transistor-2-Capacitor Ferroelectric Edge Compute-in-Memory Scheme with Disturb-Free Inference and High Endurance” IEEE Electron Device Letters , 2023 https://doi.org/10.1109/LED.2023.3274362

Funding Source:

National Science Foundation (October 2021 – September 2025)

Funding Collaborators:

  • Nikhil Shukla – University of Virginia
  • Mircea Stan – University of Virginia
  • Jon Ihlefeld – University of Virginia

SNN Benchmarking – Chromologic Inc

vdp5074 on

[et_pb_section admin_label=”section”]
[et_pb_row admin_label=”row”]
[et_pb_column type=”4_4″][et_pb_text admin_label=”Text”]

Research Overview:

The objective of the project is to develop a set of neuromorphic benchmarks that can be useful to demonstrate the ability for event-based inference hardware. As with conventional ML hardware, there are currently no established benchmarks for neuromorphic hardware. Also, this neuromorphic benchmark must show state-of-the-art performance for edge-relevant applications and run on currently available neuromorphic chips.

In this collaboration with Chromologic, we focus on monocular depth estimation with multi-modal data while targeting the Intel Loihi platform. Accordingly, we are currently working on building and training an event-based network and achieving comparable accuracy as available state-of-the-art works. Once we achieve reasonable accuracy, we will port the relevant part of the network to the Loihi platform and measure its performance compared to conventional GPU implementation.

Research Collaborator:

Chromologic Inc.

 [/et_pb_text][/et_pb_column]
[/et_pb_row]
[/et_pb_section]

3DFeM: CENTER FOR 3D FERROELECTRIC MICROELECTRONICS

vdp5074 on

Overview:

3DFeM is one of the 41 active Energy Frontier Research Center (EFRC) program funded by U.S. Department of Energy’s (DOE) office of Basic Energy Science (BES). 3DFeM will enable technologies exploiting the 3rd dimension in microelectronics for functions beyond interconnects for low power, non volatile 3D memory above CMOS logic. The 3DFeM vision leverages fundamental understanding of emergent ferroelectricity created via nanoscale inhomogeneities to create memory devices densely interconnected with logic to enable low-power, 3D non-von Neumann computation. Our research focuses on developing sophisticated models to accurately capture the intricate behavior of ferroelectric devices (FeFETs, FTJs, and FeRAMs) and harnessing their unique properties for circuit and system-level applications. We serve as a critical bridge between the high-level aspirations of various applications and the innovative materials and devices pioneered by our fellow researchers at the 3DFEM center. The Center embodies the power of collaboration, bringing together ten leading institutions and twenty-one esteemed principal investigators, with Dr. Vijay proudly serving as Associate Director

Read More

New Paper in ACM Multimedia 2022

mzk591 on

Our paper titled “Robust Multimodal Depth Estimation using Transformer based Generative Adversarial Networks” got accepted in ACM Multimedia 2022. In this paper, we propose a Transformer based GAN to recover dense depth from sparse depth samples and RGB data which obtain state-of-the-art results for ShapeNet and NYU-Depth-v2 datasets. In order make the model robust to various sensor asymmetry scenarios, we also propose a novel training recipe.