AI-Powered Augmented Reality for Advancing Human-Computer Interface in Assembly Tasks
In this I[X] Team Spotlight, take a look at the team behind the research. Dr. Wei Yan will share why this project is important to the community and how his team of students will contribute to the project. Discover what compelled the students to select this particular Innovation[X] project to work on, and what valuable skill sets and unique perspectives they bring to the project.
Dr. Yan, as the project leader, tell us why this project is so important to the community?
“Major assembly and construction projects are increasingly complex. Automation and robotics are replacing many manual, repetitive, and standard tasks. Still, more than 95% of jobs consist of activities that need human labors. Assembly tasks may take advantages of the Augmented Reality (AR) technology. AR superimposes digital images on the real-world view of human users. In the meantime, Artificial Intelligence (AI) has the potential to significantly empower AR. The project aims to research and develop innovative AI-powered AR methods for assembly tasks. Empowered by AI, AR is expected to significantly enhance human’s ability to understand their living and working environments, make informed decisions, and accomplish complex tasks of making and learning. The project will utilize our prior working prototype – BRICKxAR: a novel AR-based instruction method that guides real LEGO® brick construction with virtual bricks, step by step. In a demonstration, LEGO® Arc de Triomphe is built completely with BRICKxAR, without the instruction booklet.”
What do you think your team of students can bring to this project?
“Our graduate and undergraduate students from the Colleges of Architecture and Engineering can greatly support this research. They are assisting in the development of a prototype of AI-powered AR, and will demonstrate it through a case study with LEGO® Architecture assembly. Specifically, they will contribute to the development and testing of Computer Vision and Deep Learning methods for (1) AR model registration, i.e. the alignment between virtual and physical models, while the virtual models guide the physical assembly; and (2) assembly part recognition, which will enable efficient part finding, detection of the completion of assembly steps, and detection of errors. The project team consists of: Wei Yan, Dezhen Song, Manish Dixit, Zohreh Shaghaghian, Shu-Hao Yeh, Zaal Buhariwalla, Cheryl Cruz, Vatsal Mistry, Jesus Sarmiento, Tzu Ting Tseng, Sai Juttu, Karim Zaher, Luis De La Fuente, Jared DeLeon, Cassidy Frier, Alexander Moree, Grace Radhakrishnan, and Radhika Soni.”
As students, what made you chose this particular Innovation[X] project?
“Both Augmented Reality and Artificial Intelligence have always been interesting topics to me. After I graduate, I plan on going into the AI-Robotics field, but I have also seen the potential that AR could bring to modern society. When I heard about this project, I just knew that I had to join!”
– Karim Zaher, Computer Engineering Major
“As an engineering student, I wish to use my education, skills, and experience to continue developing products and solutions to help solve real-world problems and make a difference to real people, situations, and communities. In this project, we will learn prototyping as a research method, computer programming for basic AR, synthesize training data, set up deep learning model parameters, run the training process, and evaluate the outcomes.”
– Jared DeLeon, Computer Engineering Major
“I read carefully about the description and it looks very interesting. I am interested in AR, robotics and AI, and this project includes all. I definitely would love to learn more about them.”
– Tzu Ting Tseng, Computer Engineering Major
“It is a very interesting project that I believe will help me grow my skills and become better acquainted with coding and management.”
– Grace Radhakrishnan, Computer Engineering Major
“I wanted to expand my knowledge of applied machine learning and convolutional neural networks.”
– Vatsal Mistry, Computer Science Major
What part of the project will you be working on?
“As one of the PhD research leaders, I am responsible for one of our sub-groups working on the AI intervention for the AR app including hand detection and Piece detection (Brick Finder).”
– Zohreh Shaghaghian, Architecture Graduate Student
“I am responsible for hand recognition using CNNs at the moment and researching into optimal methods of achieving a high accuracy with fast performance.”
– Vatsal Mistry, Computer Science Major
“Research and develop innovative Artificial Intelligence-Powered Augmented Reality for advancing Human-Computer Interface. The anticipated outcomes from this project include AR and AI apps, demonstration videos, a project website, and publications about research findings.”
– Jared DeLeon, Computer Engineering Major
“LEGO® piece detection: use the documentation and run the current method. Improve the training model. Improve the training dataset.”
– Tzu Ting Tseng, Computer Engineering Major
“Machine Learning optimization and implementation. The tasks are to make it work to the best of its ability and the ability of the software and hardware we have.”
– Alexander Moree, Computer Science Major
“My task is to make the version control operable so I will be helping other members work together more efficiently.”
– Jesus Sarmiento, Computer Science Major
“I’m working on improving and designing AR marker registration.”
– Shu-Hao Yeh, Computer Engineering Graduate Student
