How can we record and share the built environment using projection, mapping, and digital tools, without rendering it static, while preserving heritage, documenting changes, and maintaining its dynamic histories?
By Jacob Mwesigwa
Jacob Mwesigwa is on a journey to safeguard what matters most by transforming 360 imaging into enduring 3D archives. He extends the encounter with the St. Francis Chapel murals beyond the physical space to the Goethe-Zentrum Kampala basement during the Musealised publication launch, through projection, mapping, and digital realities projected on the ceiling. Catch the moments captured below by Josh Dalgoo (courtesy of GZK).
Jacob Mwesigwa
Jacob is a passionate creative technologist dedicated to enhancing work and education through Real-Time 3D technology. He specialises in immersive technologies, using Extended Reality (XR), including Virtual Reality (VR), Augmented Reality (AR), and Artificial Intelligence (AI) to create innovative applications. Jacob’s work aims to improve training and learning outcomes for students, facilitate exploration for professionals, and empower storytelling for artists.
Digital World Reconstruction
3D Gaussian Splatting (3DGS): is an advanced AI-driven 3D reconstruction technique. It translates 2D photos into photorealistic, navigable 3D environments by representing scenes with millions of tiny 3D shapes.
Unlike traditional mesh-based photogrammetry, 3DGS bypasses the need for hard surfaces and textures whilst still using Input from a set of overlapping photos. Reconstructing them into sparse point clouds rendered by assigning colour, opacity, and shape (covariance) to thousands of 3D Gaussian "points" to seamlessly render a photorealistic environment.
Here are my initial tests before choosing which method to use, either Photogrammetry or 3DGS (obviously chose the later).
With the base methodology done, I embarked on the Chapel, a bigger scale that would present a bigger challenge ( Of course I accepted the challenge 😀). I scanned the St. Francis Chapel and turned it into 3D data via Polycam and used the images to train a 3DGS with the advent of Machine Learning techniques. Making it possible to render these scans interactively on consumer devices via Supersplat.
I then created an online experience hosted on Viverse where it can be experienced via their awesome cross platform service on any device (VR, mobile and Desktop).
The Future
Scan technology has revolutionised the field of built conservation and documentation playing a vital role in the preservation and restoration of heritage sites. The purpose of my research is to develop a broad approach for the application of current and new technologies in creating knowledge banks for our built environment as prototyped with St. Francis Chapel.
As a preliminary outcome, a tool is suggested for the potential of virtual restoration, where professionals can virtually reconstruct and perceive a cultural site in its former splendor including both graphical and non-graphical information. This will allow for the exploration of alternative restoration approaches and empower decision-makers to make informed choices.
Projection Mapping
For the showcase of our research work from the workshop series, I created a projection mapping experience to connect the audience to the St. Francis Chapel in an ethereal way.
Projection Mapping uses everyday video projectors, but instead of projecting on a flat screen (e.g. to display a PowerPoint), light is mapped onto any surface, turning common objects of any 3D shape into interactive displays. More formally, projection mapping is "the display of an image on a non-flat or non-white surface."
For this experience I created animated sequences using the scan data obtained, applying various image effects that warp and distort the murals all the while bringing them to life and offering the audience a chance to intimately translate their meaning as the artist intended.
Special thanks to Martha Atukunda, my teammate from Wal-e Visual who helped with the music. Sadly I was too engrossed in VJing the crowd's ohhs and ahhs to capture videos of the audience myself but that is where team work comes in, thanks for Franklin, Rebecca, and Lorna for sharing their POV recording on the GZK basement ceiling (OMG people actually laid down to watch our work!).
The first scan of the chapel.
Tap/click the play button to move through a digital twin of the chapel.
