The development of augmented reality applications is characterized by high variability, technical constraints, and frequent environmental change...
| Activities | January | February | March | April | May | June | July | August | September | October | November |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Study of the current project | ✔️ | ✔️ | |||||||||
| Study of available technologies | ✔️ | ✔️ | |||||||||
| Development | ✔️ | ✔️ | ✔️ | ✔️ | ✔️ | ✔️ | ✔️ | ||||
| Monography | ✔️ | ✔️ | ✔️ | ✔️ | |||||||
| Defense | ✔️ | ✔️ |
After evaluating the challenges of AR development, the project was forced to change its focus. The absence of a formal development methodology prevented the application from evolving as expected, resulting in months of effort with little progress toward viable prototypes or meaningful testing. Consequently, the initial proposal was compromised, motivating a shift in emphasis toward restructuring and improving the development process itself.
Initially, Professor Colli was already developing an AR application for the Matemateca. The idea did not originate with the authors of this work. Some implementation attempts had already been made. After a few conversations, it was verified that there was a project in Unity using Vuforia. As the main requirement is to perform tracking of the exhibited pieces, the chosen plugins satisfied this demand. For this purpose, image tracking was used with QR Codes, a technique that consists of identifying an image and defining it as a reference for positioning 3D models. However, this does not solve other problems, such as the fragility of the tracking.
Vuforia is an SDK in Unity and has some functionalities restricted to its usage plans. With it, one can implement an AR application and already model 3D objects within the tool itself. In addition, there are broad options of features to implement an AR scene. Although it enables the simplification of several development processes, the main concern regarding its use is versioning. We will still explore how to handle this if we choose to adopt this development method. However, we want to avoid it, as it deviates somewhat from the scope of classic mobile application development that we learned in the computer science course.
Given this estrangement, we ventured into other frameworks and languages. Initially, we sought an API with a native library for mobile AR development. We found two, namely ARCore and ARKit. There are limitations regarding their use, since although ARKit is more robust than ARCore, it lacks Android compatibility. Not only that, when testing development with ARCore, it was possible to observe the aversion of the Unity project owners to the complexity of developing in Android Native. Therefore, we soon changed our strategy.
Thus, we changed the approach once again. As React is widely known in the market, we created a new repository using React Native. With it, it is possible to develop mobile applications for Android and iOS. This solution expands the availability of the application and facilitates its development, since it is familiar to developers. Therefore, we continued development with it to test its limitations.
In contemporary times, interactive digital products saturate the daily lives of all global demographics. In particular, mobile applications contribute to users’ comfort, learning, and recreation. Technology even extrapolates the virtual world and merges with the material world. QR Codes, for example, assist in digital communication. One simply points the camera and the device interacts with reality, directing the user to the virtual world.
This blend has become clearer with advances in computer vision techniques. Inevitably, applications centered on this technology have emerged. Thus, we have objects from the virtual world that interact with the physical world and vice versa. At the British Museum, for example, it is possible to rent a tablet with a customized application installed on the device. With it, visitors can obtain more information about the exhibits by using the device’s camera.
There is no need to distance ourselves from the national context. Professor Giselle Beiguelman from the Faculty of Architecture and Urbanism (FAU - USP) carried out the demonumentaRA project. Analogous to the British museum, augmented reality was used to blend the physical and the virtual. However, the purpose was different. Instead of augmenting physical objects, the professor proposed bringing architectural monuments to be placed on the university campus. Thus, signs were installed at the Clock Square indicating their “existence.” To the unsuspecting passerby, nothing is observed. It was necessary for the observer to install the application developed by FAU in order to see the objects. Through the camera, the models appeared as if they were actually in the physical space.
Inserted into this context, the same technology is integrated into education. The Matemateca, as a small mathematical museum, plans to take advantage of augmented reality. The interactive pieces already serve the purpose of materializing mathematical abstractions for its non-academic visitors. However, with AR it will be possible to achieve many other advances to support this initiative. This expands design options, proposing new ways to interact and learn.
At the Institute of Mathematics and Statistics, since 2003, a collection of interactive objects has been built. Over the years, professors, students, and artisans have contributed by complementing this small museum with playful boards, geometric shapes, and demonstrations of mathematical concepts, whose mysteries—veiled in equations, philosophies, and abstractions—are unveiled before visitors. Although the project has volunteer monitors who mediate visits, their permanent presence is unfeasible. In fact, having someone available throughout the entire period in which the Matemateca is installed at the institute is not possible, since it may remain for days or even months without any forecast of removal from the lobby of Block B.
Therefore, there is a need for alternatives for mediation. With this in mind, Professor Eduardo Colli consolidated a project to integrate AR (Augmented Reality) into the pieces. Together with Lab3D, development of an application using Unity with Vuforia began. This simpler option, more accessible to people outside computer science, seemed like an initial alternative. However, the development flow with this tool cannot be sustained to become a mobile application with continuous delivery (CI/CD).
It is therefore desired to adopt more rigorous development methods. By applying computer vision concepts and adopting good practices for large- scale software development, this work aims to collaborate with the Matemateca project, contributing so that computing serves the social purpose of integration with the community external to the University of São Paulo.
Initially, the intention is to take advantage of what has already been defined previously. Thus, we will begin by studying the limitations of the technology so that it is accessible to the general public. This means that, by the end of this project, we want to make available an application that can be used on common smartphones without excluding any user due to operating system– or sensor-based functionality.
Afterwards, we want to define the scenes to be implemented. Design decisions will involve collaboration with the Lab 3D team and guidance from Professor Eduardo Colli. Although it is not the focus, we will be concerned with following good usability practices. This means that we will take care to refine any proposed idea before proceeding to implementation. Finally, it is essential to follow agile methodologies. Therefore, we will have sprints that will be documented on the TCC website.