RELATED TOPICS
Enter Centric: Building a Custom Software Solution
NTN reached out to Centric Consulting’s Chicago team about partnering to build a custom app aligned with its goals. Using our Centric Construct approach, we built a Microsoft-Azure-hosted custom software solution and database that provided greater stability and ownership of the application long-term. This allowed NTN to more easily configure and tailor the app to its evolving business needs.
For example, NTN recognized teachers’ need to tailor students’ math assessments to meet the array of US states’ different mathematics standards. To meet that need, we designed a functionality that allows teachers to group questions according to the standards and report students’ progress at various levels, even down to individual students. Teachers can see where students are meeting or not meeting standards as well as key achievement benchmarks in between.
We also built an interface that allows NTN to add new questions and standard assessments to their inventory, giving teachers the option to search and choose questions for their own custom assessments or choose from the NTN-recommended assessments.
After seeing the additional capabilities our custom solution opened up, NTN began looking for ways to add even more value for schools and teachers. One challenge the company faced was meeting the needs of students with visual impairments or language barriers. Their next challenge was to give its MathKEYmatics software the capability to read online math assessment questions and multiple-choice answers aloud, enabling students with special needs to complete their assessments without a teacher’s aid.
The Solution: An AI Tool That Translates Math Expressions Into Read-Aloud Speech
The speech component had several specific requirements. First, it needed to use proper verbiage that a math teacher would use while not giving hints about the answers. Second, it had to work in both English and Spanish. On top of these requirements, NTN wanted the functionality to be ready and usable by the start of the next school year, which was only a few months away.
Initially, neither our team nor the client could find any third-party technology that could translate images of written math problems into spoken language or meet the other requirements. However, our architects and onshore/offshore developers pieced together their own solution by combining three different ones: Wiris MathType, OpenAI, and Microsoft Azure Cognitive Services.
First, we leveraged our previous implementation of the Wiris MathType, a mathematics UI tool that enables users to build math expressions and equations that get rendered into an HTML image format. The HTML image already contained the SVG presentation image, the MathML syntax that allows for re-editing, and the Wiris-generated accessibility description of the math stored in the ALT attribute.
Next, we needed to parse the readable math verbiage from these HTML ALT text strings for use in the read-aloud part of the solution. However, the auto-generated math verbiage was not proper for NTN’s requirement of reading the math the way a professional math teacher reads it. The public ChatGPT website was skyrocketing in popularity at the time, so our team experimented with its ability to translate the strings into a math professional’s language.
We showed the client a demo of how we could apply ChatGPT’s capabilities to their program. Impressed by its responses, they agreed to pursue integrating OpenAI. As our team did this work, we found that various OpenAI models were inconsistent or sometimes incorrect in the translated output due to the complex nature of mathematics and the varying uses of math symbols.
Our team of experts knew we could get more out of the AI models by fine-tuning OpenAI’s application program interface (API) — but we would need a lot of before and after translated samples for training the AI model. We worked closely with the client to compile large amounts of validated training data and feed it to the AI model to control its outcomes and generate more accurate and consistent responses.
Process flow diagram of how our team used Wiris Mathype, OpenAI, and Microsoft Azure Cognitive Services to work together to translate images into readable HTML text and parse that ALT text into natural, accurate math verbiage for audio playback.
NTN’s MathKEYmatics platform now uses the AI model we trained to translate complex math problems and solutions into usable text-to-speech verbiage. We used Microsoft Azure’s text-to-speech and language translation cognitive services to convert the completed content verbiage to audio in both English and Spanish with many natural human-sounding voice options to support accessibility functionalities.
Screenshot of the new text-to-speech feature in the MathKEYmatics platform with playback in several natural human-sounding voice options in English and Spanish.
Conclusion
In today’s world, schools and technology are more connected than ever, requiring businesses in the education sector to provide the most innovative digital experiences. When NTN was not satisfied with their original solution, they took success into their own hands by working with our team to build a custom version of the software.
Once our onshore and offshore software developers delivered a custom app that NTN could tailor to their needs, they teamed up with us to take the program to the next level with new features. Our team dove into the world of generative AI to give the application the power to read math problems and answers aloud to students.
In a time when user-friendly learning technology is more important than ever, we navigated new tools and technical hurdles to help NTN make its mathematics software more accessible for all students.