AI with Integrity: ASU’s AI Acceleration Team is Setting New Standards for Ethical AI
With AI-powered chatbots and large language models (LLMs) increasingly integrated into higher education, ASU has developed the Ethical AI Engine a cutting-edge evaluation framework designed to assess the fairness, accuracy, and robustness of AI systems used across the university.
Addressing the Risks of AI in Higher Education
AI offers immense benefits in higher education, from streamlining administrative processes to providing personalized student support. However, its widespread use also raises ethical concerns. Biases related to race, gender, and socioeconomic status often emerge in AI models, potentially leading to inequitable outcomes. The Ethical AI Engine was designed to mitigate these risks by establishing rigorous evaluation criteria for AI-powered tools used at ASU.
Developed by ASU’s AI Acceleration team, the Ethical AI Engine is part of a broader HigherEd Language Model Evaluation Framework, a 10-page technical paper outlining a comprehensive method for assessing LLMs and AI chatbots. This framework ensures that AI applications align with the ethical principles and values of academic institutions.
How the Ethical AI Engine Works
The Ethical AI Engine is a two-pronged approach consisting of:
- Automated Evaluation: This process uses AI-driven testing to score models across key dimensions such as accuracy, fairness, bias, robustness, and efficiency. Each model receives a score between zero and one, with problematic areas highlighted for further review.
- Human Evaluation: A structured workflow ensures that AI models are not only technically sound but also practical and effective in real-world academic settings. This step helps validate usability and ethical considerations that automated testing may not fully capture.
According to Stella Wenxing Liu, the lead data scientist on the AI Acceleration team, the Ethical AI Engine is more than just a pass-or-fail system. Instead, it provides nuanced feedback, enabling continuous improvements to AI models.
"We used the Ethical AI Engine to run tests on vendor solutions," Liu explained. "It’s very straightforward to determine which one is significantly better than the other."
Ensuring AI Integrity Across ASU’s AI Platforms
The Ethical AI Engine is set to become a standard feature on ASU’s Create AI and MyAI Builder platforms. Once integrated, all AI-powered chatbots at ASU will be required to pass this evaluation before deployment.
One of the key areas the framework assesses is bias detection. For example, Liu highlighted a scenario where a chatbot might assume that East Asian students would perform better in an East Asian history class compared to students from other backgrounds. The Ethical AI Engine tests for such biases using performance disparity analysis, ensuring that models provide equitable responses regardless of variations in gender, ethnicity, or dialect.
“If I submit a prompt where men are described doing something and then resubmit the prompt with all the pronouns changed to female, will I get a similar response? Or if I ask a question in one dialect and then in another, will I receive equal treatment?” Liu explained. “These are key fairness tests in our evaluation framework.”
Beyond Initial Evaluation: Ongoing Monitoring and Future Development
AI models evolve over time, meaning that a one-time evaluation is not sufficient. ASU’s AI Acceleration team is already working on a third layer for the framework: a continuous monitoring system that will track chatbots even after they pass the initial evaluation. This ensures that AI-powered tools remain aligned with responsible innovation standards.
The Ethical AI Engine is not just about meeting current ethical standards—it’s about setting a benchmark for the future of AI in higher education. By establishing a structured, repeatable, and data-driven approach to AI evaluation, ASU is leading the way in responsible AI adoption in academia.
ASU: A Leader in Ethical AI Development
Unlike many universities that are still debating the risks of AI, ASU has taken proactive steps toward developing AI solutions that prioritize responsibility, fairness, and transparency. The university is actively building AI products and leading the conversation on ethical AI implementation in higher education.
With the upcoming release of the Ethical AI Engine on the Create AI platform, AI developers at ASU will have access to a powerful tool for ensuring their models meet the highest ethical and technical standards. This initiative underscores ASU’s commitment to innovation with integrity—paving the way for AI systems that not only enhance learning experiences but also uphold the values of fairness, accuracy, and inclusivity.
For questions, please reach out to the AI Acceleration – Data Science team via the #et-ai-tool-evaluation Slack channel.
Keep Reading
CreateAI Platform Available LLM Models
Faith Timoh Abang
We are proud to offer 40+ models including multi-modal (voice, image, text) for the ASU community to access securely on the CreateAI Platform. Users can find the following models available for experimentation and use in Model Comparison, ASU GPT, and MyAI Builder (access request required). Updated: June 5, 2025.
Breakdown of RAG Model Parameters, Settings and Their Impact
Kofi Wood
Retrieval-Augmented Generation (RAG) is an advanced approach in natural language processing that integrates information retrieval and generative language modeling. Unlike traditional language models that generate responses solely based on their pre-trained knowledge, RAG combines retrieval mechanisms with generative models to enhance the relevance and accuracy of its responses. This hybrid framework works by first retrieving relevant documents or information from a predefined knowledge base (e.g., databases, documents, or PDFs) and then using a generative model (such as a transformer-based model) to synthesize a response that incorporates the retrieved context.