AI with Integrity: ASU’s AI Acceleration Team is Setting New Standards for Ethical AI

---

With AI-powered chatbots and large language models (LLMs) increasingly integrated into higher education, ASU has developed the Ethical AI Engine a cutting-edge evaluation framework designed to assess the fairness, accuracy, and robustness of AI systems used across the university.

Addressing the Risks of AI in Higher Education

AI offers immense benefits in higher education, from streamlining administrative processes to providing personalized student support. However, its widespread use also raises ethical concerns. Biases related to race, gender, and socioeconomic status often emerge in AI models, potentially leading to inequitable outcomes. The Ethical AI Engine was designed to mitigate these risks by establishing rigorous evaluation criteria for AI-powered tools used at ASU.

Developed by ASU’s AI Acceleration team, the Ethical AI Engine is part of a broader HigherEd Language Model Evaluation Framework, a 10-page technical paper outlining a comprehensive method for assessing LLMs and AI chatbots. This framework ensures that AI applications align with the ethical principles and values of academic institutions.

How the Ethical AI Engine Works

The Ethical AI Engine is a two-pronged approach consisting of:

• Automated Evaluation: This process uses AI-driven testing to score models across key dimensions such as accuracy, fairness, bias, robustness, and efficiency. Each model receives a score between zero and one, with problematic areas highlighted for further review.
• Human Evaluation: A structured workflow ensures that AI models are not only technically sound but also practical and effective in real-world academic settings. This step helps validate usability and ethical considerations that automated testing may not fully capture.

According to Stella Wenxing Liu, the lead data scientist on the AI Acceleration team, the Ethical AI Engine is more than just a pass-or-fail system. Instead, it provides nuanced feedback, enabling continuous improvements to AI models.

"We used the Ethical AI Engine to run tests on vendor solutions," Liu explained. "It’s very straightforward to determine which one is significantly better than the other."

Ensuring AI Integrity Across ASU’s AI Platforms

The Ethical AI Engine is set to become a standard feature on ASU’s Create AI and MyAI Builder platforms. Once integrated, all AI-powered chatbots at ASU will be required to pass this evaluation before deployment.

One of the key areas the framework assesses is bias detection. For example, Liu highlighted a scenario where a chatbot might assume that East Asian students would perform better in an East Asian history class compared to students from other backgrounds. The Ethical AI Engine tests for such biases using performance disparity analysis, ensuring that models provide equitable responses regardless of variations in gender, ethnicity, or dialect.

“If I submit a prompt where men are described doing something and then resubmit the prompt with all the pronouns changed to female, will I get a similar response? Or if I ask a question in one dialect and then in another, will I receive equal treatment?” Liu explained. “These are key fairness tests in our evaluation framework.”

Beyond Initial Evaluation: Ongoing Monitoring and Future Development

AI models evolve over time, meaning that a one-time evaluation is not sufficient. ASU’s AI Acceleration team is already working on a third layer for the framework: a continuous monitoring system that will track chatbots even after they pass the initial evaluation. This ensures that AI-powered tools remain aligned with responsible innovation standards.

The Ethical AI Engine is not just about meeting current ethical standards—it’s about setting a benchmark for the future of AI in higher education. By establishing a structured, repeatable, and data-driven approach to AI evaluation, ASU is leading the way in responsible AI adoption in academia.

ASU: A Leader in Ethical AI Development

Unlike many universities that are still debating the risks of AI, ASU has taken proactive steps toward developing AI solutions that prioritize responsibility, fairness, and transparency. The university is actively building AI products and leading the conversation on ethical AI implementation in higher education.

With the upcoming release of the Ethical AI Engine on the Create AI platform, AI developers at ASU will have access to a powerful tool for ensuring their models meet the highest ethical and technical standards. This initiative underscores ASU’s commitment to innovation with integrity—paving the way for AI systems that not only enhance learning experiences but also uphold the values of fairness, accuracy, and inclusivity.

For questions, please reach out to the AI Acceleration – Data Science team via the #et-ai-tool-evaluation Slack channel.