Artificial Intelligence Performance Evaluation Tool

The proposed research effort aims to address critical gaps in the evaluation and validation of Artificial Intelligence (AI) systems within the Air Force by developing an Artificial Intelligence Performance Evaluation Tool. The tool will provide a standardized, adaptive, and robust framework capable of assessing the performance, reliability, and explainability of AI models, with a targeted application in sensor and information systems used in test and evaluation missions under the 412th Test Wing (412TW) of the Air Force Test Center (AFTC). This effort aligns with the Department of Defense's (DoD) thrust toward advancing Trusted AI and Autonomy and operationally focused Advanced Battle Management System (ABMS) initiatives. Problem/Unmet Need: Current AI evaluation processes lack standardization and reproducibility across varying operational environments, making it difficult for the AFTC to confidently assess AI models’ trustworthiness, reliability, and operational readiness. Furthermore, the escalation in complexity of AI-enabled systems has underscored the need for technologies that can quantitatively measure AI performance while adhering to DoD Responsible AI principles. These challenges prevent the Air Force from leveraging AI advancements at scale, impacting readiness and modernization goals. Opportunity/Desired Outcome: By creating a modular AI evaluation tool, this effort provides an opportunity to establish a scalable and adaptable solution that aligns with emerging DoD AI standards. The desired outcome is a measurable enhancement in the fidelity, speed, and consistency of AI performance assessments, reducing decision-making cycles and increasing mission readiness. The tool will enable practical implementation of AI validation in secure environments, ensuring compliance with applicable standards and operational conditions. Successful deployment will support the broader modernization priorities tied to mission-critical AI deployment and advanced autonomy. Approach The project will begin in Phase I with baseline research and feasibility studies, advancing into a scalable prototype in Phase II for integration and testing in operationally relevant environments. A summary of planned efforts by phase follows: Phase I: Feasibility and Concept Development (Initial TRL: 2; Target TRL: 4) Research: Conduct an initial landscape analysis of AI performance metrics, methodologies, and tools with a focus on gaps and challenges specific to AFTC’s mission space. Identify and assess algorithms for measuring real-time inference latency, adversarial resilience, and data drift specifically for different sensor models. This research must establish and utilize open-source or synthetically generated surrogate sensor datasets that are representative of AFTC flight test scenarios. Framework Design: Develop a software architecture for an adaptable AI performance evaluation framework designed to address reliability, robustness, and explainability metrics. Awardees will need to base their design on widely accepted DoD digital engineering and telemetry standards. The architecture must explicitly define the assumed data ingestion pipelines for surrogate sensor data and detail the computational methods that will be used to calculate reliability, robustness, and explainability metric. Simulation: Perform initial simulation tests to explore early-stage integration of explainability metrics and KPI identification methodologies. This will consist of running dry-runs within a localized software sandbox. The objective of these tests is to demonstrate that the proposed mathematical models can successfully process these inputs, identify Key Performance Indicators (KPIs) such as data drift, and output quantifiable explainability scores. Deliverables: A feasibility report describing how the tool can practically operate within the AFTC environment, a detailed framework design document, and initial simulation results demonstrating concept validity. This could include Surrogate Data Strategy and Baselining, algorithmic and mathematical validation, architectural alignment and computational overhead, and Phase II scalability and transition pathway. Phase II: Prototype Development and Validation (Initial TRL: 4; Target TRL: 6) Prototype Development: Build a functional prototype of the performance evaluation tool with a secure Python-based backend and a user-friendly Graphical User Interface (GUI). Integration: Test and integrate the tool with existing sensor and information systems in a controlled 412TW test environment to assess real-world performance. AI Metrics Library: Develop an expandable library of explainability metrics aligned with DoD Trusted AI standards to support diverse mission applications. Validation: Conduct comprehensive testing in operationally relevant scenarios to ensure reliability, scalability, and compliance with DoD standards. Deliverables: A fully functional prototype, test reports validating system performance, and a roadmap for field-level deployment. Conclusion This effort supports the Air Force’s operational imperatives by advancing capabilities for trustworthy AI deployment in mission-critical applications. Upon completion of Phase II, the AI Performance Evaluation Tool is expected to reach TRL 6, with a clearly defined transition pathway toward operational implementation across the 412TW and other DoD components. This innovation is anticipated to significantly enhance the Air Force's ability to validate AI systems, improve decision-making confidence, and maintain technological superiority in contested environments.