NIST AI RMF key to U.S. AI leadership.
NIST 19 — National Institute of Standards and Technology, an agency within the U.S. Department of Commerce, removes a major challenge to the U.S. industrial competitiveness at the time, 2019 (“U.S. Leadership in AI: A Plan for Federal Engagement in Developing Technical Standards and Related Tools,” National Institute of Standards and Technology, August 9th, Available Online at https://www.nist.gov/system/files/documents/2019/08/10/ai_standards_fedengagement_plan_9aug2019.pdf, Accessed 07-20-2022)
Emphasizing the importance of artificial intelligence (AI) to the future of the U.S. economy and national security, on February 11, 2019, the President issued an Executive Order (EO 13859) directing Federal agencies to ensure that the nation maintains its leadership position in AI. Among its objectives, the EO aims to “Ensure that technical standards…reflect Federal priorities for innovation, public trust, and public confidence in systems that use AI technologies…and develop international standards to promote and protect those priorities.”
United States global leadership in AI depends upon the Federal government playing an active and purpose-driven role in AI standards development. That includes AI standards-related efforts needed by agencies to fulfill their missions by:
• supporting and conducting AI research and development,
• actively engaging in AI standards development,
• procuring and deploying standards-based products and services, and
• developing and implementing supportive policies, including regulatory policies where needed.
The EO directs the Secretary of Commerce, through the National Institute of Standards and Technology (NIST), to issue “a plan for Federal engagement in the development of technical standards and related tools in support of reliable, robust, and trustworthy systems that use AI technologies.” This plan was prepared with broad public and private sector input.
There are a number of cross-sector (horizontal) and sector-specific (vertical) AI standards available now and many others are being developed by numerous standards developing organizations (SDOs). Some areas, such as communications, have well-established and regularly maintained standards in widespread use, often originally developed for other technologies. Other aspects, such as trustworthiness, are only now being considered.
This plan identifies the following nine areas of focus for AI standards:
• Concepts and terminology
• Data and knowledge
• Human interactions
• Metrics
• Networking
• Performance testing and reporting methodology
• Safety
• Risk management
• Trustworthiness
Trustworthiness standards include guidance and requirements for accuracy, explainability, resiliency, safety, reliability, objectivity, and security.
It is important for those participating in AI standards development to be aware of, and to act consistently with, U.S. government policies and principles, including those that address societal and ethical issues, governance, and privacy. While there is broad agreement that these issues must factor into AI standards, it is not clear how that should be done and whether there is yet sufficient scientific and technical basis to develop those standards provisions.
Standards should be complemented by related tools to advance the development and adoption of effective, reliable, robust, and trustworthy AI technologies. These tools—which often have overlapping applications—include, but are not limited to:
• Data sets in standardized formats, including metadata for training, validation and testing of AI systems
• Tools for capturing and representing knowledge, and reasoning in AI systems
• Fully documented use cases that provide a range of data and information about specific applications of AI technologies and any standards or best practice guides used in making decisions about deployment of these applications
• Testing methodologies to validate and evaluate AI technologies’ performance
• Metrics to quantifiably measure and characterize AI technologies
• Benchmarks, evaluations, and challenge problems to drive innovation
• AI testbeds
• Tools for accountability and auditing
U.S. government agencies should prioritize involvement in AI standards efforts that are: inclusive and accessible, open and transparent, consensus-based, globally relevant, and non-discriminatory. This plan provides guidance regarding important characteristics of standards to help agencies in their decision making about AI standards. It also groups potential agency involvement into four categories ranked from least- to most-engaged: monitoring, participation, influencing, and leading. The plan provides a series of practical steps for agencies to take as they decide about engaging in AI standards.