Technology

State comptroller: New York City agencies are using AI without guardrails

An audit by state Comptroller Tom DiNapoli found that New York City lacks “an effective AI governance framework.”

AI tools are often used to help sort through massive amounts of data, but they can have biases and flaws in accuracy – leading to calls for meticulous oversight.

AI tools are often used to help sort through massive amounts of data, but they can have biases and flaws in accuracy – leading to calls for meticulous oversight. Andriy Onufriyenko

New York City agencies are using artificial intelligence tools, but without standard rules in place for how to use them responsibly, according to a new audit by state Comptroller Tom DiNapoli.

 “The use of algorithmic tools, such as AI, to support agency decision making comes with significant risks, and the lack of a governance structure over these systems amplifies these risks,” the audit said.

AI tools are often used to help sort through massive amounts of data – something government agencies like police departments and school systems need help with. But the tools can have biases and flaws in accuracy – leading to calls for meticulous oversight.

“NYC does not have an effective AI governance framework,” the audit said, concluding that each city agency has ended up with their own methods for using AI. “These ad hoc and incomplete approaches to AI governance do not ensure that the City’s use of AI is transparent, accurate, and unbiased and avoids disparate impacts.”

Albert Fox Cahn, executive director of the Surveillance Technology Oversight Project, agreed that the city has still yet to address the threats posed by AI tools. “New York City’s AI has been discriminating against communities of color for years, putting BIPOC New Yorkers at heightened risk of wrongful arrest and the loss of parental rights,” Cahn wrote in a text.

Incomplete public reporting

City agencies are required to publicly report some of the AI they use, but tools that are “in production” or in pilot phases can evade disclosure, the audit found. 

It also identified a risk that some tools could just go unreported. The city Department of Education, for example, did not report the use of Teach to One 360, a tool that uses artificial intelligence to create personalized lesson plans for students. In another instance, the Department of Probation reported algorithmic tools to the city in 2020 that the city did not disclose in its public report. According to the audit, the city advised that those tools didn't need to be disclosed. In total, agencies reported a combined 23 tools to the city in 2020, but only 16 were included in the final public report. 

The audit from DiNapoli’s office is the most in-depth look at the city’s attempts to regulate the use of artificial intelligence – and other algorithmic tools – since former Mayor Bill de Blasio’s Executive Order 50 in 2019. The executive order created an “algorithms management and policy officer” and required the development of policies for the ethical use of algorithmic tools. In the 25 months that Executive Order 50 was in effect, the city created a reporting framework for use of algorithmic tools, but not policies governing the “fair and responsible use” of those tools or a mechanism to deal with complaints.

The algorithms management and policy officer position was discontinued by Mayor Eric Adams last year when he created a new centralized technology office. 

“While much of this audit focused on the work of the prior administration and a different government structure, this administration's recent consolidation of technology agencies and entities under the (new Office of Technology and Innovation) umbrella puts the city in a strong position to approach AI in a more centralized, coordinated way,” a City Hall spokesperson wrote in an email on Thursday.

No roadmap for oversight

DiNapoli’s audit surveyed the governance and use of artificial intelligence at four city agencies – the Administration for Children’s Services, the New York City Police Department, the Department of Education and the Department of Buildings. The audit found that none of those agencies have formal policies for tracking the intended use or outcomes of their tools. Only the Administration for Children’s Services maintained detailed records of its AI tools.

Use of AI by city contractors is a gray area. For example, while the Department of Buildings said that the agency itself doesn’t use AI tools, it allows their use by approved facade inspectors. “In response to our findings, DOB indicated it did not believe it is responsible for overseeing the use of AI by facade inspectors,” the audit read. “As a result, DOB itself does not know or maintain documentation for AI tools that facade inspectors may use.”

The NYPD, whose use of facial recognition technology has been the subject of debate and opposition from police reform advocates, reported that their facial recognition tool was evaluated by the National Institute of Standards and Technology. But the audit said that there was no evidence the NYPD had reviewed NIST’s evaluation or determined what level of accuracy the NYPD would find acceptable. The NYPD has long said that facial recognition is used to assist in investigations as a complement to human analysis.

Despite the lack of formal oversight found in the audit, the comptroller’s office said that the Administration for Children’s Services showed evidence that it takes steps to address possible bias in its AI tools, which include a predictive model to decide which of its investigations go through an extra review to check for and address shortcomings in the agency’s practices. The model prioritizes cases involving children most likely to experience future severe harm.

Though Adams discontinued the position of the algorithms management and policy officer, the City Council passed a law last year that largely continues the reporting requirements for algorithmic tools. DiNapoli’s report comes as the Adams administration talks about creating an AI strategy. City & State reported last month that the city is hiring for the newly created role of director of artificial intelligence and machine learning. The position, under the Office of Technology and Innovation, will spearhead a comprehensive AI strategy, which officials said would include the creation of governing principles. As of Thursday afternoon, that job notice was still posted on the city’s careers website.

X
This website uses cookies to enhance user experience and to analyze performance and traffic on our website. We also share information about your use of our site with our social media, advertising and analytics partners. Learn More / Do Not Sell My Personal Information
Accept Cookies
X
Cookie Preferences Cookie List

Do Not Sell My Personal Information

When you visit our website, we store cookies on your browser to collect information. The information collected might relate to you, your preferences or your device, and is mostly used to make the site work as you expect it to and to provide a more personalized web experience. However, you can choose not to allow certain types of cookies, which may impact your experience of the site and the services we are able to offer. Click on the different category headings to find out more and change our default settings according to your preference. You cannot opt-out of our First Party Strictly Necessary Cookies as they are deployed in order to ensure the proper functioning of our website (such as prompting the cookie banner and remembering your settings, to log into your account, to redirect you when you log out, etc.). For more information about the First and Third Party Cookies used please follow this link.

Allow All Cookies

Manage Consent Preferences

Strictly Necessary Cookies - Always Active

We do not allow you to opt-out of our certain cookies, as they are necessary to ensure the proper functioning of our website (such as prompting our cookie banner and remembering your privacy choices) and/or to monitor site performance. These cookies are not used in a way that constitutes a “sale” of your data under the CCPA. You can set your browser to block or alert you about these cookies, but some parts of the site will not work as intended if you do so. You can usually find these settings in the Options or Preferences menu of your browser. Visit www.allaboutcookies.org to learn more.

Sale of Personal Data, Targeting & Social Media Cookies

Under the California Consumer Privacy Act, you have the right to opt-out of the sale of your personal information to third parties. These cookies collect information for analytics and to personalize your experience with targeted ads. You may exercise your right to opt out of the sale of personal information by using this toggle switch. If you opt out we will not be able to offer you personalised ads and will not hand over your personal information to any third parties. Additionally, you may contact our legal department for further clarification about your rights as a California consumer by using this Exercise My Rights link

If you have enabled privacy controls on your browser (such as a plugin), we have to take that as a valid request to opt-out. Therefore we would not be able to track your activity through the web. This may affect our ability to personalize ads according to your preferences.

Targeting cookies may be set through our site by our advertising partners. They may be used by those companies to build a profile of your interests and show you relevant adverts on other sites. They do not store directly personal information, but are based on uniquely identifying your browser and internet device. If you do not allow these cookies, you will experience less targeted advertising.

Social media cookies are set by a range of social media services that we have added to the site to enable you to share our content with your friends and networks. They are capable of tracking your browser across other sites and building up a profile of your interests. This may impact the content and messages you see on other websites you visit. If you do not allow these cookies you may not be able to use or see these sharing tools.

If you want to opt out of all of our lead reports and lists, please submit a privacy request at our Do Not Sell page.

Save Settings
Cookie Preferences Cookie List

Cookie List

A cookie is a small piece of data (text file) that a website – when visited by a user – asks your browser to store on your device in order to remember information about you, such as your language preference or login information. Those cookies are set by us and called first-party cookies. We also use third-party cookies – which are cookies from a domain different than the domain of the website you are visiting – for our advertising and marketing efforts. More specifically, we use cookies and other tracking technologies for the following purposes:

Strictly Necessary Cookies

We do not allow you to opt-out of our certain cookies, as they are necessary to ensure the proper functioning of our website (such as prompting our cookie banner and remembering your privacy choices) and/or to monitor site performance. These cookies are not used in a way that constitutes a “sale” of your data under the CCPA. You can set your browser to block or alert you about these cookies, but some parts of the site will not work as intended if you do so. You can usually find these settings in the Options or Preferences menu of your browser. Visit www.allaboutcookies.org to learn more.

Functional Cookies

We do not allow you to opt-out of our certain cookies, as they are necessary to ensure the proper functioning of our website (such as prompting our cookie banner and remembering your privacy choices) and/or to monitor site performance. These cookies are not used in a way that constitutes a “sale” of your data under the CCPA. You can set your browser to block or alert you about these cookies, but some parts of the site will not work as intended if you do so. You can usually find these settings in the Options or Preferences menu of your browser. Visit www.allaboutcookies.org to learn more.

Performance Cookies

We do not allow you to opt-out of our certain cookies, as they are necessary to ensure the proper functioning of our website (such as prompting our cookie banner and remembering your privacy choices) and/or to monitor site performance. These cookies are not used in a way that constitutes a “sale” of your data under the CCPA. You can set your browser to block or alert you about these cookies, but some parts of the site will not work as intended if you do so. You can usually find these settings in the Options or Preferences menu of your browser. Visit www.allaboutcookies.org to learn more.

Sale of Personal Data

We also use cookies to personalize your experience on our websites, including by determining the most relevant content and advertisements to show you, and to monitor site traffic and performance, so that we may improve our websites and your experience. You may opt out of our use of such cookies (and the associated “sale” of your Personal Information) by using this toggle switch. You will still see some advertising, regardless of your selection. Because we do not track you across different devices, browsers and GEMG properties, your selection will take effect only on this browser, this device and this website.

Social Media Cookies

We also use cookies to personalize your experience on our websites, including by determining the most relevant content and advertisements to show you, and to monitor site traffic and performance, so that we may improve our websites and your experience. You may opt out of our use of such cookies (and the associated “sale” of your Personal Information) by using this toggle switch. You will still see some advertising, regardless of your selection. Because we do not track you across different devices, browsers and GEMG properties, your selection will take effect only on this browser, this device and this website.

Targeting Cookies

We also use cookies to personalize your experience on our websites, including by determining the most relevant content and advertisements to show you, and to monitor site traffic and performance, so that we may improve our websites and your experience. You may opt out of our use of such cookies (and the associated “sale” of your Personal Information) by using this toggle switch. You will still see some advertising, regardless of your selection. Because we do not track you across different devices, browsers and GEMG properties, your selection will take effect only on this browser, this device and this website.