State audit finds room for improvement in NYC’s AI governance

The capabilities of artificial intelligence have grown leaps and bounds in three years. But in the three years since state Comptroller Tom DiNapoli found “incomplete and ad hoc approaches” to AI governance in New York City agencies, progress on filling in the gaps has been comparatively modest.

In a new follow-up report to his office’s 2023 audit on AI governance in New York City agencies – such as guardrails for use and systems to assess risks – DiNapoli found the city has “made progress on AI oversight efforts” but that work remains on some important oversight and accountability measures. “The city still does not maintain a complete inventory of AI tools used across agencies, lacks consistent citywide standards and has not established a formal process for public concerns related to AI systems,” he said in a statement to City & State.

The initial audit, which spanned January 2019 to November 2022, surveyed AI governance systems at a sample of city agencies including the Administration for Children’s Services, the New York City Police Department, the Department of Education and the Department of Buildings.

Recommendations that DiNapoli issued to the city in 2023 have only been partially implemented as of this April, his office found in the new report.  The city has made progress on steps like developing definitions for AI, publishing preliminary use guidance on generative AI, convening a citywide steering committee and starting a risk-assessment process for AI tools. But one recurring concern raised in the report is the voluntary and self-reported nature of some of the city’s policies, from risk assessments conducted on a “consultative or voluntary basis” to relying on city agencies to self-report what AI tools and systems they use.

“We appreciate the state Comptroller’s partnership and focus on this important topic and are reviewing the report and its recommendations,” Ray Legendre, a spokesperson for the city’s Office of Technology and Innovation said in a statement. “This March, our annual Local Law 35 report featured a record number of algorithmic tools documented. Under the leadership of Chief Technology Officer Lisa Gelobter, we look forward to building on this foundation and advancing the responsible use of AI tools in city government.”

New York City isn’t alone here. DiNapoli released a report last year finding that New York state does not have an effective AI governance framework in place for its agencies. 

New York City has pursued multiple strategies for how to use and control AI tools, from then-Mayor Bill de Blasio’s creation of an algorithms management and policy officer and publishing of an AI Strategy document, to then-Mayor Eric Adams’ AI Action Plan. As City & State previously reported, Mayor Zohran Mamdani hasn’t brought about any kind of sea change in agencies’ use of AI, though he’s expressed some skepticism about AI.

But the city’s use of AI has been highly scrutinized – nowhere more than in New York City Public Schools, which has delayed its release of guidance on how students and teachers can use AI. Some advocates have pushed for a moratorium on generative AI use in schools until appropriate guidance can be worked out. With classes set to start on Sept. 10, that guidance has yet to be released. Mamdani told City & State last week that news on that is coming “soon.” 

Under Mamdani, the Office of Technology and Innovation hasn’t produced its own vision or new guidance for how city agencies should use or engage with AI tools, though existing uses appear to largely continue. Thanks to laws passed by the City Council last year, the administration is required to create a new Office of Algorithmic Accountability that will be tasked with analyzing algorithmic tools used by city agencies (which will include AI), evaluating their risks and creating a process for public complaints. That office was due to be created by June.

City Hall told City & State last month that the office was moving ahead and that six positions have been allocated in the budget, but no further announcements on the office’s work or plans have been made.