How ACS measures its providers' performance

Shutterstock

This story was originally published on NYN Media.

In its annual scorecard of foster care and preventive services providers, the New York City Administration for Children’s Services replaced letter grades with rankings in an effort to increase competition among nonprofits and to reflect new Commissioner David Hansell’s emphasis on greater accountability.

The provider scorecards are an assessment of the quality of services provided by the small army of nonprofit agencies who do much of the work to manage the cases of roughly 9,000 children in the city foster care system and those who receive preventive services.

Foster care agencies work to provide placements outside of the home for children living in dangerous situations, while preventive services providers work proactively with families at risk of having children removed from their households.

The newest set of calculations, which measure providers’ progress toward meeting targets and improving outcomes during the 2016 fiscal year – as well as to identify areas where improvement is needed – were released exclusively to New York Nonprofit Media. The rating system ranks 62 family foster care and more than 100 preventive services programs.

RELATED: ACS' reform agenda for keeping New York City's children safe

The most recent metrics rank programs based on how well they compared with the work of other agencies across similar program areas, or the level of risk their clients’ cases presented. So, a provider might rank in first among its peers, instead of earning an A, per the letter grades issued in previous years.

The preventive categories include risk levels ranging from low to very high and foster care categories included program areas such as therapeutic family foster care, residential and specialized family foster care.

The methodology for ranking providers draws upon outcomes and measures gathered during recurrent interactions between ACS and its providers as well as data from the Provider Agency Measurement System comprehensive case review. ACS’ oversight procedures often involve reviewing the contacts providers have with children and families, measuring program usage, making safety and risk-related updates and following up on critical incidents. Each preventive and foster care program is measured against a baseline – the “category score,” or “program systems score” respectively – which is calculated using a median of about 4,000 cases from across the provider system. Then each provider is compared against the peer group for their program.

“We think competition is a good thing,” Hansell told NYN Media. “We want agencies to know where they stand in relation to each other and that there are performers providing the same service they are that may be doing better in some areas, and so that they know that it is possible to do better. It’s their obligation to figure out how they can improve their performance.”

Along with fostering competition, ACS has increased its emphasis on improving child safety and reducing the length of time children spend in foster care. The effort is designed to increase the rate at which children find permanent homes through family reunification, adoption or guardianship with relatives, known as KinGap. ACS officials said that KinGap placements have increased 25 percent, while adoptions increased 5 percent. The percentage of youths who re-enter the foster care system has also dropped by 1.6 percentage points to 7.7 percent over the past two years.

In recent years, ACS itself has faced intense pressure to improve its operations following the high-profile deaths of children in the agency’s care, the resignation of Commissioner Gladys Carrion, probes by the city comptroller and city Department of Investigation, failures involving its Close to Home program, and concerns about high caseloads and inadequately supported staff. In addition, the state appointed an outside monitor that reports to the state Office of Children and Family Services.

Hansell, a former KPMG executive and public health administrator who was appointed in February to replace Carrion, said the new approach reflected his priority of bringing more accountability to the agency. ACS also recently launched an Accountability Office to track data for outside nonprofits, staffers, providers and members of the public.

“While the scorecards obviously have existed here for some time, from my perspective, they’re now embedded in a stronger accountability framework than we had at the agency previously,” Hansell said.

While the metrics are an important indicator of how nonprofits are performing, the agency typically doesn’t publicize them. Nonprofits see their own performance data regularly, against the typical performance of other foster care providers delivering the same type of programming, or preventive services providers serving similar types of families.

Overall, foster care agencies received a median score of 77 of 100 on measures of permanency, such as the approval of a Family Assessment and Service Plan and reaching KinGap targets. They scored an 81 both on measures of safety – factoring in indications of maltreatment, children who are unaccounted for and other measures – and well-being, such as sibling placements and medical reviews.

The provider earning the highest score in permanency, HeartShare St. Vincent’s, received 91 points, with the lowest, Coalition for Hispanic Family Services, earning 63 points. NYN Media’s document takes a snapshot of this data by listing the rankings for all of the providers in key areas across various programs. Here are ACS’ complete scorecards for foster care and preventive services providers.

RELATED: 5 things to know about New York City's 3-K For All program

Providers are rarely deficient across the board. Instead, some exceeded the standard in one or two areas, while lagging in others. In one example, foster care provider the Coalition for Hispanic Family Services was ranked second in the safety and well-being categories, but last among 22 providers in permanency.

Those earning lower scores in certain outcomes – which, for preventative agencies include repeated maltreatment, placements and safety, among other metrics – will get targeted help in those areas. Every provider is required to create plans to improve any deficiencies.

For providers whose performance continues to raise concerns, they are targeted for heightened monitoring. If the situation continues to be below the agency’s standards, a provider can have their intake closed, corrective measures can be taken or its contracts can be terminated.

In 2014, the nonprofit Community Counseling & Mediation had a contract canceled, though it still operated a preventive services contract last year. Last year, the Women’s Prison Association was placed on a corrective action plan and it ranked 17th out of 18 preventive services providers this year that are working with a very high-risk population.

Three nonprofits are currently in corrective action status – about the standard for a typical year – though ACS officials declined to name the providers.

Julie Farber, the ACS deputy commissioner of family permanency services, said that while providers were competing against each other, the agency is broadly sharing best practices uncovered along the way. “We look at bright spots and challenges, and we have a very disciplined approach to looking at the practice, and then we work to scale the bright spots and mitigate the challenges,” she said.

Stephen Levin, who chairs the New York City Council’s General Welfare Committee, said that over the past several decades, the foster care system – which had more than 16,000 children in 2007 and more than 40,000 in the late 1990s – now receives far fewer cases. He was heartened that the bulk of resources are now being invested in preventive services. More recently, ACS has allowed nonprofit workers to participate in the ACS Workforce Institute, which offers professional skill development that only used to be open to city workers.

He said the death of Zymere Perkins – a 6-year-old boy whose case was being investigated by ACS’ child protective specialists and supervisors – showed where more training could help.

“You had across-the-board system failure there, where you had CPS managers, supervisors all missing important aspects of what was going on in his life, but also there were important gaps in his preventive case and with his preventive provider. They need that type of training; they need that ongoing professional development.”

He said he considered the scorecards an important tool for the agency and he said he’d considered legislation to publicly provide the information, but some providers resisted because of concerns that they presented an incomplete picture.

Mary Jane Dessables, who is the director of information, research and accountability at the Council of Family and Child Caring Agencies, said the umbrella organization has collaborated with nonprofit organizations to provide feedback on the scorecards. “Overall, it’s a comprehensive, generally fair, good measure of what’s going on in the system,” she said.

But she cautioned that a simple ranking cannot convey all of the various and complex program elements that factor into the measurements. For example, one of the components of the final score has roughly 180 questions. There could also be anomalies due to geography or program area that affect a provider’s progress.

”Every program is very different and has individual reasons why they might end up scoring better or worse on different things,” she said. “It doesn’t always tell the full story.”

One main benefit to the scorecard is letting providers see how they are doing and how they are driving performance on programs such as KinGap. “It keeps people from working in a vacuum,” she said.

Click to see ACS’ complete scorecards for foster care and preventive services providers.

X
This website uses cookies to enhance user experience and to analyze performance and traffic on our website. We also share information about your use of our site with our social media, advertising and analytics partners. Learn More / Do Not Sell My Personal Information
Accept Cookies
X
Cookie Preferences Cookie List

Do Not Sell My Personal Information

When you visit our website, we store cookies on your browser to collect information. The information collected might relate to you, your preferences or your device, and is mostly used to make the site work as you expect it to and to provide a more personalized web experience. However, you can choose not to allow certain types of cookies, which may impact your experience of the site and the services we are able to offer. Click on the different category headings to find out more and change our default settings according to your preference. You cannot opt-out of our First Party Strictly Necessary Cookies as they are deployed in order to ensure the proper functioning of our website (such as prompting the cookie banner and remembering your settings, to log into your account, to redirect you when you log out, etc.). For more information about the First and Third Party Cookies used please follow this link.

Allow All Cookies

Manage Consent Preferences

Strictly Necessary Cookies - Always Active

We do not allow you to opt-out of our certain cookies, as they are necessary to ensure the proper functioning of our website (such as prompting our cookie banner and remembering your privacy choices) and/or to monitor site performance. These cookies are not used in a way that constitutes a “sale” of your data under the CCPA. You can set your browser to block or alert you about these cookies, but some parts of the site will not work as intended if you do so. You can usually find these settings in the Options or Preferences menu of your browser. Visit www.allaboutcookies.org to learn more.

Sale of Personal Data, Targeting & Social Media Cookies

Under the California Consumer Privacy Act, you have the right to opt-out of the sale of your personal information to third parties. These cookies collect information for analytics and to personalize your experience with targeted ads. You may exercise your right to opt out of the sale of personal information by using this toggle switch. If you opt out we will not be able to offer you personalised ads and will not hand over your personal information to any third parties. Additionally, you may contact our legal department for further clarification about your rights as a California consumer by using this Exercise My Rights link

If you have enabled privacy controls on your browser (such as a plugin), we have to take that as a valid request to opt-out. Therefore we would not be able to track your activity through the web. This may affect our ability to personalize ads according to your preferences.

Targeting cookies may be set through our site by our advertising partners. They may be used by those companies to build a profile of your interests and show you relevant adverts on other sites. They do not store directly personal information, but are based on uniquely identifying your browser and internet device. If you do not allow these cookies, you will experience less targeted advertising.

Social media cookies are set by a range of social media services that we have added to the site to enable you to share our content with your friends and networks. They are capable of tracking your browser across other sites and building up a profile of your interests. This may impact the content and messages you see on other websites you visit. If you do not allow these cookies you may not be able to use or see these sharing tools.

If you want to opt out of all of our lead reports and lists, please submit a privacy request at our Do Not Sell page.

Save Settings
Cookie Preferences Cookie List

Cookie List

A cookie is a small piece of data (text file) that a website – when visited by a user – asks your browser to store on your device in order to remember information about you, such as your language preference or login information. Those cookies are set by us and called first-party cookies. We also use third-party cookies – which are cookies from a domain different than the domain of the website you are visiting – for our advertising and marketing efforts. More specifically, we use cookies and other tracking technologies for the following purposes:

Strictly Necessary Cookies

We do not allow you to opt-out of our certain cookies, as they are necessary to ensure the proper functioning of our website (such as prompting our cookie banner and remembering your privacy choices) and/or to monitor site performance. These cookies are not used in a way that constitutes a “sale” of your data under the CCPA. You can set your browser to block or alert you about these cookies, but some parts of the site will not work as intended if you do so. You can usually find these settings in the Options or Preferences menu of your browser. Visit www.allaboutcookies.org to learn more.

Functional Cookies

We do not allow you to opt-out of our certain cookies, as they are necessary to ensure the proper functioning of our website (such as prompting our cookie banner and remembering your privacy choices) and/or to monitor site performance. These cookies are not used in a way that constitutes a “sale” of your data under the CCPA. You can set your browser to block or alert you about these cookies, but some parts of the site will not work as intended if you do so. You can usually find these settings in the Options or Preferences menu of your browser. Visit www.allaboutcookies.org to learn more.

Performance Cookies

We do not allow you to opt-out of our certain cookies, as they are necessary to ensure the proper functioning of our website (such as prompting our cookie banner and remembering your privacy choices) and/or to monitor site performance. These cookies are not used in a way that constitutes a “sale” of your data under the CCPA. You can set your browser to block or alert you about these cookies, but some parts of the site will not work as intended if you do so. You can usually find these settings in the Options or Preferences menu of your browser. Visit www.allaboutcookies.org to learn more.

Sale of Personal Data

We also use cookies to personalize your experience on our websites, including by determining the most relevant content and advertisements to show you, and to monitor site traffic and performance, so that we may improve our websites and your experience. You may opt out of our use of such cookies (and the associated “sale” of your Personal Information) by using this toggle switch. You will still see some advertising, regardless of your selection. Because we do not track you across different devices, browsers and GEMG properties, your selection will take effect only on this browser, this device and this website.

Social Media Cookies

We also use cookies to personalize your experience on our websites, including by determining the most relevant content and advertisements to show you, and to monitor site traffic and performance, so that we may improve our websites and your experience. You may opt out of our use of such cookies (and the associated “sale” of your Personal Information) by using this toggle switch. You will still see some advertising, regardless of your selection. Because we do not track you across different devices, browsers and GEMG properties, your selection will take effect only on this browser, this device and this website.

Targeting Cookies

We also use cookies to personalize your experience on our websites, including by determining the most relevant content and advertisements to show you, and to monitor site traffic and performance, so that we may improve our websites and your experience. You may opt out of our use of such cookies (and the associated “sale” of your Personal Information) by using this toggle switch. You will still see some advertising, regardless of your selection. Because we do not track you across different devices, browsers and GEMG properties, your selection will take effect only on this browser, this device and this website.