Return to Newsletter

The Tipping Point of Child Welfare Systems:
Decision Making, Information, and Risk Assessment

Richard J. Gelles, Ph.D.
Betty Kim, MSW

In August, 2006, Danieal Kelly, a 14-year-old girl with cerebral palsy, was found dead in her mother’s apartment. Danieal died from starvation—she weighed 42 pounds when her body was discovered. Her body lay in a bed in a dark room with no fan or air conditioning. Deep-maggot infested bed sores covered her body. The bed clothes Danieal lay in were soaked with her urine.

Danieal’s death was a tragedy to be sure. But what was astounding and truly tragic was that Danieal, her mother, and her siblings were an “open case” in the Philadelphia Department of Human Services (DHS). Multiple reports of Danieal’s suspected child neglect had been filed with Philadelphia DHS. The case had been assigned to a local agency in order for the Kelly family to receive “Services to Children in Their Own Homes” (SCOH). A caseworker from the private agency was to visit the Kelly home. The caseworker was supervised by a clinician employed by the private agency. Philadelphia DHS assigned one of its own case managers to manage the case and oversee the services provided by the private agency. The DHS case manager had her own DHS supervisor. At least three different employees of the private agency and six different employees of DHS were assigned to the case over the three years DHS was involved with the Kelly family. And yet, Danieal lost 50 pounds while “visited” and provided services by the private and public child welfare agencies. She was never enrolled in school. She never saw a physician. In fact, she rarely moved outside the dark, hot room in which she died.

A Philadelphia County grand jury issued a report in July, 2008 excoriating all those who were supposed to protect Danieal. The grand jury indicted nine individuals in Danieal’s death, including her parents, two case workers from the private agency, the director of clinical services for the private agency, and two case workers employed by Philadelphia DHS. A week later, Philadelphia Mayor Michael Nutter suspended 7 DHS workers who had direct or supervisory involvement in the Kelly case.

Seven-year old Nixzmary Brown was tortured, molested, and starved by her stepfather. Nixzmary’s stepfather beat her to death on the night of January 11, 2006. Nixzmary’s New York City Administration for Children’s Services (ACS) caseworker went weeks before seeing the child and failed to update the case files in a timely or appropriate manner. The caseworker’s supervisor failed to follow through and obtain a warrant to help find Nixzmary after the girl failed to attend school for weeks and after the school reported that Nixzmary showed up at school with a gash over her eye. Although ACS caseworkers and supervisors had multiple opportunities to protect Nixzmary, they failed.

In Rhode Island, 3-year-old Thomas T.J. Wright was beaten to death by his foster other (his aunt) and her boyfriend in October, 2004. A report issued by the Rhode Island Office of the Child Advocate, stated that the state child welfare agency, The Department of Children, Youth, and Families had seven (emphasis added) opportunities to intervene and protect T.J. Unfortunately, no one in the chain of command knew enough about the case, knew about what was being done, or knew about what was not being done, to intervene at any of the seven points and act effectively to protect the child.

When a public tragedy occurs, such as the death of Danieal, Nixzmary, and T.J., the typical response of child welfare administrators is to claim “the child fell between the cracks” of the system. Having evoked the “fell through the cracks” mantra, child welfare administrators, advocates, and legislators come together and “round up the usual suspects.” The requests go out for more funding, more workers, lower caseloads, and more training. Administrators resign, are fired, or are replaced. Both New York and Rhode Island actually changed the names of their child welfare agencies after high profile tragedies. But without fail, after the usual suspects are rounded up, the new employees hired, and new training programs rolled out, children keep falling between the cracks.

What Do Child Welfare Agencies Do?

Why, forty years after child abuse was identified as a significant social problem1 do crises and tragedies still plague our systems and agencies created to protect children and support families? In our opinion, one of the core problems is the failure to understand and recognize the key and core task of child welfare systems.

A review of mission statements of the more than 300 child welfare systems in the United States reveals a general consistency—child welfare agencies exist to assure the safety and wellbeing of children, to assist families in crisis, and attempt to preserve families, and/or to assure permanence of caregiving for children. Child welfare agencies also create and deliver, either directly or by contract to private agencies, a broad array of supportive, ameliorative, and/or preventive services to assist families in dealing with crises or deficits (put another way—to help families use their strengths). When all else fails, child welfare agencies can, and are obligated to, take the initiative to sever parental rights and find permanent homes for dependent children.

One might conceptualize child welfare agencies as social service agencies, but that would be incorrect. In reality, child welfare agencies are gate-keepers and the workers decision makers.

Child welfare workers make the following key decisions:

Figure 1 illustrates, with national data, the pyramid of gates and decision points that occur once there is a report of suspected child maltreatment.

Figure 1

  • 2,600,000 investigations
  • 990,000 substantiations
  • 200,000 child removals
  • 517,000 children were in foster care on September 30, 2004
  • 65,000 children had the rights of all living parents terminated
  • Of children reported 6% are removed
  • Of investigations, 7.6% removed
  • Of substantiations, 20% removed
  • Of those children in foster care 12.5 percent had the rights of all living parents terminated

While it is true that child welfare agencies have a wide range of responsibilities and tasks, an examination of the public and private files of child welfare agencies reveals clearly that the core of all the work and the ability to meet agency goals and missions, is decision making.

How Are Decisions Made?

Clinical Judgment

If decision making is the core task, and assume for the moment it is, how are decisions made? Historically the decisions regarding the risk of child maltreatment, whether or not to substantiate or “found” a case as child maltreatment, and the decisions regarding placement, have been based on clinical judgment. Juvenile, family, or dependency courts are the final arbitrators on decisions to remove a child, return a child, or terminate parental rights, but the evidence presented to the courts is typically framed by clinical judgment.

When exhibiting clinical judgment, the Child Protective Service (CPS) worker processes the information in his or her head and then makes a decision.2 Generally, decisions are influenced by personal characteristics, biases, and experiences of the worker, which would lead to a variety of problems concerning the reliability and validity of the predicted risk.3 Research comparing clinical judgment to actuarial methods (statistical) has shown actuarial methods to be superior in terms of reliability and accuracy. Clinical judgment, due to fatigue, recent experiences, or mood fluctuations, can produce random changes in judgment while actuarial methods always leads to the same conclusion for the given information.4 When compared to statistical models, clinical judgments of experts do an inferior job of predicting behavior due to low reliability.5 Rossi, Schuerman, and Budde6 asked child welfare experts and protective services from three states to make decisions and write summaries about actual child abuse and neglect cases. The authors found that decision-making in the child protection systems is inconsistent. “Although there appeared to be some general principles used in making decisions, in the sense that certain characteristics of cases (especially prior complaint record) played roles in custody decisions, workers and experts varied widely in how each weighed those characteristics in making decisions.”7 The researchers concluded that decision-making by way of clinical judgment in the child protection agencies may have high amounts of false alarms and high frequency of high risk cases classified as low risk.8

Consensus Risk Assessment

A second form of decision making is “Consensus Risk Assessment.” In consensus - based risk assessments, specific client characteristics are identified by the consensus judgment of experts in the context of child maltreatment.9 Generally, expert judgment uses knowledge from clinical experience and research literature.10 Child welfare workers use the consensus from experts to guide decision-making about child maltreatment while exercising their own clinical judgment about the case.11 The list of predictors or characteristics of child maltreatment is based on consensus, mainly by expert judgment and accepted practice knowledge, and/or simple correlations found in research literature.12 These instruments help organize a social worker’s clinical assessment of child abuse risk, but are not based on research specific to the area that uses this instrument.13

Some of the states using Consensus Based Risk Assessments are Washington, Illinois, and California (Family Assessment Factor Analysis, the Fresno Model).

Actuarial Risk Assessment

Actuarial risk assessment models are based on empirical research on actual child protective service cases. Empirical research is used to recognize a set of risk factors with a strong statistical relationship to the specified behavioral outcome. Actuarial-based instruments integrate client characteristics shown to be statistically predictive of future child maltreatment.14 These models are generally constructed by taking a sample of children and families in the child welfare system, examining their paths while in the system, and linking those paths to a set of characteristics or events related to each family in the sample.15 The analyzed characteristics and events are weighted and combined to form an assessment tool that categorizes families or individuals according to the “risk” they may exhibit.16 Under this approach, workers use the actuarial instruments to score whether families are low, medium, or high risk.17 The goal or purpose of actuarial instruments is to have the highest number of substantiations in the category of high risk families and the lowest amount for the low risk families. Michigan, California (California Risk Assessment, not the Fresno model), Alaska, and New Jersey are some of the states currently using actuarial measures in assessing the risk of child maltreatment.
One widely used actuarial risk assessment model is the Structured Decision Making System (SDM). Structured Decision Making was developed and implemented by the National Council on Crime and Delinquency’s Children Research Center (CRC). Space precludes a detailed discussion of the SDM approach to actuarial decision making.
Although actuarial risks assessments have been shown to be an improvement over clinical judgment and consensus-based tools, the predictive validity and reliability is still modest. Gambrill and Shlonsky,18 who have compared the two risk assessments, state “although actuarial models tend to be the best predictors of future maltreatment, they are far from perfect,” (pg. 826). The Michigan’s Structured Decision Making System Family Risk Assessment of Abuse and Neglect, as one of the most researched risk assessment demonstrating superiority over other tools, still has a level of reliability lower than desired. 19

Data Mining: Neural Networks

Artificial Neural Networks (ANN), a type of data mining computing methodology, has the potential to be more reliable and efficient and to improve predictive accuracy in child maltreatment risk assessment.20 As a computer-based learning system, ANN is able to discover patterns in a set of data, especially concerning past behavior.21

Schwartz and his colleagues22 examined data gathered by the Third National Incidence Study of Child Abuse and Neglect. The researchers trained and tested 1767 child abuse cases using an artificial neural network. The study showed that the trained network was able to successfully categorize 89.6 percent of the cases in the sample population, which resulted in a 10.4 percent predictive error. Most of the predictive errors resulted from the neural network’s inability to classify the case. About 75 percent of all the errors were due to the inability to classify. Only 0.6 percent of the cases were false positives and 1.9 percent were false negatives.23 Zandi replicated this study in 2000. Zandi was successfully able to train neural networks to classify child abuse and neglect cases just like the previous study. In one of the network experiments, 90 percent of the abused cases were correctly classified. Ten percent of the cases were false negatives and 13 percent were false positives.24

Research has also compared the effectiveness of artificial neural networks to a linear or logistic multiple regression. Marshall and English 25 applied neural network analysis to child protection services data from the State of Washington’s risk assessment model. The authors concluded that the neural network demonstrated superior prediction and classification abilities over the logistic regression models. The network models classified cases equal to, but in general, more substantially superior to linear or logistic regression. This improvement can be explained by the ability of neural networks to represent nonlinear relationships between highly interacting variables, which are generally characterized by risk assessment data.26 Marshall and English state neural networks can be a useful instrument to aid the worker seeking to model complex relationships in child maltreatment risk assessment.27

Contrary to the other research studies, Flaherty and Patterson28 did not find artificial neural networks to be a superior predictor of child abuse when compared to a statistical model. The small number of actual case examples may have been a possible explanation for the inferior performance by the artificial neural network model in this study.

Can Child Welfare Workers Make Better Decisions Using Better Tools?

The decisions made by child welfare or child protective service workers directly safeguard the rights and wellbeing of children, and necessitate significant improvement in risk assessment tools. This need for improvement warrants the exploration of better means of making decisions. Clinical judgment and consensus risk assessment are simply not up to the task of being valid and reliable decision-making tools. Actuarial methods, such as Structured Decision Making, are empirically superior to clinical judgments and consensus-constructed forms. New technologies, such as artificial neural networks demonstrate the potential to achieve higher rates of validity and reliability in decision-making, and to increase the protection and well-being of children.
See page seven for footnotes.

 

Footnotes

  1. Kempe, C.H., Silverman, F.N. Steele, B.F., Droegmueller, W., & Silver, H.K. (1962). The battered child syndrome. Journal of the American Medical Association, 282, 107-112.
  2. Dawes, R., Faust, D., & Meehl, P. (1989). Clinical versus actuarial judgment. Science, New Series, 243 ,(4899), pp. 1668-1674.
  3. Gambrill, E. & Shlonksy, A. (2000). Risk assessment in context. Children’s Youth and Services Review, 22, pp. 813 – 837.
  4. Dawes et al. (1989).
  5. Gambrill & Shlonsky (2000).
  6. Rossi, P., Schuerman, J., & Budde, S. (1996). Understanding child maltreatment decisions and those who make them. Chicago, IL: University of Chicago, Chapin Hall Center for Children.
  7. Ibid pp. 595-596.
  8. Ibid
  9. Baird, C. (2002). Comparison study of the use and effectiveness of different risk assessment models in CPS decision making processes, distributed by the National Data Archive on Child Abuse and Neglect, Ithaca, NY.
  10. Knoke, D. & Trocme, N. (2005). Reviewing the evidence on risk assessment. Oxford University Press.
  11. Baird. (2002).
  12. Gambrill & Shlonsky (2000).
  13. Baird, C. & Wagner, D. (2000). The relative validity of actuarial – and consensus – based risk assessment systems. Children and Youth Review Services, 22, (11/12).
  14. Rycus, J. & Hughes, R. (2003). Issues in risk assessment in child protective services. North American Resource Center for Child Welfare. Columbus, Ohio.
  15. Gambrill & Shlonsky (2000).
  16. Rycus, J. & Hughes, R. (2003).
  17. Baird (2002).
  18. Gambrill & Schlonsky (2000).
  19. Knoke & Trocme, (2005).
  20. Flaherty, C. & Patterson, D. (2003). Predicting child physical abuse recurrence: comparison of neural network to logistic regression. Journal of Technology in Human Services, 4, 93-112.
  21. Zandi, I. (2000). Use of artificial neural network as a risk assessment tool in preventing child abuse. Available at: http://www.acasa.upenn.edu/auto.htm.
  22. Schwartz, D., Kaufman, A., & Schwartz, I. (2004). Computational intelligence techniques for risk assessment and decision support. Children and Youth Services Review, 26, pp. 1081-1095.
  23. Ibid
  24. Zandi (2000).
  25. Marhall, D. & English, D. (2000). Neural network modeling of risk assessment in child protective services. Psychological Methods, 5 (1), pp. 192-124.
  26. Ibid
  27. Ibid
  28. Flaherty and Patterson 2003