Program Evaluation and Research

EDNA MCCONNELL CLARK FOUNDATION EVALUATION SUMMARY

Excerpt from "Evaluating The Results", Keeping Families Together and Children Safe: Facts on Intensive Family Preservation Services, October 1994.

"With two decades of experience, intensive family preservation has a solid track record. It is among the best-defined services in the field, and research shows it works on both an individual and family level, helping parents learn the skills to keep their families together safely.

Success in the social services world is difficult to measure, however, and researchers have yet to find scientific "proof" that states can successfully use intensive family preservation to lower their overall foster care rates. Part of the problem may be lack of adequately designed evaluations to test the methodology. Another source of difficulty is targeting the appropriate families to receive the service. But it is also genuinely difficult to document the impact of an intervention when each family is different and when placement decisions are made by individual caseworkers and judges. In addition, social and economic factors beyond the control of any social service can have a major impact on the numbers of children removed from their homes.

Evaluation is, therefore, a much-debated topic in the family preservation field. There are three key areas to review 1) safety of children; 2) improved family functioning; and 3) prevention of unnecessary placement and the resulting fiscal benefits of the program. On a subjective level, client satisfaction should also be included.

SAFETY
Safety of all family members is the primary focus of intensive family preservation services (IFPS). Workers do not leave children in dangerous situations, but rather help parents restructure the child's environment in order to protect him or her. If, during an intervention, a caseworker believes a child is not safe, the worker does not hesitate to recommend placement, though that is rarely necessary.

Although no social service can guarantee 100 percent safety, IFPS programs have a very strong safety record, precisely because of the characteristics of the service itself: trained caseworkers are on the scene within 24 hours of referral and begin delivering services immediately. Services take place in the home, where IFPS workers can help parents minimize the threat of violence. Workers are available daily, around the clock, to provide optimum support and protection for all of the family members. And workers have only two cases at a time, giving them the flexibility to respond to emergencies and the time to stay long enough to stabilize the household when a crisis occurs.

FAMILY FUNCTIONING
The first comprehensive evaluation of intensive family preservation services was completed in 1989, with support from the U.S. Department of Health and Human Services. It was conducted by the Social Research Institute of the University of Utah and the Washington State-based Behavioral Sciences Institute, which operates HOMEBUILDERS. The study found that in even the most difficult family situations, IFPS programs significantly increased parenting skills in dealing with a variety of economic, drug abuse, and social problems. In fact, when primary caretakers assessed problems both before and after delivery of IFPS, positive improvements were reported on 26 of 28 problems affecting family functioning. The study compared these results with those obtained from studies of more traditional service methods of handling the same problems and showed that the rate of increase in competence for families in IFPS programs was four to five times higher than for those families helped with traditional approaches. Studies in New Jersey and California (see below) also looked at family functioning and found that it had improved after families received intensive family preservation services.

COST-EFFECTIVENESS AND PLACEMENT PREVENTION
On a per case basis, intensive family preservation services cost less than foster care, and considerably less than placement in residential juvenile or psychiatric institutions. In Michigan, in 1993, it cost about $4,500 per family for IFPS, compared to $12,000 per child for family foster care. The average stay in foster care in New York State is slightly more than two years, and in 1992 the average cost per child was almost $14,000 a year; the cost of Intensive Family Preservation Services in the state is approximately $4,900. In New York City, foster care costs about $20,000 per child, for a year, and institutional or psychiatric facilities run even more. IFPS, including 10 months of follow-up care, costs about $9,000 per family.

The Targeting Conundrum
States will only save money, however, if intensive family preservation is available on a large enough scale to reduce the numbers of children being placed outside the home and if the programs reach the appropriate families. In the first decade of family preservation, most policymakers and legislators relied on qualitative and descriptive studies about the value of the service, as well as statistical evidence from across the country showing that, on average, about 80 percent of families remained together one year after the intervention was completed.

As the programs matured, IFPS advocates recognized a need for more rigorous, random-assignment evaluations. Does intensive family preservation really reach the families for whom the services are intended-that is, those families whose children would actually have been placed if intensive family preservation had not been available? If services are not reserved for these families, the case for a reduction in overall foster care numbers and an impact on social service budgets is weakened.

Several attempts to evaluate this aspect of family preservation have revealed a targeting problem in states' use of the program. Just as decisions to remove a child can be shaped by the personal criteria of individual caseworkers, decisions to refer a family to IFPS can be equally variable. If, in order to obtain the best services for their clients, caseworkers refer families with children who would not have been removed from the home, then there is no way to determine whether the services would actually have prevented placement if they had been properly targeted.

It is becoming increasingly evident that reducing states overall foster care placement rates requires more than installing a single program to prevent out-of-home care. For IFPS to be successful, administrators must reorient other parts of the system so that the system as a whole supports families and prevents unnecessary breakups. This means at a minimum: creating new fiscal incentives for use of intensive family preservation services; developing new accountability approaches; establishing clear expectations about the use of preventive programs and the need to strengthen families and ensure that stays in out-of-home care are minimized; retraining workers to keep families together and safe; and connecting IFPS to follow-up programs that build on the benefits of an intensive, short-term intervention.

California and New Jersey Evaluations
Two studies of family preservation programs, in California and New Jersey, were among the first to draw attention to the problem of targeting. Both studies showed improvement in family functioning without any reduction in safety among families receiving family preservation services. However, the California study showed strong placement avoidance in both the family preservation group and a control group receiving traditional services; about 75 percent of families in both groups were still together at the eight-month follow-up period. Had the families been properly targeted and referred, the families who did not receive family preservation services would be expected to have experienced a much higher rate of placement.

In New Jersey, where placement was defined to include any stay outside the home, even with other family members and for any length of time, a preliminary report showed significant placement avoidance for up to nine months among families receiving IFPS when compared to a control group. After nine months, the difference diminished and because of the small number of families involved, was no longer statistically significant. The final New Jersey report, released in early 1992, included an additional county and showed that family preservation did result in statistically significant placement avoidance over a one-year period, when compared to the control.

Both of these evaluations had other limitations. They used small sample sizes and were based on relatively new programs. And in California, the family preservation category grouped together HOMEBUILDERS-type IFPS programs with other, quite different, home-based programs.

Illinois Study
A more ambitious, large-scale evaluation of home-based programs in Illinois was released in 1993. This evaluation studied 61 agencies that offered a variety of approaches to help families. Placement prevention was the goal of all of the programs, and they all called themselves family preservation; the average intervention was 120 days. Only one program adhered strictly to the HOMEBUILDERS IFPS model.

Six geographic areas around the state were selected to be part of a random assignment study, comparing families receiving family preservation services with those receiving traditional child welfare services. Investigators found little difference in placement rates or reoccurrence of abuse and neglect between the two groups. But because the family preservation programs varied, it is difficult to interpret the findings. And since only 21 percent of the children receiving traditional services were placed the programs clearly missed their target group. Dissatisfied with the results or the various programs it supported under the aegis of family preservation, Illinois decided to start over in Chicago and has specified that family preservation programs in the city must use HOMEBUILDERS' intensive approach.

The Bottom Line
Intensive family preservation experts have learned a great deal from the evaluations and studies that have been done. Recognition of the need to improve the targeting process and to couple IFPS with other systemic reforms in order to make an impact on a state's overall placement rates has helped state officials with their planning. But the limitations of the studies completed to date show just how difficult it is to draw conclusions about intensive family preservation: some studies have lumped together a variety of programs under the "family preservation" rubric, and some have failed the test of using appropriate randomly assigned control groups. Interpretations are additionally complicated by small sample sizes and varying definitions of placement from state to state.

OTHER METHODS OF EVALUATION

Client Satisfaction
IFPS providers take the opinion of families seriously, and IFPS clients are routinely polled by persons other than those who work with them. Results of Washington State client surveys show that:

  • 99 percent of the clients were satisfied or very satisfied with the services they received;
  • 93 percent of the clients rated services helpful or very helpful; and
  • 97 percent would recommend HOMEBUILDERS to a family in a situation similar to their own.
Supported by:

United Way Children's Charities of America ICA CFC

The Institute for Family Development is a 501(c)(3) not-for-profit organization.

Copyright 2023, Institute for Family Development, All Rights Reserved