Sample Projects

Report & Manual Examples

Evaluation of Behavioral Health Help Line, Family Navigator and Post Adoption/Post Guardianship Services (Nebraska) [return to top]

Late in 2009 HZA was awarded an 18-month contract with the Nebraska Department of Health and Human Services to evaluate three programs newly funded by the legislature: the Behavioral Health Help Line which responds to families with children with behavioral health concerns; the Family Navigator program which provides support to families over a three-month period to access behavioral health and informal peer support services; and the Post Adoption/Post Guardianship program which provides support and services to families who have adopted children or who have become their legal guardians.

The evaluation measures the effectiveness, fidelity and outcomes of all three services using a variety of techniques. Examples are listening to the Help Line calls to determine whether the families seem to be receiving the information they are requesting; receiving extracts of the databases being operated by the agencies to determine the intensity, duration and types of services received; conducting surveys and focus groups with family members who sought help; and visiting the agencies and staff providing the services. In addition, HZA is creating a dashboard so members of the public can track the volume and nature of the services. Father Flannigan’s Boys Town and Lutheran Family Services are the lead agencies providing the services.

Quality Services Peer Review (Arkansas) [return to top]

In 2009 HZA opened its second staff unit within the Division of Children and Family Services in Arkansas whose purpose is to perform Quality Service Peer Reviews throughout the state. The new unit, with five full-time HZA employees and two full-time state employees, generally follows the federal Child and Family Services Review model to review open child protective services and foster care cases in every county of the state. At the end of the review, which encompasses 30 cases per Area, a report is generated for the Area which highlights the strengths and areas needing improvement of casework practice.

Staff of the unit also work with the caseworkers and supervisors on a more in-depth review of selected cases after the county institutes practice improvement measures. The purpose of the second review is to determine the extent of each Area’s success in addressing the issues in its program improvement plan.

The new unit represents a public-private partnership since it is staffed by both HZA and state employees. The work is complemented by HZA’s other unit, which performs data analysis, evaluation and report writing for all programs in the Division. In addition, HZA’s staff monitor the Division’s contracted services such as residential treatment care.

Evaluation of Medication Assisted Treatment for Drug Addiction (Maine) [return to top]

In 2009, HZA received a contract with the Maine Department of Health and Human Services to evaluate the impact of medication-assisted therapies such as Suboxone and methadone on the treatment of consumers of addiction services and the agencies which provide these therapies. Working for the Office of Substance Abuse which had received a two-year Advancing Recovery grant from the Robert Wood Johnson Foundation, HZA did extensive research on the practices of ten pilot agencies, the views of consumers about alternative treatment approaches, and the outcomes for consumers.

HZA conducted surveys and focus groups with individuals being treated for drug addiction at the pilot agencies. In addition, HZA reviewed Medicaid claims of individuals receiving no addiction treatment (but with emergency room claims), behavioral health services only and both behavioral health and medication assisted treatment. Consumers receiving both behavioral health and medication assisted therapies had the best results.

Co-occurring State Integration Initiative (Maine) [return to top]

In 2005, HZA was awarded the evaluation component of the five-year State Incentive Grant for Treatment of Persons with Co-occurring Substance Related and Mental Disorders. The goal of this multi-year project, funded by a grant from SAMHSA, is to institute broad and far reaching changes in the services offered to the estimated 10,000 people across Maine who experience psychiatric and alcohol or drug-related co-occurring disorders. To create a welcoming system, changes are being tested with diverse populations in twenty pilot sites and then replicated more broadly throughout the state.

HZA is conducting a rigorous evaluation of both process and outcome measures to determine whether the changes increase the percent of programs that screen, assess and treat people with co-occurring disorders and reduce the percent of clients who experience impairment following treatment. Data is being collected through a software tool developed by HZA, meetings, interviews, focus groups, document reviews, site visits, surveys and data from a variety of reporting systems. Findings from the evaluation will reveal whether interventions are consistently effective in different treatment settings serving diverse ethnic and cultural populations at various stages of the project’s implementation.

Quality Assurance to the State Central Registry (New Jersey) [return to top]

In 2005, the New Jersey Department of Human Services contracted with HZA to perform a study of the Division of Youth and Family Services’ newly-implemented State Central Registry (SCR). Implemented for the purpose of receiving and classifying reports of suspected child abuse and neglect, the SCR represents one of the reforms of the service system to protect abused and neglected children articulated in A New Beginning: The Future of Child Welfare in New Jersey. The purpose of the study was to determine whether proper determinations were being made by the SCR call takers and if the appropriate actions were taken on the calls warranting further assessment in a timely manner. HZA analyzed 600 cases to determine validity, reliability and professionalism of the SCR call takers and to identify and make recommendations for necessary changes. The findings of the study have been used by the Panel appointed by the Federal Court in a consent decree.

In a subsequent project HZA provided mentoring services to the call takers and their supervisors at the hotline. For five months, two HZA staff were deployed full-time at the hotline in Trenton. One listened to taped calls of specific workers, noting what was done correctly and what was not. The other took that information and held sessions with each worker and supervisor, re-listening to some of the calls and providing guidance on how to correct some of the problems. By the end of the project in June of 2006, there was a measurable difference in the completeness of the information gathered by the counselors and the SCR supervisors expressed their desire to have this function continue on an ongoing basis.

Workload Studies for the Office of Children Services and the Division of Public Assistance (Alaska) [return to top]

In 2005, HZA was contracted by Alaska’s Department of Health and Social Services, Office of Children’s Services (OCS) to conduct a workload study which would provide the basis for giving case workers reasonable workloads, allowing them the time they need to achieve positive outcomes for children and families. The approach was designed to define not just the time actually spent on cases, but also the time required to handle each type of case in accordance with policy guidelines and best practice standards.

Activities involved in this large and complex study included the production of:

separate estimates of the time required per month to meet practice standards for different case types, defined after review of policies and in consultation with OCS staff;
separate calculations of the maximum reasonable workload which can be handled by intake workers/child protection investigators, in-home services workers/ongoing permanency workers, rural workers with mixed caseloads, home study/licensing workers and supervisors; and
a methodology for tracking caseloads of individual workers over time, to permit OCS to maintain reasonable caseloads and equity in its case assignments.

The first phase included three deliverables: a report on specific policy requirements and geographic issues, pertinent to Alaska that impact workload; a list of case types and a task inventory for case-specific and non-case-specific activities; and workload measurement instruments. The second phase entailed implementation of the workload measurement instruments, while the third phase involved calculation of workloads in urban, rural and remote areas of Alaska and a report providing benchmarking information and recommendations concerning OCS use of case weighting formulas.

The subsequent year HZA was hired to perform a similar study for the Division of Public Assistance within the same agency. The study encompassed Medicaid, Food Stamps, Denali Kid Care and other programs. In addition to the workload study HZA and a subcontractor analyzed the work flow of the agency and made recommendations for process improvements.

Adoption Services Specialized Administrative Support Quality Assurance (Ohio) [return to top]

In 2004, HZA began a multi-year project where HZA served as Ohio’s quality assurance vendor for its adoption program. The overall goal was to help the Department of Job and Family Services to increase the number of adoptions of children in public custody by providing both information and the support to understand how the program functioned, how each initiative inter-related and what changes were needed to make it more effective. HZA served in a similar capacity for the Department between 1998 and 2001.

The work required both quantitative and qualitative methods. HZA analyzed data from the state’s information system to determine the extent to which the state achieved the federal outcomes related to adoption, which counties performed at desired levels and which needed to improve. Analyses were also done at the client level to identify which children were adopted and which were not, and which kinds of families were likely to adopt and which were not. Qualitative work was used to deepen the understanding of the data and to highlight promising practices on the part of the 88 public children services agencies. Both the quantitative and qualitative components led to recommendations as to what the state agency, the Department of Job and Family Services, and the county public children services agencies needed to do to improve their processes for finding permanent homes for children in foster care.

Examples of studies and reports produced through this contract:

Semi-annual Performance Reports. The reports, produced for the state as a whole and each of the 88 county agencies, included data and analysis of trends among children entering permanent custody, rates of success and adoption, characteristics and progress of children waiting for adoption and an assessment of the timeliness of adoptions.
Termination of Parental Rights. This study analyzed the cause of delays in the timely achievement of permanent custody. Counties were asked to provide supplemental data for a sample of cases beyond that provided in the state’s case tracking system.
Prospective Adoptive Families. Two studies about prospective adoptive families were conducted. One drew comparisons between the families waiting to adopt and the children available for adoption. The other took an in-depth look at who the prospective families were and whether or not they differ from those who actually adopted. Both studies were used to inform recruitment efforts.
Quarterly Reports. On a quarterly basis, a report was produced which measured the impact of an Ohio Administrative Code rule change.
Promising Practices. Through focus groups and interviews with county agencies, a compilation of promising practices in two areas, the federal adoption measure and Ohio’s faith-based initiatives, were assembled.

One of the key issues HZA faced during this project was the disconnect between the important initiatives Ohio had undertaken and the negative impact those initiatives had on achieving the federal outcomes. For instance, Ohio had encouraged counties to increase the number of older children being adopted, because many older children were waiting. In virtually all cases, however, these increases were followed by reductions in the percentage of children adopted within 24 months. The new federal outcome measures helped to address this issue.

Evaluation of Special Purpose Courts (Maine) [return to top]

HZA is evaluating several special purpose courts, those designed to address a particular problem such as drug abuse or child protective problems. HZA has worked in both adult and juvenile courts. One study relates to defendants with co-occurring disorders. Another relates to families in civil court proceedings for child protective custody that involves substance abuse by parents or other caregivers. The family drug court represents the coordinated efforts of judges, lawyers, treatment professionals and a variety of local, private and public sector agencies to address the complex problems associated with parental substance abuse. The model uses cooperative planning and treatment rather than punitive, adversarial approaches. Through comprehensive supervision, drug testing, integrated substance abuse treatment services and routine court appearances before a designated program judge, the goals of the Family Drug Treatment Court are to ensure the safety and welfare of the children while providing parents the opportunity to enter into treatment and learn the skills they need to become healthy, responsible caregivers.

HZA is employing a multi-method, quasi-experimental design involving a cross-site, comparative dimension. Here, child protective custody cases from two non-family drug court jurisdictions serve as comparison groups. The first component investigates the key mediating and final outcome measures of programmatic success/failure against three critical system domains: treatment, child welfare and judicial. The second component investigates cost savings both to the judiciary and to child welfare.

Voluntary Placement Program Rate Redesign Project (Washington) [return to top]

In January 2001, Washington’s Children’s Administration (CA) implemented a new foster care rate system developed through consultation with HZA. To ensure consistency within the Department, the Division of Developmental Disabilities (DDD) contracted with HZA to modify and expand upon the CA rate system to make it applicable to children receiving Voluntary Placement Program (VPP) services. Many children in VPP have extraordinary support needs and require a level of service above and beyond the basic rates of foster care or group care placement. Previously, DDD social workers relied heavily on “professional judgment” to determine reimbursement rates for needed specialized supports. Given the continued rise in the costs of VPP services, DDD wanted to implement a standardized service needs assessment for children in its care and to link the assessment results to the determination of rates.

Throughout the two-year project, HZA analyzed program costs to determine factors which affect expenditure levels, developed and pilot-tested a needs assessment instrument, conducted analyses of the instrument to develop fiscal projections, modified the existing automated rate system and trained staff in the use of the final assessment tool.

Reassessment of State Court Performance in Children in Need of Protection or Services Cases (Wisconsin) [return to top]

In 2004, HZA completed a nine-month project to reassess improvements in child welfare court procedures in Wisconsin in light of the Adoption and Safe Families Act of 1997, the final rule to implement ASFA in 2000, state legislation to implement ASFA in 2002 and CCIP reform efforts. Areas where these regulations impact court processes and which were therefore the foci of the study included: adequacy of resources; treatment of parties and witnesses; quality of legal representation; quality, depth and timeliness of hearings; training and education of involved parties; and collaborative efforts between the court, relevant agencies, tribes and the community.

HZA utilized many data collection methods. First, Wisconsin’s court rules, as defined in Wisconsin’s Children’s Code Chapter 48, were compared to federal rules and professional practice standards for compilation of a comprehensive catalog. Next, researchers conducted on-site visits to 13 counties of various sizes and geographic areas to observe court proceedings; interviews were conducted with judges, attorneys, court staff, social service agency staff and guardians ad litem; and a case file review was conducted of 800 randomly-selected cases. In addition, the researchers analyzed all Children in Need of Protection or Service cases (CHIPS) data in the Consolidated Court Automation Programs (CCAP) for two time periods, one before the changes in state law were enacted in 2002 and one after to discern whether the changes had any effect. The State of Wisconsin used the findings to secure federal funds for court improvement in its continuing effort to upgrade practices in cases affecting children in need of protection.

HZA was contracted in 2008 to evaluate Wisconsin's Court Improvement Program. HZA began by examining the strategic plans the courts have developed for each of its Court Improvement Program federal grants. HZA is developing various process and outcome measures. One evaluation technique will be to use the court's information system to determine whether children under court jurisdiction are being kept safe, are given permanency within reasonable time frames, and whether there had been changes over the time of this initiative.

Restructure Child Welfare Regulations and Policies (New Jersey) [return to top]

As part of its transformation plan, the New Jersey Department of Human Services, Division of Youth and Family Services was required to develop a new regulatory and policy framework which would implement A New Beginning: The Future of Child Welfare in New Jersey. The plan provided a new direction for DYFS which was designed to change the culture, processes and results for children and families touched by the system. In 2004, HZA was contracted by DYFS to totally revamp its regulations.

HZA began by conducting a comprehensive analysis of high-level changes necessary in either statute or regulations to support the implementation of A New Beginning. Next, a regulatory matrix of required changes was developed, as well as a process for the development of the revised regulations. Finally, HZA drafted Child Safety, Permanency, Placement, Adoption and Adolescent and Youth Transition Regulations, in addition to regulations on other system elements such as prevention, community partners, quality assurance, behavioral health, workforce development and management reporting.

The goals of the policy portion of the project were to analyze the structure of existing policy, procedures and forms manuals and suggest a revised structure; to identify where policies already determined as needing to be changed or added would fit into the manual structure; and to write policies and procedures for the manuals.

Data Analysis for the Department of Public Welfare (Pennsylvania) [return to top]

In 2002, HZA was awarded a contract by the University of Pittsburgh on behalf of the Pennsylvania Office of Children, Youth and Families to standardize data collection and reporting activities. Key components of the work involved extensive analysis of AFCARS and NCANDS data for the self-assessment of Pennsylvania’s federal Child and Family Services Review and development of data packages for each county to use in the creation of their Needs Based Budget and Plan. Throughout the contract HZA’s scope of work expanded to collect additional information from the counties, to validate the data, and to provide management training throughout the state on how to interpret the data.

Evaluation of the Strategic Prevention Framework State Incentive Grant (Maine) [return to top]

In 2005, HZA began work on a prevention initiative for Maine’s Department of Health and Human Services, Office of Substance Abuse. Over a period of four years, HZA will evaluate Maine’s Strategic Prevention Framework. The purpose of the project is to create and support a statewide prevention/health promotion infrastructure designed to reduce substance abuse; reduce risk factors and consequences with a demonstrated link to substance abuse and related problems; increase assets and protective factors with a demonstrated link to the reduction of substance abuse and related problems; and reduce health risk behaviors linked to substance abuse that are priority objectives of the Strategies for Healthy Youth workgroup partners.

The evaluation is conducted at the state, community and program levels. Major activities include the design of process and outcome evaluation instruments; collection and analyses of National Outcome Measures and other Maine data; participation in workgroups and the Executive Management Team; and collaboration with national evaluators to develop data collection tools and assist in Maine’s compliance with national cross-site evaluation requirements.

This Strategic Prevention Framework is Maine’s second State Incentive Grant. The first, One ME – Stand United for Prevention, is a multi-year grant in which HZA served as the program-level evaluator for 23 coalitions around the state. The evaluation utilized process measures such as program fidelity and outcome measures based largely on a risk and protective factor framework. HZA used various methodologies for the One ME evaluation, including: pre and post testing of youth and parent participants; surveys of coalition coordinators and members; and on-site and telephone interviews.

Quality Assurance Unit (Arkansas) [return to top]

For nearly a decade HZA has been responsible for the privatized quality assurance operation in the Arkansas Division of Children and Family Services (DCFS). HZA has seven full-time staff located inside the Division, operating largely as part of the agency. The HZA-DCFS Quality Assurance Unit is responsible for producing regularly-scheduled and ad hoc reports for both internal and external consumption and conducting special studies, including analyses of foster care caseloads, evaluations of foster care re-entry and recurrence of child maltreatment, workload analyses and estimates of the impact of welfare reform on child welfare caseloads.

For each study conducted, HZA develops a research methodology and submits it to the agency for approval. The result is a wide variety of different data collection techniques and methodologies, including:

Workload Study. HZA placed the time study forms on the agency’s intranet for workers to report case activity.
Supervisory Review. HZA created a web site to enable DCFS county supervisors to conduct their case reviews, following Division forms and protocol, on-line, which eliminated the use of paper forms, e-mailing databases and other time-consuming methods.
Evaluation of the Impact of Welfare Reform. HZA conducted a special study on the impact of welfare reform on child welfare caseloads, which necessitated creating a protocol for merging files from multiple data systems such as child welfare and income maintenance.
Quality Services Peer Review. HZA created a web site to enable the DCFS reviewers to record their rating for each item into the on-line review tool. A monthly report outlining the area’s performance for each Performance Improvement Plan item can then be automatically generated and submitted to the DCFS executive staff.
Family Preservation Services Program Evaluation. To capture all available data, HZA created a methodology in which data from both the public agency’s electronic case record and the private provider’s written case record were merged and analyzed.
Re-entry Into Foster Care Evaluation. Due to unique data collection practices by the Division, HZA developed a methodology that combined standard global quantitative data analysis with in-depth qualitative analysis of supplemental narrative information for a representative sample of study clients.
Compliance Outcome Report. Upon implementation of this report, 49 items of compliance were measured as determined by the Angela R. settlement decree. After the consent decree ended, the Division decided to re-address the measures to determine if they were both valid and useful in providing information that could drive improved performance. The result was a revised COR which focuses on 36 elements and provides the number in compliance and the percentage of compliance for both the current and previous months.

Healthy Families Program Evaluation (Maine) [return to top]

HZA has been evaluating the Home Visiting Program for first time parents since 2002.

Maine’s Home Visiting Program provides home-based education and support services to first time families and pregnant and parenting adolescents throughout the State of Maine. Managed by the Maine Department of Health and Human Services’ Center for Disease Control, the services are designed overall to help the Department achieve its goal of increasing the health status and self-sufficiency of Maine’s families. The particular focus is on new, first-time parents and their infants and young children. The program is intended to have a positive impact on child health, parent-child attachment, parent knowledge and skill in supporting childhood growth and development.

Major activities include a multi-site (13 agency) evaluation, development and implementation of an integrated database at each program site to standardize data collection and universalize program evaluation, training and technical assistance for providers in the use of evaluation methods, annual surveys of both served families and community organizations that collaborate with the agencies, and completion of reports on the progress of the evaluation.

Return to Top Home