Services

Program Evaluation

HZA evaluates specific programs, services and casework tools. Data is collected through interviews, focus groups, document reviews, site visits, surveys and from a variety of management information systems.

Nearly all of the program evaluations also involve a case reading process involving anywhere from 100 to 4,000 cases with the average being 300 cases. Case readings generally answer three kinds of questions: (1) To what degree are agency practices in compliance with policy? (2) To what degree do agency practices meet national standards of good casework practice? and (3) To what degree does the agency achieve desirable outcomes for the adults, children or the families it serves? HZA generally uses cohort analyses to track outcomes, collecting longitudinal data for one to three years, either prospectively, retrospectively or both.

Examples of HZA’s program evaluations include:

Maine Co-occurring State Integration Initiative: Using multiple means of data collection, HZA is evaluating the project’s goal of instituting broad and far reaching changes in the services offered to individuals across the state of Maine who experience co-occurring disorders.

Maine Evaluation of the Pediatric Rapid Evaluation Program: Using multiple means of data collection, HZA evaluated the program’s goal of improving the medical, dental, educational and mental health status of children in Maine’s foster care system in target communities. HZA categorized three types of measures for analysis: descriptive measures, performance measures and outcome measures. It is the hope that children receiving PREP services will be able to better access medical assessment and treatment services and realize improved outcomes in terms of their safety, permanency and well-being.

Ohio Study of Validity and Reliability of the Family Risk Assessment Matrix: With a particular focus on the utilization of the Risk Assessment tool in the later stages of a child welfare case, HZA developed a new methodology which would take sufficient account of the confounding effects of the agency’s own actions based on risk assessment scores.

Maine Evaluation Services for the State Incentive Program: HZA evaluated the degree to which the project is meeting its objectives which include coordination of funding, development of a comprehensive state prevention system and a reduction of substance abuse among 12 to 17 year olds through the implementation of science-based prevention programs in communities across the state of Maine.

Maine Inventory and Analysis of Adult Mental Health Crisis Services: HZA performed a systematic analysis of the crisis services provided by eleven providers across the state of Maine and produced an inventory of crisis services.

Wisconsin Reassessment of State Court Performance in Children in Need of Protection or Services Cases: HZA evaluated the Wisconsin Children’s Court Improvement Program (CCIP) in light of federal and state legislation designed to alter court processes and improve child outcomes. HZA is currently assessing the implementation of several additional CCIP initiatives designed to improve the courts' response to child welfare cases.

Maine Evaluation of the Healthy Families Program: HZA developed and implemented a comprehensive evaluation of three home visitation models: Healthy Families, Parents as Teachers and Parents Are Teachers, Too, which provide home-based education and support services designed to increase the health status and self-sufficiency of Maine’s first-time families and pregnant and parenting adolescents. Continued evaluation of the program is still underway.

Management Reporting, Outcome and Performance Measurement [return to top]

HZA’s outcome and performance measurement has included developing outcome statements, translating those statements into indicators, mapping the indicators to specific data elements, creating formulas for calculation of the measures, collecting the data and then computing the results. All of the company’s major evaluations have included some or all of these steps. It is HZA’s position that meeting program goals is a matter of improving outcomes for the clients served by those programs.

Examples of HZA’s management reporting, outcome and performance measurement projects include:

California Desired Results for Resource and Referral and Alternative Payment Programs: HZA created a set of process standards explicitly designed to promote the achievement of Desired Results for the California Department of Education’s child care resource and referral agencies and alternative payment programs. Development of the standards permitted for the measurement of the benefits derived by the families and children served.

Pennsylvania Results Based Management Reporting System: A series of standards by which performance could be measured were developed and available management reporting sources were then evaluated to identify a series of performance measures to assess the Pennsylvania Office of Children, Youth and Families' ability to keep children safe, achieve permanency and preserve their well-being.

Arkansas Contract and Outcome Monitoring: HZA assessed the ability of Division of Children and Family Services' management information system to provide sufficient information for measuring performance and outcomes. On a quarterly and annual basis, outcomes on safety, permanency and well-being are measured.

Quality Assurance [return to top]

HZA provides quality assurance services to a number of states and their respective programs, and serves as the Quality Assurance Unit for one child welfare agency.

Quality assurance activities often begin with an evaluation of the management information systems’ capacity to measure characteristics of clients served and program performance and outcomes of those served. Alternative data sources are developed, such as family satisfaction surveys, when additional or qualitative information is sought. Reports are produced on a regular basis to inform state and local agencies, and in some instances legislative offices, of the agencies’ achievements, areas for improvement and changes in clientele.

Examples of HZA’s quality assurance services include:

Arkansas Quality Assurance Unit: The Unit, located within the Division of Children and Family Services, produces regularly-scheduled reports for both the legislature and agency administrators; tracks specific compliance measures; and conducts special studies related to foster care, workloads, minority children and welfare reform.

Ohio Adoption Services Specialized Administrative Support Quality Assurance Program: HZA produced statewide and county-specific performance reports semi-annually, to assess how quickly children in foster care achieve permanency, and conducted other special studies including production of a semi-annual Promising Practices booklet, assessment of family satisfaction, timeliness and barriers to termination of parental rights and capacity to satisfy federal requirements.

Pennsylvania Standardization of Data Collection and Reporting: For Pennsylvania’s Office of Children, Youth and Families, HZA evaluates AFCARS and NCANDS data and state-mandated reports submitted by local counties to develop a semi-annual data package for the counties to use to create their Needs Based Plan and Budget and to monitor Performance Improvement Plan initiatives.

Florida Quality Assurance Services for the Abuse Hotline: HZA analyzes data collected by local quality assurance monitors. The purpose of the data is to show where call takers are complying with state law and policies as they receive potential reports of abuse, neglect and exploitation of children and vulnerable adults. The information is used to determine needed areas of improvement in worker knowledge of screening policy and practices.

Rhode Island, Oklahoma and Nevada Quality Assurance and Data Analysis Mentoring Services: In recognition of its achievements in the areas of quality assurance and data analysis, Arkansas and HZA, as its contractor, received a federal award to serve as mentors to other states wishing to enhance their quality assurance and data analysis capacities. HZA worked with Rhode Island, Oklahoma and Nevada to make better use of those states’ SACWIS data for quality assurance and management reporting.

Workload Studies [return to top]

HZA has conducted time and workload studies for a variety of social services agencies to determine the optimum workload for each worker given the basic policy standards they are expected to meet. The methodology involves an analysis of the time required to handle each type of case in accordance with policy guidelines, coupled with a determination of how much time staff have to devote to case work, to provide agencies with a means by which they can determine if they have adequate staff and make assignments more equitably. Several states have used the methodology successfully to advocate for additional staff. In one state, foster parents gave up an increase in their own rates in favor of additional staff, given the shortages revealed by the workload study.

Examples of HZA’s workload studies include:

Alaska Workload Studies for the Division of Public Assistance and the Office of Children's Services: HZA’s approach defined not just the time spent on different types of cases, but also the time required to handle each type of case in accordance with policy guidelines and other factors unique to Alaska. At the conclusion of each workload study, HZA provided a methodology for agencies to track and maintain reasonable caseloads and equity in case assignments.

Virginia Workload Management Study: An initial study, conducted in 2000, involved the measurement of all social services programs including Medicaid, child welfare, income maintenance, and adult services across Virginia’s local departments of social services. HZA is currently conducting a follow-up study to re-examine two pieces of information from participating local DSS agencies: (1) the time workers have available for case specific work and (2) the time required to handle cases according to state and federal policy and procedure. Thirty-four localities are participating in the follow-up study. Results from both studies will be used to guide the size of the state’s allocations to localities.

Westchester County (New York) Workload Study: To assist the Child Welfare Division with a determination of need for additional staff, HZA developed a web-based application for staff to record their time, measured the time required to handle cases in accordance with policy standards and determined the time such staff have available to devote to case work.

Rate Setting [return to top]

Rate setting systems allow public agencies to determine rates for the services of private providers in a way that is both equitable and cost effective. Recognizing that clients have differing needs and individual providers have differing capacities to meet those needs, HZA often employs a needs assessment instrument to measure those differences. Based on program costs and budget allowances, HZA works with its clients to develop a formula by which provider rates may then be set.

Examples of HZA’s rate setting initiatives include:

Washington Foster Care Rate Redesign Project: HZA modified and expanded upon the rate system it previously developed for Washington’s Children’s Administration to make it applicable to the special needs children receiving Voluntary Placement Program (VPP) services. This allowed for the implementation of a standardized service needs assessment for children in care and linked the assessment results to the determination of rates.

Alabama Revenue Enhancement/Rate Setting System: HZA worked with the Alabama's Revenue Enhancement Group to develop a formula to identify and apply administrative and training costs as a revenue maximization strategy to enhance Title IV-E revenues and recovery of prior state expenditures eligible for federal funding. The scope of work was expanded to include development of a rate setting system for children in residential and group care.

New Jersey Rate Setting System Development: HZA examined contracts with legislatively authorized Regional Diagnostic and Treatment Centers, with a focus on three key areas: accessibility, service standards and funding. HZA examined the current funding pattern for each center and the ways in which the funds were used, compared to the core services required by legislation. As a result of the study, gaps and inconsistencies in service patterns across centers and in relation to the legislative requirements were identified.

Organizational and Policy Analysis

[return to top]

HZA’s evaluations of organizations and their policies focus on factors which both contribute to and detract from the agencies’ ability to achieve its mission. The evaluations often employ a triangulation methodology using data from the agencies’ management information system; interviewing agency workers, supervisors and community representatives; and conducting interviews, focus groups and/or surveys of clients. This methodology allows HZA to offer a broader set of recommendations, pointing agencies in directions they may have failed to notice.

Examples of HZA’s organizational evaluations and policy analyses include:

Erie County (Pennsylvania) Organizational Review: The review, which considered regulatory challenges and heightened family problems confronted by child welfare agencies, included a business process engineering review. HZA was asked to design and deliver a series of supervisory training sessions at the conclusion of the organizational review.

Child Welfare League of America Organizational Review: The focus of this review was to determine how CWLA could best organize itself to support its member agencies while growing the organization nationally. Site visits to regional offices, interviews with member agencies and a staffing analysis were components of this review.

New Jersey Policy Review: HZA worked with an advisory group to develop a new regulatory and policy framework as part of the implementation of "A New Beginning: The Future of CHild Welfare in New Jersey." HZA drafted changes in regulation and policy needed to bring these documents in line with New Jersey's new initiatives which were made in response to a federal law suit.

Wisconsin Online Policy Manual: HZA is assisting with the development of an online child welfare policy manual, creating a structured template for rewriting policy, organizing existing policy according to caseflow process, identifying gaps in current policy and implementing a search engine for locating policy via Wisconsin's SACWIS.

Training and Professional Development

[return to top]

Several of HZA’s engagements have involved training and professional development and have included front-line staff, managers, supervisors and administrators. HZA develops the curriculum, handouts and reference materials and delivers the training using a combination of techniques such as formal presentations, exercises, practice sessions, discussion and feedback in single or multiple sessions. HZA also evaluates the training according to what has been learned and how it is being applied on the job.

Examples of HZA’s training and professional development projects include:

Florida Abuse Hotline Training: HZA is providing a comprehensive ongoing package of training for Call Center staff and Crime Intelligence Unit staff at the Florida Abuse Hotline. HZA developed a database to track both training needs and training received by staff, is developing and administering pre- and post-tests, developing and modifying training curricula and delivering four categories of training. Feedback from the training sessions and data from the training database are used to generate reports for the client.

Erie County (Pennsylvania) Supervisory Training Sessions: HZA conducted a series of ten training sessions for the Erie County Office of Children and Youth. The program was designed in accordance with a needs assessment conducted with supervisors as well as discussions with administrators about the direction of the agency.

Nevada Strategic Planning Retreat: The goal was to establish a well-defined direction for the management and development of youth corrections under new leadership. The process involved: working with Youth Correctional Services staff to set the agenda; conducting research in advance; preparing materials; facilitating the retreat; and reporting on findings and results.

Pennsylvania PACWIS Training: HZA trained state and local child welfare staff on the use of Pennsylvania's Automated Child Welfare Information System (PACWIS) and provided assistance to the local offices with reengineering their business processes as the case management system was implemented into day-to-day practice. In another project HZA trained all of the county managers on how to interpret data for their needs based plans and budgets.

Information Systems Development

[return to top]

Several of HZA’s engagements have involved development of information systems both as data collection instruments and as final products. These systems enable the client to standardize data collection and provide a more robust set of data for analysis. Extensive research during the planning and development stage is conducted, which may include a review of policies and practice standards developed by other states and national organizations as well as existing systems and tools HZA has designed for similar purposes for other clients. HZA works closely with the client to review the system and make sure it is capturing the necessary data, making revisions as needed. HZA develops user reference materials, provides training and support to system users and often offers ongoing technical support via a toll-free help desk.

Examples of HZA’s information systems development projects include:

Maine Integrated Tracking System for the Healthy Families Program Evaluation: Utilized by 15 program sites, the program standardizes data collection, helps home visitors manage their caseload and provides data extracts to HZA for an accurate assessment of the success of each program. HZA provides training and technical assistance in the use of the database to program representatives and operates a HELP DESK to support system users. HZA is currently creating a web-based version of the tracking and management reporting system.

New Jersey Qualitative Services Review (QSR): HZA’s instrument included questions that informed New Jersey about the state of practice on both the seven systemic factors, viewed from a case specific perspective, and program outcomes. At the conclusion of the review, the application was provided to the Division for use in future QSR activities.

Pennsylvania Technical Assistance, Training and Staff Support: A portion of that effort involved the development of computer programs and automation tools which would present data in a consistent and reliable form. The tools also validated county and state data received and allowed for ease of data preparation and submission to the federal government or for use by OCYF for various other reporting purposes.

Maine Client Enrollment Tracking System for the Co-occurring State Integration Initiative: This software, installed at project sites throughout the state, provides the ability for staff to enter baseline and follow up data on clients enrolled in service delivery and evaluation; the data is routinely uploaded to HZA for analysis.

Needs Assessment and Strategic Planning

[return to top]

Several of the firm’s engagements have included a needs assessment or strategic planning component. While any needs assessment is ultimately a matter of counting units of need on the one hand and units of resources on the other, the extraordinary range of resulting combinations creates issues of complexity which have to be addressed directly, if the project is to be successful. Strategic planning involves setting well-defined goals and a framework for direction.

HZA develops a methodology tailored to the unique characteristics of each engagement and then utilizes a combination of data collection approaches such as meetings, interviews, focus groups, document reviews, site visits, surveys, management information systems analysis and case readings.

Examples of HZA’s needs assessment and strategic planning projects include:

Georgia Foster Care Needs Assessment: This project required a complex, multi-dimensional examination of both clients and the resources needed to serve children in the custody of Fulton and DeKalb counties. HZA’s methodology involved both a needs analysis and a gap analysis. HZA designed the data collection instruments, developed a database, collected and analyzed the data, conducted a literature review and developed an ongoing monitoring tool. The resulting data provided the client with estimates of current and projected need for both foster care and placement prevention services, resources for care and placement prevention services and an identification of both current and projected gaps in resources. HZA furnished software to perform ongoing monitoring of needs and service gaps.

Ohio Needs Assessment of Child Protective Services: Conducted to satisfy a law suit, the needs assessment quantified the services children and parents needed either to prevent their placement in foster care or to aid in the children’s return home. Over 60 different types of services were examined and the assessment represented the first effort to quantify what services were actually available as well as to provide accurate and quantifiable projections of unmet need.

Child and Family Services Review (CFSR) Support

[return to top]

All states in the nation are required to participate in an in-depth review by the federal government of their child welfare services program. The CFSR is one of the most significant efforts which states have undertaken and one that can carry significant future financial penalties if improvements are not made. HZA assists states through the entire CFSR process from self-assessment to implementation of the program improvement plan.

Examples of HZA’s Child and Family Services Review projects include:

New Jersey Qualitative Services Review: HZA developed a tool and process, similar to that used by the Administration for Children and Families, to determine if cases were being handled in a manner that professionals would agree constitutes quality casework. The findings provided critical information for the development of a statewide assessment. Nevada used the tool in a similar fashion and Pennsylvania has employed the tool to measure ongoing county performance improvement efforts.

New Jersey and Nevada Analysis of Data for Statewide Assessments: HZA conducted analyses of AFCARS and NCANDS data and wrote sections of the Statewide Assessments in New Jersey and Nevada preceding the federal on-site Child and Family Service Reviews.

Pennsylvania CFSR and PIP Support: HZA has provided support to Pennsylvania in preparation of both the first and second round CFSRs, writing portions of the Statewide Assessment and analyzing AFCARS and NCANDS data to measure outcome performance. HZA analyzes the results of Quality Services Reviews as part of the state's PIP, using a case record tool it helped to develop, to assess individual county performance on 23 items measuring safety, permanency and well-being.

Return to Top Home