IN PROGRESS

Following are descriptions of the organizations with whom DRA is presently working, the approaches DRA is taking with each of the contracts, and the expected outcomes of the evaluation process.

California Health Care Foundation

DRA has been retained to complete a three year evaluation of CHCF's new major initiative "Expanding the Availability of Diabetic Retinopathy Screening." This evaluation follows on DRA's ground-breaking work in documenting the costs and benefits of tele-based systems for reducing the incidence of diabetic retinopathy. The evaluation focuses on five major aspects of CHCF's Initiative: 1.How successful is the program in affecting the identified health indicators and outcomes; 2. Is the diabetic retinopathy screening program cost-effective; 3. What is the impact of diabetic retinopathy screening on operational processes, efficiencies, and quality of care, and to what extent can such a screening program be self-sustaining; 4. What impact, if any, does diabetic retinopathy screening have on primary care provider and specialist satisfaction with this technology; and 5. What challenges are faced by federally qualified health centers in seeking reimbursement for program services based on current regulations and what progress has been made in developing reimbursement mechanisms? The evaluation will be completed by December 2010.

Employers Direct Insurance Company

DRA has been retained to assist Employers Direct Insurance Company (EDIC) in designing, refining, conducting and analyzing the responses of their Customer-Driven Quality Improvement Initiative Survey. EDIC is California’s only direct provider of workers compensation insurance, and provides coverage without the use of agents or middlemen. The purpose of this Initiative is for EDIC customers to play an active role in setting EDIC’s strategic priorities, using the Net Promoter^® model. The survey will be distributed to EDIC's entire customer base, using both electronic and traditional paper questionnaires.

DRA staff are providing a variety of services, including facilitation of team meetings during the design and refinement phases; production of iterations of the online survey format and content; testing of the survey protocol; ongoing technical support to EDIC staff and customers with respect to the survey protocol; assistance in the analysis of survey data; development of tailored follow-up surveys for customer groups; and production of survey findings and reports.

Prison Industry Authority, State of California

DRA is conducting a study of the effectiveness and associated costs of the programs administered by the California Prison Industry Authority (PIA). Specifically, the study has been designed to:

• identify a range of outcomes which best represent the impact of PIA’s work with California prison inmates, such as reductions in recidivism and/or prison violence;

• identify costs associated with administering/conducting PIA’s programs;

• access existing and/or develop new data and databases to capture outcomes and costs;

• conduct statistical analyses to isolate those factors that best account for outcomes related to PIA’s work with California prison inmates, e.g., inmate demographics; inmate criminal histories; PIA occupational training; job placement characteristics; etc.;

• report on significant and/or provocative relationships between outcome factors, outcomes and costs;

• institutionalize a process within PIA to continuously assess and report on outcomes and costs associated with its programs.

The study is utilizing a variety of data and analytical techniques to assess the cost effectiveness of PIA’s work with California prison inmates. The study has been structured as a matched cohort design with respect to recidivism and behavior change for PIA workers, with an emphasis on prison violence reduction. The study seeks to identify factors which may account for the relative success or failure of post-incarceration employment and re-integration into local communities. A report on the findings with respect to these factors will be published, with the primary intended audience being PIA’s clients, California’s Youth and Adult Corrections Agency, as well as State control agencies, including the California State Senate, the State Department of General Services, Bureau of State Audits and State’s Legislative Analysts Office.

Sonoma County HIV/AIDS Prevention Education Project

Over the last few years, the Sonoma County Department of Health Services has facilitated a community-based effort to design and develop a comprehensive approach to countywide HIV education and prevention. The result is a 3-year Strategic Plan that aims to guide local funding and intervention strategies geared toward educating community members, organizations/institutes and policy-makers on: HIV transmission, risk reduction strategies, and local resources for the prevention of HIV and the support of those who are infected. The County, along with the local Prevention and Planning Group, has engaged DRA to complete two tasks: 1) to assist in the development of evaluation plans for grantee programs, and 2) to conduct assessments of the ongoing strategic planning and program implementation process at both the County and local program levels.

As always, building the evaluation capacity of local stakeholders is a key to DRA’s role. Understanding the value of partnerships, the power of community engagement, and the importance of good information with respect to building and sustaining essential programs and activities is therefore a focus of this effort. In addition, DRA continues to assist the HIV Prevention and Education Program in evaluating the extent to which the local projects meet the goals specified in the County Strategic Plan. Likewise, DRA is assisting in assessing and documenting the extent to which the County as a whole meets the mandates of the State Office of HIV/AIDS. Tailored approaches to evaluation will continue to be central to DRA’s work with each of the individual grantees and the County operated programs. At the same time, process, outcome and impact commonalities among the full range of projects and strategies being implemented will be explored.

Sutter County Human Services Department-Health Division

Sutter County Human Services Department-Health Division,  through funding from the California Tobacco Tax and Health Promotion Act of 1988 (Proposition 99), provides health education and health services aimed at the prevention and reduction of tobacco related illness. 

In addition to working with staff to insure that State Tobacco Control Section requirements are met, DRA has teamed with Sutter County Health Division staff to: conduct local assessments and community surveys; facilitate community based strategic planning; and develop program evaluation plans and activities that support local needs and interests. 

A key facet of DRA's work with Sutter County is the aim to strengthen understanding and build appreciation for the value of information development to program planning and refinement as well as achievement of desired outcomes.  To this end,  DRA has worked with staff, Youth Tobacco Coalition members, Adult Tobacco Coalition members and other local stakeholders in order to provide orientation and/or training in basic evaluation techniques and evaluation activities in support of their current and likely future program priorities, goals and objectives.

RECENTLY COMPLETED 

PRIDE Industries

DRA was retained to lead the planning for the revision of PRIDE’s corporate information development system, including a plan for evaluating the cost effectiveness of its community re-entry programs. This planning process focused primarily on PRIDE's community re-entry and criminal justice programs, but also explored innovative uses of data to highlight the outcomes and impacts of its traditional rehabilitative programs. The consult used a team approach to build information systems that serve various program stakeholders' needs, with special attention to providing a foundation for cost-benefit analyses.

DRA staff provided both general and highly specific services, including team facilitation; review of current data systems; recommendations for integration of existing data systems; designing of both standard and ad hoc reports; and ensuring that future information development systems are guided by both current and likely future information stakeholders' interests and needs.

California State University, Sacramento – California State Department of Alcohol and Drug Programs

DRA’s staff were hired to assist in the design, coordination, technical support and evaluation of Alcohol and Drug Programs’ Statewide Methamphetamine Initiative. This Initiative seeks a variety of outcomes, including effective collaboration among State agencies whose missions are affected by methamphetamine production and use; effective coordination of resources in treating methamphetamine as both a law enforcement and public health issue; public education and information dissemination; and the building of local organizations’ capacities to deal more effectively with methamphetamine supply and demand.

DRA staff provided a full range of services, from research on other states’ approaches to evaluation of the Initiative’s outcomes and impacts. DRA was an active partner in designing the Initiative, developing the Initiative’s advisory Task Force, facilitating the Task Force’s work, developing targeted approaches to at-risk populations and working with private non-profit organizations and other local agencies to forge a unified approach to reducing the demand for and supply of methamphetamine within California. Using a stage-dependent approach, the coordination of the State’s efforts will gradually be expanded to include the collaboration of State, local and private, non-profit efforts, yielding a cohesive campaign to impress upon at-risk populations the extreme danger of this drug.

Rural Community Assistance Corporation (RCAC)

Continuing its long association with RCAC, DRA assisted staff in transferring lessons learned through its evaluation of the Agricultural Worker Health and Housing Program (AWHHP) to local agricultural worker health and housing programs throughout California. This work was in direct support of The California Endowment’s new agricultural worker health initiative, “Poder Popular,” and involved the following:

• assisting in the dissemination of ‘lessons learned’ and ‘promising practices’ gathered through evaluation of the AWHHP;

• providing technical assistance and information on affordable housing and environmental infrastructures;

• modeling collaborative decision-making skills and practices;

• working with external evaluation consultants to develop and implement practical learning applications and tools;

• conducting two statewide conferences, and two series of regional roundtable meetings;

• working with other initiatives and organizations to analyze and bring together wealth-creation strategies and resources; and

• providing staff support and coordination to the AWHI/AWHHP Advisory Committee.

Independent Adoption Center

In August of 2005, the Independent Adoption Center (IAC), a national organization based in Pleasant Hill, California, retained DRA to design and complete an evaluation of their Infant Adoption Awareness Training Program.  This program, one of six national federal grants, provided training to health and social service professionals who, through the nature of their work, may be called upon to offer information to pregnant women about open adoption.  DRA took on the role of ensuring that IAC was able to respond to the information needs mandated by their federal grant, as well as developing information vital to refining the training effort.

DRA worked with IAC to review, refine and institutionalize data collection processes and protocols, and to ensure that valid and complete data was collected regarding program participation, economic and geographical challenges faced by program participants, as well as the satisfaction and perceived usefulness of the training at the time of training and at follow-up.  All of these data were used to develop quarterly reports as mandated by the federal grant, as well as reports internal to IAC briefing staff on the program’s progress and needs for program refinement. 

The California Endowment

In April 2005, DRA completed a one year study of The Endowment’s Agricultural Worker Health and Housing Program (AWHHP). This partnership between The Endowment, the Rural Community Assistance Corporation (RCAC) and local projects accounted for over 40 projects throughout rural California. DRA’s evaluation report on the AWHHP was oriented toward the development of ‘lessons learned’ and ‘promising practices’ which could be used to help structure The Endowment’s new rural and/or agricultural worker initiatives. The three volume report provides an overview of findings; a program analysis; and individual project-specific reports on each of the AWHHP projects.  

Overall, this evaluation/information development effort was an attempt to redress the fact that the original AWHHP evaluation met with substantial challenges and, as of early 2004, had not been productive. In redressing this situation, DRA, together with its subcontractor, The Aguirre Group, organized the AWHHP projects into three groups: intensive (highly informative), secondary (less informative) and tertiary (very new or difficult to address) projects. The project-specific reports reflect this classificatory scheme, with the level of detail varying according to the information that could be developed for each project.

California Telemedicine and eHealth Center (CTEC) 

formerly the California Telehealth and Telemedicine Center (CTTC)

DRA was retained by CTTC to evaluate and measure the effectiveness and plausibility of the Center’s three year pilot telehealth and telemedicine re-granting program. The Center’s mission for this three year program was to increase access to and use of specialty medical care in rural and urban medically underserved areas through tele-based technologies. To do this, CTTC, funded by The California Endowment, initially funded 39 pilot telehealth and telemedicine programs, later adding 13 teleophthalmology programs in Indian Country. This contract presented a unique challenge for the evaluation team: DRA was retained to not only evaluate the internal capacity and newly generated organizational structure of CTTC itself, but was also contracted to evaluate all of the 52 project and network grants. To tackle this assignment, DRA worked from five basic values designed to encompass the evaluation of projects ranging in scope from basic telehealth website construction to the tele-delivery of mental health and cardio-care specialty services to the creation of community computer centers.

The first of these five values was that the evaluation was in response to a defined need, more specifically, the need to understand the effectiveness of the grant program, its cost-effectiveness, and to assess the potential for new models of health promotion based on the outcomes of the re-granting program. The second value, endemic to DRA’s work, was that the evaluation of CTTC and all of its grantees be participatory. From our standpoint, the degree to which the evaluation is useful is directly related to the degree to which it is owned by those participating in the program. The third was that the evaluation tailor its perspective to the particulars of each project site. This approach allowed the evaluation team to construct information based specifically on projects differences; focusing on each project as a model, a type. This course was complemented by a fourth value: the evaluation of CTTC and its grantees should make every attempt to distill common lessons and features among variant projects, leading to a set of “lessons learned” that inform the field at the programmatic and policy levels. The final value is capacity building: all of the previous foci were designed to help CTTC and the 52 projects learn more about evaluation and the ways in which evaluation can make a practical difference in the efficiency and effectiveness of their projects.

To read some of the various reports DRA has prepared for CTEC, please visit http://www.cttconline.org/eval.html. 

County Medical Services Program (CMSP)

DRA’s first contract with CMSP began in 1995 and has since provided DRA occasion for intense and varied work with 34 of California’s counties. Defined simply, CMSP provides primary and catastrophic care to non-MediCal indigent populations ages 21-65 in 34 rural California counties. DRA’s latest work with CMSP built on its 1995-96 system-wide needs assessment, and focused on two results: 1) the development of information development and evaluation capacities for CMSP’s Wellness and Prevention Program (W&PP) participants; and 2) the evaluation of the W&PP outcomes.

The County Medical Services Governing Board, in partnership with The California Endowment, established the W&PP grants in order to support counties' efforts to improve the health status of poor and underserved rural populations through community based wellness and prevention activities. In addition to building rich electronic data sources for use by the various W&PP grantees, DRA utilized a wide array of evaluative techniques including quantitative data analysis on client, provider and service characteristics; cost and utilization data analysis; program effectiveness and impact analysis at the client level; and qualitative data gathering and analysis using interview, group meetings and other face-to-face methods. DRA also provided every level of CMSP stakeholder with training and capacity-building exercises and resources. DRA also conducted a special study to document stakeholders’ perspectives on CMSP’s efforts to develop a new strategic direction and to make effective use of strategic analyses.

Access to coherent, concise information on client demographics and services rendered is essential to both the functioning of CMSP and the potential impact of the Wellness and Prevention Program. Because the systems of care in each county range from relatively simple to highly sophisticated, the organizational and tracking systems varied accordingly. The horizon for the overall evaluation effort was the building of counties’ capacities, especially the construction and utilization by the counties of a database built by DRA which, while able to accommodate each county's specific data needs, provided a standardized data set for analysis by CMSP and DRA. Finally, DRA assisted counties with incorporation of a wellness and prevention perspective into their overall viewpoint on indigent health care, so that local efforts best meet the needs of the clients CMSP was designed to serve.

To read the complete CMSP W&PP evaluation report produced by DRA, please visit CMSP's Grant Programs website at http://www.cmspcounties.org/about/grant_projects.html and follow the link to "Volume 2-Evaluation Findings" under the Wellness and Prevention Program.

American Indian Diabetic Teleophthalmology Grant Program (AIDTGP)

Born of the California Telehealth and Telemedicine Center’s (CTTC) initial re-granting round, the AIDTGP functioned as the second round of telemedicine grants designed to increase access to and utilization of specialty care. Unlike the first round of CTTC grants which ran the gamut of telehealth and telemedicine, the AIDTGP program was extremely specific in its focus: providing tribes specialty eye care through the use of tele-equipment. Serving a population often faced with the prospect of traveling for a day or more to receive specialty eye care, the AIDTGP program was designed to bring the care to the patient, easing massive barriers to care. DRA has provided the projects not only evaluation services - how best to meet the needs of community members while maintaining a functional program - but also a high level of technical and logistical support - how to coordinate doctors, clinics, technicians, and data into a functioning whole.