MAJOR WORK

Following are descriptions of major organizations with which DRA worked, the approaches DRA took  with each of the contracts, and the outcomes of the evaluation process.

Prison Industry Authority, State of California

DRA, under subcontract to California State University, Long Beach, was retained to design a study of the effectiveness and associated costs of the programs administered by the California Prison Industry Authority (PIA).  Specifically, the study would:

• identify a range of outcomes which best represent the impact of PIA’s work with California prison inmates;

• identify costs associated with administering/conducting PIA’s work;

• determine the availability of data sets related to PIA’s work;

• determine the processes for accessing such data;

• access existing and/or develop new data and databases to capture outcomes and costs;

• conduct statistical analyses to isolate those factors that best account for outcomes related to PIA’s work with California prison inmates, e.g., inmate demographics; inmate criminal histories; PIA occupational training; job placement characteristics; etc.;

• report on significant and/or provocative relationships between outcome factors, outcomes and costs;

• institutionalize a process within PIA to continuously assess and report on outcomes and costs associated with its programs.

The study was designed to gather an extensive and great variety of data which would be subjected to unusually rigorous analytical techniques to assess the cost effectiveness of PIA’s work with California prison inmates. At a minimum, the study was structured as a matched cohort design with respect to recidivism and behavior change for PIA workers, with an emphasis on prison violence reduction. More optimally, the study was designed to focus on factors which may account for the relative success or failure of post-incarceration employment and re-integration into local communities. These analyses would be published, with the primary intended audience being PIA’s clients, California’s Youth and Adult Corrections Agency, as well as State control agencies, including the California State Senate, the State Department of General Services, Bureau of State Audits and State’s Legislative Analysts Office.

DRA did, in fact, complete this two-year study and delivered to PIA the most comprehensive report of a state's prison industry ever produced in the United States at that time.

Consequent to the delivery of this report DRA was again retained by PIA to conduct a smaller-scale follow-up study on PIA's competitiveness in delivering goods and services to California State agencies. In addition, DRA provided continuous consulting and technical assistance to PIA in support of its newly installed data systems, which were in large part restructured to reflect the data sets DRA built during its major study.

County Medical Services Program (CMSP)

DRA’s first contract with CMSP began in 1995. During the years following DRA had many opportunities for intense and varied work with 34 of California’s counties. Defined simply, CMSP provided primary and catastrophic care to non-MediCal indigent populations ages 21-65 in 34 rural California counties. DRA’s last work with CMSP built on its 1995-96 system-wide needs assessment, and focused on two results: 1) the development of information development and evaluation capacities for CMSP’s Wellness and Prevention Program (W&PP) participants; and 2) the evaluation of the W&PP outcomes.

The County Medical Services Governing Board, in partnership with The California Endowment, established the W&PP grants in order to support counties' efforts to improve the health status of poor and underserved rural populations through community based wellness and prevention activities. In addition to building rich electronic data sources for use by the various W&PP grantees, DRA utilized a wide array of evaluative techniques including quantitative data analysis on client, provider and service characteristics; cost and utilization data analysis; program effectiveness and impact analysis at the client level; and qualitative data gathering and analysis using interview, group meetings and other face-to-face methods. DRA also provided every level of CMSP stakeholder with training and capacity-building exercises and resources. DRA also conducted a special study to document stakeholders’ perspectives on CMSP’s efforts to develop a new strategic direction and to make effective use of strategic analyses.

Access to coherent, concise information on client demographics and services rendered was essential to both the functioning of CMSP and the potential impact of the Wellness and Prevention Program. Because the systems of care in each county range from relatively simple to highly sophisticated, the organizational and tracking systems varied accordingly. The horizon for the overall evaluation effort was the building of counties’ capacities, especially the construction and utilization by the counties of a database built by DRA which, while able to accommodate each county's specific data needs, provided a standardized data set for analysis by CMSP and DRA. Finally, over several subsequent years DRA continued to assist counties with incorporation of a wellness and prevention perspective into their overall viewpoint on indigent health care, so that local efforts best meet the needs of the clients CMSP was designed to serve.

California Telemedicine and eHealth Center (CTEC) 

formerly the California Telehealth and Telemedicine Center (CTTC)

DRA was retained by CTTC to evaluate and measure the effectiveness and viability of the Center’s three year pilot telehealth and telemedicine re-granting program. The Center’s mission for this three year program was to increase access to and use of specialty medical care in rural and urban medically underserved areas through tele-based technologies. To do this, CTTC, funded by The California Endowment, initially funded 39 pilot telehealth and telemedicine programs, later adding 13 teleophthalmology programs in Indian Country. This contract presented a unique challenge for the evaluation team: DRA was retained to not only evaluate the internal capacity and newly generated organizational structure of CTTC itself, but was also contracted to evaluate all of the 52 project and network grants. To tackle this assignment, DRA worked from five basic values designed to encompass the evaluation of projects ranging in scope from basic telehealth website construction to the tele-delivery of mental health and cardio-care specialty services to the creation of community computer centers.

The first of these five values was that the evaluation was in response to a defined need, more specifically, the need to understand the effectiveness of the grant program, its cost-effectiveness, and to assess the potential for new models of health promotion based on the outcomes of the re-granting program. The second value, endemic to DRA’s work, was that the evaluation of CTTC and all of its grantees be participatory. From our standpoint, the degree to which the evaluation is useful is directly related to the degree to which it is owned by those participating in the program. The third was that the evaluation tailor its perspective to the particulars of each project site. This approach allowed the evaluation team to construct information based specifically on projects differences; focusing on each project as a model, a type. This course was complemented by a fourth value: the evaluation of CTTC and its grantees should make every attempt to distill common lessons and features among variant projects, leading to a set of “lessons learned” that inform the field at the programmatic and policy levels. The final value was capacity building: all of the previous foci were designed to help CTTC and the 52 projects learn more about evaluation and the ways in which evaluation can make a practical difference in the efficiency and effectiveness of their projects.

The California Endowment

In April 2005, DRA completed a one year study of The Endowment’s Agricultural Worker Health and Housing Program (AWHHP). This partnership between The Endowment, the Rural Community Assistance Corporation (RCAC) and local projects accounted for over 40 projects throughout rural California. DRA’s evaluation report on the AWHHP was oriented toward the development of ‘lessons learned’ and ‘promising practices’ which could be used to help structure The Endowment’s new rural and/or agricultural worker initiatives. The three volume report provides an overview of findings; a program analysis; and individual project-specific reports on each of the AWHHP projects.  

Overall, this evaluation/information development effort was an attempt to redress the fact that the original AWHHP evaluation met with substantial challenges and, as of early 2004, had not been productive. In redressing this situation, DRA, together with its subcontractor, The Aguirre Group, organized the AWHHP projects into three groups: intensive (highly informative), secondary (less informative) and tertiary (very new or difficult to address) projects. The project-specific reports reflect this classificatory scheme, with the level of detail varying according to the information that could be developed for each project.

Rural Community Assistance Corporation (RCAC)

Continuing its long association with RCAC, DRA assisted RCAC staff in transferring lessons learned through its evaluation of the Agricultural Worker Health and Housing Program (AWHHP) to local agricultural worker health and housing programs throughout California. This work was in direct support of The California Endowment’s new agricultural worker health initiative, “Poder Popular,” and involved the following:

• assisting in the dissemination of ‘lessons learned’ and ‘promising practices’ gathered through evaluation of the AWHHP;

• providing technical assistance and information on affordable housing and environmental infrastructures;

• modeling collaborative decision-making skills and practices;

• working with external evaluation consultants to develop and implement practical learning applications and tools;

• conducting two statewide conferences, and two series of regional roundtable meetings;

• working with other initiatives and organizations to analyze and bring together wealth-creation strategies and resources; and

•  providing staff support and coordination to the AWHI/AWHHP Advisory Committee.

American Indian Diabetic Teleophthalmology Grant Program (AIDTGP)

Born of the California Telehealth and Telemedicine Center’s (CTTC) initial re-granting round, the AIDTGP functioned as the second round of telemedicine grants designed to increase access to and utilization of specialty care. Unlike the first round of CTTC grants which ran the gamut of telehealth and telemedicine, the AIDTGP program was extremely specific in its focus: providing tribes specialty eye care through the use of tele-equipment. Often faced with the prospect of traveling for a day or more to receive specialty eye care, the AIDTGP program was designed to bring the care to the patient, easing massive barriers to care. DRA provided the projects not only evaluation services - how best to meet the needs of community members while maintaining a functional program - but also a high level of technical and logistical support - how to coordinate doctors, clinics, technicians, and data into a functioning whole.  DRA's report assimilated and assessed the various data and provided a model of how future programs may approach similar efforts.

Sonoma County HIV/AIDS Prevention Education Project

During the early 2000s, the Sonoma County Department of Health Services facilitated a community-based effort to design and develop a comprehensive approach to countywide HIV education and prevention. The result was a 3-year Strategic Plan that aimed to guide local funding and intervention strategies geared toward educating community members, organizations/institutes and policy-makers on: HIV transmission, risk reduction strategies, and local resources for the prevention of HIV and the support of those who are infected. The County, along with the local Prevention and Planning Group, engaged DRA to complete two tasks: 1) to assist in the development of evaluation plans for grantee programs, and 2) to conduct assessments of the ongoing strategic planning and program implementation process at both the County and local program levels.

As always, building the evaluation capacity of local stakeholders was key to DRA’s role. Understanding the value of partnerships, the power of community engagement, and the importance of good information with respect to building and sustaining essential programs and activities was therefore a focus of this effort. In addition, DRA continued to assist the HIV Prevention and Education Program in evaluating the extent to which the local projects meet the goals specified in the County Strategic Plan. Likewise, DRA was central to assessing and documenting the extent to which the County as a whole met the mandates of the State Office of HIV/AIDS. Tailored approaches to evaluation were central to DRA’s work with each of the individual grantees and the County operated programs. At the same time, process, outcome and impact commonalities among the full range of projects and strategies being implemented were explored.

Independent Adoption Center

In August of 2005, the Independent Adoption Center (IAC), a national organization based in Pleasant Hill, California, retained DRA to design and complete an evaluation of their Infant Adoption Awareness Training Program.  This program, one of six national federal grants, provided training to health and social service professionals who, through the nature of their work, may be called upon to offer information to pregnant women about open adoption.  DRA took on the role of ensuring that IAC was able to respond to the information needs mandated by their federal grant, as well as developing information vital to refining the training effort. In addition, DRA’s work with IAC involved a pilot study with Planned Parenthood of California to track and analyze the outcomes and impact of IAC’s training on adoptions where Planned Parenthood provided counseling and referral services.

DRA worked with IAC to review, refine and institutionalize data collection processes and protocols, and to ensure that valid and complete data was collected regarding program participation, economic and geographical challenges faced by program participants, as well as the satisfaction and perceived usefulness of the training at the time of training and at follow-up.  All of these data were used to develop quarterly reports as mandated by the federal grant, as well as reports internal to IAC, briefing staff on the program’s progress and needs for program refinement.