SMS has been doing program evaluation since our evaluation of the Hawai‘i Department of Education’s Title I Program, 1978-1981. Most of our formative and summative evaluations have been in education, health, and human services. We have also done many needs assessments, evaluation plans, data collection tools development, and data processing for evaluation. On the macro side, we conduct major health and education monitoring projects to measure the impact of public programs.
Recently we have been involved in some important projects in the tobacco prevention field. SMS has been responsible for the evaluation of the Hawai‘i Tobacco Trust Fund Community Grants Program, the Clear the Smoke Media Program, the HCF Innovations Grants Program (2007), and have contributed data collection services for the Tobacco Trust Fund Media Campaign Evaluation, and several major studies conducted by and for the Healthy Hawai‘i Initiative.
SMS staff includes several members with extensive experience with program evaluation:
Our evaluation section is housed in our consulting division. Business or non-profit agency, we want to produce analysis for improving programs. We plan on basing that analysis on hard data derived from your program operations and to develop from the analysis, plans that will make your program work better.
That doesn’t mean we ignore accountability, far from it. Profit-disbursing companies are accountable to their shareholders, their customers, and the communities in which they operate. Non-profit service-delivery agencies are equally accountable to their funding agencies, their clients, and their communities.
Evaluation is valuable to any organization that operates strategically, that is, that is set up to learn from its experience and apply that intelligence to improving the product or service delivered to its clients and constituents. The strategic thinking organization plans, then operates according to those plans, observing how things went, then evaluates the information and develops a new plan based on those findings. Evaluation springs from this model. Outside of this model, or one very much like it, there is no meaningful role for evaluation.
Every planning cycle should have an evaluation stage, where plan projections are compared with actual results and adjustments made as necessary to the plans for the following year. For non-profits the evaluation process is frequently more challenging because “bottom-line results” are much less tangible.
Program evaluations are as varied as there are programs. Worse yet, the field has grown to include many different vocabularies over the years. We trust that our use of the specific terms we selected won’t be taken as a sign we don’t know the others.
In general our lexicon is eclectic; our services flexible. Unified by a desire to provide reliable information for program improvement.
A representative design is shown below:
The figure shows all of the elements of a good, formal evaluation design. Some of the key elements, like the logic model and measurement devices are described in greater detail elsewhere on our site.
SMS has years of experience producing evaluation plans. We can produce all of them for your program, produce part of them, or use your evaluation designs from scratch.
SMS has all of the equipment and facilities needed to perform effective program evaluation data collection and analysis. We have all the data collection methods you might need. We have staff, hardware, software to accomplish any type of analysis you might need. We have experience in most kinds of evaluation. We can develop reports useful to a wide variety of audiences, including program staff, boards of directors, funding agencies, community groups, client groups, lawmakers, and others. Similar to our business consulting work, SMS develops operating plans for public agencies we have evaluated.
SMS did a great job in leading Hawai‘i Tourism Authority’s evaluation development process…including identifying input, output and outcome measures to track progress toward HTA’s strategic goals. SMS also provided steps to guide us in integrating the evaluation process into our overall operations and planning. Our staff was actively engaged in the process and tracking the measures will ensure a focus on driving towards outcomes.
— Leslie Dance, former VP Marketing
Hawai‘i Tourism Authority