External evaluator Renee Lemmons conducts a survey with teachers.
Nano-CEMMS serves as a site for collaboration and innovative research in nanoscience with a focus on manufacturing applications. Human Resource Development and Educational Outreach Programs (HRD) are key to accomplishing this agenda.
The success of all HRD activities is measured by written goals. Specifically, the activities must:
- Contribute to the diversity of the Center participants and the scientific community
- Promote the professional advancement of B.S., M.S., and Ph.D. Students
- Educate and transfer knowledge to K-12 students, community college, undergraduate, and graduate students
The evaluation effort is a collaboration between Nano-CEMMS and the College of Education faculty and staff at UIUC and NCA&T. Evaluation is imbedded in all human resource and education activities of the Center. The approach is designed to provide both formative and summative feedback regarding program improvement and progress toward goals. Quantitative and qualitative data are gathered using direct observation, interviews and surveys.
The Center’s Human Resource Development commitment is to:
- Develop a diverse U.S. workforce of educators, scientists, engineers, and practitioners who will advance the field of nanoscience/technology in the U.S. and beyond
- Increase the scientific literacy of the public at large
These goals fall into two overlapping categories: the internal educational program of integration and collaboration among graduate students, post-doctoral associates and fellows, researchers and professors; and the external K-12 educational efforts with students and teachers. All HRD activities are linked to appropriate assessments and analyses in order to maximize their quality and efficacy.
The purposes of this external evaluation are twofold:
- To give formative (i.e., instrumental) guidance on the design and implementation of HRD/Educational Outreach programs
- To provide summative (i.e., assessment/accountability oriented) feedback on the extent to which these programs met the intended goals
Several main questions guide the work of the evaluators:
- Centerness – What is the vision of the center? How is it communicated? Is it adaptive? Is there buy in? What does it look like? Where does it happen? How pervasive is it?
- Sustaining Power – What systems for dissemination, in both the narrow and broad sense, are being developed? What do they look like and how will they be institutionalized?
- Diversity – What is the relationship between the diversity and educational programs? How can these two foci be blended in order to move forward on both issues? What more is planned and how will continuing efforts be institutionalized through recruitment and programming?
- Interdisciplinary Training and Collaboration – How is the Center responding to this call (for each group, across universities)? How is this encouraged? Where does it happen? What does it look like?
The Center engages an external team to develop and implement a systematic plan for evaluation. The evaluation team develops data collection instruments and procedures that measure the effectiveness of each educational activity and ascertain what works best, for whom, and under what circumstances. Methods and analyses emphasize formative and continuous feedback and are being structured to enable Center staff to institutionalize evaluation efforts, building the capacity of staff to engage in ongoing evaluation. The evaluation also serves a summative function, reporting on the accomplishment of HRD goals and impact annually and at projects’ end.
Lizanne DeStefano, Professor of Educational Psychology, College of Education, University of Illinois at Urbana-Champaign, leads the external evaluation of the Center’s Human Resource Development and Educational Outreach. In addition to the Nano-CEMMS and Bureau of Educational Research evaluation staff dedicated to this project, additional resources are utilized to measure the effectiveness of the Center’s educational outreach, including measures put into place by the U.S. Department of Education and the Illinois State Board of Education, the Champaign Public Schools, and Education for Employment District #330.
Professors Jennifer Greene and Lizanne DeStefano of the Department of Educational Psychology at UIUC have support from the National Science Foundation to develop and field test a value-engaged approach to evaluating STEM educational programs. This approach emphasizes evaluation as an opportunity for learning about effective STEM education, as manifest in diverse contexts. The approach is centered on assessments of the quality of STEM content in the program being evaluated, the appropriateness of the program's pedagogy, and the ways in which the program advances equity in the relevant STEM field. This evaluation approach thus respects the importance of context in teaching and learning, and legitimizes diversity of perspective and experience.
Nano-CEMMS is a field test site for Professors Greene and DeStefano's NSF project. The Center's educational outreach offers opportunities for testing this evaluation approach in both K-12 and higher education contexts. These evaluation field tests have contributed valuable feedback to the Center's educational staff. It is especially useful to the evaluation team to have an ongoing relationship with Center staff around evaluation issues. Evaluation theory and procedures can be refined and revised from one evaluative field test to the next. Opportunities for deeper evaluative understanding on the part of Center educational staff are also possible through sustained communications over time. Beyond this, the evaluation team themselves have repeated exposure to the basic ideas of nanoscience and nanotechnology, ideas they can further share and incorporate into their own understandings of science as citizens.
The external evaluation team regularly disseminates findings to Center staff in order to provide feedback that can be used while programs are in progress, as well as in the design of future programs. The external evaluation team also provides information in formal reports to NSF site visit teams and project managers, and may present findings at professional conferences and in scholarly publications.