Scientific Reasoning Research Institute - PERG

Assessing-to-Learn (A2L) Website Moved to UNC

cmiller — Thu, 15 Mar 2012 16:35:44 +0000

The Assessing-to-Learn website (A2L.physics.umass.edu) has moved to a new server (clickercentral.net), contact Ian Beatty (idbeatty@uncg.edu) for further information. More information about A2L is on the A2L project page.

PERG Moved from UMass to UNCG

ibeatty — Sat, 21 Mar 2009 02:47:12 +0000

The UMass Physics Education Research Group has moved from UMass to the University of North Carolina at Greensboro. As of June 2008, Prof. William Gerace retired from UMass after 30 years in the Department of Physics and 16 years as Director of SRRI. He moved to Greensboro, North Carolina, and accepted a position as the first Helena Gabriel Houston Distinguished Professor of Science Education at UNCG. In addition to teaching in the UNCG Department of Physics & Astronomy, he pursues a vigorous agenda of promoting science education research, teaching, and outreach.

In January 2009, Prof. Ian Beatty accepted a position as Assistant Professor in the Department of Physics & Astronomy at UNCG. He continues his association with SRRI as a Visiting Scholar, spending about one week per month in residence to continue working with the staff and participating secondary school teachers of the Teacher Learning of Technology-Enhanced Formative Assessment (TLT) project.

In August 2009, Prof. William Leonard will also join the UNCG Department of Physics & Astronomy as Associate Professor, where — in addition to Physics Education Research and his association with the TLT project — he will continue his Engineering Education Research in collaboration with North Carolina Agricultural & Technological University.

One of the group's top agenda items is to build a graduate program in Physics Education Research and cross-disciplinary Science Education Research at UNCG.

Minds*On Physics Project

root — Sun, 13 Jan 2008 00:04:22 +0000

Developing and field testing a research-based curriculum for high school physics

Contact(s):

Leonard, William J.

Funding:

NSF MDR-9050213, NSF MDR-9255713

Starting date:

1993-03-01

To learn about the finished Minds·On Physics product, see the MOP entry in our Resources section.

When the Minds·On Physics (MOP) project began in 1990, there were no exemplary materials for high school physics instruction, the most popular textbooks were full of errors and oversimplifications, and college physics students seemed to be at a disadvantage if they had taken physics in high school. Results of cognitive research were starting to make their way into college instruction (by the researchers themselves, for the most part), but there had been little impact on high school physics instruction.

Pulling together multiple strands of educational research, such as cognitive overload, expert-novice differences, metacognition, and alternative conceptions, UMPERG created a framework for describing how knowledge is stored and used. We also identified 5 stages of cognitive development. Using the framework and stages of development, we proposed to create activities to help students explore existing concepts, interrelate concepts, analyze and reason, solve problems, and organize knowledge.

At first, the approach was so different from anything that had been done previously that NSF would fund only a small pilot project with four teachers in four very different settings. The activities were found to be useful and effective with the pilot teachers and classes. Now there are more than 180 activities in 6 volumes for students and over 2,000 pages of support materials for teachers.

The materials have many special features. For instance, activities are done first, with little or no preparation, and no need for any introduction by the teacher. The reading assignments are brief (1-2 pages per activity) and done after the activities are completed, rather than before. The activities demonstrate multiple paths to success, while encouraging thoughtfulness, communication, teamwork, and self-awareness as essential features of learning. The program is algebra-based, but it avoids over-generalizations by using graphs and mathematical principles to take into account changing quantities. The materials also support and promote a new approach to teaching that stresses modeling, conversation, advising, and mentoring over traditional, one-size-fits-all lecturing.

Having published the last Teacher's Guide early in 2003, the project has moved into an implementation and adoption phase, with visits to teachers and school systems interested in trying out a new style of teaching and learning. Grand Rapids (MI) Public Schools has recently adopted MOP, and Chicago Public Schools (among others) is currently considering the program.

Knowledge Broker

root — Sat, 12 Jan 2008 23:53:36 +0000

Exploring technology for next-generation classroom response systems

Contact(s):

Beatty, Ian D.

Funding:

Hewlett-Packard Co. (University Mobile Technology grant); Microsoft Corp. (University Relations grant)

Effective pedagogy --- identifying students' initial understanding and misconceptions, engaging their minds in the activity of learning, continually monitoring their individual progress, and adjusting one's teaching to the ever-changing circumstances of the classroom --- is difficult to practice with medium-sized and large classes. Technology can help, in the form of Classroom Communication Systems (CCS) and Classroom Response Systems. These are combinations of hardware and software that help an instructor give students questions to answer or tasks to accomplish in class, individually or in small groups; collect the answers; and immediately display a statistical summary or histogram of the answers for the instructor and potentially the entire class. In concert with appropriate pedagogic techniques and curriculum, such systems can enable interactive, active, dynamic pedagogy in large classes to a degree not previously possible.

Currently available systems, however, are pedagogically limiting and not ideally suited to forward-thinking university instruction. They are also closed and proprietary, not amenable to creative innovation, improvement, or extension. We believe a need exists for an open CCS that is:

designed according to sound, current thinking in educational research; and
not just a CCS product, but a platform for ongoing research, innovation, and growth in technology-assisted classroom pedagogy.

We have, therefore begun exploratory development of the Knowledge Broker, a next-generation CCS to serve both as a tool for practical teaching and as an instrument for research into technology-enhanced classroom pedagogy. In the process, we are finding other uses for Tablet PC technology in university instruction.

This project was born in collaboration with, and has provided support to, Prof. Gino Sorcinelli's ConferenceXP project in the UMass Isenberg School of Management.

Update: As of 2005, this project has been suspended indefinitely due to lack of support.

homeworkCentral

root — Sat, 12 Jan 2008 23:47:21 +0000

Online support for students doing electronic homework

Group Page:

PERG

Contact(s):

Leonard, William J.

Funding:

NSF ROLE-0106771

Starting date:

2003-09-01

At a large state university such as UMass, resources are necessarily limited, and students in a course with 200 or more classmates often think they cannot possibly get the assistance they need to answer questions and complete their homework. Course personnel are available for only certain hours and only during the day, and the departmental resource room has help that is regrettably spotty. A few students seek and receive help, but the majority of students work alone or in small groups, without having useful assistance.

We also use an electronic homework (eHW) system to administer out-of-class activities. While eHW has many advantages, such as immediate feedback and the opportunity to attempt each problem many times, there are also some drawbacks. For instance, students sometimes become overly focused on getting the answer correct and earning a perfect score on their assignments, and inevitably develop frenetic, superficial approaches to doing their homework.

So, even though analysis activities are being encouraged and modeled during class, since students spend most of their time outside of class, it is essential to help students engage in analysis activities while working on their homework. As part of the RRA project, we also despaired of ever monitoring what students are doing during homework, and therefore, we thought we would never be able to judge the level of engagement of individual students.

"It's very helpful most of the time; like having my own [professor]."

To address both of these concerns, we created a web site called homeworkCentral (hwC). For each homework problem assigned, there are typically 3-5 suggestions for activities designed to help students analyze the situation and solve the problem, with around a dozen hints for executing the suggested activities and a separate page of common pitfalls to avoid. The web site is available online, so students can get support from a trusted source any time they need it.

"Homework central helps us with our eHW when no one else is around to help (ie the professor) Helps us look at the problem in the correct way and names the common pitfalls so we are to avoid them."

By structuring the site with each hint on a separate page and pitfalls separate from suggestions, we anticipate being able to profile and follow students as they work through each homework problem. By correlating the hit logs in hwC with those from eHW, we hope to be able to construct a rather complete picture of how students — all students — begin, process, cope with, and finally complete their homework assignments.

"When doing homework, it's helpful to have the help immediately and from a source that is guarenteed [sic] to be reliable."

Every Decision Counts (EDC)

root — Sat, 12 Jan 2008 23:40:47 +0000

Developing and researching the impact of an alternative format for multiple-choice assessment

Contact(s):

Leonard, William J.

Funding:

NSF ROLE-0106771

Starting date:

2002-02-01

Standard multiple-choice assessments, for which there are 4 incorrect and exactly one correct choice, are efficient to implement but imprecise and difficult to interpret. To assess student course work, teachers generally accept the limitations of the standard multiple-choice format in order to take advantage of the low cost and minimal support needed to implement it.

For educational research, however, the standard format is unacceptable. Students guess; some of them guess correctly. Some are confident in their answers; others are not. With the standard format, we cannot distinguish between these states. Open-ended formats are therefore preferred, but open-ended questions and answers can be extremely time-consuming both to administer and to analyze.

_Every Decision Counts_ (EDC) is a compromise between open-ended and standard, multiple-choice formats. Students are allowed to mark more than one choice on a standard 5-bubble answer sheet, which has two consequences. The first is that students can communicate their confidence in their answer to a standard question by selecting two or more incompatible choices. The more marks they make, the less credit they earn and the less confidence they have in their answer. If they fill in all 5 bubbles, they are "just guessing."

The second consequence is that we can ask questions with possibly more than one mark in the correct answer. There are 31 possible combinations of answers, and therefore it is almost impossible for students to guess the correct one. Possible scores are 0, 1, 2, 3, 4, and 5, so there is a more precise categorization of students between completely correct (5/5), and completely incorrect (0/5). We also have a just guessing category for students who mark all 5 choices. As compared to standard multiple-choice questions, a lower percentage of students are completely correct (earning all 5 points), yet the average score tends to be noticeably higher, because nearly all students are earning at least 3 points (out of 5). In other words, in practice, almost nobody earns 0 or 1 point, and few earn only 2 points.

We would like to study the accuracy of EDC by comparing students' answers in this format to their answers in a more open-ended format. We would also like to study the potential of EDC for measuring problem-solving proficiency, which cannot currently be measured with the standard multiple-choice format. In 2006 or 2007, we anticipate being able to secure funding to study EDC, as well as identify and analyze question styles, develop and make available a database of exemplary questions, and work toward wider implementation of EDC by teachers and researchers who use multiple-choice assessments.

As a public service, we have created a website for instructors who want to use EDC. It includes web-based tools for making the scoring process simple.

EDC is a spin-off from our RRA project.

ConMap

root — Sat, 12 Jan 2008 23:34:28 +0000

Computer-based assessment tools for probing physics students' conceptual knowledge structures

Group Page:

PERG

Contact(s):

Beatty, Ian D.

Starting date:

1995-01-01

Traditional problem-based exams are not reliable tools for diagnosing students' knowledge and guiding pedagogical intervention; new tools grounded in cognitive science and educational research are needed. If one wishes to assess a students' knowledge in detail, rather than merely summarize items on which the student did or did not succeed, one needs a model of what a knowledge state is, instruments for probing a student's state of knowledge, and an understanding of the mechanism by which instruments probe the state. An effective diagnostic assessment must describe a student with reference to some suitably detailed model of physics knowing, learning, and application.

Our ConMap ("Conceptual Mapping") project has two interdependent objectives. One is developing a detailed, quantitative model of physics conceptual knowledge and its application to problem-solving and analysis. The other is creating practical tools to enhance teaching through frequent "formative assessment" and monitoring of learning. We have been investigating the potential of a set of simple, computer-administered term-association tasks for probing the concepts and interconnections within a physics student's knowledge store. By eliciting spontaneous associations between physics terms in a variety of ways and contexts, we hope to build up a map of the concepts within a topic area that a student has access to and of the web of associations providing structure to those concepts. Since experts and novices are known to structure their knowledge in qualitatively different ways - experts hierarchically around key principles with rich interlinking, novices chronologically with sparse interlinking - we hope to detect signatures of expertise in the patterns of associations elicited, and perhaps observe the onset of those signatures as learning occurs.

This project began with Ian Beatty's dissertation work in the 1990's. Since then, it has been pursued through the dissertation work of graduate students Jenny Chang and Edgardo Ortiz.

Preliminary work has established that the approach is "interesting and promising": We have found internal consistency within the data and partial correlations with other measures of knowledge, suggesting that the ConMap probes are in fact measuring something real, reproducible, and relevant. We have also experimented with quantitative dynamical models "explaining" some aspects of the observed data.

The original project is described in Ian's dissertation (2000) and, partially, in a paper published in 2002 the American Journal of Physics. Later work has not been written up (yet).

Springbok

root — Fri, 11 Jan 2008 23:23:38 +0000

A simple but rich mechanical system for teaching about the physics of jumping

Group Page:

PERG

Contact(s):

Leonard, William J.

(This is a companion website to the article Springbok: The physics of jumping, published in The Physics Teacher.)

Springbok is our name for a simple mechanical system for teaching about the physics of jumping. A springbok consists of a large mass and a small mass connected by a spring; when compressed and then released, it jumps up into the air. (The name "springbok" comes from a South African gazelle noted for its grace and its delightful habit of springing suddenly into the air.)

A springbok is easy to make and engaging to study. It provides a rich context for exploring a wide range of physics concepts and principles, and it possesses a number of features that give it broad instructional value. There is much a student can learn about the physics of jumping from a purely conceptual analysis of this toy.

However, the simplicity of the spring-loaded design also allows for a straightforward quantitative analysis of jumping. A springbok is ideal for hands-on projects and science competitions. With an appropriate focus, a springbok can be used in a variety of instructional settings, from high school physical science to graduate mechanics.

An article providing both a conceptual and quantitative analysis of the springbok (the mechanical system, not the gazelle) can be found in the published paper Springbok: The physics of jumping.

RRA

root — Tue, 27 Nov 2007 19:27:08 +0000

Researching the Role of Qualitative Analysis

Group Page:

PERG

Contact(s):

Leonard, William J.

Funding:

US National Science Foundation grant ROLE-0106771

Starting date:

2001-09-01

RRA was a research project on the combined impact of qualitative analysis and reasoning activities and formative assessment on the attitudes, conceptual understanding, skills, and problem-solving proficiency of introductory college physics students.

Too often, students in introductory college physics develop superficial problem-solving approaches - often manipulating equations and avoiding concepts. Meanwhile, their instructors generally have lofty goals for them, such as developing deep conceptual understanding and forward-looking strategic approaches to problem solving, yet somehow the day-to-day activities seem to discourage these outcomes from ever happening.

Analysis activities, in which students use concepts to reason and answer questions about problem situations, are being used as a bridge between the current reality of physics instruction and the desired state of helping students develop transferable thinking skills. The approach, called Analysis-Based Problem Solving, has been used by UMPERG members for many years, but only recently have we been funded to study the approach and its effect on student attitudes and skills.

The data consists of surveys (weekly attitudes, biweekly pretests, and end-of-semester posttests), performance (exams and quizzes using the EDC format; eHW), and hit logs (eHW and homeworkCentral). The goal is to classify students in terms of their prior knowledge and skills, as well as level and style of engagement, and then look for correlations of these variables with student attitudes and an array of performance indicators, such as conceptual understanding, analysis and reasoning ability, and problem-solving proficiency. The Every Decision Counts (EDC) format for multiple-choice assessments and the homeworkCentral (hwC) web site are critical elements of the project. EDC is a format invented to given us more precise performance data, and hwC is a web site created to allow us to monitor student engagement while working on electronic homework.

Question Driven Instruction

root — Sun, 25 Nov 2007 05:15:17 +0000

Question-driven instruction (QDI) means having students wrestle with rich, meaty, meaningful questions and problems as a context for sense-making and a vehicle for learning, not just as assessments.

(We'll be writing more about this soon, but we needed a placeholder page here now!)

Beatty Talks About TLT Project For STEM Institute

root — Sun, 25 Nov 2007 04:23:32 +0000

On Tuesday, December 4, SRRI's Ian Beatty will give a talk on the SRRI TLT project as part of the UMass STEM Institute's seminar series. All are welcome to attend. Time: 4:30 PM. Location: Hasbrouck 138.

Minds*On Physics

root — Sat, 27 Oct 2007 15:34:42 +0000

A constructivist, active-learning curriculum for high school physics

Contact(s):

Leonard, William J.

MOP is a one-year curriculum for high school physics. It is the result of a materials development project supported by the National Science Foundation, and its design was guided by educational research findings. The curriculum integrates topics traditionally taught at different times of the year, and students are expected to develop conceptual understanding of physics while improving problem-solving proficiency.

A2L Project

root — Sat, 27 Oct 2007 01:55:51 +0000

Assessing-to-Learn: Continuous Formative Assessment for Physics Instruction

Group Page:

PERG

Contact(s):

Gerace, William J.

Funding:

US National Science Foundation grant ESI-9730438

Starting date:

1998-05-01

There has been considerable interest in assessment, especially in view of the goals set forth in reform documents, such as the NRC's National Science Education Standards. Generally, those goals call for science learning to focus on conceptual understanding, problem solving, and science as inquiry. Most agree that there is a mismatch between those goals and summative tests used to rank students on those goals. Formative assessment practices provide a better match, but there are barriers to teachers adopting such practices. We argue that a good first step toward helping teachers adopt a program of formative assessment is the development of (simple-to-use) formative assessment activities and an efficient process of collecting formative assessment data.

Assessing-to-Learn (A2L) is a multifaceted project with the purpose of advancing our understanding and practice of formative assessment in the teaching of high school science. The original goal of the A2L project was to develop quality formative assessment materials (i.e., assessment materials that a teacher can use to make decisions about subsequent instruction) for teaching high school physics. Lessons learned during the development and pilot testing of formative assessment activities led us to broaden the scope of the project to address other significant issues (i.e., other than the lack of quality formative assessment materials) affecting the adoption and implementation of formative assessment practices, including the following issues:

How can one effectively integrate assessment and instruction to meet the wide variation in teaching practices among high school science teachers?
What factors influence a teacher's adoption of particular formative assessment practices?
What kinds of support do teachers need to implement a program of formative assessment?
How do we enable teachers to become authors of formative assessment activities?
What preparation do teachers need to be effective practitioners of formative assessment?
What is the effect of formative assessment practices on the classroom dynamic and the way teachers teach?

We have identified eight tangible outcomes for the A2L project (listed on the next page). Seven of the eight outcomes are framed in terms of a product that supports some aspect of formative assessment (i.e., classroom practice, teacher education, development of assessment materials, etc.). The remaining outcome concerns addressing certain research questions related to formative assessment — chosen specifically because we can make a contribution to these questions given the other goals of the project and the available resources.

A set of prototype formative assessment items (aimed at introductory high school physics) for use with a classroom communication system.
Teacher aids containing answers to the assessment items and other useful instructional information.
A design paradigm for formative assessment items.
Strategies for classroom formative assessment.
A guide to formative assessment for practicing teachers who are interested in a practical introduction to formative assessment and for teacher educators who wish to include a component of formative assessment in their teacher education programs.
Design of an on-line course.
A2L web site providing access to all published A2L materials.
Research improving the knowledge base on teacher adoption of classroom formative assessment.

As part of the project, we created a library of formative assessment "items" — multiple-choice questions for use with classroom response system teaching — designed as a vehicle for classroom interaction and learning, rather than for testing what students already know. This library is available to the public on the Assessing-to-Learn project website (A2L.physics.umass.edu or clickercentral.net). This site has a new location as of May, 2012.

The A2L project directly motivated and informed a current large project, Teacher Learning of Technology-Enhanced Formative Assessment.

TLT

Jang Kreetong — Sat, 27 Oct 2007 01:39:05 +0000

Teacher Learning of Technology-Enhanced Formative Assessment

Group Page:

PERG

Contact(s):

Beatty, Ian D.

Contact(s):

Leonard, William J.

Funding:

US National Science Foundation grant ESI-0456124 (TPC)

Starting date:

2005-07-01

TLT was a five-year research project studying how secondary science and mathematics teachers learn to use an electronic "classroom response system" to implement a specific pedagogical approach called Technology-Enhanced Formative Assessment (TEFA). Background: Classroom response systems (CRSs) were technology that helped an instructor poll students' responses to a question, displaying a graphical chart of the class' aggregated answers. These systems, although simple in concept, could have a beneficial and even transformative effect on instruction.* TEFA was a pedagogy that Gerace, Leonard, Beatty, and colleagues had developed over 15 years to support effective science teaching with a CRS. Teachers at the university, high school, and middle school levels had succeeded with TEFA, but mastering it was often challenging and time-consuming, and took extensive support.

Objectives: The TLT project was designed with three goals: (1) to better understand teacher learning of CRS technology and TEFA, and consequent changes to their practice; (2) to better understand effective and efficient methods of teacher professional development in TEFA; and (3) to develop tools and techniques for the evaluation of teachers' TEFA mastery, of suitable design and quality for use in a controlled, randomized study of the effects of TEFA on student learning.

Professional Development: Project staff had conducted (and continue to conduct) intensive, sustained, on-site professional development (PD) programs for 40 middle- and high-school science and math teachers at six schools in three Western Massachusetts school districts. PD focuses on use of CRS technology (provided to the teachers by the project), practice of the TEFA pedagogy, development of supporting curriculum elements, and attendant teaching issues. It began with a four-day summer workshop, continued with a year of weekly or biweekly after-school meetings, and ended with one or two years of monthly after-school meetings. The PD was itself conducted according to the TEFA approach.

Research: The project used a longitudinal, delayed-intervention design. Data was collected via several channels, including interviews and regular questionnaires for participating teachers, surveys for their students, videotaping of classes being taught, and video- and audio-taping of professional development meetings. Analysis was mixed-methods, focused on detailed, heavily triangulated case studies and cross-case analysis. Significant new instrumentation had been developed, tested, and refined during the course of the project.

Outcomes: Project staff had compiled detailed case studies of four participants, with partial profiles of several others, and had developed an initial model of teacher learning and pedagogical transformation called the "model for the co-evolution of teacher and pedagogy." A TEFA PD program had been developed and iteratively improved, and the TEFA pedagogy had been more clearly articulated, defended, elaborated, and disseminated. Experiences, methods, and preliminary results had been presented at several different professional conferences.

* For references, contact Ian Beatty (idbeatty@uncg.edu).

The TLT project is funded primarily by grant TPC-0456124 from the National Science Foundation. Any opinions, findings, conclusions, and recommendations expressed here or in other project publications are those of the principal investigators and do not necessarily reflect the views of the NSF.

Additional project support has been provided by InterWrite Learning (now owned by eInstruction), makers of the PRS-RF classroom response system.