Assessment and Technology

John Rueter

Introduction

Technology can change the way we communicate with students and interact with data. Because of its flexibility and power, information technology can also be used to substitute for previous techniques and strategies. It is easy to use email in the way that you might have used a phone or to use a database as a gradesheet. These are not necessarily undesireable, in fact these uses of technology might be crucial intermediate steps for faculty. Learning how to use Excel to construct a gradesheet is probably a necessary step that we all went through that gave us the insight and confidence that we could use databases for more than just keeping scores. This paper provides several examples of how technology can support assessment of student learning.

The first questions are, "what do we want to be able to communicate?" and "what do we want to assess?" These are important questions because technology can help us communicate our expectations to students in their cycle of learning activities and allow us to collect assessment data about how the students performed on the assessments and evaluative assignments (graded problem sets and quizzes). I have embedded assessments into the flow of the course, students are asked questions through a web page form and I collect there answers, create responses and answer each of them within 24 hours of the assessment. I treat these assessment exercises as transactions between each student and myself, where the quality of my answer is determined by the effort (learning quality) of their answer. A student who gets the wrong answer but shows thought and effort will get a well crafted corrective statement. Another student who scratches off some casual reply would probably get much less attention and in fact might get my dismissive answer "Your answer was not carefully enough worded for me to respond. Please put more effort into writing a concise answer." In addition, I make an effort to provide useful feedback to students who get the answer correct while reminding them that their answer was totally acceptable. These good students might get an aswer such as "Your answer was very good and covered all the points required. You might consider how this simple question relates to the more sophisticated social context."

These assessment exercises are built into the flow of course work. Several of the courses that I have used these techniques in are "hybrid" courses that are half on-line and half face-to-face. For example, the course might meet for 1.5 hours on a Thursday evening and there are other activities during the week that prepare students for the Thursday section. I have given assessments of vocabulary words that were due by Monday at midnight and provide a response back to the all students (who made the deadline) by Tuesday at 5 PM. This gives them time to work on the next assignment (due in class) after they have seen how expect them to use the vocabulary. These assessents are simple and straightforward to respond to (see below for details on the Multemail process), but they fit into the learning model and use my skills. The learning model for these courses is based on Bloom's taxonomy and explicitly states that they need to learn the vocabulary and use that vocabulary in their development of the concepts and then they need to apply these concepts to other problems. Thus the vocabulary is a building block for the rest of their work in the course. Vocabulary words for the assessment are chosen that might be abiguous with there standard English meaning (example: hypothesis and theory are used differently in the context of the scientific method than they are in casual discussions). My roles in this assessment activity are to design the activity to fit in with the flow of their learning and to provide timely feedback. An important skill that I have, as all faculty do, is to recognize when an answer is weak or potentially indicative of a misconception. This is an important role for faculty that can't be provided by the book or even context sensitive help files, and it is considered to be a cruical process in education (Laurillard ****). These simple assessment activities are run on a technology platform (Multemail) that allows me to keep my comments to students and thus provides a list of these misconceptions or unfounded assumptions that students make. This could be as important assessmment data as following individual student performance.

Technology can also allow the student to have information about their progress in the course. Of course, this is almost the same as handing back papers with grades on them, but I have used this in a much more structured manner that helps the students see there progress compared what is expected. Several detailed examples of these will be provided below. The basic idea is to store student performance data in a database that is accessible to students. I constructed several courses that used variations of these themes. First, the course must be designed with explicit lists of specific learning objectives and how these learning objectives are to be met in student assignments or evidence shown in quizzes. The time line of specific learning objectives and how previous specific learning objectives are related to subsequent ones is important in the structure of the course. For example, students might be expected to do X and Y on problem sets and on the quiz they are expected to take part of the solution to X and part of the solution to Y and synthesize a new answer. It is very informative to me to make sure they can do X and Y before I interpret how they did on the X-Y synthesis question. By making the performance on specific learning objectives available to students on the web, it becomes their individual responsibility to correct deficiencies before the quiz. This allows me to fundamentally change the way students interact with the data on their learning.

Examples of course assessmentst that use technology

The examples described here have been described elsewhere on this site.

1. Multemail

A process for grading assessment or short quiz answers that uses multiple tools (web form pages, FTP, Excel, and a PERL script "Multemail"). The process collects all the answers from students and then creates and revises answers to students' answers as the faculty works through the questions. For example, as the faculty reads and grades the first several paper they make notes that would be a good answer to that student. As they read more student answers, the teacher creates more responses and modifies previous answers. This process leads to a categorization of student answers and optimized answers. All the answers are automatically updated, the first student graded gets the same answer as subsequent students that made similar mistakes. In large classes, the teacher may choose to work through 10 or 20% of the student answers, determine categories and streamline the grading process by putting the subsequent 90 to 80% (respectively) student answers into the categories.

This process provides three important features: 1) allows for time savings in grading and returning assessment information to students, 2) provides individualized answers to students that may actually be more valuable because the teacher has identifies a type of misconception (rather than a totally personal response that would address only that student's answer), and 3) a record of the misconceptions and which students had evidence of these misconceptions. All of these for the same time (or less) that it would take to respond to the individual emails.

The considered response of faculty to student emails has another value; by allowing time and thought to go into these email responses they are more inciteful and corrective than an immediate response might be. Besides making the instructor look smarter (on the internet no one knows that you're not Disraeli) it provides more value to the student.

2. Concept maps and scaffolding for problem set to quiz answers

This process was described in both part 1 and part 2 of this site.

add in

Persistant and Transparent Assessment

Different levels of assesment

We need to be able to assess student learning once and use that information multiple times and in multiple contexts. There are different levels of assessment of student learning that range from classroom assessment techniques to programmatic assessment for accreditation purposes (Table 1).

Table 1: Different levels of assessment
name	short description	time scale
classroom	short activities that are embedded in the learning activities and that provide immediate feedback	minutes
action research	assessments that are targetted at finding out a simple detail about student learning that will be used by the instructor to modify the	days
within course	multiple points of assessment being used to improve a section of the course or entire course	weeks to terms
course	total performance in a course that can be used to modify subsequent offerings of that course	terms to years
program	collection of information for student performance on multiple courses in the curriculum	1 to 5 years
alumni	collection of indicators from the alumni of a program	2 to 40 years

The goal of persistant assessment is to collect information that helps students learn but do that in such a way that information can be useful to the longer term assessment needs. From the faculty view, this is "persistant"; from the administors' view of this data it is "transparent".

Investment in quality improvement

If we look at examples of quality improvement and data collection from other industries we realize how expensive this process can be, how crucial it can be and how adaptible it must be. A successful quality improvement program will result in a moving target for quality. Chasing the red queen, each improvement becomes the new standard procedure that must be improved. Universiteis have different resources than other industries, but it is hard to imagine that we take real improvement as seriously as some industries that are so competitive that they have to improve or perish. Some estimates put the costs of factory revision and retooling at around 5% of the GNP. Some estimates of training time for employees are in the 10-15% of their time. If we are going to be serious about assessment serving quality improvement, we need to be realistic about how much time and money are being spent in our institutions toward this goal and adjust our efforts so that we get the maximum benefit for that expenditure. It is foolhardy to oversell what we can do. One potential win-win solution is to focus on learning and evaluation activities that are already being used and use our limited resources to get the most out of these activities.

Examples of persistant assessment

effective grading - but need to identify exam questions and grade drift

in class assessments and classroom research - link to other pre-requisite courses

look for misconceptions

maximum immediate effect

sacrifice individual data for course data

past student performance in courses based on prerequisites and sequence

data mining activity

Conclusions

Creating and revising courses that serve learning and assessment needs will face two interrelated challenges: faculty development and new data interface tools. In almost all universities, faculty are responsible for creating the courses and modifying them with time. These are time consuming activities that could be enhanced if faculty understood the overall task better and had appropriate tools. Faculty need to understand the structure of the information in their discipline, how that relates to their course, the curriculum and assessment techniques. Just as other industries use simulations in training and decision making, I think that simulations of student learning could greatly enhance our ability to make curricular decisions. Some of thse simulation tools are available at the level of course and curriculum management (see for example DARS for performing "what if" scenarios for enrollement). I think that the first step in trying to simulate student learning is to be explicit about our models for how students learn. If we can start with descriptive models for what helps and what hinders student improvement, this would be a step toward decision making simulations that would help highlight particular information that we need to move forward. My view of the current assessment plans are that they are hopelessly mired in the mindset of unbounded rationality, the more data the better, so how could it hurt to collect more information? My answer is that it takes effort to collect and sort all of that information and I think we should spend that effort on making a few crucial decisions based on simple and robust models. I would ask, what are the top several factors that lead to student success and how can we improve those factors for our students.

The other part of this challenge is to have appropriate tools for examing the data that we choose to collect. Our current desktop metaphor and application metaphors are good for office and business but I don't think they are the best for following and understanding learning. The spreadsheet metaphor is good for performing repetitive calcuations but may not be the best for following the changes in performance of students unless all you want is points, grade and rank. Similarly, databases are powerful tools and are good at tracking inventory and querying the data, however it may not be the best for representing the concepts, relationships between the concepts and student performance on these. Steve Johnson (1997) addresses exactly this issue in his discussion of interfaces and how we will need a new visual language. He says that we need are "metaforms", data making sense of other data. These metaforms "prosper at the threhhold points fher the signals degenerate into noise" and the metaforms are "messages that evolves faster than the medium". When you see the way instructors are using spreadsheets and databases to track and attempt to understand student learning, the high degree of innovation and improvisation that is required is a form of these metaforms evolving past the orignial metaphor of the tools. As they say, "education is not exactly rocket science", and they're right it is more difficult to try to understand the complexity of 40 independent minds in a classroom. Table 2 lists some specific interfaces that have been developed for other particular tasks.The current interfaces (applications) that we use as faculty are too simple for the task at hand. We need more sophisticated applications that can look at student learning outcomes in new ways.

Table 2. Applications and specific tasks that might resemble assesment interface.
Application	Short description
flow diagrams, STELLA	follow path and multiple paths of fluids, electrons, etc.
LISERL	statistical tool for causal analysis
MIDI	using sound patterns to identify temporal patterns, harmonies and disharmonies
fractal generators	look for scale dependence and independence of relationships
GIS	geographic information systems with many layers for different data and/or processes
CAD/CAM	simulation for building parts that allows virtual trials of fit and performance before manufacturing
software development environments	use of visual metaphors to help design and collaborate on the construction of complicated projects includes support for object reuse

An example of such a tool or capacity that I am developing is a dataset of frequency and rank of concepts as they are presented in the book and in lecture. In the book the frequency vs. concept follows an inverse power law with a shallower slope for the most common 20 to 25 terms followed by a high slope for rarer terms. This means that there are some terms that are used very frequently as they describe the domain of environmental science and then more specific terms are used in relationship to those terms. The tool that I am developing will look at the overall frequencies and rank but allow me to look at student performances on associating common and rare terms.

Figure 1. An example relationship between frequency and rank of terms used in the book and in class. Notice that the "concept veiwer" shows that there are some areas that were emphasized more in class than in the book.
Figure 2. Frequency and rank data with a "student viewer" that shows what words an individual student chose to associate. The pull down menu on the viewer allows you to select different students. The instructor can also program in links that are indicative of misconceptions.
Figure 3. The average association window that describes the performance of the entire class or performance of sub-groups of the class. It would be possible to view sub-groups by rank in class or other indicators. In this example, it shows that 90% of the students made the connection between terms that had similar frequency but fewer students associated the more frequent to the rarer term.

This application is offered as an example of how when assessment of learning is built on an explicit learning model then he results are much more easily interpreted. The data for the students in this course could be compared to subsequent terms because it is self-consistent and complete. These characteristics of this data is crucial for persistance.