Items will be posted with the newest at the top.
Links open in new windows. Link to documents we might use.

16 January 2008

when we do rubrics or computer-based testing: check their skills with databases, spreadsheets, or whatever else they use for grading their students

leftovers from Bill's original SG for the Oregonian reflection article, to help us plan presentation of rubrics:

1. on-time (handing it in after 1 week, as assigned, gets a 4, not a 6 - I can explain why when we next talk, if you want, and we certainly should make the point in class - both to forefend howls and to make a point about assessments and to what degree timeliness is to be emphasized); percent of total grade: around 20%, not just because this is 1 of 5 sub-rubrics, but because that's about what it's worth to get our message across (=do assignments promptly, because that's when they have their best effect on progress in the course, because that leaves time for revision, and because that makes it easier for you and me), but without so heavy a value that a late assignment that is otherwise strong cannot still get a high grade.

(I may use my proposal for the ACTFL conference, which I wrote under time pressure last evening, as an example of what the profession demands of us - in timeliness, but also - see below - content and exposition.)

2. Base presentation of the recollected test (=before reading the article) - a) description - important, because we need to be able to understand the other parts of the assignment in relation to the base experience; b) subjective matters. 20% of total grade, with (a) and (b) counting equally toward the 20%. (I assume you can use a database, spreadsheet, or grading program that allows you to write formulas to calculate grades without doing all the math yourself. Say! Shouldn't being able to do that - if necessary, learning how - be part of an assessment course?)

For the 6 and 5, the assignment has not only to do what we specified, but also go beyond it. (This, too, will take some explaining class, and there may be some gripes, but it is absolutely consonant with the meaning of "exemplary", with the PSU emphasis on critical thinking, and with what it means to be outstanding in a profession: we need teachers who will (and want to be) leaders, and who have what it takes to be such. (Same goes for performance on the other features.)

3. Revisitation after reading - effect of the article on view of the test experience (student can say the article had no effect, but had better be able to explain why). 20%

4. Revisitation after reading - application of article to own world of language learning and teaching. 20%

(So the content of the reflection is worth 40%, twice as much as the base recollection.)

5. Expository skills (mechanics, rhetoric, etc.). 20% Exposition is important, of course - but of course everybody says that. When we present the SG I'll want to make it clear how I view exposition in functional-professional terms, rather than just specifying a style manual or stating how many points will be deducted for whatever. We can frame this assignment as something that a teacher might write for, say, the state/regional newsletter of a professional society. Not a scholarly article, but rather a personal but not merely subjective, casually written commentary, and in neither case for a national/ international-level publication. Those newsletters are relaxed about the details of format, but a contribution that is poorly thought out, poorly structured, poorly phrased, or - heavens! language teachers are so picky about language! - subliterate will either go unpublished, or else - worse for the contributor! - will be greeted with chuckles and sighs. In comparison to content, I'm less concerned about spelling (to a point) or even small-scale grammar (to a slight extent, anyway), because a little editing can easily make publishable a piece that contains good thought, where no amount of editing can turn garbage content into something publishable, no matter how perfect the author's mechanics.

Here's the language of factor 5 from the SG for the similar 2005 activity:

6 The reflection could be published in a professional medium (or used as a model in the course). No words are wasted. If written in native language, contains no embarrassing errors. If written in non-native language, could easily be edited by a native speaker.

4 Quality is graduate-student level, but undistinguished. Contains flaws in both major and minor features of language. Half of the following: more than one instance of verbosity that would need thoughtful cutting; one structure that is substandard; one word that is clearly chosen incorrectly; one spelling error that is not just a typo

2 Quality is even below undergraduate level. So short or so long that much would have to be (re)written to reach 4. More than one substandard structure (or repetition of one substandard structure. More than one word that is clearly chosen incorrectly. More than one spelling error that is not just a typo.

top of page

9 January 2008

replace "backward engineering" by in-class tutorials: AL - specs and item composition; FL - OPI training

need to arrange extra classrooms and have breakout sessions videotaped

When we present standards, stage breakout sessions: AL for IELP, etc.; FL for National, State, Zertifikat, etc. But be sure each group has a general idea of the other group's situation

Notes on the assignments:

evaluate own proficiency: use both ACTFL and EU scales; if necessary, evaluate someone else's proficiency (on the general principle of: if you encounter obstacles to doing an activity, let us know)

main project can be Atkinson School (assessment, not just in-class participation); PASS moderation; grant idea; research idea)

class observation: 2 visits (Atkinson, PSU, comm college, K-12 all OK); maybe read Donato (FLA) article

When we get to week 5, testing of listening and reading, we'll need to raise consciousness of FLL people about LITERATURE; this will then open up issues about program assessment, such as why 400-level literature courses are Death Valley for students who start the languages in first year of college; don't know what the corresponding issue is in AL- or maybe this is the week to take up their own hot-potato topic of World EnglisheS

top of page

3 January 2008

Hughes TOC by chapter:

1 & 2 basics and approaches; 3 Kinds of Tests and Testing; 4 Validity; 5 Reliability; 6?; 7 Stages of test Development; 8 Common testing techniques; 9 testing Writing; 10 Testing Speaking; 11 Testing reading; 12 Testing Listening; 13 Testing Grammar and Vocabulary; 14 Testing overall ability; 15 Tests for young learners; 16 Test Administration; Appendix 1: Statistical Analysis; Appendix 2: Item Banking

Major (weekly) topics, activities, readings, related somewhat to SK's excel calendar:

testing culture?

Assignments: article reflection (A1); pseudo-OPI (FL) and something equivalent for AL; ??backward-engineer an existing test??; write a scoring guide; evaluate own proficiency (A2); evaluate own professional development (start portfolio); search of prof lit (or as grad student assignment, replace here by creating an additional test); classroom observation (?); further project exploration (Atkinson, grant idea); group activity?

several reflective pieces, apart from the above or in conjunction with them

write a scoring guide for the course? Have groups produce SGs for each activity after we have introduced rubrics?

Grad students: create an additional test? or replace with search of professional lit?

top of page

14 December

to do: poster; office spaces and hours

goals/ objectives:

just checking: get beyond the "default" level of assessment (FL: GT & maybe OPI; ESL: TOEFL & ??)

learn the "culture" (including the testing culture) of the neighboring language area (FLL - ESL)

understand language and curricular standards at the various levels

understand the clients (students, but also XYZ)

learn the "guts" of a major test in own language area (FL: OPI; ESL: TOEFL)

understand rubrics

aware of curricular issues (general direction of own field; alignment / articulation issues)

know where to find tests

self-assessment

understand computerized testing: what's there, strengths and weaknesses

understand kinds of validity and reliability

recognize the statistical concepts

understand international assessments (CEDEFOP, DIALONG)

backwash

test specs

assess learning materials

nodding familiarity with assessment of programs and institutions (ex: PSU internationalization, diversity, critical thinking)

foot-on-bottom-rung familiarity with how research is conducted (mandated, financed)

major assignments:

article search (FLA/TQ; database)

conduct a major assessment (FL: OPI; ESL: ??)

construct a skills test

critique / reverse engineer a "default" test

construct a test (specs, item-writing, reflective piece, administer)

create an activity (not necessarily a "test") and a rubric for it; administer, reflect

large group presentation

grad students: presentation paper

write grant proposal

weekly reading reflections

self-evaluation for portfolio

problems:

how to maintain parity (FLs does OPI, what do ESLs do - some sort of OPI, or investigate the various OTs out there - does parity mean they do same type of student activity, or acquire the equivalent knowledge / tool?)

misc: need to sensitize the "G" topic; ditto for ESL/FL difference

12 December

Week 1: have students compare their backgrounds and do simple checkoff (intake survery) of what they have / lack in training and experience (return to this later with systematic self-assessment). Have them discuss various non-language assessments they've encountered.

Assignment: rubric for traditional literature term paper;

activity: where does language learning land on the cognitive domain of Bloom's taxonomy (see Moran, J. J. "Assessing Adult Learning", pp. 42-3

activity: how well does Moran's description of how most portfolios are rated (progress, versatility, creativity, self-directioin) fit language learning (as it is commonly encountered, as it should be)? (database 339, "Mastering Performance Portfolios")

4 December

The FLL people should acquire the vocabulary of language pedagogy in both English and their specialty language(s). Could have them read an article in that language and compile a bilingual list of key term. That would be useful for some but not all AL people. Could we come up with an equivalent activity for the rest?

3 December

Schedule / Assignments pages (FLL-only) versions of course: 2005 Fall; 2004 Winter

from my notes of our November discussions:

major topics:

• fundamental concepts of testing (kinds, purposes, clients / communities, determinations of quality)

• national / international proficiency standards and definitions of competence

• commonly used assessments, including TOEFL and ACTFL OPI

• state standards and assessment tools, and their relation to curriculum and content

• creating test specifications

• creating rubrics / scoring guides

• computer-assisted testing: IBT and WebCAPE

• assessment and statistics

• assessing resources (textbooks, etc.)

• curriculum, program, institutional and system assessment, including departmental assessment at PSU, learner-outcomes assessment at PSU and in OUS

• teacher self-assessment (professional progress)

• financing assessment and research (familiarization with grants and projects

possible major assignments:

hunt through databases / journals to see the scholarship of language-related assessment in the last ten years (2 assignments: learn the basics of the field; learn database use)

assessment project (FLL people: practice OPI; AL people: ??)

regular reflections on topics: ??

possible guest speakers: Danelle Stevens (rubrics, PSU Institutional Assessment Council); Mark Endsley (OUS prek-20 assessment & articulation); textbook company rep about careers; Mary Bastiani (Portland Public Schools - directed two-way Spanish/English program at Beech Elementary, developed JPN/SPAN distance learning, with large-scale assessment, co-directed $50K grant with me about Styles- and Strategies-Based Learning, developer of SLIP assessment, much else); Nila Friedberg and Linda Godson from FL Heritage Learning Institute, which has done recent learner attitudes assessment; Kektchison; Labissiere; PDX translation agency

••