Sherman Dorn

Peer reviewing, redux and redux

I regularly attempt some form of student peer review process for short writing assignments and am regularly frustrated with the inadequacy of the process--inevitably someone falls down on the job either in terms of participating in peer review or submitting draft work or providing advice that is counterproductive... and leaves an awkward gap and a reasonable question by students: "Do I follow my classmates' advice if I'm not sure it's wise?" That's the reaction that gets me to suspend peer reviewing for a semester or two and restructure. Yet I keep returning to it because there is value in having students look at a range of actual work and think about their own work in comparison.

In odd moments over the past few weeks I've been thinking about a different structure, focused less on providing written feedback and more on just comparing a sample of assigned writing. To give the exercise some meaning (and motivation), I've been thinking about incorporating student judgment into the grades for the assignments. (The idea is that if student ratings or rankings make a difference to peers, they will take the process more seriously.)

That's nice in theory, but how do you do that in practice in a way that is fair and does not provide perverse incentives (with students' having a reason to rate peers either high or low in some fashion)? (Please don't tell me not to grade: I'm not at UC Santa Cruz or New College!) That took me in some interesting directions, through some psychophysics literature (Thurstone's "law of comparative judgments"), some folks named Bradley, Terry, and Mallow (and Kenneth Arrow's preference/voting paradox), and eventually into BCS football rankings and the international chess fideration ranking system. Because I did some of my reading late at night, and some of this is far out of my comfort zone, my brain was hurting at several points, but I think I have stumbled onto something that is simple enough to administer, consistent with how I'd like to look at the inclusion of student judgment, and gives students a structure that is easy to respond to and gives them a reason to be honest in expressing judgments.

Again, that's in theory. It'll probably be the end of 2010 before I know whether I'm completely bonkers or if it's workable.

College visits: some good, bad, and ugly ideas

Valerie Strauss's blog entry yesterday discusses the standard advice for high school students' visiting colleges and then provides links to the College Board and ACT college-visiting pages (which are surprisingly slim). Again, Strauss is speaking directly to parents -- "Be sure to contact the school in advance... If you child can sit in..." -- which has good programmatic advice and horrid parenting advice.  The high school student should be doing the contacting, maybe, if it's her or his college experience at question? But aside from that quibble, here's my take as a parent of a high school student waiting for admissions decisions:

Good: Strauss's advice for students who can't travel long distances to visit local colleges like the ones they're interested in in some way, to get a sense of what it would be like to attend a metropolitan university, a private liberal-arts college, a public regional state university, etc. That way, questions students have can be focused on the issues most likely to be relevant for a type of institution.

Good: Strauss's advice to visit classes. Wrong: her assertion that it's  "probably best [to leave that] for schools at the top of the desired list." As a parent and citizen, I think high school students should see what it's like to be in a college class as often as possible, especially in discussion classes that are never going to be on iTunes. My not-so-hidden agenda in pushing that with my daughter was to make sure that by the time she went to college, she'd know she belonged. 

Good: Strauss's advice to read student publications, and the College Board's advice to scan campus bulletin boards. I'd expand that: during tours of dorms and classroom/office buildings, look to see what's taped, plastered, and otherwise attached to the walls. 

Good: the College Board's advice to browse in the college bookstore. My daughter was interested to see if the general-books sections had her favorite authors. I wanted to know what was on the shelves for classes.

Good: Strauss's point that visits when classes are out of session or students are preparing for or taking exams is ... uh, not so wise. She takes for granted that the only way that you can visit colleges without taking an unexcused absence is during spring break. Not true! Many school districts will assign excused absences for college visits, and my local district will let students take up to three days per school year as official school business for college visits.

Good: eat in the dining halls. Caveat: this is assuming a residential college or university. But it's definitely a good idea, and many admissions offices will let one parent and the student eat lunch without paying (one reason why you head to the admissions office first).

Bad -- in fact, pretty horrible: The College Board's advice to "Try to see a dorm that you didn't see on the tour." This is a fine way to get arrested or warned off campus by the campus police or security. Dorms should not be accessible to strangers, such as high school students or their parents. Of course, if you try to get into a dorm that's not on the tour and no one stops you, that tells you something about the place, too, but I suspect you can find that out simply by observing whether the tour guide has to unlock a dorm entrance.

Not mentioned by anyone: how to ask questions of current students. The ACT page has several suggested questions, but there's something very important to keep in mind, both in asking questions of campus tour guides and other students: you can ask about the college in general, about the student's experience, and about the students' friends and classmates. Campus tour guides are trained to talk about the campus in general, but if you're a high school student, you need to know about real experiences. Best suggestion: ask about the experiences of friends and classmates. That's a healthy compromise between the generic "students here" and the privacy-invading "so tell me why you're really about to transfer at the end of the term." 

Not mentioned by anyone: if you've been accepted to a place and you don't think you can travel to visit it, ask the admissions office to put you (the high school student) in touch with a few current students who are willing to talk to you about the college. If you're interested in specific majors, ask for people majoring in those  (or related) subjects. See if it's possible to use Skype rather than a cell phone because if it's a Skype video call, you might get the benefit of talking not only with the current student but also with whoever's in the dorm room at the time. And here is where visiting a similar type of college locally can help you figure out the crucial questions to ask.

Another stupid article on "the dating scene" in college

Some of the clues that the latest article on the "dating scence" in colleges with 60% female enrollment was written by a reporter with an axe to grind and a preset angle at which to grind:

• The featured photograph from a university with 60% female enrollment (a) is of college seniors (or I hope they're seniors) in a bar, (b) is of an all-white group of students, (c) has six women and one man, (d) has no older students.
• Every photograph features white students.
• All the women interviewed for the story appear to be members of sororities.
• One of the interviewees is a former student who happens to be hanging out in a bar near campus. (So why is he representative? Why didn't the reporter step a few minutes away from a bar?)
• The focus is entirely at a flagship public university.
• There are no older students interviewed for the story.

Since the primary world of colleges is at the regional state university and community-college level, maybe we should skip the flagship campuses and look at the statistics of an institution such as Miami-Dade College. MDC has more than 150,000 students enrolled, and while 60% of them are women, only about 35% are right out of high school (under 21). About two-thirds are attending MDC on a part-time basis, and while MDC is now a four-year institution, I don't think there are any dorms, so every one of those students are commuters and live somewhere in the Miami area. In other words, the dating scene for straight, gay, or bisexual students is where they live as well as on campus. That's the reality for the majority of college students in the United States, not the preppy picture that the New York Times reporter and photographer portrayed.

But if you want to look at residential colleges and universities, maybe a little reality should intrude: the average age at civil marriage for women in the United States has moved back up to the mid-20s, where it has been historically for well over a century, with the exception of the immediate postwar years. College students' meeting and marrying in college is common enough but not dominant. 

And the history of colleges is not one filled with demographic "balance" in some hypothetical way. For many years, the ranks of elite residential institutions were filled with single-sex colleges and universities with single-sex undergraduate colleges, and the students in those colleges and universities had to go off-campus for a hetereosexual dating scene. And in the first decade after World War II, the GI Bill pushed enrollment in public universities in the other direction, towards majority male enrollment. If you can find more than a decade or two when the dominant demographic profiles of residential colleges, community colleges, and public universities were all fairly evenly split by gender, I'd be surprised. My guess is that maybe a decade or two will fit with the peak of the Baby Boom through the mid-1980s... when people worried about the social consequences of the sexual revolution. As one of Gilda Radner's characters would say, if it's not one thing, it's another... so let's stop obsessing with the on-campus dating opportunities of college students.

Sloppy journo skewered; readers await fix

Reporting is a hard job. These days, reporters are being asked to cover more subjects in less time with an even smaller news hole for newspapers that are losing money, laying off colleagues, and may be out of business within a matter of months. Even in good times, reporters knew that errors were going to be read by thousands of subscribers and that even if they worked twice as many hours in a day (usually impossible), they'd never catch all factual goofs or grammatical mistakes, or never quote enough interviewees to satisfy all readers. Great beat reporters are inherently improv artists.

Having said that, I know it should not be too much of a surprise that even reporters with solid reputations such as Ed Week's Debra Viadero sometimes get caught taking shortcuts. Thus far, no response from Viadero, but it's another part of journalism (and a reflection of the craft) to print corrections publicly. So let's wait and see how Ed Week acknowledges error.

Grading the "Grades" reports

I'm back from Toronto today--had a great time talking with Canadian faculty, had my head chewed off in a thoroughly polite, Canadian way for one bone-headed error I made in discussion, survived subzero temperatures for a few mornings, and completely failed to enter the Hockey Hall of Fame building--and back in Florida the temperatures are a bit lower-than-average for this time of year but discussion of the Ed Week Quality Counts "grades" given Florida is apparently heating up. So maybe I need to revisit my idea from last summer of grading the grading reports. Last June, I pointed out that professional grading practices generally provide scoring criteria in advance, so that those who are being evaluated will have a chance to... you know... meet the standards. Let me list all of the facets on which I think one can grade such "grade reports" of states and the like:

• Purpose. Is there a clear public rationale for issuing such a report? How broad or narrow is the public purpose?
• Scoring criteria. Described in June.
• Description of sources and analysis. How systematic is the collection of source material (as opposed to anecdotal or convenience sampling)? Is there a clear chain described from collection of data to the application of labels? Is there a discussion of relevant caveats/alternatives?
• Robustness of sources. Are the sources publicly verifiable or replicable? Are they subject to gaming, falsifiability, or manipulation?
• Relevance of sources. Is the material relevant to the criteria, and does the "grade report" use the most relevant obtainable information? Is the source information analyzed appropriately to warrant the application of the grade labels?
• Sponsorship. Are funding sources and potential related interests stated clearly? Is there a separation between the real or likely perceived material interests of sponsors, on the one hand, and editorial control of the project?

It strikes me on impression that different types of periodic "grading" exercises have different types of weaknesses. An advocacy organization whose reports rely on anecdotal evidence and give higher grades to states that are more extreme towards its position might receive lower grades on description of sources and analyses and sponsorship than in other categories. A news organization that makes millions of dollars by selling a volume ranking colleges and universities using reputational surveys of institution heads and data on institutional wealth is likely to receive low grades on public purpose, robustness of sources, and relevance of sources. A news organization that ranks states on categories that change every year using no apparent criteria that also change every year is likely to receive its lowest grades in the area of scoring criteria and description of sources and analyses.

As a faculty member who has assigned thousands of grades to students, where the grades affect student progress towards degrees and financial-aid eligibility, I know from experience that the process of grading is imperfect and in my field depends on judgment rather than objective cut-and-dried methods. That's why I state criteria as early as I can, display model work from prior semesters if possible (with the permission of their creators), answer questions about assignments, look at drafts, structure revision opportunities into a number of courses, and always let students correct me when they document that I have recorded individual assignment grades incorrectly.  I know from student complaints about grading in general that they hate being judged on criteria they feel the evaluator keeps secret, or that is designed to make the evaluator look good, or that serves some other purpose that isn't for the general purposes grading is accepted by at least some to serve. In other words, if you're going to assign grades, especially if the clear intent is to shame certain entities into changing, you need to take at least a few minutes of care to address common-sense ethical expectations. I'd have far more patience with these publicity-seeking exercises if there were more care evident in the process.

Why the "college hunt" genre is unrepresentative, and the shame of the College Board Profile

This morning's blog entry by Valerie Strauss is typical of the genre: a perspective on what it's like to apply to a number of selective colleges and universities and hunt for financial aid. And it's all wrong, both from a policy perspective and (I'd argue) even a hypercompetitive parents' perspective.

Policy perspective: the colleges most students attend are not very selective. Even for the ones that don't accept all applicants, most accept the majority of applicants (including most public universities). And even in the world of "very" selective institutions, you might be surprised. Sure, both Harvard and Stanford will reject more than 90% of their applicants this year, but most of the "very" selective private liberal arts colleges accept 25% or more of applicants... and we're at the peak of the baby boom echo, so it's only going to head up from here. (Math problem: If you're a high school senior and apply to colleges where you have a 50% probability of being accepted, and the decisions of each college are random and independent, how many do you need to apply to to have at least a 98% chance of being accepted into at least one?)

So the problem is generally not getting accepted into one college but being able to pay for it and being able to take all the classes you need and succeed at them. My daughter is applying to a few places where the tuition/board combination is high enough where some institutional aid would be very nice, and last night we completed the FAFSA, which is one half of the financial-aid paperwork for one of her desirable colleges. (I'll have more to say about the other half later.) The administration's promise on a simplified FAFSA has been fulfilled, at least from my experience: you don't need a CPA to fill it out, especially for families who are eligible for Pell grants and state assistance. The administration's proposal for a 10% cap on income you owe on college loans would be another step, and a definite improvement on the new income-based repayment option. Given the gap between Pell grants and tuition at a number of public universities, pushing on income-based repayment may be more valuable in the long run than expanding Pell grants.

Where Strauss is correct from a public perspective is the gap between the time high school counselors can spend shepherding students through the admissions process and the reality of the need. I'm thinking here primarily of high school students who would be first-generation college students. There aren't too many guidelines for a ninth-grader to keep in mind, but they're probably not repeated often enough: get your act together now to make sure your first semester grades are at least a mix of Cs and Bs, and they need to head up from there; read more than what's required; go as far in math as you can; take SATs or ACTs in your junior year; tell your parents to put their financial information in one place starting early fall of senior year; expand your college possibilities in one dimension from what you're being told by those around you. I suppose there are others that high school counselors use, but for the barebones, students whose parents never attended college can get into a fine public university following this.

If there's something that worries me apart from the high school curriculum and funding for poor students, it's the narrow way most high school students think about where and how to look for colleges, and the way that adults encourage that narrowness in part from their experiences or perceptions or because of tacit knowledge. There are sometimes circumstances that restrict students--those who need state assistance will be staying in-state, and often first-generation college students (especially young women) live at home while attending classes at a public university, at least for a year or two. (I know of one very large community college where faculty get the benefit of teaching incredibly talented first-generation students because their parents wouldn't let the students move away for a few years.) High school students can be creative in working with family preferences--Orlando high school students often prefer the University of South Florida (here in the Tampa area) and Tampa area students often prefer the University of Central Florida (Orlando) as a "far enough away from home so I'm not visited by my mom twice a week, but close enough to drive home on weekends" solution. But that's like chain migration: if you hear about an option from someone you know, you can use it.

What about the options you don't personally know? I've had some conversations with teenagers and parents in the past year or two where presumptions have become stereotypes and blinders. One parent completely dismissed a nationally-known public liberal-arts college because she knew some students with learning disabilities who saw that as a friendly place to attend... so it must not be good enough (i.e., prestigious). A student who is one of the most hard-working teenagers I have ever known and interested in engineering schools didn't know the difference between tuition-dependent private schools and those with endowments and substantial institutional aid. She was thinking very hopefully on an engineering school within driving distance that is tuition-dependent and where there was no way that she could get aid (and thus attend). She hadn't thought of CalTech at all, though it's well off and where she might get a boost because of the dominance of men in their undergraduate enrollment. Another student who moved to the U.S. four years ago was disappointed in her board scores and thought colleges wouldn't want her. She's another incredibly hard-working student, one who admissions officers would drool over in reality. For the students in these cases, I'm not worried because it didn't take much to persuade them or their parents to think a bit more broadly (and optimistically). For the millions of talented high school students I can't persuade personally to think a little more broadly about colleges, I worry about the mental shortcuts we take when looking for colleges. It's an understandable but sad statement about our country when some of the most effective recruitment of college students is done through Saturday television broadcasts in the fall.

Private perspective: As I wrote above, the FAFSA is one of the pieces for institutional aid for a college my daughter is keenly interested in. The other is the College Board Profile. Last night, I printed out their 19-page worksheet and filled in answers for the several-hundred questions about parental income and assets so my daughter can enter the data this afternoon. I'll just say this to the admissions officers for the private institutions using the College Board Profile: you've just demonstrated to me why your efforts at recruiting a diverse population of students is often a facade. When your chosen tool (which you don't have to pay for) is several orders of magnitude more difficult to complete than the old, more complicated FAFSA, it's clear that you don't have a clue about how to get poor students to apply for financial aid. And College Board? Shame on you for requiring poor families to pay for the privilege of having one more barrier to receiving financial aid.

My daughter will do fine, and unlike other college seniors, she hasn't panicked. Several years ago, when it was clear she was interested in Type X college, her mother and I talked about the financial feasibility of that. (I'm a public-university professor in a relatively low-paid field. Well-off? Definitely with respect to human history! Able to send my daughter to Type X college on my and my wife's income alone? .... uh, what type of cat food tastes good?) We figured we could expand her horizons, but given that her spine is stiffer than mine, I expected it would be in one direction.  Let's see: ask her to consider Type Y college? Not going to happen. Z? Not a chance. Type X-public? Hmmn... that worked. In the fall of her sophomore year, I told her that if she could find a Type X college that would let her visit classes, either public or private, I'd take her. And she found such a place, so we went. As a result, we spread out college visits over a few years, not a few weeks. That first college is still on her "very interested" list, and overall she liked (and applied to) roughly half of the places we visited, most of which were Type X colleges. Her interests have changed a bit, but she'll do fine in any of the places she's applying to, and it's her life, not mine. Yes, she's been accepted to at least one. As I stated above, if you've worked hard in high school and you're not set on getting into the One True Place for You, you'll get in somewhere you can learn a great deal in.

Horizon 2010 report mostly wrong

At least this year's EDUCAUSE Horizon report on emerging IT doesn't predict the tremendous growth of Second Life. But it has plenty of misjudgments in what it predicts will be Big Higher-Ed IT in the near and medium term. Below are my quick judgments of what Horizon 2010 thinks will be big:

• Semi-correct: the impact of "mobile computing." The sloppy use of the term indicates that the report writers have bought into the hype. There is just too much fragmentation of operating systems and too many students of moderate means who cannot afford smartphones for this prediction to be anything but wishful thinking. Mobile computing will work for certain professional programs, largely at graduate levels, where either there is a reasonable expectation that students will buy equipment as demanded or where there is support for a specific set of devices. My guess for the most common application of mobile devices today? Clickers. Maybe some company will figure out how to combine clicker technology with prepaid (term-length) cell service for specific purposes. Until then, mobile computing will generally be project-specific.
  
• Semi-correct: the likely impact of ebooks. Again, this is going to be more selective than the report indicates (and I say this as a relatively early adopter). What ebook readers may provide is more flexibility to read generally-formatted text documents (such as PDF), rather than expansion of types of formats (such as multimedia).
  
• Largely incorrect: expansion of open content. In a few subsidized areas this will continue, but we've already seen the shuttering of one major open-content project. The reality of open content is that it requires resources to create and maintain; witness Valley of the Shadow Project, a wonderful online history project that is now officially "archived." Obvious sign of the report's failure to connect with reality: no discussion of the shutting down of Utah State's OCW project. Ouch.
• Largely incorrect: gesture-based computing. These applications will be quite complicated and expensive, and they will be limited to disciplines where the investment pays off. 
• Philosophically problematic: the hype of "visual data analysis." I use graphs in teaching. I do not assume that because I use graphs, students can competently conduct data mining just by looking at pictures. For some reason I cannot fathom, the report highlights Wordle; a tag cloud is the humanistic equivalent of USA Today "infographics." Horrid. Kill this idea now, please, before you do more damage.
  
• Major goofball hype: augmented reality. Yeah, right, in the same way that Second Life took off and CAD is used in English courses. Whenever the most obvious use of a particular tool is in the field of architecture, you know that you're not talking about a tool that is going to be used widely across higher ed.
  

I need to return to my Sunday copyediting task (a wonderful but very long and editing-needy article MS). Maybe my focus on copyediting today is making me a bit grouchy with the Horizon report; I know that since I've criticized the report, I should probably provide an alternative perspective, and I'll think about that over the next week. In a year or five, you'll be able to see who was correct.