Texas A&M and the Texas Public Policy Foundation's "Breakthrough" Solutions, Part II

This is the second part of my commentary on the Texas Public Policy Foundation's "Breakthrough Solutions" and their application to Texas A&M.  (The first part is here.)  This post deals with Solutions 1, 2, and 4, which propose a set of reforms to improve the quality of undergraduate teaching at public universities.

Without further ado:

Solution 1. Measure teaching efficiency and effectiveness.

Goal: Improve the quality of teaching by making use of a public measurement tool to evaluate faculty teaching performance that makes it possible to recognize excellent teachers.

This reform is a mix of some good and bad elements.
 

First, collecting and making public instructor-level data on teaching performance and grades is a great idea.  Current and prospective students should have access to information about who the best teachers are, which classes they teach, and how often those classes are taught.  I wouldn't buy a car without information on the relative quality of various makes and models.  Students shouldn't be forced to choose a college or plan a schedule without being able to compare the relative quality of the faculty who will be teaching their classes.

Data on teaching effectiveness could also be used by the university to more effectively direct resources to enhance undergraduate teaching.  For example, less strong teachers could be directed to support services. 

My only criticism of the evaluation and public information component of this solution is its exclusive reliance on student evaluations.  I am not a student evaluation alarmist, but I am certainly sensitive to some reasonable concerns about using them without supplementary assessment tools.  Though student evaluations are often useful indicators of an instructor's proficiency, there is some evidence of gender bias and bias against more demanding courses (i.e. those that require essay exams versus multiple choice exams), for example. Including peer evaluations, retrospective alumni evaluations, and other assessment tools along with student evaluations in a record of teaching effectiveness seems like a sensible way to augment student assessment data and maybe tap information about the prospects for long-run teaching value as opposed to short-run satisfaction.

Information on grade distributions by course is already public information at Texas A&M, so releasing the "average percentage of As and Bs awarded" isn't much of a reform.  Indeed, web resources like http://www.myedu.com/ have already aggregated a pretty impressive array of information about instructors' grading history overall and for particular courses.  Combining information on teaching performance with grading information is also pretty harmless, since, my impression is that student evaluations, at least, are more associated with a course's "demandingness" in terms of work than its "difficulty" in terms of grades.  (Someone who is more in touch with research on student teaching evaluations is welcome to correct me.)

In any event, publicizing grading data would probably have a couple of positive consequences.  First, I think it would result in an informal harmonization of grades across various instructors teaching the same courses.  Few people want to be an outlier, and simply letting faculty know what other faculty could support a more fair aggregate grading scheme without the stricture of firm curves of targets.

Second, I think it would provide students with more reasonable expectations about their performance in college courses.  Most students who attend A&M were in the top 10% of their high school classes.  Many have never received a C as a grade for their work before.  Showing them historical data on class performance can help them form realistic expectations about how they are likely to perform and, perhaps, inspire individual students to perform at a higher level than they might have otherwise. 

On the bad side, the whole faculty cost per student hour computation is awful.  Awful.  Big (profitable) lecture courses have their place in higher education.  The instructor can transmit information and ideas to large numbers of students and be around as a resource to answer questions, advise, deal with current events, etc.  However, the size of those courses makes it impossible to provide much attention to individual students or their work.  Teaching a smaller course, I can meet with each student individually outside of class to discuss the readings and lectures, go over term paper ideas, review outlines and read drafts of written work, go over old exams, discuss prospective tests, etc.  That is physically impossible in a larger class.

The point is, larger classes offer higher numbers of less dense faculty-student interaction; smaller courses offer a smaller number of more dense interactions.  At the level of the student, much more teaching is packed into the average small course than the average large course. Lumping the high density teaching and low density teaching together in a faculty cost analysis is unreasonable and relies on an illusion created by the fact that the university charges students the same tuition for large courses and small courses.  We overcharge for big courses on a student-hour basis and undercharge for small courses.  In essence, the big courses subsidize the small courses.  If we unbundled the two, charging a lower rate for big courses and a higher rate for smaller courses, then the "value" of faculty who principally teach smaller courses would look much more like the "value" of faculty who principally teach large courses than it does with the lumped pricing structure in place.

As an analogy, imagine that the owner of a McDonald's franchise performed the sort of analysis proposed by the breakthrough solution on the components of the Happy Meals it sold.  He'd take the total amount of money received for the Happy Meals, divide the revenues among the various components of the meals---hamburgers, fries, and Cokes---and then subtract the total cost of providing each element of the Happy Meals for the number of meals sold.  If he "force ranked" those components, he'd find that burgers are losers, fries are a wash, and sodas are big winners.  By the reasoning implicit in the TPPF solution, he'd conclude he needed to sell three Coke Happy Meals at the price he was charging for the burger, fries, and drink model.  This reasoning, though, completely ignores that the greatest part of the actual value in the meal comes from its most expensive components.

Any per student hour computation of faculty profitability is absolutely useless unless it corrects for the implicit under-pricing of high-density teaching in small classes (and the complementary over-pricing of low-density teaching in large classes).

A real "breakthrough" reform might be to actually unbundle tuition costs by class size and department (or the quality of the professor's teaching evaluations or even time of day and other factors that make different courses or sections more or less valuable to students).  Credit hours in large lecture courses in the humanities would be very cheap; credit hours in small laboratory courses would be more expensive.  Students could actually make more efficient consumer decisions about what kinds of courses they wanted to take based on a more informative pricing scheme, and the university and its departments could adjust course offerings and prices as a way of controlling demand for over-subscribed courses and enticing students into important but under-subscribed areas.

In any event, here's the second breakthrough solution:

Solution 2. Recognize and Reward Extraordinary Teachers

Goal: Breakthrough Solution #2 will offer voluntary cash bonuses to improve the effectiveness and efficiency of teaching on college and university campuses and attract the best teachers.

Bonuses will be widespread, significant and based on how well a course delivers on its learning objectives Since up to 25% of the faculty each semester will receive a teaching bonus, every faculty member will have an incentive to improve his or her teaching skills. The top awards will be up to $10,000 a class, so once a teacher receives a lower-level bonus, there will be a strong incentive to continue to improve.

Basing bonuses on how well students judge that a course delivered on its promises will encourage faculty to be as explicit as possible about the learning objectives for each course, making it easier for
administrators to judge the effectiveness of the curriculum and coordinate class offerings.

Finally, since the bonuses depend not only on the quality of teaching but on the number of students taught, the best teachers will have an incentive to teach even more students.

I have more-or-less the same orientation toward this solution as the previous one.  Recognizing outstanding teaching is a great idea in general.  Recognizing those faculty whom the students regard as the most outstanding teaching is a great idea in particular.  I am surprised there wasn't already a program in place to do so at Texas A&M.  (As a graduate student, I won exactly this kind of teaching award from the student body at the University of North Carolina---The Student Undergraduate Teaching and Staff Award.)  I am perplexed that anyone would object to this being only one of several ways that A&M recognizes great teaching.

I think the favoritism toward large classes is off-base, though, for the same reasons I mentioned above.  The implicit assumption that large class teaching is better and more valuable because it serves more students is wrong.  Large classes and small classes provide comparable amounts of per-student value, a fact obscured by the way the university prices its classes.

Solution 4 is more ambitious and more rightly controversial. 

Solution 4. Require evidence of teaching skill for tenure.

Goal: If universities are to serve students, parents, employers and taxpayers, a large number of the tenured faculty need to be good teachers as well as productive researchers.

This reform ensures that teaching will be considered as an important qualification for tenure by requiring evidence that a majority of those nominated for tenure have demonstrated the ability to teach well.

The principle is totally fine.  This is a public university with a public education mission.  High standards for teaching as well as research should be part of the tenure review here.  Reviewing and perhaps amending the mechanisms in place to ensure that faculty are given tenure only on the condition that they are effective teachers is a fine idea.

The devil (and the controversy) is in the details here.  These are the two items listed under "Carrying out the Reform."

1. Require evidence of teaching skill for tenure.

The majority of new tenure appointments (say 75%) will be granted to professors who have proven that they can teach well by having taught on average three classes per semester and thirty students per class for the seven or more years that a teacher is on the tenure track.

2. Customer (student) satisfaction ratings would be used to determine teaching effectiveness.

Average teaching ratings must be a minimum of 4.5 on a 5.0 scale. Limits may be placed on the number of A’s and B’s awarded if the efficacy of customer (student) satisfaction ratings are questioned.

The first problem here is that faculty don't pick and choose their own courses  We are assigned courses by our departments.  So, this whole reform misplaces responsibility for administrative-level decision on individual professors in the tenure process.

The second problem is the continued prominence of the production line mentality---more students in bigger classes is better.  More students in bigger classes is more profitable for the university under its current pricing scheme, but it does little for the overall quality of the education students receive, the value-added to students and graduates, or the value the university provides to the state by, among other things, enhancing the quality of its workforce.  Naively increasing the number of students and courses taught by most (say 75%) faculty members will decrease the average quantity and quality of faculty-student interactions.

Third, if tenure decisions are going to be made on the basis of firm cutpoints in teaching evaluations, you better have one heck of a valid and reliable instrument.  I am not confident we have one at Texas A&M currently, and I am not confident that one exists.

Fourth, the extended reform document also hinges on the severability of research and teaching.  Dealing with that point will be a focus of my next post, so I'll keep it short for now.  Suffice it to say that it is my belief and personal experience that research improves teaching.  As a student, my best teachers were always those who taught in the fields in which they were scholars or practitioners.  As a professor, I am at my best when I teach material that I know from personal experience as a scholar.  There is no substitute for active engagement with some topic for helping me constantly update my courses to reflect cutting-edge knowledge about my field. Again, I'll have more to say about this later.

Finally, I am also concerned about this reform because of how ad hoc it seems.  This is a major policy statement by a major policy research organization relating to one of the largest systems of higher education in the world.  Instead of providing concrete guidance about how to evaluate teaching in the tenure review process, it just says things.

Where does the "say 75%" figure come from?  Are we OK with having say 25% of the faculty be awful teachers? What about the 4.5 cutoff?  Where did that come from?  Is that just a roundish number or does it relate to some demonstrated distribution of teaching ability?  What instrument uses the 5 point scale?  What is the time frame under which teaching will be evaluated for tenure?  If this is supposed to be a statement of principals, why the precision?  If this is supposed to be a specific proposal, why the lack of detail?

It's sloppy policy work. This is a serious issue, and Texas A&M and TPPF are serious organizations.  The subject matter and the institutions involved deserve better.

Craftsmanship aside, I would like to repeat that I am sympathetic to teaching being a more highly weighted component of the tenure process.  Increasing teaching loads or inflating class sizes by themselves doesn't serve that goal.  Hiring good teachers, supporting teaching development, making teaching more important in the tenure and promotion processes, and providing a sensible balance of large-class (low density) and small class (high density) coursework along with experiential learning, like internships, seems like a more sensible set of targets for reforming teaching at Texas A&M.

For next time, Solutions 3 and 5.