OpenFiction [Blog]

Failure to define success, part 2

I’ve had my share of existential moments, but this is one I didn’t realize I was having. Or rather my profession was having. I like Taylor’s book for the most part, and think it serves as a useful examination of the field, but I do think it does miss on a couple of fronts, which I will discuss in later posts.

But for now, the soundbite: Ira Fuchs quote “If you take away OCW completely, I’m not sure that higher education would be noticeably different.” Sure, especially US higher education. The same could be said of Wikipedia. And once again I am filled with the sense that as a movement, we are failing to adequately define success and so leaving ourselves open to having others define it for us.

When OCW was announced, I think there were many out there who hoped it would provide the leverage to break away from the artisan model of teaching to something that was more scalable. There seem to be two varieties of this hope: One that, faculty around the world would just pick up and use MIT’s curriculum, saving time and improving quality in one fell swoop (the “dirty underwear” model); two, that OCW would repeat Wikipedia’s success and that teachers around the world would collaborate on one “killer app” curriculum. A third variety that emerges in Taylor’s book is that online resources might supplant live teachers entirely-the OLI model.

All three I think grow from a view of education that holds it is essentially knowledge transfer, and that there ought to be one “best” way to do it, measurable and precise. Education, at least for me, is intensely local and personal, learning how to learn. I won’t dwell, and plenty of people have spoken more intelligently and articulately on the issue. Comments like Ira’s I think express the frustration of revolutionaries expecting a revolution.

OCW by its nature, though, reinforces the artisanal model of education by providing an example of one of the best artisanal communities of educators in the world hard at work. When we were first going to faculty and encouraging OCW participation, one of the constant refrains we heard was that MIT’s materials were designed for MIT students, and likely weren’t going to be appropriate for most people out there. Not that they were necessarily too high level, just that they were created for a specific community working with in specific conditions.

However, OCW materials do provide educators a window into how the MIT faculty community operates, how they craft educational experiences, and other craftsmen and women around the world can draw inspiration and resources from OCW as they create their own educational experiences. But this is not the kind of activity you see writ large on the face of institutions. Nor does it change the fundamental model.

Large parts of the OCW story also take place outside the walls of institutions as well, offering educational opportunity to people who previously had none, and Ira’s comment completely ignores this issue. OCW has the potential of impacting a great many lives, and appears at some level to have already done so for hundreds of thousands. But this is a difficult story to document and tell, not measured in pre-tests and post-tests.

Which doesn’t mean we shouldn’t be doing it, it just means we have a lot of hard work ahead of us. This again is not a post about Ira Fuch’s comment, it’s a post about our own failures to make the case. To define success. To share what we know about the ways OCW is making a difference around the world. Ira’s right, we haven’t noticeably changed higher education. But we have noticeably changed lives all around the world, and we need to be getting that message out there.

Failure to define success

I guess the thing I’m feeling most strongly out of the most recent exchange over the issue of cost savings and OCW is my own failure to get more data out there about how OCW generates benefit. I have several years’ worth of survey data that we simply haven’t had the time to package up neatly and get out there, and so if we aren’t showing how it works, I guess we can’t complain when others attempt to define success for us.

So this is in small measure an attempt to insert some fact into the speculation. David is currently postulating that cost savings from adopting OER is the key benefit, and if that benefit isn’t generated the movement has failed. The data below is from a 2008 survey of OCW users with more than 5,000 self-selected respondents. Educators were 9% of respondents and so we are already looking at a small portion of overall traffic. How does this subsection of the audience use the content?

Graph showing educatior scenarios of use

As it turns out fewer than 1/3 are using it in a way that would require direct adoption (20.2% incorporating materials + 7.9% developing curriculum).  The other three modes of use—personal learning, learning new teaching methods and finding reference materials for students—have nothing whatsoever to do with adoption of materials in the way David describes (although the reference materials for students might be done via linking).

We can look even more granularly at these modes of use and the benefits they produce.  So, for example, personal learning, which is 30.5% of educator use:

Table of personal learnign uses

As can bee seen, educators use the site largely to learn new material, either within or outside their field, and secondarily to refresh their knowledge of the basics.  What are the benefits in doing this?

Table of  educator personal learning benefits

Mostly making them better teachers, it would seem, through the availability of better information and the motivation it provides.

What about learning new teaching methods, the second most prevalent mode of use in Table 1 at 22.9%?

Table of ways educators learn teaching methods

Primarily they learn new methods for themselves, and secondarily for their wider community of educators.  What benefits does this activity generate?

Table of teaching methods benefits

I’m sure this will be interpreted in some circles as promoting backwardness, but interestingly, most educators learning teaching methods from the site believe they become better lecturers, and only secondarily learning to make instruction more interactive or project based.

OK, onto the issue that has David’s attention, incorporation of materials at 20.2% of educator use.  What does this actually look like?

Table of incorporation modes

No surprise most faculty are incorporating materials into an existing course, which means the formats, approaches, language, etc. all have to be a pretty good match for direct adoption.  Interestingly, the second most prevalent way is looking for ideas on how to design a course, which may or may not be the kind of direct adaptation David is interested in.  The last mode, adoption for a new course seems to me to be the clearest path for adoption in the way David describes it.  And the benefits?

Table of adoption benefits

Time savings fall pretty low here, and cost savings even lower.  So the benefit David is focusing on is 37% (24.9% time + 12.1% cost) of 20.2% (% of educators incorporating materials) of 9% (% of educators) of the benefit we’ve identified.  That’s less than 1% of the benefit.  If indeed this is the key benefit of OCW, we are truly in trouble.

Let’s complete the data set with the student reference use (15.1% of educator use):

Table on educator refence uses

Largely, educators appear to use the materials to help students better learn concepts in the class or to study more advanced topics, and less for remedial work.  How do they characterize the benefits?

Table on reference benefits

Increased student learning and increased student motivation appear to be the key benefits here.

Finally, curriculum development, at 79% of overall educator use:

Table on curriculum development

Here is an even split between existing and new curriculum.  And the benefits?

Table on curriculum development benefits

Cost was not addressed directly in this instance (the possible benefits were generated through an analysis of open-ended questions regarding benefit in previous surveys).  Generously assuming all “other” responses to be cost- or time-savings related is still only 5.9%.

So scanning the benefits in all of the tables above, most of the responses have to do with increases in quality of instruction or learning, or with student and faculty motivation.  There are two ways to make a system more efficient, make it cost less or increase the output.  I don’t see a lot of evidence in this that OCW can make education cheaper (though open texts and open access journals may), but I see lots of evidence that it can help us all get more out of the investments we do make in education. The above analysis does not even take into consideration the benefits generated for students, who are 42% of our audience.

Of course there are things that could be said about the data–it’s self-selected, the benefits were preselected and incorporate bias, it represents what is rather than what should be and if OCW were more adoptable more people would adopt it…  All of these are likely true to an extent, and show just how difficult it is to conduct evaluation on a resource like OCW.  But I doubt the cumulative effect is enough to change the picture dramatically.

Why is this important enough to spill so many pixels over?  Because selling chickens as a source of milk will disappoint the customer.  If the institutional administrators, Department of Education staff, grantors, donors and other sources of support are brought in on the cost savings argument, and those savings fail to materialize, the movement will lose support.  For formal education, it’s very important that we actively promote OCWs ability to increase quality of education through transparency and open publication, and to also look for cost savings where they emerge.  If we are going to do OpenCourseWare, we ought to do it for what it does—increases the quality of education at our institution and provides educational opportunity for millions—not for what we wish it did.

Anyway, in future posts, I’ll share parallel data for students and independent learners.  But more immediately, I have a follow on case study to this educator data that illustrates how “adoption” is not simply plug-and-play or a matter of a few localizations, and why the impact is more quality and less money.

Mythbusters Special: MIT OpenCourseWare Paywall

Posted in Uncategorized by scarsonmsm on September 16, 2010

The University World News is erroneously reporting that MIT is considering putting OCW behind a paywall. Don’t believe it. This is apparently a misinterpretation of ideas presented in last year’s Institute-Wide Planning Task Force Report.

The Chronicle and David Wiley are taking the opportunity to recycle information that’s been out there a while from Institute planning activities, which is fine (MIT would be silly to not be evaluating its position in the online environment regularly), but the impression generated is that there has been some substantial change in OCW’s financial situation, which isn’t the case. We’re still working on a set of less sensational sustainability efforts.

Gates praises MIT OpenCourseWare

Posted in Uncategorized by scarsonmsm on April 21, 2010

At his talk today on MIT’s campus, Bill Gates discussed the impact OCW is having on both global education and his own personal education. He says he’s watched the lectures from 11 of our 33 courses with full video lecture recordings, and has his sights set on two more. He does a great job in just a few minutes of capturing both the promise and the challenges of open education at this point in the game. Full video of the event is available from MIT AMPS.

Share the load

Posted in Uncategorized by scarsonmsm on February 5, 2010

I’m always on the lookout for revealing OCW numbers. I spent a chunk of today resorting my e-mail archive to finally fully clean up the mess from last year’s hard drive crash. I sorted things into files by year, one for sent and one for received, and was thus able to generate the following data set:

Year: Received/Sent

2003: 5,587/4,219
2004: 3,849/3,116
2005: 6,203/3,256
2006: 8,041/3,620
2007: 12,498/5,720
2008: 13,474/7,465
2009: 11,884/5,509

To some extent, there is a personal history of the project written in these numbers. ’03, which was a full year (I started January 6th, 2003), was arguably the busiest year I’ve spent with the program–a year in which I managed the publication of 100 courses, developed a major database we still use, and launched our department liaison program. In ’04, the pace was a little slower and I worked quite a bit on the program’s evaluation, which required less communication and more quiet desk work.

’05-’08 chart in part the growth of the Consortium and the increased communication it generated. They also chart increased internal responsibilities as I took on more comunication and management tasks. ’08 was a peak year as we pushed the Consortium through to incorporation and more independance. ’09 was lower as I was less split between the OCWC and MIT OpenCourseWare.

The larger point though, and I think this is true for many people I know, is that my work life is increasingly fragmented into (literally) 10,000 splinters. I’ve felt in the last year or two like I am developing a bit of e-mail induced ADD. I’m finding it more work to focus on tasks requiring an hour or more of attention. I’m looking forward to working on another big evaluation report in the coming months, simply for the opportunity to shut off the e-mail and work on an extened project for several hours at a time.

Tagged with:

CED Report

Posted in Uncategorized by scarsonmsm on November 4, 2009

The Committee for Economic Development, a non-profit, non-partisan business led public policy organization (their description), has issued a really excellent report on openness in higher education.  It addresses an exhaustive array of facets to openness in the higher education context including OER, Open Access, and openness in administration and certification. The report is a great opportunity to understand the breadth of the open landscape as it stands today.

There are a few factual errors regarding OpenCourseWare that ought to be noted.  On page 18, the report vastly undercounts the number of OCW courses available through the Consortium at 5,000, when the most recent self-reported figures from the membership are closer to 13,000.

More concerning is the characterization of MIT OpenCourseWare’s agreement with Elsevier, which on page 28 is described as:

…a more straightforward and operationally simple definition of fair use so as to ease rights clearance for MIT’s OCW; Elsevier now provides blanket clearance for up to three tables and 100 words per article for thousands of Elsevier’s articles.

Our agreement with Elsevier governs our use of their materials under our open CC license and in no way defines or limits our recourse to fair use with respect to Elsevier’s or any other content owners materials.  Were we to use materials employing a fair use approach, they would have to appear on our site with all rights reserved.  Elsevier has agreed to allow OCW to publish materials under our CC By-NC-SA 3.0 license, which makes the materials available for downstream reuse.  It’s a very important distinction.

Of profiles and pop-ups

Posted in Uncategorized by scarsonmsm on September 9, 2009

For as long as I’ve been evaluating the use of MIT OpenCouseWare, our highest-level user profile has been relatively consistent. Our first evaluation put educators at around 13% of our audience, students around 40%, and self learners at 53% (these figures are from memory).

The numbers shifted somewhat over the years, with educators moving steadily upward to 15%. Students settled somewhat lower to around 30%, and self learners fell to around 50%. The numbers all seemed to be moving in comfortably predictable trend lines.

At the same time, I knew as far back as 2006 that Firefox was skewing our data collection with its efficient pop-up blocker. In the notes from the report released that year, I described this impact. I also had it in my to-do list to implement a system that would correct for this issue.

For the 2009 survey I finally got a system implemented that did not rely on pop-ups, and while I understood that Firefox was having an impact, I didn’t understand the implications of the impact until after I looked over the numbers. What I might in retrospect have predicted but only became clear in data analysis was the differential adoption of Firefox across the user profiles.

Firefox, it turned out, was being adopted at a significantly higher rate by students than by educators or self learners.  In the 2009 survey, 57% or educators and self learners were using Firefox, while a staggering 67% of students reported using the browser.   As market share of Firefox among students grew, they were being disproportionately underrepresented in our survey results.

For the ’09 survey, the profile numbers are 9% educators, 42% students and 42% self learners.  This shift isn’t earth-shattering, I think, but it does raise some interesting questions.  With the old numbers, it could be easily argued that the predominance of MIT OpenCourseWare’s impact was in the informal learning sphere, and certainly US self learners continue to be the single biggest block of OCW users.  With the new numbers, it appears that OCW is having more of an impact on formal educational systems than has been apparent to date.

No doubt some of this impact in educational systems is “informal,” use of OCW as supplementary resources not directly incorporated into formal instruction.  Among both students and educators, enhancing personal knowledge (informal study) is a primary scenario of use.  But that informal study is occuring within a specific context, with other resources—libraries, peers, instructors—available.

It may be a while until I understand all of the implications of this, but I do think it means we need to look more closely on campus, and try to better understand how OCW-type resources are reshaping the formal educational experience.

Welcome to the new home of tOFP [Blog]

Posted in Uncategorized by scarsonmsm on September 3, 2009

Please excuse the mess as I’m still moving in from my old home. Yes I intend to keep the CC licensing and I’ll be adding a fresh blog roll as time allows. Hope to get back to blogging a bit more in the coming months.

Update on Daniel

Posted in Uncategorized by scarsonmsm on August 19, 2009

Just wanted to follow yesterday’s post with an update on Daniel.  His scans today were all clear, and there is no evidence of the return of the cancer. We are obviously relieved and thankful.   I appreciate all the kind messages and thanks to those of you who have made donations to our Jimmy Fund team.

We are deeply touched by the many notes from friends whose loved ones have suffered–and too many who have passed away–from cancer.  The SuperDans will keep all of them in our thoughts as we walk on September 13th.  Also, thanks to your generosity, I am approaching my goal of raising $500 individually.

As a team, though, we are still far short of our $10,000 overall goal, so if you are considering a donation, we still need your help.  Donations may be made on the team site (http://www.jimmyfundwalk.org/superdans) and my individual site (http://www.jimmyfundwalk.org/scarson).

Again, thanks for all the notes of support, and thanks for helping us to support this cause.

Tagged with:

Help us end cancer

Posted in Uncategorized by scarsonmsm on August 18, 2009

Tomorrow, Lori and I are taking Daniel to his next appointment at the Jimmy Fund Clinic to ensure he remains cancer free. Thankfully, the appointments are now every six months instead of every three, and now we go with a little less fear but still knowing that relapse for him is not entirely out of the question.  We are thankful he has remained free of cancer, but every visit to the Jimmy Fund Clinic is a reminder to us of the many children still battling this disease, the many families struggling to find their way back to a normal life with healthy, happy children.  As much as we would like to put the experience of Daniel’s illness behind us, we cannot forget their pain.

These children need better cures, and their caregivers need better funding.  For the third year, we will be walking in the Dana Farber/Jimmy Fund Boston Marathon Walk to ensure the care and cures will be available.  In each of the past two years, our team–the SuperDans–has raised over $10,000, and we’re aiming to do this again.  I have personally committed to raising $500, and I am asking you to help me reach that goal.  Please visit my page at (http://www.jimmyfundwalk.org/scarson) and make a donation.  I appreciate any support you can provide.

Tagged with: