**** ******* ******* *** ******** Report

January 2005

** ***** **** *** **** info rmation, please contac t:

Stephen Carson

Eval uation Sp ecialis t

MIT Op enCo urs eWare

Building 9-251

** ***sachusetts A venue

Camb ridg e, MA 02139

Phone: 617-***-****

Fax: 617-***-****

Email : sc arso n@mi t.edu

MIT OpenCo urseW are is funded jointly by the William and Flora Hewl ett Foundation,

the Andrew W . Mellon Fo undation, and the Massachusetts Institute o f Technology.

MIT OCW Evaluation Findings January 2005 ii

MIT OpenCourseWa re

2004 Program Evaluat ion Findings Report

Contents

I. Execu tiv e Summary 1

A. About MIT Op enCourseWar e 1

B. Histor ical context 1

C. Evalu ation design 1

D. Evalu ation data sources 2

E. Summary of findings 3

1. Access 3

2. Use 4

3. Impact 6

II. Findings: Access 8

A. MIT Open CourseW are content access levels 8

1. Overall O CW site traff ic 8

2. First time and returning O CW tr affic 8

3. Page v iew s per visit 9

4. OCW visitor g eographic orig in 10

5. Access to translated OCW mater ials 10

6. Offline access to O CW materials 11

B. Visitor awar eness of O CW 11

1. Overall v isitors awar eness channels 11

2. Visitor awar eness via sear ch eng ines 12

C. Visitor education al roles and profiles 13

1. Educational ro le distribution 13

2. Overall v isitor profile 14

3. Self learner profile 14

4. Student profile 16

5. Educator profile 17

6. Prior visits and visit frequen cy by edu cational ro le 18

D. Visitor languag e and English proficiency skills 20

E. Traffic by organ ization typ e 20

1. Visits from educational institutions 20

2. Visits from non-educational organizations 21

F. Visitor technical profiles, site p erformance and recommended improvements 22

1. Visitor technical profiles 22

2. Site technical performance 22

III. Findings: Use 24

A. Content use by subject 24

1. Use d istr ibution by subject 24

2. Course traff ic levels 25

B. Scenar ios of use by role 26

1. Self learner scenar ios 26

2. Student scenar ios 27

3. Educator scenar ios 28

C. Material use by role 30

1. Overall mater ials use 30

2. Materials use by self learners 31

3. Materials use by students 32

4. Materials use by educators 32

D. Visitor satisf action 32

MIT OCW Evaluation Findings January 2005 iii

1. E-mail feedback 32

2. Satisfaction by role 33

3. Suggested content improvemen t 33

E. Special focus: MI T student use 34

1. MIT student access 34

2. MIT student use 36

3. MIT student impact 37

IV. Findings: Impact 39

A. Visitor per cep tions of impact 39

1. Visitor current and futur e impact ratings 39

2. Visitor per cep tions of impact on educational exp erience 40

3. Visitor intent to r eturn 41

B. Distr ibution of MIT O CW content 41

1. Universia.n et 42

2. Chin a Open Resources for Education 42

3. Opensource Op encoursew are Proto type System 43

4. Makerer e Univ ersity, Uganda 43

5. Unaffiliated mirror sites 43

6. Impact of O CW on dev eloping r egions 43

C. Adoption of the open coursew are concep t 44

1. Affiliated op encoursew are pro jects 44

2. Unaffiliated opencourseware projects 44

Appendix 1 Background on MIT O CW 46

Appendix 2 Program Ev alu ation Methodology and D ata Collection 49

Appendix 3 O CW Inter cep t Survey 55

Appendix 4 Interview Proto col 68

Appendix 5 Interview Notes 72

Appendix 6 O CW MIT Studen t Survey 89

MIT OCW Evaluation Findings January 2005 iv

Tables and Figures

Figure 1. Month ly visits .. ... ... 8

Figure 2. Monthly f irst time v isitors ... ... 9

Figure 3. Month ly returning visitors .. 9

Figure 4. First time and returning visitors 9

Figure 5. First time and repeat visits . 9

Figure 6. Distr ibution of page views per v isit ... 9

Figure 7. Mon thly visits to translated con ten t ... 10

Figure 8. Mon thly visits to OCW and translated conten t .. 10

Figure 9. Course traff ic d istr ibution by visits .. 25

Figure 10. MI T domain visits .. 36

Figure 11. Univ ersia.n et Sp anish-language O CW ho mepag e . 42

Figure 12. V isits to OOPS translation /mirror site 43

Figure 13. RAI Un iversity open coursewar e homepage ... 45

Figure 14. O CW site for Environmental Flu id Mechan ics 1 ... ... 45

Table 1. Visitor geographic d istribution ... . 10

Table 2 . Wor ld population and intern et use . 10

Table 3. Use cases: Offline uses of OCW mater ials 11

Table 4. O CW v isitor aw areness ch annels .. 11

Table 5. Top ref erring sites .. 12

Table 6. Top top ic-based search phrases . 13

Table 7. Visitor educational role distr ibution .. 13

Table 8. Visitor role distribu tion by reg ion . 13

Table 9. Overall visitor education lev el ... 14

Table 10. Over all v isitor area of inter est/d iscipline . 14

Table 11. Self learner g eographic origin . 15

Table 12. Self learner education lev el .. .. 15

Table 13. Self learner professional experience 15

Table 14. Self learner ar ea of in terest/discip lin e . 15

Table 15. Use cases: Self learner profiles 16

Table 16. Studen t g eographic distribu tion 16

Table 17. Student education level 16

Table 18. Studen t institution type 16

Table 19. Student area of interest/d iscip line 17

Table 20. Use cases: Student profiles .. 17

Table 21. Educator highest degree earned . 17

Table 22. Educator institution type 17

Table 23. Educator experience level 17

Table 24. Educator geographic distribution 18

Table 25. Educator ar ea of interest/d iscipline . 18

Table 26. Use cases: Educator profiles 18

Table 27. Pr ior visits by ro le . 19

Table 28. V isit frequen cy by role 19

Table 29. O ctober 2004 estimated r eturning v isitors by role and frequency 19

Table 30. Use case: V isit frequen cy 19

Table 31. Visitor native language 20

Table 32. ESL r ead ing skills self assessment . . 20

Table 33. Use case: Eng lish language proficiency . 20

Table 34. Top 50 .edu or .ac domains by visits . . 21

Table 35. Top non-educational domains by visits ... ... 21

MIT OCW Evaluation Findings January 2005 v

Tables and Figures

Table 36. Visitor operating sy stem .. 22

Table 37. V isitor browser typ e . ... . 22

Table 38. V isitor connection type . 22

Table 39. Visitor satisfaction with site performan ce .. .. 22

Table 40. Use cases: V isitor techn ical suggestions .. 23

Table 41. Use cases: Visitor exp eriences with PDF formats .. .. 23

Table 42. Site u se d istr ibution by discip lin e . 24

Table 43. Use cases: Cross-curricular uses of the site . 25

Table 44. Self learner scen arios of use . 26

Table 45. Use cases: Self learner goal statements by use scenario . . 26

Table 46. Use cases: Self learner e-mail feedb ack .. . 27

Table 47. Studen t scenar ios of use . . 27

Table 48. Use cases: Student goal statemen ts by scenario .. . . 28

Table 49. Educator scenar ios of use . 28

Table 50. Use cases: Sample edu cator goal statements by scenar io .. .. 29

Table 51. Edu cator adoption of site materials .. 29

Table 52. Uses cases: Educator adoption and ad aptation of site materials . 30

Table 53. Mater ial type use by role . . 31

Table 54. Use cases: Visitor difficu lties in obtaining referen ced full text material . 31

Table 55. Use case: Self learner materials use . 31

Table 56. Use cases: Positiv e e-mail feedback . 32

Table 57. U se cases: Neg ative e-mail feedback ... 33

Table 58. 2004 Visitor site content satisf action ratings . . .. 33

Table 59. 2003 Visitor site content satisf action ratings .. .. 33

Table 60. Use cases: Sample visitor site content improvement suggestions . .. . 34

Table 61. MIT incoming fresh men awar eness 35

Table 62. MIT in coming freshmen awar eness channel .. 35

Table 63. MIT incoming freshmen prior visits .. 35

Table 64. MIT in coming freshmen decision inf luence . .. 35

Table 65. MI T upperclassmen awar eness and use 35

Table 66. MIT upper classmen aw areness channel 35

Table 67. MIT upperclassmen visit frequency . 35

Table 68. MI T upperclassmen usag e by major . 35

Table 69. Use cases: MIT upperclassmen most valu able aspect descrip tions 36

Table 60. MI T upperclassmen scenarios of use .. .. 36

Table 71. MIT upper classmen mater ials use 36

Table 72. Use cases: MIT upperclassmen material use descr iptions ... . 37

Table 73. MIT upperclassmen site content satisf action ratin gs ... . 37

Table 74. MI T upperclassmen site imp act on student experience .. 37

Table 75. Use cases: MIT student impact statemen ts . .. 37

Table 76. Current positive impact per cep tions by role .. 39

Table 77. Fu ture positive imp act exp ectations by role . . 39

Table 78. 2003 current or future imp act per cep tions by role . 39

Table 79. Use cases: V isitor perceptions of imp act .. 40

Table 80. Self learner per ceptions of impact on educational experience . 40

Table 81. Studen t percep tions of imp act on educational ex perien ce .. 40

Table 82. Educator p ercep tions of impact on educational exp erien ce . .. 41

Table 83. U se cases: Educator perceptions of impact on edu cational experience 41

Table 84. Edu cator feedback of impact on teaching and materials 41

Table 85. Use cases: Visitor per cep tions of impact on d eveloping regions .44

MIT OCW Evaluation Findings January 2005 vi

I. Executi ve Summa ry

A. About MIT OpenCourse Ware

MIT Open CourseW are (OCW) is a large-scale, w eb-based electronic publish ing in itiative, accessible on the

Internet at o cw.mit.edu. Through OCW, MIT mak es its cor e teach ing materials lectur e notes, problem sets,

syllabi, reading lists, simu lations, etc. openly available for non-commercial educational purposes. O CW

publishes those mater ials in stand ards-based formats for anyone with access to th e Internet.1 O CW h as a dual

mission:

To provide fr ee access to v irtually all MIT course materials f or educators, stud ents, and

individual learners around th e world .

To ex tend the r each and impact of MIT O CW and the opencourseware concept.

B. Historical conte xt

The MI T Open CourseWare project in clud es three phases: a p ilo t phase (now complete) that encompassed th e

publication of the f irst 500 sets of course materials; an expan sion phase (curr ently underway), during which

OCW will publish 1,300 additional courses, in clud ing virtu ally all of MIT s curr iculu m; and a steady-state phase

in which O CW will con tinue to update and improve th e site conten ts as an ongoing activ ity of the Institute.

OCW publish ed a proof-of-concept site in Sep tember 2002 that included mater ials from 50 courses, follow ed by

463 additional courses in September 2003. The Septemb er 2 003 publication marked the comp letion of the pilot

phase and was accompanied by signif icant med ia cover age. OCW has sin ce comp leted two additional six-mon th

publication cy cles, in cluding materials from 190 and 214 cou rses r espectively, br inging th e total number

published to 915 2 in September 2004. This mileston e marks the in clusion of half of MI T s curricu lum.

C. Evaluation design

In order to understand the ex ten t to which O CW is meeting its goals and estab lish a thorough and continuous

feedback process, we have in tegrated a substantial evalu ation program. Ev aluation is essen tial for two reasons:

Track ing both the usefuln ess and usability of O CW, as w ell as our internal eff iciency, w ill h elp u s to

make improvemen ts to O CW features and serv ices and to set longer ter m dir ection to k eep O CW

relev ant over time.

By measuring the use and by demonstrating th e impact of O CW and the course materials MIT off ers

through it, w e can p ersuade others to consider sh aring their materials through similar efforts.

We segment the O CW evaluation strategy in to program evaluation and process ev alu ation. Within each

category, w e h ave d evised an "ev aluation portfolio" approach that comprises a v ariety of data collection methods

in order to achieve both breadth and dep th in th e evalu ation. This r eport pr esen ts f indings from th e program

evalu ation activities conducted from October 1 through December 31, 2004. Appendix 2 d escr ibes th e logic

model that structures th is evaluation. The evaluation probes three gener al ar eas of user behav ior and user

profile:

Access. Who is accessing O CW, what are their profiles (edu cator, student, self-learner, other), what are

their discip lin es (or o ther interests), and where ar e they located?

Use. How do educators and learners use O CW and is O CW designed appropriately to facilitate that

use? To what ex ten t and in what way s ar e MI T course mater ials adopted or adap ted for teaching

purposes?

1

See appendix 1 for more background information on the OCW initiative at MIT.

2

2 courses have been retired from the site.

MIT OCW Evaluation Findings January 2005 1

Impact. Wh at effects positive or negativ e, in tended or unin tended ar e being realized through the use

of OCW ?

D. Evaluation data sources

OCW undertook this evaluation from October 1 through December 31, 2004. W e used multip le d ata collection

strateg ies (an integrated portfolio approach ) that included the following data sources:

Web analy tics. Ak amai, O CW s web hosting and conten t d istribution network provider,

captur es aggregate usage data such as p age views, object views and user location. Akamai

also offers a mor e sophisticated an aly tic tool called SiteW ise,3 which O CW employed starting

November 1, 2003. Most w eb usag e statistics in this report h ave b een drawn from the SiteWise

tool, with the no tab le exception of geograph ic traff ic information, which is drawn from

Akamai due to its gr eater accuracy. Unless o therw ise noted, web statistics for th is r eport cover

the p eriod of Nov ember 1, 2003 to October 31, 2004.

Online in ter cep t surveys. Betw een October 25th and Nov ember 22nd, 2004, a survey tool

invited (via pop-up window) 103,741 of the 253,597 OCW v isitors for the period to complete

an online surv ey.4 Of those promp ted, 14,308 people b egan the survey, and 5,000 completed it

fully, with a dropout rate of 60% and an over all completion r ate of 4.8 %. The sample provid es

a margin of error of not more th an 1.5 %. Self-learners as o pposed to educators and students

wer e mor e likely to complete the survey once star ted . 5 G eographically, over all completion

rates do not vary signif icantly from d istribution of O CW tr affic; note that in th e 2003

evalu ation, completion r ates w ere higher from Eng lish-sp eak ing regions.6 The sample

significan tly over repr esen ts r eturning as opposed to first time visitors. Returning visitors ar e

11% of all O CW visitors, and make up 52 % of surv ey respon dents. For the 2003 survey,

returning v isitors w ere 58% of r espondents.

Interviews. Interviews w ere conducted w ith a small subset o f people in various target groups

and geographies to gather textured qu alitative data about the use and imp act of OCW.

Interviewees w ere selected from those whose responses sp ar ked the cur iosity of the evaluation

team. Members of th e O CW resear ch team conducted 18 in- depth interv iew s w ith w illing

particip ants from intercept survey responden ts, distribu ted across sev eral target r egions ( Latin

America, Chin a, Sub-Sahar an Africa, and th e Middle East an d North Africa) and education al

roles ( educators, students and self-learners). The interview q uestions and protocol ar e included

in Appendix 4. In addition, 2 follow-up in terviews with sub jects from the 2003 program

evalu ation w ere conducted to gather information on how their use of O CW and their attitudes

about the imp act of the Open CourseW are project h ave chang ed. Cand idates wer e selected

from 2003 interview subjects based on geographic d istributio n, user ro le, and insightfuln ess of

prior responses.

Site f eedback. O CW h as imp lemen ted a d atabase to support the processing and analysis of

user e- mail feedback. The system includes e- mail feedback collected since October 1, 2003.

The f eedback system allows users to self- identify role, geogr aphic region and type of

feedback; fur ther, the system supports tagging of e-mail feed back by topic, correlation of

feedback to related course sites, and full-text searches of f eedback messages. We have

contacted users as appropriate to gather addition al insight into access, u se, and imp act. Unless

otherwise no ted, e- mail feedb ack addressed in th is evaluatio n is the 3,722 feedback messag es

collected from November 1, 2003 to October 31, 2004.

3

Site Wise tracks users anonymously via cookies, and so identifies unique visitors by browser; for the evaluation period, the SiteWise system

reported 7.9% of OCW traffic had cookie support disabled. SiteWise also r elies on JavaScript; for the evaluation period, the SiteWise

system reported 1.7% of OCW traffic had JavaScript disabled.

4

See Appendix 3 for the complete text of the intercept survey. Note that the online survey is built with research logic that dynamically

presents a logical subset of the survey questions based on the respondents answers.

5

Self-learners made up 7.0% less of partially completed surveys than fully completed surveys; students accounted for 5.3% more of the

partially completed surveys than fully completed surveys; educators made up 2.4% more of the pool of partially complete surveys than fully

completed surveys. These figures are comparable to those from the 2003 evaluation.

6

In the 2003 evaluation, North American respondents accounted for 47.0% of completed surveys and 34% of partially completed surveys;

Pacific region respondents made up 1.4% of fully complete surveys and 0.8% of partially complete surveys; in all other regions, ratios were

either statistically equal or indicated a higher percentage of partial respondents by 3-6%.

MIT OCW Evaluation Findings January 2005 2

MIT student surveys. In order to b etter understand th e usefulness of O CW to MI T students as

a w indow in to th e sustainab ility of opencourseware projects, OCW surveyed MI T

undergraduate stud ents using a w eb survey and e-mail invitation. On Nov ember 8, 2004, 3,153

upperclass undergraduates w ere invited to comp lete the surv ey; by Nov ember 19th, 800

students had begun th e survey and 708 had comp leted it fully, for a dropout rate of 11.1% and

an overall co mpletion rate of 22.4%. The margin of error for the resu lts is calcu lated to b e no

greater than 3.2 %. The tex t of the student survey is included in Append ix 6. In addition,

OCW in cluded a limited number of questions on a g eneral In stitute fr eshman survey

admin istered to the 1,081 2004 in coming fr eshmen. 869 stud ents returned th e survey, and 597

responded to the O CW qu estions, providing a 55% response rate. Margin of error for these

responses is calculated to be no gr eater than 2.7%

The d ata sources emp loyed for this evaluation provide a r ich statistical picture of site usage through the web

analy tics and intercep t survey, complemented by qualitativ e information from feedback e-mails and interviews,

with no appar ent contrad ictions.

E. Summary of f indings

1. Access

a. Traffic to th e O CW site is steady, incr easing, and progressiv ely g lobal.

In total, 2.3 million uniqu e visitors came to the site from No vember 1, 2003 to October 31, 2004,

generating n early 4.2 million visits.

The site aver aged 348,000 visits p er mon th; month ly visits r anged from 235,000 in November 2003

to 523,000 in October 2004.

Returning visitors w ere 11 % of all v isitors, and gener ated 46 % of all v isits; r eturning v isitors

averaged 6.8 v isits per month.

36% of O CW visitors come from North A mer ica; 16% each come from East Asia and Western

Europe; 11 % each from Latin America and Eastern Europe; and the remaining 9 % from the Midd le

East, Africa, th e Pacific, Centr al Asia and the Caribbean co mbined.

OCW distribution affiliates Un iversia, OOPS and CO RE7 generated an estimated 1.3 million v isits

(31% of MIT OCW traffic) to translated MI T O CW con tent.

b. Most users continu e to lo cate th e site v ia onlin e and offlin e media.

32% of visitors report learn ing of the site via online med ia, and 23% learn ed of the site from offline

media.

Near ly 60% of visits to th e O CW site came as r eferrals from other sites, largely the main MIT site,

sear ch eng ines, O CW-aff iliated sites, news sites, and technical community forums.

The p ercen tag e of v isitors locating th e site v ia search engin es in creased from 10% in 2003 to 18%

in 2004, and the search es used are incr easingly topic sear ches rather than v ariants of

"OpenCourseWare."

c. Visitor education al role distribu tion and profiles remain abou t the same.

Self learners con tinue to mak e up the bu lk of traffic to O CW (48%), followed by studen ts (31%),

and educators (15%).

7

Opensource Opencourseware Prototype System and China Open Resources for Education; see Sections IV. B. 2 and 3 for more information

on these initiatives.

MIT OCW Evaluation Findings January 2005 3

51% of O CW visitors come from North A mer ica or Western Europe; they typically hold a

bachelor's or master's degree (66 %); and they are most frequ ently in ter ested in electrical

engineering, business, man agement, physics and mathematics.

55% of self learners come from North A mer ica or Western Europe; th ey typically hold a bach elor's

or master 's d egree (74%); 60% have 10 years or less professional exper ien ce; and th ey ar e mo st

frequently in terested in eng ineer ing, business/manag emen t, and sciences.

49% of students co me from North America or Western Euro pe; they typically hold bachelor's

degrees or high school d iplomas (62%); most attend 4 year colleg es or graduate schools (86%); and

nearly half ar e pursuing engineering d egrees.

42% of edu cators co me from North America or Western Eur ope; 81 % hold master's or doctor ate

degrees; 62 % hav e been teach ing for 10 years or less; and ed ucators ar e most fr equently interested

in eng ineer ing and scien ces.

d. OCW visitors eith er speak English as a nativ e languag e or read English as a second language v ery

proficiently.

36% of O CW visitors speak Eng lish as a native language.

84% of visitors who are no t native English sp eakers rate th eir proficiency at reading English as

"excellent" or "good."

66% of visitors hav e their browser language pref erence set to English and 13% to Chin ese.

e. The O CW site receives significan t traff ic from educational in stitutions, th e United States military, and

high technology co mpanies.

3,100 .edu or .ac domains gener ated mor e than 434,000 visits last year; the top 50 domains

generated more than h alf of th e v isits.

US military domains gener ated 22,000 visits last year, 11th most of any domain and below only

major commer cial ISPs ( e.g . Comcast, AOL, Road Runner).

In total, Intel, Boeing, I BM, Ray theon, Mo torola, and Hew lett-Packard gener ated nearly 35,000

visits last year .

f. Visitors using a w ide r ange of p latform and connection types are overwhelmingly satisf ied with th e

technical performan ce and usability of the site.

95% of visitors express satisf action with the site organization, visu al design, and performance.

Site enhancements requested in clude improved search function; downloadab le compr essed files of

all conten ts for a course; and discussion forums.

2. Us e

a. OCW use is centered on subjects for which MI T is a recognized leader.

Courses from Electrical Engineering and Co mputer Scien ce, Sloan School of Manag ement,

Mathematics, Physics, Economics and Mech anical Eng ineer ing togeth er repr esen t 35% of the

conten t published on the O CW site, and attr act 65 % of the tr affic.

43% of course visits were to engin eer ing courses, 24 % to science courses, 18 % to humanities and

social scien ce courses, 12 % to managemen t/business courses and 4 % to ar chitecture and urban

planning courses.

MIT OCW Evaluation Findings January 2005 4

Courses aver aged 9,520 per course v isits for th e year, 8 with a med ian of 5,978; the most visited

course r eceived 100,653 visits, the least v isited received 542 visits.

Course usage is largely deter min ed by subject, special con tent typ es, and conten t completeness.

b. Visitors larg ely use the site for prev iously iden tif ied scenario s of use.

Self learners most frequ ently use th e site to enhance person al knowledge (58%), keep curren t in

field developments (18%) or p lan a fu ture course of study (1 1%).

Students primar ily use the site to comp lemen t materials from a course they are taking (44 %),

enhance personal know ledge (32%) or p lan a course of study (12%).

Educators primar ily use the site to enhan ce personal knowled ge (25%), develop a course (23%),

prepare to teach a specific class (18%) or enhance r esearch ( 14%).

Near ly a th ird of edu cators hav e already adop ted mater ials fr om th e site, and over half ind icate they

plan to do so in the future.

c. Lecture notes, fu ll text read ings, assignments and sy llabi ar e identified by v isitors as the most useful

types of content.

65% of visitors identified lecture notes as b eing among the most valuab le types of conten t9; 42%

chose fu ll text r ead ings, 24 % selected assignments, 23 % cited syllabi, and 21% iden tif ied lecture

videos.

Educators found more value in sy llabi than other roles; students cited lecture v ideos as v alu able

more often that oth er roles; self learners found more v alu e in full text read ings than o ther ro les.

Calendar s w ere cited by only 4 % of v isitors as being among the most valuable types of content,

ranking last among content typ es.

d. Visitors are gener ally su ccessful at ach ieving their goals in coming to th e site, and are largely satisfied

with the bread th, depth and quality of site content.

91% of visitors are completely or somewhat su ccessful in achieving their goals for v isiting th e site.

Of feedback e-mail that expressed an op inion abou t the site, 97.5 % h as b een positive and 2 .5%

negative.

81% of visitors express satisf action with site sub ject matter breadth ; 73% express satisf action w ith

course mater ials d epth; and 89 % express satisfaction with course materials quality.

Visitors suggestions for site content improvemen ts include increased course conten t d epth,

additional courses, add itional curricular information, more video/audio content, and tr anslations of

site conten t.

e. More th an half of incoming MIT freshmen are awar e of the site and many indicate it influ enced th eir

decision to attend the Institu te; MIT upperclass undergraduates are almost all aware of the O CW site,

nearly three-quarters use the site for a range of educational activities, and n ear ly all feel th e site h as h ad

a positive effect on their studen t experience.

53% of in coming MIT fr eshman ar e awar e of the site; 12 % o f those aw are have mad e more than 15

prior visits; 16 % of those aw are indicate th at th e site influen ced th eir d ecision to attend MIT.

8

These figures include only the 511 courses published for the complete period of October 1, 2003 to September 30, 2004.

9

Respondents were asked to choose the three most valuable types of content.

MIT OCW Evaluation Findings January 2005 5

92% of MIT upperclass undergraduates ar e awar e of the MI T O CW site; 73 % of MI T upperclass

undergraduates report accessing th e site; of that 73%, 54 % u se the site at least mon thly.

MIT upperclass undergraduates, wh en asked to identify all w ays they hav e used th e site, most of ten

cited comp lementing materials from a current course (76%), planning a course of study (44%),

review ing materials from a pr eviously comp leted course (36%) and enhan cing per sonal knowledg e

(29%).

54% of MIT upperclass undergraduates rate th e impact of O CW on their studen t experience as

extr emely positive or positive ; a furth er 41% rate th e impact as moderately positive or

somewhat positiv e.

3. Impact

a. Visitors indicate th at OCW has already had sign ifican t impact on their teaching and learn ing and expect

even greater impact in th e future.

80% of site v isitors indicate the site has alr eady had ex tremely positive or positiv e impact on th eir

educational activities, and a furth er 18% reported moder ately positive to somewhat positiv e imp act.

Almost 88% of site v isitors expect extr emely positive or positive impact in the future, and near ly

12% expect moder ately positiv e to somewhat positive impact.

Site v isitors strongly agree th at th e site h as h elp ed th em b e more productiv e (81%), helped them

learn (88%), and increased th eir motivation to learn (80%) .

85% of edu cators strongly agr ee or agree they h ave improved their courses using O CW .

93% of visitors would reco mmend O CW to oth ers.

b. MIT O CW content h as b een widely d istr ibuted beyond th e MIT site through translations and mirror

sites.

MIT O CW translation aff iliates OOPS and CO RE ar e mirror ing O CW conten t in Taiwan and

main land Ch ina; a pilot mirror site has been established w ith OCW s assistan ce at Makerer e

University in Ug anda.

Unaffiliated mirror sites (http://www. scien tif ic- library .net, h ttp ://mit.handsbrain.co m/,

http://mit.blogchin a.com/) have been id entified in Russia and Ch ina.

Universia.n et has tr anslated 80 O CW courses into Por tuguese and 79 in to Span ish; CORE h as

selected 95 courses for translation into simplified Chinese an d completed 2 ; OOPS has selected 110

and comp leted 11 .

c. The open coursewar e model is being adop ted by institutions in the US and in ternationally, both though

affiliations w ith MI T O CW and independ ently.

MIT O CW is partn ering w ith groups of universities in the U nites States, Ch ina, Jap an and Spain

who are actively adopting th e open coursew are model.

Universities unaffiliated w ith MIT OCW in Fran ce and India are moving forw ard with

opencoursewar e projects.

MIT OCW Evaluation Findings January 2005 6

The Op enCourseWar e site is attr acting an increasing ly glob al aud ien ce of self learners, students and educators

includ ing a core group of return ing visitors who av erage mor e th an one visit a week to th e site. Self learn ers and

students ar e incorporating the site in to th eir learning routin es, as w ell as using th e site to p lan for fu ture learning.

Educators ar e emp loying th e site to improve their own subject matter expertise, plan courses, and prepar e

teaching mater ials; n early a th ird of educators hav e already adopted O CW mater ials for use in their own

teaching . Site visitors ar e high ly satisfied w ith materials th ey find, indicate that the site has already had

significan t imp act on their teaching and learning, and exp ect even greater impact in th e future. Additionally, the

site is provid ing significan t ben efits to internal aud iences at MIT, wh ich will support the sustain ability of the

project in th e long term. Op enCourseWar e is h aving a w ider imp act on the open sh aring of education al

resources, inspir ing numerous institutions and organizations to undertake similar projects, and aiding many in

these endeavors. These findings demonstrate continued progr ess toward the fulf illment of OCW s mission and

provide data with which w e can gu ide program planning and decision-making.

MIT OCW Evaluation Findings January 2005 7

II. Findi ngs: Access

The 2004 Program Ev aluation in cludes the f irst full year of d ata co llected for th e O CW project following the

launch of the pilot site in October 2003. A t the h ighest level, the data related to access show:

Traffic to th e O CW site is steady, incr easing, and progressiv ely g lobal.

Most users continu e to lo cate th e site v ia onlin e and offlin e media.

Visitor education al role distribu tion and profiles remain abou t the same.

OCW visitors eith er speak English as a nativ e languag e or read English as a second language v ery

proficiently.

The O CW site receives significan t traff ic from educational in stitutions, th e United States military, and

high technology co mpanies.

Visitors using a w ide r ange of p latform and connection types are overwhelmingly satisf ied with th e

technical performan ce and usability of the site.

A. MIT OpenCourseWare content access leve ls

1. Ove rall OCW site traffic

OVERALL SITE TRAFFIC

In October 2003, MIT Open CourseW are comp leted its p ilo t phase

with the publication of materials from 511 MI T courses. The In total, 2.3 million unique visitors came to the

site from No vem ber 1, 2003 to Octo ber 31,

publication of these courses w as accomp anied by significan t press

2004, gener ating nearly 4.2 million visits.

coverage, wh ich r esulted in a signif icant spik e in tr affic.

The site aver aged 348,000 visits per month;

By November 2003, this media-driven traff ic spik e had subsided, monthl y visits ranged from 235,000 in

and OCW had Novem ber 2003 to 523, 000 i n Octo ber 2004.

also fully

Figure 1. Monthly v isits

implemented its

web metr ics softw are, SiteW ise. Between

November 1, 2003 and O ctob er 31, 2004, OCW collected

600,000

its first fu ll year of site traffic d ata.

500,000 Visits

400,000

As seen in Figure 1, tr affic patterns in the f irst year appear

300,000

to be influ enced by th e Nor th American and W estern

European academic calendar, with lower traff ic levels

200,000

during December holidays and summer months, and higher

100,000

traffic during the spring school term. In September and

0

October 2004, MIT Open CourseW are r eceiv ed significan t

r

r

y

ril

ly

er

ry

r

ay

ne

st

ch

be

be

be

ar

Ju

gu

Ap

ua

ob

M

ar

Ju

nu

m

em

m

Au

M

br

ct

ve

ce

nation al and international coverage on CNN, resu lting in

Ja

pt

O

Fe

No

De

Se

dramatically in creased traff ic.

So ur ce: Site Wise

2. First time and returning OC W traff ic

FIRST TI ME AND R ETUR N TRAFFIC

The site aver aged 188,000 first time visitors

Throughout the p eriod from Novemb er 1, 2003 to O ctober 3 1, 2004,

per mo nth, which ranged from 153,000 in

the O CW site r eceived an aver age of 188,000 of monthly first time

Novem ber 2003 to 258, 000 i n Octo ber 2004.

visitors; as with over all traff ic, these figures (see Figure 2 below)

were r elatively stab le from January through May 2004, and slightly OCW r eceived an aver age of 24,000 retur ning

lower in June, Ju ly and August. Sh arp increases in Sep temb er and visitors per month, which ranged from 14, 000

in No vem ber 2003 to 37,000 in October 2004.

October coin cid ed w ith coverag e of the site on CNN.

Retur ning visitors wer e 11% of all visitors, and

As illustrated in Figure 3 below, a similar pattern ex ists for r eturning gener ated 46% of all visits; retur ning visitors

visitor levels, w ith an averag e of 24,000 monthly r eturning visitors averaged 6.8 visits per mo nth.

coming to the site.

MIT OCW Evaluation Findings January 2005 8

Figu re 2. Month ly first tim e vis itors F igure 3. Monthly retu rning v isitors

300,000 40,000

Monthly Return Visitors

35,000

250,000 First Time Visitors

30,000

200,000

25,000

150,000 20,000

15,000

100,000

10,000

50,000

5,000

0 0

r

r

y

ril

ry

y

r

r

ay

ne

st

ch

r

r

ry

ril

ry

er

Contact this candidate