Data Site
Resume:
MIT OpenCourseWare
Program Evaluation Findings Report
March 2004
For more information, please contact:
Stephen E. Carson
Evaluation Specialist
MIT OpenCourseWare
** ************* ******, *-***
Cambridge, MA 02139
Phone: 617-***-****
Fax: 617-***-****
Email: *******@***.***
MIT OpenCourseWare
Program Evaluation Findings Report
March 2004
Contents
I. Executive summary 1
A. About MIT OpenCourseWare 1
B. Evaluation design 1
C. Evaluation data sources 2
D. Summary of findings 3
1. Access 3
2. Use 4
3 Impact 4
II. Findings: Access 6
A. Site traffic 6
1. Historical context 6
2. Overall traffic level 7
3. First time and returning visitors 8
4. Visit frequency 8
B. Geographic profiles 9
C. User roles 9
D. Educator profiles 11
E. Technical profiles and performance 13
F. User awareness of OCW 14
1. Media 15
2. Peer referrals 15
3. Search engines 15
4. Awareness by geography 16
III. Findings: Use 18
A. Subject areas of interest to users 18
B. Uses of materials 19
1. Uses by educators 19
2. Uses by students 21
3. Uses by self-learners 22
C. User satisfaction 22
1. Usefulness by task 22
2. Usefulness by scenario 23
3. Content satisfaction 24
4. Site usability 25
5. User-recommended improvements 25
IV. Findings: Impact 26
A. User perception of impact 26
1. Impact on educators 27
2. Impact on students 28
3. Impact on self-learners 29
B. Adoption of OCW materials 29
C. OCW and users perception of MIT 30
MIT OCW Evaluation Findings March 2004 Page I
Appendices
1. Background on MIT OCW 31
2. Program Evaluation Methodology and Data Collection 34
3. Evaluation Indicator Worksheet 40
4. OCW Intercept Survey 49
5. OCW Supplemental Survey 65
6. Interview Protocol 80
7. Representative Suggestions from Users 85
Figures and Tables
Figure 1. OCW Historical Monthly Site Usage 6
Figure 2. OCW Site Visits: October - November 2003 7
Figure 3. OCW Site First-Time and Daily Returning Visitors (November 2003 8
Table 1. Summary Site Traffic Statistics (10/1/03 - 11/30/03 7
Table 2. User Frequency/Number of Visits 8
Table 3. Visitors by Geography 9
Table 4. OCW Visitors by Role 10
Table 5. OCW Visitors by Role versus Geography 10
Table 6. OCW Visitors by Role versus Educational Level 10
Table 7. Frequency of Visits by Role 11
Table 8. Prior Visits by Role 11
Table 9. Educator Experience by Geography 12
Table 10. Type of Institution Where Educators Teach 12
Table 11. Educator Areas of Interest/Expertise 12
Table 13. Desktop Operating System Usage 13
Table 14. Top 50 User Host Organizations 13
Table 15. OCW User Internet Connection Type by Geography 14
Table 16. OCW Site Performance - Satisfaction by Connection Type and Geography 14
Table 17. OCW Visitor Awareness 15
Table 18. Top 50 referring URLs 16
Table 19. Top Referring Search Engines 16
Table 20. Awareness Triggers by Geography 17
Table 21. Student and Self-Learner Areas of Interest 18
Table 22. OCW Content Use by Department (Subject Area) 19
Table 23. Educator Usage Scenarios versus Level of Experience 20
Table 24. Educator Scenarios by Geography 21
Table 25. Student Scenarios by Geography 21
Table 26. Self-Learner Scenarios by Geography 22
Table 27. Overall Success in Using OCW 23
Table 28. Educator Usefulness Ratings by Scenario 23
Table 29. Student Usefulness Ratings by Scenario 23
Table 30. Self-Learner Usefulness Ratings by Scenario 24
Table 31. User Satisfaction with OCW Content 24
Table 32. User Satisfaction with OCW Site Usability 25
Table 33. User Satisfaction with Additional Usability Attributes 25
Table 34. Perceptions of Impact by User Role 26
Table 35. Perceptions of Impact by User Role versus Frequency of Visits 26
Table 36. Attributes of OCW that Facilitate Impact 26
Table 38. User Plans to Return to OCW 27
Table 39. OCW Impact on Educators' Teaching Practices 27
MIT OCW Evaluation Findings March 2004 Page II
Table 40. Perception of Impact by Educators by Geography 28
Table 41. Perception of Impact by Students by Geography 28
Table 42. Perceptions of Impact by Self-Learners by Geography 29
Table 43. OCW Content Adoption by Educators 30
Table 44. User Perception of OCW Consistency with MIT Image 30
MIT OCW Evaluation Findings March 2004 Page III
MIT OpenCourseWare
Program Evaluation Findings Report
January 2004
I. Executive summary
A. About MIT OpenCourseWare
MIT OpenCourseWare (OCW) is a large-scale, web-based electronic publishing initiative, accessible on the Internet
at ocw.mit.edu. Through OCW, MIT makes its core teaching materials lecture notes, problem sets, syllabi,
reading lists, simulations, etc. openly available for non-commercial educational purposes. OCW publishes those
materials in standards-based formats for anyone with access to the Internet.1 OCW has a dual mission:
To provide free access to virtually all MIT course materials for educators, students, and individual
learners around the world.
To create an efficient, standards-based model which other universities may emulate to publish
their own course materials.
B. Evaluation design
To measure the success with which OCW is fulfilling its mission as well as to establish a thorough and continuous
feedback process that guarantees its improvement over time OCW conducts substantial ongoing evaluation, which
includes assessments of external access, use and impact (program evaluation), and internal effectiveness and
efficiency (process evaluation). This report presents findings from the program evaluation activities conducted from
October 1 through November 30, 2003. Appendix 2 describes the logic model that structures this evaluation, and
appendix 3 provides the evaluation indicator worksheet that connects the logic model to the data collection methods
we used in this evaluation. The evaluation probes three general areas of user behavior and user profile:
Access. Numbers and characteristics of OCW users, including their geographic locations,
educational backgrounds, methods of locating the OCW site, technical contexts through which
they access OCW, and how well the OCW technical architecture performs in supporting access.
Use. How visitors use the site and how well it meets their needs. Evaluation is based on
hypothesized scenarios of use (described in appendix 2) for the following user roles:
Educator. Educators are defined as users professionally employed in providing
o
instruction to others at any level. Educator users are of particular interest to OCW
because of the potential of reaching secondary audiences though educator adoption of
OCW materials.
Student. Students are defined as learners enrolled in a formal institutional program of
o
study at any level.
Self-learner. Self-learners are defined as learners not enrolled in a formal institutional
o
program of study at any level. Generally, learners in employer-sponsored professional
development programs are considered self-learners.
Impact. Outcomes resulting from OCW site use. In particular, this area of inquiry focuses on
OCW s impact on individual teachers and learners, as well as its impact on the open sharing of
educational materials.
1
See appendix 1 for more background information on the OCW initiative at MIT.
MIT OCW Evaluation Findings March 2004 Page 1
C. Evaluation data sources
OCW undertook this evaluation from October 1 through November 30, 2003.2 We used multiple data collection
strategies (an integrated portfolio approach ) that included the following data sources:
Web Analytics. Akamai, OCW s web hosting and content distribution network provider, captures
aggregate usage data such as page views, object views and user location. Akamai also offers a
more sophisticated analytic tool called SiteWise,3 which OCW employed starting November 1,
2003. Some measures in this report synthesize data from these two sources through correlations.
In particular, site visit and page view statistics for the month of October are extrapolated from
Akamai hit data; after November 1, site visit and page view information is a direct measure using
the SiteWise tool. All geographic traffic information is drawn from Akamai, due to its greater
accuracy. 4
Online intercept surveys. Between November 6 and 19, a survey tool invited (via pop-up window)
a random sample of 21,467 OCW visitors to complete an online survey.5 Of those prompted, 3573
people began the survey, and 1220 completed it fully, with a dropout rate of 66% and an overall
completion rate of 5.7%. The sample provides a margin of error of not more than 3%. While
overall completion rates roughly parallel high-level geographical distribution of OCW traffic, self-
learners were slightly more likely to complete the survey once started (as opposed to educators
and students), and partial completion rates were lower in North America and the Pacific region
(primarily Australia), indicating language played a role in survey completion.6 The intercept
survey is included in appendix 4.
Supplemental surveys. A supplemental survey was distributed to 600 individuals around the
world with emphasis on some target regions (Latin America, Africa, Asia and Eastern Europe) and
roles (primarily educators and students). 62 respondents started to fill out the survey, and 29
completed it, with a dropout rate of 53% and overall completion rate of just under 5%. This
sample set is not representative of the overall OCW visitor base, but provides qualitative insights
into the experiences and attitudes of OCW visitors in those target geographies. See appendix 5 for
supplemental surveys.
Interviews. Interviews were conducted with a small subset of people in various target groups and
geographies to gather textured qualitative data about the use and impact of OCW. Interviewees
were selected from those whose responses sparked the curiosity of the evaluation team. Members
of the OCW research team conducted twenty-five in-depth interviews with willing participants
from intercept and supplemental survey respondents, distributed across several target regions
(Latin America, Asia, Eastern Europe, North America) and educational roles (educators, students
and self-learners). The interview questions and protocol are included in appendix 6.
The data sources employed for this evaluation provide a rich statistical picture of site usage through the web
analytics and intercept survey, with additional qualitative information derived from the supplemental surveys and
2
Please note that OCW announced the publication of the 500th course on October 1, with an accompanying surge of media coverage and
publicity. This led to unusually high levels of site access and usage patterns during that period (particularly an unusually high number of first time
visitors to the site).
3
SiteWise tracks users anonymously via cookies, and so identifies unique visitors by browser; for the month of November, the SiteWise system
reported 5.6% of OCW traffic had cookie support disabled. SiteWise also relies on JavaScript; for the month of November, the SiteWise system
reported 0.6% of OCW traffic had JavaScript disabled.
4
Due to limitations in SiteWise regarding accuracy of geographical distribution data, the total visitors by region data shown is created from hit
ratios measured in Akamai tools, which are then applied to unique visitor data from SiteWise to generate the approximate breakdown of where
visitors originate. Data between these two systems correlates at a 98% level.
5
See appendix 4 for the complete text of the intercept survey. Note that the online surveys (intercept and supplemental, see below) are built with
research logic that dynamically presents a logical subset of the survey questions based on the respondents answers.
6
Self-learners made up 7.5% less of partially completed surveys than fully completed surveys; students accounted for 5% more of the partially
completed surveys than fully completed surveys; educators made up 2.5% more of the pool of partially complete surveys than fully completed
surveys. North American respondents accounted for 47.0% of completed surveys and 34% of partially completed surveys; Pacific region
respondents made up 1.4% of fully complete surveys and 0.8% of partially complete surveys; in all other regions, ratios were either statistically
equal or indicated a higher percentage of partial respondents by 3-6%.
MIT OCW Evaluation Findings March 2004 Page 2
interviews. Web analytic statistics combined with survey and interview data provide a complementary picture of site
access, use and impact, with no apparent contradictions.
D. Summary of findings
At the highest level, the program evaluation data show:
1. Access
a. OCW traffic volume is high, and there is a core of repeat visitors.
The OCW site recorded 718,000 visits between 10/1 and 11/30/2003 an average of almost
12,000 visits per day for that period.
Returning visitors account for around 25% of daily visits for the month of November.
Over 95% of OCW users plan to return to the site in the future.
Almost 10% of visitors report daily use of the site; a further 25% at least weekly use; over 40% of
visitors report more than 10 previous visits to the site.
b. OCW has attracted international attention, with over half the site traffic coming from outside North America.
45% of OCW visitors come from North America (USA/Canada).
Western Europe is the second most common point of origin (19%) and East Asia is third with
18%.
The Middle East and North Africa (1.6%) and Sub-Saharan Africa (0.4%) represent small but
measurable portions of OCW s traffic.
c. Educators, students and self-learners access the site extensively.
Numerically, self-learners predominate, representing almost 52% of visitors with an average of
over 6000 daily visits. The self-learners are most likely to come from North America (60% of
North American visitors).
Students represent approximately 31% of visitors or an average of over 3600 daily visits.
Educators represent over 13% of the visitors, an average of 1550 visits per day.
The OCW user base is well educated; almost 70% have earned a bachelors or graduate degree.
d. Educators from around the world visit the site; about half have less than five years teaching experience and
most often have expertise in electrical engineering and computer science, or business and management.
Educators represent a higher percent of visitors for several regions outside North America, e.g.
Latin America (18%) and Eastern Europe (20%).
Almost 49% of educators using OCW have less than 5 years teaching experience; the balance of
educator use is distributed across the remaining spectrum of experience levels.
55% of educators using OCW teach at 4-year colleges or the equivalent and their expertise is most
commonly focused in electrical engineering and computer science (26%) or business and
management (14%).
e. OCW s technical platform results in high levels of satisfaction with site performance across a wide range of
connection types.
Over 95% of current OCW users reported they were satisfied with the site performance.
Approximately 16% of OCW users accessing the site via dial-up connections reported only
slightly lower satisfaction at 94%.
MIT OCW Evaluation Findings March 2004 Page 3
f. User awareness of OCW comes via a range of channels.
Almost 63% of visitors became aware of OCW via online or offline media articles.
Over 25% of users report that they became aware of OCW through a colleague, peer, or teacher.
While search engines drive just over 10% of OCW traffic, the prevailing search phrases are
variants of OpenCourseWare rather than topic-based searches, indicating those searching
became aware of OCW through other channels.
2. Use
a. OCW use is centered on subjects for which MIT is a recognized leader.
Electrical Engineering and Computer Science course sites attract 34% of traffic for users who
accessed specific course materials on OCW while only accounting for 10% of the total courses
published.
Mathematics, Management, Economics and Physics sites account for an additional 26% while
representing a further 21% of courses published.
b. Users are largely confirming hypothesized scenarios of use for OCW.
Educators primarily use the site for planning, developing, improving and teaching courses or
classes (44%), and secondarily to enhance their personal knowledge (25%).
Students most frequently use the site to find subject matter and materials for use in conjunction
with a course they were currently taking (43%), and secondarily to enhance their personal
knowledge (39%).
Self-learners overwhelmingly use the site to enhance personal knowledge (80%).
No significant new scenarios of use emerged from the evaluation.
c. Users are largely satisfied with the quality, breadth and depth of content available; they find OCW useful in
supporting their educational activities, and are highly satisfied with the usability of the OCW web site.
In attempting to complete a specific scenario-related task, more than 90% of users reported they
were either completely or somewhat successful.
More than 92% of users express high levels of satisfaction with the overall quality of course
materials published on OCW.
79% of users are satisfied with the breadth of subject matter and course areas available on the site,
(currently less than a third of MIT s overall curriculum).
71% of users express satisfaction with depth and completeness of materials for courses available
on OCW.
Over 97% reported satisfaction with the site visual design and presentation of materials.
3 Impact
a. OCW users overwhelmingly find that OCW has, or will have, significant positive impact on both teaching and
learning activities.
Over 80% of all users report either positive impact or extremely positive impact, 18% report
moderate or some positive impact and less than 2% report no positive impact.
Over 95% of all users report an intention to return to OCW in the future.
Over 92% agree that they will recommend OCW to someone else.
MIT OCW Evaluation Findings March 2004 Page 4
76% of educators agree that OCW will impact their future teaching practices.
b. Educators have already reused OpenCourseWare materials or are planning to do so in the future.
Over 97% of educators expressed satisfaction with the quality of the course materials published in
OCW.
Over 47% have reused MIT OCW materials (or plan to); 41% may reuse materials in the future.
c. OpenCourseWare is perceived as being consistent with MIT s overall brand.
Over 84% of users report OCW is consistent or extremely consistent with their perceptions of
MIT.
Less than 2% indicate the site is somewhat or extremely inconsistent with their perceptions of
MIT.
At this early stage of the OpenCourseWare project, educators, students and self-learners from around the world
come to the OCW site and return in great numbers. Many educators worldwide are already incorporating MIT
teaching materials and practices into their own instruction. Site visitors are highly satisfied with the materials they
find and expect OCW to have significant impact on teaching and learning. These findings demonstrate early
progress toward the fulfillment of the mission of OCW and provide baseline data with which we can guide program
planning and decision-making, and against which we can measure improvement over time.
MIT OCW Evaluation Findings March 2004 Page 5
II. Findings: Access
OCW materials are meant to be accessible to users across geographies using various web browsers and accessing the
Internet through high- and low-bandwidth connections. OCW intends that all users encounter a reliable technical
infrastructure, and have technical access to the full range of content on the site. OCW also engages in an ongoing
communication effort (newsletter, press relations) to make educators, students and self-learners aware of the site
through a variety of channels and media. The current evaluation provides a baseline measure of usage levels, user
profiles, and awareness channels.
A. Site traffic
1. Historical context OCW traffic volume is high,
and there is a core of repeat visitors.
The MIT OCW site was first made available to
The publication of 456 course sites from June through
the public in late September 2002. At that
September 2003 (in addition to the 50 proof-of-concept sites),
time, there were 32 courses published on the
and the ensuing press coverage, resulted in a dramatic initial
proof-of-concept site. A large initial spike in increase in OCW site usage.
site traffic accompanied the October 2002
The OCW site recorded 718,000 visits between 10/1 and
public announcement, and throughout this
11/30/2003 an average of almost 12,000 visits per day for
initial year, the site received significant traffic. that period.
Beginning in June 2003 the OCW team
Returning visitors account for around 25% of daily visits for the
published an increasing number of courses, month of November.
continuing through late September 2003, when
Over 95% of OCW users plan to return to the site in the future.
the 500th course was published. As the
Almost 10% of visitors report daily use of the site; a further
number of courses grew, the level of traffic on
25% at least weekly use; over 40% of visitors report more than
the site grew rapidly, as shown in Figure 1
10 previous visits to the site.
below.
Figure 1. OCW Historical Monthly Site
Usage
9,000,000 600
OCW Traffic (Page Views)
8,000,000
500
Total Courses Available
7,000,000
6,000,000 400
5,000,000
300
4,000,000
3,000,000 200
2,000,000
100
1,000,000
0 0
Launch Nov Dec Jan Feb March April May June July Aug Sept Oct Nov
200*-****-**** 200*-****-**** 200*-****-**** 200*-****-**** 2003 2003
Source: Page Views: Akamai and SiteWise
Courses Available: OCW CMS
MIT OCW Evaluation Findings March 2004 Page 6
2. Overall traffic level
Overall site traffic statistics from October 1 through November 30, 2003 (see Table 1) show a total of just under 12
million page views, from over 718,000 visits. An average of nearly 12,000 visits to the OCW site occurred daily
during that period.7
Table 1. Summary Site Traffic Statistics (10/1/03 - 11/30/03)
Daily Average During
Period
Site Traffic Statistic Total For Period
8
Website Page Views 11,738,898 192,441
Website Visits 718,866 11,785
Source: Akamai and SiteWise
Traffic in early October 2003 reflects the high media and public interest as the site reached the 500-course
milestone, and traffic levels in late October and November are expected to more closely reflect the activity of core
user groups.
Figure 2. OCW Site Visits: October - November 2003
70,000
60,000
Total Daily Visits
10-day rolling average
50,000
40,000
30,000
20,000
10,000
0
10/1/03 10/6/03 10/11/03 10/16/03 10/21/03 10/26/03 10/31/03 11/5/03 11/10/03 11/15/03 11/20/03 11/25/03 11/30/03
Source: Akamai and Sitewise
As illustrated in Figure 2, the number of daily visits was somewhat consistent through the workweek, with a distinct
pattern of lower usage on weekends. Many users interviewed, particularly self-learners, reported accessing the site
through higher bandwidth links available at work or their place of study (versus home). One self-learner interviewed
in Argentina described how he frequently arrives at work 30 minutes early to browse the web (including OCW) and
will sometimes print or download materials to review in more depth outside of work hours.
7
Due to a transition in web analytics tools that occurred during the month of October, raw usage and geographic data was being captured in the
basic Akamai analytics product. This data was correlated with November SiteWise usage and visit data to produce an integrated view of daily site
visits from October 1 Nov 31. Additional usage data related to visitor technology, referring URLs, entry points, and department and course
section level visit activity were captured in SiteWise starting November 1.
8
Site page views only counts HTML page views. Approximately 66% of the content on the OCW site is contained in PDF document format
(approximately 5000 HTML pages vs. approximately 10,000 PDF and other format documents and other materials). Thus, the page view number
is a conservative measure of total site activity.
MIT OCW Evaluation Findings March 2004 Page 7
3. First time and returning visitors
OCW began implementation of more sophisticated web site metrics software than previously used, starting October
1, 2003, and we are still accumulating the data necessary to support accurate analysis of returning versus first-time
visitors to the site. Significant numbers of visitors counted as first-time in this measurement have likely visited the
site prior to the October 1st implementation. The statistics for November (Figure 3), however, give an early
indication of first time versus daily returning visitor traffic, with a strong flow of around 5,000 first time visitors to
OCW per day.
Figure 3. OCW Site First-Time and Daily Returning Visitors (November 2003)
12000
Daily Returning Visitors
First-Time Vistors
10000
8000
6000
4000
2000
0
11/1/2003 11/8/2003 11/15/2003 11/22/2003 11/29/2003
Source: SiteWise
Measurement of returning visitors shows a solid base of daily returning visitor traffic, running at an average of
around 1700 visitors per day for the month of November. This is supported by the intercept survey results in Table
2, which show a significant base of returning visitors.
4. Visit frequency
Over 57% of all survey respondents report that this was not their first visit to the site. Of the returning visitors, over
40% report having visited OCW on more than ten occasions previously, indicating a strong core of longer-term
repeat visitors. The most common frequency of use for this group is weekly visits to OCW, with 23% of all repeat
visitors reporting more than 10 visits based on a weekly basis.
Table 2. User Frequency/Number of Visits
Number of Times Previously Visiting
% of
Once 2-5 Times 6-10 Times > 10 Times
Users
Visit Frequency
This is the first time 42.3% N/A N/A N/A N/A
Daily 8.9% 1.6% 3.3% 1.4% 8.8%
Weekly 24.8% 1.4% 10.2% 8.2% 23.0%
Monthly 9.8% 0.1% 6.3% 5.1% 5.1%
Occasionally (
Contact this candidate