Tag Archives: E-Learning / Distance Learning

In praise of the open-book exam

Whether or not students should be required to take proctored exams in an online course has been debated for as long as there have been online courses and I suspect it will not be resolved anytime soon. Nevertheless, there has been a good deal written about the subject that is worth considering.

Open book online exam

CC-BY-NC by Jared Stein on Flickr

A recent study compared proctored and non-proctored student outcomes in online courses. Lee William Daffin, Jr. and Ashley A. Jones from Washington State University, found that with the non-proctored online exams, students took about twice as long to complete their exams and performed between 10% and 15% better than students taking proctored exams.

Despite the instructions that the exam was to be taken closed-book, we might reason that students spent the extra time looking up answers in the text or searching online. Again, this brings up the issue of how to prevent cheating in an online course without requiring students to visit the testing center.

Daffin and Jones suggest trying the following methods to discourage cheating:

  • Explaining the technology used to discourage cheating (e.g. Safe Assign, Bb Monitor)
  • Creating a large enough test bank from which to draw questions
  • Randomizing both questions and answers
  • Including essay questions
  • Posting class and/ or institutional policies and penalties for cheating
  • Shortening the time frames to complete tests
  • Having students sign Code of Conduct / Honor statements

Daffin and Jones offer yet one more option…

A final possibility is to make online exams open notes/book from the start but increase the difficulty of such exams so that they are not simply testing the recollection of facts (Feller, 1994; Williams & Wong, 2009; Stowell, 2015). Though students would be permitted to utilize outside sources, eliminating student misconduct issues, they would still need a good understanding of the material to be able to accurately apply it and could not simply look up answers as they took the exam. – L.W.Daffin & A.A.Jones

In another study, Beth Johanns, Amber Dinkins, and Jill Moore, from Indiana State University, looked at the research around open-book vs closed-book exams and found there are some unique advantages for student learning with both formats. Open-book exams, in particular, were found to help learners develop critical thinking skills. While the closed-book required students expend more time and effort in preparation for the exam, they were also more likely to cram and therefore rely on short-term memorization. On the other hand, open-book exams were shown to engage learners more deeply by encouraging them to gather and critically analyze information – oftentimes from multiple sources.

In their study, Johanns, Dinkins, & Moore explored yet a third type of exam – collaborative testing. In this approach, students prepared for a closed-book exam but were permitted to collaborate with other students while completing the examination. This approach has been shown to promote creative thinking and increased metacognition.

Finally, it is clear there are both opportunities and challenges with any single approach to online assessment. Perhaps the best solution is to provide multiple options. Rather than requiring students to schedule weekly visits to the testing center, consider reducing the closed-book to the midterm and/or final exams and maybe mix things up with open-book and collaborative exams in the interim.

References:

Daffin, Jr., L.W., & Jones, A.A. (2018). Comparing student performance on proctored and non- proctored exams in online psychology courses. Online Learning, 22(1), 131-145. doi:10.24059/olj.v22i1.107

 

Johanns, B., Dinkins, A., & Moore, J. (2017). A systematic review comparing open-book and closed-book examinations: Evaluating effect on development of critical thinking skills. Nurse Education in Practice, 27, 89-94. doi.:10.1016/j.nepr.2017.08.018

 

 

Advertisements

Whether synchronous online course activities meet accessibility guidelines may depend on how you use them.

Man web-conferencing using an iPhone

CC-BY Public.Resource.Org on Flickr

Adding synchronous activities to your asynchronous online class can help to foster social presence and instructor immediacy. Using such tools as Google Hangouts, Skype, Zoom, etc. can add an element of interactivity and engagement to the online course that might otherwise be missing. Web-conferencing can be used to prepare students for upcoming assignments, review recent material, offer virtual office hours, and more…

but what about accessibility?

Our campus recently adopted a policy that essentially requires all online instruction meet accessibility guidelines. While our facilities department is well versed in ensuring campus physical learning spaces meet ADA requirements, faculty may struggle with how to approach accessibility within the virtual learning environment. When the disabilities office notifies an instructor that a student in their class needs further accommodation they also provide the direction and support necessary to meet the student’s needs.

Universal design vs. Accommodation

To accommodate a specific student with a disability is to retrofit instruction to meet that student’s needs. However, by applying Universal Design principles we anticipate a wide range of learners’ needs and design the instruction and learning environment accordingly.

Universal design for learning is achieved by means of flexible curricular materials and activities that provide alternatives for students with differing abilities.

Employing universal design principles in instruction does not eliminate the need for specific accommodations for students with disabilities. There will always be the need for some specific accommodations, such as sign language interpreters for students who are deaf. However, applying universal design concepts in course planning will assure full access to the content for most students and minimize the need for specific accommodations. DOIT Center

So how can we go about ensuring online synchronous activities are accessible to all students regardless of ability? Part of the answer may depend on how the synchronous tools are being used.

Several web-conferencing solutions permit attendees to participate by dialing in on their phones rather than using computer VoIP. A phone connection may be used with other devices and services – such as Telecommunications Relay Services (TRS) – to translate voice to text. Also by dialing in for the web-conference rather than using the browser or app, students can avoid exceeding their data cap when they are not connected via WiFi.

When presenting to the entire class in real-time, provide links to accessible digital materials in advance for participants to review and follow along during the session. Provide detailed descriptions and annotations to go along with any images, charts and graphs used within the presentation. Take the time to describe the significance of each graphic in detail during the presentation. These practices will also help when transcribing or captioning the session for recordings after the event.

By using the chat feature provided in most web-conferencing solutions all participants can submit questions. Assign a moderator to keep track of the chat window and type in the presenter’s answers to posted questions. Whether a question is being asked by someone in the lecture hall or via chat, it is always a good idea to repeat the question before responding.

Rather than using only video and voice features, offer chat and/or text messaging as options for conferences with individual students. Be sure the information on how and when to connect is also made accessible. Web links should provide a description and any further instructions in plain text (as opposed to images, arrows, highlighted text, etc.) to be read by screen-reader software. Include a phone number in the invite as well in the event description for hearing impaired students to access via TRS services.

By taking the time to create accessible web-conferences we not only accommodate the student with a visual or hearing impairment, we make the presentation more usable for all participants.

References:

DOIT Center: Universal design vs. accommodation

Federal Communications Commission: Telecommunications Relay Service (TRS)

W3C: Understanding WCAG 2.0 – Time-based Media

IBM Human Ability & Accessibility Center: Overcoming accessibility challenges of web-conferencing

 

Closing the gap between online and classroom student outcomes

For the past few years community colleges have shifted their focus from one of access to one of completion. Offering online programming is a great way to provide access to higher education but closing the gap between online and classroom student outcomes is an ongoing challenge.

Student studying at computer

eLearning CC-BY Wolfgang Greller on Flickr

Efforts on bridging the gap have mostly centered around learner characteristics: GPA / SAT scores, whether students have previous experience with online learning, their capacity for self-regulated learning (self-efficacy, time management, organizational skills), etc.. Although these can be helpful indicators in predicting online student achievement, another approach that offers promise focuses on the at-risk course.

Ferris State University has been offering Structured Learning Assistance (SLA) for the past 25 years focusing not on the at-risk student, but on the high-risk for failure course. Although this has not been offered as an online option the impact on student success has consistently meant better than a 10% higher pass rate than of those sections without supplemental instruction. By focusing on courses with a history of high failure and withdrawal rates SLA has been able to support students with an additional 45 hours of supplemental instruction.

A recent study at Borough of Manhattan Community College at CUNY looks at online course-level predictors of learning outcomes (Wladis et al 2015). The study found that there was a significant gap in course completion between online courses taken as electives and those that were required for a given major. Also lower-level courses had a much higher attrition rate than higher-level courses. The study suggests that interventions such as embedded supplemental instructional support (tutoring, mentoring, advising, extra technical assistance) within the more challenging courses could significantly improve – and possibly even eliminate – the performance gap between online and face-to-face outcomes.

A few years ago I attended the Online Learning Consortium (OLC) conference and sat in on a panel presentation describing strategies to keep students enrolled in online learning. The panel was represented by faculty and staff from Penn State World Campus. The strategies included embedding tutors in some of the more challenging online courses. A single embedded tutor might support as many as eight sections of the course.

Tutors meet virtually with students one-on-one or with groups by appointment as many as five to six days a week – including evening and weekend hours. The tutors also scheduled “drop-in sessions” when they would go over some of the more challenging concepts, answer questions, and provide more detail on upcoming course assignments. In addition to the virtual meetings, tutors posted helpful tips on study skills and supplemental web resources. The goal of the initiative was to increase retention by 2% per year over a five year period. However, the results showed a 75% reduction in withdrawal and late drops, and a 15% reduction in course failure rates.

Implementing an online supplemental instruction program sounds like a big undertaking but focusing on the most challenging online courses sounds like a great place to begin.

References:

C. Wladis, K. Conway, A. C. Hachey (2015). Using course-level factors as predictors of online course outcomes: A multilevel analysis at a U.S. urban community college. Studies in Higher Education. Vol 42 (1). Taylor & Francis Online. http://www.tandfonline.com/doi/abs/10.1080/03075079.2015.1045478

Stitz & Zeager Open Algebra 3rd Edition

Can OER improve learning outcomes?

The June issue of the International Review of Research in Open and Distributed Learning (IRRODL) is dedicated to Open Educational Resources (OER). Several of the studies focus specifically on OER and student learning outcomes.

In, Exploring Open Educational Resources of College Algebra, Marcela Chiorescu, of Georgia College describes a case-study in which the instructor of a hybrid college algebra course switches from commercial software and textbook after the first two semesters, to an open text and supplemental low-cost software for a semester, and then back again to the commercial content.

For the semester using OER, Chiorescu estimates students collectively saved over $13,500  – spending approximately 75% less than students using the commercial text and software. Certainly one of the main advantages of adopting OER is lowering the costs to students. However, there may be other advantages more closely tied to student success.

Analyzing the grade distribution over the four semesters, the percentage of students earning a grade of C or better was significantly higher at 84.3% for the students enrolled in the semester using OER over the previous or subsequent semesters. The percentage of students earning an A in the course was also higher for those using OER. In addition, the OER sections reported significantly fewer withdrawals.

The decision to return to commercial content after one semester was related to technical issues with the low-cost software (quizzes locking up and slow download speeds). Chiorescu was also concerned that the low-cost software was not comparable in to the commercial version, due to a “lack of resources”. To compensate for the deficiency of course materials, she developed a LibGuide to accompany the course including supplemental videos and tutorials.

In another study: The Impact of Enrollment in an OER Course on Student Learning Outcomes, Kim Grewe and Preston Davis, of Northern Virginia Community College, compared learning outcomes for students enrolled in an online history course using OER with a similar number of non-OER sections over two semesters. The study took into account student cumulative GPA and, as expected, found a correlation between prior academic achievement and student course achievement. However, an even stronger correlation was found between student achievement and OER section enrollment.

Both studies build upon previous research looking at the efficacy of OER and student achievement (Grew & Davis) finding that students enrolled in courses using OER perform as well, if not better, than students enrolled in non-OER courses. In addition, OER supported courses are more affordable and students are more likely to enroll in a higher number of credit hours per semester – and thereby, achieve their academic goals in a more timely manner.

Another subtle, but important, takeaway was the use of LibGuides to supplement OER textbooks. One of the challenges of adopting OER is that the open textbook may not include all of the supplemental materials that oftentimes accompany the commercial texts. LibGuides offer the opportunity to engage the academic librarian in the course design process and potentially improve the overall quality of OER course offerings.

References:

College Algebra, 3rd edition by Stitz C. and Zeager, J. (2013) Available at http://www.stitz-zeager.com/

CHIORESCU, Marcela. Exploring Open Educational Resources for College Algebra. The International Review of Research in Open and Distributed Learning, [S.l.], v. 18, n. 4, jun. 2017. ISSN 1492-3831. Available at: <http://www.irrodl.org/index.php/irrodl/article/view/3003/4223>. Date accessed: 26 Jun. 2017. doi:http://dx.doi.org/10.19173/irrodl.v18i4.3003.

GREWE, Kim; DAVIS, William Preston. The Impact of Enrollment in an OER Course on Student Learning Outcomes. The International Review of Research in Open and Distributed Learning, [S.l.], v. 18, n. 4, jun. 2017. ISSN 1492-3831. Available at: <http://www.irrodl.org/index.php/irrodl/article/view/2986/4211>. Date accessed: 26 Jun. 2017. doi:http://dx.doi.org/10.19173/irrodl.v18i4.2986.

 

 

 

Pittsburgh Bridges by Don O'Brien CC-BY on Flickr

Navigating the online course

After several trips to Pittsburgh in the past few years, I have learned to turn off my GPS navigation system while driving through the downtown area. With its tall buildings, numerous highway exits, over-passes and bridges, my GPS loses the signal and begins offering less than helpful suggestions at a time when the importance of a clear useable navigation system is most critical.

Pittsburgh Bridges by Don O'Brien CC-BY on Flickr

With its tall buildings, numerous highway exits, over-passes and bridges, my GPS loses the signal and begins offering less than helpful suggestions at a time when the importance of a clear useable navigation system is most critical. Pittsburgh Bridges by Don O’Brien CC-BY on Flickr –

A colleague shared with me an incident with a student requesting a refund when, after week four of an eight week math course, they had yet to log into the publisher’s course site. According to the student they had logged into their Blackboard course and perused the course space but were unaware of the need to log into the supplemental publisher’s site, where the quizzes were to be completed. Of course, by week four, they were already halfway though the course and had missed several quizzes. When the instructor suggested they drop the course, the student stated they were not made aware of the second site and therefore should not have to pay for the course.

Unfortunately, this isn’t the first time I’ve heard these kinds of concerns and it seems to me that this is yet another example of how important it is to make clear the expectations early on in the semester or term. One way to help avoid these problems is to consider the course interface from the student’s perspective.

In a Quality Matters research project, Does Findability Matter? – Findability, Student Motivation and Self-efficacy in Online Courses (2012), researchers found that time-on-task, student self-reported difficulty and frustration levels were significantly higher for students enrolled in control group (non-QM-recognized) courses. On the other hand, students enrolled in the experimental group (QM recognized courses) reported a significantly better experience in regards to ease of finding course materials and assignments. The were more likely to enjoy their course experience and to recommend the course to friends. Instructor ratings were higher and frustration levels, lower than for the control group.

“Findability” was rated using QM Standard 6.3 – Navigation throughout the online components of the course is logical, consistent, and efficient.

When preparing the course ask a colleague and / or student(s) who are unfamiliar with the course, to look over the homepage and provide helpful suggestions. Here are some questions for which they might provide feedback…

  • Is it obvious where to begin?
  • Can you easily find the course syllabus?
  • Is it clear how to contact the instructor, when they are available and when students can expect a response?
  • Are the course goals and objectives clearly stated?
  • Is the schedule of assignments and course activities easy to find and understand?
  • Can students easily navigate to ancillary materials (e.g. publishers quizzes)?
  • Can you see at first glance how to find both technical and course-related assistance?

Keep in mind that people are different and will intuit where to find things by their own experience with similar interfaces. Although you may feel that putting all of the important information in one place (such as the syllabus) should suffice, your students may benefit by finding the same information in multiple locations – or rather – multiple paths to the same information.

These considerations may seem obvious but getting a second or even third perspective can help to ensure students can easily navigate the course and focus on the learning, as opposed to getting lost along the way.

References:

Simunich, B., Robins, D., Kelly, V. (2012) Does findability matter?. Quality Matters.org https://www.qualitymatters.org/files/webform/Quality%20Matters%202012%20Findability.pdf

CC-BY-NC-SA by Luke Chan on Flickr

From Community College to College community

When students drop courses at our community college, we ask them to complete a course drop survey form explaining their reasons for dropping.  The majority state for “personal reasons” the second most frequent response is for “other” reasons. Apparently we are not unique. Neal Raisman recently shared on his blog – “Great Service Matters” – the results of a study of why students leave college. Many of these students at the point of departure state they leave for “personal reasons”.

CC-BY-NC-SA by Luke Chan on Flickr

CC-BY-NC-SA by Luke Chan on Flickr

In the study, 864 students were interviewed after at least six months following their leaving college. The break in time was intended to give students an opportunity to be more reflective and open about their reasons for leaving.

Twenty-six percent of students reported they left because the “college doesn’t care”. Another twenty-four reported “poor service” as their main reason for leaving – which could be interpreted as another way of saying that the college doesn’t care. Together, these two responses account for fifty percent of students leaving school.

That’s huge.

In a previous post I shared the results of our own student survey of online learners asking what we could do to help them be more successful. It was clear from these student responses that they valued courses where the instructor was available and accessible – in other words – cared about them and how they were doing in the course.

What does it mean for a college to care?

Raisman talks about the importance of customer service. Certainly responsiveness is an important component to quality customer service, including something as obvious as having people available to answer calls, emails, texts, etc. in a timely manner. Listening to what students have to say, and then putting ourselves in their shoes is key to responsive and caring customer service.

Inclusiveness is another essential consideration. Many students at the community college are first generation college students. Their parents, friends and family may not be in a position to advise them about college life, expectations, and what it takes to succeed. Keeping this in mind, we need to consider ways of connecting students to the campus. For on-campus students this may mean student organizations or study groups. For online students we need to consider virtual connections that may leverage social media, or collaborative class projects, virtual office hours, etc. With more than 80% adoption of mobile technology by college students, there has never been a better time to leverage social media as a means of connecting all students to the college community.

Student support is more than a responsive friendly help desk or online tutoring – it includes a creating a sense of community for both the on-campus and the online student. If we are to succeed in retaining our distance learners,  the community college must become more of a college community.

Antique Video Camera

Students ask for more video with their learning

A few years back we invested in the streaming video service – Films on Demand. The service integrates with our campus portal and the Lakeland Library to permit faculty the means of integrating educational video into their course materials. Compared with the former system of ordering individual media (VHS, DVD), this has proved to be a very good investment. Nevertheless, the level of adoption has been less than expected.

Fims on Demand - Collections

Films on Demand – Collections

In our recent survey of online learners, students reported that they would like to have more video content in their online courses. Streaming video is becoming an extremely popular way for students to view content as is evidenced by the rate of adoption of such services as iTunes U and Khan Academy.

According to a recent article in Inside Higher Ed reporting on thePearson Higher Ed Survey on Student Mobile usage

“Eight in ten (83%) college students regularly use a smartphone, up significantly from 72% in 2013.  Smartphones are now close to laptops (89%) as the mobile device students are most likely to use on a regular basis.”

Perhaps the increased use of smartphones by students helps to explain the demand for more video content. Although I enjoy reading journal articles and other text-based material on my iPad because of the size and orientation, I am not a big fan of reading text on my iPhone. That being said, the iPhone works great with streaming video – especially with WiFi available both at work and home.

iPhone CC-BY-NC-SA by Alex Bartok on Flickr

CC-BY-NC-SA by Alex Bartok on Flickr

Films on Demand is available to both faculty and students and although we have promoted its use primarily with the faculty – demonstrating how to search for relevant content from among the more than 15,000 titles and 200,000 plus segments – students also may access these materials and find value in searching through the collections themselves.