*Note: I have changed my research question to the following quantitative research question: Does multimedia teaching methods affect a student’s academic success in elementary school?

Article #1: Achievement Effects of Embedded Multimedia in a Success for All Reading Program

This study examined the incorporation of video content (embedded multimedia) within teachers’ lessons to see if embedded multimedia enhances the effectiveness of beginning reading instruction.  The study design was a year-long experimental study that used a cluster randomized trial design with random assignments of schools comparing first graders who learned beginning reading through the Success for All reading program either with or without short video modules.  The Success for All reading program is designed for beginning readers to help them with “…letter sounds, sound blending strategies, vocabulary, and and comprehension strategies” through visual models (Chambers, Cheung, Gifford, Madden, & Slavin, 2006, p. 233).  There were 450 first grader participants that attended one of the ten different inner-city Hartford, Connecticut elementary public schools (all within the same inner city district), and 394 completed the pre-tests and post-tests.  62% of the students were Hispanic, 35% were African American, and 3% were White.  Almost all of the students were qualified for free or reduced-price lunches.

In regards to the data collection methods, the study consisted of an experimental group and a control group.  The experimental group incorporated the use of multimedia content, while the control group did not use any multimedia content.  The schools were randomly assigned to either the experimental group or the control group.  The experimental and control group were about equal in terms of having about the same  number of schools, number of students, percentage of students, qualified for free lunch, ethnicity, and percentage of students whose first language is not English.  The experimental group used multimedia content included in teachers’ every day ninety minutes Success for All reading lessons, and included thirty seconds to three minute skits and other multimedia components incorporated with the teachers’ lessons to demonstrate reading, comprehension, and vocabulary strategies to the students.  The control group (no embedded multimedia) schools used the every day ninety minutes Success for All reading program by the teachers using picture cards to demonstrate the letter shapes and vocabulary in the student books, as well as activities and games to help improve a student’s reading and comprehension skills.  The participants were given the Peabody Picture Vocabulary Test and the Word Identification sub-test from the Wood-Cock Reading Mastery Test-Revised as pretests to the study.   For the post-test, the participants were given a reading fluency test from the Dynamic Indicators of Basic Early Literacy Skills and three scales from the Woodcock Reading Mastery Tests-Revised (Word Identification, Word Attack, and Passage Comprehension).

In term of data analysis, the data were analyzed with hierarchical linear modeling with students within the schools to cluster the randomized design.  Classroom-level analyses were not conducted because the participants changed reading teachers over the course of the year.  The condition was the independent variable, and the Wood-Cock Reading Mastery Test-Revised sub-tests and the Dynamic Indicators of Basic Early Literacy Skills fluency test were the dependent measures.  The Peabody Picture Vocabulary Test and the Word Identification sub-tests were used as covariates to help regulate for starting differences between the experimental and control groups so that statistical power could be raised.  Analyses were also performed for both the overall sample and for a Hispanic subsample.

The researchers of this study found only one significant experimental-control difference as conclusions to this study.  The only significant experimental-control difference was with the Word Attack outcome measure in which the experimental group scored higher than the control group on the Word Attack test.  This finding supports past theoretical expectations in which it was believed that students who were given multimedia components dealing with letter sounds and sound blending would have a greater effectiveness of beginning reading instruction than no multimedia components.  The control group scored higher than the experimental schools on Word Identification, Passage Comparison, and the Dynamic Indicators of Basic Early Literacy Skills (DIBELS).  Even though it was expected that the Hispanic participants would not benefit the same from the embedded multimedia treatment as the other participants, the multimedia effects for Hispanic participants were almost the same as the other participants (mostly African-American).  Thus, no interaction between ethnicity and treatment were found.

In regards to whether or not the conclusions are valid given the study design, I feel that even though the data analyses were thorough and detailed, this study is somewhat weak because it only measured the achievement effects of multimedia content on an individual’s reading program.  As the researchers suggest, I feel that it would be important to examine the effect multimedia components have on teaching practices and whether or not multimedia improves teachers’ performance of the reading program to students.   I feel that there are also several threats to internal validity given the study design.  One threat of internal validity that pertains to this study is the selection of participants.  It is not possible to say the conclusions are necessarily valid because researchers only studied ten schools in Hartford, Connecticut.  In addition, the majority of the participants were either African-American or Hispanic, with only a minority percentage of white participants and from a lower socio-economic background, which does not give a representative sample of the total population of first grade students around the United States.  Maturation is another threat to internal validity in this study due to the possible mental developmental changes in the participants over the one-year duration.  Treatment replication is another threat to internal validity because this study was only conducted once instead of several times, which could lead to invalid results.  Diffusion of treatment can also be a threat to internal validity of this study because the student participants may have been talking about the multimedia components within the school, which could impact the results.  This could result in the participants taking the reading assessments to already have ideas before taking the assessment from talking with other participants who took the measurement before they did. Even though I feel that this study used several valid assessments to measure what it was intended to measure (essential reading skills), I feel further research with a more representative sample needs to be conducted.

Article #2: Science Education in Primary Schools: Is an Animation Worth a Thousand Pictures?

This study examined fourth grade and fifth grade elementary school teachers’ methods for incorporating animated movies, the teachers’ views about the role of animation in improving fourth grade and fifth grade elementary school-aged students’ thinking skills, and the effect of animated movies on fourth and fifth grade students’ conceptual understanding of science and their reasoning ability.  The study design was experimental, and consisted of an experimental group and a control group.  There was a pilot study to create the reliability and validity of the research tools.  The main study was performed to examine the fourth grade and fifth grade school teachers’ methods for incorporating animated movies, the teachers’ views about the role of animation, and the effect of animated movies on fourth and fifth grade students’ learning.  The schools in the sample were separated into experimental and control groups according to the school principal and the science teachers’ preferences.  Stratified sampling was used to make sure students from different demographics and age groups would be represented in the sample.

The participants were from primary schools in the central part of Israel.  1,335 primary schools students were separated into experimental and control groups.  The experimental group students included 926 students from five elementary schools, in which there were 435 fourth graders and 491 fifth graders.  The control group student participants included 409 students from two elementary schools, in which there were 206 fourth graders and 203 fifth graders.  The student participants gender distribution was equal, meaning there were 50% girls and 50% boys between both the experimental and control group.  Approximately 11% percent of the student participants’ parents’ career choices involved a scientific field, such as medical doctors, scientists, and engineers.  12.8% of the student participants’ participated in extracurricular activities that related to science education.  In terms of the teacher participants, there were 15 science teachers who incorporated who incorporated animated movies into their teaching.  All of the science teacher participants’ were females, 62% had a Bachelor’s of Education degree, and 85% had taught for more than 10 years.

In regards to data collection methods, both quantitative and qualitative methodologies were used.  Inform discussion was performed with the science teachers’ participants during the breaks between classes.  Five science teachers were selected to the experimental group in which they shared their beliefs and experience in their classrooms about their teaching style, and their thoughts about incorporating animated movies to improve elementary school-aged students’ thinking skills.  Data was also collected through surveying teachers’ methods for incorporating web-based educational technologies before and after the BrainPop animated movies program (three to five-minute animated movies that explain scientific concepts, interactive quizzes, and experiments that correspond with Israeli national science education standards).  Pre- and post-questionnaires were given to the experimental group student participants and control group student participants at the beginning and at the end of the school year.  The questionnaire used was the Science thinking skills, in which two versions were included, one for fourth grade student participants and one for fifth grade student participants, according to Israeli national standards and topics.  The questionnaires for both grade levels included the same questions, but in a different order.  Both questionnaires included looking further into students’ understanding of scientific ideas (i.e., motion and forces, life on earth, environmental issues) through multiple choice questions and true/false questions that mandated explanations.

The data analysis of this study included content analysis of informal discussions with both experimental and control group teachers.  The students’ scores in the pre-post Science thinking skills questionnaire was analyzed by comparing between the experimental group and control group, gender (boys and girls), parents career choice (science and non-science occupations), and extracurricular activities (science-related or not science-related).  An ANCOVA-Analysis of Covariance test was used for comparing the pre-questionnaire results when testing for statistical significant differences in the post-questionnaire results.  Eta Squared analysis was also used to measure the development of students’ science thinking skills.  An ANCOVA test was used for comparing the fourth grade students’ levels of explanations.

The conclusions of this study showed that teachers felt that animated movies can be shown in the classroom to help promote class discussions, teamwork between groups of two or three students, and individual learning.  There was a statistical significant difference between the experimental group and control group was that students who experienced the use of animated movies as part of their science learning improved their understanding and academic performance towards science concepts and reasoning, compared to students who used only textbooks and photos.  The experimental group that consisted of fourth and fifth grade students had statistically significant higher scores on the Science thinking skills questionnaires compared to the control group of fourth and fifth grade students.  The study found significantly higher percentage of experimental group students’ correct explanations than the control group participants’ explanations for the true/false questionnaire.  Therefore, this study found that animated movies could support the use of animated movies as a multimedia teaching tool because it can promote scientific interest, the development of using scientific language, and better understanding scientific reasoning through integrating visual, auditory, and kinesthetic learning styles and using sight, hearing, and feeling senses.

For the most part, I feel that the conclusions are valid given the study design.  The researchers had a pilot study stage performed in order to create the reliability and validity of the research tools, including the Science thinking skills questionnaire.  The questionnaires were also validated by four experts in science education and three elementary school teachers.  In order to make sure that the study was designed to measure what it intended to measure, the experimental teachers received a two-hour workshop for incorporating the web-based animations (BrainPop) and received extra help if needed by BrainPop experts.  The ANOVA analysis also seems that it was thorough and detailed throughout this study.  However, I feel that incorporating surveys and informal discussions are hard ways to examine the teacher participants, and this could be a threat to validity.  I also feel that there is a threat to internal validity because the sample of participants was only conducted in Israel, the participants are not representative and should not be used to generalize a wider population of all fourth and fifth grade students.  In addition, the study did not state the age of the fourth and fifth grade student participants, which could be different from fourth and fifth grade aged students in the United States.  Even though there are threats to internal validity, I do feel that this study was well-performed.  I think that the conclusions are valid in terms of studying the effectiveness of animated movies on students’ participants scientific learning and reasoning based on the validity of the Science thinking skills instrumental measure and thorough data analysis.

References:

Barak, M., & Dori, Y. J. (2011). Science education in primary schools: Is an animation worth a thousand pictures? Journal of Science Education and Technology, 20(5), 608-620. doi: 10.1007/s10956-011-9315-2.

Chambers, B., Cheung, A. K., Gifford, R., Madden, N.A., & Slavin, R.E. (2006). Achievement effects of embedded multimedia in a success for all reading program. Journal of Educational Psychology, 98(1),233-237. doi: 10.10370022-0663.98.1.232.

Advertisements