The Cult of Pedagogy Podcast, Episode 177
Jennifer Gonzalez, Host
GONZALEZ: When I put things out into the world about teaching, I do my best to support those things with research. But honestly, I’ve never felt really secure about it, because I don’t have any formal training in academic research. I have a Bachelor’s degree in secondary education and an M.A. in writing and publishing, so my direct experience with academic research has been mostly self-directed.
Although plenty of people in the education world do have this training, I’m guessing quite a few others—especially K-12 classroom teachers— are in a similar boat to mine: You have a vague understanding of how a peer-reviewed, published piece of research works, but a lot of it is still pretty confusing. It’s like the university folks who study best teaching practices are standing at the edge of a canyon and all the way on the other side are the actual practicing teachers. There should be steady, clear communication between these two groups—lots of good bridges that are easy to access—but so far we only have a few, and far too many of us on the teaching side don’t even know about them.
That’s what this episode is for: to teach teachers how to get ahold of research that can inform what you do in your classroom, hopefully in ways that won’t require crazy amounts of time.
This is not going to be the most exciting episode I’ve ever put out—I mean we’re looking at academic research and talking about the finer points of interpreting it, and it’s a lot longer than most of my other episodes—but I think it’s incredibly valuable because it will empower teachers with the tools they need to back up their instructional decisions with solid research. So think of this episode as a mini-course. My guest, Kripa Sundar, is an educational psychologist who agreed to help me get better at understanding how to read research. After talking about some of the excellent publications that are out there that already summarize this research for us—making it unnecessary to ever read a single study—she then walks me through how to find the actual research, then shows me how to read a meta-analysis and a sample study. Because the second half of our interview is highly visual—we did it over a Zoom call—I am making the video recording of our conversation available on YouTube. Just go to Cult of Pedagogy, click podcast, and choose episode 177 to get to the page where the video will be embedded. On that same page you’ll find links to all the publications we mention, the studies we look at, and a summary of the key points from our conversation.
Before we start I’d like to thank ISTE for sponsoring this episode. Future-proof yourself and your career by becoming a member of the International Society for Technology in Education—ISTE. Joining ISTE’s passionate community of global educators helps you find your network, build your skills and grow your career. Plus, members save on dozens of PD options and ISTE books. Hundreds of thousands of educators have trusted ISTE for relevant professional learning for over 40 years, so you can count on ISTE to provide the gold standard in educator PD. Sign up now at iste.org/membership. Use discount code CULT—press enter to activate it— to save $10 when you join. That’s iste.org/membership.
Support also comes from simpleshow. simpleshow’s mission is to make the world easier to understand for everyone. In more than 20 languages, their explainer videos and learning experiences help users worldwide explain, communicate, educate and learn. With their scientifically proven AI-based methodology, which combines charming storytelling and easy video creation, simpleshow video maker is helping make the world become a simpler place. Let simpleshow transform your message into a convincing explainer video and enjoy the interactive experience! You can register for FREE to test it out by visiting videomaker.simpleshow.com. If you are an educator, you can apply for the FREE classroom plan with all premium features.
Now here’s my interview with Kripa Sundar about academic research.
GONZALEZ: I would like to welcome Kripa Sundar to the podcast. This is the second time that we’ve recorded this because we did sort of a trial run yesterday and it was a hot mess. So we’re trying it again. Welcome.
SUNDAR: Thank you. Thanks for having me, Jennifer.
GONZALEZ: So what we are going to be doing today is helping me, and through me other teachers, learn how to make better use of academic research. And so before we get started and actually do, we’re doing this as a video and audio because there’s some stuff that would be really helpful to show visually. Before we start, just tell us a little bit about yourself.
SUNDAR: Sure. So my name’s Kripa Sundar. My full name’s NarayanKripa Sundararajan. So I finished my PhD in educational psychology a couple of years ago, and most of my work has been focused on multimedia learning, but I also do, I’ve also published meta-analysis on retrieval practice, and I’ve done a handful of experiences on concept mapping. So to sum it up I guess, most of my research has been in multimedia learning and cognitive instructional strategies.
GONZALEZ: Okay. And I found you because I think I saw a tweet where you had published something that was sort of helping teachers learn how to look at or how to read a meta-analysis. And I thought, this is exactly what we need because one of the big differences between you and someone who, they’ve been working, teaching 10th grade for the last 15 or 20 years is that you have got formal training in academic research and a lot of practice in it. And that’s one of the things that I think a lot of practicing teachers need because we want to be doing research-based practices, but a lot of teachers don’t really know how to find that research. So we decided that we were going to start with the topic of concept mapping, which you’re sort of just pretend that I am a teacher. I’ve heard about this strategy of concept mapping. I’ve heard it’s effective, and so I want to find out what does the research actually say about it? So, you know, one of the things we did yesterday is we started by looking for a study, and then we realized that you were actually aware of a lot of other places where we can find academic research without having to go directly to the academic research. So why don’t we, let’s talk about alternatives first, and then later we’re going to actually look at a study and you’re going to give me some tips on how to decipher it.
SUNDAR: That would be wonderful, yes. Yeah. I’m a strong believer of let’s not reinvent the wheel. If we know there’s an easier way, why not? Especially and when someone else does the hard work, I’m like, ah. So there are a whole bunch of different places where both teachers who’ve been super invested in research and have been reading the research translated and posted as blogs or books and stuff like that, but there are also some organizations in books. Like when speaking of blogs, there’s A Chemical Orthodoxy that kind of focuses explicitly on chemistry education. But there are lots more popular blogs, I think. And if I’m not wrong, you’ve hosted a majority of them on your podcast before with Learning Scientists, retrievalpractice.org, Effortful Educator that’s Blake Harvard, Learning and the Brain that’s led by Andrew Watson. Retrievalpractice.org is by Dr. Pooja Agarwal with whom actually the guy that you mention on decoding the meta-analysis on. There are also some, some of the articles in Edutopia and EdSurge are really evidence-based and really tie in tightly. And some of them are moments, are teachers kind of sharing their experience. So if you’re looking for something quick, these are some handy places to go to. And the one thing I would put in is a lot of times they focus on a handful of strategies and not like the whole universe. So you might not find concept mapping in them, right?
GONZALEZ: Right. This is like retrieval practice, which is really, she’s really been just devoted to that specific section of [crosstalk].
SUNDAR: Yeah, and I think in retrievalpractice.org she’s done retrieval practice, spacing, interleaving and feedback kind of in that realm. And I think Learning Scientists also kind of picked on the five or sex. And the reason behind that isn’t that that’s all that the research has to offer, which I’ve sometimes come across from educators like, “Oh. So I know those five things. I know education research. It’s most because those five to six findings have repeatedly stood the test of time for decades.
GONZALEZ: Right.
SUNDAR: So that’s, you know, I definitely recommend those. There’s also the American Educator published by AFT, which is a practitioner-focused journal. And they have researchers kind of write their research to the educators. So you actually hear from the researcher, but it’s not written with like the heavy stats or explaining the methods and all of that. It’s more focused on, oh, what are the key findings and how can we implement this in our classroom and the why behind it.
GONZALEZ: Okay. And that’s the American Educator, and I clicked on the link while you were talking. We’re going to provide links to everybody for these places. This is just aft.org/ae, and it looks like it’s free?
SUNDAR: It is free and it’s open, yeah.
GONZALEZ: Okay.
SUNDAR: the book, I mean like a hard copy I think.
GONZALEZ: Okay.
SUNDAR: The online one’s free, and then, you know, there are always books. I think the best thing about teachers and educators is they still read books, so it feels good to still write them.
GONZALEZ: Yeah.
SUNDAR: There are plenty, plenty of books out there, and I can’t shout out to all of them, but I do think three that I’ve personally liked, and I’ve heard a lot of educators kind of bring it up and recognize them. My favorite of them all is “The ABCs of How We Learn: 26 Scientifically Proven Approaches, How They Work, and When to Use Them.” This group of researchers literally used the alphabet and give you one strategy per alphabet, and kind of make it a very simple, digestible, okay, here is the big picture. Here’s how you use it. And then we have “Understanding How We Learn: A Visual Guide.” That’s by Dr. Yana Weinstein and Dr. Megan Sumeracki who were also part of the founding team for Learning Scientists.
GONZALEZ: Yes, and they have been on the podcast, yes. Absolutely.
SUNDAR: So that one’s, again, and I really like that book because it’s not just about the strategies, but they try to help give educators the hook and the framework of how do we learn and what’s the big, what’s the underlying mechanism in broad strokes as well? Which I think is helpful because educators, if you know what, how we learn, it’s easier for you to tease out the strategies which people come in and say, oh do this. It’s going to, you know, it’s the magic bullet of education. And you’re like, that’s not even in sync with how we learn. Like, how can that be the magic bullet?
GONZALEZ: Yeah. It sounds like what you’re saying is because the angle that I was originally coming at this from, and we’re going to do this example later, is that I have the strategy in mind and then I want to find the research. And you’re saying yes and it’s also a good idea to just have more comprehensive understandings of all of these things that research has found time and time again to be good. And so it’s almost like having a really good grounding in nutrition and somebody comes along with some new food trend, and you say, well, how does this already fit in with the framework that I already know?
SUNDAR: Exactly, exactly.
GONZALEZ: Okay.
SUNDAR: And it’s a great starting place, right? Because you know these, like retrieval practice. When I published the meta-analysis in 2016, we had covered 75 years of research on it, which kind of also speaks to the testament as to how the science of learning or cognitive psychology, none of this is new. It’s just got more funding recently to become more famous and popular. It’s finally become the cool kid on the block, right?
GONZALEZ: Yeah.
SUNDAR: So yeah. I think those are great. There are also some initiatives, like I’m a steering committee member of the TLRC, of the Teaching and Learning Research Committee, which is a subsection of like the i4tl organization. And what we’re trying to do in TLRC is connect educators with researchers. So if you have a big question or if you have a question like, oh, in my classroom, I have this group of kids and I’m trying out this strategy. I want to know if it’s effective. Maybe we could connect you with a researcher who’s interested in that topic to see if you could actually conduct that study even.
GONZALEZ: Does the TLRC publish anything, or is this more like an individual consulting kind of a thing?
SUNDAR: No, we’ve got podcasts. I actually hosted one season, but that was just kind of fun.
GONZALEZ: Okay.
SUNDAR: There’s, there are podcasts, and we’re usually there in a lot of the practitioner-focused conferences with a pre-conference event to talk about research and practice and how we can bring those two together.
GONZALEZ: Okay. And that is not the same thing because I see this on your list of things to talk about at some point too. Oh, conferences like researchED. Is that the same group?
SUNDAR: No, they’re not.
GONZALEZ: That’s different?
SUNDAR: Yeah.
GONZALEZ: Okay.
SUNDAR: So researchED is a group of brilliant folks. I think it originated in UK and it slowly moved into the US as well. Of all the conferences I’ve seen educators talk about, and we must say, like my sample is very biased just because of who I follow and all that kind of stuff. But they actually have a central focus on research and evidence supporting the instruction and the talks that they are speaking of. So researchED’s a nice way to go. John Catt ed publishers. They also started in the UK and moved over to the US, also have a whole host of books on different strategies for different subjects, which are also pretty evidence-based.
GONZALEZ: Okay. John Catt?
SUNDAR: Yes. John Catt publishing.
GONZALEZ: Okay.
SUNDAR: Oh. I know I’m throwing out a whole bunch of sources out there, but just because there is actually so much out there that’s much easier to access. We’ve talked, like most of the resources I’ve shared so far are very driven by the people who run it or like specific organizations, right? Like, the American Educator publishes, I think, once or twice a year, maybe. And blogs, you really never know when you’ll get what you want or can I really find it? Am I finding the blog?
GONZALEZ: Yeah.
SUNDAR: So actually two or three other ways that you can do it is there are handbooks of research on topics. So if you’re into reading comprehension, there’s a handbook for reading research. If you’re into multimedia learning like I am, then there is The Cambridge Handbook of Multimedia Learning. So if you want the whole shebang [inaudible] have that book handy to kind of look through and see what’s happening.
GONZALEZ: And is that written as a textbook that just sort of discusses different things? Or is it just a collection of studies on that topic?
SUNDAR: No, it’s written like a textbook so you can just read through it.
GONZALEZ: Okay.
SUNDAR: They don’t go into the, typically they don’t go into the stats too much, so it’s a fairly easier read.
GONZALEZ: Okay, but it’s all, it’s all research-based, cited stuff?
SUNDAR: yeah.
GONZALEZ: Okay, okay.
SUNDAR: Think of it like if you were in grad school and you’re going to an intro class, that might be the book there the teacher asks you to have in hand because it covers the topic.
GONZALEZ: Okay, okay. I’m embarrassed by how much of this you’re telling me and I’m hearing it for the first time.
SUNDAR: But Jennifer, it’s not your fault.
GONZALEZ: Yeah.
SUNDAR: The problem is we’ve never really gone out there to educators and said, it’s already there.
GONZALEZ: Yeah.
SUNDAR: So many times I hear people say, oh teachers should use evidence-based practices, teachers don’t care about research. I’m like, we tell them to use research. We don’t give them the research.
GONZALEZ: We don’t know where to find it, and I don’t know if this is, my graduate degree is in writing. I wanted to specialize in my content area, so maybe it’s because I didn’t get a master’s in curriculum instruction. Maybe people who study education as their graduate thing know about these things, but maybe not.
SUNDAR: I’d say maybe not. Actually, another organization, Deans for Impact, is trying to work on that exact problem you mentioned.
GONZALEZ: Let’s put that on here. That sounds really familiar to me. That’s come across my radar.
SUNDAR: So Deans for Impact are actually working with the teacher prep programs to include the science of learning and more research in the teacher prep programs itself. So you’re kind of introduced to all of this when you’re training.
GONZALEZ: Okay. I wrote that in there, so it’s there now.
SUNDAR: Okay, thank you.
GONZALEZ: Okay. So anything, any other places before we kind of move beyond this? Because I’m going to make a list of this. It’s all going to be over on Cult of Pedagogy, a list of links to this stuff. Anything else people need to hear about?
SUNDAR: Yeah, absolutely. I’d be remiss if I don’t mention the IES, the Institute of Education Sciences report. So all of, not all, but a good chunk of the research is actually funded by the government. So any research that’s funded by the government typically requires public reporting, so they, there are two places you can look for. You can always pull up the original articles. We will talk about that in a little bit, but even reports. They’ve actually summarized and synthesized reports for educators. So they look through decades of research and publish it as a report, and you also have the What Works Clearinghouse where they’ve created practice guides, which is like the, here’s the big idea. This is how you apply it now.
GONZALEZ: Talk about not reinventing the wheel. This is, this is, you’re giving me material for the next 10 years of my website, so thank you for that.
SUNDAR: A lot of different sources, but I hope it makes people feel a lot more comfortable that if you are interested in research, you don’t have to know a ton of stats and dig deep through the supposed dungeons of internet to figure out where [crosstalk] is hiding.
GONZALEZ: Well, and also, and we’re going to get to this soon too, one thing that happens to me all the time is I get, I hit paywalls all the time where it’s like, this study is amazing. You’re going to have to give us $45 for it. I’m like, okay, well, never mind then because I’m not even sure if it’s what I need, which is really the biggest barrier is not knowing as you’re just fumbling through. So knowing that there are all these people who are working on basically synthesizing, summarizing the research, and making it digestible and it’s there, we just need to now know about it and take advantage of it. So I’m going to spread the word hard about that. So, okay, so the next question before we dig into an actual study is when I am thinking of it, this whole concept of like the research says, the research says that, and boy, if 2021 is not the perfect year to talk about how one cites research. What should we be keeping in mind about using this concept of academic research to support our decisions?
SUNDAR: There are a handful of points that I’d want to kind of bring up to the surface. The first one, and I think since you mentioned 2020, one that a lot of people might resonate with is the idea that research is ongoing. As you collect more data, the evidence can suggest other things. So what, once we said, hey, this works, maybe in a decade or five years or 20 years we’re saying, that doesn’t work. The perfect example is learning styles. In the ‘60s, ‘70s, I think there was a bunch of research that said, oh, you know, you should give kids instruction based on how they think they learn best. And then fast forward a couple of years, and that finding has not been able to be replicated across multiple studies. There’s plenty of research on learning styles there. Doesn’t always mean that the research says that it’s effective.
GONZALEZ: Right.
SUNDAR: So keep in mind that the research can change, but that doesn’t discount the value of research in its whole. It’s a mere factor of how education research happens. When we do education research, we work with students who are living beings in real times of changing context. So as we collect data from these folks, they’re going to show us different sets of data, right? Their understanding is different. Their level of knowledge is different. Their skill sets are different. All of that’s also going to shape what findings we have. So be a little open-minded with the sense of, hey, this is evidence-based, yes, when was the evidence collected, and is it still applicable to your context, I think would be the overarching theme. So I guess long story short, anything you see, give it both benefit of doubt but still take it with a pinch of salt.
GONZALEZ: Mhmm, mhmm.
SUNDAR: And, you know, speaking of that, it’s, like I said, it’s done by humans, right. We as researchers are collecting data, and there will always be inherent bias and typically because of the rigors required by research when they’re trying to test out if this X works, it’s going to be in very controlled settings. I’ve spoken with some educators who kind of felt like, but if it worked in a lab setting, it’s never going to work in mine. I can’t tell you that it will work, or it won’t work. What I can suggest is why not try it on a small test lesson. Don’t go with your big, you know, don’t experiment with your hardcore lessons if you don’t want to, but try a sister lesson, like a super small one. Test it out and see how your students respond, and that way you can kind of have the lived experience of, does this work in my context or a classroom?
GONZALEZ: Yeah, yeah. So even if you’re seeing it in the research, it’s still, you should still try it and try it on something that’s low stakes first before you’re going to change an entire program based on that research, for example. Okay.
SUNDAR: Yes. Take it small. Give it time.
GONZALEZ: Yeah.
SUNDAR: Balance it out. There’s, unfortunately, there is no magic bullet or unicorn strategy. There just isn’t. And I guess the last piece, and this kind of ties into the next part that we’ll be talking about, is without methodological training like you very well mentioned, it can be very hard to isolate, is this a good study or a study that I want to be cautious of?
GONZALEZ: Right.
SUNDAR: And that’s OK cause sometimes even with the training, there are always new methods that sometimes I read papers, and I’m like, wait, whoa. What just happened there? It’s like the numbers floating over the head kind of scenario happens. And sometimes you’ll find research pro and against. Now learning styles is a wonderful example. I used to get very annoyed with it, and recently it’s become like my pastime just kind of seeing people going back and forth [inaudible] yay, no, yay, no. And it’s always interesting because we’ll always dig out a study where it says learning styles is effective. As a researcher, you look at it and you’ll be like, oh, they didn’t control for this, or they didn’t do this, or they didn’t do that. Like there’s methodological problems.
GONZALEZ: Right.
SUNDAR: Or it’s just bad interpretation, which then gives you this awkward result, right.
GONZALEZ: Yeah.
SUNDAR: But that’s kind of why I, well, it’s one of the hundred reasons why I love meta-analyses where meta-analyses are like a summary of studies done on a particular topic. So you get like this big bird’s eye view, and then you can go into the details.
GONZALEZ: Yeah.
SUNDAR: So if you don’t have method training, it doesn’t kick you out. Just take it with a little more salt, I guess.
GONZALEZ: Right. You know what, another thing too, I’m thinking while you’re talking about not having the training, I don’t necessarily also know, sometimes I can read in the summary, and I can be like, okay, this study was done on 17 participants, so maybe it’s not transferable to any situation. But a lot of times I’ll just think, if it’s published, it must be peer-reviewed, it must be okay. But there are so many journals out there that I don’t necessarily know the reputation of even the journals that are out there. Or even if something is, if you get something that’s got nice desktop publishing and a good font and is called The Journal of Blah Blah Blah, and I’m like, yeah, it must be good, and it’s, you know, I think that’s why training is a good thing. So we’re going to be talking next about meta-analyses, so that’s kind of if we’re ready to move to the next question. If I actually do want to find a study, and none of these other avenues have really led me to what I’m looking for or I just want to find the real thing. Where would I go? And we’re going to do a little, like a little mini training here of how I would actually go about that.
SUNDAR: Sure. So you want the, you don’t want the coffee, you just want the coffee beans?
GONZALEZ: Right.
SUNDAR: There are three places you can look for studies. The most common one and the easiest one I guess is Google Scholar. So to go there you type out scholar.google.com. Do you want to share your screen so we can have that?
GONZALEZ: We tried that last time. Let’s have you share yours because it was a lot of you saying, can you scroll down a little more, can you scroll down a little more? And this way, you can just show me what you’re talking about.
SUNDAR: Okay.
GONZALEZ: Yeah.
SUNDAR: Then give me a minute. I’ll just kind of, here we go. So to go to Google Scholar we type in scholar.google.com. It works, the search engine works kind of similar to regular Google, so it’s adaptive based on your interests, based on what you’ve searched for, you’re going to get similar kind of recommendations or even the order of articles kind of come.
GONZALEZ: Okay. Underneath, what we’re looking at right now underneath, that’s stuff from your own history, so somebody coming to Google Scholar is not going, that’s not your search results, for example.
SUNDAR: Yes.
GONZALEZ: That’s just almost like an ad basically for stuff you’ve looked at in the past.
SUNDAR: Exactly or things that, either I’ve looked at it in the past or it could just be things that they, that, based on my search history, they think it’s interesting to me.
GONZALEZ: Yes, okay.
SUNDAR: Yeah. So we’ll just go, so if I was to search on a topic, so for concept mapping, right, I can just go concept map, concept mapping. You can see how many suggestions there are. If I can fangirl about concept maps for just a teeny second, it’s one of my favorite strategies ever because it does everything. It’s time consuming. I’m not going to, you know, I’m not going to disregard that. It is time consuming, but it does everything. It helps you plan. You can use it as assessment, you can study it like just study a concept map for learning or you can construct your own concept maps. You can create them as worksheets. It’s like there’s almost nothing you can’t do with concept maps.
GONZALEZ: I totally agree with you. Yes. No, I’m with you on that. It’s, yeah. This is a good one.
SUNDAR: I actually, for my master’s thesis I had done a study on concept map with kindergartners and since they can’t use words what we did was printed out pictures of like the weather and activities that you do in the weather across the seasons and kind of had kids even do it with pictures, which is always a nice strategy if someone wants to try it.
GONZALEZ: Cool, yeah. Because it still shows the relationships.
SUNDAR: Yes.
GONZALEZ: Yeah.
SUNDAR: Okay, now when we type in concept map, see how there’s like, is this 4,520,000 results?
GONZALEZ: Yep.
SUNDAR: Yep. And that’s a lot. And even if you assume only 10 percent of it is even valid, you’re still sitting at about 452,000 studies, right?
GONZALEZ: Mhmm.
SUNDAR: So a couple of ways that this can, that you can work through this is click on custom range, and I like to choose the last 10, 12 years of research, just because like I mentioned, research is continuously evolving and seeing 10 years of work can kind of help you know that it’s in the ballpark or at least in the decade of what you’re looking at.
GONZALEZ: Right, and this is a point that we brought up yesterday that wouldn’t be in today’s video if I didn’t say it, is that any study done in the last 10 years, if there’s something really pivotal that happened before that, it’s likely to be included in the literature review or in the citations or something of that newer study.
SUNDAR: Yes.
GONZALEZ: They’ll be building on that.
SUNDAR: Yes, absolutely. And I would recommend just leaving that second box empty because sometimes journals have advanced publications, and they call it 2022 but you’ll find it now.
GONZALEZ: Okay.
SUNDAR: Because online comes faster than book. Why do [crosstalk] I don’t know but that’s a different conversation.
GONZALEZ: Yeah.
SUNDAR: So if I click search, just because of my time range, I’ve brought it down to 1,870,000. That’s still quite a lot, and it can be like, oh, I just want to explore what concept map is. And sure, you could start out with something like this and kind of go, oh, let me see what happens. But a smarter way to do it, especially for teachers who are short on time — which I think is pretty much every teacher — is to specify, so put concept map, actually no, and then put reading, for example. And then when you search for it, you’re down to 513,000.
GONZALEZ: Okay.
SUNDAR: That is —
GONZALEZ: Totally manageable.
SUNDAR: A little bit.
GONZALEZ: Yeah.
SUNDAR: You can also always just specify and say, high school.
GONZALEZ: Okay.
SUNDAR: Well, actually, let’s do middle school because I usually think, I don’t know, this is my personal opinion, not my professional opinion, that strategies that work in undergrad likely might work in high school. That’s just my thinking.
GONZALEZ: Yeah.
SUNDAR: I mean there’s some difference between high school and college.
GONZALEZ: Yeah.
SUNDAR: Okay. Now look at that. When we break it down to, I don’t actually know if the number changed.
GONZALEZ: It did. It went from 500,000 to a little over 100,000.
SUNDAR: Yeah. So that’s, like every time you add another label in here, it kind of chops down how many studies you have to look through.
GONZALEZ: Okay.
SUNDAR: Then a couple of things, like I do have two studies we can look at. A couple of things just about using Google Scholar is they link out to PDFs if they’re available on the internet. So let’s say I wanted to read this one, “The Effect of Learner Constructed, Fill in the Map Concept Map Technique, and Summarizing Strategy on Iranian Pre-university Students’ Reading Comprehension.” I could just go click Open and voila. I have the paper.
GONZALEZ: That is my favorite thing about Google Scholar because I don’t have access to a university library, and so for those of us outside of those systems, we can’t see any of these studies unless there’s a PDF.
SUNDAR: And sometimes even if you are within a university or you have access, databases can sometimes not give you some articles or some journals.
GONZALEZ: Okay.
SUNDAR: It’s always a handy thing. One other trick that I wanted to share about just Google Scholar is let’s say I read this one, right? We opened it up, we read it, and we’re like, oh man. This is wonderful. I love it. What you can then do is click on “cited by 27.” Click on that and then it gives you the list of articles that cited the paper that you liked.
GONZALEZ: Mhmm.
SUNDAR: And this is a great way for two things. One, you can read more around the same topic or the same idea. It can also sometimes show you papers that have critiqued your original paper.
GONZALEZ: So these are not all necessarily fans of the paper.
SUNDAR: No.
GONZALEZ: Okay.
SUNDAR: So I guess I don’t have, I’m not old enough to do this, but if someone wanted to get super famous super quick, just publish a really controversial paper and then you get cited left, right and center. Not yet. Not ready for that.
GONZALEZ: Yeah. What about this star? Is there a way of starring, and then you’ve got, then it goes into your library of saved articles?
SUNDAR: It does, yes.
GONZALEZ: Okay.
SUNDAR: So if you just click, I don’t usually do that, but if you click Save, and then you go to My Library, it will pop up.
GONZALEZ: Okay, and this is all tied to, oh gosh, it looks like you can tag and everything. So you can really get, get deep in terms of organizing. It’s all tied to your Google account.
SUNDAR: Yes.
GONZALEZ: Okay.
SUNDAR: You can download, you can export the list too if you wanted to create like a reading list for yourself.
GONZALEZ: Okay.
SUNDAR: The other thing that I wanted to show was the alert system.
GONZALEZ: That’s right.
SUNDAR: So one super nice thing about Google Scholar is — that I use a lot mostly because of my work on meta-analysis — is this Create Alert function. So we’re back on that original search page where we did concept map and reading in middle school. And we had just looked at this paper, and we’d looked through the cited by. Now when I come back to this main page, if I was to create an alert, it will show me, I can just enter my email in here and then press on Create Alert. What that does is it will send me an email with the link of the newly published study as soon as it’s available. Most of the times it’s pretty accurate. Most of the times it comes in at the exact same time you see authors kind of tweeting, saying, “My paper got accepted,” right? It’s super cool.
GONZALEZ: Right.
SUNDAR: Well, not accepted, but it’s like now live. Whenever it’s live, it comes through. So that’s definitely a nice way to go about it if you’re thinking of, oh, I really want to go through this topic in-depth or I want to stay abreast.
GONZALEZ: Okay.
SUNDAR: Okay. So all of this shebang is Google Scholar. I have a lot of tricks on this one because I use this the most. Two other sources where you could always get articles is ERIC. This is by the government, so it’s eric.ed.gov, and you can put in any topic. And this is very gentle. Like you were saying, how do I know if it’s a good quality? You can select only for peer reviewed. Again, peer-reviewed isn’t bulletproof, but it is better than not having it, right?
GONZALEZ: Right.
SUNDAR: I also like that it lets you just look at the ones that are available.
GONZALEZ: Okay, yeah. That’s great. I remember ERIC from college. Haven’t really used it since then, but yeah, it looks great.
SUNDAR: And let’s, so for example, let’s, the science of reading comprehension instruction, there’s a direct link and does it, oh no, this one actually you can download the PDF. But sometimes when you click through, if you hit a paywall, one thing you can do is email the authors. I usually mark everyone on the paper because you never know. Someone might have graduated or not using that account, or they’ve changed jobs or whatever. But if you email everybody, most of the times they’ll reply.
GONZALEZ: You’re emailing them to say, can I see a copy of the study? Is that what you’re asking?
SUNDAR: Yeah. Just send them an email saying, hey, I’m an educator. I came across your paper titled this, or send them the link, whichever, and say, would you, I don’t have access to it. Would you be willing to share it?
GONZALEZ: Yeah, definitely didn’t know that trick. Okay. And that, you get results sometimes from that?
SUNDAR: I’d say 9 out of 10 times.
GONZALEZ: That’s fantastic, okay. Yeah. Every time I hit a paywall, I just give up and that’s it.
SUNDAR: Oh no, no need.
GONZALEZ: Okay.
SUNDAR: I think, no, I think most researchers would be more than happy to send that through.
GONZALEZ: Cool, okay.
SUNDAR: With ERIC, the other nice thing about it is just it also breaks it down for you a little more based on your topic.
GONZALEZ: So you put in a search term and it gives you lots of other sort of variations and subcategories?
SUNDAR: Yeah. So these are the tags that the authors have put in as well.
GONZALEZ: Okay, okay cool.
SUNDAR: So you run into reading comprehension, you’re like, oh no, what I really mean is reading skills.
GONZALEZ: Okay, good.
SUNDAR: Then you kind of just look at those papers.
GONZALEZ: Okay.
SUNDAR: And similar to that is the What Works Clearinghouse. And this is also where I’d mentioned that you get the reports, like the practice guides.
GONZALEZ: Yes.
SUNDAR: You can also see reviews of individual studies, intervention reports and find what you need.
GONZALEZ: Okay. The reviews of individual studies, I was going to ask you this yesterday, and I, the question left my brain before I had a chance to ask you, but is there any kind of, I mean, you know, if I want to buy a TV, I can go on the Best Buy website and look at customer reviews. Do they have anything like that for academic research where people can just rate it?
SUNDAR: No, but that’s a smart idea.
GONZALEZ: A way of crowdsourcing this stuff.
SUNDAR: There is though, like the What Works Clearinghouse tries to do extremely rigorous research and sometimes, I heard the joke, I promise this isn’t me. I’ve heard it from someone else, was like it should be called a What Doesn’t Work Clearinghouse. Just because the studies that qualify for the What Works Clearinghouse are extremely rigorous.
GONZALEZ: Interesting.
SUNDAR: So it’s like very tightly controlled, across multiple settings, and you’ve got like lots of things going on but you’re focusing really on one. But it is a great space for you, especially if, let’s say, you are applying for ESSA funds, right, and you want research to back it up. Now the resources here would be like a great way to search for it.
GONZALEZ: Got it.
SUNDAR: So yeah, that’s, that’s the three big places: Google Scholar, ERIC, What Works Clearinghouse.
GONZALEZ: Okay. So are we, are we ready to look at a study and actually kind of drill down into how one reads a study?
SUNDAR: Yes, absolutely.
GONZALEZ: Okay.
SUNDAR: My, I think I mentioned this yesterday, but it would be good for me to say again. The first time that I have ever read an academic journal publication in any field whatsoever was the first day of graduate school.
GONZALEZ: Wow, okay.
SUNDAR: I had never seen a published article before that. So it was kind of overwhelming, right?
GONZALEZ: Yeah.
SUNDAR: Like I’m sitting there in grad school and everyone else is like, the assignment is like, oh, write up an article critique and just find an article and then write about it. I’m like, exactly like what you said, right?
GONZALEZ: Yeah.
SUNDAR: I can pull up articles. Right now I cringe to think that I thought that first article critique I wrote was a good article to even write on. I’m like, why. Right? Very embarrassing but one thing that I’d learned pretty soon in grad school was this magic called meta-analyses.
GONZALEZ: Okay.
SUNDAR: Meta-analysis or even a systematic review. Both meta-analysis and systematic reviews are summaries of the research literature, and they give you a big picture of what’s happening. So it’s more breadth, and for me, if it’s the first time that I’m learning about a topic, like you mentioned, I’ve heard about concept mapping. I want to know a little more about it, like just high level to start with to test if it works. I’d definitely go with a meta-analysis or a systematic review. The difference between a meta-analysis and the systematic review is a meta-analysis is quantitative summary, so we’re trying to say, on average, across all of these studies, how effective is this strategy?
GONZALEZ: Okay.
SUNDAR: On the other hand, a systematic review gives you more a thematic analysis and more, yeah, a thematic analysis of the papers that have been published.
GONZALEZ: And they’re both looking at multiple, multiple, multiple studies. It’s just that one of them’s a little more conversational, and the other one is actually crunching the numbers.
SUNDAR: Yes. It’s conversational as awkward academics can be, yes.
GONZALEZ: Okay, okay.
SUNDAR: And I want to tell people, like this is something which, you know, I want to use this chance to say researchers, we write in a particular way, but we’re still human and when you talk to us, most of us don’t talk in that same language. So if you’re feeling tripped up or you really like the study, but you don’t understand it, email and ask, hey, can I chat with you for half an hour? I don’t know how likely that’s going to be a yes, but I’m pretty sure if someone emailed me or a few others there’d be like, yeah, sure. I’d love to talk to you about my study.
GONZALEZ: Okay.
SUNDAR: The academic writing is a learned skill, not our natural tongue.
GONZALEZ: Yes. I have actually worked with some people to try to get them to write blog posts for me who are trained in academic writing, and we have to help them unlearn that and make it a little more conversational and accessible to the average human. So yes, yeah. So we’re going to be looking at a meta-analysis?
SUNDAR: Yes. Let’s start with a meta-analysis.
GONZALEZ: Okay.
SUNDAR: Head’s up, meta-analysis will always feel intimidating, even to a researcher.
GONZALEZ: Okay.
SUNDAR: They are 35 to 50 pages long and have about 10 big tables with lots of numbers, a few plots and it can look and feel very smart, but it’s still a really nice read. Break it over days, though. Break it over days definitely.
GONZALEZ: Okay.
SUNDAR: I know the Review of Educational Research is a wonderful journal to keep an eye out for meta-analysis and systematic reviews. They almost exclusively only publish that across a variety of topics.
GONZALEZ: Okay.
SUNDAR: So like speaking up, is this a good journal? Review of Educational Research definitely is.
GONZALEZ: Yeah.
SUNDAR: There’s also Educational Psychology Review. Those two are the ones that I’m familiar with in my world of work.
GONZALEZ: Okay.
SUNDAR: So staying on task of, like, how do we read the meta-analysis, the first step is to read the abstract. The abstract is basically a quick high-level summary of what’s happening in the paper. It will tell you how many studies. So in this, they have looked at 55 studies involving 5,818 participants. And they give you a small description of what, the students were from Grade 4 to post-secondary, and they used concept maps in science, psychology, stats, and nursing. This is not typically comprehensive, but if it’s super important or if it’s substantive, it will mostly find a place in the abstract, right? It will also tell you a quick summary, I guess, of the results.
GONZALEZ: Yeah. And this is the place that I always go when I’m trying to find research to support something is I’ll look, and usually by the end of the abstract, you get a sense of what they uncovered or decided by the end, okay.
SUNDAR: Mhmm. And the next trick is, one hurdle that I’ve often heard from folks is I don’t understand that language. There’s far too much jargon in it.
GONZALEZ: Yeah.
SUNDAR: I 100 percent resonate with that. It’s been something I’ve been trying to stay mindful of as even I work on my writing journey, but there are some terms that as researchers you can’t use another term, simply because that’s going to muddy up waters. And a lot of water has been muddied in that way, right. X calling it something, and then Y calls it something else.
GONZALEZ: Yeah.
SUNDAR: But most authors will define their main idea topic.
GONZALEZ: Mhmm.
SUNDAR: So that way you can check, is this the same as how you’re thinking about it? And that’s true whether it’s a meta-analysis or an individual study. So in this one, it’s in the very first line.
GONZALEZ: Okay. And so usually the first few paragraphs are good to read after the abstract as an introduction to the study, a little bit of background and understand.
SUNDAR: For the definitions and stuff, yeah.
GONZALEZ: Yeah, okay.
SUNDAR: When it comes to a meta-analysis, feel welcome to kind of scroll down to, look at over here: hypothesize cognitive and self-regulatory effects. Sounds super fancy.
GONZALEZ: Yep. This is where my brain would be like, bye, yep.
SUNDAR: So if you see the word “hypothesis” or “theoretical framework” and these words are fairly similar in both like your individual studies and your meta-analysis, that’s your hook, your why.
GONZALEZ: Okay.
SUNDAR: Why do I expect an effect? Why am I expecting that the strategy will work, and in what way do I think it’s going to work?
GONZALEZ: Okay.
SUNDAR: And this is where I was telling you, right? Getting that overall framework in mind so you can kind of assess strategies as they come in.
GONZALEZ: Yes. So this hypothesized is in a passive voice. They’re saying, here’s what we are looking for, this is what we’re expecting to find in these, and this is what we think will be the results.
SUNDAR: And the why.
GONZALEZ: Yes.
SUNDAR: And why we think it’s so, and sometimes it’s just called theoretical framework, which is—
GONZALEZ: Okay, okay.
SUNDAR: So if we scroll down, like in this paper they look at multiple reasons. They say, hey, maybe it’s dual coding, then they’re like, maybe it’s verbal coding or it’s just the elaboration part of learning strategies, and that concept maps could be effective for different types of students, typically those with low verbal ability, and just the different ways that you can do collaborative need cooperative. So these authors went, looked at the same problem from multiple different frameworks. More often, people just look at one or two frameworks at most to kind of look at what they’re studying. So it won’t typically be as long.
GONZALEZ: Okay.
SUNDAR: The next piece that you want to look at in a meta-analysis is the studies selection criteria or what they call an inclusion criteria. Why that’s important is it’s going to tell you what kind of studies were included in a meta-analysis. No meta-analysis is like 100 percent complete because then we will never publish it. So the inclusion criteria tells you which studies we decided we’re taking in, until what time. So like the meta-analysis under retrieval practice, I think we closed the doors on that in 2014. Yeah, we closed the data collection in 2014, and it got published later. So it’s helpful to read through that to see, oh, okay. What have they, whom have they included? But let’s say you said all that, you can kind of, and you’re like, oh, I don’t have the time to read this 36 pages, I don’t care about it. You can always just scroll all the way down.
GONZALEZ: That’s the stuff that I always skip. As soon as I start to see what looks like trigonometry to me, I’m like, okay, it’s just, yeah. And I was a good math student. That’s the frustrating part of it is that I was a good math student, but still it’s gone now.
SUNDAR: Well most meta-analyses now don’t necessarily include the equation. They typically don’t.
GONZALEZ: Okay.
SUNDAR: So you’re all good. So when we go into the results section, typically either the first or second table is going to have, where is that second table, here. Weighted mean effect sizes for concept maps constructed and studied by geographical location. You see this category all?
GONZALEZ: Uh huh.
SUNDAR: That’s the table that you want looked at where it either says overall or all. And this tells you across all of the studies that they’ve studied, what is the result? So what this sentence here is telling me is across all of the studies, they had 5,818 students, and they pulled out 67 unique experiments.
GONZALEZ: That’s what K stands for, the number of experiments?
SUNDAR: Yes.
GONZALEZ: Okay.
SUNDAR: And the .604, .28 is your mean and your standard error, which is basically your average, right, .604 is the average gain in the score. So that’s not, so for example, if I had a test out of 10 points and my kids typically scored 5 out of 10, if I did concept maps with them, this paper suggests that they’ll be scoring 5.6 the next time.
GONZALEZ: Okay.
SUNDAR: For kind of translating that, the guide that we published on retrieval practice has more information on, like, how do we translate that effect size, so that would be handy resource to have in hand when reading these tables. Just one more point on the table, and then we’ll jump shop to the individual study.
GONZALEZ: Okay.
SUNDAR: I really want folks to look at this piece. Where it says confidence interval, think of it like the range of impact. Your lowest impact could be .55 all the way up to about .66, and that kind of helps you weigh how much you’re going to trust that result.
GONZALEZ: Anything above zero, so when it comes to effect size, if I’m looking at any study, effect size is positive or negative, this is how much this thing impacted the learning.
SUNDAR: Yeah.
GONZALEZ: Up means it positively impacted it, down, negative means it’s negatively. And generally, I mean it seems like when I’ve seen effect sizes, I don’t think I’ve ever seen anything higher than like the number 4. Is that even a reach? Usually it’s —
SUNDAR: That’s a very big reach, I think.
GONZALEZ: That would be very high, okay, so I’m thinking it’s like 1.3 or something like that would be very, very high, if it’s over 100 percent increase.
SUNDAR: Yes.
GONZALEZ: Okay, okay. So these are, something that’s in these decimal ranges is still very significant.
SUNDAR: Yes.
GONZALEZ: Okay.
SUNDAR: And we need to remember that this is all in averages, right? It’s on average. This is [crosstalk].
GONZALEZ: Right.
SUNDAR: So let’s say if you take this study and you’re like, let’s do concept maps in class, well, yeah, it’s going to take time to actually hit that result.
GONZALEZ: Okay.
SUNDAR: Because your students need to be trained in concept mapping, and then you implement it. So it might take you two to three lessons to even start seeing a difference. So if you’re really interested in the strategy, maybe applying it a few times will help you look through it.
GONZALEZ: Okay.
SUNDAR: But the reason I want to show this is if you pulled up a meta-analysis and saw this and you didn’t read through the rest of it, which I love reading through it because there’s a story there kind of saying what and why and how and all of that.
GONZALEZ: Yeah.
SUNDAR: But this kind of gives you this idea that, okay, that’s not too bad, and I’m like kids are in a 10, if it was a 10 point and is jumping up by .6 then hey, maybe if I had 100, imagine that jump.
GONZALEZ: Right.
SUNDAR: Right? Like it could very well be the difference from a B to an A-minus or a C to a B-plus. So, and then you’re like, okay, I love this. Then how do I read more, right? How do I know whether this works for my context? Scroll down to the reference list and look for the ones that have the asterisk because that one means they included that in the meta-analysis.
GONZALEZ: Okay, okay. And those would be the ones to drill down more deeply?
SUNDAR: Yeah. You can look into it more deeply.
GONZALEZ: Okay.
SUNDAR: And then just looking into that. So I’ve pulled one up here.
GONZALEZ: Can I, hold on, can I stop you before we do that? And then, I want to go back to the meta-analysis one last time. I wanted to make sure that we also get down to the discussion because we looked at the numbers.
SUNDAR: Yes.
GONZALEZ: And I know that even if I’m somebody who is willing to try looking at the tables a little bit, I, untrained, I would absolutely want to get to the results section to confirm that what I read there is actually what they’re concluding also.
SUNDAR: Sure. This particular paper, they combine their results in discussions.
GONZALEZ: Okay.
SUNDAR: So it’s actually built into this, but I was thinking I’ll use the other paper to kind of walk through the [crosstalk].
GONZALEZ: Okay. But with any time you’re actually looking at studies, whether it’s a meta-analysis or an individual study, you do want to always make sure you go down to the discussion and the results to —
SUNDAR: Yeah.
GONZALEZ: — yeah, in the conclusion, okay.
SUNDAR: Yeah, so in this meta-analysis they combined the results, just part of the discussion but they do have a conclusion section that’s like a, long story short, this is what we think.
GONZALEZ: Okay. Got it.
SUNDAR: These are the key findings. This is where the evidence is. Here are some things that we need to work on, that kind of stuff.
GONZALEZ: Okay. So now we can go back over to the study. I just wanted to make sure we didn’t leave people hanging because that part is really important. If someone’s really new to this, I want to make sure that they know to go to that end section.
SUNDAR: Absolutely.
GONZALEZ: Okay.
SUNDAR: So the one that I chose for us to look individual is one that I think you’ve talked quite a bit about retrieval practice on your blog and your podcast.
GONZALEZ: Yeah.
SUNDAR: So I was like, may be fun for folks to see retrieval practice and concept mapping compared. In most individual studies, you’re going to see the strategy that they are testing against either instruction as usual or just [inaudible] or just nothing at all. So an important thing to keep in mind when you’re reading an individual study is to always look for what the comparison is. In this study, we know that, oh, and one good way to find out what that is, is again just read the impact, the abstract, sorry. Reading the abstract, again, it gives you a kind of summary of what the work was, what their comparisons was, what they basically found. So again, like I had mentioned, the definitions are almost always in the first paragraph. Like in here, they’ve just mentioned retrieval practice, like a high-level definition. It mentioned elaborative coding, and I think down here somewhere they mentioned, they defined concept mapping, look at that. Concept mapping and kind of explaining why. So the first few pages kind of help with that. The next thing that you want to look at in an experimental study is this section called method. Almost always will be the subtitle of method and looking at this method will help you think about the implementation in your classroom, right? Like, what did the researchers do and what did they use to see the impact that they’re doing. As an educator, I think this is perhaps the most critical piece to see if it fits, tells you who it was. It tells you what the comparisons were. So in this one, there were three comparisons. The study only, a retrieval practice condition, and a concept map and retrieval practice condition. Then there is a, so that tells you, okay, maybe I can try this in my classroom, right? Like, I got high schoolers and that’s close enough, and I want to try it. So in the materials they tell you what they read. So in this particular one, the text was three paragraphs, 27 sentences and 262 words in length, and that really helps you put in context what are your kids doing in your classroom. Are they really only reading three paragraphs? Are they reading more than that? That’s going to also influence the overall results, and in this study, I think that’s what happened but I’m not getting into that right now. Looking in the materials will help you think about, oh, this is good. For example, like in retrieval practice, a lot of time they use word lists and folks are like, oh, so can retrieval practice only work with word lists? Not really, it’s just for experimental ease. It’s easier to kind of limit, limit how much they’re using.
GONZALEZ: Okay.
SUNDAR: So once you see that, look at the procedure, which will tell you exactly what they did, how long they spent, that kind of stuff. So over here, I think they put in, yeah, that learning phase students in all three conditions studied the text, and then after that, the retrieval practice condition practiced recalling for 10 minutes and then restudied it. In the concept map and retrieval practice condition, they constructed their concept maps for 10 minutes, and then they completed their retrieval practice procedure.
GONZALEZ: Okay.
SUNDAR: It tells you what the process is.
GONZALEZ: Mhmm.
SUNDAR: And in individual studies, the nice thing is even if all these texts with numbers in between kind of makes your eye go crossed, you can always jump down to a discussion session. In this particular journal article, there are three experiments, so they have a small discussion up here and then a bigger one down.
GONZALEZ: Okay.
SUNDAR: So in here, in this discussion, they kind of summarized what they found and whether that was expected, why, why not. And if there were any lessons that they learned, how are they going to fix that in the next experiment? So I’m going to skip through Experiment 2 and 3 for time.
GONZALEZ: Okay.
SUNDAR: And if we go down and see the general discussion, these are usually very interesting to read because in this kind of paper they talk about all three studies, how they came together, where they converged, where they diverged. But even in a study, even in an article where there’s just one experiment, the discussion’s a wonderful place for a teacher to look at if they think the study’s a fit because it’s going to help you ground that finding in the broader context of the literature. Like what we had mentioned. If I’m restricting my search only to 2011, what if there was an influential paper before that? That’s most likely going to get cited in the discussion.
GONZALEZ: Okay.
SUNDAR: Or even like if there was a whole host of studies, like what happened with deductive details or retrieval practice where suddenly in one year there was like, boom, 15 studies. Like just, boom, and it was like, okay.
GONZALEZ: Yeah.
SUNDAR: But the discussion section will really start piecing all of that together, saying like X found this, Y found this, we found this. Ours is similar to what they found or ours is different than what they found, and here is why we think. So the discussion section’s really fun and kind of bringing those different pieces of evidence together, and also helping you think through, so this is why perhaps it’s worked or hasn’t worked.
GONZALEZ: Yeah. I’m also thinking, you know the value for like a K-12 teacher to get into the habit occasionally of reading something like this is I’m just, I’m seeing a lot of the really measured thinking that one gets trained to do as an academic researcher as opposed to just being like, well this worked this one day, so let’s just go and do that, and let’s just go change everything. Like, there’s just much more, which is part of that frustration that I think we talked about yesterday where you know if you’re busy, you want to know what’s going to work, and I want to apply it right now, and within academia, things just move a lot more slowly. And it’s this kind of thinking where you really want something to be accurate as opposed to quick.
SUNDAR: Yes. That is definitely a big difference between the research and practice is the speed with which what happens. And the other thing also is as an educator, if you take the initiative to start reading a research article, and you’re not sure if the interpretation’s right, I wouldn’t self-doubt or kind of beat down on myself at all, even if I didn’t have the research training. Because one, it takes prior knowledge to know what they’re talking about, right? Like, the terms, the more you read them, the more familiar you get. The more number of studies you read, you start noticing a pattern, and you’re kind of like, oh, I see what you mean.
GONZALEZ: Yeah.
SUNDAR: This very study, for example, when I read it, it definitely caught my attention because I do retrieval practice, I do concept mapping, and I looked at it, and I’m like, wait, what? In my head, it didn’t even make sense for me, as, like, for me to compare retrieval practice and concept mapping because based on the meta-analysis on each of them separately, they have very similar effect sizes. So why would we compare one effective strategy with another effective strategy? I mean I understand why, but it’s like —
GONZALEZ: Yeah.
SUNDAR: — it’s like .7, .7.
GONZALEZ: But they’re combining them in one of the groups, right?
SUNDAR: And that’s a great point, and that’s where like the theory piece that I was telling you —
GONZALEZ: Yeah.
SUNDAR: — because if you look into the theory of concept mapping, it’s not like a lot of folks just do mind mapping, and then say, hey, it’s a concept map. But in actual rigorous concept mapping where there’s that big effect size, you actually write out a description label between the two concepts that you’re connecting. And like the steps to it starts straight from writing down, you’re literally doing a brain dump as your first step. And if that’s not retrieval practice, what is?
GONZALEZ: Right, right, right.
SUNDAR: So I’m like, so I’ve done that. I’ve done the concept map, including the very nature of developing a concept map forces you to do retrieval practice. It forces you to do elaboration. So why are we adding retrieval practice again at the end? It’s like, in my head, right? I mean this is just how I think about it.
GONZALEZ: Right.
SUNDAR: I’m not saying anything is right or wrong, but that comes to my mind because I’ve read about both.
GONZALEZ: Yeah.
SUNDAR: But someone who’s just read this study, which is diss my concept map, well not my, but diss concept mapping saying, oh, it doesn’t, you know, we don’t need this. True we don’t need it, right.
GONZALEZ: Yeah.
SUNDAR: You don’t need it if retrieval practice is enough for what you’re doing, and that’s where the materials make sense, like 250 words, and if you read through the details, you’d see like they’re fairly straightforward connections between like X does Y, A does B, kind of very straightforward concept, no, concept kind of very straightforward connections.
GONZALEZ: Right.
SUNDAR: In an instance like that, I don’t see why a concept map would be more helpful.
GONZALEZ: It’s not complex enough.
SUNDAR: Exactly.
GONZALEZ: To need it, yeah.
SUNDAR: Yeah. But if it was a longer piece of text where we had more characters in it or anything that’s bigger to condense it —
GONZALEZ: Yeah.
SUNDAR: — as an educator, if you pick up an article and you feel like, God, I don’t understand this, I wouldn’t fret about that. Just read what you can. If you’re not sure, look up, talk to the author or talking to the author might be a little harder, but they will email back.
GONZALEZ: Yeah.
SUNDAR: So you can always send an email or ask to chat or just find another resource from that big list of other resources we found.
GONZALEZ: For people to find you online, where would they go?
SUNDAR: I’m definitely active on Twitter. It’s just @KripaSundar.
GONZALEZ: Okay.
SUNDAR: I also have a website, kripasundar.com.
GONZALEZ: Okay.
SUNDAR: I super love learning, and I kind of hop around all of everywhere doing what I can to help learning be applied in practice. And if I may, a little shoutout to my book that I wrote.
GONZALEZ: Yes.
SUNDAR: Which was on my to-do list before my 30th birthday. It was like, I gotta do something before my 30th birthday. And writing a book was like on top of the list, and I’m like there’s no way I’m doing all the research I need for a full book. And I realized this like seven months before my birthday, so I wrote a book for little kids. I got two little ones myself, so it’s called “How do I learn?” and it’s a picture book on some of the key ideas behind learning, like, hey. Instead of saying, pay attention, I say, to learn something you’ll, I learn when I, what do I say? I think like, I learn when I grew my thinking, or something like that, or just for each of these, yeah, for each of the different principles I just put in something very light. But my favorite piece is at the end there’s a parent section where it kind of explains what the principle was and gives some activities and like conversation prompts. Because I’m like, why are we waiting until high school and college when the bad habits in knowledge and [inaudible] the system sets to talk about learning habits. Like, that should happen when kids start to learn.
GONZALEZ: Yeah, yeah. Oh, that’s cool, okay.
SUNDAR: I’m still experimenting on my kids, so we’ll see how that turns out.
GONZALEZ: Well, I will link people to that book as well.
SUNDAR: Thank you.
GONZALEZ: And just thank you for meeting with me twice about this now and re-recording and waiting for my computer to restart yesterday and all of that stuff. And I’m going to condense this into something that’s really going to, going to help teachers lots. So thank you so much.
SUNDAR: Yeah, absolutely. I’m happy to help.
To read a summary and transcript of this interview, or to find links to all the resources mentioned today and the video of Dr. Sundar walking me through the studies we talk about, visit cultofpedagogy.com, click Podcast, and choose episode 177. To get a weekly email from me about my newest blog posts, podcast episodes, courses and products, sign up for my mailing list at cultofpedagogy.com/subscribe. Thanks so much for listening, and have a great day.