Much, Zach, for that very, very kind introduction, and thank you all for being here. Um, just so you know, I can't see you, so I'm sort of just speaking into space and hoping that there are some Hoping that there are some nods and people sort of following along. As Zach said, I have condensed a lot into 20 minutes, and I'll do my best to step you through. In such a way that, um, is easy to follow. Just working out my slides… Okay. All right, can… Zach, I can see you nod, can you see my slides? Yes, awesome. All right, um, today I'm going to start off by giving a definition of interpretive research. Qualitative research. And then I'm going to give an overview of the approach that Zach was speaking about, uh, the ProQL approach to designing interpretive research studies. And then I'm going to spend all the rest of the time working through an example, so you can really see how the approach that we've developed applies to interpretive research design. All right, let's start with the definition. So, what is interpretive research? I just want to highlight some key parts. So. Interpretive research is a systematic study of social systems, where researchers, like ourselves, create knowledge by interacting with people or texts. Um, to find out about their lives, and then we draw conclusions about the social systems, functioning, or structure. From interpreting these accounts. I'm now going to contrast this to researching the technical sciences, which is what a lot of the people who we work with are more familiar with. And some of those contrasts that people in the technical sciences typically work with physical systems. We work with social systems. Another big difference is that in interpretive or qualitative research, the researchers themselves are the instruments where the where we are co-creating the data with our participants. Uh, and then we subjectively, or interpret the accounts that we gather. So here are some key differences, and these Key differences pose some of the challenges. That, uh, a lot of the people that I work with often encounter. So there are a wealth of traditions, doctoral programs and texts, in interpretive research, and these comes from… these… come from all sorts of disciplines, as you can see on this slide. And, in fact, there's so much out there that it can feel pretty overwhelming for people who are new to this type of research. And that's really the key reason why my colleague Jochen Walter and I developed the ProQual approach. Really to help researchers leapfrob that leap… leapfrog, that initial feeling of overwhelm, and uh… and so that people can just get on with the designing, with designing a study that makes sense. All right, I'm now going to… Try it and summarize the ProQL approach in six points, and then I'm going to come back to this slide multiple times as I go through the example. So the ProQL approach offers an overarching framework for thinking about research design, and it does a few things. So, one is that we really try and draw on the strengths that technical scholars bring to interpretive research. Things like visual thinking, design. Considering alternatives, brainstorming, so really that kind of thing. Uh, our approach is problem-led, so that means that instead of starting with discussions of ontology, epistemology, words that took me you know, a long time to even be able to pronounce. We start with the interest that, uh, our participants That beginning interpretive researchers come to us with. So it's really problem-led. Um, our approach really supports collaborative and team-based approaches, which is quite different to a lot of those disciplines that I showed you earlier. A lot of those disciplines really work with a solo researcher approach, whereas in the technical sciences, we see a lot more appetite for team-based approaches? Uh, one thing that we've been having a lot of fun with recently is, uh, really purposefully using AI to accelerate what has traditionally been a really time-intensive process. I'm going to give you a tiny taste of that. And, uh, and all of this, we feel, really gives us the opportunity to produce results that can make a big change in the world. All right, let's, uh, expand on that first point, that overarching framework. So, what does that overarching framework entail? Again, I've tried to summarize this into four points. Uh, so really the first step is drawing a pictorial systems map of the social system of interest, and we are going to do that in a minute. So, a picture of the social system that we are interested in. Then, once we draw this fairly large map, we're going to start to consider different areas that could be the focus of our research in that map. After doing that, I'm going to introduce to you some ways of thinking about the, uh, essential functions of some of the core components of interpretive research. So, in contrast to technical research. Interpretive research has a big, uh, place for theory. And also, in theory is quite implicit in technical research, and so that's why a lot of beginning and even advanced interpretive researchers stumble over the role of theory. And methodology is also something that's a little bit more implicit in the technical sciences. And so we've worked out ways to really think about what these components essentially do in interpretive research studies. And I'll come back to these with the example. And then finally, we've developed a way of thinking about research quality, so that we can make sure that the results that we're making in this social subjective space are high quality and have the potential to really make a difference in the world. That we're working in. All right, let's get into an example. I'm going to start with this pictorial systems map. So, we really see a pictorial systems map as A dependable process to turn big ideas into purposeful research projects. So, people come to us with I'm interested in this big thing. Well, that's great. Well, how can you clearly define a researchable unit Uh, within that big area of interest. And the key question to ask when you're drawing a pictorial systems map is to really think about, in concrete terms. Where does your interest happen in people's lives? I'm gonna… show you this through an example. And that process of drawing pictorial systems map involves brainstorming elements, drawing the system. And this process allows… acts as a boundary object, so by that I mean that it allows a lot of people in a team to get on the same page, so everyone knows what they're talking about. And it allows us to slow our thinking down, which is not what I'm going to do today. Today, I'm going to be talking really fast. But the process of drawing pictorial systems map makes us slow down our thinking, which can be really powerful. Brainstorming involves across lots of dimensions, people, artifacts, relationships, etc. Alright, the example I'm going to use comes from a real-life example of a research incubator that I used to run at the University of Georgia. Uh, and this idea came from an instructor who taught statics, who had just, over the last few years, transitioned to mastery grading. He was super excited about the impacts he was seeing of this new grading approach on his students, and he wanted to do research on it. He didn't have any experience in interpretive research, and he came to the incubator and said, mastery grading. What do I do? How do I start? And so we started. With a pictorial systems map. Alright, let's jump into it. So, where does mastery grading happen in people's lives? Well, it involves students. This is my attempt at drawing a student. Uh, it involves students thinking about status, sum of forces equals zero thinking about their grades, and what are some of the relationships? Well, they're interacting with, uh, instructors, and here's an instructor giving a tick For, yes, cross for no, this is how Mastery of grading works, and then a redo if they didn't do it right. Until the student reaches mastering. I'll flick through the next few slides a bit quickly. This came out of brainstorming. Some people were asking. Well, how does the student change over time from the beginning to the end of the semester? Uh, there were some words that came up. Typically, we don't include words in pictorial systems maps, because we're interested in how this interest happens in people's lives, these theoretical constructs, but it was a brainstorming session, so… We threw them on there. Um… Uh, some people also suggested how mastery grading changes how instructors think about a particular subject. Uh, there was a lot of interest in the, uh, transition of this instructor from traditional grading to mastery grading. Some people also suggested, well, what about the different experiences that different students have in their class? Maybe they're not all the same. And then we can start, and that was the second point on the previous slide. Starting to think, well, what would make researchable units? And the most obvious researchable unit May well be to look at the student. The impact of mastery grading on the student, but there are other potential areas we could research as well. Um, for example, the relationship, you can see here, the relationship between instructors and students. How does that change in a mastery grading context? Um, or what about what I spoke about before, that transition? You know, how does that happen? If we… If we think that mastery grading is a good thing, maybe we need to understand how professors transition, instructors transition from traditional to mastery grading techniques. And then we brainstormed more, so this is this iterative process of exploring, and we talked about things like, well, what about the relationship to industry? You know, isn't mastery grading really closer to industry? Because there's no grades in industry, you have to go back and get it done. Until you get it right. Alright, let's go back to this, um, what the ProQL approach offers. So, I spoke about how the ProQL approach draws on the strengths of scholars with technical backgrounds, and here, one of the strengths that we're drawing on is this visual approach, this visual thinking. Also, um… Uh, uh, yes, so… Sorry, I'm just catching my breath here, hopefully giving you a chance to catch yours as well. Um, another thing that this example shows is that problem-led quality of the ProQL approach, and so we started with this big idea of mastery grading, and then we drew that social system, so this is an example of that point. And uh… and we can also see how that… drawing that pictorial systems map supports collaborative and team-based approaches. That drawing happened in a meeting where people were throwing ideas out. And instead of everyone having their own ideas and having slightly different understandings, we all reached somewhat of a consensus, or a shared understanding, through this, uh, artefact of a Victoria Systems map. Uh, going back to that other overview slide. We've done the pictorial systems map, we've started to identify some, um, researchable what we call social realities under investigation, that's what I put the dotted line around. So we started to investigate some of those, and Um, in the actual approach, you then go and look at the literature and see what the saturation is of literature across those various potential areas to investigate, and that might also influence what you decide to research. All right, let's move into these fundamental, um, understandings of what theory methods and methodologies offer in interpretive research with this example. So, for this, I have picked, um… This is, uh, this transition of an instructor from traditional to mastery grading, as an example, to speak about theory, methods, and methodology. Now, theory… this is our theory about theories, if you will, and it's really a starting point for thinking about what kinds of theories could I use in my work But getting back to that fundamental role of theory, theories help us better understand what's happening with our social reality under investigation. So. Here, if we say our social reality under investigation, is this instructor transition How can theory help us better understand what's happening there? And let's just say that we're looking at that at the level of an individual. Maybe we're interested in one instructor, or separate instructors doing this in different contexts. What's happening inside their heads when they're making that transition? And here, um… Yesterday, actually. I, um, decided to ask AI what theories might help me understand this transition. And, um, it came up with some interesting theories, and we'll just look at the second one for now. So, teacher identity theory was one suggestion, that transitions in grading practices often require instructors to renegotiate who they are as teachers, their values, and their relationships with students. Um, and so that could help us, so how could theory help us understand what's going on can then help us understand how to see it. And so if we use this theory, we might ask questions about how the teacher now sees themself. Selves, now that they're using mastery grading. And this is a really cool way to purposefully use AI, Previously, we advised people to have a sense, um, of at what level their theory is, and then go and speak to prof… Speak to people who have a lot of experiences in that area. Speak to people far and wide, read purposefully. Now we can really, uh, in an awesome way, I think. Use AI. Because we have this overarching framework that we know what we're looking for. Um, I'm going to skip over this example in the interest of time. Alright, now methods The essential function of methods in interpretive research is that They help us to see our social reality under investigation. And, for example, if we… look at that instructor transition, and let's just say we're looking at one instructor, or single instructors in different contexts. And we want to use this theory of identity We, uh, interviews might be a really good way to see that social reality under investigation, because it allows us to look inside people's heads. Um, in contrast, if we were… perhaps looking at a social reality under investigation that involved maybe a group of instructors. That all together had decided to make this transition. Maybe they were part of a community of practice. Maybe a focus group where we brought all those instructors in would allow us to see that shared social transition. And so, methods allow us… and again, this is where we can start to experiment with those different options in our research design process, what would this method allow us to see? What would this method allow us to see? Likewise, written reflections, they allow us to see something different. Okay, uh, methodologies. Methodologies, uh, are things like narrative analysis, granted theory, phenomenography, phenomenology, as you can see, I still have trouble pronouncing some of these words And that can be a real obstacle to a lot of, um, people who are starting out interpretive research. There's this whole Other layer of studies that needs to be integrated into the research design. And so we have defined the essential function of methodologies as as methodologies illuminate certain patterns in the data. And again, for this, um… oh, well… First of all. Um, and here are some pictograms that we've developed that allow people to easily grasp what those patterns are that are being illuminated in the data. And so, for example, with narrative. If we apply that lens, that methodology, we're saying that If our results take in narrative form, then that will illuminate something new. And provide impactful results. Likewise, if we, um, use the pattern of thematic analysis. Then we're saying that some kind of hierarchy of themes and sub-themes is going to be a pattern That will, um, provide a way of showing our results that will be impactful, and I will show you what this means with our example of instructor transitions. So, going back to our pictorial systems map, which was the first step in this ProQL approach, and really. The basis for research design. If we put this pattern of a narrative across this trajectory, we can already see some alignment. You know, we're talking about a change in time. Um, if our results took that form of a narrative, then we might uh, focus on the initial inspiration, we might focus on Um, discuss challenges that came along the way, key characters that helped people. Um, uh… challenges, successes, etc, and then they get to the end. And that could be a really insightful Finding that could help other researchers who are considering that transition anticipate what their journeys might look like. Alternatively, we might decide that we want to know what factors influence their decisions, you know, what factors helped them. And have a list of factors, what factors provided barriers, and have a list of those barriers. And that format, um, of results might help administrators in maybe setting up, um, institutional conditions to support their faculty in moving to mastery grading. So that's another pattern that we could consider. And, um, and this is always a fun pattern, just… Because pictograms are. Great way to explore different methodological fits with our studies. This is a pictogram of a research approach called phenomenography. And let's just say we were doing research not just on one individual faculty member, but maybe 20 or 30 faculty members from a across the United States who have transitioned to mastery grading, we might decide to do a phenomenography of that. Now, the phenomenography, the essential pattern there is that there's variation in understandings of a particular phenomenon, in which case we'd be looking at Variation in the way that these instructors understand and experience mastery grading. And these results often take the form of less nested less to more comprehensive ways of understanding or experiencing something. And as the… just sort of brainstorming, as the least, um, comprehensive way of understanding mastery grading, people might say, well. Instructors might say, well, students need to reach a certain level of mastery. Um, as a more sophisticated way of understanding, people might say, yes, they have to reach a level of mastery. And it's important for students to learn how to fail, and… then correct their mistakes. Once again, more sophisticated. Those two things, plus maybe other understandings are, yes. Mastery grading… correct their mistakes, but that's also the way that, um these technical students, when they get into their jobs, are going to, um, work in industry, when they get to industry, that… and so I'm actually teaching them professional skills. And then, right up at the more, most comprehensive, it might be all of that, and, well, this completely dismantles power dynamics, and playing for grades, and shifts the focus from learning to metacognition, something like that. And so this might also be an interesting pattern That we look for in our data. Now, the reason I went into so much detail about that particular pattern is because our fourth part of this overarching framework is a range of quality and um, constructs. Dealing with validity and reliability. That help us better understand how our design choices, so what theory are we using? What method are we using? What methodology are we using? How are design choices are measuring up to these essential functions is our method allowing me to actually see my social reality under investigation? Is my methodology illuminating a pattern that's in the data and will be useful for people to read about in my paper. Obviously, can't go into all of that right now, but I'll pull out one example of pragmatic validation. So, pragmatic validation is a quality construct that concerns the fit between the theoretical constructs we bring to the study. And the empirical realities. So some of the theoretical constructs we bring to the study are our theory. And our methodology. And the empirical reality is What we're actually looking at, the social reality under investigation, what people actually say. And here, this… quality construct, as an example, prompts us to check these things. And so let's just say that we design a study where we're looking at these 20 to 30 faculty, and we want to do a phenomenography because it's really cool. But we're not seeing. These nested less to more comprehensive ways of understanding. It's just… it's just not in the data. This methodology just doesn't resonate. Well, this notion of pragmatic validation First of all, prompts us to check prompts us to ask the question, are the theoretical constructs Fitting the social reality under investigation, and then gives us permission to say, well, no, not really. I need to try another methodology. Um… So I will leave it there. So I know I have thrown a lot at you. In a very short amount of time, but I wanted to leave Plenty of time for Kelly and plenty of time for questioning. So I'll just finish with 3 broad statements, and that is that interpretive research is a really powerful approach for understanding important educational challenges. Um, and designing and conducting interpretive research is a lot of work, but it's also a lot of fun, especially when you've got a framework like this, that can guide you in that brainstorming process, and it's even more fun if you can do it with collaborators. Um, technical researchers also bring a lot of unique skills to this work. And so, if you're interested in diving into qualitative integrative work, then I would say Don't hold back, do it. Thank you so much, and I'll hand over to you now, Kelly. Killing that, you can just kick me off, I'm not quite sure, can't find the screen. So, thanks, Nikki. Yeah, I think I can do that. Awesome. Well, thank you so much, uh, Nikki, for kind of putting together the presentation. So, um, I got to see her presentation, uh, last week, um, just to have a chance to interact, ask a few questions, and think about what, um, I might share for kind of a 10-minute follow-up. Um, so I'll introduce myself again quickly. Um, I'm a teaching associate professor of Statistics at the University of Illinois. I have a master's degree in statistics and a PhD in curriculum and instruction, so I kind of come at this discussion as someone who started in a more quantitative space, but pivoted to more qualitative methods as I actually started, um, becoming more of an educational researcher. Um, and so I wanted to kind of highlight some of the things that Nikki's talking about with another example to give you another chance to kind of digest some of those things that she was talking about, um, while also really focusing on this idea of framing, because I think when I started in qualitative research. I had a really hard time wrapping my mind around what people meant when they would say framing, because I felt like it meant a lot of different things. And the truth is, it does mean a lot of different things. Um, it's something that has a lot of layers. So, I'm gonna kind of unpack this into three layers, um, based mostly on what Nikki was saying, but kind of reframing it, um, a way that makes sense for me as well. Um, so, Nikki talked about framing as scoping your research aim. She talked about selecting appropriate methods and methodology to unpack and parse your data, and then one more thing that I think was implicit, but maybe not explicit in the way that she talked about it, but framing as… choosing a lens through which to see your data and tell a story. So I'm gonna walk through an authentic example that I had with a colleague of mine, and give you a sense of kind of how we grappled with some of these layers of framing. So, Nikki had that really cool pictorial representation. That's something that she and her colleagues do as part of this ProQual approach, is actually physically representing the research situation or the phenomenon that you're dealing with. In scoping, then, is thinking about different kind of mechanisms or interactions or pieces of this puzzle that you really want to focus your attention on. Um, so I want to bring another kind of metaphor to that. When I think about scoping, I think about just the idea of taking a picture, um, and choosing how and from what angle to take your picture. So, this is a picture of CloudGate in Chicago, but if you live around Chicago, you probably just call it The Bean. That's what most people think of it as. Um, and so here's a picture of the bean. Um, so if I asked you, go take a picture of the bean in Chicago. Maybe this is the picture that you would take. That's certainly not the only picture that you could take, um, but something that we might notice about this picture is it's really framing the shape of the bean. It's also shaping, uh, or framing kind of the background, the cityscape behind it. Here's another picture, um, but this time your attention's a little bit more drawn to the reflective aspect of the being. You're still seeing the cityscape, but now you're seeing it as a reflection, rather than seeing it as kind of a backdrop, um, to, um, the sculpture. Here's another picture focusing even more zoomed in on the reflection. Now I'm really not thinking about the shape at all, I'm really focusing on the reflection, and maybe this particular reflection's kind of guiding me to think about the personal aspect, um, the people who come around and sit around or take pictures of the being, as opposed to just kind of its more solitary sitting here. Here's a picture from underneath, um, showing the distortions of the reflections that can happen when you're looking at that particular angle. Here's an opposing image showing from above. Kind of situating the bean within this space. Not really seeing the shape of the bean as we'd see it from the ground, but kind of seeing it from a different space. And so the thing about scoping research is there isn't really a right or wrong picture. They're just kind of pictures that give you different perspectives and focus your attention on different things. So, here's an example of kind of a general research question or, um, query that a colleague of mine and I kind of had together, and it was… Is there any reason for our students to come to our statistics classes in person if we're making recordings available? A lot of students just choose to watch recordings, they don't want to come to class. So, should we incentivize them to come? Do we think there's value? Maybe more importantly, do they think there's value? Um, can we just record our classes once and never have to teach in person again? So, we could kind of address this question from a lot of different angles, and so I'll give you a couple that, um, had kind of come up for us. Um, one question we might ask is, what are some features of classes in which students can learn just as well through recordings as through in-person classes? Um, so obviously there's a lot of online classes these days. Do they have certain features? Are there certain things that those classes have, um, that make them just as equivalent as an in-person class? Are there certain topics or certain types of classes that lend themselves well to being virtual like that. That really comes at this from a kind of a core-slash-instructor-oriented perspective, thinking more about course versus course. We could also ask, um, which particular students benefit most from in-person classes, focusing more on the students. How do students compare their in-person class experiences to their experiences watching a recording? Now we're focusing a little bit more on an experience, um, oriented perspective. Do students engage with tasks differently in person, rather than when following along with the video? Now we're thinking about different tasks and the nature of the tasks involved. Does peer interaction provide value to students in class? Focusing a little bit more on small group interactions, not so much the individuals as kind of the small group dynamics that happen in our classrooms. So, depending on the research angle that we want to take, how we kind of want to focus our research, um, that might inform the types of data that we want to collect. Nikki talked about several of these on different slides. But if I'm focusing on a question like, how do students compare their in-person class experiences to their experiences watching a recording, um. I would think I probably could choose interviews, focus groups, or surveys, maybe depending on the kind of data I want to collect. I think Nikki might have talked a little bit about grain size at some point, but maybe that would kind of connect here, this idea that the nature of data that I collect from a survey is probably going to be a little bit more coarser-grained, um, as opposed to the kinds of data that I might collect in an interview or a focus group where I have the ability to pro students, um, and really dig into certain ideas with more depth. But then surveys, maybe I can collect data from a lot more people, whereas interviews and focus groups are going to limit how many people, interviews, probably even more so, but with the possibility of more depth, perhaps, in a focus group or more of an individual perspective. Um, and then she also talked about methodology. Um, so if you're new to qualitative research, there's a lot going on here, um, and so I think just the idea to take away is that when we do qualitative research, just like in quantitative, there's so many different kind of methods of analysis or approaches that we can take when we grapple with our data. And so, we think about in qualitative, um, which methodology kind of lends itself well to the nature of data I'm collecting, and the kinds of insights that I'm trying to draw out. So I think for the question and the study that my collaborator and I were thinking about, I think we could take a case study approach, where we really think about the individual perspectives and experiences. Phenomenology would be more like, what is kind of the common experience, or the common ideas, maybe, that we're seeing from our students that we want to capture. But they just kind of lend themselves to slightly different Orientations, um, with the data that we're collecting. And then this third piece that I want to bring in is framing as a lens and a medium for storytelling, which to me is, I think, the most fun about doing qualitative work, is it's a place to be kind of creative. Um, and so, when I think about this idea of framing, um, one way that we might think about it is framing through a metaphor. Um, so maybe in this example. Maybe I'd be inspired by the idea of how are professors trying to attract students to class? Related to businesses trying to attract customers. Is there something about that idea that might inspire me or inform the way that I kind of write about and think about the data that I'm working with, and the way that I present it to, um, readers? Uh, maybe I'm framing more through a concept or a construct. So maybe I want to think about this idea of self-regulated learning. Maybe that will help me understand why some students are more successful with recordings than others. If I think about students' self-regulated learning, does that help me understand my data with more depth? Or through a theory. Um, I think Nikki had two different theories that AI kind of helped her think about or discover as she was looking at her data. Another example in my case, how does feminist theory help me understand the experiences of women participating in my class? Does that give me a perspective to take when I understand my data, or even when I'm thinking about how I approach my study? So, I'll kind of end with just a few suggestions for, um, if you're new to qualitative research, where do you get started? How do you get started Um, so one thing that really made a difference for me was just having a good qualitative research book that I actually kind of enjoyed reading. Um, so one that I've had is a pretty, like, standard book. It's Corbin and Strauss's Basics of Qualitative Research. Um, and I have a few different books, but this is the one that I have also found is just the most readable. It's the one that I actually enjoy just kind of reading through on my own, as opposed to just using it more as a reference book. Um, an alternative, or perhaps And, um, in addition to that, it's just reading good qualitative research papers in your area. And I wanted to highlight, too, there's many good qualitative research and statistics education that we could point out, but here's two that I'm familiar with. In that I think have very good methods sections that really detail how they were addressing these questions, how they were framing their data, and how they were analyzing their data. Um, the first is from Allison Theobald and Stacey Hancock on coding code. It looks at students' code in a data science course as a unit of analysis to think qualitatively about how students are doing data science. Um, and then another by Nicola Justice and her, uh, collaborators, paint-by-number, Picasso, a lovely example of framing through metaphor, by the way, um, of how students think about statistics, um, through different. Painting experiences, paint-by-numbers, step-by-step class, realist or abstract painting. I really love that paper. And then another suggestion that Nikki said, just collect some data. You don't have to necessarily plan everything out before you start collecting qualitative data sometimes. Um, you just have a direction that inspires you, but not necessarily a prescription for what you're gonna do. And it's totally okay for framing and questions to evolve. As you sit with qualitative data, that's kind of part of the freedom and process of qualitative research. And also, Nikki said, ask someone to collaborate with you, even if it's both your first time, it's more fun to do qualitative research with a partner than it is to do it on your own. Well, at least it is for me, maybe it isn't for everyone else, but, um, I think it's, um, usually elevates your insights when you have somebody else that you can discuss. Your data with. And that's where I'll stop. All right, thank you, Kelly. Thank you, Nikki. Um, folks, we are moving into the Q&A. I noticed that some folks have already started to post questions into the Q&A. If you haven't found that button. Go ahead and hunt around in your Zoom interface. Feel free to write us questions here, but I can get us started, because some folks were already kind enough. To start to fill the queue. First question is directed at Nikki. Um, if we jump too soon into collecting data and recording interactions with students, is there anything we can do to rescue our research and improve its quality at the data analysis stage? Well, I think Kelly just touched on that, that it's fine to start collecting data. Um, uh… But maybe not too much. Um, but if you already have a dataset, then I would start at the methods part of the research design, and try and understand what that data is allowing you to see. So, is that data allowing you to see, well, I guess you're saying recording interactions with students. What is it about those interactions with students that's interesting? And then you can work backwards. And think, okay, well, um… you know, what is my social reality under investigation? Am I interested in Um, how students interact when they're facing challenges. Am I interested in how students interact Um. Based on gender, am I seeing, you know, mostly men sit together on a table and women sit together on another table, etc. So, I guess the… what I'm really trying to get at is you need to see what phenomenon that data is allowing you to see, and then you can Find out, okay, well, what theory can help me better understand that? Um, etc. So, it's… it's quite normal for people to start with data, but it is a little bit easier to, um, start at the beginning, and then collect purposeful data, because I… see a lot of people who already have data. Um, it's not… it's big in volume, but there's maybe not a lot um… that's useful compared to the amount of volume. Whereas if you really purposefully collect your data, then you're going to have a one-to-one useful data to what data you have. So I guess I would also say be ready to let go of some of that data, and just focus on what's really going to help you with your study. I remember, Nikki, one of the things you told me, um. Yeah. Was, like, the first thing you should do is just go on a pilot. Like, you know, try… try a small-scale to start to learn, um, and make some of those early mistakes, and I feel like that's something that I've tried to really take to heart in my own work, and also to sometimes look at things and say, like. I'm now going to call this a pilot. This is… Uh, this is going to be a learning opportunity for me to revisit the framing. Um, and I feel like that, at least for me, has been a liberatory view of how to do this kind of work effectively. Um, Kelly, anything else you'd like to add on this question? Yeah, I mean, I think I agree with the pilot idea. Is it, you know, if nothing else, it's always good to, like, interview like, low stakes or interview a friend sometimes if you're just, like, doing qualitative research for the first time. Um, but in terms of, kind of, that first question of, is there anything that we can do to rescue our data, I think I would just say what What Nikki said is you just kind of have to ask what is here. It may not be what I initially was hoping to answer, maybe it's not rich enough, um, to kind of do what I initially thought, but then I just have to ask the question, well, what… what is rich about this data? What is here? Maybe it's just a different direction than I intended. In some cases, maybe you just come to the conclusion of, well, you know what, maybe this data just isn't, um, rich in any way that I can find, and I do just have to, um, kind of do a second round of data collection where I've learned from my mistakes, but… but yeah, I think just being free to realize that our framing can shift quite a bit, our questions can shift. Um, from where we initially thought, just depending on where the data actually goes. Yeah, and I think that can probably be hard for, um, maybe especially hard for statisticians, since we're used to thinking in a confirmatory mode. Um, really saying, I'm going to take an exploratory approach and potentially go somewhere quite different. Might be a bit scary at first. Yeah, I wanted to say, um, when I talked about framing, technically, when we do quantitative research, we're framing too, we just don't think about it, we're not confronted with those decisions in quite the same way. And a lot of that framing happens before we collect any data, whereas in qualitative, we have a lot of framing liberty as we're collecting and after we collect data as well. But technically and quantitative, we frame, but not maybe as explicitly, um, thinking about it. So, next question here. What advice do you have for researchers who start with having rich, multidimensional data? Working backwards with data to select a framework slash methodology and shape a story. I think that question is very similar to the first one, and it… and it is possible, and… and quite common, I would say. Um, uh, like I said before, it's certainly easier to start at the beginning, but If you have a framework and, um, and know where you are, it certainly is possible to work backwards. And sometimes, even with the best intentions, we have to work backwards. I mean, I have, um, designed a phenomenography before. And when we got to the data analysis stage, the graduate… undergraduate student I was working with just kept saying, no, no, no. Not seeing it, not in the data, and at that point, we were forced to shift our methodological choice. And so, in that particular context, I was in the same, um, situation. I had a lot of data, and my methodology no longer fit, and I had to work backwards and really think about, well, what am I actually seeing here? What research questions can I actually answer? And so, yes, it is certainly possible to work backwards, and… and this is where really understanding the essential functions of theory, methods, and methodologies can help you do that. Yeah, I don't know that I have anything to add to that. I think, um, I had a collaborator who's has said before, you know, like, how do you… how do you find… frameworks? Like, how do you find a good framework? Like, do you just have to be well-read in the literature? Um… Something Nikki did in her presentation was mentioning, sometimes I just ask AI, do you have any suggestions? And sometimes that's a way to brainstorm, um, or talking to somebody else. But I do kind of say that, like, there is framing through formal theories for… through, like, existing frameworks, um… But sometimes framing can be a little bit more loose, like, I think framing with a metaphor sometimes, um, is just about being creative with your data. Um, but uh… but yeah, I think just… In my experience, it's… It's number one, trying to be well-read, um, and number two, trying to intentionally go look at research that's answering similar questions to see, do I want to… Um, be consistent with frameworks that other folks have used who are just in this general area, too. Yeah, there's definitely, um, an ecological part to doing this kind of work where, um, if you're bringing in ideas. That are maybe… Foreign to communication community, or potential… either foreign in the sense of unfamiliar, or foreign in the sense of, like, actually challenging. You probably need to… be ready to, um, defend those choices and work harder to actually have them be accepted. And I think this is also touching on questions of alignment Kelly, where you were talking about being creative, I'm all for creativity and interpretive research, as long as you have that alignment between the sexual reality on investigation. Your theory's helping you see that, your methods are helping you, your theory is helping you illuminate it, you're… methods of helping you see it, your methodologies illuminating some interesting patterns, so you can get that through not following a recipe, we can get that through, you know, metaphor can very, you know, very easily fit into that alignment. Another question here, um, it's actually a many-part question, so… Bear with me. Um, in the description of this webinar, you describe qualitative research as a family of small n research approaches. Um, that was me. I wrote the description. I will own that. I'm not sure if Nikki and Kelly would fully endorse the way I frame that. But the question continues, can you talk more about this? Do you view qualitative research as a tool to be used when you have a small sample size? As a tool, you can only use for a small sample size, because it is labor-intensive. Both, neither. How do you feel about quantitative and qualitative research approaches? In terms of how they complement each other, fit together. Sorry, I know this is several questions in one. Um, yeah, a lot of really great questions. I don't think I'm going to do justice to all of them now, but I'll just touch on some of them. Um, uh, so qualitative research, uh, yes, works very well with small n, and that small, I've, you know, there are papers that look at just one participant, and that can be really powerful if that participant is, uh, really uniquely situated. Um, typically. Qualitative research studies have 8 to 20 participants because it's so labour intensive. Um, with AI now, I'm doing a lot of experimentation with using AI for qualitative data analysis. Maybe that will change in the future, but at the moment, typically, 8 to 20. Um, there's a lot of work that combines, you know, mixed methods, qualitative and quantitative, and there are a whole different way… lots of different ways to do that. Creswell, and maybe… can we share resources, Zach, as part… can we attach resources to this webinar? Yeah, um, what you might be able to do is, if you… if you throw something into chat, um, in your chat window, there's a way to switch it to everyone. Okay. Um, so if you had, say, a URL or, like, even, like, a reference, like, just a title, author, etc, you could paste it in there. Okay, one of, um, Creswell's earlier works defines all sorts of different ways of mixed methods, so where you do qualitative first, then quantitative, quantitative first, then qualitative. And if you're interested in doing that, that description, um, is… is really helpful. And I'll just… We leave it there for now, and I'll put that reference in the chat. Yeah, I think I would agree with the sentiment of your comment, Eric, that it's… It's not exclusively a small n. I think we can also qualitatively examine survey, like, open-ended survey responses where maybe there's a lot of different responses, but we still want to kind of do the… The… the work of thinking through what the essence of people's responses are, we could probably do that with quantitative methods, too. Um, so then, I think there are spaces where quantitative, qualitative, and or both can kind of be used to look at the same data, but maybe just with our attention in different ways, or the methods of analysis might just look a little bit different. Nikki, you started to touch on this, so I want to bring up this next question, um… I noticed Nikki used AI for brainstorming. What is a researcher's ethical responsibility for using AI in qualitative research? This is a massive question, and there… there is no answer. To that at the moment. Um… But there are many ethical issues. The most obvious of which I would say, is putting participants' data in, uh, an AI. Um, and I… find that difficult myself to justify at the moment, but things are moving so quickly. Um, I think that… Uh, having a purposeful framework where you understand what you're trying to do, AI can be a great Um, brainstorming partner. Um, and in terms of addressing Well, I'll just tell a little story about what I've done recently, where I found AI to be extremely powerful, and that is I'm working with someone who's doing a thematic analysis of maybe 15 interviews, and they did their thematic analysis, and it was… Very strong, and then we used AI to, um, what we came up We asked the question, if this thematic analysis was the most impactful way to present the findings, then we asked ourselves. And we came up with, um, some other ideas, um, looking across this thematic analysis. It was, like, 7,000 words long. Of other ways that could be more impactful, and we came up with these because statements. So the question was. Why do high-achieving women in computer science? Um, uh, leave computer science. And we came up with these because statements, because this, because this, because this, because that. And then we worked with AI to um, expand upon, refine those, because statements. By feeding in our… analysis, so it didn't feed in any raw data, we fed in our analysis, and so it was sort of In the past, I would have worked with an undergraduate student to do that, and it would have taken a long time. But that was a really, um… purposeful and, um… efficient way to brainstorm those ideas that could make your research findings more impactful. But yeah, ethics, I think at the moment. It's very hard to… justify putting your raw data into AI. And AI does a very poor job at analysing it. It's very basic. Um… But, look, it's moving so fast that I really can't say anything for certain. Kelly, anything you'd like to add? Um, I haven't used it, um, for kind of the same reason that Nikki said, so I don't know that I have any, like, additional thoughts beyond that, but yeah, I think just my concern would always be the privacy of responses that I'm feeding. To an AI, even if I don't imagine they would be… available somewhere, but just… you know, just for complete confidentiality. I think that's totally reasonable. Another question over here, um… Nikki, I feel like you'll love this one. Um, what separates good quality qualitative from lower quality? I know what phenomena to study. The study units, as well as the artifacts to collect, but I'm nervous about whether I'm missing any crucial alternatives to the artifacts to be collected. Top of the current list is finding a collaborator who is more experienced than I am, but I would love other suggestions as well. Thank you. What separates a good qualitative study from a lower qualitative study? Um, yes, that is… a question that I… I always find fascinating. Um, and my colleague and I have developed a a quality framework, and it's something that I always have on my desk. It's this, um, one-pager that I showed you before. That can help you, sort of. Um, tease out, um. Some answers to this question, and… and I would just say. Just to pick on a few points of what differentiates a high-quality study from a low qualitative study, and the first point is that Your findings have to focus on what you actually want to focus on. So you need to have this alignment that this was your research question. And this is the data that you collected that aligns with your research question, and this is your analysis that aligns with your initial focus, so you need that alignment throughout. You would be surprised how many studies are out there where people have a research question, and then their findings discuss something completely different. So you need this alignment, and we call this Theoretical validation, that your results actually focus on what you want them to focus on. And then there are a whole bunch of other things that differentiate a high qualitative study from a lower qualitative study. Um, uh, language is an important one, that you're using appropriate language at every stage in your study, so a higher quality… higher quality qualitative study. We'll think about the language they use that's appropriate when they're speaking to the participants, um. So that participants can understand what you're talking about, you're not talking about in too high a theoretical terms, all the way through to speaking about the language that you're using in your publication, that you're using the terms That, uh, um, are recognizable by your colleagues, so that you can Fit your knowledge contribution into that particular area. And… And in this framework, we speak about a whole lot of other markers as well. But, um, actually, as a general answer, I would say that a high qualitative study, a high-quality qualitative study. Has considered quality issues from the very beginning all the way to the end, whereas a lower quality qualitative study you know, the tongue twister, might just start to consider issues at the end of the study, and think retroactively, okay, what can I write in my research quality section in my paper? So, just having that Mindset and orientation to think about quality at every single stage. I was gonna say something similar of… it's… it's not that it's… there's any, like… there's no perfection standard in qualitative research. There isn't in quantitative either, but in qualitative, we're embracing the subjectivity a lot more. And while there is things that we can and should do to try to… make a validity argument or a reliability argument that what we said we're going to do matches up with what we actually did, that we believe other people would… Would see or have some agreement if they were looking at this data as well. There's just this acknowledgement and a transparency, um. In the paper about, you know, who am I as the researcher, and what did I bring in? What subjectivities are involved at each stage of this process? Um, and I liked… That Nikki said alignment. I think that's a really good word to think about, is that, um, when you think about the way the paper's been framed, that there's consistency and the choices that they're making, that if they're using some kind of framework, that that framework makes sense, um, with the data, that it makes sense with the methodology, with the nature of the data that they're collecting, with the way that they're telling the story, um, that there is A cohesive story being told that… seems to align with the data, um, that the researchers have collected, and that they're presenting to you in excerpts. Awesome. Um, we've made it through all the questions, and we've also just about landed on time, so I feel like this is the time to call it. Um, thank you to everyone who came And joined us in the webinar, and thank you especially to our two presenters, Nikki and Kelly. Um, I've really enjoyed getting to see you two react to each other. I think this was a… I think this is a really informative session for our audience. Well, thank you so much for inviting us. Wonderful. Thank you for inviting me. This will be recorded, so, um… Take a look at the, um, the cause site that, um. You know, hosted this particular webinar, listed it. Um, I think we can probably send out the recording to the usual channels. Uh, but thanks, everyone, for joining us.