Mini: The Secret Sauce: Multidisciplinary Curiosity (Shawn Weil, Courtney Dean, and Evan Oster)

When studying human performance, we need multidisciplinary curiosity along with the combination of methods to crack the nut of the difficult problem that capturing human performance represent. What does that look like? MINDWORKS host, Daniel Serfaty, sits down with Dr. Shawn Weil, Courtney Dean, and Evan Oster of Aptima to learn more!

Listen to the entire interview in Mission-Critical Environments: Can we improve human performance? With Shawn Weil, Courtney Dean, and Evan Oster. 

Transcript

Daniel Serfaty: Okay. So we have a lab scientist, a golfer, and a teacher. Sounds like the beginning of a wonderful joke, but it is actually not just a joke. He's actually the secret ingredient of this very challenging but extremely rewarding domain of studying human performance. We need that multidisciplinary curiosity and that combination of methods to crack the nut of the difficult problem that capturing human performance represent. So perhaps in one or two sentences each one of you can take our audience, that is, what do you do? Now we understand your background, but what do you do at Aptima? You can just pick one project or one activity that really you think represents what you do. And I'm going to scramble a little bit the order and start with Courtney. What do you do, Courtney?

Courtney Dean: So at Aptima, my primary focus is the most directly human performance, you could just about put that as line one on my resume. I've been focused on developing measures of human performance and training contexts for a variety of domains almost since the first day that I stepped through the doors of the company. This involves sitting down with subject-matter experts in their respective domains and identifying what it is that differentiates somebody who's competent from somebody who's incompetent or somebody who's excelling in their field versus someone who's not. Breaking that down to the micro level, what are the specific behaviors that we can observe that indicate that somebody possesses the knowledge, skills, and abilities that are necessary to complete said tasks? And we've developed a pretty effective methodology for eliciting that information. And I've just run with that methodology to apply it to many different domains and utilize that to both gain an understanding about the domain that we're focused on there, and then produce a series of metrics that those individuals can then take with them into their training environment and utilize to achieve some goodness on the part of the trainees or the learners in their environment.

Daniel Serfaty: So in a sense, you have a scalpel like a surgeon, which is a method you mentioned. And you are trying to understand, deconstruct the nature of mastery or expertise in the domain that you study, whether they are fighter pilots or police officers or surgeons actually. Is that what you're saying, you basically are an analyst that decompose that and say, "These are the ingredients of mastery"?

Courtney Dean: Yeah. I would say that it's a little bit less elegant than a scalpel. The truth is that it's a little bit more along the lines of a sledgehammer and some super glue.

Daniel Serfaty: Right. Well, we'll talk about those tools in a minute. So what is it that you do at Aptima, Evan?

Evan Oster: That's a great question because when friends asked me that, I have a different answer each time, and it really depends on the day. When I am looking at human performance at a high level, that's me reviewing and conducting research on training to improve human performance. But more specifically, what that looks like is working with colleagues who are experts in their field in small teams to be able to innovate some solution that satisfies a customer's need. So that can be improving decision-making, it can be improving the instructors, it can be helping to improve the students. And I think that's what's really unique is you can take a look at human performance challenges from multiple different perspectives and multiple different ways. And each time you improve something, it improves the whole.

Daniel Serfaty: Okay. So you're not an engineer, you're not a software engineer. Many of the solutions you dream up or Courtney dreams up end up in instantiation software. How do you establish a dialogue with folks that are actually about coding and architecting software systems?

Evan Oster: So that's also a really good question, something that we face every day. And I think what it comes back to is really in understanding and good communication with one another. Relationships are huge, right? So understanding how different people work, how they view problems, how they see things and valuing those differences. And being able to clear out space for them to work as they can work best. At the same time, having a common framework and lexicon for what it is that need to be accomplished. How many times have you heard someone say the word domain? And that means one thing to one person, something to someone else, something to someone else. And so being able to have that common language and framework to operate from really helps to inform that end goal and form the solution.

Daniel Serfaty: Well, we'll come back again to this dialogue. But Shawn, I know what you do. You're the executive vice president for strategy at Aptima, but that's not the only thing you do. It sounds pretty managerial and executive, but actually you're still scientists. So how do you bring that science into your job?

Shawn Weil: It's a really good point, Daniel. I think about this in a number of ways. I wear different hats in the company, and I wear different hats professionally. Because I have a corporate role, it allows me to think about human performance in a systemic way using systems thinking. So construed broadly, that could be looking at human performance of teams and how they communicate or how they interact with artificial intelligence. It could be looking at how we bring together measurement from different modalities, observer-based or systems-based. Or it could be understanding the link between the physiological, the behavioral, and the cognitive, and trying to make sense of that. But the other hat that I wear, that executive hat is the one where I'm helping our engineers and scientists, both Aptima staff and our partners really understand what the end users' needs are. There's something intrinsically wonderful about human performance that satisfies the intellectual curiosity of scientists and engineers. But then you need to figure out how to frame that in a way that's going to be really beneficial societally, and that requires a different perspective. So it's a pleasure of mine to be able to take on that role, put that hat on and work with our diverse staff to help them help those customers.

Daniel Serfaty: That's a very good way to describe this constant stitching of ideas into something that not just the market, it's too abstract, but the human user, the human learner out there needs. So I'm going to ask you Shawn, think of an instance of an aha moment in your professional life when you suddenly realized something new or how to articulate insight into a scientific fact, a project, something, an aha moment?

Shawn Weil: I've had a couple of those aha moments. But there's a common thread in all of them, and that is messiness and complexity. So when I was first exposed to ideas of human cognition and human performance, human behavior, they were very neatly compartmentalized. You had these kinds of behaviors in these situations, this decision-making in these kinds of environments. This kind of communication patterns with these kinds of people, and you could study them independently. When I was in graduate school, I first got exposed to the complexity of just having two people talk to each other and two people trying to coordinate towards a common goal. So I think an aha moment came early on in my career at Aptima when I was working on a program about command control, and we were doing some experimentation. And when you start looking at not two people, but just five people in a controlled environment, the amount of complexity, the multiplicity of ways that you can look at human performance, it's just staggering. So how do you then do something useful, collect something useful? It may not be collecting everything. So the aha moment for me in that scenario was really saying to myself, what questions do we need to answer? And from all of the chaos and all of the complexity, do we know how to zero in on that subset that's really going to give us some insight that's going to help improve the performance of that organization?

Daniel Serfaty: That's interesting because that's a dimension that all of us had to face quite early in our carrier, you're right. This notion of not everything is neatly compartmentalized the way it was in the lab in graduate school. And dealing with these, you call it chaos or complexity or messiness is really a skill that eventually we all need to acquire. Courtney and Evan, any aha moment you want to share with the audience?

Evan Oster: I had one for sure pretty early on. So we were at an army installation, and we were there to help instructors provide tailor feedback to their students. And that was going to be through an adaptive instructional decision tool. And I was conducting a focus group. And I asked the instructors what made their job challenging? And they started off by complaining about the students. They started to tell all these stories about how lazy they were, and they constantly make these common mistakes. And I looked up one of the main instructors there and asked, so why do you think they're doing this? And he paused and thought about it, and he said, "Well, it's because the job is really hard." And he gave the example of, if you stacked all of their manuals on top of one another, they'd be over six feet tall," taller than most of the trainees who were there. And then soon some of the other instructors started talking about other things that were challenging and why they're hard. And really just from one question, the conversation and the culture shifted from complaining to more of a sense of compassion where then they're starting to put themselves in the student's shoes and they're able to resonate with why it's so difficult. And as the instructor, I mean, this is easy for them, they'd have lots of practice and they've done it for years. But really what we were able to do is before we even did any software, any development, we were able to set a culture and a framework for why are the instructors there? They're really there to help. And from that moment forward, it helps shape what that final solution would look like. That was a big aha moment for me that this human performance, similar to what Shawn said, it's messy. You look at it from so many different perspectives, and it's not clean-cut and clear. You need to go in and feel your way around and figure out what's going on and see what the best path is moving forward.

Daniel Serfaty: Courtney, you got one of those ahas?

Courtney Dean: I have one, but I actually had to relearn. I remember watching a retired army operator taking me and bring the soldier down with him. And they had this conversation, it was very quiet and very reserved, and it was technical. He was pantomiming and gesturing with his hands to mimic the use of a rifle. And there was probably a little bit of engineering and physics associated with the drop of the bullet or the rise of the muzzle, et cetera. I don't know, I wasn't totally privy to the words that were coming out of her mouth. But I watched the body language of this settled, isolated conversation. And it was before I had been on a firing range with drill instructors and new privates or private recruits where the screaming was only overshadowed by the sound of muzzle fire. Then we started to see drill instructors taking knees next to young privates who were practically quaking in their boots if they don't hit the target just right.  And you started to see some change in behavior because they went from, "I'm trying to avoid this screaming, this punishment," to, "I'm starting get some context and some understanding and some support. And I understand that maybe my inability to hit the target as accurately as I'm supposed to right now is not a personal flaw that I'm never going to be able to get over. And then I constantly forgot that. And one day I had a friend in town and my new walking one and a half year old son, and we were going sledding. And he didn't have very good mittens on or children can't keep their mittens on and snow was creeping in between his hands and his sleeves. And I wanted to put hi, on this sled because it's going to be so exciting and he didn't want to. And he started to get scared and started to cry. And I found myself trying to force this child onto the sled. You just get on the sled, and everything is going to be wonderful. And then I stopped for a second because I was realizing that wasn't happening. But in my head, the gears were just barely clicking. What is going on here? And my friend who hadn't had a child yet took a knee and talked to my son and calmed my son down, and he got the situation back under control. And I looked at that and I thought, "Well, I guess I'm one of those old drill instructors right now, and it's time to become one of those new drill instructors."

Daniel Serfaty: These are wonderful stories, all the three stories that you're telling me. Which really remind all of us that human learning, which is the other side of human performance and the ability to improve a skill or to acquire a skill is more extremely complex and messy. But at the same time, there is an intimacy to it between the instructor and the learner, between the tool and the person reading the data that we as engineer, as designer, as scientists have to take into account. Thank you for using those stories, they illustrate very well better than any theory our job is both so hard but also so rewarding.