Anyone who has attended college in recent years is extremely familiar with the process of course registration.

You pick out your classes in advance. You’re excited for the potential of the new semester, and you’re ready to register.

Your slot opens and … the class you needed for your major is full. You scramble to find a last-minute replacement, and thankfully, it turns out there’s another section. But now the backup section conflicts with an interesting elective that’s offered only once every two years. Your perfect schedule is shot.

For Richard Hoshino, associate teaching professor at the Khoury College of Computer Sciences in Vancouver, this isn’t a problem. It’s an opportunity.

Hoshino solves these kinds of issues—and many more—through scheduling optimization.

Scheduling optimization in a nutshell
High school students have tricky schedule concerns too; they spend four years taking specific classes to prepare for college. Coupled with fulfilling their high school general requirements, students find themselves in a difficult situation when the school registrar doesn’t account for their course preferences. This is where Hoshino swoops in.

“I think it’s extremely important to ensure that teachers can teach the courses they want, while maximizing students getting into the courses they need,” he said.

Hoshino’s goal is to enroll every student into all of their core courses, and as many elective courses as possible.

His most recent client was a high school that employed two teachers who were married to each other and were parents of a young child. Using Python, Hoshino created a rule in the code that these two teachers couldn’t teach in the same timeslot. Considering teacher scheduling preferences, class times, and many other constraints, Hoshino developed the school’s 2021-2022 master timetable.

Five hundred students made more than 2,900 course requests that semester. Hoshino’s master timetable satisfied all but twelve.

“Through my research, I’m able to solve these real-world problems,” explained Hoshino. “Then I’m able to achieve results like this, where I can get nearly every single student into every single one of their classes while satisfying hundreds of constraints that need to be met from the perspective of the school administrators as well as the teachers themselves.”

Those twelve students weren’t completely out of luck, though. Hoshino spoke to the school registrar, who found backup options for each of them.

Scheduling optimization isn’t just for school timetables though. Hoshino has also helped Nippon Professional Baseball, Japan’s top-flight league, slash their carbon emissions by creating a schedule that minimizes the distance that teams travel. On his work in Japan, he said, “It’s really exciting to see how math can be used in such elegant and surprising ways. During my three years in Tokyo, I became one of the world’s experts in a field called sport tournament scheduling.”

Governments can leverage scheduling optimization as well. In the case of the Canada Border Services Agency, Hoshino created an improved risk-scoring algorithm for marine cargo containers coming into Canada so the CBSA could forgo the common practice of geographical profiling—a method that judged the relative security of the cargo solely based on the export country. Hoshino designed the algorithm to consider the importer and exporter’s prior business transactions, the freight forwarder’s criminal record, and the transported item’s origin not logically fitting with the country it supposedly came from, among others. These factors are risk indicators for cargo ships that may be carrying weapons or drugs.

Hoshino explained, “It was quite exciting to use techniques in machine learning—and techniques in data mining that we teach at Northeastern—to help design algorithms that led to improved rates of drug and weapon seizures, with no geographical profiling.”

As the owner and principal of Hoshino Math Services, he runs professional development workshops for high school teachers and math workshops for students. The bulk of his consulting work is in using his expertise and scheduling optimization to create timetables for businesses and schools. His endeavors landed him the Adrien Pouliot Award—the Canadian Mathematical Society’s recognition of sustained, significant contributions to Canadian math education. Oh, and he’s the youngest winner in the 26-year history of the award.

From practice to pedagogy
So how did Hoshino land at Northeastern? He stumbled across an article about the university’s new Vancouver campus and was immediately drawn to it. “I became very attracted to the Align MSCS program whose values and mission were amazingly parallel to my own and what I wanted to do as an educator.”

During the summer of 2021, Hoshino taught a math prep course to nearly 100 incoming Northeastern students across Northeastern’s campus network. On the course, he enthusiastically remarked, “In this math prep course that I’m teaching right now, I’m working with a paralegal, a nurse, a welder, a lawyer, and several U.S. Army veterans. It’s just so exciting to have such a diverse array of life experiences in the classroom. That’s what attracted me to teach at Northeastern.”

Hoshino’s creativity doesn’t stop at the classroom, though. He said, “By picking rich problems, we can teach an entire curriculum where students uncover the material for themselves, rather than a typical class where the teacher covers the material for the students.” He added, “I hate using the word ‘lecture.’ I call my classes ‘problem-solving workshops’ because I provide students with meaningful problems to tackle.”

In the process of solving those problems, Hoshino’s students uncover key concepts, techniques, and methods on their own.

“When I run workshops for teachers, they’re often looking for problems to share with their students so that they can teach in this way. So in these workshops, I actually give them the same problems that they would give their students to watch them work on them, wrestle with them, and argue the solutions together so that they can make the key discoveries themselves,” said Hoshino.

Inspired by his students’ backgrounds, professional experience, and academic experience, he gets a sense of fulfillment when he can help make the computer science field more inclusive. “The college’s mission, ‘computer science for all,’ doesn’t work if every single graduate of our program looks like me.”

Hoshino concluded, “Teaching at Northeastern has been rewarding, because so much of success in this program is about confidence. For these students, whether you believe you can or you can’t, you’re right.”

Subscribe to the Khoury College newsletter

Dear Khoury College Community,

This week, the appointment of Elizabeth Mynatt as the new Dean of the Khoury College of Computer Sciences was announced by Provost Madigan, on behalf of President Aoun. Her term begins January 2, 2022.

At the college, we are thrilled by this news. Many of us, including me, had several opportunities to meet incoming Dean Mynatt in a series of interviews, conversations, and public forums throughout the recruitment process. She has extensive experience as a researcher, teacher, and educational leader. I am especially enthusiastic about Beth’s commitment to the emerging interdisciplinary research and education here in Khoury, as well as her passion for understanding the impact that computing has on our everyday lives.

She impressed others across the university. I’d like to share with you an excerpt from Provost Madigan’s announcement on Monday, November 15, which he emailed to the entire Northeastern community of faculty, staff, and students:

Elizabeth is currently a Distinguished Professor of Interactive Computing and Executive Director of the Institute for People and Technology at the Georgia Institute of Technology. Prior to her role as Executive Director, she served as Director of the GVU Center, the home of computing research innovation at Georgia Tech. She has served in various leadership roles, including Associate Dean of the College of Computing and Director of the Aware Home Research Initiative

Throughout her tenure at Georgia Tech, Elizabeth has played a key leadership role in the design and management of two premier educational programs: the Ph.D. in Human-Centered Computing and the M.S. program in Human-Computer Interaction—including directing the HCC program from its inception until Fall 2006. Once she took over the M.S. program, she spent six years building the quality, curriculum, and opportunities for students to integrate within faculty and research communities

Elizabeth’s research interests focus on Human-Centered Computing, HCI, Health Informatics, Ubiquitous Computing, and Assistive Technologies. Currently she is focused on the implications, as well as the opportunities, that pervasive computing technology has on everyday life, particularly within home health care and information sharing in office environments

Elizabeth’s innovative approach to interdisciplinary collaboration and expertise in tackling issues that impact everyday life are key components of Northeastern’s ethos. I look forward to the new heights the college will achieve under her leadership

Provost Madigan’s words capture Beth’s impressive achievements, personal qualities, and sincere commitment to her new role at Khoury College. I look forward to working with Beth as both a colleague and a leader.

Please join me and our entire Khoury Leadership Team in extending a warm welcome to Beth as she continues to advance our work and support our students!

Growing up in the 1980s, David Makar was fascinated with the development of the internet once it was introduced in 1994—which just so happened to be the year he began studying at Khoury College of Computer Sciences.

“I liked the possibility of technology making things easier for other people, to save time, and the feeling that what we could do with computers was magic,” said Makar. “We could now do stuff people thought was impossible, and I wanted to be a part of that.”

After graduating in 1999 with an undergraduate degree in computer science, Makar began working in software deployment training. With the desire to use tech to help others, Makar explored other careers in entrepreneurship.

Soon after leaving his job, Makar launched a website design business in 2003, where he built developed software for small businesses. This venture made him realize how much he enjoyed working with small business owners—helping them grow their business and giving them an online presence.

In 2010, Makar worked as a coach and consultant, helping business owners and entrepreneurs market their business through networking. Over the course of five years, he became an international speaker— delivering keynotes at conferences and hosting business coaching workshops.

Reflecting back on his experiences, Makar found that he could make a larger impact by working with others.

Fast forward to the present day where Makar assists independent businesses on a larger scale through his role as the chief customer officer (CCO) of Rosie.

Rosie’s pursuit to grow the local economy

Rosie is an e-commerce software development company for the independent grocery space. With over 42,500 supermarkets and grocery stores in the US, Rosie gives local grocers a platform to sell their products online for delivery or pickup. In an ever-changing retail space, the company enables local businesses to make their products accessible to as many consumers as possible.

As the CCO, Makar’s primary role is to lead the customer success team, which acts as the liaison between the customer—either a retailer or an online shopper—and the software team. Makar’s success is powered by his knowledge of the computer science industry.

“The degree [in computer science] helps me build the bridge from the online shopper, who may or may not have a lot of software experience, to the grocery store, which is really key to customer service and operations and getting all their shelves stocked for what customers want,” said Makar. “It helps me bridge that to the software team who builds the software that replicates that entire in-store experience, like rewards programs and coupons, and brings that to the online platform.”

In leading the customer success team, Makar ensures that the platform is working properly for all of its users. Another key function is to condense user feedback and it send to the software team so that they can build the right tools and features to improve customer experience.

Beyond giving local retailers the ability to enter the world of online groceries, Rosie also strives to make food more accessible for individuals who may struggle with shopping, because of a lack of time or perhaps a disability. Over its last eight years in business, Rosie has been able to grow within the e-commerce space and continuously help more local businesses by updating its software and bringing new local grocers on board.

Since the start of the pandemic, individual enrollment in the Supplemental Nutrition Assistance Program (SNAP) has increased by four million. In line with its mission to increase accessibility, Rosie is planning to roll out a system for online SNAP payments in 2022. This project would increase the availability of healthy and nutritious groceries for SNAP recipients.

“Often these are people who are homebound, low-income, or sometimes in food deserts,” said Makar, referring to urban neighborhoods that lack access to a local source of affordable or good-quality food. “So, this really just opens up food access.”

Another exciting project in the works is the rollout of retailer-specific apps, which would give “hundreds and thousands of grocery stores” their own online presence, which could help their local business immensely, explained Makar.

Looking at all the work that Rosie is doing to help independent businesses and individuals who are facing food issues, Makar is grateful that he is able to engage in work that helps him fulfill his own goals of helping others by impacting an integral part of their everyday life. From an early age, he knew he wanted to help people in some way. By working at Rosie, he helps small business owners find success and make food more accessible for those who need it.

“There are people who are hungry, even in this country, who just don’t have access to food,” said Makar. “So being part of this company is helping me live that part of my personal mission [to help increase food accessibility] every day is huge.”

More than just a degree

Reflecting back on his time at Khoury College, Makar explained that while his computer science courses taught him the skills he needed to succeed in the field, he especially valued the Khoury College community for all of its networking value.

One way he built his network was by getting together with fellow students to re-establish Northeastern’s chapter of the Association for Computing Machinery (ACM), the largest professional association for computer science.

“Networking is more than just computers and your peers,” said Makar, reminding current students to take advantage of the resource around them, “it’s upperclassmen, administrators, professors, and people in other colleges in the university.”

Being in a role that requires him to correspond with many people across a variety of teams, Makar also commented on the importance of communication in any field.

“A lot of success is driven by just listening— less communication of how to say the right thing and more of, ‘how do I listen to people and really hear what they are saying?’ How do I practice better empathy? How do I understand what they are trying to say even if it isn’t exactly what they are saying?” explained Makar.

By taking a mix of business, communication, and computer science classes at Northeastern, Makar built a strong professional foundation for himself. His main takeaway for current students is to take a look at how they might want to apply their degrees to their career and make use of Khoury College’s community and that of the greater Northeastern University as well. “A few times a month, I reach out to my Northeastern network for advice and ideas. My relationships from the last 25 years have been invaluable to my success.”

“I know I wouldn’t have been able to do what I do professionally, or do it this well, without Khoury College.”

If you sit down to have a conversation with someone, you begin to build the foundation of your communication before either of you open your mouths. As soon as two people become aware of each other they begin to form mental models, attempting to understand each other by ascribing mental states to them based on their behavior, their words, and your own previous, similar interactions with others. The better your mental models of one another, the more easily you’ll be able to understand the other person’s motivations and perspective, and the more effectively you’ll be able to communicate.

This interaction, according to Khoury College of Computer Sciences professor Stacy Marsella, is the essence of the psychological concept called theory of mind.

An interdisciplinary professor jointly appointed to Northeastern’s Khoury College of Computer Sciences and the Department of Psychology in the College of Science, Marsella is deeply fascinated by the human mind. His research interests run the gamut from psychology to disaster relief to robotics, and over the course of many years a key question has consistently made its way to the forefront: What role can theory of mind play in creating useful human-computer interactions?

Early experiments with AI and theory of mind

Initially, Marsella’s background actually wasn’t in computer science at all—as a young researcher, he was focused on psychology and economics. He first became interested in AI as a way to create better economic models, more rooted in human psychology. But Marsella has always loved to build things, and when he discovered the potential AI had to build detailed psychological tools he was hooked.

For many years, he researched ways to build narrative-based training simulations driven by AI that used theory of mind. A narrative-based training situation takes an emotionally charged situation — for instance, a doctor informing a patient of their terminal illness — and allows that doctor to practice their approach and delivery ahead of time. The simulations can be a little like video games, in which the dialogue options you select impact how your conversation partner responds. As all video game players know, though, selecting dialogue options to advance a selected storyline bears little resemblance to real life, in which each interlocutor will have unique biases, emotions, needs, and ways of understanding the world that affect the unfolding narrative.

What role can theory of mind play in creating better training simulations? Here’s where Marsella’s work comes in. Giving that virtual conversation partner a complete mental model gives the trainee something to attempt to understand. By first crowdsourcing a vast array of potential scenarios and then using AI to weave them into a dense web of potential avenues of interaction, Marsella and his team created simulations that could go in any number of ways—but always in a way that’s consistent with the specific virtual conversation partner’s personality, beliefs, desires, and emotions.

“[The simulated conversation partners are] acting in very autonomous and novel ways, essentially, which forces the person interacting with them to think about, what are their motivations? What are they trying to achieve?” explained Marsella. “It was an attempt to not only give the agent a model of the human, but to force the human to develop a model of the agent.”

Although this research has since concluded, it laid much of the groundwork for Marsella’s other, more recent projects. If theory of mind can successfully create better virtual conversation partners, where else can it be applied to human-computer interactions?

Robots, earthquakes, and disaster relief

Robots are massively useful disaster relief tools — but at least for now, tools are all they are. Though the work environments are dangerous, humans are still needed to ‘supervise’ the robots. Marsella believes that they have the potential to become true collaborators in our disaster relief efforts though, and that theory of mind may be one of the keys to unlocking that potential. The Robot Human Interaction Project is run in conjunction with assistant professor Chris Amato, whose focus is on the robotic side of the interaction, while Marsella focuses on the human side.

In order to develop a theory of mind model, the team is first studying humans in simulated rescue environments to create models of how people are likely to act. “Then, using those models to inform the robots, the team is able to help their robots develop a theory of mind in regards to their human counterparts. This facilitates a level of collaboration that’s otherwise impossible.

“Effective teamwork usually assumes that people have an understanding, and an expectation about, what everyone else is doing in the team. If that breaks, if these entities don’t have that, then they can’t be responsive to each other,” Marsella explained. “To be adaptive as a team and respond to uncertainty, you need to have an ability to predict the other entity, and predict that other entity under changing conditions. That, essentially, is what theory of mind is giving.”

If a novel situation arises — say, a wall in a building where a robot is conducting search-and-rescue operations has collapsed — the robot without a theory of mind would likely need to either stop and await new instructions, or risk injuring itself and its collaborators. The robot recognizes that the wall is down, and the human collaborator might know as well, but without a theory of mind, the robot is unable to know if the human knows the wall is down (or if the human knows that the robot knows that they know, etc.). If the robot was to react autonomously — for instance, by digging through the wall — without an understanding of the human’s knowledge, goals, and likely reactions, it could negatively impact them both — for instance, when the wall collapsed because people were still walking on top of it, unaware that the robot had proceeded with the rescue.

“In order to get the robots to really work well with people, they need to understand people better than they do now. You can see this with your robot vacuum cleaner, or your other robots like Alexa…they are very, very limited in what they can do, what they expect, and how they can interact with you,” explained Amato. “I think having things like theory of mind are necessary in order to have the robots get to the next level, and be able to think about why the human is saying this, what they’re going to do as a consequence, and how the robot’s actions will influence the people’s thoughts of them.”

“That is essentially where human-AI interaction is going,” added Marsella. “This is something that’s evolving in the whole AI community, this notion of human-AI interaction where there is much more of a balance. There’s a transparency between the robot and the human in the sense that they have models of each other.”

Theory of mind on a superstorm scale

Marsella is also working on another project that combines disaster relief and theory of mind: a program to understand and simulate if and when people evacuate in the face of an oncoming hurricane. The project accounts for factors like what kind of official warnings have been issued, people’s likely emotional states, and the time before the storm makes landfall to model when people flee — and what factors might make them more likely to flee, more quickly.

“There’s an interesting close connection between theory of mind and emotion, because a lot of emotions are social emotions. Things like anger, or shame, or guilt — these are emotions that we form because of the mental models we have (of) ourselves, and how they relate to other people in the environment,” said Marsella. In this way, one’s individual beliefs about the minds of themselves and others, and the emotions they feel about those beliefs, help to build the framework they would use to make decisions.

In progress since 2017, the project is now to a stage where the model is ready to get into the hands of policymakers. Marsella hopes it can be used to hone evacuation messaging to ensure the maximum number of people make it to safety.

“If you have a model that makes predictions about whether or not people are going to evacuate, and what are the reasons they’re not evacuating, then you can actually systematically play with that model and see, how do we convince them to evacuate?,” said Marsella. Local leaders could use the model to run simulations with different messages to predict what techniques would effectively motivate people to evacuate.

Marsella’s work displays how computer science is, at its heart, an intensely human discipline. In order to make robots and virtual agents who can communicate smoothly with real people, researchers like Marsella and Amato need to understand the parts of our minds that are difficult to communicate. To create a model maximizing the number of people who will evacuate in the face of a coming storm, researchers and policymakers need to be able to understand and empathize with the reasons they might want to stay. Time and time again, interdisciplinary research brings an element of humanity to computer science that just makes it click.

Amato was able to neatly sum up what it is that Marsella and his theory of mind-based approach uniquely bring to his research.

“It’s [work that is] not doable by someone that doesn’t know about humans. How do you get robots to work with humans, if you don’t know about humans?”

---