If you work for a salary or an hourly wage, you get paid for all the tasks you do during work hours—answering emails, making sure your paycheck is deposited correctly, figuring out what task you’ll complete next. You may even be getting paid to read this article right now as you work. But if you’re a crowdsourced worker on a platform like Mechanical Turk cleaning and labeling AI training data, the critical tasks that keep your work, well, working—but that aren’t the actual job of cleaning and labeling data—are unpaid. For a long time, they weren’t even measured.

This invisible labor by crowdsourced AI workers is the crux of new research by Khoury College of Computer Sciences assistant professor Saiph Savage, her doctoral student researcher (and first author) Carlos Toxtli, and Microsoft researcher Siddharth Suri. Their work was recently honored with the Impact Award at the 24th ACM Conference on Computer-Supported Cooperative Work and Social Computing (CSCW), one of the top conferences in the field of human-computer interaction. Their paper, “Quantifying the Invisible Labor in Crowd Work”, examines how much of their time crowdsource workers are spending on tasks that are critical to their job, but unpaid.

To measure invisible labor, the team built a browser extension that labeled and measured the different tasks being performed on crowdsource platforms. They had 100 workers who varied across axes like skill level, country, language, disability, and working style install the browser, and then the research team collected and analyzed the data.

“We found that it’s a big part of the [total] labor, invisible labor. In fact, it is one-third of all the labor that they do on the platform,” said Toxtli. “In other types of work…you have a fixed amount per hour, independent of how productive you are. For [crowdsource workers], they have to be productive all the time.”

The most common types of invisible labor were related to payments—checking earnings, requesting their pay for hours worked, etc.—and hypervigilance. Because the supply of crowdsource workers often exceeds demand, many of the tasks on offer pay very little, and snapping up well-paid tasks requires constantly watching the platform. Companies aren’t paying workers to perform this hypervigilance, but without it, workers won’t get paid at all.

All this invisible labor has a serious impact on their wages. When accounting for the time put into invisible labor, the researchers found that a number of crowdsource workers are making below the federal minimum wage of $7.25 an hour.

The Impact Award is granted to papers that demonstrate critical real-world impact, which Savage said has been the goal of this project all along.

“Our goal is that, with this type of tool, we can empower workers to go to policymakers and say ‘hey, a large number of us are impacted by invisible labor. It’s consuming 33 percent of our time, and it’s costing us this amount of money. We need to do something, change needs to happen here,” explained Savage. “If we truly want to create change for these platforms, positive change, we need to understand what is happening inside.”

In addition to giving crowdsource workers the data they need to push for policy change, the team hopes their research can bring visibility to a part of AI that’s often overlooked, despite being critical to the industry’s success.

“They are invisible workers, because most people don’t know the AI [we] use is powered by them,” said Toxtli. Training a single algorithm takes thousands of pieces of data, each cleaned and labeled by a human being; at least for now, machine learning can’t learn without being powered by people. “They really want to be recognized.”

Melanie Tory strives to create real-world impact—a goal that led her to the Roux Institute at Northeastern University. Here, she’s taking on a brand-new role as director of data visualization research.

Appointed in May, Tory aims to lead and collaborate on projects that will affect change for the institute’s partner organizations, and to realize the Roux’s mission of fostering innovation and generating economic prosperity to the region.

“I want people to think of the Roux Institute as the place for data viz research — where they can do research that has real impact in the world, that is right at the forefront of the field, and that is also very relevant to outside organizations that they get to work with,” Tory said. “My vision is to grow this team and bring it visibility and impact in the world.”

Tory’s expertise lies in both research and teaching. For nine years, she taught as a faculty member at the University of Victoria in British Columbia, and up until this past May, she worked at Tableau Software in Palo Alto, California.

“I’ve always been at the boundary between academia and industry,” Tory said. “When I was a professor before, I felt like a bit of an imposter, in teaching students how to do software engineering when I hadn’t actually done it myself. I really wanted to do something that would allow me to have more real-world experience and to see how the sausage got made in industry, so that got me to Tableau.”

Her love for impact-driven work then solidified at Tableau. For much of her career, Tory’s work was purely research-focused until she worked on a project at Tableau with the goal of making data visualization more accessible to non-experts. This involved building a product prototype and getting feedback directly from the public, shifting her role from an academic researcher to an applied user researcher.

“The work I was doing was no longer for the purpose of writing publications but instead to figure out how to make that product work in practice. That was a big shift [in my career],” she said. “It forced me to learn a lot more about how software gets made … and how I could use those same research skills I had in a different way and in a different context to bring value to a product.”

She believes her new directorship is the culmination of her past experiences, as she’s found herself back at an academic institution, but this time with a focus on industry impact.

“I like to push the boundaries of my field and do things that are new and novel, but also I really care about solving problems that matter to people in the world. I want to see the results of my work make it into the hands of people who can use them.”

Before Tory found her love for data visualization, she was a microbiology major at the University of British Columbia and completed co-ops related to microbiology and chemistry. Though she realized her aversion to wet lab work, Tory discovered that she enjoyed manipulating and analyzing her data. Thus, after graduating with her undergraduate degree in biology, Tory decided to pursue her doctorate in computer science at Simon Fraser University.

To her, the best part about data visualization is how it intersects, and benefits, those in other fields of study.

“I approach visualization from the discipline of computer science, but it also draws on all kinds of other things, like design, cognitive science, and psychology,” Tory said. “You can apply data to any other field, so when I go into a design study, I get to learn about a discipline that I never knew about before, like construction management or engineering. In a way, I never had to pick a discipline because I always get to be learning about new ones.”

She particularly emphasizes the significance of psychology when creating data visualizations, since they serve to help the public consume information in ways that are best suited for them.

“The human side is the most important part of the equation,” she said. “It’s the reason we build visualization tools—to enable humans. We have to understand how humans behave and how they work and design to fit that, rather than expect people to change to fit the tool.”

Currently, Tory is in the process of building a research team. She envisions her lab to be inherently collaborative, thriving off working with others, especially those in other domains. Her lab at the Roux will also be more applied than a traditional academic lab and more akin to an industry research lab, where “the measurements of success are less about the number of grants, research papers published, and patents, and more about the amount of impact that we have on partners and the topics that they care about.”

Her main active project now is in partnership with Northeastern’s PROTECT Project, which examines the effects of environmental contaminants on preterm birth in Puerto Rico. Through visualization, Tory and Liudas Panavas, a doctoral student from the Khoury College of Computer Sciences, are transforming the data to inform the project’s researchers.

At the same time, she is brainstorming future projects, such as exploring the intersection of visualization and artificial intelligence as well as creating “situated visualizations.” The latter are designed to fit the context of a place, whether it be a factory floor or a home, so that users can interact with them more effectively.

Looking back on her career thus far, Tory is grateful for those she’s met along her professional journey and believes they are the reason for her growth.

“The visualization field is special because it is so collaborative. All of the people who I’ve interacted with in my field are excited to work on things together and are very supportive of each other,” she said. “It’s an amazing community to be part of, and I attribute a lot of my success to the support of that whole research community.”

“How you break down stereotypes is by taking up space—be loud, be comfortable, and speak out for yourself. Take care of yourself, and take care of others like you,” said Arturo Herrera, founder of the entertainment tech startup ShoCast. “By simply doing that and being present and visible—it’s a big contribution to the environment you are in.”

Herrera was one of four panelists for Out in Tech, a conversation hosted by Khoury College of Computer Sciences featuring LGBTQ+ professionals in tech roles. The panelists shared stories and advice based on their unique professional, academic, and personal experiences with current students. The goal of the event was to shed light on what it is like as an LGBTQ+ person working in tech, and how to be comfortable within the workspace.

“[This event] is not only an opportunity to celebrate National Coming Out Day but also LGBT History Month,” said Taylor Sprague, one of the event’s organizers and a co-op coordinator at Khoury College.

Ricky Lee, a software engineer at Google, expressed his gratitude for the opportunity to participate in Out in Tech, saying, “I have found meaning in participating in these LGBTQ panels. It just means a lot to be the representation you wanted to see when you were younger.”

Having come from different academic and career paths, the panelists presented their unique journeys as members of the queer community working in tech. However, in reflecting and sharing their unique experiences, these panelists gave students tips on how to be their true selves all while navigating challenges of the workplace.

John Huân Vu, engineering program manager at PayPal and another one of the event’s panelists, acknowledged that for queer people, “Being authentic at work can be very very difficult, just being out and being who you are completely.”

For Vu, he feels lucky that the company he currently works at supports him fully and has an inclusive work environment.

“[Your] career is not fixated on your salary—it is also fixated on your whole authentic self,” advised Vu, encouraging students to find companies that will respect them rather than settling for jobs with only good pay.

The other panelists agreed that being true to who you are plays a large role in finding success and comfort within your job.

“I don’t want to act while I’m at work, and it was just so tiring to be the person who they wanted me to be,” said panelist Grace Collins, CEO of Snowbright Studios, reflecting on one of their past jobs. “You can’t give into erasure. When you start giving in to that, you start losing little bits of your identity.”

In Herrera’s situation, he found that when he censored his personality during work meetings, it limited his potential.

“I’ve been able to let go of those notions [that I can’t truly be myself], which has opened me up to more opportunities,” said Herrera. “When you can be pretty authentic, people connect to that.”

The panelists also emphasized that students should take advantage of mentors and the mentorship resources available at the university level.

READ: https://devsite.khoury.northeastern.edu/northeasterns-chapter-of-ostem-equips-lgbtq-students-with-academic-personal-and-professional-success/

“I didn’t have mentors back then and it was during a difficult time,” said Vu. “You are lucky to have this because so many people paved the way for you to be safer in this wonderful place.”

As current students prepare to enter the workforce, having a mentor by their side can provide invaluable assistance because mentors use what they have learned throughout their time in the industry to guide their mentees—often acting as role models.

Reflecting on having a mentor and being integrated into the queer community at his company, Lee shared that it has “not only helped me grow in terms of my career, but also as a person.”

Although times are changing and the workplace is becoming more accepting, “discrimination still happens,” said Vu.

On that sentiment, the panelists urged students to use their voices to change the industry—shaping it for themselves and those who will enter the field after them.

“You have to connect with people on an individual level,” said Collins, encouraging students to stay true to themselves and find a space in their workplace where they can feel included.

On an industry level, Vu finds it important to also make changes within the actual work being done in the field. In his work, Vu is pursuing “Justice in Design,” an ideology concerned with identifying implicit bias and racism that may be embedded in code or design. In doing so, the coder/designer signals that “I’m here for equality, and my product is allowed to be used by anyone regardless of their background,” explained Vu.

By sharing all of their success stories, the panelists hoped to leave students feeling confident that they have enough resources and power to feel safe and accepted in the tech industry as members of the queer community.

Examining the many facts on Michelle Borkin’s resumé inevitably provokes one question:

How did a lifelong astronomy devotee, one who spent several years earning a doctoral degree in applied physics, just win an uber-prestigious data visualization award?

On October 26, the IEEE Visualization and Graphics Technical Committee (VGTC) bestowed their first-ever Significant New Researcher Award on the Khoury College of Computer Sciences professor. The award recognizes important data visualization research conducted within seven years of the recipient’s most recent degree.

“I’m thrilled that Professor Borkin has been awarded the IEEE VGTC award for early career researchers,” Interim Dean Alan Mislove said. “This is the highest award that junior faculty members can receive in the field of data visualization.”

The committee’s recognition of Borkin’s “work on visualization memorability, visualization evaluation methods, and their application to astrophysics and medicine” speaks to the unpredictable, interdisciplinary path she has carved out in the last 15 years. It’s a path that switched academic tracks without compromising Borkin’s intellectual passions, one shaped by collaborations with colleagues and students, and one with plenty of chapters left to write.

Computer science, then astrophysics

“I was extremely fortunate to have early exposure to computer science, as my father has a PhD in the subject,” Borkin recalled. “So I grew up with a personal computer, reading BASIC and playing games. My parents always supported my interests. However, my passion for so many sciences—from geology to archaeology to astrophysics—captured my attention.”

Astrophysics took the wheel for a while, forming the basis of Borkin’s undergraduate studies.

However, during a coffee chat aimed at finding her an astronomy project, her undergraduate advisor mentioned a talk she’d given about challenges in astronomy data analysis and visualization. A radiologist in the audience at the event bounded up and started chatting about the similarities between medical and astronomical data analysis.

An immediately intrigued Borkin, to her advisor’s confusion, zeroed in on the idea as a potential thesis, proposing that she “learn the radiology software, the algorithms, and the image processing, and see if we can use this to better analyze and visualize our astronomy data.”

“At the time it was pretty risky, because it could have failed,” Borkin explained. “But thank goodness it didn’t. I was hooked.” She recounted that pivotal change in direction: “I withdrew my astronomy grad school applications and took a couple of years between college and grad school to be a research assistant and figure out, ‘What is this thing called data visualization?’”

Around this time, her research advisor sent her to her first IEEE VIS conference, where she was thrilled by the scope and breadth of the field and its community. Fifteen years later, her contributions have helped shape the field.

Data visualization research and education

Most of Borkin’s research, and the impetus for the IEEE VGTC award, focuses on visualization evaluation methodologies and evaluation studies at the intersection of cognition, perception, vision, and data visualization.

“One area of work where I’ve made a significant contribution is better understanding how people see, interpret, and remember visualizations,” she explained. “Two of my most impactful papers were about what makes a visualization memorable, which has implications for how we learn and communicate. It’s really important for journalism, education, and scientific communication.”

Borkin recommends that visualizations be unique, clean, easy to read, and well-titled, with labeled axes and (preferably visual) annotations to highlight key takeaways.

“This is what I teach my DS 4200 undergrads,” Borkin said, referring to her “Information Presentation and Visualization” course. “These were loose qualitative guidelines, but the work I did with my collaborators pinned them down with actual data, confirmed them, and explained why.”

Mislove added that Borkin’s research and philosophy made her an integral part of Khoury College, noting that “her data visualization class is one of the most popular in the college, and her service-learning approach is both innovative and well-aligned with the college and university’s experiential learning mission.”

That experiential component is present in all aspects of Borkin’s career. It manifests in the classroom, where her service-learning-based curriculum helps students learn more effectively, impresses on them the value of their work on their communities, and inspires Borkin’s own research.

It also manifests in the research itself, where she develops theory, crafts software and tools, and places them in the hands of practitioners. The award—which was also separately awarded to the University of Edinburgh’s Benjamin Bach—marks the reach and significance of that work.

“Right out of the blue, I get this email: ‘Congratulations!’” Borkin said. “I’m really shocked and surprised and honored.”

Impact on software, astronomy, and medicine

Fifteen years after the decisive coffee chat with her advisor, nothing and everything has changed for Michelle Borkin.

Her research is a unique blend of data visualization, software, astronomy, and medicine. She’s working to improve clinical decision support systems for hospitals while simultaneously serving as visualization lead for Glue, a versatile data analytics tool employed in astronomy by the James Webb Telescope, which will replace the vaunted Hubble.

READ: Novel framework for implementing design studies wins best paper at CHI 2020

Some of Borkin’s graduate students came to work with her because of her research profile. Others, she says, enlighten her and challenge her to consider new research areas. Laura South made her think more about accessibility. Uzma Haque Syeda pushed her to think about visualization for social good.

“The research I’m getting this award for is near and dear to my heart, and it is the core of the work I do now with my graduate students,” Borkin noted. “Now that I’m lucky to be an assistant professor with a growing lab and wonderful, brilliant graduate students, I’ve been able to expand it.”

READ: Epilepsy can make routine internet use life-threatening. Three Khoury researchers want to change that.

After winning the most prestigious award her field offers to young researchers, Borkin reflected on her gratitude for the academic home she’s found at Khoury College, one based on the interdisciplinary philosophy that has infused her career.

“I’ve flourished in an environment that supported interdisciplinary research,” Borkin said. “It accepted me coming in with a different background, embraced my diversity, and supported me as I grew into a professor.”

To write good code, every variable needs to be well defined. To make the written code written do positive, impactful work in the wider world, everything else—from goals to the strategies used to achieve them—needs to be well defined too. At least that’s what Don Fallis, a professor of philosophy and computer science at the Khoury College of Computer Sciences, believes.

Professor Fallis works at the intersection of computer science and philosophy, focusing specifically on adversarial epistemology. Adversarial epistemology, a term Fallis coined himself, refers to the study of how we acquire knowledge (epistemology) when someone is trying to stop us (adversarial). Much of his work centers around creating and refining definitions, be that figuring out exactly what it means to tell a lie or considering the ways that defining your epistemological goals changes what tools serve you best.

Taking an interdisciplinary, philosophical approach to the questions of deception posed by modern technology allows Fallis to grasp the wider landscape of the field and see connections he might miss working as a programmer or a software developer.

Why should computer science students study philosophy?

Although Fallis created the term adversarial epistemology himself, the concept is nothing new. People have been lying to and deceiving each other for thousands of years. Modern technology opens up new tools and techniques to make practicing deception easier, from deepfakes to fake news to AI. As such, it’s unsurprising that Fallis finds a lot of classical philosophy still relevant today, and he provides a foundation on it in his classes.

“I’m always finding quotes in David Hume and Descartes that I say, ‘Look, it sounds like he’s talking about Twitter there!” said Fallis.

Fallis’s primary course, “Knowledge in a Digital World,” is cross-listed in philosophy and computer science. Fallis believes the best epistemology is inherently goal-oriented, so he starts the class by trying to figure out exactly what students mean when they say they want to acquire knowledge.

“Does it mean I want to have a lot of true beliefs? Does it mean that I want to have very few false beliefs? Does it mean that I want my beliefs to be justified? Does it mean that I want a lot of people that have these things? Does it mean that I want to have these things very quickly?” explained Fallis. He elaborated that, while any of these goals would qualify as acquiring knowledge, the tools and methods used in each scenario look very different. “Getting clear on your goals is going to put you in a better position to decide what sorts of particular technologies and policies are going to be the best way to achieve those things.”

Once students have considered their epistemological goals, Fallis has his students each pick a specific technology—anything from a specific subreddit to the GPS app on their phone. They spend the course learning to evaluate its epistemic costs and benefits using a range of philosophical frameworks. He intends for students to leave the class understanding not just their own technology, but how to evaluate future technologies in order to engage with them more critically and avoid the deceptions they could create. Fallis believes that learning to think in this way is particularly useful for computer science students, who will someday be designing technologies of their own to help people acquire knowledge.

“Computer science students are basically in the business of what you might want to refer to as information intermediaries. They’re helping people acquire information, if they’re a technologist or a librarian or whatnot. If they’re a cybersecurity expert or a software developer of some other sort, then they’re developing the technology that various people in the world are using to communicate and share information,” explained Fallis.

He added, “Having this theoretical understanding of how people acquire knowledge, how technology can affect how people acquire knowledge—that can all be useful, especially if you’re interested in being an ethically enlightened information intermediary.”

Using philosophy to spot connections

Much of Professor Fallis’s research work is focused on finding or creating definitions. He’s written papers in the past attempting to define lying and disinformation; he’s now working on a taxonomy of the different kinds of deception that he hopes will be useful in the world of cybersecurity.

Fallis has found that taking the time to step back and create exact definitions and categories allows him to see powerful connections between different kinds of deception that he would have otherwise missed. He gave the example of a browser that automatically clicks on all the ads in the background of a page.

“Say you’re shopping for shoes—you click on the shoe [ad] and then, the next day, the next hour you start getting ads on your feed saying, ‘here’s some shoes you might like’,” said Fallis, describing an interaction that users are accustomed to. But such a browser adds a, well, sneakier interaction that is far less familiar to users. He continued: “What this browser does is it hides what you’re up to in this mass of information, so the person tracking you can’t tell that you’re [shopping for shoes].”

The second interaction is deliberately misleading, among a set of what he called “techniques for preventing somebody from acquiring a particular piece of knowledge that you don’t want them to acquire.” According to Fallis, it would be insufficient to approach this as a one-off example of deception. “That counts as what counterintelligence agents call Dazzling,” he explained, adding that “there’s these other ways you can Dazzle that you might not even have thought of, but they have the same structure so you should watch out for that.”

Once a user knows what a technique looks like, they’re empowered to recognize and combat it when it’s being used against them, and additionally equipped to imagine scenarios—like the browser—where it could be used to protect you instead.

Unlike many more traditional computer scientists, Fallis doesn’t have a straightforward checklist of how to stay safe online — change your passwords regularly, set up two-factor authentication, don’t click on suspicious links, and so on. (Although he does think those are all good ideas.) Instead, he believes that the best way to protect yourself is to understand your adversaries and what they can do.

“[My pitch is basically] if you study philosophy with me, you’ll have a better understanding of the world,” said Fallis, “and the better that understanding it is, the better position you’ll be in to deal with these different phenomena.”

---