Jan 15, 2015 · 21m · mad
Hanna Wallach, Microsoft Research // Data Driven #33 // Jan 2015 (Hosted by FirstMark Capital)
0:00 / 0:00
▶ Watch on YouTube →
gold bands on the timeline = statements, start to end. Hover to read, click to jump. CC turns on captions
In this presentation at Data Driven NYC, researcher Hanna Wallach examines computational social science, emphasizing how interdisciplinary collaboration, question-driven research, and rigorous model evaluation are essential for addressing algorithmic bias and ethical challenges in big data.
How this conversation actually went
Every chapter scored 0–10 on four independent dynamics. Hover any point for the reasoning behind the score. Matt holds 1% of the talking time here. How this is scored →
Matt as informed peer 0.0 Guest teaching 0.0 Guest disagreement 1.2 Matt pushing back 0.0
speaking balance: gold is Matt, purple is the guest (3 minute bins)
Sharpest disagreement ▶ 8:50 Critique of data-first and tool-first approaches
Wallach offers her strongest intellectual pushback against mainstream computer science norms, criticizing researchers who create tools looking for nails or pick convenience datasets without framing meaningful social questions.
Hardest push from Matt ▶ 19:27 Host time management wrap-upBecause the presentation was a uninterrupted monologue, the only host intervention was Matt Turck stepping in at the end to limit Q&A to one quick question due to time constraints.
Biggest teaching moment ▶ 16:50 Refuting human intuition in social analysisWallach explicitly refutes the idea that computer scientists can rely on intuition about human behavior, educating the audience on unconscious implicit bias and the need for rigorous social science methods.
Matt holds his own ▶ 19:27 Host post-talk feedbackMatt Turck reassumes control of the session, contextualizing the presentation as a unique, highly substantial talk compared to typical data science pitches.
the scores for every segment, with the reasoning behind each
| Chapter | Topic | Matt as informed peer | Guest teaching | Guest disagreement | Matt pushing back | Why |
|---|---|---|---|---|---|---|
| Defining Big Data and Social Granularity | 0 | 0 | 1 | 0 | Hanna Wallach delivers a solo keynote presentation defining big data and highlighting why granular social data makes people uncomfortable. As this is a monologue presentation without host participation, host expertise and pushback are non-existent. | |
| Ethics, Bias, and Interdisciplinary Collaboration in CSS | 0 | 0 | 1 | 0 | Wallach advocates for meaningful interdisciplinary collaboration between computer scientists and social scientists to tackle ethics, bias, and privacy. The segment is a solo lecture monologue with no host interaction. | |
| Computational Challenges in Analyzing Heterogeneous Data | 0 | 0 | 2 | 0 | Wallach gently critiques data-first and method-first research approaches in computer science, urging research to start with social questions rather than convenience datasets. Host scores remain zero due to the monologue format. | |
| Machine Learning Models, Error Analysis, and Uncertainty | 0 | 0 | 1 | 0 | Wallach explains machine learning model error analysis, emphasizing the necessity of representing uncertainty when modeling social data. The host does not interrupt or participate during this presentation section. | |
| Drawing Fair Findings and Improving Scientific Communication | 0 | 0 | 1 | 0 | Wallach discusses implicit bias, relying on methodology over human intuition, and public understanding of data science to conclude her talk. The segment contains no host dialogue. |
Statements from this episode (13)
Wallach: Big data sets differ from physics data by documenting human behavior
“Unlike the data sets arising in physics, the data sets that typically fall under the big data umbrella are about people, Their attributes, their actions, and their interactions. That is to say these are social data sets that document people's lives, behaviors …”
Wallach: Big data's defining feature is granularity, not size
“The issue is granularity. In other words, not only do these data sets document social phenomena, they do so at the granularity of individual people and their second-to-second activities.”
Wallach: Research shows diverse teams best facilitate rapid breakthrough innovation
“In fact, there's even substantial research in social psychology and sociology indicating that rapid breakthrough innovations are best facilitated by bringing together people with really diverse backgrounds.”
Wallach: Tech and government players must hire social scientists to address bias
“So as a result, if technology companies and government organizations, the biggest players in the big data game, Are going to take issues like bias and fairness and inclusion seriously. They need to hire social scientists, the people with the best training and …”
Wallach: Addressing algorithmic bias requires focusing on big data's granular nature
“When it comes to addressing issues like bias, fairness, and inclusion, perhaps we should instead be focusing our attention on the granular nature of big data.”
Wallach: The big data game will be won by asking interesting questions
“Ultimately, the big data game will be won by those who know how to ask and answer interesting questions.”
Wallach: Data-first research approaches amplify issues with algorithmic bias and fairness
“Although this is kind of conducive to fast-paced work, these data-first or method-first approaches can actually amplify issues related to bias, fairness, and inclusion of minorities.”
Wallach: Convenience datasets bias analytical models toward demographic majorities
“But the problem with these convenience data sets is that they typically reflect only a particular segment of society. For example, young people with smartphones. And so, as a result, many of the methods developed to analyze these data sets end up prioritizing …”
Wallach: ML fairness research overwhelmingly focuses on predictive over exploratory models
“So much of the existing work on fairness and transparency in machine learning, not that there's very much of it, focuses on predictive models rather than models for exploratory or explanatory analyses”
Wallach: Bias in exploratory models distorts subsequent predictive models
“As a result, the models used to perform these previous analyses and any kind of bias or unfairness in them will necessarily influence the resultant findings, and hence the representations that we then choose to use in our predictive models.”
Wallach: Predictive social data models must maintain and report uncertainty
“When building and using predictive models for social data, whether resultant decisions can actually affect real-world people, representing, maintaining, and reporting uncertainty, along with any subsequent decisions, should be standard practice.”
Hanna Wallach: Most machine learning methods will be deployed without creator involvement
“It's really likely that most machine learning methods or data science methods will at some point be used by some people, end users, without the involvement of those people who created them.”
Wallach: Investigating correct model predictions helps contextualize how models treat certainty
“I don't know if there are necessarily any sort of general purpose, like this is going to fix everything kind of solutions, but I would say yes, digging into why your model is making certain predictions, even when those predictions are correct, can kind of help…”