Blind Analysis in High-Stakes Survey Science: When, Why, and How?

March 13-15, 2017 @ KIPAC/SLAC


Blind analyses are an effective way to reduce or eliminate e xperimenter's bias, the unintended biasing of a result by the scientists making a measurement. While blind analysis methods have become common in certain sub­fields of physics and astronomy, they are not yet widespread in the community of scientists preparing for LSST data and future CMB surveys. This workshop will bring together members of LSST science collaborations (DESC, transients, statistics) and CMB collaborations with interdisciplinary blinding experts, and blinding critics.


We thank the LSST Corporation's Enabling Science program and KIPAC workshop program for their support of this workshop.

Registration

Registration is now closed. Everyone is welcome to attend the plenary talks. If you would like to participate in the round table discussions, please contact the Organizing Committee .
Thanks to workshop support from KIPAC, there will be no registration fee. Thanks to the LSST Corporations's Enabling Science program, we have limited tavel support for students and postdocs, please email Elisabeth Krause to inquire about travel support.

SOC and LOC

  • Steve Allen - KIPAC
  • Pat Burchat - KIPAC
  • Elisabeth Krause - KIPAC
  • Ashish Mahabal - Caltech
  • Adam Mantz - KIPAC
  • Maria Elena Monzani - KIPAC
  • Phil Marshall - KIPAC
  • Aaron Roodman - KIPAC
  • Risa Wechsler - KIPAC

Invited Speakers

  • Charles Lawrence - JPL
  • Ben Loer - PNNL
  • Robert MacCoun - Stanford
  • Aaron Manalaysay - Davis
  • Bill Molzon - Irvine
  • Holger Müller - Berkeley
  • Jessie Muir - Michigan
  • Saul Perlmutter - Berkeley
  • Caroline Simard - Stanford
  • Massimo Viola - Leiden
  • Alan Weinstein - Caltech
  • Nathan Whitehorn - Berkeley
  • Bonnie Zhang - ANU

Schedule

Meeting location: Kavli Auditorium (1st floor), Kavli Building, SLAC

Plenary talks will be live-streamed at https://bluejeans.com/777333541/.
We are collaboratively taking live notes in this document.

Talks

Panelists

Charles Lawrence is Chief Scientist for Astronomy and Physics at the Jet Propulsion Laboratory in Pasadena. He was twice awarded the NASA Outstanding Leadership Medal -- in 2010, for leadership of the US Planck science and data analysis effort from concept through spacecraft launch and commissioning; and in 2004, for scientific and technical leadership related to the Spitzer Space Telescope. He is the chair of the Planck Editorial Board, which is responsible for the scientific and technical papers from Planck, an extremely high-stakes cosmological survey, and the lead for the US Planck Project.

Bill Molzon is Professor of Physics and Astronomy at UC Irvine. Since the mid-1980s he has worked on extremely precise experiments to test the conservation of additive quantum numbers associated with electron- and muon-type leptons. In a series of experiments that he co-led at BNL, he and collaborators placed an upper limit on the probability of decay of neutral kaons to a muon and an electron of ~5x10-12. He now works at PSI on the MEG search for anti-muons decaying to a positron plus photon, which has already set an upper limit of ~5x10-13. He also participates in a collaboration building the mu2e experiment to search for muons converting to electrons in muonic atoms, which should detect one event if the conversion probability is 3x10-17. He has incorporated blind analyses in all his rare search experiments.

Saul Perlmutter is Professor of Physics at UC Berkeley, co-Director of the Berkeley Center for Cosmological Physics, and Director of the Berkeley Institute for Data Science. He led one of two teams that simultaneously discovered the accelerating expansion of the universe using Type Ia supernovae, and shared the 2011 Nobel Prize in Physics for the discovery. He has incorporated blind analysis techniques in his team’s supernova research, and has collaborated across fields to understand the role blind analyses can play in minimizing confirmation bias.

Nathan Whitehorn is a member of the Berkeley Cosmology Group and works on the South Pole Telescope and POLARBEAR experiments. He was awarded the 2014 APS Division of Astrophysics “Young Star” award for his contributions to the IceCube experiment while a graduate student and postdoc at U of Wisconsin-Madison. His thesis research resulted in an upper limit on energetic neutrinos from gamma-ray bursts that was significantly lower than predicted rates (Nature 2012). He was also one of the lead researchers of an analysis that presented the first evidence for high-energy extraterrestrial neutrinos (Science 2013).

Round Tables

The Round Table is named after King Arthur’s fabled congregation of Knights; given the shape of the table, it has no head — everyone has an equal voice. Each Round Table will have a moderator and a scribe/reporter.
The role of the moderator is to ensure each participant is invited to voice their opinion or ask probing questions about the topic at hand. We have suggested questions for each Round Table, but the moderator should feel free to adapt or expand the questions as the discussion unfolds.
The role of the scribe is to take notes on the discussion (as well as participate) and then synthesize the main points and give an informal ~five-minute summary to the Workshop later in the schedule. The scribe and moderator may want to work together preparing the synthesis.
Each physical round table (which is actually rectangular...) will have 12 participants, including the moderator and scribe. There will be a mix of experts and non-experts discussing the topic of each Round Table. We are hoping the 'non-experts' will challenge the 'experts' on some of their underlying assumptions. Sometimes it is the 'non-expert' questions that are the most probing... or the 'non-expert' scribe who gives the most refreshing summary.

Round Table Assignments

Round Table 1: Blinding -- Motivation, perceptions and responsibilities

  • What motivates scientists or collaborations to conduct blind analyses?
  • What are the objections to blinding that scientists articulate?
  • Do individuals, collaborations, or scientific communities have a responsibility to adopt procedures that minimize experimenters' bias?

Round Table 2: Challenges and experience with blinding

  • What challenges have you experienced -- or do you anticipate experiencing -- in implementing blinding techniques?
  • Are the challenges primarily technical, or philosophical / cultural / sociological?
  • How can the two types of challenges be addressed?

Round Table 3: Solutions for implementing blinding techniques

  • Which blinding methods are best suited to different types of experiments or analyses?
  • What are the greatest implementation challenges that each type of experiment faces? Are there any perceived “show stoppers”?
  • When is cross-working group or cross-collaboration coordination needed to effectively minimize experimenters’ bias?

Round Table 4: Blind analysis methods for specific experiments or probes

One table per experiment or probe: CMB (Havasu), DM (FKB3), individual LSST probes (Toluca), LSST joint probes (Tulare).
  • What are the specific technical solutions that can be used for your experiment or probe, to minimize experimenter bias?
  • What are the specific technical challenges?
  • Are there resources or support -- from the collaboration or the community -- that would help you address these challenges?

Round Table Reports

Logistics

Locations

Getting there

The nearest airports are San Francisco International Airport (SFO) or Mineta San Jose International Airport (SJC). From the airport, transportation options include: