— Late January —

Subject: Fw: Your submission J410934
To: kaxiros@naic.edu
From: nicot@naic.edu

Well, I guess we’re putting off the new project…

Subject: Your submission J410934
From: editor@astrochem-a.springer.com
To: nicot@naic.edu

Dear Mr. Tenelli,

We are pleased to inform you that your article, “Chemistry of Saturn’s rings from Bayesian analysis of radio telescopy”, has been Accepted Pending Major Revisions.

Saturn, in a radio image.

“It’s not that bad, Nico — Reviewer 1 only thinks that the prior has too much methane, given how surprising and unexplained the Cassini results were. How long will it take to rerun with a new prior?”

“It should only take about a week.”

— Early February —

“Dr. K, the results are… very different with the tweaked prior. Much, much less methane, and a spike in CO2 — it’s not a small change.”

“You know what they say in showbiz, Nico: give the people what they want.”

— Early April —

Subject: Fw: Your submission J410934
To: kaxiros@naic.edu
From: nicot@naic.edu

Damn it! Changed my ass…

Subject: Your submission J410934
From: editor@astrochem-a.springer.com
To: nicot@naic.edu

Dear Mr. Tenelli,

We are pleased to inform you that after review the status of your article, “Chemistry of Saturn’s rings from Bayesian analysis of radio telescopy”, has been changed to Accepted Pending Major Revisions.

Subject: Re: Fw: Your submission J410934
To: nicot@naic.edu
From: kaxiros@naic.edu

What’s their problem now?
— Dr. K

Subject: Re: Fw: Your submission J410934
To: kaxiros@naic.edu
From: nicot@naic.edu

Reviewer 2:
“The startling lack of methane in the results lacks credibility; the chosen prior dramatically undercuts the direct experimental evidence from Cassini and so calls into question the meaningfulness of the final concentrations. A more typical prior would give me much greater confidence in the conclusions.”

“Oh, Nico. My poor little Sribatsa. You’ve put Reviewer 1 on a methane-less throne, Reviewer 2 on a methane-full one, and now they’re going to throw your mattress in the river.”

“Dr. K, you’re being even more cryptic than usual.”

“Have you ever heard the story of the Evil Eye of Sani? No?”

Dr. Kaxiros cleared her throat and began:

“Once upon a time, Sani — that’s Saturn — who was the god of bad luck” — Dr. Kaxiros wiggled her eyebrows — “had a fight with Lakshmi, the goddess of good luck.”

“Well, the gods took up the two sides of the fight in equal numbers, so Sani and Lakshmi sought judgement from the righteous and wise Sribatsa. Sribatsa was rightfully leery of angering either powerful god, and so did his damnedest to avoid the whole question. Unfortunately for him, they insisted, and he eventually gave in and invited them to his house for a judgment.”

“In preparation, he had two beautiful seats made, one of silver and one of gold. When the gods came, he invited Lakshmi to sit on the gold one and Sani on the silver.”

“Needless to say, Sani was, well, pissed. And as a result, Sribatsa had to run away, his mattress got lost in a river, and then he got separated from his wife who he thought was dead.”

There was an expectant pause. Dr. Kaxiros settled back into her chair:

“The end.”

Nico raised an eyebrow.

“Usually your stories are a lot more poetic and helpful, Dr. K.”

“Oh, I know.”

“So what do I do?”

“Well, you have a choice: push ahead and keep resubmitting till we get lazy reviewers who let it through, or we acknowledge that the difference means our results are meaningless, boring, or both, and so go back to the drawing board.”

Nico sighed.

“Remember what Feynmann said about fooling yourself, Nico. I know you’ll make the right choice.”

“Yeah. I know,” Nico grumbled.

Dr. Kaxiros grinned. “Besides, you’re not a good enough liar to fool yourself anyway.”

Background: Many scientific processes are very sensitive to initial conditions: seemingly small changes in assumptions can have enormous effects on results. In popular culture, this is sometimes known as the Butterfly effect.

A common source of such assumptions are priors for Bayesian statistical methods, which encode what you already know about something and how confident you are about that information. Such models are everywhere in science.

A great example of this was presented recently by one of my research mentors, Dr. Sean Crowell, the staff scientist of NASA’s GeoCarb mission here at OU. “Sensitivity of OCO-2 and In Situ Inversions to Transport, Prior Mean, and Prior Uncertainty: Results from the OCO-2 Model Inter-comparison Project and Beyond” demonstrated that prior choice and prior uncertainty had dramatic effects on the estimations of CO2 emissions from satellite data for a wide variety of models. In some models, indeed, he found that only strict priors were forcing the models to produce sensible data, and that when these constraints were relaxed, the models’ behavior was entirely nonsensical, calling their broader results into question.

Critically, this sensitive dependence on initial conditions applies also to the models themselves, both in their forms and parameters. Basically, “you can be arbitrarily close to the correct equations, but still not be close to the correct solutions” — Erica Thompson and Leonard Smith in this great introduction to the so-called “Hawkmoth Effect.” A great discussion of these issues in climate modeling is presented by Judith Curry, a remarkable scientist who has served on the guiding committees for a number of the major U.S. government funding bodies.

For those of a more mathematical bent, bifurcations — dramatic changes in model behaviour resulting from tiny perturbations of parameters — are a fascinating branch of the theory of dynamical systems. The Wikipedia article isn’t bad, and here’s a beautiful video of the bifurcation diagram of the logistic map, just for kicks.

Disclaimer: In case you didn’t read the intro: I am not an astrophysicist, and most of this astrophysics is made up. These stories are allegories for general classes of failings and issues of rigor in science. Cassini is a real NASA mission, however, and data from it did recently suggest some very surprising results about the chemistry of Saturn’s rings.

The Moral: You cannot assume that small changes to assumptions will have small results, nor, as a result, that “reasonable” assumptions are always good enough.

Image Source: National Radio Astronomy Observatory, via the NSF. The image shows Saturn, as “seen” by the Very Large Array (VLA) radio telescope.

Story Source: The Bengali story, “The Evil Eye of Sani,” with text from Folk-Tales of Bengal, via the course UnTextbook. Though in the original a great deal of misfortune does befall Sribatsa and his wife, in the end they do pull through by perseverance and faith, and are restored to fortune and prosperity.