If you are in the world of astronomy, you probably have heard or read about the infamous letter written by senior staff of an unnamed “famous” astronomy department sent out to their grad students (though it is not difficult to find out which department sent out the letter). If you don’t know what I’m talking about, please read the “inspirational” e-mail here.

The first reactions and comments were naturally ones of outrage (see e.g. astrobetter). There were tweets and facebook posts that were shocked and indignated, mostly on the fact that we, as astronomers, should be working on the subject all of our waking hours and expressing “Thank you, sir, may I have another“. I think that the main source of the scorn was the claim that you have to work 80-100 hours per week on astronomy as a grad student if you want to have a succesful path as a researcher.

First of all, I don’t think anybody can work that long over prolonged periods of time (there’s even evidence for that). As an exception, because there were so many deadlines, I worked 65 hours last week and feel tired today. I hardly saw my family last week and am pretty unhappy about that. Now add 15 hours to that and I would feel downright atrocious.

However, I feel like that putting a number to the hours you work in a week missed the point completely. I am especially sad that the letter put this point first without any nuance or diplomacy – a great way to discourage your students. I can smugly say that even though my advisor was trying to impart similar wisdom to me about working hard, his way of doing it had a much more positive effect on me. Let me veer a bit into personal anecdote time here:

I found out in 2005 I was pregnant. The timimg is never perfect. I had just gotten the Lawrence Fellowship for Graduate Students. Ironically, while that made my measly grad student salary multiply by about 1.5 times, it carried a pretty horrendous health insurance making me basically pay for a big part of the costs during pregnancy and birth, basically negating all the gains in salary (heh!). Anyway, there I was, freshly married and quite frankly scared of telling my advisor that I was about to devote a large part of my life to a new being growing inside of me.

He couldn’t have been any more supportive. First questions were of the lines of “Are you ok?”, but then he did the smart thing, appealing to my ego: “You’re not thinking of leaving astronomy, are you? That would be a big loss! You can make a great scientist. You can make it happen”. Anyway, I left that meeting with a big sense of confidence, that yes, I could make it happen! He didn’t sugarcoat anything, he never said I could succeed by just being half as productive, but he saw the potential that it was doable.

Fast forward about 14 months. I was on my first trip since having my daughter. We were observing at Keck. Funnily, many people complain that they feel tired all the time while observing, but let me tell you, with an 8-month old at home, I felt refreshed and slept like I hadn’t in months, i.e. contiguous 6 hours. heh! Anyway, it was September of 2006 and I was supposed to be graduating less than a year from that date. I had produced just one paper (my deal with my advisor had been three) and I was hitting a roadblock with the analysis of the two big papers. For the Hubble paper, I feel like I wasn’t getting the fitting program, how to tweak the parameters, etc. I just wanted to scream at the world: “Look at the pictures! They tell all the story, what do I need to analyze here!?!?!”.

So we were driving back from a pretty unsuccessful run. We had just gotten a few hours of data out of 2 or 3 nights, I forget. I was feeling down, it was just too hard, the baby, the roadblocks, the stupid clouds. So somewhere on the road between Waimea and Kona my advisor asked me if I really wanted it, that the time and commitment I was putting in wasn’t gonna cut it. Especially not when I was doing other things than the analysis needed for the papers. “It’s all about decisions, Tanya. The person observing after you tonight went diving today. It is a nice thing to do, especially if you’re in Hawai’i. But to really want that job in astronomy, you will need to give up some nice things sometimes.”

All about decisions. That point really resounded with me. I can’t tell you that suddenly everything worked out fine. But I worked hard and by November, close to Thanksgiving I had *the* analysis, *the* main point of the Hubble paper done. There’s even a funny story about my excitement over that result. I had gotten the correlation with which my most disturbed objects were also the most reddened ones on the Thursday before Thanksgiving. I know because the next morning was a “Baby and me” class that I attended every Friday (those who know me will cringe, since they already know this story). We were asked around the circle to say what we were thankful for. I was to go first and happily blurted out that I had just found out the last piece of the puzzle of the paper and that I would be one step closer to graduating. Oh man, that was the wrong answer in that group and I only realized it slowly after every. single. mom. was thankful for having a healthy baby. Ooops!

I did not get a job for the 2007 job season (but only half a year later), that would’ve been too convenient of a story, wouldn’t it have been?. But I can tell you that 2007 was a year which I did focus on work, very much. I shooed colleagues away wanting to come in and gossip, the hours at work were too precious. It was about decisions. The fact that I did not even know what was playing on the TV during those years (I know my friends were obsessed with Lost!, heh), what was going on in the news. I may not have been the best paper reader, may not have had the best code written, it was sometimes hard going at it a bit alone (not completely alone, of course, thank you, Mark!). But the focus was there, the motivation was sparked that kept me going, I was gonna to whatever it took to get that PhD and hopefully that postdoc job!

Mind you, I didn’t suddenly work some magical fixed number of hours a week or spent my time at home with my baby on one arm and the laptop in the other. Rather, and it sounds weird to say it like that, I became more efficient. I most definitely missed some things. I couldn’t have done it without the enormous help and encouragement of my husband. But I can’t explain it better than I was really into everything astronomy during that time (as I feel like I have been this year, too). And it’s like an avalanche, it just multiplies and gets bigger.

Anyway, boy did I veer off the point. It was about this fixation on the number of hours worked, when in reality you should get away from this thinking. It’s about the focus, the flow and the results which you should measure. And thankfully the posts and blogs that have come out since have been a bit more nuanced and addressed much more eloquently what I was wanting to say. The two articles I liked I liked the most were:

a) The post by Elizabeth Lovegrove over at astrobites. Its main point is that there is no clear line where work ends and when fun begins and that we should step away from the computer once in a while. That doesn’t mean that you suddenly should seek out weird ways of doing science in the cafeteria, but rather that we are not in a factory where some fixed amount of product needs to be produced in a certain amount of time. It is science, insights don’t come at the n-th hour worked on the subject.

b) Another amazing blog on the subject was written by Lucianne Walkowicz. http://tangledfields.com/2012/10/26/free-advice/ It’s downright inspirational for any field you work in. Right in the beginning she tackles the issue about the number of hours worked with this simple sentence: “If you are counting how many hours a week you are working, you probably don’t like your job very much.” Exactly! I couldn’t have conveyed it better. I would suggest you just sit down and read the whole article, it deals with health, goals and a whole bunch of other things that I was just nodding along to.

Lastly, I do want to say that I hope if anything good comes out of this discussion is: yes, it’s about decisions, but there are a few things that you should not decide away – it is your health, be it physical or mental and your family and close friends! Now go on, keep on truckin’!

I have a long commute to work. It takes me about 90 minutes each way if I am taking public transportation, about an hour if I take the car. There are some personal reasons on why I live so far away from work, but I am neither sad or stressed about it. Those 2-3 hours I spend on the road are wonderful times in which I either get caught up on my TED Talks, Jon Stewarts and Colbert Reports or in which I listen to podcasts when I’m in the car.

One of these podcasts is Econtalk with Russ Roberts. It deals with more than the pure money economics that is “Planet Money“, but is way more scientifically rigorous than “Freakonomics“, which is more entertainment than anything else. The podcast is quite libertarian, with a healthy does of skepticism towards the government, but it isn’t preachy or even biased. Russ does a good job of moderating the issues, he’s always well prepared and I like his insights into all aspects of economics. He’s made me be a believer of Hayek (and Mieses to some extent) and appreciate the nuances of Friedman’s policies. His podcasts with Mike Munger are entertaining, but some of the most conflicted I have been when hearing a podcast.

And because econtalk is about the social sciences in general, not just about finance, a few weeks ago he had Brian Nosek on to talk about his latest paper: Scientific Utopia: II – Restructuring Incentives and Practices to Promote Truth Over Publishability. I was very impressed with this podcast not only because of the interesting topic, but because I saw many traits of this among my astronomer friends, even though we are in the hard, natural sciences, not in the social sciences. If you can spare an hour, I implore you to listen to the podcast, I think it is very important to be aware of our unconscious biases when we are doing science and finding publishable results.

The main and salient point that Nosek raises is that we as scientists try to genuinely do good science but that does not mean that we aren’t vulnerable to some of the reasoning and realizations that leverage the incentives to be successful at science. Read: we need to make sure to get published, so we have a hidden bias to pursue publishable results (journals will not publish null results). I mean, if I get data at a telescope, I’m expected to publish a paper on that data. To massage some scientific insight out of there can often be quite difficult, but the incentive to publish is there.

Russ reads aloud two of his quotes and I think they are so good that they bear repeating:

“The real problem is that that the incentives for publisheable results can often be at odds with the incentives for accurate results. This produces a conflict of interest. The conflict may increase the likelihood of design, analysis and reporting decisions that inflate the proportion of false results in the published literature.”

“Publishing is also the basis of a conflict of interest between personal interests and the objective of knowledge accumulation. The reason? Published and true are not synonmyms. To the extent that publishing itself is rewarded then it is in scientist’s personal interest to publish regardless of whether the published findings are true.”

The sentence that published and true are not synonyms is the heart of the problem and it is a pretty depressing idea, too. We like to see ourselves as the truthseekers, but personal interests can derail this. Incentives matter, certain kinds of results are valued more than others. We are subtly influenced to take the path that is most beneficial to our career, i.e. seek out results that I can publish. But does that push our analysis in a certain way?

Nosek identifies 9 tricks or “things we do” where our bias shows:

• – leverage chance by running many low powered studies than a few high powered ones.
• – Uncritically dismiss failed studies as pilots or due to methodological flaws but uncritically accept successful studies as methodologically sound
• – selectively report studies with positive results and not studies with negative results every time
• – stop data collection as soon as a reliable effect is obtained
• – continue data collection until a reliable effect is obtained
• – include multiple independent or dependent variables and report the subset that worked
• – Maintain flexibility in design and analytical models, including the attempt of a variety of data exclusion or transformation methods for to subset
• – Report a discovery as if it had been the result of an exploratory test
• – once a reliable test is obtained, do not do a direct replication; shame on them.

Now you may say, Tanya, that is all fine and dandy, but it only applies to the social sciences. Well, unfortunately no. I have seen the above over and over again in astronomy, too. When some observation does not agree with the model the scientists who proposed suddenly dismisses it, many projects are lying around in drawers, because “the discovery” didn’t pan out. But even worse, when an observation begins to even show signs of agreeing with the proposed, it gets sent to a prestigious journal, even if there isn’t critical statistical mass – hey it’s a pilot study, but it’s *really* important!

What do we do? Fix the journal / peer review system? Journals have own sets of incentives that encourage publications. They want attention and prestige, be at the forefront of innovations. So can’t do that. Should we maybe fix the university system? Can’t do that either, because here we face the same challenges. The way that universities gain prestige is the same as journals, so they don’t have the incentive to change either. Nosek suggests to rather start from the bottom up, from the practices of the single scientist to be accountable rather than from the top.

At the boundaries, we will have risk, we will be wrong a lot. That’s what science is supposed to do. But what happens is that by that design the discovery component suddenly outweighs the other side of science which is verification, which is kind of boring. Verification is just as important, but not as exciting! We value innovation more. Just last week this was my facebook status update:

“We had a big discussion at our conference dinner yesterday about science, its history and its impact. Yes, there was wine flowing :). One person in the group stated matter-of-factly “the biggest driver in science is discovery!”. Fortunately, I was not the only one who disagreed and vehemently at that. We need to get away from this thinking – science is incremental! It’s important that we assign glory to the people checking results, improving statistics, interpreting data, improving on some technique etc. as to the people who discover something “first”!” with a link to an ode to incremental research.

By the way, the person in the group is a brilliant scientist, I was rather perplexed by this attitude, but I am finding more and more, that it is prevalent among astronomy. We will face a big challenge is in trying to rebalance this. Can we find some incentives to make verification more interesting and provocative? It’s gonna be difficult. By the way, verification is not attacking the original scientist, disagreements over a subject does not mean that science is broken, that’s actually how science works! You gather some evidence here, somebody tries to disprove it there and then you converge on a result. Well, if that’s how science works, won’t it self correct? Isn’t then discovery the main driver still? Only eventually; it takes a very, very long time! That’s not ok! We can correct and clarify quicker and more efficiently.

So what are the suggestions to counter that thinking. In the podcast they delve into a nice anecdote about a study that was about people acting slower whenever they heard about old age. When the study couldn’t replicate, the original scholar just scowled: “well, you didn’t do it right!”. In the original paper they even reproduced the result, so obviously there was some methodology that was giving them those results. They are still talking about psychology, but even in astronomy, we have our way of doing things. What is a speck, some background variability to somebody is a Lyman alpha blob at a sigma of 2.5 at extremely high redshift to somebody else. Some people have great observing ability, positioning the slit exactly right, some are masters of getting that last little photon out of the data. But when is the science gotten from that photon reproducible? There is often so much subtlety and nuance! Failing to replicate also does not mean that the original result is false!

So the recipe is actually quite simple: be accountable! Describe each detail of the methods. I like the approach that people are putting their scripts now online via the VO or github (e.g. https://github.com/nhmc/H2) to really make their methods transparent and accessible. Some people provide diaries to their colleagues / collaborators on what they do (I know, I have and I find the ones of others extremely helpful as some sort of cookbook), if we could make this even more open it would be great – documenting your workflow is of real value! Every researcher does more research than he/she actually publishes. There may be diamonds in the rough there if all that data is open as opposed to the biased representation in the published literature.

This way of opening up methods also raises and could correct another thing: when you write a proposal, you already have the expectation of a confirmatory stance. So to correct for the strong expectation of a result you can present the tools you will use to analyze the data. This is why simulations of data in proposals are encouraged. When the data comes in, all you have to do is to run it through the already developed tools and just confirm or deny your hypothesis without fiddling, adjusting or even fudging the output. Register the analysis software in advance, it reduces your degrees of freedom, but makes you fairer, even if the data may not look as pretty. Better yet, have two competing camps work on an analysis script together!

Last, but not least, there is some fame and glory for shooting studies down. But only for the really famous ones. For the simple data analysis paper, we are often met hesitantly: “Why do you feel the need to question that result?”. But it actually is quite doable for high impact studies. This makes it so that one doesn’t constantly replicate things and will actually discover things, too, but it at least makes sure that the most “high impact” results are validated.

These are issues that have been of concern for years. Scientists don’t want to waste their time on things that aren’t true, so obviously they want to get at the heart of the problem. It’s actually great that people are looking critically at the research methods that scientists use. Even just knowing that we might have a skewed view in published results is valuable in it by itself. So if you made it this far, I have now made you aware of even another unconscious bias we carry around and need to account for. I leave you with a funny comic from xkcd, that a commenter on the podcast linked to – very relevant to the discussion.