What to do when you get results that don't make sense

A few different recent conversations and this blogpost on list experiments by Andrew Gelman have made me think about the nature of the file drawer problem.

Gelman quotes Brendan Nyhan

I suspect there’s a significant file drawer problem on list experiments. I have an unpublished one too! They have low power and are highly sensitive to design quirks and respondent compliance as others mentioned. Another problem we found is interpretive. They work best when the social desirability effect is unidirectional. In our case, however, we realized that there was a plausible case that some respondents were overreporting misperceptions as a form of partisan cheerleading and others were underreporting due to social desirability concerns, which could create offsetting effects.

and Lynn Vavreck:

Like the others, we got some strange results that prevented us from writing up the results. Ultimately, I think we both concluded that this was not a method we would use again in the future.

Many of the commenters on the blog said that the failure to publish on these results reflected badly on these researchers and that they should publish these quirky results to complete the scientific record. 

Both of these examples as well as many other stories I've heard make me think that the major causes of the file drawer effect in social science are not null results but inconclusive, messy and questionable results. The key problem is when you get a result from an analysis that makes you reassess some of the measurement assumptions that you were working with. For instance, a secondary correlation with a demographic variable comes out in an unexpected direction or the distribution of the responses is bunched up in 3 places on a 10 point scale.

The problem comes down to this. If I design a survey or other study to test an empirical proposition, the study is likely not to be ideally designed to test the validity of the measures involved and how the design effects are impacting them.

The results you get from a study designed to test an effect are often enough to cast doubt on validity but rarely are enough to demonstrate the lack of validity in a convincing way (i.e. that would be of publishable quality). The outcome is therefore that the paper can either be written up as a poor substantive article (i.e. the validity of the measures is in doubt or a poor methodological article (the evidence about the validity of the measures is weak either way because the study wasn't designed to be a test of the measure's validity).

One answer to this is to do more pre-testing. This can help to establish the validity of measures prior to working with them and can certainly identify the most obvious problems. However, unless the pre-test is nearly as large as the actual sample, the correlations with other variables won't be particularly clear in advance. In addition, pre-testing won't help understand design effects unless it tests different combinations.

However, what is really needed is whole studies devoted to examining design effects experimentally and establishing the measurement attributes. But until that happens for methods such as list experiments, researchers will be stuck with questionably valid results that are hard to publish as good empirical or methodological pieces.

A more radical approach would be to encourage journals of ideas that didn't quite work out. Short research articles that explain why the idea should have worked out nicely but ended up being a damp squib. These would be useful for meta-analysis of why certain techniques are problematic in practice without having the same time requirements for writing up as a full methodological piece.

Why emotional intelligence leads to poorer job performance: a hypothesis

The Atlantic has an interesting summary of the recent literature focusing on the negative effects of emotional intelligence. Essentially, emotional intelligence not only allows for better interpersonal relations and cooperation but also a greater ability to manipulate others.

One of the most interesting examples that the article gives is that, in non-emotional work (data analysis or car repair rather than counselling or teaching), there is actually a negative correlation between emotional intelligence and job performance (see here for the review article). The Atlantic article proposes that emotional intelligence distracts people from their work in these types of jobs: people spend their time reading their colleagues rather than their spreadsheets.

I have an alternate explanation that should probably be considered. While emotional intelligence may not make you better at low emotion jobs, it probably makes you more likely to be promoted or hired (conditional on prior job performance). If this is the case, then the negative correlation is simply the result of selection into jobs on the basis of emotional intelligence (due to bosses liking the employee or good interview performance).

Essentially a person with low emotional intelligence needs to be better at their job than a person with high emotional intelligence to get hired for the same position. It certainly fits better with my anecdotal observations than people being distracted by their emotions (surely people with more emotional intelligence need to expend less energy on reading those around them).

This hypothesis is also compatible with the finding that emotional intelligence is associated with better job performance in emotional work. In emotional work, emotional intelligence is a good signal for job performance (indeed it may be better than formal indicators), so promoting someone based on it probably improves the job/employee fit.

I've not read the literature in much depth so I'd be interested to hear if this hypothesis has been tested somewhere.

Making bribe paying legal in India: how can we make it work?

Bribery is a huge problem in many countries. India, the world's largest democracy has a particular problem with this form of corruption. Many studies show that Indians routinely face bribe requests for services they are legally entitled to.

Former chief economic adviser of India and now chief economist of the World Bank, Kaushik Basu, has suggested a radical proposal. Make paying bribes legal, while maintaining the illegality of requesting bribes.

He argues that criminalizing both sides of the transaction aligns the interests of the corrupt official and bribe payer. Neither one wants to report the transaction because they would both suffer. In fact Basu goes further and suggests returning the bribe to the bribe payer in the event that they report that the bribe took place.

This final detail is important for incentivizing bribe givers to go to the trouble of reporting the bribe takers. However, as Basu notes, it does create a new set of incentives to falsely report bribes and this could just create a whole new problem of public official harassment and a court system too overloaded to actually deal with the real claims of bribery.

I think there might be a potential fix to this proposal that would get round these problems. The answer is that we don't return the bribes but we do make not reporting a bribe an illegal act. In this way we create the heavily divergent interests between bribe payer and bribe taker but don't create the perverse incentive to falsely report bribes.

The power of the "duty to report" law will depend partially on the likelihood of being caught. To increase this probability I would suggest running a small number of high publicised sting operations where well audited and video recorded officials request bribes from the public. These bribes are returned in full with a reward in the event of the person reporting the mock bribe and the person is prosecuted if they fail to report it.

The fear that a bribe requester could be a sting operation will heavily skew incentives to report any bribe requesters in fear of being prosecuted if you do not do so.

See here for the Planet Money write up
http://www.npr.org/blogs/money/2011/04/01/135011027/why-paying-bribes-should-be-legal

and here for Kaushik Basu's paper
http://finmin.nic.in/workingpaper/act_giving_bribe_legal.pdf

Is the US airbase in Kyrgyzstan completely empty?

Brief background first. Kyrgyzstan is currently the only country in the world to host both a Russian Airbase and a US Airbase: the Manas Transit center.  Manas is the main air route that the US uses to fly personnel and equipment in and out of Afghanistan. The Kyrgyzstan government has threatened to close the base several times, sometimes coinciding with newly generous aid packages from Russia. At present the US government has been given until June 2013 to close the base.

But putting aside the exciting story of Russian and US meddling in Central Asia, I have a much more mundane question. How come the US Department of Defense generally reports that it only has between 5 and 20 personnel present in Kyrgyzstan in its official statistics for each month between 2001 and the present? I'm not a military expert but I would have thought that it took more than  20 people to run an airbase that is reported to move up to 50,000 US and coalition troops to and from Afghanistan each month. Incidentally an ABC news report mentions that Manas is staffed by 1,200 people.

I came across this issue when checking the Heritage Foundation's compilation of the American troop deployment data.  Their data does seem to match the DoD's figures but I'd like to know whether these numbers have any validity or if the Kyrgyzstan data is a single oddity in an otherwise high quality dataset.

I'm interested in hearing if anyone has an explanation. Is there a counting technicality that makes the staff on base not technically active duty? Is the US downplaying its presence in Kyrgyzstan in its military statistics to try and avoid creating local tensions? Is the Manas airbase staffed by mercenaries or ghosts?

Sidenote: this was the first figure in the dataset that I checked because I've been following the disputes over the airbase for a while and wanted to see how large a force was deployed there.

The heritage data has also been used in a couple of academic articles so this is an issue that affects things other than my own research.