What to do when you get results that don't make sense

A few different recent conversations and this blogpost on list experiments by Andrew Gelman have made me think about the nature of the file drawer problem.

Gelman quotes Brendan Nyhan

I suspect there’s a significant file drawer problem on list experiments. I have an unpublished one too! They have low power and are highly sensitive to design quirks and respondent compliance as others mentioned. Another problem we found is interpretive. They work best when the social desirability effect is unidirectional. In our case, however, we realized that there was a plausible case that some respondents were overreporting misperceptions as a form of partisan cheerleading and others were underreporting due to social desirability concerns, which could create offsetting effects.

and Lynn Vavreck:

Like the others, we got some strange results that prevented us from writing up the results. Ultimately, I think we both concluded that this was not a method we would use again in the future.

Many of the commenters on the blog said that the failure to publish on these results reflected badly on these researchers and that they should publish these quirky results to complete the scientific record. 

Both of these examples as well as many other stories I've heard make me think that the major causes of the file drawer effect in social science are not null results but inconclusive, messy and questionable results. The key problem is when you get a result from an analysis that makes you reassess some of the measurement assumptions that you were working with. For instance, a secondary correlation with a demographic variable comes out in an unexpected direction or the distribution of the responses is bunched up in 3 places on a 10 point scale.

The problem comes down to this. If I design a survey or other study to test an empirical proposition, the study is likely not to be ideally designed to test the validity of the measures involved and how the design effects are impacting them.

The results you get from a study designed to test an effect are often enough to cast doubt on validity but rarely are enough to demonstrate the lack of validity in a convincing way (i.e. that would be of publishable quality). The outcome is therefore that the paper can either be written up as a poor substantive article (i.e. the validity of the measures is in doubt or a poor methodological article (the evidence about the validity of the measures is weak either way because the study wasn't designed to be a test of the measure's validity).

One answer to this is to do more pre-testing. This can help to establish the validity of measures prior to working with them and can certainly identify the most obvious problems. However, unless the pre-test is nearly as large as the actual sample, the correlations with other variables won't be particularly clear in advance. In addition, pre-testing won't help understand design effects unless it tests different combinations.

However, what is really needed is whole studies devoted to examining design effects experimentally and establishing the measurement attributes. But until that happens for methods such as list experiments, researchers will be stuck with questionably valid results that are hard to publish as good empirical or methodological pieces.

A more radical approach would be to encourage journals of ideas that didn't quite work out. Short research articles that explain why the idea should have worked out nicely but ended up being a damp squib. These would be useful for meta-analysis of why certain techniques are problematic in practice without having the same time requirements for writing up as a full methodological piece.

Something every multi-national survey should do

This table is from the Afrobarometer website. You would think that every cross-national survey would have a page like this but I've never managed to track one down on some surveys I'll not mention, while others hide it so effectively it might as well not be there.

Your site can have as much on it as you like but before anything else the following three things should be linked prominently from the front page:

1. A table like the one above
2. Links to the original questionnaires for each country
3. Links to the raw data files, preferably as completely merged as possible

Why emotional intelligence leads to poorer job performance: a hypothesis

The Atlantic has an interesting summary of the recent literature focusing on the negative effects of emotional intelligence. Essentially, emotional intelligence not only allows for better interpersonal relations and cooperation but also a greater ability to manipulate others.

One of the most interesting examples that the article gives is that, in non-emotional work (data analysis or car repair rather than counselling or teaching), there is actually a negative correlation between emotional intelligence and job performance (see here for the review article). The Atlantic article proposes that emotional intelligence distracts people from their work in these types of jobs: people spend their time reading their colleagues rather than their spreadsheets.

I have an alternate explanation that should probably be considered. While emotional intelligence may not make you better at low emotion jobs, it probably makes you more likely to be promoted or hired (conditional on prior job performance). If this is the case, then the negative correlation is simply the result of selection into jobs on the basis of emotional intelligence (due to bosses liking the employee or good interview performance).

Essentially a person with low emotional intelligence needs to be better at their job than a person with high emotional intelligence to get hired for the same position. It certainly fits better with my anecdotal observations than people being distracted by their emotions (surely people with more emotional intelligence need to expend less energy on reading those around them).

This hypothesis is also compatible with the finding that emotional intelligence is associated with better job performance in emotional work. In emotional work, emotional intelligence is a good signal for job performance (indeed it may be better than formal indicators), so promoting someone based on it probably improves the job/employee fit.

I've not read the literature in much depth so I'd be interested to hear if this hypothesis has been tested somewhere.

Making bribe paying legal in India: how can we make it work?

Bribery is a huge problem in many countries. India, the world's largest democracy has a particular problem with this form of corruption. Many studies show that Indians routinely face bribe requests for services they are legally entitled to.

Former chief economic adviser of India and now chief economist of the World Bank, Kaushik Basu, has suggested a radical proposal. Make paying bribes legal, while maintaining the illegality of requesting bribes.

He argues that criminalizing both sides of the transaction aligns the interests of the corrupt official and bribe payer. Neither one wants to report the transaction because they would both suffer. In fact Basu goes further and suggests returning the bribe to the bribe payer in the event that they report that the bribe took place.

This final detail is important for incentivizing bribe givers to go to the trouble of reporting the bribe takers. However, as Basu notes, it does create a new set of incentives to falsely report bribes and this could just create a whole new problem of public official harassment and a court system too overloaded to actually deal with the real claims of bribery.

I think there might be a potential fix to this proposal that would get round these problems. The answer is that we don't return the bribes but we do make not reporting a bribe an illegal act. In this way we create the heavily divergent interests between bribe payer and bribe taker but don't create the perverse incentive to falsely report bribes.

The power of the "duty to report" law will depend partially on the likelihood of being caught. To increase this probability I would suggest running a small number of high publicised sting operations where well audited and video recorded officials request bribes from the public. These bribes are returned in full with a reward in the event of the person reporting the mock bribe and the person is prosecuted if they fail to report it.

The fear that a bribe requester could be a sting operation will heavily skew incentives to report any bribe requesters in fear of being prosecuted if you do not do so.

See here for the Planet Money write up
http://www.npr.org/blogs/money/2011/04/01/135011027/why-paying-bribes-should-be-legal

and here for Kaushik Basu's paper
http://finmin.nic.in/workingpaper/act_giving_bribe_legal.pdf

An idea for a more useful Google Trends

I've written a couple of articles (the second one is coming out in JEPOP some time soon but I don't have a link yet) about the limitations of using Google Trends data for social science research. The major issue is that many more search terms seem like plausible measures than actually turn out to correlate with public opinion. Search terms don't even necessarily work across different countries that speak the same language!

As a result, any use of these trends has to go through the process of matching up the data to equivalent survey data before it can be used validly.

But what if we didn't have to do all that?

One of the reasons I suspect the Google Trends sometimes don't match up as well as we would hope is that it counts searches not people. A handful of furiously searching journalists and politicos can drive the trend as much as widespread searching across the population. This means that issues may be ignored by 99% of the population but still result in a lot of Google searching.

So the graph above tracks what percentage of all searches in the United States were for the term "Syria" on different dates (these percentages are then scaled to a 1 to 100 index so we don't know the actual percentages).

This representative problem is easily solvable for Google. Simply report the trends for number of people searching for a term instead of the number of searches for a term. Google could simply offer us the option of tracking the percentage of people using Google on each date who searched for the term "Syria". Even if journalists search for Syria a thousand times, it will only count as one person.

It's not hard for Google to identify different people either. While there are some complexities to tracking an individual over time, Google has been building profiles on its users for a long time and even a measure of the number of unique IPs that searched for a term would go a long way towards solving this problem.

Having both of these settings as an option would give much greater insight into the breadth and depth of opinion on an issue.

It might even make offhand references to Google Trends as a proxy for public opinion a little more accurate.

Note: There are other reasons why Google Trends data might not match up to public opinion (see the papers) but this is certainly one major concern.

Would changing law school to two years reduce tuition fees?

Barack Obama's recent comments on reducing law school length to two years have prompted discussion about the trade off between reducing costs for students and the quality of legal education.

However, there's a fairly major point being overlooked, that reducing the course to two years is unlikely to reduce the upfront cost of legal education to students.

Let's assume that Obama is right and law school can be shrunk to two years without reducing quality. Given the huge variation in legal training in different countries, this seems reasonably plausible. We'll make a second assumption that law firms will recognize this fact at least over the medium term.

If this is the case, then the opportunity cost of taking a three year law course is a year of earnings and the interest accrued on loans over the extra year. By this logic, the two year course actually has higher expected value than the three year course and so students should be willing to pay more for a two year course than for a three year one. Because there is no regulation of tuition costs, law schools will quickly adjust prices to fit the demand for the shorter programs.

The key point is that a yearly tuition fee is best understood as a payment plan towards the whole cost of a degree, than a price for each year.

One point that is worth considering is whether fees might be reduced because of lower costs to the law schools of holding the courses. This is possible, but probably only if the market as a whole shifts in that direction so that all law schools are competing on the basis of two year costs. In a mixed market it probably makes sense for two year courses to maintain prices: 1) because they can and 2) to avoid signalling a lower quality product.

So if upfront costs are unlikely to go down, is the plan a bad idea? Not at all. The reason prices won't go down is because two year courses offer a better deal for the student. A chance to be a lawyer a year earlier and an extra year of earnings over a career.

This last point is also important for society. A student who enters the jobs market a year earlier provides an extra year of lawyer. And this difference isn't an extra year of a junior lawyer but an extra year of being an experienced lawyer at the end of the career. The lawyer becomes a lawyer "with 30 years of legal experience" a year earlier and a lawyer "with 40 years of legal experience" a year earlier.

So reducing law school length sounds like a good option, just don't expect it to reduce fees.

Automatically creating citations for your R packages within knitr

Health warning: the following post is about solving a fairly small problem when using R, latex and knitr. If you don't already use two out of three of these you might want to skip over this post!

Workflow issues are some of the most irritating issues in academia. On one extreme you can use stata and Microsoft word to analyse and write your documents. However, there are a whole host of formatting and replicability issues that go along with this setup. At the other extreme you can use an ever increasing cocktail of software to finely hone every aspect of creating a paper with the consequent problems of updates, conflicts and looking for missing parentheses in five different scripts.