What to do when you get results that don't make sense

A few different recent conversations and this blogpost on list experiments by Andrew Gelman have made me think about the nature of the file drawer problem.

Gelman quotes Brendan Nyhan

I suspect there’s a significant file drawer problem on list experiments. I have an unpublished one too! They have low power and are highly sensitive to design quirks and respondent compliance as others mentioned. Another problem we found is interpretive. They work best when the social desirability effect is unidirectional. In our case, however, we realized that there was a plausible case that some respondents were overreporting misperceptions as a form of partisan cheerleading and others were underreporting due to social desirability concerns, which could create offsetting effects.

and Lynn Vavreck:

Like the others, we got some strange results that prevented us from writing up the results. Ultimately, I think we both concluded that this was not a method we would use again in the future.

Many of the commenters on the blog said that the failure to publish on these results reflected badly on these researchers and that they should publish these quirky results to complete the scientific record. 

Both of these examples as well as many other stories I've heard make me think that the major causes of the file drawer effect in social science are not null results but inconclusive, messy and questionable results. The key problem is when you get a result from an analysis that makes you reassess some of the measurement assumptions that you were working with. For instance, a secondary correlation with a demographic variable comes out in an unexpected direction or the distribution of the responses is bunched up in 3 places on a 10 point scale.

The problem comes down to this. If I design a survey or other study to test an empirical proposition, the study is likely not to be ideally designed to test the validity of the measures involved and how the design effects are impacting them.

The results you get from a study designed to test an effect are often enough to cast doubt on validity but rarely are enough to demonstrate the lack of validity in a convincing way (i.e. that would be of publishable quality). The outcome is therefore that the paper can either be written up as a poor substantive article (i.e. the validity of the measures is in doubt or a poor methodological article (the evidence about the validity of the measures is weak either way because the study wasn't designed to be a test of the measure's validity).

One answer to this is to do more pre-testing. This can help to establish the validity of measures prior to working with them and can certainly identify the most obvious problems. However, unless the pre-test is nearly as large as the actual sample, the correlations with other variables won't be particularly clear in advance. In addition, pre-testing won't help understand design effects unless it tests different combinations.

However, what is really needed is whole studies devoted to examining design effects experimentally and establishing the measurement attributes. But until that happens for methods such as list experiments, researchers will be stuck with questionably valid results that are hard to publish as good empirical or methodological pieces.

A more radical approach would be to encourage journals of ideas that didn't quite work out. Short research articles that explain why the idea should have worked out nicely but ended up being a damp squib. These would be useful for meta-analysis of why certain techniques are problematic in practice without having the same time requirements for writing up as a full methodological piece.

Making bribe paying legal in India: how can we make it work?

Bribery is a huge problem in many countries. India, the world's largest democracy has a particular problem with this form of corruption. Many studies show that Indians routinely face bribe requests for services they are legally entitled to.

Former chief economic adviser of India and now chief economist of the World Bank, Kaushik Basu, has suggested a radical proposal. Make paying bribes legal, while maintaining the illegality of requesting bribes.

He argues that criminalizing both sides of the transaction aligns the interests of the corrupt official and bribe payer. Neither one wants to report the transaction because they would both suffer. In fact Basu goes further and suggests returning the bribe to the bribe payer in the event that they report that the bribe took place.

This final detail is important for incentivizing bribe givers to go to the trouble of reporting the bribe takers. However, as Basu notes, it does create a new set of incentives to falsely report bribes and this could just create a whole new problem of public official harassment and a court system too overloaded to actually deal with the real claims of bribery.

I think there might be a potential fix to this proposal that would get round these problems. The answer is that we don't return the bribes but we do make not reporting a bribe an illegal act. In this way we create the heavily divergent interests between bribe payer and bribe taker but don't create the perverse incentive to falsely report bribes.

The power of the "duty to report" law will depend partially on the likelihood of being caught. To increase this probability I would suggest running a small number of high publicised sting operations where well audited and video recorded officials request bribes from the public. These bribes are returned in full with a reward in the event of the person reporting the mock bribe and the person is prosecuted if they fail to report it.

The fear that a bribe requester could be a sting operation will heavily skew incentives to report any bribe requesters in fear of being prosecuted if you do not do so.

See here for the Planet Money write up
http://www.npr.org/blogs/money/2011/04/01/135011027/why-paying-bribes-should-be-legal

and here for Kaushik Basu's paper
http://finmin.nic.in/workingpaper/act_giving_bribe_legal.pdf

An idea for a more useful Google Trends

I've written a couple of articles (the second one is coming out in JEPOP some time soon but I don't have a link yet) about the limitations of using Google Trends data for social science research. The major issue is that many more search terms seem like plausible measures than actually turn out to correlate with public opinion. Search terms don't even necessarily work across different countries that speak the same language!

As a result, any use of these trends has to go through the process of matching up the data to equivalent survey data before it can be used validly.

But what if we didn't have to do all that?

One of the reasons I suspect the Google Trends sometimes don't match up as well as we would hope is that it counts searches not people. A handful of furiously searching journalists and politicos can drive the trend as much as widespread searching across the population. This means that issues may be ignored by 99% of the population but still result in a lot of Google searching.

So the graph above tracks what percentage of all searches in the United States were for the term "Syria" on different dates (these percentages are then scaled to a 1 to 100 index so we don't know the actual percentages).

This representative problem is easily solvable for Google. Simply report the trends for number of people searching for a term instead of the number of searches for a term. Google could simply offer us the option of tracking the percentage of people using Google on each date who searched for the term "Syria". Even if journalists search for Syria a thousand times, it will only count as one person.

It's not hard for Google to identify different people either. While there are some complexities to tracking an individual over time, Google has been building profiles on its users for a long time and even a measure of the number of unique IPs that searched for a term would go a long way towards solving this problem.

Having both of these settings as an option would give much greater insight into the breadth and depth of opinion on an issue.

It might even make offhand references to Google Trends as a proxy for public opinion a little more accurate.

Note: There are other reasons why Google Trends data might not match up to public opinion (see the papers) but this is certainly one major concern.

Would changing law school to two years reduce tuition fees?

Barack Obama's recent comments on reducing law school length to two years have prompted discussion about the trade off between reducing costs for students and the quality of legal education.

However, there's a fairly major point being overlooked, that reducing the course to two years is unlikely to reduce the upfront cost of legal education to students.

Let's assume that Obama is right and law school can be shrunk to two years without reducing quality. Given the huge variation in legal training in different countries, this seems reasonably plausible. We'll make a second assumption that law firms will recognize this fact at least over the medium term.

If this is the case, then the opportunity cost of taking a three year law course is a year of earnings and the interest accrued on loans over the extra year. By this logic, the two year course actually has higher expected value than the three year course and so students should be willing to pay more for a two year course than for a three year one. Because there is no regulation of tuition costs, law schools will quickly adjust prices to fit the demand for the shorter programs.

The key point is that a yearly tuition fee is best understood as a payment plan towards the whole cost of a degree, than a price for each year.

One point that is worth considering is whether fees might be reduced because of lower costs to the law schools of holding the courses. This is possible, but probably only if the market as a whole shifts in that direction so that all law schools are competing on the basis of two year costs. In a mixed market it probably makes sense for two year courses to maintain prices: 1) because they can and 2) to avoid signalling a lower quality product.

So if upfront costs are unlikely to go down, is the plan a bad idea? Not at all. The reason prices won't go down is because two year courses offer a better deal for the student. A chance to be a lawyer a year earlier and an extra year of earnings over a career.

This last point is also important for society. A student who enters the jobs market a year earlier provides an extra year of lawyer. And this difference isn't an extra year of a junior lawyer but an extra year of being an experienced lawyer at the end of the career. The lawyer becomes a lawyer "with 30 years of legal experience" a year earlier and a lawyer "with 40 years of legal experience" a year earlier.

So reducing law school length sounds like a good option, just don't expect it to reduce fees.

Could the pope make everyone in the world a European citizen?

The Vatican occupies an interesting place in international law with certain functions of its government assumed by Italy but with independence in other respects. While reading around this, I noticed an oddity that might allow a pope to grant Italian citizenship (and hence European citizenship) on as many people as he wishes.

This stems from the Lateran treaty that setup the legal relationship between Italy and the Vatican. Certain employees of the Vatican are made citizens of Vatican state for the period of their employment. However, this citizenship is removed once their term ends. The Lateran treaty refers specifically to this group as follows:

"Ceasing to be subject to the sovereignty of the Holy See, the people mentioned in the preceding paragraph may, if in terms of Italian law, regardless of the circumstances of fact set forth above, are not considered to be equipped with other citizenship, will be regarded in Italy as citizens certainly Italian." (Google translated)

Essentially, former citizens of the Vatican who no longer have any other citizenship are automatically granted Italian citizenship instead of being made stateless. This opens up the tantalizing possibility of the pope hiring large numbers of temporary staff for 2 days. Then, once they have renounced their previous citizenship- firing them- making them automatic Italian citizens. This logic suggests that the Pope can essentially create as many Italian (and therefore European) citizens as he wishes.

However, what would happen next is more murky. Since Italy hasn't signed the Convention on the Reduction of Statelessness, which generally disallows the removal of citizenship  when it would leave an individual stateless, they might just pass a law revoking post-Vatican citizenships. I'm not a legal expert so I'm unsure whether this counts as breaking the Lateran Treaty. EU law might also have some sway but it generally defers to Member States on their right to determine rules around citizenship.

Given the new pope's greater focus on poverty, perhaps opening up Europe's borders will appeal as a poverty reduction measure.

Notes:
1) I certainly don't think Francis will do this and the effects of it would likely be very uncertain.
2) This is basically a "real life" version of the ploy from Elysium to get everyone in the world healthcare.

Zombie nation? (an update on a sleepless world)

Sadly this isn't a post about German trance band made famous by this piece of popular music (slightly NSFW) but a follow up to my previous post about sleep reduction and the economy. It stirred up plenty of debate here and at Marginal Revolution and EconLog so I thought I'd follow up with some amendments and replies. Thanks to everyone who linked and commented- working through a new idea like this is exactly what I hoped the blog would allow.

Is 2.5 hours realistic?

Plenty of responses were sceptical whether modafinil could reduce sleep to 2.5 hours a night regularly. This is a fair point and the figure is definitely toward the extreme end of reported experiences on the Internet and most of these were not long term users. I used the 2.5 hours as an extreme case (which it's fair to say is probably not achievable for most people with current medicine) but the more common experiences seem to be people who reduce their sleep by a few hours habitually and people who use the drugs to stay up for extended periods once in a while without suffering the drastic cognitive declines insomnia normally entails. It'd probably be possible to keep people up for this long but I'll agree with the commenters that this is likely unsafe. So current drugs such as Modafinil are probably not capable of changing a whole population over to a 2.5 hour sleep schedule, so we would probably be looking at smaller increases in the work day than the 34% discussed. Still, the mechanisms of increased productivity should still apply albeit at a smaller magnitude.

40 hour weeks are as much as we can do

Other commenters suggested that a 40 hour work week is the maximum productive number of hours people could manage. This may be true at present but most of the studies on these drugs do suggest they have a substantial impact on reducing fatigue and maintaining performance. I'd see this 40 hour limit as a contingent fact of our normal mental state rather than a cast iron rule. These drugs are doing more than keeping users awake in an undead state but actually prolonging the time that they are able to be alert.

An arguably stronger effect of this class of drugs would be increasing the productivity of those already working long hours. The current usage of these drugs seem to support this view. A study of doctors found that their performance during long shifts was significantly improved by taking modafinil. The authors conclude that:

 Our results suggest that fatigued doctors might benefit from pharmacological enhancement in situations that require efficient information processing, flexible thinking, and decision making under time pressure. However, no improvement is likely to be seen in the performance of basic procedural tasks.

Sleep is important

At the moment this is definitely true and there is convincing evidence that sleep promotes memory consolidation. What we don't seem to know (it'd be good to hear from any sleep experts who have a more complete answer) is how much sleep is required and whether these functions are affected (positively or negatively) by wakefulness drugs.

People enjoy sleep

I am genuinely not sure if this is true. I certainly enjoy dreaming (which is only a small proportion of total sleep), snoozing (that half asleep phase after you wake up) and feeling refreshed. But I'm not sure I like sleeping 8 hours simply because it meant I was asleep for 8 hours. If I could feel equally refreshed and remember just as many dreams after 3 hours I'd probably be just as satisfied. I'm not sure if the number of hours in which I was not experiencing anything is something I value in itself so much as for the effects it has on me.

I don't want to work more hours

This was probably the most commonly made point. Whether this is valid comes down to whether we care about the proportion of our time that we are at work or the total amount of work we do in a day. The American Time Use Survey in the previous post suggests that (at least on work days) people spend more of their waking hours working than all other activities combined (this includes household tasks and caring for dependents). If we do get extra wakeful time it could all go to leisure but this would be a huge increase in the proportion of our waking lives spent in leisure. This of course is no bad thing but our current tendency spend a high proportion of our time on work suggests that people do not value leisure so much as to forgo wages substantially.

The split I suggested would increase leisure time much more proportionally than it would work time. Nevertheless, it is an open question whether we fundamentally dislike working more than 8 hours a day regardless of the time we spend in leisure. It is possible that regularly working 11 hour days would be unpleasant regardless of how much leisure time was available for relaxing. There may be timetable that can allow extended breaks of several hours during the day but everyone knows lunch breaks don't feel as good as real time off.


Signalling?

Nelly wrote that:
"These drugs may make someone more productive on a daily basis, but wouldn't necessarily make someone more productive on an hourly basis. Maybe more importantly, it is quite possible that those who would choose to take these drugs would be negatively selected. They would likely be taking them to become more competitive, which would be in line with your increasing the labor supply argument."


This is an interesting point and might be true to begin with. However, if the negative selected individuals are managing to compete with those with a higher underlying skill, it might make sense for skilled people to take these drugs as well in order to maintain their position. In a jobs market where the payoff to human capital is partially positional (i.e. almost every jobs market), people with higher talent are disadvantaged by people with lower talent using drugs to catch up. If the talented can't stop the dopers from using drugs, then they may be pushed towards joining them.

This creeping economic pressure to use these drugs is still the most convincing argument against their use. But this is roughly analogous to the economic pressures to use any other technology. Workers who refuse to use any motorized transport limit their job opportunities and only a few farmers still use pre-industrial methods. That's not to say that cars and fertilizer are purely positive developments but they still spread rapidly.

These pressures mean that if we do decide that we don't want to live in a society that relies on these drugs, we may have to actively regulate them rather than leaving the decision to individuals.

The Sleepless World: One more piece of low hanging fruit?

Update: more detail on the argument and responses to it here and minor edits in this post

Tyler Cowen at Marginal Revolution is a major proponent of the Great Stagnation thesis: that new innovations are not having the same impact on productivity as those we saw in the previous 150 years. Think of iPads versus electricity and cloud computing versus the railroad. Hence, we can expect to see slowing growth in GDP per capita as future productivity gains will take much more effort to unlock. This week The Economist took up this line of thought with a thorough briefing on the subject that broadly agreed with Cowen, although with some equivocation.

Overall, I think the argument has a lot of merit but there may be at least one more piece of low hanging fruit: a vast reduction in our need for sleep.

The American Time Use survey reports that an average American work day includes 8.8 hours of work and 7.6 hours of sleep. Sleep is the second largest single use of time. However, new drugs such as Modafinil appear to vastly reduce the need for sleep without significant side effects (at least so far). Based on anecdotal reports from users, it seems that people could realistically [edit: potentially (see update)] cut their sleep requirements to as few as 2.5 hours a night without a decrease in mental acuity. That gives us another 5 hours to distribute over the day.

Workers would probably prefer to allocate the bulk of that extra time to leisure but I doubt employers will let that happen. Let's make a generous breakdown and give work an extra 3 hours and let workers spend another 2 as they wish. This increases working hours by around 34% and potentially increases leisure time by 80%. This increases the number of hours a worker spends at work from around 1800 hours a year now to about 2,400.

So would this lead to an increase in productivity? That partially depends on how we measure it. If we look at output per worker, then a 34% increase in hours worked would be a substantial boost in productivity and would surely lead to increased economic growth. This alone makes a sleepless world a classic example of low hanging fruit (although it may be closer to an increase in labour force participation than a productivity boost from innovation).

However, there are also reasons to think that output per worker hour may be improved by longer hours. Let's assume that the additional hours worked allow companies to keep the same production with a quarter fewer workers.  Why hire four workers for a task, when you can hire the three workers who are most productive to do more hours?

That process would be a major gain for firms. They can hugely reduce costs by spreading the fixed cost per worker over more hours of work. More hours worked shouldn't increase costs of healthcare, training and fringe benefits so the fixed costs fall in line with their reduced workforce.

There are also productivity boosts by lengthening the work day itself in terms of travel time, starting up computers and lunch breaks that we wouldn't expect to increase at the same rate as hours worked. The longer work day may also push forward globalization as workdays overlap for longer periods across time-zones.

Once again though, the gains go even deeper than this. Probably the largest productivity improvement  of all will be in human capital accumulation and the returns an individual can expect from it. From the perspective of a student, a sleepless world is an increase of a third in expected working lifespan giving a longer time period for her to benefit from the investment she has made in her human capital. It also means she can potentially finish her education in a shorter time or use the additional waking hours in employment, reducing the opportunity cost of an extended university education.

The sleepless world may finally begin to reverse the age inequality in employment (at least temporarily). Young people will accumulate job experience at a faster rate through more hours of work experience. They may also be more willing to adopt the new drugs than older generations increasing their relative value as employees.

So given the benefits, do we want to grab this piece of low hanging fruit?

The main argument against it is that it will not be a voluntary choice for an individual. If these drugs become widely available then workers who are willing to use them will easily out-compete those who do not. In addition, if the drugs are expensive then it will further increase inequality between those who can afford them and those for whom the price is prohibitive. However, a Google search of various dubious websites suggest the current drugs retail for around $5 a day. At that price, even those working at minimum wage would find it worthwhile to buy the pills.

The final argument against is that a rapid introduction of these pills would amount to an increase in the labour supply and cause a fall in hourly wages or unemployment. However, it's likely that individuals would generally still see an increase in their overall income and their additional leisure time (2 hours extra) would allow this to be translated into an increase in demand in the economy through increased consumption.

Overall the transition to a sleepless world seems beneficial to humanity. There's nothing special about the 7 hours of sleep we get right now and I think people would rightly be opposed to a change that made everyone spend an extra hour asleep every day.

Caveats:

• I've never used Modafinil. This is because I don't know where to buy it, I have some moral qualms about using it when the rest of the world is not and because it is still a bit early to conclude that there are no long term health effects;
• Some people I've talked to have raised the issue of environmental damage. I think the total environmental impact of a sleepless world could be positive or negative but surely the damage would be lower per unit of output (because there are a lot of fixed carbon outputs per work day such as commuting and building overheads). At the very least, a sleepless world looks like a more environmentally friendly growth strategy;
• This argument is premised on the safety of these drugs. Clearly the calculus will change if they are shown to have negative long term consequences;
• For those people who already work long hours with little sleep, these drugs should at least make that lifestyle less dangerous. There is convincing evidence that chronic lack of sleep is harmful in normal circumstances;
• The precise amount of sleep that a Modafinil user can get by with seems to vary but all sources I've seen suggest it is dramatically lower;
• The short term costs of a rapid change might be substantial so gradual adoption is probably preferable from the standpoint of welfare.