What to do when you get results that don't make sense

A few different recent conversations and this blogpost on list experiments by Andrew Gelman have made me think about the nature of the file drawer problem.

Gelman quotes Brendan Nyhan

I suspect there’s a significant file drawer problem on list experiments. I have an unpublished one too! They have low power and are highly sensitive to design quirks and respondent compliance as others mentioned. Another problem we found is interpretive. They work best when the social desirability effect is unidirectional. In our case, however, we realized that there was a plausible case that some respondents were overreporting misperceptions as a form of partisan cheerleading and others were underreporting due to social desirability concerns, which could create offsetting effects.

and Lynn Vavreck:

Like the others, we got some strange results that prevented us from writing up the results. Ultimately, I think we both concluded that this was not a method we would use again in the future.

Many of the commenters on the blog said that the failure to publish on these results reflected badly on these researchers and that they should publish these quirky results to complete the scientific record. 

Both of these examples as well as many other stories I've heard make me think that the major causes of the file drawer effect in social science are not null results but inconclusive, messy and questionable results. The key problem is when you get a result from an analysis that makes you reassess some of the measurement assumptions that you were working with. For instance, a secondary correlation with a demographic variable comes out in an unexpected direction or the distribution of the responses is bunched up in 3 places on a 10 point scale.

The problem comes down to this. If I design a survey or other study to test an empirical proposition, the study is likely not to be ideally designed to test the validity of the measures involved and how the design effects are impacting them.

The results you get from a study designed to test an effect are often enough to cast doubt on validity but rarely are enough to demonstrate the lack of validity in a convincing way (i.e. that would be of publishable quality). The outcome is therefore that the paper can either be written up as a poor substantive article (i.e. the validity of the measures is in doubt or a poor methodological article (the evidence about the validity of the measures is weak either way because the study wasn't designed to be a test of the measure's validity).

One answer to this is to do more pre-testing. This can help to establish the validity of measures prior to working with them and can certainly identify the most obvious problems. However, unless the pre-test is nearly as large as the actual sample, the correlations with other variables won't be particularly clear in advance. In addition, pre-testing won't help understand design effects unless it tests different combinations.

However, what is really needed is whole studies devoted to examining design effects experimentally and establishing the measurement attributes. But until that happens for methods such as list experiments, researchers will be stuck with questionably valid results that are hard to publish as good empirical or methodological pieces.

A more radical approach would be to encourage journals of ideas that didn't quite work out. Short research articles that explain why the idea should have worked out nicely but ended up being a damp squib. These would be useful for meta-analysis of why certain techniques are problematic in practice without having the same time requirements for writing up as a full methodological piece.

An idea for a more useful Google Trends

I've written a couple of articles (the second one is coming out in JEPOP some time soon but I don't have a link yet) about the limitations of using Google Trends data for social science research. The major issue is that many more search terms seem like plausible measures than actually turn out to correlate with public opinion. Search terms don't even necessarily work across different countries that speak the same language!

As a result, any use of these trends has to go through the process of matching up the data to equivalent survey data before it can be used validly.

But what if we didn't have to do all that?

One of the reasons I suspect the Google Trends sometimes don't match up as well as we would hope is that it counts searches not people. A handful of furiously searching journalists and politicos can drive the trend as much as widespread searching across the population. This means that issues may be ignored by 99% of the population but still result in a lot of Google searching.

So the graph above tracks what percentage of all searches in the United States were for the term "Syria" on different dates (these percentages are then scaled to a 1 to 100 index so we don't know the actual percentages).

This representative problem is easily solvable for Google. Simply report the trends for number of people searching for a term instead of the number of searches for a term. Google could simply offer us the option of tracking the percentage of people using Google on each date who searched for the term "Syria". Even if journalists search for Syria a thousand times, it will only count as one person.

It's not hard for Google to identify different people either. While there are some complexities to tracking an individual over time, Google has been building profiles on its users for a long time and even a measure of the number of unique IPs that searched for a term would go a long way towards solving this problem.

Having both of these settings as an option would give much greater insight into the breadth and depth of opinion on an issue.

It might even make offhand references to Google Trends as a proxy for public opinion a little more accurate.

Note: There are other reasons why Google Trends data might not match up to public opinion (see the papers) but this is certainly one major concern.

Will general election turnout stop UKIP repeating their performance?

UKIP's record breaking performance has the world (or that subset of it that follows English local elections closely) talking about whether they might repeat this performance at the general election. If they did, then they might replace the Liberal Democrats as the third party in the House of Commons.

Before getting carried away with speculation on Nigel Farage's role in a future coalition government, it is worth considering some factors that might limit this. One factor is the odd electorate that votes in local elections. While the BBC's projected national share accounts for the difference in the areas that vote, it doesn't account for the difference in the electorate that turns out. These differences can be large, just 31% of eligible voters made it to polling stations yesterday compared to the 65% who voted on election day.

These differences aren't random either. In particular, local elections voters are much older  those at general elections (broadly, the elderly will turnout in every election whereas the young tend to only show up for high profile contests). Conservatives and liberal democrats have been the traditional beneficiaries of this differential turnout but UKIP has a strong base of support among the elderly.

So what would these results have looked like if the turnout had been 65% rather than 31%. To give a rough answer to this question I looked at how much share UKIP gained since 2005 (a general election) in each ward depending on how turnout changed between the 2005 General Election and 2013's local contests.

	UKIP vote change 05-13
Turnout change 05-13	-0.065
	(2.42)*
_cons	17.810
	(15.14)**
R2	0.03
N	163

* p<0.05; ** p<0.01

As expected, UKIP improved their performance more in contests that saw a sharper drop in midterm turnout. However, this would not have been sufficient to dent their performance greatly: they would lose a total of 2 percentage points bringing them from a 23 point PNS to a 21 point result.

While UKIP benefited from low turnout, it is not enough to begin to explain their huge electoral gains. It will take more than robust turnout to reverse their success in these elections.


Note: These results are also robust to a set of controls:

	UKIP vote change 05-13
Turnout 05-13	-0.061
	(2.25)*
Population Density	-0.015
	(0.29)
% aged 65+	0.293
	(2.05)*
% white	-0.156
	(0.67)
% aged 18-24	-0.193
	(1.27)
_cons	28.916
	(1.31)
R2	0.10
N	160

* p<0.05; ** p<0.01

Caveats:

• See previous post
• I realise I'm in danger of contributing to the "questions to which the answer is no" genre of blog writing. In my defense I didn't know that the answer would definitely be no in advance. 

Blown away: How much impact did wind farms have on the UKIP vote?

UKIP's rapid rise has led to certain gaps in their policy platform. The party's core issues of immigration and Europe have been fully articulated but the rest of the platform is still in a state of flux. Another line of policy has been to take up various "NIMBY" (not in my back yard) issues. For instance, their Yorkshire and Lincolnshire webpage gives high prominence to the wind farms. In fact it's the only policy area mentioned on the site other than Europe.

So what should we make of this NIMBY focus? Is it a key part of their appeal or just window dressing around their core anti-immigration/EU message? 

Taking a first look at this, I've compared the UKIP performance in wards where there is a wind farm to those without one. I was helped in this by a wikipedia article listing the coordinates of all onshore wind farms in the UK (I am very intrigued about who put this together) and the ever useful mapit API. 

For now I'm simply analysing the difference across the BBC's keywards (those councils that they analyse in detail) that have declared results  (as of 3.31pm 3/5/2013), so these results are very preliminary. Among these wards, 19 have wind farms and 14 of these have UKIP candidates standing.

In these 14 wards UKIP averaged 32.7% of the vote, this compares with a share elsewhere of 24.6% (n = 1170). If this is robust (a big if), it would make the presence of wind farms one of the biggest effect on UKIP share.

Of course, it may simply be that UKIP does well in rural areas, which also tend to be the ones containing wind farms. 

A quick regression analysis suggests that this isn't the case. Although the difference is not quite as large as the raw figures, they still perform 6 percentage points better in wards with wind farms that those without. 

	UKIP 2013 share
Population density	-0.054
	(3.49)**
Wind Farm	6.160
	(2.86)**
_cons	25.689
	(73.17)**
R2	0.02
N	1,083

* p<0.05; ** p<0.01

So the analysis so far suggests that NIMBY issues may have some potential for UKIP. 

But wind farms are only relevant to a small number of wards. A second NIMBY issue that might have more wider relevance is High Speed 2. The proposed route of the train line cuts through many councils being counted today. UKIP have been slower to jump on this issue, but we can look at whether it's helped their vote. The potentially affected postcodes are listed by safe-move.co.uk and were coded up using mapit.

Unlike their wind farm success, the regression suggests that there is little difference in UKIP performance in wards affected by HS2.

	UKIP 2013 share
Population density	-0.051
	(1.94)
HS2	-2.919
	(0.81)
_cons	11.580
	(18.25)**
R2	0.02
N	291

* p<0.05; ** p<0.01

So far then, UKIP's share does not appear to have been driven primarily by NIMBY issues. Their possible success in mobilizing support around wind farms has not been replicated for HS2, which is potentially much more widely relevant. 

The lack of an effect of HS2 might be seen as a missed opportunity for UKIP. However, it also underscores a positive result for them: their strong showing is not merely the result of canny use of local issues but a genuine national shift in their favour. 

Caveats:

• These are obviously preliminary results and there may be other factors to control for. I hope to analyse some of these in future posts.
• The worst affected HS2 postcodes are in Buckinghamshire which the BBC is not covering due to large boundary changes. 
• It is questionable whether these effects should be seen at ward level or perhaps at district level. I'll look at this question in more detail later.
• Obviously correlation =/= causation.
• Ideally, I would show the regression for the changes in the shares since 2009 and 2005 but UKIP has fielded candidates in so many new locations, that there simply aren't enough results for comparison.
• This blogpost does not reflect the opinion of the BBC or my department.