​Research Enfield Council Based The Decision To Commence Licensing Of Private Landlords On.

Enfield Council based their decision to commence licensing of private landlords on a flawed study by an obscure consulting group known as NKM titled "Understanding The Relationship Between Private Rented Properties And Anti-Social Behavior In Enfield".  Here are extracts from a critical appraisal of the NKM Study:

ASB Data Highlights

1. Data did not verify property addresses of ASB incidents.

2. Out of 29,481 reported cases of ASB between April 2011 and July 2013 by NKM, only 1,530 were matched to actual addresses.

3. Reported incidences were compared to properties identified as most likely to be privately rented and categorised as such.

Analysis

1. Model used was “probabilistic” hence did not give a definite answer on whether a property is rented or not.

2. Model consisted of:

• 4 probable risk factors with best predictive power giving rise to 16 possible risk factor combinations.

1. “Goodness-of-fit” of the statistical model used was not calculated or was weak hence was not disclosed by NKM for scrutiny.

2. P-values not stated for statistical significance of probable risk factors.

3. Confidence intervals of probable risk factors not stated.

4. Odds ratio of probable risk factors were not calculated from the data analysis but based on extrapolation.

5. The report authors clearly state that some of their so-called risk factors may overstate the risk.

6. The analysis did not include any power calculations in the published report and hence could not make any categorical statements.

Sources Of Error In The Analysis.

1. The predictive power of probable risk factors established by follow-up visits by NKM was unblinded and therefore likely to be biased.  To be bias-free, both the investigator and the study population must be blinded to the study and its effect.[i]  Further, a control population must be in place.

2. The report authors admit that they had no comprehensive data to base the predictive power of the model they used. To do this, they would have needed to establish new data collections or undertaking large-scale surveys.  In other words, the NKM study lacked statistical power to make any categorical statements.

3. Risk factor weights that NKM previously identified in the London Borough of Newham, an inner city borough, may not be applicable in a suburban borough like Enfield.

4. The data analysed may have been subject to manipulation towards a pre-determined outcome as it was never locked and a specific data custodian was never appointed as would have been normal in this type of study to safeguard against the data being “fiddled”.

Weaknesses Of The Study

1. No research question was asked.

2. No hypothesis was proposed.

3. The investigators did not carry out a pilot study in LBE to generate a hypothesis before conducting the main study.

4. The investigators never asked neither did they obtain informed consent from the people whose data they used in the study.

5. The data the investigators generated in the study was never locked to prevent manipulation at the end of data collection. 

6. The investigators never appointed a data custodian.

7. The statistical model used was “probabilistic” and could not verify outcomes.

8. The so-called risk factors identified by the investigators have never been identified by any other investigator.

9. The investigators only studied 1,530 incidents out of 29,481 supposed incidents of ASB in the borough during the study period.  That is 5% of reported ASB incidents. To be able to make a power statement to cover the 128,500 residential properties in the borough of Enfield, the NKM study should have recruited 7,900 verified incidents. 

10. The figure of 29,481 ASB incidents during the study period contradicts the ASB plus all other crimes in the borough recorded by the Metropolitan Police Service as 28,587 during the same period and published by the Mayor of London in the MOPAC report.8

11. The report authors did not provide for the differences in the characteristics of Enfield compared to Newham, They treated the 2 boroughs as though they were one. These differences include:

• Properties in the 3 highest risk categories in Enfield accounted for 3.1% of properties versus 12.7% in Newham.

• Not receiving Council Tax benefit, which was the cornerstone of the statistical model the authors developed was 69% in Enfield versus 36.8% in Newham.

• Higher turnover of Council Tax liable persons in 36 months in Newham (high turnover of this index is one of the supposed risk factors identified by the report authors)

• Higher turnover of Electoral Roll registrants in 36 months. (another supposed risk factor for ASB identified by the report authors despite admitting that the dataset they analysed covered only 27 months)

1. Standard error or standard error of the mean, which is a measure of how precisely the sample mean approximates to the population mean was not calculated. Thus, the report authors could not construct a key statistical measure known as the “confidence interval” in the study population.

2. The analysis did not include a funnel plot and as such did not rule out asymmetry due to combining data from multiple sources.

3. The analysis did not provide for important confounders like:

• ASB due to missed bin collections.

• ASB from commercial organisations like betting shops.

• ASB from late-opening establishments like nightclubs.

• Street gangs.

• London riots of 2011 despite its occurrence during the study period.

1. The report authors admit that they did not uniquely ascribe ASB to individual households.

2. Neither the Null hypotheses nor the Alternate Hypotheses was tested in this study. [i]

3. The study was silent on the whether or not any outcome measure reached statistical significance.

4. The study was not powered to make any categorical statements. This wrongful use of the statistical analysis of retrospective data to estimate the power resulted in uninformative and misleading values.[ii]  A test's power is the probability of correctly rejecting the null hypothesis when it is false. A test's power is influenced by the choice of significance level for the test, the size of the effect being measured, and the amount of data available. A hypothesis test may fail to reject the null hypothesis, for example, if a true difference exists between two populations being compared by a t-test but the effect is small and the sample size is too small to distinguish the effect from random chance.[iii]

5. When two continuous variables like ASB and privately rented properties are studied, it is usual to calculate the correlation coefficient from the data generated to see if there is a linear association between the two variables.  This is usually a figure between -1 and +1.  This was not done in this study and as such no association was proved.[iv]

6. The study used nonsensical terms like “co-location” that is more appropriate for warehouses or IT servers  than an unbiased scientific report.

7. The study was not peer-reviewed.

Does this study stand up to scrutiny?

1. No, it does not.

2. This is the scientific research was not subject to peer review and yet was publicly funded.[v], [vi]

3. We submit that the appearance of misleading statistical analysis is not surprising considering the existence of data irregularities and the other biases that may have occurred in the process of locating, selecting and combining data from multiple unverified reports. [vii], [viii],

4. Given the glaring inconsistencies in this study and its conclusions, it is very likely that this is not a simple failure of research governance but a case of fabrication of data and analysis - research fraud. [ix], [x]

References

[i] Ellis PD. The Essential Guide to Effect Sizes: An Introduction to Statistical Power, Meta-Analysis and the Interpretation of Research Results.  Cambridge University Press. 2010

[ii] Cramer D.  Advanced quantitative data analysis.  Open University Press, Maidenhead. 2003

[iii] Hoenig JM, Heisey DM. The Abuse of Power. The American Statistician 2001; 55:19-24

[iv] Ellis PD. The Essential Guide to Effect Sizes: Statistical Power, Meta-Analysis, and the Interpretation of Research Results. Cambridge University Press 2010; p. 52.

[v] Altman DG. Practical statistics for medical research.  1st Edition. Chapman & Hall 1991

[vi] Mulrow CD. Rationale for systematic reviews. BMJ 1994; 309: 597-9

[vii] Bacchetti P. Peer review of statistics in research: the other problem. BMJ 2002; 324: 1271 – 1273

[viii] Eysenck HJ. An exercise in mega-silliness. Am Psychol. 1978; 33: 517

[ix] Huque MF. Experiences with meta-analysis in NDA submissions. Proc Biopharmaceutical section. Am Statist Assoc 1988; 2: 28-33

[x] Egger M, Davey Smith G. Misleading meta-analysis. Lessons from “an effective safe, simple” intervention that wasn’t.  BMJ 1995; 310: 752-4

[xi] Egger M, Davey Smith G, Meta-analysis: bias in location and selection of studies. BMJ 1998 316:61-66