Tuesday, 24 May 2016

Who wants the TEF?

I'll say this for the White Paper on Higher Education "Success as a Knowledge Economy": it's not as bad as the Green Paper that preceded it. The Green Paper had me abandoning my Christmas shopping for furious tirades against the errors and illogicality that were scattered among the exhausted clichés and management speak (see here, here, here, here and here). So appalled was I at the shoddy standards evident in the Green Paper that I actually went through all the sources quoted in the first section of the White Paper to contact the authors to ask if they were happy with how their work had been reported. I'm pleased to say that out of 12 responses I got, ten were entirely satisfied, and one had just a minor quibble. But what about the twelfth, you ask. What indeed?
When justifying the need for a Teaching Excellence Framework (TEF) last November, Jo Johnson used some extremely dodgy statistical analysis of the National Student Survey to support his case that teaching in some quarters was 'lamentable'. I was pleased to see that this reference was expunged from the White Paper. But that left a motheaten hole in the fabric of the argument: if students aren't dissatisfied, then do we really need a TEF?  One could imagine the civil servants rushing around desperate to find a suitably negative statistic. And so they did, citing the 2015 HEPI-HEA Student Academic Experience Survey as showing that "Many students are dissatisfied with the provision they receive, with over 60% of students feeling that all or some elements of their course are worse than expected and a third of these attributing this to concerns with teaching quality." (p 8, para 5).  The same report is subsequently cited as showing that: ".. applicants are currently poorly-informed about the content and teaching structure of courses, as well as the job prospects they can expect. This can lead to regret: the recent Higher Education Academy (HEA)–Higher Education Policy Institute (HEPI) Student Academic Experience Survey found that over one third of undergraduates in England believe their course represents very poor or poor value for money." The trouble is, both of these quotes again use spin and dodgy statistics.
Let's take the 60% dissatisfaction statistic first. The executive summary of the report stated; "Most students are satisfied with their course, with 87% saying that they are very or fairly satisfied, and only 12% feeling that their course is worse than they expected. However, for those students who feel that their course is worse than expected, or worse in some ways and better than others, the number one reason is not the number of contact hours, the size of classes or any problems with feedback but the lack of effort they themselves put in." So how do we get to 60% dissatisfied? This number is arrived at from the finding that 12% said that their experience had been worse than expected, 49% said that it had been better in some ways and worse in others. So it is literally true that there is dissatisfaction with 'some or all elements', but the presentation of the data is clearly biased to accentuate the negative. One is reminded of Hugh in 'The Thick of It' saying "I did not knowingly not tell the truth".
But it gets worse: As pointed out on the Wonkhe blog, among 'key facts' in a briefing note accompanying the White Paper, the claim was reworded to say over 60% of students said they feel their course is worse than expected. The author of the blogpost referred to this as substantial misrepresentation of the survey. This is serious because it appears that in order to make a political point, the government is spreading falsehoods that could cause reputational damage to Universities.
Moving on to perceptions of 'value for money', there are two reasons for giving this low ratings  - you are paying a reasonable amount for something of poor quality, or you are paying an unreasonable amount for something of good quality. Alex Buckley, one of the authors of the report replied to my query to say that while the numeric data were presented accurately, crucial context was omitted. This made it crystal clear it was the money side of the equation that concerned students. He wrote:
"Figure 11 on page 17 of the 2015 HEPI-HEA survey report shows that students from England (paying £9k) and students from Scotland studying in Scotland (paying no fees) have very different perceptions of value for money. And Figure 12 shows that the perceptions of value for money of students from England plummeted at the time of the increase in fees. Half of 2nd year students from England in 2013 thought they were getting good or very good value for money. In 2014, when 2nd years were paying £9k, that figure was a third. (Other global perceptions of quality - satisfaction etc. - did not change). There is something troubling about the Government citing students' perceptions of value for money as a problem for the sector, when they appear to be substantially determined by Government policy, i.e. the level of fees. The survey suggests that an easy way to improve students' perceptions of the value for money of their degree would be to reduce the level of fees - presumably not the message that the Government is trying to get across."
So do students want the TEF? All the indicators say no. Chris Havergal wrote yesterday in the Times Higher about a report by David Greatbatch and Jane Holland in which students in focus groups gave decidedly lukewarm responses to questions about the usefulness of TEF. Insofar as anyone wants information about teaching quality, they want it at the level of courses rather than institutions, but, as an ONS interim review pointed out, the data is mostly too sparse to reliably differentiate among institutions at the subject level. Meanwhile, the NUS has recommended boycotting the National Student Survey, which forms a key part of the metrics to be used by TEF.
This is all rather rum, given that the government claims its reforms will put students at the heart of higher education. It seems that they have underestimated the intelligence of students, who can see through the weasel words and recognise that the main outcome of all the reforms will be further increases in fees.
It's widely anticipated that fees will rise because of the market competition that the White Paper lauds as a positive stimulus to the sector, and it was clear in the Green Paper that one goal of the reforms was to tie the TEF to a regulatory mechanism that would allow higher fees to be set by those with good TEF scores. Perhaps less widely appreciated is that the plan is for the new Office for Students to be funded largely by subscriptions paid by Higher Education Providers. They will have to find the money somewhere, and the obvious way to raise the cash will be by raising fees. So students will be in the heart of the reforms in the sense that having already endured dramatic rises in fees and loss of the maintenance grant, they will now also be picking up the bill for a new regulatory apparatus whose main function is to satisfy a need for information that they do not want.

Saturday, 7 May 2016

Would paying by results improve reproducibility?

Twitter and Facebook were up in arms last week. "Merck wants its money back if University research is wrong" was the headline to the article that set off the outrage.

Commentators various described the idea as dangerous, preposterous and outrageous, and a 'worrying development', while at the same time accusing Merck of hypocrisy for its history of misleading claims about its vaccines and drugs.

In the comments beneath the article, similar points were made: 'No way any academic institutions will agree to this'; 'If you want absolute truth take up religion'; 'Merck wants risk-free profit'.

But if you follow the link to the article that the story was based on, it's clear that the headline in the Technology Review piece was misleading.

For a start, the author of the piece, Michael Rosenblatt, was clear that he was not representing an official position. Under 'competing interests' we are told: "M.R. is an employee of and owns stock and stock options in Merck & Co. The opinions expressed in this article are those of the author and do not necessarily correspond to the views of the author’s employer."

Rosenblatt's focus is irreproducible scientific research: a topic that is receiving increasing attention in biomedicine among other disciplines. He notes the enormous waste that occurs when a pharma company like Merck attempts to use basic biomedical research to develop new drug targets. Whether or not you approve of Merck's track record, there is no question but they have legitimate concerns. He explains that the costs of building a translational project on shaky biomedical foundations are so great that pharma companies will now routinely try to replicate original findings before taking them forward. Except that, all too often, they find they cannot do so. Since it can take between 2-6 scientists one to two years to conduct a replication, the costs of irreproducible science are substantial.

Rosenblatt notes that there have been numerous suggestions for improving reproducibility of biomedical science; mostly these concern aspects of training, and altering promotion practices so that academic scientists will have an incentive to do reproducible research. His proposal is that pharmaceutical companies could also address incentives by making their financial deals with universities contingent on the reproducibility of the results.

In effect, Rosenblatt wants those who do the research in universities to ensure it is reproducible before they bring it forward for pharmaceutical companies to develop. His proposal does indeed include a 'money back' guarantee, so that if a result did not hold up, the pharmaceutical company would be compensated. But, Rosenblatt argues, this would be compensated for by additional funds made available to universities to enable them to adopt more reproducible practices, including, where necessary, replicating findings before taking them forward.

The Technology Review headline misses all this nuance and just implies that Merck is being naïve and unrealistic in assuming that scientists have some kind of pre-cognition about the outcomes of experiments. That is far from being the case: the emphasis is rather on the importance of ensuring that findings are as solid as possible before publishing them, and suggesting that pharma companies could play a role in incentivising reproducible practies by tweaking their funding arrangements with Universities.

Obviously, the devil would be in the detail, and Rosenblatt is careful to avoid being too specific, suggesting instead that this is not intended as a panacea, but rather as an idea that would be worth piloting. I'm as cynical as the next person about the motives of pharmaceutical companies, but I have to say I think this is an interesting suggestion that could be in the interests of scientific progress, as well as benefiting both universities and pharmaceutical companies.

Tuesday, 19 April 2016

Talking about tax avoidance: weasel words

With recent revelations about rich people legally avoiding tax, it seems a good time to tell this story.

In late 2013, I received an email telling me that I could potentially save huge amounts in tax on my pension. I thought this must be a particularly sophisticated spam: poverty in the UK was rising fast, and I have high earnings, so why would pension rules be changed to benefit me? But no, I checked it out and it was all legal and above board.

All is made clear in this piece*. The phraseology has some gems: I particularly enjoyed 'the appropriate crystallisation event, such as the point of death', but behind all the talk of accrued benefits and recovery tax charges, the message seems to be that the Government had introduced a new measure that would increase the amount of tax paid by rich people, but HMRC had immediately provided a legal means for avoiding this, 'Fixed Protection'.

I'm fascinated by the use of language: the word 'protect' has entirely positive connotations: my online dictionary defines it as 'keep safe from harm or injury'. We hear of tax 'shelters', 'tax-advantaged savings vehicles' and suchlike. But what is achieved here is tax avoidance by rich people – all aided and abetted by HMRC.

If you have money and you talk to solicitors or financial advisers, you'll find that the default assumption is that you want to pay as little tax as possible. I first came across this when making a will: the solicitor concerned started describing how I could set up a complex trust that would mean less tax would be paid on my estate. I don't have dependents so this struck me as particularly pointless. I'd be dead and wouldn't care. The solicitor looked at me aghast and clearly thought I was barking mad.

I'm not a saint and I'm not an idiot. I have some savings. But I live a comfortable life and don't need to squirrel away vast amounts of dosh. And I grew up in the 1960s when, although things weren't perfect we had a reliable National Health Service; schools had adequate resources; there were grants to support people through university. The current generation may find it remarkable that until I was about 30, I never saw a beggar on the streets of any British city. Now we have food banks. An equable society that looks after the most vulnerable costs money, and that money comes from taxes.

I'll leave the last word to J.K.Rowling, who is one of the few rich people who seems to get it:

I chose to remain a domiciled taxpayer for a couple of reasons. The main one was that I wanted my children to grow up where I grew up, to have proper roots in a culture as old and magnificent as Britain’s; to be citizens, with everything that implies, of a real country, not free-floating ex-pats, living in the limbo of some tax haven and associating only with the children of similarly greedy tax exiles. 

A second reason, however, was that I am indebted to the British welfare state; the very one that Mr Cameron would like to replace with charity handouts. When my life hit rock bottom, that safety net, threadbare though it had become under John Major’s Government, was there to break the fall. I cannot help feeling, therefore, that it would have been contemptible to scarper for the West Indies at the first sniff of a seven-figure royalty cheque. This, if you like, is my notion of patriotism. On the available evidence, I suspect that it is Lord Ashcroft’s idea of being a mug.

*I found this on Google search: I don't have any dealings with this firm of solicitors, but as far as I can tell, what they say matches the advice I had earlier, and is pretty standard