Report No 21: Enhancing the value of health statistics: User perspectives

This report is the first by the Statistics Commission to take a broad look at one entire field of official statistics - in this case health statistics. It was prepared under the guidance of a project board chaired by commissioner Sir Kenneth Calman and with the support of York Health Economics Consortium who carried out a detailed review on behalf of the Commission.

The decision to look first at health reflects the central importance in all our lives of messages drawn from health data. These messages are so pervasive that we tend to forget they are based on the analysis of statistical data. Many people will not have seen the tables of figures but we all know roughly how long we are expected to live, that smoking and excessive alcohol will shorten that period, that heart disease or cancer will kill most of us eventually, that you have to wait for much hospital treatment - and in some cases for longer than anyone would wish. These messages, and many others, come from analysis of statistical data and their validity is directly dependent on reliable statistics being collected and properly analysed.

A moment's thought points to the importance of health data in other walks of life too, in setting life insurance premiums, ensuring the safety of our food, in government advice on diet and obesity, measures to protect sexual health and prevent drug abuse, in the design of our buildings and cars (no asbestos in the first; air-bags in the second), the identification of the most deprived parts of the country, the need for social services and care for the elderly. Less familiar, but equally important, is the role of health data in government formulae for allocating vast sums of public money to different areas of the country.

The Statistics Commission approaches all statistical issues from the perspective of those who use, or may one day need to use, the figures and the messages from them in decision-making. In fields such as health this includes us all to some extent. We decide whether to smoke, what to eat, when to seek medical advice, how much exercise to take.

This report, building on the one prepared by York Health Economics Consortium, recognises the great wealth of health data now available, or planned, and does not attempt to focus on the occasional gaps or flaws in this rich fabric - that can safely be left to others.

Our approach has rather been to identify a small number of high-level proposals that will enhance the capacity of health statistics to support decision-making and the wider user community. In the event, our proposals have relevance beyond the field of health statistics and raise a number of issues of importance for official statistics more generally. We believe our recommendations are balanced and achievable and will prove good value for money when the costs and benefits are fully assessed. We will be seeking in the coming months to establish with all the relevant government departments and agencies the best way of taking them forward.

I thank all those who contributed to this important exercise.

Introduction

1. This report by the Statistics Commission forms Part One of Statistics Commission Report No. 21. It looks at UK official statistics relating to health, and builds on a review carried out for the Commission by York Health Economics Consortium (YHEC)². A summary of YHEC's conclusions and recommendations is included at Section V. The full report from YHEC follows as Part Two of this volume.

2. There is now a great wealth of health statistics available and the focus of this report is not on adding to that infrastructure so much as on ways to enhance the value of the statistics to decision-makers. The decisions in question include those taken in the context of managing health services but extend much further. At the most fundamental level, health statistics underpin society's understanding of what is good for us and what is not - and so inform advice about diet and smoking, life insurance premiums, the laws on drug abuse and so on. Health data also indicate which parts of the country are most deprived and in greatest need of services - and so inform the allocation of resources to local authorities, social services and the targeting of national schemes.

3. Thousands of organisations from GP practices to national institutions, university research departments, drug companies, insurance companies, charities and of course NHS management, draw regularly on statistical information either in the form of data or as messages taken from analysis of the data. Its importance in shaping the institutions and values of society cannot be overstated.

The Health Service context

4. In the context of health service management, the scale of decision-making is vast. Public expenditure on health services this year will be of the order of £80 billion and is rising rapidly. This equates to some £1,350 annually for every man, woman and child in the UK. The NHS, government, Parliament and the citizen all want to know whether this money is being spent as effectively as possible, whether the advice and treatment provided by the NHS is as good as possible, and whether services are as good where we live as they are elsewhere in the UK, and elsewhere in the world.

5. We live in an age of audit and accountability. Every important decision is now expected to be supported by evidence and be subject to review and public scrutiny. This applies at all levels of health service management from the GP deciding on a diagnosis to government deciding on the allocation of funding, the price of prescriptions, or the salaries of doctors and nurses. Official statistics of many different kinds form the bedrock of the evidence base. Although their collection undoubtedly places an additional, and at times unwelcome, burden on public services - not least the staff of the NHS - they are the key to informed decision-making and public accountability.

6. To illustrate the importance of statistics in NHS management, it is instructive to look at the way the NHS Improvement Plan, published in June 2004, describes progress since 1997. It states that:

• "The maximum waiting time for an operation has fallen from 18 months to less than nine months.
  
  
• The maximum waiting time for an outpatient appointment has fallen from 26 weeks to 17 weeks.
  
  
• 97% of patients are now able to see a GP within two days. Growing numbers of patients are taking advantage of new services such as NHS Direct and NHS Walk-in Centres.
  
  
• 94% of patients are seen, diagnosed and treated within four hours of arrival at accident and emergency.
  
  
• Increases in staff numbers since 1997 are also improving services for patients. They include a 22% increase in doctors, a 21% rise in the total number of nurses and a 27% expansion in scientific, therapeutic and technical staff.

· In national surveys patients are increasingly positive about the quality of their care. Beyond the clinical arena, there has been progress on issues where patients have asked for more focus, with cleaner hospitals, better food and better provision of bedside phones and televisions."

7. Every one of these statements rests on the collection and analysis of statistics and could not have been made without that analysis. And looking to the future, the NHS Improvement Plan draws on further statistical indicators, for example:

• "A maximum of 18 weeks from referral by their GP".
  
  
• "A 40% fall from 1997 in death rates from heart disease and stroke".

8. So the critical role of statistics in shaping and steering decisions inside and outside the health services is beyond question. The aim must therefore be to take all reasonable steps to enhance their value. One focus of the YHEC review was to find out more about the non-government users of health statistics - who they are, what they use the data for, and what changes might be made to better meet their needs.

9. The review sought to identify the main users of health statistics, and their views through interviews and questionnaires. A number of themes emerged from this evidence-gathering process, some of which were subsequently pursued in more depth. The Statistics Commission has drawn heavily on the YHEC report, and its conclusions and recommendations. We believe that the review has identified scope for actions in a number of areas, which could improve the usefulness and accessibility of the statistics. Based on the review, the Commission has developed a set of six recommendations that we believe should be priorities for producers of health statistics. The background to the Commission's recommendations is described in the following section of this report. The recommendations themselves are set out in full in Section III.

²Information about a statistical series that enables the user to gauge quality, context, comparability, limitations etc.

Background to Commission recommendations

10. The Commission's recommendations are directed at the government departments, administrations and agencies that produce official health statistics - the Department of Health and its agencies, the Office for National Statistics, and the health and statistical departments of the devolved administrations. All the Commission's recommendations derive from, and are consistent with, the YHEC review conclusions and recommendations.

Use made of health statistics

11. A better understanding of the use made of health statistics was a prime concern for the review. YHEC collected some information on use made through questionnaires and interviews. For the most part the information collected was broad-brush in nature; for example, nearly 60 per cent of respondents said that their main use of health statistics was for 'research' or 'in reports'. The review report does however provide some tangible examples of the use made of specific statistics. These examples include the use of cancer registry data by regional health observatories, the use of morbidity data for 'stop smoking' campaigns and the use of health survey data by local authorities.

12. It proved more difficult to obtain robust information about which data ultimately informed which kinds of decisions. Whilst we can speculate with varying degrees of confidence about this, it would be valuable to all those concerned with official statistics to have sound evidence and we believe that producers of statistics need to know more about the way statistics are used in practice (see Recommendation 6).

Production of health statistics - structural issues

13. Official health statistics are produced and published by the Department of Health (DH), by the Office for National Statistics (ONS) and by the health departments of the three devolved administrations. ONS produces a range of statistics relating to births, deaths and public health from surveys and from NHS and other administrative sources. Some of these statistics have UK-wide coverage; others relate to England and Wales only. Other statistics produced by DH relate to England only. The equivalent statistics for the other countries of the UK are produced by the devolved administrations. This reflects the fact that provision of health services is a devolved function of government but tends to ignore the potential user requirement for consistent data across the UK.

14. The multiplicity of UK organisations producing health statistics is one of the factors that have led to difficulties in compiling figures that are consistent across the four countries. Users who want UK figures, or who want to make comparisons across countries, will often need to go to a number of different sources in order to access data - which is inconvenient at best. Worse, many figures are not directly comparable across countries because of differences in definitions and/or coverage.

15. The Department of Health has recently announced the creation of a Health and Social Care Information Centre (see Reconfiguring the Department of Health's Arm's Length Bodies, 22 July 2004, and accompanying DH press release). The intention is that the new agency will be established as a Special Health Authority from 1 April 2005. The Centre will merge the statistics and information policy functions of DH with some of the 'information' related functions of the NHS Information Agency. The aim is to "'reduce burdens on the frontline by co-ordinating information requirements across a wide range of bodies" (DH press release 22 July 2004).

16. By itself, the creation of the Health and Social Care Information Centre will not reduce the multiplicity of organisations in the UK producing official health statistics. Nevertheless the Commission believes that the new Information Centre will be in a good position - probably a better one than DH is now - to take on the enhanced central co-ordinating role that the Commission believes is required in order to improve coherence and comparability of health data across the entirety of the UK. For example, we believe that the new Information Centre will be well placed to develop and manage the online index of UK heath statistics and associated metadata that we propose in Recommendations 1 and 2. The Commission recognises that the Information Centre will only be able to do this with the full co-operation of the devolved administrations and of ONS.

Disaggregation versus confidentiality

17. The YHEC review identified a demand for more disaggregated data, both by geographical area and by indicator, in several different areas of health statistics. The Commission believes that producers of health statistics should make every effort to satisfy these demands (Recommendation 5). However it needs to be recognised that there is a potential conflict between, on the one hand, the production of more disaggregated data, and, on the other hand, data confidentiality and data protection issues that argue against the release of individual estimates derived from a small number of observations.

18. The Commission recognises the need to protect data confidentiality. It is a key principle of the National Statistics Code of Practice that "no statistics will be produced that are likely to identify an individual unless specifically agreed with them". This is consistent with the Caldicott Principles that govern, and restrict, the dissemination of patient-identifiable information within the NHS.

19. In some circumstances, eg the release of detailed disaggregated data for research purposes, the principles regarding data confidentiality and protection can be met by release of data in a suitably anonymised format. But, as well as guarding against releasing information that directly identifies individuals, it is also necessary to protect against both inferential disclosure - where information about an individual can be inferred with a high degree of confidence - and inadvertent disclosure, eg as a result of differencing of two datasets with largely but not completely overlapping populations.

20. Nevertheless the Commission is concerned that there is a risk of data confidentiality being cited as a reason for withholding disaggregated data in circumstances that do not justify it. We think that decisions as to the level of disaggregation below which data cannot be released need to be based on actual confidentiality issues regarding each individual dataset, and not on arbitrary rules that suppress all data derived from less than a specific (arbitrary) minimum number of observations. (The YHEC review reports that such a rule of thumb - the 'no less than 5 in a cell' rule - appears to be in quite wide use in the health statistics area.)

21. We believe that such blanket bans are bound to lead to suppression of more data than is necessary to meet the principles of the Code of Practice, and are best avoided. As a general rule, we would urge data producers to look for ways of releasing as much data as possible.

Performance measurement

22. Changes in the organisation and management of public services are creating new demands for statistics that can enable monitoring of activity and performance at a local level. This has been particularly the case in the health service (for example at the level of Primary Care Trusts and within PCTs). This was noted by a number of contributors to the YHEC review.

23. Given finite resources, there is always a risk that meeting a growing demand for a particular kind of information will impact on the supply and availability of data for other purposes. The YHEC review found, in the responses to interviews and questionnaires, some who argued this was occurring. The review report notes that implementation of performance targets was perceived to have had "a detrimental input on the quantity of data available for areas that are not subject to such targets".

24. It is difficult to comment on assertions by review respondents that were not backed with any clear evidence. It is certainly the case that more performance information is being collected than hitherto, but it does not follow that other statistics will necessarily suffer.

25. This is another area where the new Health and Social Care Information Centre should be in a good position - for England at least (and more widely, we hope) - to take an overview, recognising and correcting any inefficiencies or duplications in data collection, taking and publicising decisions on priorities, and ensuring that full use is made of all the information collected. In doing this, the Commission would urge the Information Centre to take into account the needs of users of health statistics outside government and the NHS, as well as of those within.

Costs and burdens

26. There are often costs associated with making improvements to data collection and analysis. Meeting the recommendations of the review and of the Commission report is likely to require additional resources, both financial and human.

27. The Commission recognises that a strong value-for-money case needs to be made for any new investment in the statistical infrastructure. This is why our Recommendations 3 and 4 call for 'fresh assessments' of the value to users of statistics before any resource commitment is made. We believe that, where it can be demonstrated that there is a clear case for improved data, the costs of those improvements are very likely to be more than offset in the longer term by the savings flowing from better decision-making and enhanced public confidence in the decision-makers.

28. It also needs to be recognised that new demands for information are likely to place an additional burden on data providers. An important aspect of controlling the burden on data providers is to ensure that information is collected and used in as efficient a way as possible, and in particular that the same information is only collected once. In this respect the Commission notes, and welcomes, the proposed focus for the new Health and Social Care Information Centre on co-ordinating information requirements, with the aim of better managing the burden on data providers.

Issues and recommendations

29. The following paragraphs set out the Statistics Commission's recommendations and explain the context in which they have been made. Recommendations are made in three broad areas:

a. supporting user access to data and analyses
b. consistency and coherence of data
c. identifying and meeting user needs.

Supporting user access to data and analysis

30. Accessibility of information is crucial for all users of statistics. This was an issue raised by a number of respondents to YHEC's interview and questionnaire enquiries. A common theme was the inconvenience of having to search multiple sources, which can be both complex and time-consuming, and result in important data not being found. Users suggested that a centralised web-based repository could greatly simplify the process of obtaining data. Such 'clearing houses' are now commonplace in some other fields and countries.

31. The Statistics Commission endorses this idea, but questions remain as to who should do it, and how. As regards how, we think that the objective should be to provide all users with a web portal that would be a first point of reference for information on, and enquiries about, statistics on all aspects of health and care relating to all countries of the UK. As regards who, we believe that the new Health and Social Care Information Centre should be well placed to take on the co-ordination function, though it would need the active and ongoing co-operation of other producing bodies - notably ONS and the devolved administrations.

32. Putting this web-based repository in place will entail collecting information, including web locations, on all health-related statistics from all the various sources, and then constructing an Index which can guide the user to the appropriate statistics. Figures 2.1 and 2.2 in the YHEC report could assist with this and offer a good example of how such an index could be 'mapped'. The Index might also indicate which of these series and reports have been produced to the standards of the existing National Statistics Code of Practice and which have not.

33. Whilst we are currently quite a way from having anything like a comprehensive repository of this kind, there are a couple of features of existing websites that the developers of the proposed Index could build on. The Health and Care pages of the ONS website - National Statistics Online - provide a web portal to a wide selection of health and care statistics published by ONS, some of which have a UK-wide coverage. And the existing DH website has an Index of Useful Links, which includes links to data series on the ONS website and on the websites of the devolved administrations, as well as elsewhere on the DH site.

34. These two features of existing websites could provide a starting point for the Index that we are proposing, but quite a bit of further development would be required. The coverage in the ONS pages is for the most part restricted to statistics published by ONS, including series published in statistical compendia such as Social Trends. And whilst the current DH Index has good links to DH data and to health data on the ONS website, the links to data held by the devolved administrations are at present very limited. It is also not in a very prominent place on the site, and certainly not located where it would be a 'first port of call'.

Recommendation 1

UK producers of official health statistics should give high priority to developing an easy to use and up-to-date online Index to available health statistics, and statistical reports, covering all four countries of the UK. Wherever possible the Index should guide the user directly to the figures and supply relevant contextual information and advice on interpretation (see Recommendation 2).The new Health and Social Care Information Centre will be well placed to lead on this development, with the agreement and co-operation of other producers.

35. A crucial component of an on-line Index would be metadata³. YHEC found that user perceptions of current metadata for health statistics were generally quite good. Nevertheless the format and content varied by type of publication and producing organisation. With implementation of Recommendation 1 above, it should become easier - but at the same time essential - to adopt a common approach that ensures the presentation of consistent metadata. The keepers of the online Index will have a key role in establishing and co-ordinating metadata standards with all UK producers of statistics.

36. A standard template would ensure that the data source attaches information such as quality indicators and limitations of use in a consistent format. If this information cannot be provided then the keepers of the online Index will need as much background information regarding the methods used to gather the data as is available, so as to help them make their own judgements in support of users.

Recommendation 2

UK producers of official statistics should work together to produce metadata in a consistent format. This metadata should be accessible from the Index proposed in Recommendation 1.

Consistency and coherence

37. A case study described in the YHEC report noted substantial differences in the availability of detailed data for specific diseases. Data on diabetes are produced for England and Wales, disaggregated to a regional level only, whereas cancer data are available disaggregated in a number of different ways - by Strategic Health Authority (SHA), by Primary Care Trust (PCT) - or equivalent in Scotland - or by postcode. The greater availability of detailed statistics for cancer is a reflection of the much larger amounts of consistent and comparable data available from the well-established network of regional cancer registries. There is no equivalent network of regional registries for diabetes, or for most other diseases and medical conditions.

38. The existence of a network of regional cancer registries, and the consequential differences in quality of detailed data for cancer as compared with diabetes, is at least in part a consequence of the higher level of funding available for data collection for cancer, reflecting past priorities. If more detailed analyses of diabetes (or of other diseases/medical conditions for which no comprehensive network of registries exists) are required, then an obvious solution is improved source data through the development of a regional registry approach along the lines of the registries for cancer.

39. The Commission believes that there could be substantial benefits from the development of networks of regional registries for more diseases and medical conditions, but accepts that this could be expensive. We therefore propose that the costs and benefits from development of regional registries for diabetes and other diseases should be further investigated, with a view to taking a decision as to whether this would be a justifiable use of public funds. Before launching a new network of data registries, we need to be sure that there is a clear user need for more detailed data, and that the costs of meeting this user need are not excessive.

Recommendation 3

Producers of official health statistics should consult users afresh about whether there are any specific diseases or medical conditions on which fuller statistical datasets - perhaps corresponding to the information available from cancer registries - are required and where the cost would be justified.

40. As YHEC identified in their review, comparative analysis of the countries within the UK is a key area of work for a wide range of organisations and individuals. These include public health observatories, voluntary organisations, Members of Parliament, central government departments, local government authorities and the Royal Colleges. The issue of comparative data also arises for international comparisons within the EU and the OECD. Yet there are real problems in pulling together UK-wide data in some areas - different definitions are used in different countries, and /or there are gaps in coverage in specific countries as compared to others.

41. There is a clear need for consistent UK-wide data. The question is how to generate such data when, for reasons of coverage and/or definitions, UK-wide data cannot be simply produced by pulling together existing and comparable data for the four countries. One answer would involve full harmonisation across the four countries of definitions and outputs, but this may have drawbacks if it results in breaks in time series, or if it involves imposing a breakdown that is sub-optimal when viewed from the perspective of a particular individual country. Imposing anything across the UK countries also has the potential to become a political issue in the devolution context.

42. There are differences in policies and targets for health as between the four countries of the UK, and meeting decision-makers' requirements may dictate differences in the statistics collected and produced. In such circumstances complete harmonisation across the UK of all definitions and coverage may not be possible or desirable. The solution here may instead involve finding means of making adjustments to individual country data so as to put those data on a common basis for purposes of inter-country and inter-regional comparisons as well as for compilation of total UK estimates.

43. Nevertheless there still needs to be agreement on what the common basis (definitions, coverage) for compilation of UK data should be. For this to be forthcoming, there first needs to be broad agreement between producers and users on the extent of the requirement for UK-wide data. We think that this is best achieved through a fresh assessment of the potential value of good aggregate data and of comparative data across countries and regions. Information of this kind may be of importance in relation to the allocation of public funding and its value in that context may more than justify addressing the challenges of providing it.

Recommendation 4

The Department of Health, in consultation with the devolved administrations, should address the inconsistencies in definition and availability of key health statistics and related indicators for the four countries of the UK. In doing so, a fresh assessment should be made of the potential value to researchers and decision-makers of UK aggregate data and of the value of consistent data for the four countries within the UK.

Identifying and meeting user needs

44. Health statistics for smaller geographical areas and for key administrative areas such as PCTs, as well as disaggregations by other indicators such as ethnicity and age, are required to monitor disease trends and population health, identify inequalities, target resources, compare performance, and plan services. The Statistics Commission believes that these are entirely legitimate demands and uses; indeed, many of them follow directly from government policies and priorities. Data producers should therefore do as much as possible to realise them, whilst recognising that data confidentiality considerations may on occasion preclude publication of data at the lowest levels of aggregation.

Recommendation 5

The producers of official health statistics across the UK should address more systematically the statistical requirements of the large number of organisations that need health data for particular geographical areas. They should agree steps to deliver as much relevant health data as possible for smaller geographical areas and for key administrative areas.

45. The Commission believes that having good information on the actual 'use made' of health statistics is important. Its importance can be seen when decisions about future statistical priorities and value for money are being considered. Without robust information on the practical use that is made and planned to be made of data there is no way that rational judgements on data collection priorities can be made - and there is a real risk of a decision being made to discontinue the supply of a particular stream of data in ignorance of its value.

46. In this respect, we think that it is desirable to have rather better information than is now available on 'use made' of statistics, both by non-government and government/NHS users. We think that producers are probably in the best position to take this forward, subject to certain qualifications.

47. The YHEC review identified a number of different fora, whose purpose was enabling communications between users and producers of statistics. These include the Statistics Users' Council, the Health Statistics User Group and workshops/groups run by DH and other producers. These groups might be enlisted to help DH and other producers to identify the use made outside of government and the NHS of health statistics. A different approach would be needed for gathering information on 'use made' of health statistics by government and the NHS.

Recommendation 6

The producers of official health statistics should seek, in consultation with bodies that represent user interests, to identify systematically the use made of health statistics in research and in decision-making across the UK.

48. This section sets out the background to the review carried out for the Statistics Commission by York Health Economics Consortium (YHEC). The review's conclusions and recommendations are listed in Section V.

Seminar on health statistics

49. The Commission hosted a seminar on health statistics in July 2003, which brought together a number of experts in the field, including academics and representatives from the Department of Health, the Health statistics User Group and the Audit Commission. The purpose of this seminar was to highlight some of the key issues and questions in health statistics, as a prelude to the proposed review. A report of the seminar is available on the Statistics Commission website: www.statscom.org.uk/media_pdfs/reports/Health Seminar Final Report.pdf.

The Review of Health Statistics

50. Key points from the specification of the YHEC review included:

• The review would look primarily at the statistics used in monitoring health services in the four countries of the United Kingdom. It would look at the adequacy of such data and the uses to which they are, and might, be put.
  
• The review would examine the perceptions of key users of health statistics inside and outside government, but with a particular emphasis on those users outside the NHS and Department of Health. The nature of the use people and organisations make of the data would be documented along with any concerns they have about the adequacy of the available statistics in relation to those uses.
  
• The emerging issues would then be examined and the Commission would test out its findings and make recommendations on appropriate points.

51. The review started by identifying users of health statistics. It then proceeded to examine ways to gather evidence from those users on use made by them of the statistics, and on the main issues and concerns that users had about these statistics. Interviews were conducted with 16 key users. The remaining identified users were invited to complete a questionnaire. Over 200 users responded.

52. Following consideration of the various issues that emerged from this evidence-gathering phase, five topics were selected for further investigation. These topics were: availability of disaggregated data, in particular for small areas; inter-country comparability of data; communications between producers and users; accessibility of data to users; and availability of metadata. The results are written up in the final report of the review.

Part 2: Review of Health Statistics:
report to the Statistics Commission by York Health Economics Consortium 23

Executive Summary

The Statistics Commission contracted York Health Economics Consortium (YHEC) to review the use of statistics in monitoring health services in the four countries of the UK. The main aims of the review were to:

• Identify users of health statistics;
  
• Explore the use made of health statistics;
  
• Investigate issues that users have with their use of health statistics.

To meet these aims, this review undertook the following four areas of work:

• Identification of users of health statistics;
  
• Interview and questionnaire surveys of users;
  
• Identification of emerging issues;
  
• Detailed exploration of selected issues.

The main recommendations of the review are as follows.

General Recommendations

The procedure for classifying data as National Statistics should be transparent.

Action: Government ministers and National Statistics.

Interviewees mentioned that they viewed the National Statistics classification as an indication of the quality of statistics. By ensuring that the procedure for classifying National Statistics is transparent, users will be aware of the requirements placed on National Statistics.

The voluntary status of the National Statistics Code of Practice should be reviewed.

Action: Statistics Commission and government ministers.

If adherence to the National Statistics Code of Practice became compulsory, then this would increase users' trust in statistics. A formal mechanism for monitoring compliance with National Statistics could also be established.

A list of all data that are classified as National Statistics should be readily available from a prominent location.

Action: National Statistician through the Government Statistical Service.

Our review found that a definitive list of National Statistics was not readily available. Moreover, we found that few users were able to distinguish between data that were classified as National Statistics and those that were not. By making such a list available to users, it would help to clarify what data series are classified as National Statistics and therefore, increase users' trust in these statistics by raising awareness that they are using National Statistics.

The data available on particular disease areas, such as those covered by NSFs, should be reviewed.

Action: DH and equivalent organisations in devolved administrations.

The detailed exploration of issues highlighted the differences in the coverage of statistics available for two important disease areas - Cancer and Diabetes. The Cancer registry network has meant that extensive, consistent data are collected on Cancer at a regional level. To adopt such registers for other disease areas would have implications for funding allocation. Therefore, the review should also take account of the level of demand for such data before investment in particular disease areas is undertaken.

Disaggregation

The level of disaggregation, by geographical area and indicator, for some statistics should be reviewed (e.g. PCT populations, recording ethnicity at birth/death, etc.).

Action: Producers.

A number of respondents commented that it would be useful to have data presented at a PCT level. This data would be used for analyses such as comparative studies. However, changing NHS boundaries and organisational structures may make it costly to revise data.

Future government changes in all administrative boundaries, not only those within the NHS, should take account of their impact on the availability and usability of National Statistics should be considered.

Action: Government in general, in particular the Office of the Deputy Prime Minister, the Boundary Commission and ministers.

A number of users commented that changes in administrative boundaries (e.g. the move from health authorities to PCTs) lead to difficulties in using and analysing statistics. Therefore, any future changes in boundaries should carefully consider the consequent costs in terms of their impact on National Statistics.

The implications of a move to a patient-centred system on areas such as data confidentiality should be examined.

Action: Statistics Commission.

The move to a patient-centred system was viewed positively by a number of users and producers. Indeed, such a system was already being considered by a number of producers. One of the main advantages of introducing such a system is that it allows patients to be tracked through the health care system. However, such a system may have implications for data confidentiality.

Inter-country Comparability

A review should be undertaken to identify key health areas where comparable UK statistics are not available. Recommendations could be made as to the appropriate actions to address these areas.

Action: Health and Care Theme Working Group, producers, and government ministers.

The lack of comparable data was seen as problematic by a number of users. However, ensuring that data are compiled consistently may be costly in resource terms and may also affect the future usability of data. There should be collaboration between producers to devise an agreed methodology which would allow the comparability of data series identified in this proposed review.

There should be cooperation at a European level to identify the areas for which comparable statistics can be produced. Recommendations could be made as to the appropriate actions to address these areas.

Action: Producers.

While the most immediate concern to most participants in the interview and questionnaire surveys was comparability at a UK level, the importance of consistent data at a European level is increasing. In this review, we found that such cooperation was already underway.

Communication

The list of forthcoming publications of National Statistics should be readily available in a prominent position on the Websites of producing organisations.

Action: Producers.

A number of users commented that they would find a list of forthcoming publications useful. Such lists are available from producers' Websites, in accordance with the National Statistics Code of Practice. Therefore, this suggests that some users are not aware that these lists exist. Placing these lists in more prominent positions may increase awareness.

There should be an easily accessible method for communication with producers which users can use at the time of searching/accessing statistics.

Action: Producers and user groups.

It would be useful for users to be able to contact producers at the point of using/searching for statistics. Our review found that a number of means of contacting producers were already in place. However, not all of these methods may allow users instant access to producers.

The means for communication between users and producers should be evaluated to ensure that the optimum approach is adopted.

Action: Producers and user groups.

Our review found that producers use a range of routes for establishing contact with users (e.g. relevant contact e-mail addresses on Websites, telephone numbers, e-mail alert systems). These methods should be reviewed to ensure that they satisfy users' requirements efficiently. Other potential methods should also be included in the review.

Accessibility

The format of the presentation of data, particularly the introduction of data cubes, should be considered.

Action: Producers.

Comments from more experienced users of health statistics suggest that they felt that the presentation of data was targeted towards the more inexperienced user. This proposed review would address the most appropriate format to fulfil the various users' needs. Alternative means of presenting the data should be included in this review. International evidence suggests that data cubes may be a means of allowing greater user customisation of data which may be more useful for experienced users.

A review should be undertaken to consider the structure of a central enquiry point. This would address users' queries at the point of use.

Action: Producers.

Some users muted that a single point of contact worked well in Scotland and Wales. Adopting such a central enquiry point may also work well for all health statistics. The review would address whether this centralised point would cover all countries in the UK and what health statistics would be included in its remit. This central point could also be used to monitor complaints and issues raised by users of National Statistics. This recommendation will be addressed to some extent by the establishment of a Health and Social Care Information Centre in 2005.

Metadata

A standardised template should be designed for all health data.

Action: Producers.

(Sophie Beale, Louise Carr, Jacqueline O'Reilly and Peter West, York Health Economics Consortium (Market Square, University of York, Vanbrugh Way, Heslington, York YO10 5NH;Tel: 01904 433620; Fax: 01904 433628; Email: yhec@york.ac.uk; http://www.york.ac.uk/inst/yhec/)

1.1 Background

The Statistics Commission contracted York Health Economics Consortium (YHEC) to review the use of statistics in monitoring health services in the four countries of the UK. The main aims of the review were to:

• Identify users of health statistics;
  
• Explore the use made of health statistics;
  
• Investigate any issues that users have with their use of health statistics.

The research can be divided into the following four areas of work:

• Identification of users of health statistics;
  
• Questionnaire and interview surveys of users;
  
• Identification of emerging issues;
  
• Detailed exploration of selected issues.

A wide range of health statistics are used within the four countries of the UK. There is no central source to identify all of these statistics. One of the first areas of work within the project was to identify the health statistics produced. This list concentrated on official health statistics published by government organisations. This is discussed in Sections 2 and 3.

1.2 Identification of Users of Health Statistics

The task of identifying users of health statistics was challenging. A number of approaches were used including a literature search and discussions with the Project Board.

The users of health statistics were categorised into primary and secondary users of data (see Section 2). This categorisation was based on discussions with the Project Board to highlight areas of interest to the Statistics Commission and searches by the research team to identify main organisations using health statistics where the use had an impact on policy. (Supplementary information was obtained from a survey of users of the ONS Website. This survey included a number of questions aimed at collecting information about people who used the ONS Website to access health data. The analysis of responses from the ONS survey is reported in Appendix A.)

1.3 Interview and Questionnaire Surveys

Organisations identified as a primary user were asked to participate in an interview survey, whilst secondary users were surveyed by questionnaire. The surveys identified a number of issues of concern to users of health statistics. The findings from the interviews and questionnaires are reported in Sections 4 and 5 respectively.

1.4 Identification of Emerging Issues

As noted, the research highlighted a number of issues that are of concern to users who responded to the surveys. These issues were aggregated into the following five main areas for further detailed exploration:

• Inter-country comparability;
  
• Communication;
  
• Metadata;
  
• Disaggregation;
  
• Accessibility of Data.

1.5 Detailed Exploration

For each area listed above, a series of issues/questions were formulated. These questions and issues were researched over a period of four weeks. The results of this research are detailed in Sections 7 to 11 of this report. Case studies, which investigate the five issues in two specific disease areas, are included in Section 12. Section 13 contains a summary of the conclusions for each issue and a discussion of the findings. Recommendations from the review are reported in Section 14.

1.6 Caveats

This report contains views from respondents to the interview and questionnaire surveys. It must be remembered that these views are perceptions and may not necessarily be factually correct. For example, a user may criticise the lack of data on a specific health issue. This may reflect the fact that the user is not aware that these data are in fact available.
The findings detailed in this report are based on a sample of health statistics users. However, the size and characteristics of the entire population of users is not known and, therefore, statistical significance cannot be calculated. Nonetheless, the research highlights a number of significant issues.

There is also the potential for self-selection bias within the sample. As previously stated, the organisations selected for interview were selected on the basis of being a primary user of health statistics. In contrast, the questionnaire survey was sent to a wide range of organisations and individuals. Only the more experienced and frequent users of data may have responded to the survey. A large number of individuals/organisations may use health statistics infrequently. Although these users may have issues or concerns about finding or using the data, their infrequent use may mean the importance they attach to these issues is lower than that of a more frequent user. Hence they may have been less inclined to respond to our survey.

2.1 INTRODUCTION

One of the first tasks undertaken as part of the review of health statistics was to compile a list of health statistics. The aim of the list was two-fold:

• To highlight key health statistics;
  
• To inform discussions with users of health statistics.

The list focused on official and primary sources of health statistics. Separate lists were compiled for the statistics produced by the DH in England and appropriate organisations in the devolved administrations, and the ONS.

These lists, although not exhaustive, were compared with the statistics contained in the Compendium of Health Statistics produced by the Office of Health Economics (OHE). This comparison ensured that relevant areas were included and also helped to identify main areas of health statistics.

These lists are lengthy¹ and have been summarised in Figures 2.1 and 2.2. These figures distinguish between health statistics collected as part of the assessment of public health in Figure 2.1 and those that provide an overview of NHS activity in Figure 2.2 (e.g. using immunisations or infectious disease notifications as a measure of population health but using total hernia operations as a measure of NHS activity, given the likelihood of untreated disease). The separation is not precise for NHS-collected data but can be seen as mainly delineated by the existence of registers, such that data collection is a formal goal of health services, as opposed to the routine recording of activity. Other categories of health statistics, not included in these Figures, relate to areas such as education and crime.

Figure 2.1: Statistics on Public Health

Figure 2.2: NHS Statistics (replicated for Social Services)

2.2 Need for Health Statistics

To identify the need for health statistics, it is firstly important to understand the operation and structure of health care systems. Health care systems comprise of a number of elements, including:

• Financing. This involves the collection of funds, which are used to finance the provision of health care services. In a publicly-funded system, funds are raised through taxation. In contrast, an insurance model will collect funds through insurance premia.
• Fund pooling. Once funds have been collected, they are then pooled by some central authority. The objective is to ensure that funds are allocated in an equitable manner to different groups within the population.
• Purchasing. Funds are then used to purchase health services. The decision on the type and quantity of services required will depend on the characteristics of the population.
• Provision. A number of different models can be used to provide health services. The type of organisations involved in the provision of services ranges from public and private hospitals to individual clinicians. Different payment arrangements may be involved. For example, clinicians may receive block grants to complete a particular amount of work. Alternatively, payment may be made on a fee-for-service basis.

The operation of each of these elements will impact on the efficiency and equity of the health care system - the two standard broad yardsticks used to assess a system.

Efficiency relates to both technical and allocative efficiency. The former ensures that services are provided at minimum cost. The latter adds to this the requirement that the services provided must correlate with the needs of the population. Therefore, to assess technical and allocative efficiency, data are needed on:

• The cost and efficacy of supplying various interventions and services;
  
• The health of the population.

Data on the first of these will be used in cost-effectiveness studies which ensure that the costs of interventions are justified by their benefits. These studies of economic evaluation, together with the introduction of clinical guidelines to recommend best practice, represent moves towards ensuring both types of efficiency. One such guideline is discussed in Section 2.3.

The second broad standard relates to equity of access. In an equitable system, access to health services should be similar for patients with similar conditions. Moreover, access to services would be related to need, rather than ability to pay. Consequently, systems funded through general taxation are considered more equitable than those financed through private health insurance.

Information on waiting lists will help to determine whether access is similar for patients with similar conditions. Furthermore, data on the health service workforce and the geographical location of the population may help to explain differences in access, since the latter may reflect local variations in the allocation of resources. Such local variations should be taken into account in the formula used to allocate health care funding.

Health care is the largest category of government expenditure. During 2002/3, departmental expenditure limits for health were set at £57,491 million - 24% of total government expenditure (£240,572 million).ⁱ The next largest expenditure category was local government, set at £37,598 million. With such a substantial level of expenditure on health, it is important to ensure that the health service is satisfying the targets of efficiency and equity.

Donabedian suggested that performance could be assessed in more detail on three levels:ⁱⁱ

• Structure - The quality and appropriateness of the available inputs and their organisation;
  
• Process - The quality of the delivery of care;
  
• Outputs - The ultimate quality of care.

Each of these three levels are interdependent since information on the quality of outputs alone would not indicate the reasons behind these findings, which may be related to the quality of inputs or the delivery of care.

To assess structure, it is necessary to obtain data on all inputs to the provision of health services, including staff, drugs, equipment, and physical assets, such as hospitals. Similarly, in considering process, information would be required on length of stay, the type of treatment received (e.g. rehabilitation), patient satisfaction, and hospital and patient management.

Quantifying outputs is considerably more difficult than the measurement of inputs. A number of instruments can be used to measure output, albeit imperfectly - for example, mortality, life expectancy, quality of life, etc. The main difficulty in deriving a definitive measure is that the relationship between output and the health services is not obvious since it may be influenced by a large number of exogenous factors, such as underlying patient characteristics.

The recent Atkinson Review examined the issue of the measurement of outputs in health.ⁱⁱⁱ This review found that the current method of measurement reflects 'the volume of goods and services produced as health care outputs', which are 'those which benefit or increase the welfare of recipients'. The output measure incorporated inpatient episodes and day cases, as well as visits from health visitors and district nurses. The fact that this measure is actually focused on quantifying inputs highlights the difficulty of measuring outputs. The Atkinson Review raised concerns about the quality of the current measure, particularly in relation to general practitioner services. The issue of quality was raised by interviewees in our review who queried the reliability of self-reported measures of activity, which are not directly related to the work of the individual reporting them.

Having established the general need for health statistics in assessing the achievement of goals, such as efficiency and equity of the various parts of the health care system, and monitoring performance, Section 2.3 proceeds to examine specific cases of the use made of health statistics.

2.3 USE MADE OF HEALTH STATISTICS

This section contains examples of the use made of three categories of health statistics by different organisations.

Example 1: Registers of diseases

The Eastern Region Public Health Observatory (ERPHO) publishes regular briefing papers on topical public health issues; these are based on health statistics. For instance, in May 2004 a briefing paper was written on 'Cancer incidence in the East'.^iv

This analysed data from the Cancer register for the East of England in detail, (e.g. an analysis of the annual incidence of Cancer in females by Primary Care Trusts (PCTs) was included). Other Observatories also use Cancer registers to carry out in-depth analyses of the regions they cover. However, the level of detail varies by each Observatory as the PHO work programmes are based on local areas of interest.

Cancer registries can undertake a range of public health surveillance and health protection functions. Data from the registers are used to monitor trends in Cancer and evaluate the impact of environmental and social factors on Cancer risk. For example, the ERPHO briefing paper states the ERPHO has shown an increased incidence of leukaemia in the residential areas around East Anglian waterways and has found an association with deprivation and specific Cancers.^v

Cancer registries and organisations within the NHS use data from Cancer registers to monitor the effectiveness of the existing national screening programmes for breast and cervical Cancer.

Cancer Research UK is a source of information on Cancer. They provide a free information service (CancerHelp UK) about Cancer and care for people with Cancer and their families.^vi This service utilises all health statistics available on Cancer. Other national voluntary organisations also provide similar services on different diseases areas.

Example 2: Morbidity/activity statistics - smoking cessation

Various statistics are collected on smoking. These include average daily cigarette consumption per smoker by sex and age, and statistics from services monitoring smoking cessation. Again, these statistics are used by various groups, including voluntary organisations, to highlight the potential health dangers of smoking. The media will often use the DH press releases about forthcoming statistical releases and write a summary article about the 'headline statistics' or write a more in-depth article. For example, following the press release by the DH, entitled 'Statistics on smoking cessation services in England, April 2002 to March 2003' published on 24 July 2003, most national newspapers included related articles.^vii One such article was headed 'NHS targets smashed as smokers kick the habit'.^viii

Health statistics on smoking were used in the evidence submitted to the National Institute for Clinical Excellence (NICE) for its review on Nicotine replacement therapy (NRT) and bupropion for smoking cessation. This review included a section on clinical need and practice, which incorporated a range of health statistics as shown in Box 2.1. This is a common use of health statistics as most reviews undertaken by NICE will tend to include a section which outlines current practice using a range of health statistics. When NICE release guidance, the media tend to report the more topical ones in detail such as smoking cessation and will tend to include health statistics in their articles.

Box 2.1: Example of use of health statistics

Source : Extract from NICE Guidance on the use of nicotine replacement therapy (NRT) and bupropion for smoking cessation - full guidance, www.nice.org.uk

The media frequently use health statistics in their articles. They also report on the findings of studies containing health statistics. Occasionally newspaper stories may make claims regarding new medical discoveries or may generate health scares. Such stories often have consequences for patients and may create extra pressures for health professionals who have to respond to patients' questions relating to these articles.

The National electronic Library for Health (NeLH) has commissioned the NHS Centre for Reviews and Dissemination (CRD) at the University of York, to assess the reliability of both the journalists' reporting of health stories and the research on which they are based. This project, entitled Hitting the Headlines, comprises a rapid assessment of the original research and an evaluation of the accuracy of the research findings.^ix These assessments are produced by CRD within 48 hours of the publication of the original news stories. An example of this work, relating to newspapers' reporting of a combined pill to reduce the risk of heart attacks and strokes, is contained in Appendix B.

Example 3: Surveys

Local government will use the results from surveys to inform local authorities about the health of their populations. For example, the Tees Valley Joint Strategy Unit has produced a detailed report analysing the results from the 2001 census and compared this with the previous census. A section in the report concentrates on health statistics.^x

The Manchester Joint Health Unit was established on 1st April 2002. It is funded by the three Manchester PCTs and Manchester City Council. It was set up in recognition of the fact that Manchester has some of the most challenging health problems in the country. The main aim of the Unit is to co-ordinate efforts to reduce inequalities within the city, and between the city and the rest of the country. To meet these aims the Unit makes use of health statistics, in particular small area statistics. It mainly uses data at three different levels: Manchester as a whole, PCT and Electoral ward. Within their Website the unit makes the following statement:^xi

'The level at which the statistics are presented is to a large extent dependent on the nature of the underlying data. For example, the number of events underlying some of the tables may be too small to present at a ward level without threatening the reliability of the indicator itself. In other cases, for example, cervical screening uptake, the data are based on the registered population of a GP practice, making it difficult to convert the information to a resident base, such as an electoral ward.'

The results from lifestyle surveys are reported frequently within the media and are used to influence policy. For example, the National Audit Office report, entitled Tackling Obesity in England, cites figures from the Health Survey for England (HSE) and uses these figures to make future projections about the prevalence of obesity.^xi The Health Select Committee's report on obesity also quotes figures from the HSE.^xii

Other health statistics related to such surveys are consumption and personal health statistics. These include statistics on alcohol consumption that are fed into various public health campaigns regarding sensible drinking.

3.1 Introduction

This section outlines the methods adopted in the following areas:

• The literature search;
  
• The identification of users of health statistics;
  
• The interview and questionnaire surveys;
  
• The identification of issues.

3.2 Literature Search

The aim of the literature search was to identify studies containing examples of how health statistics are used and any work on user opinions of health statistics. Supplementary to this, it was hoped that the searches would help to reveal the main users of health statistics. Users were also found either from publications, or through broader searches of the Internet and relevant organisations' Websites, as well as from directories of organisations.

It was recognised from the beginning that it would be very difficult to devise a workable search strategy. The term 'health statistics' is a very broad definition. It describes collected, summarised, and analysed data as well as the vast array of different statistical techniques that are used. A broad spectrum of published statistics, from national statistics to very specific detailed analyses, is produced. These statistics cover a range of topics about illness, disease, morbidity and mortality as well as about the health service and delivery of health care. An initial attempt to produce a list of relevant UK statistical resources only pointed out the plethora of health statistics available from numerous sources and in many different formats.

3.2.1 Search Strategy

The initial stage of the search strategy was to identify the most appropriate places to search (journal articles, books, reports, consultation documents, unpublished material). It was decided that the major medical databases should be searched as well as any UK specific health-related databases. Medline, Embase, Cinahl, Health Management Information Consortium database (HMIC), the British Nursing Index (BNI), and the grey literature database, SIGLE, were chosen.

Search terms are usually identified by looking at relevant MeSH or subject headings, looking through publications already obtained and through discussions between the information officer and the research team. In order to have a strategy that is both sensitive and precise (identifies all relevant studies without retrieving too many irrelevant studies), strategies usually entail a combination of search facets. In this case the only option was to combine 'statistics' with 'user' and 'user views/opinions'. Search terms for user, views, opinions, and feedback are as noisy (retrieve totally irrelevant references) as those for statistics. MeSH terms in Medline and indexed subject headings in the other databases for statistics were not used as they were found to be either too broad or concentrated on specific statistical techniques (e.g. cluster analysis, analysis of variance, models, etc.). There were no MeSH or subject heading terms for user views.

The searches had to be pragmatic rather than systematic and so looked to combine free text terms in close proximity. Furthermore, it was decided to use 'statistics' alone rather than additional terms such as data and survey. The searches were limited by date (2000-2004) and by studies relating to the UK. It was hoped that relevant references would be identified to help refine and improve the original search strategy. However, despite the date and place limits imposed, the searches still retrieved over 600 references. These references were inspected, and few seemed to be relevant. The retrieved results included references to statistical publications, press releases, research studies using statistics in their results, surveys about the use of health care services, social care services and libraries, guides about how to use statistics and debate about statistical techniques, etc.

The search strategy used in Medline is included in Appendix C. This strategy was translated and adapted for use in the other databases that were searched.

3.2.2 User Groups

The poor results of the database searches meant that a different approach was required. Statistical e-mail discussion groups were accessed to see if experts could help by naming any relevant publications, suggesting useful sources and giving advice. Although there was a good response, the overall picture was that little had been done, and that any relevant work would be likely to be in the form of consultation documents and feedback questionnaires: not in a form easily found in electronic databases or even on the Internet. The three discussion groups contacted were: Health Statistics User Group (HSUG), RadStats (Radical Statistics Group), and All Stats Group. Contact was also made with the Office for National Statistics (ONS), the Department of Health (DH) Statistical Division, the Association of Public Health Observatories (APHO), and the eight Regional Public Health Observatories (PHOs).

3.2.3 Internet Searches

Internet searches were undertaken. Searches were carried out using the general search engine Google and the Meta-Search engine Copernic, but without success: the search terms and any attempted combination of search terms found nothing new of interest other than the organisation Websites already known. Appendix D details the Websites that were identified and inspected.

3.2.4 Users of Health Statistics

Alongside the literature review, searches were made to compile a list of users and user groups who might influence policy. Any users identified during the literature review were included on the list. Searches of the health Internet gateway OMNI were undertaken to identify more user groups, and known user group Websites were searched for links to other user groups. Directories of organisations and associations were also searched (Directory of UK Associations, Directory of British Associations, Directory of Health Library and Information Services).

3.3 Identification of Users

The literature search confirmed that, to date, very little published work has been carried out on user views of health statistics. However, it helped to identify the list of statistics to be included in the review and gave an insight into the user groups relating to health statistics. A list of organisations using health statistics was compiled based on this information and discussions between the research team and the Statistics Commission.

Figure 3.1 indicates the breadth of organisations using health statistics. Discussions were held with the Project Board to identify those of primary interest. These discussions resulted in the inclusion of a number of categories of organisations in the review (see Appendix E).

Figure 3.1: Schematic representation of links between organisations using health statistics

From this list, discussions were held between the research team at YHEC and the Statistics Commission, to identify primary and secondary users of health statistics. The primary users were to be included in the interview survey and secondary users were to be a questionnaire survey. Further explanation of primary and secondary users can be found in Sections 3.4 and 3.5.

3.3.1 ONS Survey

The ONS commissioned a web-based survey to investigate user perceptions. The survey was designed by MORI, an independent research agency, and could be accessed via a link on the ONS Website. ONS inserted some questions within their survey to assist us with identifying users of health statistics. Unfortunately the timeframe for the ONS survey meant that it was not possible to use findings from this in the early stages of our work. The questions included in the ONS survey and associated analyses can be found in Appendix A.

3.4 The Interview and Questionnaire Surveys

In order to identify concerns of users of health statistics, two approaches were adopted:²

• An interview survey;
  
• A questionnaire survey.

3.4.1 Interview Survey

The selection criteria for potential interviewees (primary users) focused on organisations that influence policy. Moreover, our methodology recognised that the process of influencing policy may involve a chain of users of health statistics. We assumed that only one institution - a government body - may directly influence policy, although a number of other organisations, from producers of health statistics to those who analyse and/or cite the data in reports, may have an indirect impact. Therefore, in identifying potential interviewees, we strove to ensure that contributions from representatives of each stage of the user chain were involved.

In addition, we targeted key policy areas, which were defined as those areas in which National Service Frameworks (NSFs) have been developed. Further justification for our selection of these areas is that they account for a subsul>

• Cancer;
  
• Coronary heart disease;
  
• Diabetes;
  
• Mental health;
  
• Older people;
  
• Paediatric intensive care.

We recognise that some organisations may potentially influence policy in a number of different areas. Similarly, institutions may be primary users of health statistics in a particular area, but secondary users in another area.

To ensure that consistent and comparable data were collected, the interviews followed a semi-structured format and covered the following main areas:

• Frequency and purpose of use;
  
• Frequency of publication of health statistics;
  
• Quality, reliability and accessibility;
  
• Coverage;
  
• Influence on policy.

A list of the interview questions is contained in Appendix G.

Interviews were conducted by researchers from YHEC and/or the Statistics Commission. As part of the interview process, users were sent a synopsis of the issues raised during the interview, which they were asked to agree or amend as appropriate.

3.4.2 Questionnaire Survey

To capture the views and concerns of health statistic users from a broad range of organisations, a questionnaire was circulated to a large sample of potential users.

Content

The questionnaire was designed in conjunction with the Project Board and gave recipients an opportunity to contribute to this research project. Due to the fact that the number of health statistics published is extremely large it was decided to create a questionnaire that tackled broad issues. A copy of the questionnaire and accompanying letter or e-mail can be found in Appendix H. Section 1 of the questionnaire asked general questions that gave background to how, where and why health statistics are used, whilst Section 2 concentrated on views about quality aspects of health statistics. Included within the second section was a question that asked whether the respondent felt that their use of health statistics influenced policy, either directly or indirectly. Questionnaire recipients also had the opportunity to express views on issues not directly tackled by the questions within the questionnaire.

Circulation

We aimed to target users of health statistics from a wide range of organisations. Recipients fell broadly into six categories:

• Management within the National Health Service (NHS);
  
• Government bodies - central and local;
  
• Political Parties;
  
• Think Tanks;
  
• Educational Bodies (including the Royal Colleges);
  
• Statistics User Groups.

The structured methodology used to compile the list of recipients in NHS management and a list of names of other recipients is included in Appendix H.

Questionnaires were circulated either as an e-mail attachment (in a Microsoft Word document), by surface mail, or else recipients were e-mailed details of how to access the questionnaire via the web. We also set up an AdWords Ad on the search engine Google in order to direct other potential users of health statistics to the online questionnaire. Upon entering pre-identified search terms into Google, an AdWords Ad appeared on the right hand side of the screen that displays the search results. This advertisement encouraged potential respondents to complete an online version of the questionnaire and directed them to the relevant site.

Most questionnaires were sent out as an e-mail attachment. Wherever possible e-mails were addressed to named recipients. In cases where this was not possible, e-mails were directed to the information, enquiries or help desk as applicable to that organisation. This primary recipient was asked to forward the e-mail to the appropriate recipient or recipients within their organisation.

Surface mail was used where no e-mail address was available. It was generally possible to send these letters to named individuals.

The DH in England and equivalent organisations in Northern Ireland, Scotland and Wales put a summary of the project and a link to the questionnaire on their Websites. The link was circulated to co-ordinators of statistics user groups who distributed it to their members. A link was also sent to the Society of Social Medicine (SSM) who, as well as placing it on their Website, also incorporated the summary and link in their Newsletter. Thus each link will potentially have been seen by a number of users of health statistics.

Over 900 questionnaires were sent out, giving many people the opportunity to share their views and issues pertaining to health statistics. The breakdown of the questionnaire delivery format is shown in Table 3.1.

Table 3.1: Breakdown of questionnaire delivery format

Note: ^a Each link will potentially have reached a number of health statistics users.

3.5 Identification of Issues

The responses from the interview and questionnaire survey were analysed in detail. As part of this analysis, the issues identified by the users were summarised and aggregated into relevant categories. This is discussed in Section 6.

4.1 Sample

In total, 28 organisations were invited to be interviewed. Sixteen interviews were carried out. The characteristics of the organisations, from which representatives were interviewed, are reported in Table 4.1. The sample consisted of a wide range of organisations from across the UK.

Table 4.1: Characteristics of organisations

Note: ^a Covers England and Wales.

Among this sample, the frequency of the use of health statistics varied from quarterly to daily. The following are examples of the purposes of interviewees' use of health statistics:

• To produce reports, press releases and fact sheets;
  
• To inform the public and the media;
  
• To support campaigns and lobbying;
  
• To create risk factor simulations;
  
• To assess government targets;
  
• To produce trends;
  
• To investigate current issues;
  
• To conduct comparative analysis;
  
• To develop policies and policy responses;
  
• To construct models;
  
• To study the workforce;
  
• To examine the factors influencing delayed discharges;
  
• To customise local data for use by others;
  
• To produce health technology assessments;
  
• To support local governments and local authorities;
  
• To calculate deprivation measures and estimate populations;
  
• To monitor health trends and provide advice.

Table 4.2 shows the types of organisations who were not interviewed.

Table 4.2: Characteristics of remaining organisations not interviewed

4.2 Findings

This section summarises the main points raised by users during the interviews. Comments have been classified into the following categories:

• Level of coverage;
  
• Level of detail;
  
• Frequency of publication;
  
• Time lag between collection and publication;
  
• Communication;
  
• Compilation;
  
• Quality and accuracy;
  
• Accessibility;
  
• Other.

4.2.1 Level of Coverage

Almost all interviewees identified areas where there was a lack of data. A number of interviewees recognised that the production of health statistics was determined by government policies and performance targets. This was confirmed by producers who acknowledged that the production of health statistics was mainly driven by the demands of government priorities. However, one of these producers further commented that as a result of these changing priorities, a number of gaps had been identified (e.g. data missing on children's services and mental health). One user also recognised data series that would be required to monitor policy, but were not currently available (e.g. data to support the Government's agendas on poverty reduction and health inequalities).

The main areas where statistics were missing are reported in Table 4.3. The small numbers reported in this Table reflect the relatively small sample of organisations that participated in interviews and their divergent demands for additional data.

In the absence of data from official sources, one interviewee commented that the relevant information had to be derived from alternative sources, which could introduce calculation errors. Similarly, another user raised comparable concerns about the implications for reliability of using alternative data sources. As one interviewee pointed out, using data from alternative sources may be inappropriate because these data may not be generalisable. Another organisation remarked that as a consequence of official data, particular statistics were calculated by an external, non-government institution. One user criticised the existing data relating to the NHS as 'abstract' and argued that qualitative data would help to explain trends and inform patient choice.

In contrast to the views about insufficient data reported in Table 4.3, one participant commented that new data seem 'un-balanced'. This interviewee felt that there was a significant amount of data relating to mental health being generated and that these were rarely used in their experience. If this were the case, it may suggest that there was scope to optimise the use of statistics by reallocating resources from areas where statistics are less frequently used to those where gaps have been identified.

Table 4.3: Gaps

Notes:

^* Interviewees may have identified more that one area.
^aThis interviewee was particularly interested in the hospital readmission rates for people aged 75 years or over, which could be used to assess early discharge and subsequent readmissions. This data series was discontinued because, according to this interviewee, the DH considered it to be ageist. However, the interviewee argued that this argument could be overcome by producing this information for all adults by age band.
^b This interviewee suggested that a registry for diabetes, similar to the one for Cancer, would help to identify disease prevalence.
^c There have been recent moves towards providing this information.
^dThis interviewee stated that local authorities currently produced registries of people with disabilities, but these were not updated.
^e This interviewee suggested that a survey of these core indicators could be undertaken annually to help with trends.
^fThis interviewee emphasised sensitive personal areas such as sexual orientation and political opinion.
^g This interviewee stated that additional data on care in private hospitals, and in local authority and private care homes, and care given in the community are required to provide data on care after miscarriages, and antenatal and postnatal care. Currently, these data are only provided if care was given in a hospital setting.
^h This instrument was used to collect information on hospital expenditure by function, but was abolished.
ⁱ This interviewee proposed that primary care systems and the new GMS contracts provide an opportunity to collect data at a local level on such risk factors.

4.2.2 Level of Detail

A number of users mentioned that it would be useful if data were presented on a disaggregated level. Where detailed data were not available, some of these organisations mentioned that they contacted data providers directly with specific requests for this information. A number of these participants acknowledged that detailed data may result in confidentiality issues. In addition, statistical analysis of these data might become increasingly problematic if the resulting samples were small. One interviewee noted that although they may request small area data, these may not always be provided because they may contravene the Caldicot Principles.

Specific areas, mentioned by interviewees, where disaggregated data would be useful are reported in Table 4.4.

Table 4.4: Disaggregated data