Report No 21: Enhancing the value of health statistics: User perspectives

This report is the first by the Statistics Commission to take a broad look at one entire field of official statistics - in this case health statistics. It was prepared under the guidance of a project board chaired by commissioner Sir Kenneth Calman and with the support of York Health Economics Consortium who carried out a detailed review on behalf of the Commission.

The decision to look first at health reflects the central importance in all our lives of messages drawn from health data. These messages are so pervasive that we tend to forget they are based on the analysis of statistical data. Many people will not have seen the tables of figures but we all know roughly how long we are expected to live, that smoking and excessive alcohol will shorten that period, that heart disease or cancer will kill most of us eventually, that you have to wait for much hospital treatment - and in some cases for longer than anyone would wish. These messages, and many others, come from analysis of statistical data and their validity is directly dependent on reliable statistics being collected and properly analysed.

A moment's thought points to the importance of health data in other walks of life too, in setting life insurance premiums, ensuring the safety of our food, in government advice on diet and obesity, measures to protect sexual health and prevent drug abuse, in the design of our buildings and cars (no asbestos in the first; air-bags in the second), the identification of the most deprived parts of the country, the need for social services and care for the elderly. Less familiar, but equally important, is the role of health data in government formulae for allocating vast sums of public money to different areas of the country.

The Statistics Commission approaches all statistical issues from the perspective of those who use, or may one day need to use, the figures and the messages from them in decision-making. In fields such as health this includes us all to some extent. We decide whether to smoke, what to eat, when to seek medical advice, how much exercise to take.

This report, building on the one prepared by York Health Economics Consortium, recognises the great wealth of health data now available, or planned, and does not attempt to focus on the occasional gaps or flaws in this rich fabric - that can safely be left to others.

Our approach has rather been to identify a small number of high-level proposals that will enhance the capacity of health statistics to support decision-making and the wider user community. In the event, our proposals have relevance beyond the field of health statistics and raise a number of issues of importance for official statistics more generally. We believe our recommendations are balanced and achievable and will prove good value for money when the costs and benefits are fully assessed. We will be seeking in the coming months to establish with all the relevant government departments and agencies the best way of taking them forward.

I thank all those who contributed to this important exercise.

Introduction

1. This report by the Statistics Commission forms Part One of Statistics Commission Report No. 21. It looks at UK official statistics relating to health, and builds on a review carried out for the Commission by York Health Economics Consortium (YHEC)². A summary of YHEC's conclusions and recommendations is included at Section V. The full report from YHEC follows as Part Two of this volume.

2. There is now a great wealth of health statistics available and the focus of this report is not on adding to that infrastructure so much as on ways to enhance the value of the statistics to decision-makers. The decisions in question include those taken in the context of managing health services but extend much further. At the most fundamental level, health statistics underpin society's understanding of what is good for us and what is not - and so inform advice about diet and smoking, life insurance premiums, the laws on drug abuse and so on. Health data also indicate which parts of the country are most deprived and in greatest need of services - and so inform the allocation of resources to local authorities, social services and the targeting of national schemes.

3. Thousands of organisations from GP practices to national institutions, university research departments, drug companies, insurance companies, charities and of course NHS management, draw regularly on statistical information either in the form of data or as messages taken from analysis of the data. Its importance in shaping the institutions and values of society cannot be overstated.

The Health Service context

4. In the context of health service management, the scale of decision-making is vast. Public expenditure on health services this year will be of the order of £80 billion and is rising rapidly. This equates to some £1,350 annually for every man, woman and child in the UK. The NHS, government, Parliament and the citizen all want to know whether this money is being spent as effectively as possible, whether the advice and treatment provided by the NHS is as good as possible, and whether services are as good where we live as they are elsewhere in the UK, and elsewhere in the world.

5. We live in an age of audit and accountability. Every important decision is now expected to be supported by evidence and be subject to review and public scrutiny. This applies at all levels of health service management from the GP deciding on a diagnosis to government deciding on the allocation of funding, the price of prescriptions, or the salaries of doctors and nurses. Official statistics of many different kinds form the bedrock of the evidence base. Although their collection undoubtedly places an additional, and at times unwelcome, burden on public services - not least the staff of the NHS - they are the key to informed decision-making and public accountability.

6. To illustrate the importance of statistics in NHS management, it is instructive to look at the way the NHS Improvement Plan, published in June 2004, describes progress since 1997. It states that:

• "The maximum waiting time for an operation has fallen from 18 months to less than nine months.
  
  
• The maximum waiting time for an outpatient appointment has fallen from 26 weeks to 17 weeks.
  
  
• 97% of patients are now able to see a GP within two days. Growing numbers of patients are taking advantage of new services such as NHS Direct and NHS Walk-in Centres.
  
  
• 94% of patients are seen, diagnosed and treated within four hours of arrival at accident and emergency.
  
  
• Increases in staff numbers since 1997 are also improving services for patients. They include a 22% increase in doctors, a 21% rise in the total number of nurses and a 27% expansion in scientific, therapeutic and technical staff.

· In national surveys patients are increasingly positive about the quality of their care. Beyond the clinical arena, there has been progress on issues where patients have asked for more focus, with cleaner hospitals, better food and better provision of bedside phones and televisions."

7. Every one of these statements rests on the collection and analysis of statistics and could not have been made without that analysis. And looking to the future, the NHS Improvement Plan draws on further statistical indicators, for example:

• "A maximum of 18 weeks from referral by their GP".
  
  
• "A 40% fall from 1997 in death rates from heart disease and stroke".

8. So the critical role of statistics in shaping and steering decisions inside and outside the health services is beyond question. The aim must therefore be to take all reasonable steps to enhance their value. One focus of the YHEC review was to find out more about the non-government users of health statistics - who they are, what they use the data for, and what changes might be made to better meet their needs.

9. The review sought to identify the main users of health statistics, and their views through interviews and questionnaires. A number of themes emerged from this evidence-gathering process, some of which were subsequently pursued in more depth. The Statistics Commission has drawn heavily on the YHEC report, and its conclusions and recommendations. We believe that the review has identified scope for actions in a number of areas, which could improve the usefulness and accessibility of the statistics. Based on the review, the Commission has developed a set of six recommendations that we believe should be priorities for producers of health statistics. The background to the Commission's recommendations is described in the following section of this report. The recommendations themselves are set out in full in Section III.

²Information about a statistical series that enables the user to gauge quality, context, comparability, limitations etc.

Background to Commission recommendations

10. The Commission's recommendations are directed at the government departments, administrations and agencies that produce official health statistics - the Department of Health and its agencies, the Office for National Statistics, and the health and statistical departments of the devolved administrations. All the Commission's recommendations derive from, and are consistent with, the YHEC review conclusions and recommendations.

Use made of health statistics

11. A better understanding of the use made of health statistics was a prime concern for the review. YHEC collected some information on use made through questionnaires and interviews. For the most part the information collected was broad-brush in nature; for example, nearly 60 per cent of respondents said that their main use of health statistics was for 'research' or 'in reports'. The review report does however provide some tangible examples of the use made of specific statistics. These examples include the use of cancer registry data by regional health observatories, the use of morbidity data for 'stop smoking' campaigns and the use of health survey data by local authorities.

12. It proved more difficult to obtain robust information about which data ultimately informed which kinds of decisions. Whilst we can speculate with varying degrees of confidence about this, it would be valuable to all those concerned with official statistics to have sound evidence and we believe that producers of statistics need to know more about the way statistics are used in practice (see Recommendation 6).

Production of health statistics - structural issues

13. Official health statistics are produced and published by the Department of Health (DH), by the Office for National Statistics (ONS) and by the health departments of the three devolved administrations. ONS produces a range of statistics relating to births, deaths and public health from surveys and from NHS and other administrative sources. Some of these statistics have UK-wide coverage; others relate to England and Wales only. Other statistics produced by DH relate to England only. The equivalent statistics for the other countries of the UK are produced by the devolved administrations. This reflects the fact that provision of health services is a devolved function of government but tends to ignore the potential user requirement for consistent data across the UK.

14. The multiplicity of UK organisations producing health statistics is one of the factors that have led to difficulties in compiling figures that are consistent across the four countries. Users who want UK figures, or who want to make comparisons across countries, will often need to go to a number of different sources in order to access data - which is inconvenient at best. Worse, many figures are not directly comparable across countries because of differences in definitions and/or coverage.

15. The Department of Health has recently announced the creation of a Health and Social Care Information Centre (see Reconfiguring the Department of Health's Arm's Length Bodies, 22 July 2004, and accompanying DH press release). The intention is that the new agency will be established as a Special Health Authority from 1 April 2005. The Centre will merge the statistics and information policy functions of DH with some of the 'information' related functions of the NHS Information Agency. The aim is to "'reduce burdens on the frontline by co-ordinating information requirements across a wide range of bodies" (DH press release 22 July 2004).

16. By itself, the creation of the Health and Social Care Information Centre will not reduce the multiplicity of organisations in the UK producing official health statistics. Nevertheless the Commission believes that the new Information Centre will be in a good position - probably a better one than DH is now - to take on the enhanced central co-ordinating role that the Commission believes is required in order to improve coherence and comparability of health data across the entirety of the UK. For example, we believe that the new Information Centre will be well placed to develop and manage the online index of UK heath statistics and associated metadata that we propose in Recommendations 1 and 2. The Commission recognises that the Information Centre will only be able to do this with the full co-operation of the devolved administrations and of ONS.

Disaggregation versus confidentiality

17. The YHEC review identified a demand for more disaggregated data, both by geographical area and by indicator, in several different areas of health statistics. The Commission believes that producers of health statistics should make every effort to satisfy these demands (Recommendation 5). However it needs to be recognised that there is a potential conflict between, on the one hand, the production of more disaggregated data, and, on the other hand, data confidentiality and data protection issues that argue against the release of individual estimates derived from a small number of observations.

18. The Commission recognises the need to protect data confidentiality. It is a key principle of the National Statistics Code of Practice that "no statistics will be produced that are likely to identify an individual unless specifically agreed with them". This is consistent with the Caldicott Principles that govern, and restrict, the dissemination of patient-identifiable information within the NHS.

19. In some circumstances, eg the release of detailed disaggregated data for research purposes, the principles regarding data confidentiality and protection can be met by release of data in a suitably anonymised format. But, as well as guarding against releasing information that directly identifies individuals, it is also necessary to protect against both inferential disclosure - where information about an individual can be inferred with a high degree of confidence - and inadvertent disclosure, eg as a result of differencing of two datasets with largely but not completely overlapping populations.

20. Nevertheless the Commission is concerned that there is a risk of data confidentiality being cited as a reason for withholding disaggregated data in circumstances that do not justify it. We think that decisions as to the level of disaggregation below which data cannot be released need to be based on actual confidentiality issues regarding each individual dataset, and not on arbitrary rules that suppress all data derived from less than a specific (arbitrary) minimum number of observations. (The YHEC review reports that such a rule of thumb - the 'no less than 5 in a cell' rule - appears to be in quite wide use in the health statistics area.)

21. We believe that such blanket bans are bound to lead to suppression of more data than is necessary to meet the principles of the Code of Practice, and are best avoided. As a general rule, we would urge data producers to look for ways of releasing as much data as possible.

Performance measurement

22. Changes in the organisation and management of public services are creating new demands for statistics that can enable monitoring of activity and performance at a local level. This has been particularly the case in the health service (for example at the level of Primary Care Trusts and within PCTs). This was noted by a number of contributors to the YHEC review.

23. Given finite resources, there is always a risk that meeting a growing demand for a particular kind of information will impact on the supply and availability of data for other purposes. The YHEC review found, in the responses to interviews and questionnaires, some who argued this was occurring. The review report notes that implementation of performance targets was perceived to have had "a detrimental input on the quantity of data available for areas that are not subject to such targets".

24. It is difficult to comment on assertions by review respondents that were not backed with any clear evidence. It is certainly the case that more performance information is being collected than hitherto, but it does not follow that other statistics will necessarily suffer.

25. This is another area where the new Health and Social Care Information Centre should be in a good position - for England at least (and more widely, we hope) - to take an overview, recognising and correcting any inefficiencies or duplications in data collection, taking and publicising decisions on priorities, and ensuring that full use is made of all the information collected. In doing this, the Commission would urge the Information Centre to take into account the needs of users of health statistics outside government and the NHS, as well as of those within.

Costs and burdens

26. There are often costs associated with making improvements to data collection and analysis. Meeting the recommendations of the review and of the Commission report is likely to require additional resources, both financial and human.

27. The Commission recognises that a strong value-for-money case needs to be made for any new investment in the statistical infrastructure. This is why our Recommendations 3 and 4 call for 'fresh assessments' of the value to users of statistics before any resource commitment is made. We believe that, where it can be demonstrated that there is a clear case for improved data, the costs of those improvements are very likely to be more than offset in the longer term by the savings flowing from better decision-making and enhanced public confidence in the decision-makers.

28. It also needs to be recognised that new demands for information are likely to place an additional burden on data providers. An important aspect of controlling the burden on data providers is to ensure that information is collected and used in as efficient a way as possible, and in particular that the same information is only collected once. In this respect the Commission notes, and welcomes, the proposed focus for the new Health and Social Care Information Centre on co-ordinating information requirements, with the aim of better managing the burden on data providers.

Issues and recommendations

29. The following paragraphs set out the Statistics Commission's recommendations and explain the context in which they have been made. Recommendations are made in three broad areas:

a. supporting user access to data and analyses
b. consistency and coherence of data
c. identifying and meeting user needs.

Supporting user access to data and analysis

30. Accessibility of information is crucial for all users of statistics. This was an issue raised by a number of respondents to YHEC's interview and questionnaire enquiries. A common theme was the inconvenience of having to search multiple sources, which can be both complex and time-consuming, and result in important data not being found. Users suggested that a centralised web-based repository could greatly simplify the process of obtaining data. Such 'clearing houses' are now commonplace in some other fields and countries.

31. The Statistics Commission endorses this idea, but questions remain as to who should do it, and how. As regards how, we think that the objective should be to provide all users with a web portal that would be a first point of reference for information on, and enquiries about, statistics on all aspects of health and care relating to all countries of the UK. As regards who, we believe that the new Health and Social Care Information Centre should be well placed to take on the co-ordination function, though it would need the active and ongoing co-operation of other producing bodies - notably ONS and the devolved administrations.

32. Putting this web-based repository in place will entail collecting information, including web locations, on all health-related statistics from all the various sources, and then constructing an Index which can guide the user to the appropriate statistics. Figures 2.1 and 2.2 in the YHEC report could assist with this and offer a good example of how such an index could be 'mapped'. The Index might also indicate which of these series and reports have been produced to the standards of the existing National Statistics Code of Practice and which have not.

33. Whilst we are currently quite a way from having anything like a comprehensive repository of this kind, there are a couple of features of existing websites that the developers of the proposed Index could build on. The Health and Care pages of the ONS website - National Statistics Online - provide a web portal to a wide selection of health and care statistics published by ONS, some of which have a UK-wide coverage. And the existing DH website has an Index of Useful Links, which includes links to data series on the ONS website and on the websites of the devolved administrations, as well as elsewhere on the DH site.

34. These two features of existing websites could provide a starting point for the Index that we are proposing, but quite a bit of further development would be required. The coverage in the ONS pages is for the most part restricted to statistics published by ONS, including series published in statistical compendia such as Social Trends. And whilst the current DH Index has good links to DH data and to health data on the ONS website, the links to data held by the devolved administrations are at present very limited. It is also not in a very prominent place on the site, and certainly not located where it would be a 'first port of call'.

Recommendation 1

UK producers of official health statistics should give high priority to developing an easy to use and up-to-date online Index to available health statistics, and statistical reports, covering all four countries of the UK. Wherever possible the Index should guide the user directly to the figures and supply relevant contextual information and advice on interpretation (see Recommendation 2).The new Health and Social Care Information Centre will be well placed to lead on this development, with the agreement and co-operation of other producers.

35. A crucial component of an on-line Index would be metadata³. YHEC found that user perceptions of current metadata for health statistics were generally quite good. Nevertheless the format and content varied by type of publication and producing organisation. With implementation of Recommendation 1 above, it should become easier - but at the same time essential - to adopt a common approach that ensures the presentation of consistent metadata. The keepers of the online Index will have a key role in establishing and co-ordinating metadata standards with all UK producers of statistics.

36. A standard template would ensure that the data source attaches information such as quality indicators and limitations of use in a consistent format. If this information cannot be provided then the keepers of the online Index will need as much background information regarding the methods used to gather the data as is available, so as to help them make their own judgements in support of users.

Recommendation 2

UK producers of official statistics should work together to produce metadata in a consistent format. This metadata should be accessible from the Index proposed in Recommendation 1.

Consistency and coherence

37. A case study described in the YHEC report noted substantial differences in the availability of detailed data for specific diseases. Data on diabetes are produced for England and Wales, disaggregated to a regional level only, whereas cancer data are available disaggregated in a number of different ways - by Strategic Health Authority (SHA), by Primary Care Trust (PCT) - or equivalent in Scotland - or by postcode. The greater availability of detailed statistics for cancer is a reflection of the much larger amounts of consistent and comparable data available from the well-established network of regional cancer registries. There is no equivalent network of regional registries for diabetes, or for most other diseases and medical conditions.

38. The existence of a network of regional cancer registries, and the consequential differences in quality of detailed data for cancer as compared with diabetes, is at least in part a consequence of the higher level of funding available for data collection for cancer, reflecting past priorities. If more detailed analyses of diabetes (or of other diseases/medical conditions for which no comprehensive network of registries exists) are required, then an obvious solution is improved source data through the development of a regional registry approach along the lines of the registries for cancer.

39. The Commission believes that there could be substantial benefits from the development of networks of regional registries for more diseases and medical conditions, but accepts that this could be expensive. We therefore propose that the costs and benefits from development of regional registries for diabetes and other diseases should be further investigated, with a view to taking a decision as to whether this would be a justifiable use of public funds. Before launching a new network of data registries, we need to be sure that there is a clear user need for more detailed data, and that the costs of meeting this user need are not excessive.

Recommendation 3

Producers of official health statistics should consult users afresh about whether there are any specific diseases or medical conditions on which fuller statistical datasets - perhaps corresponding to the information available from cancer registries - are required and where the cost would be justified.

40. As YHEC identified in their review, comparative analysis of the countries within the UK is a key area of work for a wide range of organisations and individuals. These include public health observatories, voluntary organisations, Members of Parliament, central government departments, local government authorities and the Royal Colleges. The issue of comparative data also arises for international comparisons within the EU and the OECD. Yet there are real problems in pulling together UK-wide data in some areas - different definitions are used in different countries, and /or there are gaps in coverage in specific countries as compared to others.

41. There is a clear need for consistent UK-wide data. The question is how to generate such data when, for reasons of coverage and/or definitions, UK-wide data cannot be simply produced by pulling together existing and comparable data for the four countries. One answer would involve full harmonisation across the four countries of definitions and outputs, but this may have drawbacks if it results in breaks in time series, or if it involves imposing a breakdown that is sub-optimal when viewed from the perspective of a particular individual country. Imposing anything across the UK countries also has the potential to become a political issue in the devolution context.

42. There are differences in policies and targets for health as between the four countries of the UK, and meeting decision-makers' requirements may dictate differences in the statistics collected and produced. In such circumstances complete harmonisation across the UK of all definitions and coverage may not be possible or desirable. The solution here may instead involve finding means of making adjustments to individual country data so as to put those data on a common basis for purposes of inter-country and inter-regional comparisons as well as for compilation of total UK estimates.

43. Nevertheless there still needs to be agreement on what the common basis (definitions, coverage) for compilation of UK data should be. For this to be forthcoming, there first needs to be broad agreement between producers and users on the extent of the requirement for UK-wide data. We think that this is best achieved through a fresh assessment of the potential value of good aggregate data and of comparative data across countries and regions. Information of this kind may be of importance in relation to the allocation of public funding and its value in that context may more than justify addressing the challenges of providing it.

Recommendation 4

The Department of Health, in consultation with the devolved administrations, should address the inconsistencies in definition and availability of key health statistics and related indicators for the four countries of the UK. In doing so, a fresh assessment should be made of the potential value to researchers and decision-makers of UK aggregate data and of the value of consistent data for the four countries within the UK.

Identifying and meeting user needs

44. Health statistics for smaller geographical areas and for key administrative areas such as PCTs, as well as disaggregations by other indicators such as ethnicity and age, are required to monitor disease trends and population health, identify inequalities, target resources, compare performance, and plan services. The Statistics Commission believes that these are entirely legitimate demands and uses; indeed, many of them follow directly from government policies and priorities. Data producers should therefore do as much as possible to realise them, whilst recognising that data confidentiality considerations may on occasion preclude publication of data at the lowest levels of aggregation.

Recommendation 5

The producers of official health statistics across the UK should address more systematically the statistical requirements of the large number of organisations that need health data for particular geographical areas. They should agree steps to deliver as much relevant health data as possible for smaller geographical areas and for key administrative areas.

45. The Commission believes that having good information on the actual 'use made' of health statistics is important. Its importance can be seen when decisions about future statistical priorities and value for money are being considered. Without robust information on the practical use that is made and planned to be made of data there is no way that rational judgements on data collection priorities can be made - and there is a real risk of a decision being made to discontinue the supply of a particular stream of data in ignorance of its value.

46. In this respect, we think that it is desirable to have rather better information than is now available on 'use made' of statistics, both by non-government and government/NHS users. We think that producers are probably in the best position to take this forward, subject to certain qualifications.

47. The YHEC review identified a number of different fora, whose purpose was enabling communications between users and producers of statistics. These include the Statistics Users' Council, the Health Statistics User Group and workshops/groups run by DH and other producers. These groups might be enlisted to help DH and other producers to identify the use made outside of government and the NHS of health statistics. A different approach would be needed for gathering information on 'use made' of health statistics by government and the NHS.

Recommendation 6

The producers of official health statistics should seek, in consultation with bodies that represent user interests, to identify systematically the use made of health statistics in research and in decision-making across the UK.

48. This section sets out the background to the review carried out for the Statistics Commission by York Health Economics Consortium (YHEC). The review's conclusions and recommendations are listed in Section V.

Seminar on health statistics

49. The Commission hosted a seminar on health statistics in July 2003, which brought together a number of experts in the field, including academics and representatives from the Department of Health, the Health statistics User Group and the Audit Commission. The purpose of this seminar was to highlight some of the key issues and questions in health statistics, as a prelude to the proposed review. A report of the seminar is available on the Statistics Commission website: www.statscom.org.uk/media_pdfs/reports/Health Seminar Final Report.pdf.

The Review of Health Statistics

50. Key points from the specification of the YHEC review included:

• The review would look primarily at the statistics used in monitoring health services in the four countries of the United Kingdom. It would look at the adequacy of such data and the uses to which they are, and might, be put.
  
• The review would examine the perceptions of key users of health statistics inside and outside government, but with a particular emphasis on those users outside the NHS and Department of Health. The nature of the use people and organisations make of the data would be documented along with any concerns they have about the adequacy of the available statistics in relation to those uses.
  
• The emerging issues would then be examined and the Commission would test out its findings and make recommendations on appropriate points.

51. The review started by identifying users of health statistics. It then proceeded to examine ways to gather evidence from those users on use made by them of the statistics, and on the main issues and concerns that users had about these statistics. Interviews were conducted with 16 key users. The remaining identified users were invited to complete a questionnaire. Over 200 users responded.

52. Following consideration of the various issues that emerged from this evidence-gathering phase, five topics were selected for further investigation. These topics were: availability of disaggregated data, in particular for small areas; inter-country comparability of data; communications between producers and users; accessibility of data to users; and availability of metadata. The results are written up in the final report of the review.

Part 2: Review of Health Statistics:
report to the Statistics Commission by York Health Economics Consortium 23

Executive Summary

The Statistics Commission contracted York Health Economics Consortium (YHEC) to review the use of statistics in monitoring health services in the four countries of the UK. The main aims of the review were to:

• Identify users of health statistics;
  
• Explore the use made of health statistics;
  
• Investigate issues that users have with their use of health statistics.

To meet these aims, this review undertook the following four areas of work:

• Identification of users of health statistics;
  
• Interview and questionnaire surveys of users;
  
• Identification of emerging issues;
  
• Detailed exploration of selected issues.

The main recommendations of the review are as follows.

General Recommendations

The procedure for classifying data as National Statistics should be transparent.

Action: Government ministers and National Statistics.

Interviewees mentioned that they viewed the National Statistics classification as an indication of the quality of statistics. By ensuring that the procedure for classifying National Statistics is transparent, users will be aware of the requirements placed on National Statistics.

The voluntary status of the National Statistics Code of Practice should be reviewed.

Action: Statistics Commission and government ministers.

If adherence to the National Statistics Code of Practice became compulsory, then this would increase users' trust in statistics. A formal mechanism for monitoring compliance with National Statistics could also be established.

A list of all data that are classified as National Statistics should be readily available from a prominent location.

Action: National Statistician through the Government Statistical Service.

Our review found that a definitive list of National Statistics was not readily available. Moreover, we found that few users were able to distinguish between data that were classified as National Statistics and those that were not. By making such a list available to users, it would help to clarify what data series are classified as National Statistics and therefore, increase users' trust in these statistics by raising awareness that they are using National Statistics.

The data available on particular disease areas, such as those covered by NSFs, should be reviewed.

Action: DH and equivalent organisations in devolved administrations.

The detailed exploration of issues highlighted the differences in the coverage of statistics available for two important disease areas - Cancer and Diabetes. The Cancer registry network has meant that extensive, consistent data are collected on Cancer at a regional level. To adopt such registers for other disease areas would have implications for funding allocation. Therefore, the review should also take account of the level of demand for such data before investment in particular disease areas is undertaken.

Disaggregation

The level of disaggregation, by geographical area and indicator, for some statistics should be reviewed (e.g. PCT populations, recording ethnicity at birth/death, etc.).

Action: Producers.

A number of respondents commented that it would be useful to have data presented at a PCT level. This data would be used for analyses such as comparative studies. However, changing NHS boundaries and organisational structures may make it costly to revise data.

Future government changes in all administrative boundaries, not only those within the NHS, should take account of their impact on the availability and usability of National Statistics should be considered.

Action: Government in general, in particular the Office of the Deputy Prime Minister, the Boundary Commission and ministers.

A number of users commented that changes in administrative boundaries (e.g. the move from health authorities to PCTs) lead to difficulties in using and analysing statistics. Therefore, any future changes in boundaries should carefully consider the consequent costs in terms of their impact on National Statistics.

The implications of a move to a patient-centred system on areas such as data confidentiality should be examined.

Action: Statistics Commission.

The move to a patient-centred system was viewed positively by a number of users and producers. Indeed, such a system was already being considered by a number of producers. One of the main advantages of introducing such a system is that it allows patients to be tracked through the health care system. However, such a system may have implications for data confidentiality.

Inter-country Comparability

A review should be undertaken to identify key health areas where comparable UK statistics are not available. Recommendations could be made as to the appropriate actions to address these areas.

Action: Health and Care Theme Working Group, producers, and government ministers.

The lack of comparable data was seen as problematic by a number of users. However, ensuring that data are compiled consistently may be costly in resource terms and may also affect the future usability of data. There should be collaboration between producers to devise an agreed methodology which would allow the comparability of data series identified in this proposed review.

There should be cooperation at a European level to identify the areas for which comparable statistics can be produced. Recommendations could be made as to the appropriate actions to address these areas.

Action: Producers.

While the most immediate concern to most participants in the interview and questionnaire surveys was comparability at a UK level, the importance of consistent data at a European level is increasing. In this review, we found that such cooperation was already underway.

Communication

The list of forthcoming publications of National Statistics should be readily available in a prominent position on the Websites of producing organisations.

Action: Producers.

A number of users commented that they would find a list of forthcoming publications useful. Such lists are available from producers' Websites, in accordance with the National Statistics Code of Practice. Therefore, this suggests that some users are not aware that these lists exist. Placing these lists in more prominent positions may increase awareness.

There should be an easily accessible method for communication with producers which users can use at the time of searching/accessing statistics.

Action: Producers and user groups.

It would be useful for users to be able to contact producers at the point of using/searching for statistics. Our review found that a number of means of contacting producers were already in place. However, not all of these methods may allow users instant access to producers.

The means for communication between users and producers should be evaluated to ensure that the optimum approach is adopted.

Action: Producers and user groups.

Our review found that producers use a range of routes for establishing contact with users (e.g. relevant contact e-mail addresses on Websites, telephone numbers, e-mail alert systems). These methods should be reviewed to ensure that they satisfy users' requirements efficiently. Other potential methods should also be included in the review.

Accessibility

The format of the presentation of data, particularly the introduction of data cubes, should be considered.

Action: Producers.

Comments from more experienced users of health statistics suggest that they felt that the presentation of data was targeted towards the more inexperienced user. This proposed review would address the most appropriate format to fulfil the various users' needs. Alternative means of presenting the data should be included in this review. International evidence suggests that data cubes may be a means of allowing greater user customisation of data which may be more useful for experienced users.

A review should be undertaken to consider the structure of a central enquiry point. This would address users' queries at the point of use.

Action: Producers.

Some users muted that a single point of contact worked well in Scotland and Wales. Adopting such a central enquiry point may also work well for all health statistics. The review would address whether this centralised point would cover all countries in the UK and what health statistics would be included in its remit. This central point could also be used to monitor complaints and issues raised by users of National Statistics. This recommendation will be addressed to some extent by the establishment of a Health and Social Care Information Centre in 2005.

Metadata

A standardised template should be designed for all health data.

Action: Producers.

(Sophie Beale, Louise Carr, Jacqueline O'Reilly and Peter West, York Health Economics Consortium (Market Square, University of York, Vanbrugh Way, Heslington, York YO10 5NH;Tel: 01904 433620; Fax: 01904 433628; Email: yhec@york.ac.uk; http://www.york.ac.uk/inst/yhec/)

1.1 Background

The Statistics Commission contracted York Health Economics Consortium (YHEC) to review the use of statistics in monitoring health services in the four countries of the UK. The main aims of the review were to:

• Identify users of health statistics;
  
• Explore the use made of health statistics;
  
• Investigate any issues that users have with their use of health statistics.

The research can be divided into the following four areas of work:

• Identification of users of health statistics;
  
• Questionnaire and interview surveys of users;
  
• Identification of emerging issues;
  
• Detailed exploration of selected issues.

A wide range of health statistics are used within the four countries of the UK. There is no central source to identify all of these statistics. One of the first areas of work within the project was to identify the health statistics produced. This list concentrated on official health statistics published by government organisations. This is discussed in Sections 2 and 3.

1.2 Identification of Users of Health Statistics

The task of identifying users of health statistics was challenging. A number of approaches were used including a literature search and discussions with the Project Board.

The users of health statistics were categorised into primary and secondary users of data (see Section 2). This categorisation was based on discussions with the Project Board to highlight areas of interest to the Statistics Commission and searches by the research team to identify main organisations using health statistics where the use had an impact on policy. (Supplementary information was obtained from a survey of users of the ONS Website. This survey included a number of questions aimed at collecting information about people who used the ONS Website to access health data. The analysis of responses from the ONS survey is reported in Appendix A.)

1.3 Interview and Questionnaire Surveys

Organisations identified as a primary user were asked to participate in an interview survey, whilst secondary users were surveyed by questionnaire. The surveys identified a number of issues of concern to users of health statistics. The findings from the interviews and questionnaires are reported in Sections 4 and 5 respectively.

1.4 Identification of Emerging Issues

As noted, the research highlighted a number of issues that are of concern to users who responded to the surveys. These issues were aggregated into the following five main areas for further detailed exploration:

• Inter-country comparability;
  
• Communication;
  
• Metadata;
  
• Disaggregation;
  
• Accessibility of Data.

1.5 Detailed Exploration

For each area listed above, a series of issues/questions were formulated. These questions and issues were researched over a period of four weeks. The results of this research are detailed in Sections 7 to 11 of this report. Case studies, which investigate the five issues in two specific disease areas, are included in Section 12. Section 13 contains a summary of the conclusions for each issue and a discussion of the findings. Recommendations from the review are reported in Section 14.

1.6 Caveats

This report contains views from respondents to the interview and questionnaire surveys. It must be remembered that these views are perceptions and may not necessarily be factually correct. For example, a user may criticise the lack of data on a specific health issue. This may reflect the fact that the user is not aware that these data are in fact available.
The findings detailed in this report are based on a sample of health statistics users. However, the size and characteristics of the entire population of users is not known and, therefore, statistical significance cannot be calculated. Nonetheless, the research highlights a number of significant issues.

There is also the potential for self-selection bias within the sample. As previously stated, the organisations selected for interview were selected on the basis of being a primary user of health statistics. In contrast, the questionnaire survey was sent to a wide range of organisations and individuals. Only the more experienced and frequent users of data may have responded to the survey. A large number of individuals/organisations may use health statistics infrequently. Although these users may have issues or concerns about finding or using the data, their infrequent use may mean the importance they attach to these issues is lower than that of a more frequent user. Hence they may have been less inclined to respond to our survey.

2.1 INTRODUCTION

One of the first tasks undertaken as part of the review of health statistics was to compile a list of health statistics. The aim of the list was two-fold:

• To highlight key health statistics;
  
• To inform discussions with users of health statistics.

The list focused on official and primary sources of health statistics. Separate lists were compiled for the statistics produced by the DH in England and appropriate organisations in the devolved administrations, and the ONS.

These lists, although not exhaustive, were compared with the statistics contained in the Compendium of Health Statistics produced by the Office of Health Economics (OHE). This comparison ensured that relevant areas were included and also helped to identify main areas of health statistics.

These lists are lengthy¹ and have been summarised in Figures 2.1 and 2.2. These figures distinguish between health statistics collected as part of the assessment of public health in Figure 2.1 and those that provide an overview of NHS activity in Figure 2.2 (e.g. using immunisations or infectious disease notifications as a measure of population health but using total hernia operations as a measure of NHS activity, given the likelihood of untreated disease). The separation is not precise for NHS-collected data but can be seen as mainly delineated by the existence of registers, such that data collection is a formal goal of health services, as opposed to the routine recording of activity. Other categories of health statistics, not included in these Figures, relate to areas such as education and crime.

Figure 2.1: Statistics on Public Health

Figure 2.2: NHS Statistics (replicated for Social Services)

2.2 Need for Health Statistics

To identify the need for health statistics, it is firstly important to understand the operation and structure of health care systems. Health care systems comprise of a number of elements, including:

• Financing. This involves the collection of funds, which are used to finance the provision of health care services. In a publicly-funded system, funds are raised through taxation. In contrast, an insurance model will collect funds through insurance premia.
• Fund pooling. Once funds have been collected, they are then pooled by some central authority. The objective is to ensure that funds are allocated in an equitable manner to different groups within the population.
• Purchasing. Funds are then used to purchase health services. The decision on the type and quantity of services required will depend on the characteristics of the population.
• Provision. A number of different models can be used to provide health services. The type of organisations involved in the provision of services ranges from public and private hospitals to individual clinicians. Different payment arrangements may be involved. For example, clinicians may receive block grants to complete a particular amount of work. Alternatively, payment may be made on a fee-for-service basis.

The operation of each of these elements will impact on the efficiency and equity of the health care system - the two standard broad yardsticks used to assess a system.

Efficiency relates to both technical and allocative efficiency. The former ensures that services are provided at minimum cost. The latter adds to this the requirement that the services provided must correlate with the needs of the population. Therefore, to assess technical and allocative efficiency, data are needed on:

• The cost and efficacy of supplying various interventions and services;
  
• The health of the population.

Data on the first of these will be used in cost-effectiveness studies which ensure that the costs of interventions are justified by their benefits. These studies of economic evaluation, together with the introduction of clinical guidelines to recommend best practice, represent moves towards ensuring both types of efficiency. One such guideline is discussed in Section 2.3.

The second broad standard relates to equity of access. In an equitable system, access to health services should be similar for patients with similar conditions. Moreover, access to services would be related to need, rather than ability to pay. Consequently, systems funded through general taxation are considered more equitable than those financed through private health insurance.

Information on waiting lists will help to determine whether access is similar for patients with similar conditions. Furthermore, data on the health service workforce and the geographical location of the population may help to explain differences in access, since the latter may reflect local variations in the allocation of resources. Such local variations should be taken into account in the formula used to allocate health care funding.

Health care is the largest category of government expenditure. During 2002/3, departmental expenditure limits for health were set at £57,491 million - 24% of total government expenditure (£240,572 million).ⁱ The next largest expenditure category was local government, set at £37,598 million. With such a substantial level of expenditure on health, it is important to ensure that the health service is satisfying the targets of efficiency and equity.

Donabedian suggested that performance could be assessed in more detail on three levels:ⁱⁱ

• Structure - The quality and appropriateness of the available inputs and their organisation;
  
• Process - The quality of the delivery of care;
  
• Outputs - The ultimate quality of care.

Each of these three levels are interdependent since information on the quality of outputs alone would not indicate the reasons behind these findings, which may be related to the quality of inputs or the delivery of care.

To assess structure, it is necessary to obtain data on all inputs to the provision of health services, including staff, drugs, equipment, and physical assets, such as hospitals. Similarly, in considering process, information would be required on length of stay, the type of treatment received (e.g. rehabilitation), patient satisfaction, and hospital and patient management.

Quantifying outputs is considerably more difficult than the measurement of inputs. A number of instruments can be used to measure output, albeit imperfectly - for example, mortality, life expectancy, quality of life, etc. The main difficulty in deriving a definitive measure is that the relationship between output and the health services is not obvious since it may be influenced by a large number of exogenous factors, such as underlying patient characteristics.

The recent Atkinson Review examined the issue of the measurement of outputs in health.ⁱⁱⁱ This review found that the current method of measurement reflects 'the volume of goods and services produced as health care outputs', which are 'those which benefit or increase the welfare of recipients'. The output measure incorporated inpatient episodes and day cases, as well as visits from health visitors and district nurses. The fact that this measure is actually focused on quantifying inputs highlights the difficulty of measuring outputs. The Atkinson Review raised concerns about the quality of the current measure, particularly in relation to general practitioner services. The issue of quality was raised by interviewees in our review who queried the reliability of self-reported measures of activity, which are not directly related to the work of the individual reporting them.

Having established the general need for health statistics in assessing the achievement of goals, such as efficiency and equity of the various parts of the health care system, and monitoring performance, Section 2.3 proceeds to examine specific cases of the use made of health statistics.

2.3 USE MADE OF HEALTH STATISTICS

This section contains examples of the use made of three categories of health statistics by different organisations.

Example 1: Registers of diseases

The Eastern Region Public Health Observatory (ERPHO) publishes regular briefing papers on topical public health issues; these are based on health statistics. For instance, in May 2004 a briefing paper was written on 'Cancer incidence in the East'.^iv

This analysed data from the Cancer register for the East of England in detail, (e.g. an analysis of the annual incidence of Cancer in females by Primary Care Trusts (PCTs) was included). Other Observatories also use Cancer registers to carry out in-depth analyses of the regions they cover. However, the level of detail varies by each Observatory as the PHO work programmes are based on local areas of interest.

Cancer registries can undertake a range of public health surveillance and health protection functions. Data from the registers are used to monitor trends in Cancer and evaluate the impact of environmental and social factors on Cancer risk. For example, the ERPHO briefing paper states the ERPHO has shown an increased incidence of leukaemia in the residential areas around East Anglian waterways and has found an association with deprivation and specific Cancers.^v

Cancer registries and organisations within the NHS use data from Cancer registers to monitor the effectiveness of the existing national screening programmes for breast and cervical Cancer.

Cancer Research UK is a source of information on Cancer. They provide a free information service (CancerHelp UK) about Cancer and care for people with Cancer and their families.^vi This service utilises all health statistics available on Cancer. Other national voluntary organisations also provide similar services on different diseases areas.

Example 2: Morbidity/activity statistics - smoking cessation

Various statistics are collected on smoking. These include average daily cigarette consumption per smoker by sex and age, and statistics from services monitoring smoking cessation. Again, these statistics are used by various groups, including voluntary organisations, to highlight the potential health dangers of smoking. The media will often use the DH press releases about forthcoming statistical releases and write a summary article about the 'headline statistics' or write a more in-depth article. For example, following the press release by the DH, entitled 'Statistics on smoking cessation services in England, April 2002 to March 2003' published on 24 July 2003, most national newspapers included related articles.^vii One such article was headed 'NHS targets smashed as smokers kick the habit'.^viii

Health statistics on smoking were used in the evidence submitted to the National Institute for Clinical Excellence (NICE) for its review on Nicotine replacement therapy (NRT) and bupropion for smoking cessation. This review included a section on clinical need and practice, which incorporated a range of health statistics as shown in Box 2.1. This is a common use of health statistics as most reviews undertaken by NICE will tend to include a section which outlines current practice using a range of health statistics. When NICE release guidance, the media tend to report the more topical ones in detail such as smoking cessation and will tend to include health statistics in their articles.

Box 2.1: Example of use of health statistics

Source : Extract from NICE Guidance on the use of nicotine replacement therapy (NRT) and bupropion for smoking cessation - full guidance, www.nice.org.uk

The media frequently use health statistics in their articles. They also report on the findings of studies containing health statistics. Occasionally newspaper stories may make claims regarding new medical discoveries or may generate health scares. Such stories often have consequences for patients and may create extra pressures for health professionals who have to respond to patients' questions relating to these articles.

The National electronic Library for Health (NeLH) has commissioned the NHS Centre for Reviews and Dissemination (CRD) at the University of York, to assess the reliability of both the journalists' reporting of health stories and the research on which they are based. This project, entitled Hitting the Headlines, comprises a rapid assessment of the original research and an evaluation of the accuracy of the research findings.^ix These assessments are produced by CRD within 48 hours of the publication of the original news stories. An example of this work, relating to newspapers' reporting of a combined pill to reduce the risk of heart attacks and strokes, is contained in Appendix B.

Example 3: Surveys

Local government will use the results from surveys to inform local authorities about the health of their populations. For example, the Tees Valley Joint Strategy Unit has produced a detailed report analysing the results from the 2001 census and compared this with the previous census. A section in the report concentrates on health statistics.^x

The Manchester Joint Health Unit was established on 1st April 2002. It is funded by the three Manchester PCTs and Manchester City Council. It was set up in recognition of the fact that Manchester has some of the most challenging health problems in the country. The main aim of the Unit is to co-ordinate efforts to reduce inequalities within the city, and between the city and the rest of the country. To meet these aims the Unit makes use of health statistics, in particular small area statistics. It mainly uses data at three different levels: Manchester as a whole, PCT and Electoral ward. Within their Website the unit makes the following statement:^xi

'The level at which the statistics are presented is to a large extent dependent on the nature of the underlying data. For example, the number of events underlying some of the tables may be too small to present at a ward level without threatening the reliability of the indicator itself. In other cases, for example, cervical screening uptake, the data are based on the registered population of a GP practice, making it difficult to convert the information to a resident base, such as an electoral ward.'

The results from lifestyle surveys are reported frequently within the media and are used to influence policy. For example, the National Audit Office report, entitled Tackling Obesity in England, cites figures from the Health Survey for England (HSE) and uses these figures to make future projections about the prevalence of obesity.^xi The Health Select Committee's report on obesity also quotes figures from the HSE.^xii

Other health statistics related to such surveys are consumption and personal health statistics. These include statistics on alcohol consumption that are fed into various public health campaigns regarding sensible drinking.

3.1 Introduction

This section outlines the methods adopted in the following areas:

• The literature search;
  
• The identification of users of health statistics;
  
• The interview and questionnaire surveys;
  
• The identification of issues.

3.2 Literature Search

The aim of the literature search was to identify studies containing examples of how health statistics are used and any work on user opinions of health statistics. Supplementary to this, it was hoped that the searches would help to reveal the main users of health statistics. Users were also found either from publications, or through broader searches of the Internet and relevant organisations' Websites, as well as from directories of organisations.

It was recognised from the beginning that it would be very difficult to devise a workable search strategy. The term 'health statistics' is a very broad definition. It describes collected, summarised, and analysed data as well as the vast array of different statistical techniques that are used. A broad spectrum of published statistics, from national statistics to very specific detailed analyses, is produced. These statistics cover a range of topics about illness, disease, morbidity and mortality as well as about the health service and delivery of health care. An initial attempt to produce a list of relevant UK statistical resources only pointed out the plethora of health statistics available from numerous sources and in many different formats.

3.2.1 Search Strategy

The initial stage of the search strategy was to identify the most appropriate places to search (journal articles, books, reports, consultation documents, unpublished material). It was decided that the major medical databases should be searched as well as any UK specific health-related databases. Medline, Embase, Cinahl, Health Management Information Consortium database (HMIC), the British Nursing Index (BNI), and the grey literature database, SIGLE, were chosen.

Search terms are usually identified by looking at relevant MeSH or subject headings, looking through publications already obtained and through discussions between the information officer and the research team. In order to have a strategy that is both sensitive and precise (identifies all relevant studies without retrieving too many irrelevant studies), strategies usually entail a combination of search facets. In this case the only option was to combine 'statistics' with 'user' and 'user views/opinions'. Search terms for user, views, opinions, and feedback are as noisy (retrieve totally irrelevant references) as those for statistics. MeSH terms in Medline and indexed subject headings in the other databases for statistics were not used as they were found to be either too broad or concentrated on specific statistical techniques (e.g. cluster analysis, analysis of variance, models, etc.). There were no MeSH or subject heading terms for user views.

The searches had to be pragmatic rather than systematic and so looked to combine free text terms in close proximity. Furthermore, it was decided to use 'statistics' alone rather than additional terms such as data and survey. The searches were limited by date (2000-2004) and by studies relating to the UK. It was hoped that relevant references would be identified to help refine and improve the original search strategy. However, despite the date and place limits imposed, the searches still retrieved over 600 references. These references were inspected, and few seemed to be relevant. The retrieved results included references to statistical publications, press releases, research studies using statistics in their results, surveys about the use of health care services, social care services and libraries, guides about how to use statistics and debate about statistical techniques, etc.

The search strategy used in Medline is included in Appendix C. This strategy was translated and adapted for use in the other databases that were searched.

3.2.2 User Groups

The poor results of the database searches meant that a different approach was required. Statistical e-mail discussion groups were accessed to see if experts could help by naming any relevant publications, suggesting useful sources and giving advice. Although there was a good response, the overall picture was that little had been done, and that any relevant work would be likely to be in the form of consultation documents and feedback questionnaires: not in a form easily found in electronic databases or even on the Internet. The three discussion groups contacted were: Health Statistics User Group (HSUG), RadStats (Radical Statistics Group), and All Stats Group. Contact was also made with the Office for National Statistics (ONS), the Department of Health (DH) Statistical Division, the Association of Public Health Observatories (APHO), and the eight Regional Public Health Observatories (PHOs).

3.2.3 Internet Searches

Internet searches were undertaken. Searches were carried out using the general search engine Google and the Meta-Search engine Copernic, but without success: the search terms and any attempted combination of search terms found nothing new of interest other than the organisation Websites already known. Appendix D details the Websites that were identified and inspected.

3.2.4 Users of Health Statistics

Alongside the literature review, searches were made to compile a list of users and user groups who might influence policy. Any users identified during the literature review were included on the list. Searches of the health Internet gateway OMNI were undertaken to identify more user groups, and known user group Websites were searched for links to other user groups. Directories of organisations and associations were also searched (Directory of UK Associations, Directory of British Associations, Directory of Health Library and Information Services).

3.3 Identification of Users

The literature search confirmed that, to date, very little published work has been carried out on user views of health statistics. However, it helped to identify the list of statistics to be included in the review and gave an insight into the user groups relating to health statistics. A list of organisations using health statistics was compiled based on this information and discussions between the research team and the Statistics Commission.

Figure 3.1 indicates the breadth of organisations using health statistics. Discussions were held with the Project Board to identify those of primary interest. These discussions resulted in the inclusion of a number of categories of organisations in the review (see Appendix E).

Figure 3.1: Schematic representation of links between organisations using health statistics

From this list, discussions were held between the research team at YHEC and the Statistics Commission, to identify primary and secondary users of health statistics. The primary users were to be included in the interview survey and secondary users were to be a questionnaire survey. Further explanation of primary and secondary users can be found in Sections 3.4 and 3.5.

3.3.1 ONS Survey

The ONS commissioned a web-based survey to investigate user perceptions. The survey was designed by MORI, an independent research agency, and could be accessed via a link on the ONS Website. ONS inserted some questions within their survey to assist us with identifying users of health statistics. Unfortunately the timeframe for the ONS survey meant that it was not possible to use findings from this in the early stages of our work. The questions included in the ONS survey and associated analyses can be found in Appendix A.

3.4 The Interview and Questionnaire Surveys

In order to identify concerns of users of health statistics, two approaches were adopted:²

• An interview survey;
  
• A questionnaire survey.

3.4.1 Interview Survey

The selection criteria for potential interviewees (primary users) focused on organisations that influence policy. Moreover, our methodology recognised that the process of influencing policy may involve a chain of users of health statistics. We assumed that only one institution - a government body - may directly influence policy, although a number of other organisations, from producers of health statistics to those who analyse and/or cite the data in reports, may have an indirect impact. Therefore, in identifying potential interviewees, we strove to ensure that contributions from representatives of each stage of the user chain were involved.

In addition, we targeted key policy areas, which were defined as those areas in which National Service Frameworks (NSFs) have been developed. Further justification for our selection of these areas is that they account for a subsul>

• Cancer;
  
• Coronary heart disease;
  
• Diabetes;
  
• Mental health;
  
• Older people;
  
• Paediatric intensive care.

We recognise that some organisations may potentially influence policy in a number of different areas. Similarly, institutions may be primary users of health statistics in a particular area, but secondary users in another area.

To ensure that consistent and comparable data were collected, the interviews followed a semi-structured format and covered the following main areas:

• Frequency and purpose of use;
  
• Frequency of publication of health statistics;
  
• Quality, reliability and accessibility;
  
• Coverage;
  
• Influence on policy.

A list of the interview questions is contained in Appendix G.

Interviews were conducted by researchers from YHEC and/or the Statistics Commission. As part of the interview process, users were sent a synopsis of the issues raised during the interview, which they were asked to agree or amend as appropriate.

3.4.2 Questionnaire Survey

To capture the views and concerns of health statistic users from a broad range of organisations, a questionnaire was circulated to a large sample of potential users.

Content

The questionnaire was designed in conjunction with the Project Board and gave recipients an opportunity to contribute to this research project. Due to the fact that the number of health statistics published is extremely large it was decided to create a questionnaire that tackled broad issues. A copy of the questionnaire and accompanying letter or e-mail can be found in Appendix H. Section 1 of the questionnaire asked general questions that gave background to how, where and why health statistics are used, whilst Section 2 concentrated on views about quality aspects of health statistics. Included within the second section was a question that asked whether the respondent felt that their use of health statistics influenced policy, either directly or indirectly. Questionnaire recipients also had the opportunity to express views on issues not directly tackled by the questions within the questionnaire.

Circulation

We aimed to target users of health statistics from a wide range of organisations. Recipients fell broadly into six categories:

• Management within the National Health Service (NHS);
  
• Government bodies - central and local;
  
• Political Parties;
  
• Think Tanks;
  
• Educational Bodies (including the Royal Colleges);
  
• Statistics User Groups.

The structured methodology used to compile the list of recipients in NHS management and a list of names of other recipients is included in Appendix H.

Questionnaires were circulated either as an e-mail attachment (in a Microsoft Word document), by surface mail, or else recipients were e-mailed details of how to access the questionnaire via the web. We also set up an AdWords Ad on the search engine Google in order to direct other potential users of health statistics to the online questionnaire. Upon entering pre-identified search terms into Google, an AdWords Ad appeared on the right hand side of the screen that displays the search results. This advertisement encouraged potential respondents to complete an online version of the questionnaire and directed them to the relevant site.

Most questionnaires were sent out as an e-mail attachment. Wherever possible e-mails were addressed to named recipients. In cases where this was not possible, e-mails were directed to the information, enquiries or help desk as applicable to that organisation. This primary recipient was asked to forward the e-mail to the appropriate recipient or recipients within their organisation.

Surface mail was used where no e-mail address was available. It was generally possible to send these letters to named individuals.

The DH in England and equivalent organisations in Northern Ireland, Scotland and Wales put a summary of the project and a link to the questionnaire on their Websites. The link was circulated to co-ordinators of statistics user groups who distributed it to their members. A link was also sent to the Society of Social Medicine (SSM) who, as well as placing it on their Website, also incorporated the summary and link in their Newsletter. Thus each link will potentially have been seen by a number of users of health statistics.

Over 900 questionnaires were sent out, giving many people the opportunity to share their views and issues pertaining to health statistics. The breakdown of the questionnaire delivery format is shown in Table 3.1.

Table 3.1: Breakdown of questionnaire delivery format

Note: ^a Each link will potentially have reached a number of health statistics users.

3.5 Identification of Issues

The responses from the interview and questionnaire survey were analysed in detail. As part of this analysis, the issues identified by the users were summarised and aggregated into relevant categories. This is discussed in Section 6.

4.1 Sample

In total, 28 organisations were invited to be interviewed. Sixteen interviews were carried out. The characteristics of the organisations, from which representatives were interviewed, are reported in Table 4.1. The sample consisted of a wide range of organisations from across the UK.

Table 4.1: Characteristics of organisations

Note: ^a Covers England and Wales.

Among this sample, the frequency of the use of health statistics varied from quarterly to daily. The following are examples of the purposes of interviewees' use of health statistics:

• To produce reports, press releases and fact sheets;
  
• To inform the public and the media;
  
• To support campaigns and lobbying;
  
• To create risk factor simulations;
  
• To assess government targets;
  
• To produce trends;
  
• To investigate current issues;
  
• To conduct comparative analysis;
  
• To develop policies and policy responses;
  
• To construct models;
  
• To study the workforce;
  
• To examine the factors influencing delayed discharges;
  
• To customise local data for use by others;
  
• To produce health technology assessments;
  
• To support local governments and local authorities;
  
• To calculate deprivation measures and estimate populations;
  
• To monitor health trends and provide advice.

Table 4.2 shows the types of organisations who were not interviewed.

Table 4.2: Characteristics of remaining organisations not interviewed

4.2 Findings

This section summarises the main points raised by users during the interviews. Comments have been classified into the following categories:

• Level of coverage;
  
• Level of detail;
  
• Frequency of publication;
  
• Time lag between collection and publication;
  
• Communication;
  
• Compilation;
  
• Quality and accuracy;
  
• Accessibility;
  
• Other.

4.2.1 Level of Coverage

Almost all interviewees identified areas where there was a lack of data. A number of interviewees recognised that the production of health statistics was determined by government policies and performance targets. This was confirmed by producers who acknowledged that the production of health statistics was mainly driven by the demands of government priorities. However, one of these producers further commented that as a result of these changing priorities, a number of gaps had been identified (e.g. data missing on children's services and mental health). One user also recognised data series that would be required to monitor policy, but were not currently available (e.g. data to support the Government's agendas on poverty reduction and health inequalities).

The main areas where statistics were missing are reported in Table 4.3. The small numbers reported in this Table reflect the relatively small sample of organisations that participated in interviews and their divergent demands for additional data.

In the absence of data from official sources, one interviewee commented that the relevant information had to be derived from alternative sources, which could introduce calculation errors. Similarly, another user raised comparable concerns about the implications for reliability of using alternative data sources. As one interviewee pointed out, using data from alternative sources may be inappropriate because these data may not be generalisable. Another organisation remarked that as a consequence of official data, particular statistics were calculated by an external, non-government institution. One user criticised the existing data relating to the NHS as 'abstract' and argued that qualitative data would help to explain trends and inform patient choice.

In contrast to the views about insufficient data reported in Table 4.3, one participant commented that new data seem 'un-balanced'. This interviewee felt that there was a significant amount of data relating to mental health being generated and that these were rarely used in their experience. If this were the case, it may suggest that there was scope to optimise the use of statistics by reallocating resources from areas where statistics are less frequently used to those where gaps have been identified.

Table 4.3: Gaps

Notes:

^* Interviewees may have identified more that one area.
^aThis interviewee was particularly interested in the hospital readmission rates for people aged 75 years or over, which could be used to assess early discharge and subsequent readmissions. This data series was discontinued because, according to this interviewee, the DH considered it to be ageist. However, the interviewee argued that this argument could be overcome by producing this information for all adults by age band.
^b This interviewee suggested that a registry for diabetes, similar to the one for Cancer, would help to identify disease prevalence.
^c There have been recent moves towards providing this information.
^dThis interviewee stated that local authorities currently produced registries of people with disabilities, but these were not updated.
^e This interviewee suggested that a survey of these core indicators could be undertaken annually to help with trends.
^fThis interviewee emphasised sensitive personal areas such as sexual orientation and political opinion.
^g This interviewee stated that additional data on care in private hospitals, and in local authority and private care homes, and care given in the community are required to provide data on care after miscarriages, and antenatal and postnatal care. Currently, these data are only provided if care was given in a hospital setting.
^h This instrument was used to collect information on hospital expenditure by function, but was abolished.
ⁱ This interviewee proposed that primary care systems and the new GMS contracts provide an opportunity to collect data at a local level on such risk factors.

4.2.2 Level of Detail

A number of users mentioned that it would be useful if data were presented on a disaggregated level. Where detailed data were not available, some of these organisations mentioned that they contacted data providers directly with specific requests for this information. A number of these participants acknowledged that detailed data may result in confidentiality issues. In addition, statistical analysis of these data might become increasingly problematic if the resulting samples were small. One interviewee noted that although they may request small area data, these may not always be provided because they may contravene the Caldicot Principles.

Specific areas, mentioned by interviewees, where disaggregated data would be useful are reported in Table 4.4.

Table 4.4: Disaggregated data

Notes:

* Interviewees may have identified more than one category
^a This interviewee commented that these categories should be defined by a standardised methodology.
^b One interviewee answered that it would be useful to have information for small areas over time rather than the 'snapshots' that the DH usually reports.
^c This interviewee stated that this information is available from the DH on request.
^d This participant mentioned that it is currently difficult to map PCT areas to parliamentary constituencies.

In contrast to these remarks requesting in-depth data, one interviewee commented that there is more detail than is ever actually needed, although they did find disaggregation by medical specialties useful.

4.2.3 Frequency

Interviewees were asked if they would like data to be produced more frequently. Three interviewees did not have any complaints about the frequency of data. One interviewee admitted that although timeliness used to be a problem, in their opinion it has improved for some statistics (e.g. Hospital Episode Statistics, (HES)). Three participants identified a trade-off between the frequency of data collection and reliability. Moreover, one of these participants also acknowledged that it was not realistic to produce some statistics more often (e.g. Vital Statistics). In particular, producing some datasets at more frequent time intervals may raise issues about small sample sizes and consequently, increased uncertainty. Although one interviewee identified an area where it would be useful to collect data more frequently (e.g. monthly rather than quarterly waiting lists), they considered increased coverage and consistency to be relatively more important.³

Another interviewee mentioned that it would be useful if datasets were produced for consistent time intervals. This participant indicated that using different datasets in conjunction may be problematic if they refer to different time periods. Relatedly, another respondent mentioned that there are differences in the target timescales of statistics collected in England and Scotland. Again, this makes comparative analysis difficult.

Areas where respondents suggested frequency could be improved are reported in Table 4.5.

Table 4.5: Suggested areas for increased frequency

Notes:

^a One of these respondents explained that the Health Survey concentrates on different subject areas in different years on a cyclical basis. Consequently, data relating to specific topics may be out-of-date. In addition, this also raises problems for establishing longitudinal data and identifying trends. This interviewee suggested that an electronic patient record system may overcome these problems.
^b This interviewee suggested that a mid-decade or running census would be useful.
^c The interviewee proposed that this source would be electronic patient records and could be used to identify patient pathways and provide information on disease progression.
^d This interviewee mentioned that it would be useful to have detailed monthly waiting lists. Currently, this information is only available on a quarterly basis.
^e This respondent answered that even though delayed discharges are an important political issue, no regular information on these is produced.

4.2.4 Time Lag

The lag between collection and publication did not appear to be a major concern to several interviewees. However, one interviewee commented that they would like real time data on, for example, HES and deaths. One participant commented that ideally there should be a shorter time lag, but recognised that there may be problems inherent in tackling this issue. This point of view was confirmed by another interviewee who stated that they would like immediate access to data, but would prefer to wait for accurate and complete data. Another acknowledged that they would like to reduce the time lag yet accepted that there was a trade-off between the speed with which data are produced and their accuracy. Two other contributors suggested that electronic collection of data may help to reduce release time (e.g. electronic Cancer registries and an electronic patient record system).

4.2.5 Communication

One respondent commented that it would be useful to know when new datasets are published. This interviewee thought that the ONS was quite good at publicising this information. Similarly, the issue of notifying users about forthcoming publications was mentioned by two interviewees. These two respondents suggested that a calendar of forthcoming publications for all health statistics may be utilised as a means of achieving this. For one of these organisations, this comment was related to a subsequent suggestion that there should be greater communication between producers and users of health statistics. Another interviewee made the related comment that public awareness of statistical releases could be heightened if the DH held press conferences and did not concentrate its releases on a small number of days.

4.2.6 Compilation

The majority of interviewees commented on the approaches used to compile statistics. Specifically, a subgroup of these viewed knowledge of methodology as important in assessing data quality and undertaking comparative analyses. One producer commented that generally users appear to be more interested in coverage and the level of disaggregation rather than compilation. This was confirmed by one user who did not have detailed knowledge of the methodological approach, even though they considered this important in determining the quality of data. One organisation commented that knowledge of compilation depended on the data source. This user suggested that clear definitions were vital in understanding the limitations of the data. The definitions reported by ONS were deemed to be 'good' by this interviewee. According to two respondents, information on the methodological approach to compiling data was available, but was not necessarily easily accessible to inexperienced users.

The comments raised during interviews concerning consistency related to individual datasets, between datasets and inter-country datasets. One organisation mentioned that diabetes was not well recorded on death certificates because it was not always noted as the main cause of death. Similarly, two interviewees mentioned that there may be biases in data inputs (e.g. some medical professionals may overstate the number of visits that have actually conducted). Related to these arguments, two participants commented that the quality of data collection was particularly poor in areas where the information being collected did not have a direct impact on those collecting the information. One of these organisations gave the example of the unit of measurement currently used in community services data. According to this interviewee, the use of these 'contacts' is meaningless not only to users, but also to the professionals to whom the data pertain. Consequently, this interviewee recommended that the process of data collection should be classified in line with the work of professions to which the data relate. The second organisation expected that the relevance of data collection to professionals would increase with the introduction of a new Person-centred Information System for community and personal social service statistics. Related to this point, one organisation suggested that through such a patient-centred data collection system, health statistics could be improved by focusing on outcomes for individuals, as is currently the case in the HSE and the Northern Ireland Health and Well-being Survey.

According to some interviewees, there was also scope to improve the consistency across datasets produced by different agencies. For example, differences in the definition of colorectal Cancer across different sources imply that these data series are not directly comparable. Similarly, where used, age bands should be consistent.

A number of interviewees commented on the lack of comparable data for the four countries of the UK. Three producing organisations explained that these differences had evolved due to different paces of technological progress and different relationships with the health service. One interviewee suggested that differences in data may reflect underlying differences in the treatment of patients across the four countries of the UK. These differences in treatment or the compilation of data have made comparisons between the four countries difficult and in some cases impossible. One interviewee commented that there seemed to be a greater volume of data for England. This meant that this organisation was forced to estimate parameters for the other three countries using data for England. However, this was not appropriate due to the differences in the demography of the countries. One interviewee suggested that it would be useful to have comparable small area statistics across the UK, which would allow comparisons between neighbourhoods across the four countries.

In contrast to these reported differences in the data produced by the four countries in the UK, two interviewees professed that the variation in different countries was not a problem. However, one of these organisations did add that they would prefer to have separate data available for England alone rather than the current merged format of England and Wales. Furthermore, the second organisation was only concerned with England and Wales.

Two interviewees expressed concern about the potential for users to misunderstand to what the data relate. One of these organisations stated that when terminology is unclear, there may be poor interpretation of the data. The other organisation mentioned that users do not always understand the statistical techniques applied to data and consequently, do not take account of this in their interpretation and analysis of the data. Moreover, one interviewee suggested that simpler and more understandable definitions of health statistics were required.

One interviewee commented that the notes attached to the HSE were 'clear and highlighted caveats'. Similarly, another participant answered that metadata should be available for all datasets. Another respondent thought that in general, although the methodology used to compile the data were well documented, it was often hard to find. This participant also suggested that a guide (in paper format) should be published and be easily accessible to ensure that users have access to data definitions and compilation methodology.

Two organisations mentioned that structural changes within the NHS and boundary reorganisations have made it difficult to produce consistent time series. For one of these organisations, this problem was compounded by frequent revisions of the data. Another interviewee mentioned a similar problem with changes in definitions in a particular dataset.

4.2.7 Quality and Accuracy

Interviewees were asked how they would define quality. Most participants perceived quality to be related to fitness for purpose, including areas such as accuracy, reliability, frequency, relevance/appropriateness, robustness and consistency. Five organisations viewed quality of data to be variable across datasets, sources, and over time. For example, one organisation regarded hospital data to be of good quality, but thought the standard was lower for data in areas where there were no performance targets. Another found reliable information on the amount of NHS expenditure, although more detailed information was required at a Trust level. One organisation specifically mentioned the scope for improvement in the quality of individual level data, and in personal social services and children services. The poor quality of these data series is compounded by different IT systems. This latter issue may be negated by the introduction of a new Patient-centred Information System. One respondent raised specific concerns over the coding of the cause of death and the absence of ethnicity from mortality data; the quality of coding ethnicity in HES; and the accuracy and completeness of HES, regional drug misuse data, and the Cancer registry.

Six interviewees reported that they were generally happy with the quality of data. One of these organisations specifically praised the reliability and accuracy of the Health Survey for England. This was attributed to a number of factors including its production by a 'reputable organisation', sponsorship by the DH, and because results are comparable with other surveys. While another of these organisations acknowledged that in general data were of good quality, they may be subject to poor interpretation. Furthermore, this interviewee also regarded the data as accurate, although there were some areas where there was missing information. Another of these organisations deemed health service data to be of good quality, but acknowledged that political objectives may affect the accuracy of the data, although these effects are not known.

As discussed in Section 4.2.6, one organisation thought that the quality of some data inputs was poor. Another interviewee mentioned that boundary and NHS reorganisations may have an impact on data quality.

A number of interviewees stated that they were confident in the quality of National Statistics because they are subject to rigorous quality checks. In Northern Ireland, the stamp of the Northern Ireland Statistics and Research Agency carried similar, if not more, weight because of its long-standing reputation.

One organisation mentioned that it would be useful if information on the degree of uncertainty associated with data were presented. This interviewee specifically referred to population statistics, which were presented as point estimates; here confidence intervals would provide useful information. One institution recognised that the accuracy of data may be affected by the workload of those completing data returns. This interviewee also commented that macro-level data were generally of high quality, but the accuracy of more detailed information was poor.

4.2.8 Accessibility

The organisations were asked how they accessed data and how they found this process. A few organisations answered that they had systems in place for finding the relevant data. However, they conjectured that searching for relevant information is considerably more difficult for inexperienced users. One interviewee explained that the difficulty arises due to the large number of sources. As a solution to this problem, another respondent advocated the use of a 'central enquiry point'. According to this interviewee, this system had worked well in Wales and Scotland, although there may be difficulties in implementing such a system on a larger scale in England due to the larger number of organisations involved. We note that the creation of a new Health and Social Care Information Centre has been announced. One of the main aims in establishing this new centre is to 'reduce the burdens on the frontline by co-ordinating information requirements across a wide range of bodies'. Regarding the functions of the Centre, it 'will retain some of the information-related functions of the current NHS Information Authority, (NHSIA) which will be abolished, and take on the statistics and information management functions of the Department [of Health]'.

One organisation expressed concern that information on a specific disease was not readily available to people with this disease. Another interviewee mentioned that they had difficulty in accessing individual data and data on child health services. The barriers to accessing data on individuals were data protection and consent constraints. Access to information on child health services was difficult because of data sharing constraints since the database was held by another institution. One organisation argued that the Compendium of Clinical Health Outcomes should be more widely available - it is currently only accessible via the NHS Intranet or on CD to those working within the NHS. In addition, two organisations contended that there should be wider access to Vital Statistics datasets. Another participant commented that speedy access to data occasionally depends on the dataset and who at the DH is dealing with the query. One interviewee suggested that online query tools and the generation of online maps could be used to access local data. This user stated that ONS and the Eastern Public Health Observatory have embryonic mapping tools.

Generally, interviewees obtained data from the Internet, paper publications, CD-ROM or directly from the producing organisation. The availability of electronic data was generally viewed as an improvement, although a number of users mentioned problems with specific Websites, including:

• The SHOW Website is difficult to access information and a lot of searching required;
  
• The National Statistics Website is difficult to search and is not well structured;
  
• The Census Website is good, although it is difficult for the inexperienced user to use;
  
• It is not clear if Census tables will be available on the Neighbourhood Statistics Website, which implies that this information must be obtained from an alternative source;
  
• It would be useful if more customisation of tables was possible on the Department of Health's Website;
  
• The classification of health statistics on the ONS Website is not helpful. Health-related information is posted in the Health and Care theme, and also under other themes.

A number of organisations also commented on the recent reorganisation of the DH Website. The general consensus was that since its overhaul, the DH Website is difficult to use. However, one organisation did find the DH Website easy to use. Although the move to Internet-based data was generally viewed positively, one organisation mentioned that there were also disadvantages with disseminating information through the Internet. The main drawback was that people without Internet access could not obtain information.

4.2.9 Format

One interviewee commented that data were currently presented in an 'inefficient format', which was aimed towards inexperienced users. This respondent proposed that it would be helpful if data were presented in a single database rather than a series of spreadsheets. This would simplify the data extraction process. Another organisation made a similar suggestion about the use of a single spreadsheet, rather than individual files. This interviewee also added that the DH could 'add value to its own collections by making the information more user-friendly'. If the DH adopted a more user-friendly approach, then this user's organisation would be able to use the service routinely. To aid with tight deadlines, one user suggested that it would be useful to have an online comprehensive interactive service. One institution commented that other issues regarding their use of health statistics had greater priority than the format of the data.

4.3 Summary

A number of issues regarding the production, compilation and distribution of health statistics were raised by interviewees. Almost all organisations proposed areas where additional data could be produced. A number of interviewees argued that there was an emphasis on collecting data in areas subject to performance targets to the detriment of other health areas. Moreover, the pressure imposed as a result of performance targets may have implications for the accuracy of data.

The majority of users remarked on the need for more detailed data. However, several acknowledged that this may not be possible due to issues regarding disclosure. For most interviewees, consistency and comparability within and across datasets were important. Comparative analyses between the four countries in the UK were almost deemed impossible due to differences in methodological approaches to data collection.

A number of interviewees viewed quality and accuracy of data as variable, although some were reassured that datasets classified as National Statistics had to satisfy high standards. However, one organisation viewed such procedures as a deterrent to the production of National Statistics. This interviewee suggested that the process should be less bureaucratic, particularly for the devolved administrations. A number of organisations also envisaged that a move to a patient-focused system of data collection would lead to improvements in the quality and accuracy of data.

5.1 Introduction

Data from 223 questionnaires were analysed in full. Comments were extracted from a further 11 questionnaires that arrived during the week following the final deadline. This represents a return rate of 36% of questionnaires by letter and 18% of those sent by email as shown in Table 5.1. As noted, the results should be viewed with caution. The findings are based on a sample of health statistics users, however, the size of the entire population of users is not known. Therefore, the results illustrate areas of concern and statistical significance cannot be calculated. However the responses provide a useful summary of views about health statistics.

Table 5.1: Breakdown of questionnaire recipients/returns

Note: ^a Links were circulated to co-ordinators of statistics user groups, the DH in England and equivalent organisations in Wales, Scotland and Northern Ireland, and in the SSM Newsletter.

5.2 Summary of Quantitative Questionnaire Responses

5.2.1 Frequency of Use of Health Statistics

Table 5.2 below shows that respondents' use of health statistics was fairly evenly spread over the five areas detailed. Usage was also fairly evenly distributed, although was slightly higher in the 'Once or twice a month category' (20%) and slightly lower in the 'Once every six months' category (9%).

The use of 'Other' statistics that respondents used was varied. However, 19% (15) of the 80 respondents who gave details of 'Other' statistics used Census and demographic data and 8% (6) used data on deprivation. An alphabetical list of 'Other' data sources used by questionnaire respondents is included in Appendix H.

Table 5.2: Frequency with which the health statistics in the listed areas are used

Notes:

^a One respondent cited daily use but gave no details of the data area used.
^b An alphabetical list of 'other' data sources can be found in Appendix H.

5.2.2 Main Sources of Health Statistics

The DH, ONS, and NHS were quoted 175, 133 and 129 times respectively as sources for health statistics. These frequencies are comparable to the figure quoted for 'Other' as a health statistics source (129). Only 17 of the 206 questionnaire respondents who completed this question indicated that they used the OHE Compendium as a source for statistics, which may be related to the fact that this publication is not freely available. See Table 5.3 below.

Table 5.3: Main source of health statistics

It should be noted that many respondents accessed health statistics from multiple sources, with 39% and 31% of the respondents accessing two and three sources respectively. See Table 5.4 below.

Table 5.4: Number of sources of statistics accessed by respondents

5.2.3 Country to which Statistics Used Refer

Statistics for England, followed by those for the UK, were the most frequently used by respondents. Twenty-eight respondents reported that they used statistics from all of the listed 'countries'.

One hundred and thirty-four respondents used statistics from at least two of the specified countries and 115 of these (86%) said that they felt that their work influenced policy. See Table 5.5 below.

Table 5.5: Country to which statistics used refer

5.2.4 The Purposes for which Health Statistics are Used

Table 5.6 below shows that respondents felt that they used health statistics most frequently within a report (mentioned 566 times) and that they used them least often for marketing or press releases (173). Use may be a reflection of the jobs carried out by questionnaire respondents rather than any indication of the actual balance of health statistics usage.

Public Health and Health Care statistics (mentioned 459 and 462 times respectively) make up 52% of statistics disseminated by respondents. Workforce, Social Care and Expenditure and Finance are all disseminated with comparable frequency (249, 227 and 225 mentions respectively).

Table 5.6: The purposes for which health statistics are used

5.2.5 Dissemination of Information Containing Health Statistics

Table 5.7 below shows that almost half (47%) of statistics were disseminated to respondents' colleagues within their own organisation.

Table 5.7: Dissemination of information containing health statistics

5.2.6 Coverage of Health Statistics

Table 5.8 below shows that approximately half of respondents felt that they would like to see other specific health statistics made available. The statistics mentioned covered a broad range of areas and many tended to be specific to the respondent's area of interest. However, many of the areas mentioned are linked, for example there were several requests for more data on lifestyle (including such areas as smoking, alcohol consumption, obesity) and these data, combined with factors including ethnicity, morbidity and mortality, would be used to forecast the incidence and prevalence of common disease areas and conditions. The other area frequently mentioned was primary care.

Generally respondents have just stated that they would like more statistics in given areas, few have provided details of how they would like that data presented. A tabulated list of health statistics that respondents would like made available is contained in Appendix H.

Table 5.8: Number of respondents who mentioned that they would like to see other specific health statistics made available

5.2.7 Influence of Health Statistics on Policy

In total, 80% of respondents who answered this question felt that their use of health statistics directly, or indirectly, influenced policy (see Table 5.9). A tabulated list of how respondents felt their work influenced policy is included in Appendix H.

Table 5.9: Number of respondents who felt that they used health statistics to influence policy, either directly or indirectly

5.3 Summary of Qualitative Questionnaire Responses

5.3.1 Views on Quality Aspect of Health Statistics

Respondents were asked whether they agreed with a number of statements about quality aspects of health statistics. The breakdown of participants according to those who agreed, disagreed and neither agreed nor disagreed, with the statements are reported in the following tables. In addition to reporting the number and percentage of respondents who agreed, disagreed, and neither agreed nor disagreed, the tables also report the number of respondents in each category who made comments in relation to the statements.

No one issue stood out in terms of the level of agreement indicated by respondents. Respondents who disagreed with the statements made most of the comments. Each statement is studied in detail in the following tables. (Specific comments made by respondents are reported in Appendix G.)

Statement 1

On balance, respondents appeared to be satisfied with the coverage of health statistics, with 47% agreeing and 25% neither agreeing nor disagreeing with the statement 'In general, I am satisfied with the coverage of health statistics'. Not all of those who disagreed with the statement explained their reasons. Comments covered a range of issues including access, timeliness, co-terminosity and areas where statistics were perceived to be lacking.

Statement 2

Respondents' opinion was spread relatively evenly across each of the three response options. Of the 59 respondents who made comments in relation to the statement, 38 (65% of the total number of comments) came from respondents who disagreed with the statement. Both those who agreed and those who disagreed with the statement wrote similar comments, most of which related to absence of data in specific areas of health or the wish for more disaggregated data.

Statement 3

Over half of respondents (52%) agreed with the statement 'I am satisfied with the frequency with which health statistics are compiled'. The 24 comments received by respondents were relatively evenly distributed between respondents who neither agreed nor disagreed with the statement and those who disagreed with it (42% and 46% of total number of comments respectively). However, only 11% of respondents supplied comments. The comments largely expressed the view that it would be helpful if a particular statistic were produced more frequently.

Statement 4

Compared to the proportion of respondents who neither agreed nor disagreed with the statement, 'In general, health statistics are disseminated with the appropriate timeliness (the time lag after the period to which they pertain)', a slightly higher percentage tended to disagree with the statement. Respondents who disagreed with the statement provided approximately nearly three-quarters (72%) of the total number of comments. Comments could broadly be split into two categories, namely those expressing general dissatisfaction with timeliness and those expressing dissatisfaction with delays associated with the publication of specific statistics.

Statement 5

Respondent's opinion was spread relatively evenly across the three response options. Only 28% of respondents made comments. Of these comments, 34 (57% of the total number of comments made) came from respondents who disagreed with the statement, 18 (30%) were made by respondents who neither agreed nor disagreed, and 8 (13%) were from those who agreed with the statement. The comments largely reflected difficulty in accessing specific statistics.

Statement 6

Just under half of respondents agreed with the statement, 36% neither agreed nor disagreed and 20% disagreed. Only 10% (21) of respondents made comments: 13 (62% of the total number of comments made) were made by respondents who disagreed with the statement, 6 (29%) by respondents who agreed with the statement, and only 3 (10%) by those who neither agreed nor disagreed. In general, the comments emphasised difficulty in accessing information either generally or for particular statistics, however, one respondent commented that 'Coppish⁴ datasets are well documented'.

Statement 7

Less than 50% agreed with the statement, 34% neither agreed nor disagreed and 21% disagreed. Only 8% (17) respondents made comments: 10 (59% of the total number of comments made) were made by respondents who disagreed with the statement, 6 (35%) by respondents who neither agreed nor disagreed with the statement and only 1 (6%) by those who agreed with the statement. Most comments were very general in nature and were directed at gaps in information on compilation.

Statement 8

Almost half neither agreed nor disagreed with the statement, 27% agreed and 24% disagreed. Only 43 (20%) of respondents supplied comment: 62% of which came from those who disagreed, 26% from respondents who neither agreed nor disagreed and 12% from respondents who agreed with the statement. It should be noted that over half of the respondents who disagreed with the statement made comments. The comments made tended to query the accuracy or bias of individual datasets rather than health statistics in general.

5.3.2 Additional Comments

Questionnaire recipients were given the opportunity to provide additional comments. These are detailed in Appendix H and include a wide range of subject matter ranging from a request for wider access to more NHS sites to a view that information relating to children should be more clearly defined.

5.4 Summary

Data from 223 questionnaires were analysed.

Analysis of the questionnaires showed that respondents:

• Indicated that their use of health statistics was spread fairly evenly between the five areas;
  
• Most frequently used health statistics within a report, or for research;
• Indicated that information containing health statistics was most often disseminated to colleagues within the respondents' own organisation;
  
• Were generally satisfied with the coverage of health statistics. However, a large variety of specific areas were mention as needing more coverage. Recurring comments were lack of:
• Localised (Regional/PCT/Ward level etc.) data;
  
• Primary care data.
  
• Were generally satisfied with the frequency with which health statistics are produced;
  
• Lent towards the view that health statistics are not generally disseminated with appropriate timeliness;
  
• Had mixed views on level of detail and access. Many felt that it was difficult to navigate around Websites containing official statistics;
  
• Were generally happy about compilation issues;
  
• Were uncertain as to whether statistics are accurate and unbiased;
  
• Generally felt that their use of health statistics influenced policy (76%).

6.1 Identification of Issues

The analysis of the interview and questionnaire survey highlighted the following main issues:

Compilation

• Consistency within and between datasets, and between inter-country datasets;
  
• Changes of definitions in datasets;
  
• Clear and simple definitions and explanations of terminology;
  
• Lack of comparable data for the four countries of the UK.

Frequency

• Notification about forthcoming publications;
  
• Greater communication between producers and users of health statistics;
  
• Datasets should be produced for consistent time intervals;
  
• Differences in the target timescales of statistics collected in England and Scotland.

Timeliness

• Respondents would have ideally preferred more up to date statistics. However, many recognised there was a trade off between timeliness and accuracy / reliability.

Level of coverage/detail

• Requirements for further information/data and more disaggregated formats. These areas were determined by the respondent's area of interest.

Quality and accuracy

• Quality and accuracy of data were viewed as variable by some users, although some were reassured that datasets classified as National Statistics had to satisfy high standards;
  
• Useful if information on the degree of uncertainty associated with data were presented.

Accessibility

• Searching for relevant information is considerably more difficult for inexperienced users due to the large number of sources;
  
• Specific concerns about Websites;
  
• Use of a 'central enquiry point'.

Format

• More user-friendly approach required/ online comprehensive interactive service.

The above areas were discussed between the researchers and the Project Board. It was agreed that the majority of views could be aggregated into the following five areas:

• Disaggregation;
  
• Inter-country comparability;
  
• Communication;
  
• Accessibility of Data;
  
• Metadata.

7.1 Introduction

Health statistics are available for a number of different geographical areas and levels of detail, depending on the type of statistic. The geographical areas covered may include one or more of the following levels:

• UK;
  
• Country;
  
• Regional;
  
• Health Authority;
  
• Local Authority;
  
• Parliamentary Constituency;
  
• Ward/Enumeration District;
  
• Postcode.

Health statistics are also produced at different levels of detail. For example, childhood, infant and perinatal mortality statistics are available as a total number of deaths. These statistics are also available by age group, gender and the cause of death using the International Classification of Diseases (ICD-10). There is also an analysis by some of the key risk factors affecting stillbirths and infant deaths. Furthermore, these analyses are also available for different geographical areas and for different time periods.

7.2 Aims

The aims of this section are to look at the following:

• What issues did the respondents to the interview and questionnaire survey raise in relation to disaggregation?
  
• Where is the demand for disaggregated data coming from? For which data is there a demand for greater detail?
  
• Why do organisations require more disaggregated data (i.e. what are the perceived specific benefits)?
  
• At what level of disaggregation do disclosure and confidentiality issues become a problem?

7.3 Issues Arising from Interviews and Questionnaires

A number of issues were raised in the responses to the questionnaire survey about disaggregation. Although the questionnaire did not directly ask about disaggregation, it was an issue that was highlighted by 29 respondents. This represents 13% of total respondents. In particular, the following two issues concerning disaggregation were raised by questionnaire respondents:

• Statistics should be available at small area level, including ward, practice and PCT level;
  
• More disaggregation of specific statistics (e.g. mortality statistics, and specialty level within a Trust).

During the interviews, a number of users also mentioned that it would be useful if data were presented on a disaggregated level. Ten interviewees gave specific examples on the type of disaggregation. These are reported in Table 7.1.

Table 7.1: Disaggregated data*

Notes:

* Ten respondents in total commented, however, some respondents raised more than one
issue.
^a This interviewee commented that these categories should be defined by a
standardised methodology.
^b One interviewee answered that it would be useful to have information for small areas over
time rather than the 'snapshots' that the DH usually reports.
^c This interviewee stated that this information is available from the DH on request.
^dThis participant mentioned that it is currently difficult to map PCT areas to parliamentary
constituencies.

The findings from the questionnaire and interviews indicate that the main areas of concern raised include:

• Lack of data for smaller areas including ward, practice and PCT levels;
  
• More detail required for specific areas of statistics, for example, the utilisation of NHS resources by age bands and the Health Survey by defined social classes, socio-economic groups, smaller geographical areas, age, education attainment and ethnic groups.

7.4 Reasons for Requiring Disaggregated Data

A number of respondents from PCTs highlighted the need for more disaggregated data at a PCT level. As PCTs are relatively new organisations, many datasets are still reported at old Health Authority level rather than PCT level. Within PCTs, health statistics are used for a range of purposes, including monitoring, planning and resource allocation. For example, PCTs are interested in monitoring inequalities to identify areas where local action is required to achieve the Government's national inequalities targets.

Monitoring can be difficult for small geographical areas such as wards within PCTs. One respondent indicated that this applies to teenage conceptions by ward. The data are only available for wards where the number of teenage conceptions is high. This is reportedly due to data confidentiality issues in order to prevent individuals being identified in those areas where there are only a small number of teenage conceptions.

A small number of respondents indicated that population figures for PCTs are not available as a National Statistic. To overcome this problem, local mapping of the population of PCTs has occurred. For example, the West Midlands Public Health Observatory produces a series of PCT profiles. Within these profiles PCT populations were derived using Census Area Statistic Output Areas for the West Midlands. Furthermore, the DH published updated figures on PCT populations in July 2004.

The DH uses a resource allocation formula, to inform revenue allocations to PCTs. Within the resource allocation formula the size of a PCT's population is a primary determinant of health need. As the ONS do not currently produce population estimates for each PCT, the DH have to estimate the population for the purpose of the resource allocation exercise. The DH calculates PCT populations using registered populations based on GP registers and resident populations based on the ONS Census. These figures are also used to project the PCT population, in order to allocate resources for 2005/06 . This is a significant area of policy and we are aware from other YHEC projects that some PCTs dispute the level of resources allocated to them.

Respondents commenting on disaggregation raised issues regarding the breakdown of statistics by gender, age, socio-economic group, education and ethnicity. Such breakdowns would enable any inequalities to be identified and targeted. Related to these points, the London Health Observatory has been highlighting the case for recording ethnicity at birth and death registration. This Observatory compiled a response on behalf of the APHO for England and Wales to the ONS consultation on 'Civil Registration' . This response indicated that, without information on ethnicity at birth or death, the health and health care needs of minority ethnic communities is difficult to establish. This in turn means it is difficult to plan for appropriate public services such as schools, housing and transport that underpin good health and address inequalities between ethnic groups.

These examples indicate health statistics are used to underpin national resource allocations to PCTs and at a more local level to monitor health and highlight areas for action. This may involve identifying inequalities in health that need to be tackled or allocating resources to projects involved in targeting particular health issues.

7.5 Case Studies

Information about the prevalence and incidence of diseases are used to inform public health and the delivery and monitoring of health services. There are different levels of health statistics available for different disease areas. For example, there is a national dataset for Cancer. Data from this Cancer register can be used to analyse historical trends and to monitor changes in Cancer incidence and survival rates. There are currently no other disease registers that provide national incidence data on a population basis over a prolonged time period. However, plans are in place to establish other registers such as for Diabetes. The NSF for Diabetes outlines plans for practice-based Diabetes registers by 2006.

7.5.1 Cancer Registration System

Cancer registration systems in the UK and in other countries were established with the aim of systematically collecting data on the incidence and characteristics of cancer. These data help in the planning and operation of cancer care services. According to the UK Department of Health, in addition to data collection, the main functions of cancer registries are as follows:

• To monitor longitudinal and spatial trends in cancer incidence, prevalence and survival;
  
• To evaluate the effectiveness of cancer prevention and screening programmes;
  
• To assess the quality and outcomes of cancer care by providing comparative data about treatment patterns and outcomes;
  
• To support investigations into the causes of cancer, especially to the evaluation of the impact of environmental and social factors on cancer risk;
  
• To provide information in support of cancer genetic counselling services for individuals and families at high risk of developing cancer.

The cancer registration system operates as organisations within the NHS (including Trusts), providing a core set of data to the regional registries. The latter are responsible for submitting these data to the National Cancer Intelligence Centre (NCIC) at the ONS. The NCIC are responsible for coordinating the national collation of cancer registration data and undertaking further analysis. Since January 1993, it has been mandatory for the NHS to provide this information.

The information collected by the regional registries can be classified into the following three main areas:

• The patient;
  
• The tumour;
  
• The treatment.

The National Cancer Minimum Data Set specifies the subset of data which must be sent by the registries to the ONS. This subset of data can be divided into fields which are core and those which are optional, as shown in Table 7.2.

Table 7.2: Core and optional fields contained in minimum cancer data set

Note:

^a From incidence year 1993.
^b From incidence year 1993; phased introduction - initially only for breast and cervix.
Source: Quinn M, Babb P, Brock A, Kirby L, Jones J. Cancer Trends in England and Wales 1950 - 1999. Appendix G: The Cancer Registration System in England and Wales.

While adherence to a minimum data set ensures that data are collected in a comparable and consistent manner across cancer registries, it is worth noting the optional fields listed in Table 7.2 will not be collected by all cancer registries. However, the findings from our interviews and questionnaire analysis indicated that some of this optional information may be useful to users - such as data on ethnic origin.

The cancer registration minimum data set is currently under review. As part of this review, the possible extension of the data set to include additional core fields is being considered. In particular, two new fields have been proposed as possible additions to the data set: the stage of the disease for all cancers, and details of treatment. The former would require information on stage to be explicitly given by clinicians. The demand for these two additional fields was borne out during our interviews with users. However, it has been recognised that in devising the fields to be included in a minimum data set, there is a trade-off between the amount of data collected and data quality, accuracy, and the cost of collection.

Section 12 details the level of data available for Cancer incidence and mortality in the UK. This has been completed, based on data available from the ONS Website. As the Section discusses, considerable amounts of data exist for different types of Cancer and different geographical areas. Furthermore specific data requests can be made to the different regional Cancer registries. In comparison the amount of data available on Diabetes is more limited. Again, this is also discussed in Section 12.

7.6 Data Confidentiality Issues

The ONS have a protocol on data access and confidentiality as part of the National Statistics Code of Practice. This protocol sets standards for protecting confidentiality in order to ensure that statistics are not produced that enable individuals to be identified. The producers of data must adhere to the protocol. However, the protocol does not specify exactly how producers should tackle the issue of confidentiality.

Many producers unilaterally adopt the 'no less than 5 in a cell' rule of thumb to avoid inadvertent disclosure. The nature of the analysis is a significant factor in assessing the risk of disclosure. For example, it may be possible to identify an individual by comparing several tables which, individually comply with the rule but when taken together, and because they involve different dimensional breakdowns, do allow the identity of an individual cell member to be revealed. Such problems would restrict the level of disaggregation possible.

To illustrate the application of the code, the disclosure protection measures for the 2001 Census for England and Wales detail how confidentiality has been maintained. For example, for the release of Standard Tables, an area must contain at least 1,000 residents and 400 resident households, and for the release of Census Area Statistics (CAS), an area must contain at least 100 residents and 40 resident households.

7.7 Conclusions

This section covered the following area in relation to disaggregation:

• Reasons for requiring disaggregated data;
  
• Data confidentiality.

Disaggregation at a geographical area and indicator level is a concern to a number of users of health statistics. The main points arising from this survey indicate:

• Health statistics for small areas are required to monitor disease trends and population health, identify inequalities, target resources, compare performance, and plan services. These are crucial areas for local priority setting but at small area level could compromise confidentiality (e.g. where a very small number of HIV infections are reported);
  
• The demand for more disaggregated data by indicators, such as ethnicity and age, needs to be reconciled with data confidentiality and data protection issues;
  
• Producers of statistics are not always able to accommodate administrative changes in health boundaries immediately. For example, there is currently a lack of National Statistics for the population of PCTs. Again, the issue of data confidentiality and data protection may have an impact on the ability to produce data for such small areas though potentially the Statistics Commission might want to consider the benefits of blanket bans or targeted bans on "sensitive" information;
  
• There does not appear to be a consistent policy on datasets for different disease areas. For example, the amount of data available about Cancer is more detailed than for many other disease areas such as Diabetes.

Section 8: Inter-country Comparability

8.1 Introduction

Within the UK, comparative analyses can be undertaken to monitor the performance and efficiency of health care systems. Comparable data allow countries to study the experiences of others. A lack of consistency in health data may lead to misinterpretation. To avoid this problem, the National Statistics Code of Practice emphasises "coherence and common standards to maximise the value of available statistical and administrative sources" and requires that:

• 'Common statistical frames, definitions and classifications will be promoted and used in all statistical surveys and sources;
  
• Statistical systems will be designed in ways that maximise the potential to add value through data integration.'

The Protocol on Statistical Integration outlines the means of achieving integration among separate sources on the compilation of statistics, such as :

• 'The integration strategy for National Statistics will aim to provide a mutually consistent and supportive portfolio of complementary statistical resources derived from a variety of sources that would otherwise be disparate in timing, form and subject.'

The harmonisation of publications and outputs is also discussed in the Protocol on Data Presentation, Dissemination and Pricing.

Following these guidelines negates the problem of comparability for those statistics classified as National Statistics. However, comparability problems can still arise with non-National Statistics. These comparability problems have been identified not only at a UK level, but also at a European level. The European Union (EU) has undertaken a project to enhance the comparability of data by devising guides for countries' health care systems.

8.2 Aims

The aims of this section are to report some examples of uses of comparative health data and the particular concerns relating to the lack of comparable data within the UK raised by respondents to the questionnaire and interviewees. A number of case studies are then presented, to highlight areas where comparable data are available and some where it is not (or where we have been unable to find it).

8.3 Issues Arising from Interviews and Questionnaires

Seven of the 16 interviewees commented on the lack of comparability between data produced by each of the four countries of the UK, although for one of these organisations such disparity was not a concern. In the absence of consistent data, cross-country comparisons are difficult, if not impossible, and time consuming. For example, one interviewee commented that there seemed to be a greater volume of data for England. This meant that this organisation was forced to estimate parameters for the other three countries using data for England, even though this was not appropriate due to the differences in the demographies of the countries. Another interviewee suggested that it would be useful to have comparable small area statistics across the UK, which would allow comparisons between neighbourhoods across the four countries.

The comments from questionnaire respondents also confirm the need for consistent data (see Table 8.1).

Table 8.1: Comments on comparability from questionnaire respondents

Discussions with three producing organisations highlighted the following issues that may affect the comparability of statistics produced by each country:

• Discrepancies in the pace of technological progress;
  
• Different structures of the health care systems;
  
• Variations in the relationships between the producing/disseminating organisation and the health service;
  
• Different methods of delivering care.

Any of these issues may affect the organisation's ability to comply with the National Statistics Code of Practice.

An example of the impact of different structures is evident in the health care system in Northern Ireland. Northern Ireland has a unique classification system related to delivering integrated health and social care involving nine Programmes of Care (PoCs).⁵ These classifications are not used in England, Scotland or Wales. These PoC classifications are not comparable to similar distinctions used in the other countries. For instance, dementia cases are included in the Elderly PoC in Northern Ireland, but classified as Mental Illness in the other countries.

8.4 Examples of Comparative Analysis

Comparative analyses may be conducted at a country or regional level. For example, during this study, we learned that comparisons were drawn between Northern Ireland and the North East of England on the basis that these two areas have similar socio-economic profiles. The Executive Committee in Northern Ireland identified the need for 'some fundamental analysis of the need for, and effectiveness of, existing patterns of spending to inform decisions on prioritising expenditure'. In relation to health, this review involved 'an assessment of the needs for and effectiveness of the resources provided for HPSS (Health and Personal Social Services) in Northern Ireland' to 'ensure a better framework for policy decisions' and 'the achievement of better value for money'. Comparative analysis played a significant role in that review.

Data for the UK have also been used to examine geographic variations in health such as mortality, congenital anomalies, Cancer incidence, infant mortality, births, conceptions and abortions. A National Statistics publication was produced which reported data on demographic and socio-economic indicators, health status, determinants of health, health education and promotion, and health care resources and expenditure for the UK. The ONS is currently consulting with producers to identify changes in the comparability of data as part of the preparation for a second edition of this publication. Similarly, the APHO was commissioned by the Chief Medical Officer to analyse public health of English Regions. Expenditure on general medical services within the UK was compared in a publication produced by the Royal College of General Practitioners. This report also detailed international comparisons of health expenditure.

From discussions with producing organisations, it appears that efforts are made to ensure that data are comparable. For instance, the Surveys and Research Management Branch of Information and Analysis Directorate (IAD) in Northern Ireland answered that, in their surveys, they ask the same questions as those used in other UK surveys to maintain comparability. However, the producers also highlighted some areas where differences are apparent. The Drug and Alcohol Information and Research Unit (DAIRU) in Northern Ireland explained that drug-related information is submitted by all four countries within the UK to the DH. This information is then forwarded to the European Monitoring Centre for Drugs and Drug Addiction. Although the organisations submit this information in accordance with a standardised list of tables, different methodologies may have been used to compile these tables. The Family Practitioner Services Directorate in Northern Ireland also commented that information on dispensed prescriptions and definitions/labels differ. Finally, for workforce statistics, there are a number of differences in definition/coverage across various staff categories, which impact on the ability to undertake comparability studies.

8.5 Case Study

In their publication, entitled 'Regional Trends', the ONS note that 'there are differences between countries in the ways that waiting times are calculated', which means that 'comparisons between countries should be made with caution'. Furthermore, data from each of the four countries are used to present aggregated statistics on waiting lists and waiting times for the UK in the publication, United Kingdom Health Statistics.xxv However, caveats relating to this aggregated series stress that, due to differences in the approach to the compilation of waiting times, cross-country comparisons should be made with caution. Waiting lists and waiting times are classified as National Statistics. This section investigates the methods used, by each country, to compile waiting lists and waiting times.

The approaches to the compilation of waiting lists and waiting times are summarised in Table I.1 in Appendix I. The information in the following sections is based on responses from the DH, ISD, DHSSPS and Health Statistics Wales, who were asked to outline the methodological approach to collating waiting lists for their respective countries. Other information was obtained from the Websites of these organisations.

8.5.1 England

In England, waiting list information is collected from:

• PCTs on a responsible population basis (commissioner basis);
  
• NHS Trusts on a hospital basis (provider basis).

Responsible population refers to:

• Patients resident within the PCT boundary;
  
• Patients registered with GPs who are members of the PCT, but are resident in another PCT.

Patients who are resident within the PCT, but who are registered with a GP who is a member of another PCT are deducted from these figures. This measure also excludes patients who are resident outside England, and privately funded patients waiting for treatment in NHS hospitals. Patients, resident in England, but waiting for treatment in Scotland, Wales, Northern Ireland, abroad, and at private hospitals are included in commissioner-based lists, but not in provider-based lists. The difference between commissioner- and provider-based lists are shown in Figure 8.1.

Figure 8.1: NHS hospital trusts and NHS commissioners waiting lists

Source: DH, http://www.dh.gov.uk/assetRoot/04/07/29/46/04072946.pdf, date accessed: 11/06/04.

Figures based on provider-based waiting lists are typically larger than those derived from commissioner-based lists. The magnitude of this difference ranges from 1% and 3%.

Patients are included on the waiting lists if they are waiting for to be admitted to hospital either as a day case or ordinary admission. The following categories of patients are not included:

• Patients admitted as emergency cases;
  
• Outpatients;
  
• Patients undergoing a planned programme of treatment (e.g. a series of admissions for chemotherapy);
  
• Expectant mothers booked for confinement;
  
• Patients already in hospital but included on other waiting lists;
  
• Patients who are temporarily suspended from waiting lists for social reasons or because they are known to be not medically ready for treatment.

Waiting time commences from the date the clinician decides to admit the patient. For patients who were already offered a date, but unable to attend (known as self-deferred patients), waiting time is calculated from the most recent date offered. Self-deferred patients are included in the total waiting.

8.5.2 Northern Ireland

All patients awaiting elective inpatient or day case procedures are included on inpatient and day case waiting lists in Northern Ireland.⁶ There is a distinction between Board- and Trust-based information. The former exclude all patients living outside Northern Ireland and all privately funded patients waiting for treatment in Health Service hospitals, although these categories of patients are included in Trust-based returns.

The following categories are not included in the compilation of waiting lists:

• Patients admitted as emergency cases;
  
• Patients undergoing a planned course of treatment (e.g. a series of admissions for chemotherapy);
  
• Outpatients;
  
• Maternity;
  
• Patients already in hospitals but included on other waiting lists;
  
• Patients who are temporarily suspended from waiting lists.
  Waiting time commences from the date the clinician decides to admit the patient. The waiting time for self-deferred cases is calculated from the most recent date offered. Such cases are included in the total waiting.

8.5.3 Scotland

Waiting lists for Scotland are presented by NHS Board of Residence. Figures by NHS Board of Residence exclude patients with the following characteristics:

• An unknown area of residence;
  
• No fixed abode;
  
• From outside Scotland.

Prior to 1 April 2003, there were two waiting lists in Scotland - the True Waiting List (TWL) and the Deferred Waiting List (DWL). Those waiting on the TWL were mostly waiting for hospital resources to become available. However, this list also included some patients whose waiting time was affected by personal circumstances. The waiting time for patients on the DWL was significantly affected by their own personal circumstances. Due to the difficulties in distinguishing between these two lists and consequent inconsistencies in entering patients on the appropriate list, it was decided that the DWL should be discontinued. Patients who should have been placed on the DWL were to be added to the TWL, together with information on the reason for the delay in treatment. This information was provided using an Availability Status Code (ASC), which is presented in Table 8.2.

Table 8.2: Definitions for ASC

Source: ISD, Changes in the recording of waiting list information in Scotland and the impact on published Statistics - National Statistics Notification of Change of Methodology.

The waiting time for inpatient or day case admissions is calculated as the difference between the date the decision was made to admit the patient (the "waiting list date") and the actual date of admission. This more recent definition differs from one published in 2000 in which the start date was actually the date the patient was entered on the waiting list. This date did not necessarily coincide with the date of the decision to admit the patient.

8.5.4 Wales

In Wales, patients are included on the waiting list if they are awaiting an admission from the active consultants' waiting list on the final day of the month. The following cases are included:

• Booked cases;
  
• All patients waiting for their first treatment for the particular condition;
  
• Patients whose planned admission is delayed;
  
• Self deferrals.

Exclusions from the list include:

• Patients waiting for planned admissions for subsequent treatments;
  
• Postponements due to medical reasons;
  
• Emergency admissions and cases where admission is required immediately on medical reasons, but the patient requests a delay;
  
• Transfer cases (i.e. patients already occupying beds in the hospital but waiting for admission to another department of hospital).

8.5.5 Summary

This Section has demonstrated the complexity of determining if waiting times and waiting lists statistics for the UK are comparable - a problem which is influenced by statistical factors and also compounded by policy considerations. Publications that contain UK waiting times and waiting list data contain caveats warning that different methods have been used.

There appears to be consistency between the four countries in the collation of waiting lists. The previous sub-sections have shown that, for example, the treatment of patients undergoing a planned treatment programme, or cases of self-deferral, are included in the waiting lists in each of the four countries. On the calculation of waiting times, all four countries appear to use the same start and end points. However, it is not clear how the countries treat patients who have been deferred or suspended from lists in the calculation of waiting times. Consequently, publications that report waiting list/time figures for all four countries have provided a cautionary note indicating that differences exist in the calculation of waiting times across the four countries. Greater comparability may be achieved with the introduction of patient-centred collection as this would allow more detailed and flexible analysis of data.

8.6 CONCLUSIONS

Comparative analysis of the countries within the UK is a key area of work for a wide range of organisations and individuals. These include public health observatories, voluntary organisations, members of parliament, central government departments, local government authorities and the Royal Colleges. Moreover, the issue of comparative data also arises for international comparisons within the EU and the OECD. The main concerns from our survey, on comparative health statistics are:

• There are inconsistencies in the availability of comparable key health statistics for the four countries of the UK. This review concentrated on the specific example of waiting lists;
  
• he importance assigned to inter-country comparisons and the availability of data varies depending on the perspective of the organisation and the use made. For example, a voluntary organisation, covering the UK, places a high level of importance on the ability to make inter-country comparisons and is obviously restricted to the availability of health statistics within its field of interest. However, a central department concerned with comparisons in a wide range of diseases and services may place less emphasis on some if comparable health statistics are available for others.

The lack of comparable health statistics in certain areas has arisen due to a number of reasons including:

• Discrepancies in the pace of technological progress;
  
• Different structures of the health care systems;
  
• Variations in the relationships between the producing/dissemination organisations and the health service;
  
• Different methods of delivering care.

The implications of changing definitions for key health statistics, such as waiting lists, within the four countries should be carefully considered. While such changes may enable cross-country comparisons to be made easily, the ability to undertake time series analysis within a country may be adversely affected. The production of a list of all health statistics that lacked comparable inter-country data was not within the remit of this review. However, this is an area that would benefit from further research.

There are a number of possible solutions which could be implemented to ensure that comparable data are collected across the four countries of the UK, such as definitional changes or the use of consistent units. Further research is required to examine the most effective and efficient method for harmonising data from the four countries.

9.1 Introduction

There is a National Statistics Code of Practice, which sets out the key principles and standards that official statisticians are expected to follow and uphold. This Code of Practice refers to notifications about forthcoming releases of statistics. In particular, the following extracts from the National Statistics Code of Practice , Protocol on Release Practices, relate to the release of statistics:

• 'Release arrangements will be open and pre-announced. Release will be orderly and as early as possible after compilation';
  
• 'All information relating to the release of National Statistics must be publicly available. Every organisation which releases National Statistics will maintain and publish a statement describing its release practices and how they conform to the 'Code of Practice' in general and to this protocol in particular';
  
• 'A programme of National Statistics releases will be maintained and regularly updated for a full year ahead'.

The ONS, the DH and equivalent departments in the devolved administrations all publish details of forthcoming releases on the web. However, they do not all publish details for the full year ahead, although, as stated above, this is required for National Statistics. These organisations also publish statements of compliance with the Release Protocol on their Websites.

9.2 Aims

Communication issues can cover a wide range of problems. The aims of this section are to explore the following:

• Notification about forthcoming release of statistics;
  
• Mechanisms for communication between users and producers.

The issues raised within the interviews and questionnaire surveys can be readily aggregated into these main categories.

9.3 Issues Arising from Interviews and Questionnaires

Two interviewees mentioned the issue of notifying users about forthcoming publications. These respondents suggested that a calendar of forthcoming publications for all health statistics may be utilised as a means of achieving this. One of these interviewees also made the general comment that there should be greater communication between producers and users of health statistics. Another interviewee similarly commented that it would be useful to know when new datasets are published, but commented that the ONS was quite good at publicising this information.

One interviewee suggested that by having press conferences, rather than concentrating its releases on a small number of days, the DH could raise public awareness about statistical releases.

The questionnaire did not directly ask about the issue of communication with producers of National Statistics. However, as Table 9.1 shows, a number of respondents did provide specific comments relating to communication issues.

Table 9.1: Comments on communication from questionnaire respondents

The responses highlighted by the questionnaire and interview survey generally related to requirements for a schedule and timeframe of forthcoming publications. This would appear to indicate that not all users of health statistics are aware that schedules of forthcoming releases of National Statistics are published.

9.4 Mechanisms for Communication

Establishing communication between producers and users can be difficult because the availability of data on the Internet implies that producers are not aware of who is using the data. In light of this, this section examines the following mechanisms:

• Communications between users and producers;
  
• Communications between users.

9.4.1 Communication between Users and Producers

A number of forums exist for communication between users and producers of statistics. The main channels are group communications and individual communications.

Group communication

The National Statistics Website has an area set up for the Statistics User's Council. Within this area notes explain that:

'The Statistics Users' Council has provided a forum for the exchange of information, ideas, expertise and views between users of statistics, National Statistics and the Statistics Commission, ever since its formation in 1970. The Council has a number of roles, which includes organising conferences and seminars, encouraging the formation of new User Groups and providing a channel for representation of users' views.'

This Website lists a number of papers/articles that users have posted. However, it is not clear how well this forum is used. In June 2004, the Website indicated the information was last updated in May 2003; this suggests the use by those posting information may be fairly infrequent.

In the past the DH and other organisations have hosted a set of workshops on different aspects of health statistics. For example, the last workshop, which took place in November 2003, was organised between the DH and the HSUG. Another such workshop is scheduled to take place in 2004. These provide a forum for dissemination and discussion.

The ONS has a number of routes through which it maintains regular contact with users. These can be generic (e.g. the Statistics Users' Council, Royal Statistical Society); subject related (e.g. HSUG, British Society of Population Studies, Society of Social Medicine); or source related (e.g. Census user groups). There are a number of means through which the ONS maintains contact with these groups, such as participating in committee work and attending information or discussion meetings. These groups may also respond to ONS consultations on proposed changes to sources or outputs. Moreover, the ONS also organises conferences to communicate directly with the main users of its health statistics. These meetings have largely been attended by users in the Health Services, although users from local authorities and other agencies also attend.

In Wales, the following groups provide a means of communication between producers and users of statistics:

• The Welsh Statistical Liaison Committee, which was established for local authority users of data produced by the Welsh Assembly;
  
• The Technical Working Group on PSS statistics, which considers data requirements placed on local authorities. Public sector users, such as the Audit Commission and CIPFA, are represented on this Group;
  
• The Welsh Health Information Managers Group, which was established as a forum for information staff in the NHS.

The DAIRU in Northern Ireland has discussions with a number of groups including:

• The Northern Ireland Drugs and Alcohol Strategy Team;
  
• The Information and Research Working Group (one of the six Working Groups set up under the Joint Implementation Model for the Drug and Alcohol Strategy);
  
• The Drug Misuse Database Project Management Group;
  
• The Needle-Exchange Co-ordination Group.

The Regional Information Branch (RIB) of IAD in Northern Ireland is represented on four Nations Clinical Indicators Group and has also joined the UK Data Standards Forum. The Branch also coordinates a number of groups for representatives from the Health and Social Services Boards and Trusts.⁷ These groups meet on a quarterly basis. The Surveys and Research Management Branch also conduct their activity with input from a number of groups that have representation from both inside and outside the DHSSPS.

Individual communication

The producers of health statistics all have links from their Websites to helpdesks or information desks to which enquires can be addressed or comments directed. Most of these sites also display telephone and fax numbers, as well as postal addresses. These links are to encourage individual communications.

Methods of informing users of forthcoming releases

The publishers of National Statistics employ a number of different measures to inform users of forthcoming publications.

For example, the National Assembly for Wales Website offers users of health statistics the opportunity to sign up to an e-mail alert system. On signing up users specify which statistics they are interested in and the system sends the user an e-mail notification when their chosen statistic is released.

The ONS pre-announces all its releases through updates on the National Statistics website. The ONS produces a publication called 'Health Statistics Quarterly'. This journal is published in February, May, August and November of each year and includes a section that details recent related publications. It is available both on the web and also in hard copy. Access to online issues is free. National Statistics Updates is the monthly Release Dates diary published by the ONS on behalf of the Government Statistical Service (GSS) and other organisations. It contains dates (up to four months in advance) of all National Statistics releases and where information is available of other releases by government departments and organisations. Releases are listed in date order. The diary is published monthly in hard copy and weekly on the Website.

The DH uses the Press Release area of its Website to inform readers that particular datasets have become available. Each news item has a link that takes the reader to the newly published material.

The Website of Information Services Division (ISD) Scotland has special pages set aside that display lists of, and links to, current and forthcoming releases. These can be reached via a drop down menu that is situated on the top of all their web pages. The IAD of the DHSSPS in Northern Ireland have a section on their Website which details publications and release dates. Furthermore, each publication normally includes the date of the next release in the series.

9.4.2 Communication between Users

A number of user groups exist for health statistics users. Fifteen percent of questionnaire respondents (34 respondents) said that they were members of statistics user groups. Table 9.2 lists these groups. Membership of these groups may reflect respondents' professional qualifications or areas of interest.

Table 9.2: User Groups identified in the questionnaire survey^a

Note: a Although respondents were asked specifically to name the statistics user groups to which they belonged, this list includes groups which are not primarily statistics user groups, and also those not directly related to the area of health.

9.5 Conclusions

This section has explored three areas of communication:

• Notification about forthcoming release of statistics;
  
• Mechanisms for communication between users and producers;
  
• Issues around the organisation of the production of statistics.

The main issues arising from this research regarding these communication areas can be summarised as:

• Not all users are aware that producers of National Statistics publish a timetable of forthcoming releases for the year ahead, in accordance with the National Statistics Code of Practice;
  
• The structure of Websites can mean that it is not always immediately obvious to the inexperienced user that details of forthcoming statistical releases are published. Similarly it is not always clear to the user where to find the schedules;
  
• A number of different forums exist to enable communication between users and producers of statistics. These include the Statistics User Council and workshops run by the DH;
  
• Communication between users can be carried out formally through membership of a statistics user group. Membership of a statistics user group was not widespread amongst respondents to the questionnaire survey although a number of different groups do exist;
  
• The availability of health statistics on the Internet means that it is difficult to know who users are.

10.1 Introduction

The availability of National Statistics is determined by the following areas:

• Access to the source or producer of the data;
  
• ccess to the required dataset;
  
• Format in which data is presented.

This section is concerned with the ability to gain access to the source of data and the dataset itself. In addition to these areas, access to metadata about the relevant dataset may also be an issue for users. This is discussed in Section 11. The impact of data protection on accessing statistics is discussed in Section 7.

The Statement of Principles of the National Statistics Code of Practice states that:

'Access to National Statistics will be fair and open'.

In particular, the Statement of Principles requires that:

• 'National Statistics will promote equality of access';
  
• 'Data will be presented to a standard that clearly and accurately expresses the contents to the widest possible audience, with choice and flexibility in the format where possible'.

To achieve these objectives, the Protocol on Data Presentation, Dissemination and Pricingix (2004) names the web as the 'primary means of providing general access to National Statistics'. However, the Protocol also recognises that since access to the web is not universal, other forms of dissemination will also be maintained (e.g. paper versions, CD-ROMs). The Protocol also recognises the relationships between all producing and disseminating organisations and requires them to 'integrate and harmonise their publications and products in accordance with users' needs and give users easy access to related statistics through common gateways or interlinked Websites'.

In their compliance statements, all producers have detailed how they have met the requirements set out in the Statement of Principles and Protocols. Specifically, the compliance statements mention the format of releases (hard and/or electronic copies) and confirm that the organisations endeavour to make the different formats of releases available at the same time.

10.2 Aims

The aims of this section are to look at the following areas:

• What are the issues regarding accessibility raised during discussions with, and a survey of, users of health statistics?
  
• What are the existing and future means of accessing data?

10.3 Issues Arising from Interviews and Questionnaire Survey

10.3.1 Access to Producing Organisations

In general, the move to web-based data was viewed positively by both interviewees and survey respondents (e.g. 'web-based access has clearly made life easier'). Only one organisation mentioned that there were also disadvantages with disseminating information through the Internet, the main drawback being that people without Internet access could not obtain information. However, the magnitude of this problem is not clear and should be mitigated by the producers' confirmation that releases are available in other formats.

Three respondents explained that access difficulties arose due to the large number of sources. Searching multiple sources is not only complex but also time consuming. One questionnaire respondent suggested that a centralised web-based repository would greatly simplify the process of obtaining electronic data. Similarly, one interviewee also advocated the use of a 'central enquiry point'. According to this interviewee, this system had worked well in Wales and Scotland, although there may be difficulties in implementing such a system in England due to the larger volume of statistics produced. In addition, a member of the HSUG expressed concern that 'we will not continue to have what we now have, once a substantial proportion of the Department's statistical work is hived off into the Information Centre'. It should be noted that this view is a conjecture made anonymously and, therefore, cannot be substantiated by the authors. One questionnaire respondent mentioned that 'it would be nice to have a breakdown "crib sheet" detailing where to find different statistics'. In their opinion, this would be particularly helpful to those new to the health environment.

Table 10.1 lists comments received from interview and questionnaire respondents in relation to the Websites of producing organisations.

Table 10.1: Website comments

Producers were asked if they kept a record of users' queries that may arise from the difficulties users encountered in accessing Websites or data. Most producers maintain a log of users' queries, although this record is not always held centrally. One producer commented that, due to the various routes of communication available to users (e.g. e-mail, telephone, personal communications), it is not possible to hold this log centrally. Another producer stated that records of enquiries are kept, but not centrally. This producer also indicated that many regular users have direct contact with individual members of staff on an ongoing basis. There is no standard method of logging ongoing contacts.

10.3.2 Dataset Accessibility

Table 10.2 reports areas where respondents specifically reported access problems. These problems have been classified into those which are:

• Difficult to access;
  
• Not available;
  
• Not available due to data confidentiality/sharing constraints.

As some of the comments in Table 10.2 show, there are inequalities of access to some health statistics. For example, the Compendium of Health and Clinical Indicators is only available to users within the NHS. However, a number of users, outside the NHS, commented that they would find this publication useful. Moreover, one interviewee mentioned that local authorities were only to be granted restricted access to Vital Statistics, an ONS publication.

Interestingly, very few respondents mentioned cost as an issue affecting the accessibility of health statistics.⁹ The free availability of publications on the Internet may mean that cost is not a significant barrier to access.

Table 10.2: Areas with problematic access

Note: ^a One interviewee explained that the ONS were reducing the Vital Statistics dataset to which local authorities have access.

10.3.3 Format

Generally, interviewees obtained data from the Internet, paper publications, CD-ROMs or directly from the producing organisation. One interviewee suggested that online query tools and the generation of online maps could be used to access local data. This user cited that the ONS and Eastern Public Health Observatory have embryonic mapping tools.

One interviewee commented that data were currently presented in an 'inefficient format', which was aimed towards inexperienced users. This respondent proposed that it would be helpful if data were presented in a single database rather than as a series of spreadsheets, thereby simplifying the data extraction process. Another organisation made a similar suggestion about the use of a single spreadsheet, rather than individual files. This interviewee also added that the DH could 'add value to its own collections by making the information more user-friendly'. If the DH adopted a more user-friendly approach, then this user's organisation would be able to use the service routinely.

To aid with tight deadlines, one user suggested that it would be useful to have a comprehensive online interactive service.

10.4 Examples of Accessibility

This section discusses how selected series of health statistics can be accessed. The health statistics examined include some of those mentioned by respondents. The following series were considered:

• Vital Statistics;
  
• 
  
• Compendium of Clinical Outcomes;
  
• HSE;
  
• Abortion statistics.

Table 10.3 reports the location and means used to access these series. This shows that all of the statistics selected are available electronically either in part or in full. However, not all of these series are accessible to the general public. For instance, the Compendium of Clinical and Health Outcomes is only available within the NHS. Similarly, one interviewee thought that access to the ONS publication, Vital Statistics, was to be restricted for local authorities. The ONS was asked to confirm the latter point. In their response, a contact at the ONS explained that the amount of detail released to different users is related to what can be disclosed to them under legislation. For instance, the Registrar General currently has limited powers to release information at an individual level from the registers for which he is responsible. No identifiable information collected under the Population Statistics Acts can be released at all. There is no relevant legislation that allows the release of information held on the register to local authorities to manage specific services. Therefore, local authorities are entitled to access the same statistics as other users not covered by legislation. However, information held on the registers can currently be made available to some users for specific purposes. For example, the Social Security (Notification of Deaths) Regulations 1987 allow the release of information from the death register to the Department of Work and Pensions for list cleaning purposes. The ONS is aware of the difficulties for local authorities in not having access to this information and is currently pursuing changes to the legislation through the Regulatory Reform process.

Typically, the data are presented in a PDF document, a CSV file, or as Excel tables. NHS Scotland publish an interactive list of statistics that they produce. By clicking on the relevant statistic on this list, the user can gain access to PDF or Excel versions of the data.¹⁰

Table 10.3: Examples of accessibility of selected health statistics

Notes:

^a As the Abortion Act 1967 does not extend to Northern Ireland, abortion is illegal in this country. However, data is available on abortions to Northern Ireland residents, which are carried out in England and Wales. These data are produced by the ONS in the Abortion Statistics series.
^b Following communication with the DH, we understand that there are plans to develop a web-based version, which will be available to users outside the NHS.

10.5 Means of Accessing Data

Given the comments from interviews and the questionnaire survey, this section examines a number of ways of presenting data which, through user interaction, allow improved flexibility to meet users' needs. In particular, this section focuses on data cubes. A data cube is defined as 'a multidimensional representation of data'. Each dimension in the database represents a parameter or variable and each cell represents the measure of the statistic. There are a number of advantages to presenting data using this method. Principally, data cubes may be a more efficient way of disseminating information as they permit rapid retrieval of data in a format which is most appropriate for the user. Whilst this addresses the issues raised by users regarding the presentation of data, it does not deal with the other accessibility issues of producers' Websites and availability of datasets. The following are examples of data cubes:

• European Health for All database allows users to download longitudinal data on more than 600 health indicators by countries or country groupings;
  
  
• StatLine, operated by Statistics Netherlands, is a central database which permits users to customise their data requests.

This section focuses on the particular example of the Interactive National Hospital Morbidity Data in Australia. Box 10.1 outlines the various steps in extracting data from the Interactive National Hospital Morbidity database and Figure 10.1 shows the breakdown of ICD-9 used in this database. This shows a user has an extensive range of options to select and can tailor analysis to their own requirements.

Box 10.1: Case study

Figure 10.1: Breakdown of principal diagnosis for ICD-9-CM classification

Figure 10.2: Breakdown of principal diagnosis for Diseases of the Respiratory System

10.6 Conclusions

This section has discussed accessibility issues around health statistics. In particular:

• Access to the source or producer of the data;
  
• Access to the required dataset;
  
• Format in which data is presented.

The main points arising within this section can be summarised as:

• The move to web-based dissemination of data was generally viewed positively. However, the format of some data on the web was criticised for being 'inefficient'. The development of web dissemination alongside more traditional methods has meant data are often presented in separate tabular format in individual spreadsheets rather than as one complete file. Demands for all formats still appear to exist, which is indicative of the range of user requirements;
  
  
• Improved access to data can be achieved by the use of data cubes. Data cubes permit rapid retrieval of data in a format that is determined by the user. The Interactive National Hospital Morbidity Data in Australia provides an example of a data cube;
  
  
• Access difficulties can arise due to the large number of sources of statistics. Searching multiple sources is complex and time consuming. The development of a central enquiry point was suggested to overcome these problems. In establishing such an enquiry point, a number of issues need to be addressed including the method of delivering information (e.g. central search facility, an e-mail or telephone helpdesk), and the geographical and statistical coverage;
  
  
• Various series of health statistics have different access restrictions. For example, a 'Compendium of Clinical and Health Outcomes' is only available within the NHS. Such restrictions confine assessments of the performance of the NHS by external organisations. The reasons for limiting access for certain data series were not addressed in this review.

⁸A map of the Health and Care theme of the ONS website is contained in Appendix J. This demonstrates the large number of categories within this section.

11.1 Introduction

Metadata is the term used to describe the summary information or characteristics of a set of data. Essentially it is a record of the Who, What, Where, When and How of the data. It shares many similar characteristics to cataloguing records used in places such as libraries, museums and archives. As part of the National Statistics Code of Practice, the Protocol on Data Management Documentation and Preservation describes metadata as 'data about data'. This document also explains that metadata is 'the information that is made available to users in order to improve their understanding of data'.

A metadata record comprises all relevant elements that describe the information source. The section of the National Statistics Code of Practice that details the Protocol on Data Management, Documentation and Preservation.xxvi states that 'Statistical resources will be documented in a standard manner. Documentation will include easily accessible metadata designed to foster greater awareness, usability and understanding of the data and enhance their functionality'.

Metadata records displayed by producers of health statistics come in a variety of formats. Those produced by the ONS are formal in structure and consist of a table consisting of two columns. In the first column is a field name, e.g. 'Size of data collection' and in the second is information relating to that field (an example is included in Appendix K). Other metadata records may be less rigid in structure, for example, that published by Northern Ireland Statistics and Research Agency (NISRA) for 2001 Census Data (see Appendix L).

Metadata records are generally reached via links from the data to which they refer.

11.2 Aims

The use and demand for metadata is increasing. The aims of this section are to look at the following areas:

• What is metadata?
  
• How is metadata currently accessed?
  
• What are the perceived benefits of metadata?
  
• What policies do the producers of datasets have on metadata

11.3 Perceived benefits of metadata

A clear understanding of the methodology used to compile datasets is important when assessing and interpreting data. The following examples highlight where it is important to know how data were compiled.

Quality

Details such as the extent of geographical coverage, population and the size of data collection will allow the user to assess data quality.

Timeliness

Data collected a number of years ago may no longer be relevant, due to a changed situation.

Generalisability

In the case of a survey, it is important to know how the sample was selected. This knowledge will enable the experienced user to judge how representative its findings might be.

Comparability

Knowledge of compilation is also important when undertaking comparative analyses. One interviewee mentioned that, within a particular dataset, the definition of colorectal Cancer had changed, making it extremely difficult to compare data. Metadata allows this assessment to be made.

Limitations

Knowledge of information, such as periods during which there were breaks in data collection, help avoid false conclusions being drawn from the data.

Interpretation

Unclear terminology may lead to poor interpretation, for example, lack of clarity in the definition of a disease or condition. One interviewee mentioned that some inexperienced users were unclear about the difference in definition between incidence and prevalence of a condition. Misunderstandings of this kind can lead to misrepresentation of findings drawn from data. Also, lack of information on subjects such as population included, or geographical area covered, could lead to the data being misinterpreted.

11.4 Issues Arising from Questionnaires and Interviews

11.4.1 Questionnaire Responses

Within the questionnaire, respondents were asked to what extent they agreed or disagreed with two statements that related to metadata. The statements sought respondents' opinions on the ease of access of information on compilation and the compilation information actually supplied for datasets.

11.4.2 Ease of Access to Information on Compilation

Respondents were asked to state whether they agreed, disagreed or neither agreed nor disagreed with the statement 'I can easily access information relating to the health statistics I use (e.g. explanatory notes, methodological descriptions, etc.)'. Out of 210 responses 92 (44%) agreed, 75 (36%) neither agreed nor disagreed and 43 (20%) disagreed with this statement.

Only 10% of questionnaire respondents supplied comments on ease of access to information on compilation. These 18 comments are detailed in Table 11.1 and fall into three categories, namely comments in support of the view that information was easy to access, comments saying that information was difficult to access and the remainder which mainly documented views on the quality of information on compilation.

Table 11.1: Comments: Ease of access to information on compilation

Access is easy

Mostly easy to access

Easy to access once I have found the data

Yes - once I have found/accessed the appropriate Website

Yes

Difficulty with access

Can take a lot of effort at times. Location on some Websites obscure

Often very difficult to find what you are looking for - often too much information to search through

Some sites are not user-friendly

Not easy to access first time round

Not always easy to find or understand

Yes, though easily is not always the case

Other

These are virtually non-existent - probably because the detail required in the query implies understanding anyway - for methods. It has been left up to me to enquire

Methodology is not always a major concern of HSS publications

Not always available

The amount of background detail provided on data files is often variable - some are very poor

This is much better with the Observatories now but could be improved even more

Mostly use figures from National Assembly for Wales Website - not many methodological notes or explanatory notes there

Varies depending on statistic

COPPISH datasets are well documented

11.4.3 Information on Compilation

The questionnaire asked respondents to state whether they agreed, disagreed or neither agreed nor disagreed with the statement 'Information on compilation of health statistics I use is sufficiently clear and at an adequate level of detail'. Out of 211 responses, 94 (45%) agreed, 72 (34%) neither agreed nor disagreed and 45 (21%) disagreed with this statement.

Only 7% of respondents to this question submitted comments. These 15 comments are detailed in Table 11.2. It can be seen that most of the comments are very general in nature and directed at gaps in information on compilation.

Table 11.2: Comments: Information on compilation

Respondents' comments

Not always clear or readily accessible as to how statistics have been compiled

Older data much less complete

Not for the professional researcher and often too vague to replicate or spot anomalies

Not always available

The amount of background detail provided on data files is often variable - some are very poor

For example, I got the ICD-10 - ICD-9 Bridge coding from a lucky break with the Scottish Executive

Inexact definitions, e.g. reducing inequalities in smoking

Access to a greater level of detail and sometimes source data would be useful

Sometimes data are provided rather than information

Varies depending on statistic

Many statistics are too detailed in definition (not necessarily too complex) to be taken seriously by the public

Very variable

Now I know about them

Some of the forms are available but info on compilation across the board is generally not

11.4.4 Summary of Questionnaire Responses

In general, respondents to the questionnaire survey seem satisfied both with current methods in place to access information on the compilation of health statistics and with the detail contained within the compilation information. However, we note that respondents are likely to be relatively experienced users of health statistics and, thus, their views are not necessarily representative of all users. On the other hand, it may be that experienced users would identify problems most readily.

11.4.5 Interview Responses

The majority of interviewees commented on the approaches used to compile statistics. Specifically, a subgroup of these viewed knowledge of methodology as important in assessing data quality and undertaking comparative analyses. One producer commented that, generally, users appear to be more interested in coverage and the level of disaggregation than compilation. One user, who did not have detailed knowledge of the methodological approach, (even though they considered this important in determining the quality of data), confirmed this. One organisation commented that knowledge of compilation depended on the data source. This user suggested that clear definitions were vital in understanding the limitations of the data. The definitions reported by ONS were deemed to be 'good' by this interviewee. According to one respondent, information on the methodological approach to compiling data was available, but was not necessarily easily accessible to inexperienced users.
Two interviewees expressed concern about the potential for users to misunderstand to what the data relate. One of these organisations stated that when terminology is unclear, there may be poor interpretation of the data. The other organisation mentioned that users do not always understand the statistical techniques applied to data and consequently, do not take account of this in their interpretation and analysis of the data. Moreover, one interviewee suggested that simpler and more understandable definitions of health statistics were required.

One interviewee commented that the notes attached to the HSE were 'clear and highlighted caveats'. Similarly, another participant answered that metadata should be available for all datasets. Another respondent thought that the metadata on the ONS Website was uninformative and that in general, although the methodology used to compile the data is well documented, it is hard to find. This participant also suggested that a guide (in paper format) should be published and be easily accessible to ensure that users have access to data definitions and compilation methodology.

11.5 Policies that the Producers of Datasets have on Metadata

Producers from England, Northern Ireland, and Wales confirmed that they produce metadata. In their responses, producers explained that the main objective of metadata was to provide background information to aid the understanding and interpretation of data. Generally, however, there is no standard metadata template used by producers. In Wales, metadata was disseminated through StatsWales. Another producer commented that they contributed to the NS Online metadata repository (formerly StatBase). The ONS recognised that there was inconsistency in the way in which metadata presented for its products. However, as part of a major Statistical Modernisation Programme, systems will be developed to ensure consistency in the presentation of metadata and improved search facilities, together with documentation of sources as a series progresses to first release.

11.6 Conclusions

This section has discussed metadata. In particular:

• What is metadata?
  
• The perceived benefits of metadata;
  
• he policies that producers of datasets have on metadata.

The main points arising within this section can be summarised as:

• Metadata was generally viewed positively;
  
• Respondents to the interview and questionnaire survey recognised the benefits of metadata and advocated its use. In particular it can be used to assess: quality, timeliness, generalisability, comparability, limitations and it can also assist with interpretation when undertaking analyses;
  
• The format and content of metadata varies by type of publication and producing organisation. The use of a common approach, such as StatBase, ensures that consistent information is recorded.

Section 12: Case Studies

12.1 Introduction

This section presents two case studies which examine some of the claims of users on the five issues for further investigation. In particular, the purpose of the case studies is to assess the accessibility, disaggregation, comparability, metadata and communication for two disease areas. Data on Cancer and Diabetes were selected as case studies for the following reasons:

• Both disease areas are key policy areas, as indicated by their own NSFs;
  
• Cancer is the only disease area in the UK where regional and national registers have been set up. Therefore, it is interesting to compare the Cancer data with that available for Diabetes, where a register does not currently exist.

12.2 Case Study I: Cancer

The purpose of this case study is to examine one publication containing data on Cancer incidence and mortality in relation to the issues selected for further exploration. The publication of interest is the electronic version of Cancer Trends, which is classified as a National Statistic and is available from the National Statistics Website. This is a publication of 45 datasets, which contain data on Cancer incidence and mortality. The characteristics of this data series are summarised in Table 12.1.

Table 12.1: Summary of characteristics of Cancer tends

12.2.1 Accessibility

Cancer Trends data are available from the National Statistics Website. It is interesting to note that searching for 'Cancer incidence' or 'Cancer mortality' on the National Statistics Website will yield data results which refer to specific datasets within the Cancer Trends group, rather than to the entire group of datasets. Upon selecting a particular dataset, the user can view the table on screen or download it to a CSV file. Furthermore, the user may customise the contents of the table according to their requirements.

12.2.2 Disaggregation

Data from the Cancer Trends series are disaggregated by geographical location and indicator. Bottom-up and top-down analyses are possible since data are presented for the UK, the four countries within the UK, and regions within England. The system of regional registries means that the latter level of disaggregation is possible.

Furthermore, although further disaggregation does not appear to be available at a national level, Table 12.2 shows that statistics on Cancer incidence in Scotland are produced according to a greater level of detail. Similarly detailed data are collected in Northern Ireland.

Table 12.2: Level of detail of Cancer data in Scotland

Sources: ¹ISD Scotland, http://www.isdscotland.org/isd/files/Cancer_all_types_inc_f.xls and http://www.isdscotland.org/isd/files/cancer_all_types_mort_d.xls, date accessed 21/06/04.

Similarly, Cancer data for the East of England have been further disaggregated by Strategic Health Authority (StHA) and PCT in a publication produced by the ERPHO, based on data collected by two regional registers.

Data in the Cancer Trends series are presented on the types of Cancer classified according to the standard ICD-10 codes. Further breakdowns are available by gender. However, in these datasets, data are not available by age, ethnicity or socio-economic groups. Some of this detailed information is collected from regional Cancer registers, although it does not appear to be aggregated to a national level. In particular, the type of information collected by one regional register is reported in Table 12.3. Moreover, information on treatment is collated at a regional level, but not at a national level.

Table 12.3: Detailed data collected by regional Cancer registries

Source: The Northern and Yorkshire Cancer Registry and Information Service (NYCRIS), www.nycris.org.uk/about_us/nycrabou.htm; date accessed: 15/06/04.

In conclusion, there is some disparity between the data published on a national level and that available regionally. It seems that there is scope to replicate at national level the detail produced at regional level.

12.2.3 Comparability

Collecting information through registers has meant that data are comparable. Indeed one of the main aims of the regional registers is to 'ensure a uniform process for registering Cancers region-wide, which will deliver timely, comparable, high-quality data'. Moreover, using standardised classifications (such as ICD-10 codes) also provides definitional clarity and helps to ensure that data are consistently compiled.

12.2.4 Metadata

Background information is available on each dataset within the Cancer Trends series. Tables 12.4a and 12.4b show examples of the detailed background information available for the series representing Cancer incidence amongst females.

Table 12.4a: Background information on Cancer incidence for females

Table 12.4b: Background information on Cancer incidence for females

12.2.5 Communication

As Tables 12.4a and 12.4b show, the background information contains contact details (telephone and e-mail) which provide users with a contact for queries, e.g. accessibility/access, editorial or content issues.

12.3 Case Study II: Diabetes

The purpose of this case study is to look at the information that can be accessed on Diabetes through a search of the National Statistics Website.¹⁴

The only datasets relating to Diabetes that are available via the National Statistics Website are those for prevalence. The most up-to-date figures (located by us) available are those for 1998. Unlike the statistics disseminated on this Website for Cancer, those for Diabetes, which is also an area of health that is perceived to be important to the nation, are not classified as National Statistics.

12.3.1 Accessibility

The datasets on prevalence of Diabetes can all be accessed in electronic format via the National Statistics Website. An electronic copy of the complete publication 'Key Health Statistics from General Practice 1998' is also available. This contains tables on the prevalence of Diabetes by age, gender and calendar year (1994 - 1998).

12.3.2 Disaggregation

Within the National Statistics publication, 'Key Health Statistics from General Practice 1998', data on the prevalence of Diabetes distinguish between those who are, and those who are not, treated with insulin. Thus, this generates proxy values for prevalence of type I and type II Diabetes respectively. Table 12.5 shows the areas of disaggregation available for the two datasets.

Table 12.5: Disaggregation of two Diabetes datasets

Notes:

^aONS area classifications are suburbia; rural areas; rural fringe; industrial areas; middling Britain; prosperous areas; inner/deprived city; established owner-occupier; metropolitan professionals; lower status owner-occupier; mature populations; and deprived industrial areas.
^bThere are five deprivation categories ranging from Q1 representing least deprived to Q5 for most deprived.

Three other National Statistics Publications also publish data on Diabetes:

• Social Focus in Brief: Ethnicity;
  Prevalence of Diabetes: by ethnic group and sex;
  (Spreadsheet, last update 11/12/02);
  
• Social Trends 31;
  Prevalence of insulin treated Diabetes: by gender and age 1994 - 1998 (Spreadsheet, last updated 09/05/01);
  
• Social Trends 33;
  Prevalence of diagnosed Diabetes: by sex and age, 1998;
  (Spreadsheet, last update 29/01/03).

12.3.3 Comparability

Although a regional breakdown of prevalence of Diabetes is produced for England and Wales, a comparison of prevalence of Diabetes that includes Scotland and Northern Ireland is not (at least immediately) possible, as this information is not available either on, or via links from, the National Statistics Website.

12.3.4 Metadata

Metadata is produced for all the datasets on prevalence of Diabetes that are listed above. It can be accessed electronically through links associated with each dataset.

12.3.5 Communication

Information on who to contact in the event of a query relating to any of these datasets is given within the metadata held for each dataset. Several contact methods are given, including telephone number and e-mail address.

There is no indication on the National Statistics Website that up-dated figures for the prevalence of Diabetes are to be produced in the near future. However, as these data are not classified as National Statistics there is no requirement for their forthcoming release to be announced.

12.4 Conclusion

This section has examined the differences in the main issues of accessibility, disaggregation, comparability, communication and metadata for two disease areas - one of which (Cancer) has well-established regional registries and is classified as a National Statistic. In the absence of regional registries, data on Diabetes are produced for England and Wales, and disaggregated to a regional level only, whereas Cancer data may be presented by StHA, PCT (or equivalent in Scotland), or postcode (although not on a national level). Moreover, since Diabetes statistics are not classified as National Statistics, there is no requirement to comply with the National Statistics Code of Practice. This has implications for the comparability and accessibility of Diabetes data. However, in the areas of metadata and communication, the two data series are similar.

13.1 Introduction

The purpose of this research was to examine the use of health statistics and to identify issues of concern to users. The project was conducted in two phases. In the first phase, users' views were obtained through interviews and a questionnaire survey. On the basis of the findings from this first stage, five issues were identified for further investigation in the second phase. In particular, the second phase explored the concerns made by users in greater detail and presented relevant case studies. The five issues for further investigation were:

• Disaggregation
• Inter-country comparability;
  
• ommunication;
  
• Accessibility;
  
• Metadata.

This Section collates the conclusions set out in this report for each of these areas and discusses some emerging issues. A number of these issues have implications that are common across the areas.

13.2 Conclusions

13.2.1 Disaggregation

• Health statistics for small areas are required to monitor disease trends and population health, identify inequalities, target resources, compare performance and to plan services. These are crucial areas for local priority setting;
  
• The demand for more disaggregated data by indicators, such as ethnicity and age, needs to be reconciled with data confidentiality and data protection issues;
  
• Producers of statistics are not always able to accommodate administrative changes in health boundaries immediately. For example there is currently a lack of National Statistics for the population of PCTs. Again, the issue of data confidentiality and data protection may have an impact on the ability to produce data for such small areas;
  
• There does not appear to be a consistent policy on datasets for different disease areas. For example, data available about Cancer is more detailed than for many other disease areas such as Diabetes.

13.2.2 Inter-Country Comparability

· There are inconsistencies in the availability of comparable key health statistics for the four countries of the UK;

• The importance assigned to inter-country comparisons and the availability of data varies, depending on the perspective of the organisation and the use made. For example, a voluntary organisation covering the UK places a high level of importance on the ability to make inter-country comparisons and is obviously restricted to the availability of health statistics within its field of interest. However, a central department concerned with comparisons in a wide range of disease and service areas may place less emphasis on specific areas if comparable health statistics are available for other areas.

13.2.3 Communication

• Not all users are aware that producers of National Statistics publish a timetable of forthcoming releases for the year ahead, in accordance with the National Statistics Code of Practice;
  
• The structure of Websites can mean that it is not always immediately obvious to the inexperienced or less frequent user that details of forthcoming statistical releases are published. Similarly it is not always clear to the user where to find the schedules;
  
• A number of different forums exist to enable communication between users and producers of statistics. These include the Statistics User Council and workshops run by the DH;
  
• Communication between users can be carried out formally through membership of a statistics user group. Membership of a statistics user group was not common in respondents to our questionnaire survey, although a number of different groups do exist;
  
• The availability of health statistics on the Internet means that it is difficult to know who users are, compared to e.g. mail order.

13.2.4 Accessibility

• The move to web-based dissemination of data was viewed positively. However the format of some data on the web was criticised for being 'inefficient'. The development of web dissemination alongside more traditional methods has meant data are often presented in separate table format in individual spreadsheets rather than as one complete file;
  
• Improved access to data can be achieved by the use of data cubes. Data cubes are viewed as an efficient way of disseminating information, as they permit rapid retrieval of data in a format that is determined by the user. The Interactive National Hospital Morbidity Data in Australia provides an example of a data cube;
  
• Access difficulties can arise due to the large number of sources of statistics. Searching multiple sources is complex and time consuming. The development of a centralised enquiry point was suggested to overcome these problems;
  
• Various series of health statistics have different access restrictions. For example a 'Compendium of Clinical and Health Outcomes' is only available within the NHS.

13.2.5 Metadata

• Metadata was generally viewed positively;
  
• Respondents to the interview and questionnaire survey recognised the benefits of Metadata and advocated its use. In particular it can be used to assess: quality, timeliness, generalisability, comparability, limitations and it can also assist with interpretation when undertaking analyses;
  
• The format and content of Metadata varies by type of publication and producing organisation. The use of a common approach, such as StatBase ensures that consistent information is recorded.

13.3 Discussions

13.3.1 Demand for Health Statistics

The demand for health statistics comes from a wide range of users who have different perspectives, levels of experience and requirements. It is a challenge to producers of health statistics to meet these demands from such a heterogeneous group. One particular area, highlighted by this research, where the disparity of experience across users was evident, was the accessibility of health statistics. Experienced users had systems in place to search and extract the appropriate data. However, given the plethora of sources of health statistics, the searching process may be considerably more difficult for those who are less experienced. Internet sites do not always provide a good "Contents" structure and may lose the inexperienced user in the "Index". One possible solution is to establish a central enquiry point which can direct users to the relevant data series or organisation.

The conflict between the requirements of experienced and inexperienced users was also apparent in the presentation of data. The introduction of systems which permit users to customise the data according to their requests may be more efficient for experienced users.

The area of communication between users and producers was also subject to similar problems. Some users were unaware that producers publish a list of forthcoming releases. This may mean that increased dissemination or more prominent schedules are required to make users aware that the information they require is available. This could be overcome by producers placing links to lists of forthcoming releases in easily accessible locations on Websites. The effectiveness of the range of methods (e.g. e-mail alerts, press releases etc.) used to publicise forthcoming releases should be evaluated. Similarly a wide range of forums exists for communications between users themselves, and between users and producers. However, it is likely that more experienced users of health statistics generally use these forums.

Metadata also serves an important function in describing the data to all users. In providing information about the data, metadata may reduce the probability of mis-interpretation. Again, it is necessary to ensure that the terminology and definitions used can be easily understood by all users, including those with a non-medical background.

Generally, this research found that users were not aware of which statistics were classified as National Statistics. This problem may be addressed by posting a list of National Statistics in a prominent position on a producer's Website. This would have the additional advantage of allowing users to make an assessment regarding the quality of these statistics since the data have to satisfy certain requirements to receive the status of National Statistics.

13.3.2 Supply of Health Statistics

The supply of health statistics is driven by demand from a range of users (e.g. government, and healthcare professionals and managers). One of the key determinants of demand arises from the need to monitor compliance with performance targets. The implementation of these targets was perceived by respondents to have had a detrimental impact on the quantity of data available for areas that are not subject to such targets.

This research also highlighted discrepancies in the quantity of data produced in the key health areas of Cancer and Diabetes, which both have dedicated National Service Frameworks. A significantly greater amount of data is produced for Cancer compared to that for Diabetes. The role of regional Cancer registries has undoubtedly had a significant positive effect on Cancer data. However, there appears to be some divergence between the data collected by regional registries and those presented at a national level. In spite of this, the introduction of regional registries for other diseases may present a solution to the existing lack of data in these areas.

Constraints mean that the level of detail required by users is difficult to provide. These constraints include patient confidentiality and data protection issues which arise when data are presented at a disaggregated level. In addition, given that the resources of producers are finite, there may be cost implications of reporting disaggregated data or establishing other disease registries.

The increasing demand for comparable data, not only within the UK, but internationally, was apparent from this research. Again, compiling comparable data may involve financial costs. In addition to these, there is a trade-off between achieving comparable data and the usability of existing data. While revising definitions for key health statistics may be one means of facilitating comparability, it may make time series analysis within a country difficult, if not impossible. Therefore, changes to existing data collection to allow comparability should be carefully considered before being implemented. Implementing comparable methodologies and definitions across the four countries in the UK requires collaboration between producers.

13.3.3 Other Issues

In addition to the five issues selected for further detailed exploration, users also raised a number of other points during the first phase of the research. Users associated the National Statistics (or equivalent) logo as an assurance that data were of a high standard. However, some users viewed data quality and accuracy as variable.
Ideally, respondents would have preferred more up-to-date data. But they recognised that there may be a trade-off between timeliness and reliability. Whilst the time lag can be reduced by allowing subsequent revisions after publication, this may be confusing and inconvenient for users. The frequency of publication was generally satisfactory as some participants recognised that it was not always feasible to publish data series with greater frequency.

13.4 Summary

This research provides an invaluable insight into the uses of health statistics and the concerns of users. It has identified a number of areas of particular concern to users and analysed these problematic areas in detail. This review contributed to the Statistics Commission's programme of investigative reviews of statistics in key policy areas.

Section 14: General Recommendations

14.1 General Recommendations

The procedure for classifying data as National Statistics should be transparent.
Action: Government ministers and National Statistics.

Interviewees mentioned that they viewed the National Statistics classification as an indication of the quality of statistics. By ensuring that the procedure for classifying National Statistics is transparent, users will be aware of the requirements placed on National Statistics.

The voluntary status of the National Statistics Code of Practice should be reviewed.
Action: Statistics Commission and government ministers.

If adherence to the National Statistics Code of Practice became compulsory, then this would increase users' trust in statistics. A formal mechanism for monitoring compliance with National Statistics could also be established.

A list of all data that are classified as National Statistics should be readily available from a prominent location.
Action: National Statistician through the Government Statistical Service.

Our review found that a definitive list of National Statistics was not readily available. Moreover, we found that few users were able to distinguish between data that were classified as National Statistics and those that were not. By making such a list available to users, it would help to clarify what data series are classified as National Statistics and therefore, increase users' trust in these statistics by raising awareness that they are using National Statistics.

The data available on particular disease areas, such as those covered by NSFs, should be reviewed.
Action: DH and equivalent organisations in devolved administrations.

The detailed exploration of issues highlighted the differences in the coverage of statistics available for two important disease areas - Cancer and Diabetes. The Cancer registry network has meant that extensive, consistent data are collected on Cancer at a regional level. To adopt such registers for other disease areas would have implications for funding allocation. Therefore, the review should also take account of the level of demand for such data before investment in particular disease areas is undertaken.

14.2 Disaggregation

The level of disaggregation, by geographical area and indicator, for some statistics should be reviewed (e.g. PCT populations, recording ethnicity at birth/death, etc.).
Action: Producers.

A number of respondents commented that it would be useful to have data presented at a PCT level. This data would be used for analyses such as comparative studies. However, changing NHS boundaries and organisational structures may make it costly to revise data.

Future government changes in all administrative boundaries, not only those within the NHS, should take account of their impact on the availability and usability of National Statistics should be considered.
Action: Government in general, in particular the Office of the Deputy Prime Minister, the Boundary Commission and ministers.

A number of users commented that changes in administrative boundaries (e.g. the move from health authorities to PCTs) lead to difficulties in using and analysing statistics. Therefore, any future changes in boundaries should carefully consider the consequent costs in terms of their impact on National Statistics.

The implications of a move to a patient-centred system on areas such as data confidentiality should be examined.
Action: Statistics Commission.

The move to a patient-centred system was viewed positively by a number of users and producers. Indeed, such a system was already being considered by a number of producers. One of the main advantages of introducing such a system is that it allows patients to be tracked through the health care system. However, such a system may have implications for data confidentiality.

14.3 Inter-Country Comparability

A review should be undertaken to identify key health areas where comparable UK statistics are not available. Recommendations could be made as to the appropriate actions to address these areas.
Action: Health and Care Theme Working Group, producers, and government ministers.

The lack of comparable data was seen as problematic by a number of users. However, ensuring that data are compiled consistently may be costly in resource terms and may also affect the future usability of data. There should be collaboration between producers to devise an agreed methodology which would allow the comparability of data series identified in this proposed review.

There should be cooperation at a European level to identify the areas for which comparable statistics can be produced. Recommendations could be made as to the appropriate actions to address these areas.
Action: Producers.

While the most immediate concern to most participants in the interview and questionnaire surveys was comparability at a UK level, the importance of consistent data at a European level is increasing. In this review, we found that such cooperation was already underway.

14.4 Communication

The list of forthcoming publications of National Statistics should be readily available in a prominent position on the Websites of producing organisations.
Action: Producers.

A number of users commented that they would find a list of forthcoming publications useful. Such lists are available from producers' Websites, in accordance with the National Statistics Code of Practice. Therefore, this suggests that some users are not aware that these lists exist. Placing these lists in more prominent positions may increase awareness.

There should be an easily accessible method for communication with producers which users can use at the time of searching/accessing statistics.
Action: Producers and user groups.

It would be useful for users to be able to contact producers at the point of using/searching for statistics. Our review found that a number of means of contacting producers were already in place. However, not all of these methods may allow users instant access to producers.

The means for communication between users and producers should be evaluated to ensure that the optimum approach is adopted.
Action: Producers and user groups.

Our review found that producers use a range of routes for establishing contact with users (e.g. relevant contact e-mail addresses on Websites, telephone numbers, e-mail alert systems). These methods should be reviewed to ensure that they satisfy users' requirements efficiently. Other potential methods should also be included in the review.

14.5 Accessibility

The format of the presentation of data, particularly the introduction of data cubes, should be considered.
Action: Producers.

Comments from more experienced users of health statistics suggest that they felt that the presentation of data was targeted towards the more inexperienced user. This proposed review would address the most appropriate format to fulfil the various users' needs. Alternative means of presenting the data should be included in this review. International evidence suggests that data cubes may be a means of allowing greater user customisation of data which may be more useful for experienced users.

A review should be undertaken to consider the structure of a central enquiry point. This would address users' queries at the point of use.
Action: Producers.

Some users muted that a single point of contact worked well in Scotland and Wales. Adopting such a central enquiry point may also work well for all health statistics. The review would address whether this centralised point would cover all countries in the UK and what health statistics would be included in its remit. This central point could also be used to monitor complaints and issues raised by users of National Statistics. This recommendation will be addressed to some extent by the establishment of a Health and Social Care Information Centre in 2005.

14.6 Metadata

A standardised template should be designed for all health data.
Action: Producers.

Metadata was generally viewed positively. However, the information classified as metadata varied across producing organisations. Adopting a standardised approach would ensure that consistent data were recorded. This is currently being examined by the ONS.

References

Appendix A: ONS Web-based Survey

A.1 Questionnaire

ASK ALL WHO HAVE USED HEALTH CARE DATA (CODE 1 TO 7 AT Q1), OTHERS GO TO Q8

A.2 Analysis of ONS Survey Responses

A.2.1 Introduction

The ONS commissioned a Website survey to capture the views of National Statistics (NS) Website users. Some questions were incorporated into this survey to inform our research. The questions asked how many times respondents used health data available on the NS site, the type of health data used and the use made of these data. This section summarises the questions of interest to our review.

A total of 2,998 NS Website users completed the questionnaire.

A.2.2 Frequency of Use of Health Data

Users were asked how often they have used health data from www.statistics.gov.uk in the past twelve months. Table A.1 shows this use. The majority of responders reported that they never used health statistics (60%). Of those who did use health statistics, 290 used them less often than once every six months. Very few responders use the Website regularly (35).

Table A.1: Frequency of use of health data

For the purpose of this research, analysis was carried out on responders who reported that they used the Website (i.e. responders who stated they never or did not know when they used the Website were excluded). Therefore, the following analysis covers 1,048 respondents.

A.2.3 Frequency of Use by Sector

The distribution, by sector, of responders that use health statistics is shown in Table A.2. This table shows that the majority of responders (63%) come from five sectors:

• Further/Higher Education (student);
  
• Health Sector;
  
• Local Government;
  
• The General Public;
  
• Central Government.
  

Looking at the sectors with a high number of users, it can be seen that of students within higher education, the majority used the Website less often than once every six months (26.5%) and only 0.8% used the Website every day. Within the Central Government responders, most used the Website less often than every six months (24.3%) or once every two to six months (22.9%). Of the general public who used the Website, most, (30.1%), used the Website less often than once every six months. Within the health sector, the pattern was slightly different as the responders' number of uses were less evenly distributed, 34.0% used the Website once or twice a month and 20.4% used the Website once every two or five months.

A.2.4 Type of Health Statistics Used

Table A.3 provides details regarding the total number of types of health statistics a responder uses. For example, 304 responders used one type of health statistic and 133 responders used three types of health statistics. The table shows that there are very few responders that use the Website to access more that 11 types of health statistics. Most responders only use one, two or three types of health statistics.

Table A.3: Total number of types of health statistics used by any responder

Note: Responders could identify more that one type.

The types of health statistics referred to are provided in Table A.4. Users were able to select 'Don't know', this table implies 157 users were unsure about the type of health data they had used. According to this table, the following types of health statistics are most frequently used:

1. Deaths;
2. Morbidity;
3. Child health;
4. National Health Service;
5. Drug abuse misuse of alcohol and smoking.

The health statistics looked at the least often are:

1. Disabilities;
2. Sexual health;
3. Financial expenditure on health;
4. Long-term illnesses;
5. Diabetes.

Table A.4: Number of responders using each type of health statistic

Note: Survey relates to all health statistics on NS Website, including those not classified as National Statistics.

A.2.5 Uses of Health Data

The survey asked respondents to identify the purposes for which the statistics they accessed were used. The uses varied, with many being for educational or work purposes, or for general interest. Table A.5 provides details of the uses health statistics are put to and how many responders reported each use. The table shows that the majority of uses are educational ones, work related and business uses or for preparing reports and for research purposes.

Table A.5: The types of uses health statistics are put too and how many people use them

Note: Thirteen respondents provided more than one use.

A.2.6 Conclusion

The ONS questionnaire provided information about the sectors using health data, the type of health statistics used and to what use the statistics were put. The results show that 1,048 responders from a wide range of sectors use health statistics from the ONS Website for a variety of reasons.

The majority of responders (63%) were in five sectors:

• Further/Higher Education (student);
  
• Health Sector;
  
• Local Government;
  
• General Public;
  
• Central Government.

The main health statistics accessed were related to:

• Child health;
  
• Deaths;
  
• Drug abuse misuse of alcohol and smoking;
  
• Morbidity;
  
• National Health Service.

The use made of the statistics was concentrated within:

• Educational/academic use/university assignment/dissertation/thesis/ course work;
  
• Work related/business use;
  
• Preparing reports/used to write a report;
  
• Research purposes.

B.1 Extract from Hitting the Headlines

£1 Superpill' could cut heart attacks by 80%'

The news that a combined pill could reduce the risk of heart attacks and strokes by 80% was reported in ten newspapers on 27 June 2003 (1-10). The research (11) behind this story was based on meta-analyses of the individual drug components proposed and the findings were generally accurately reported by the newspapers.

• Ten newspapers (1-10) reported that a new pill combining heart drugs and vitamins could cut the number of deaths from cardiovascular disease by 80%. They report that trials of the 'Polypill' are planned and most of the articles acknowledge that it would take years before the product would be available.
  
• The research was based on existing meta-analyses of different drugs that reduce cardiovascular risk factors. The authors conclude that the proposed Polypill could largely prevent heart attacks and stroke if taken by everyone aged 55 and older and everyone with existing cardiovascular disease. However, these conclusions are based on meta-analyses of the individual components of the Polypill which have not yet been tested in combination.
  
• The newspapers were generally accurate in their reporting of the research, and reported that trials were planned. Most acknowledged that it would be years before this drug could be made available.

Evaluation of the Evidence Base for 'Polypill' to reduce Cardiovascular Disease

Where does the evidence come from?

The evidence comes from research conducted by Professors Wald and Law of the Wolfson Institute of Preventive Medicine at the University of London. The authors have filed a patent application on the formulation of the combined pill described here, and a trademark application for the name 'Polypill'.

What were the authors' objectives?

The objective was to determine the combination of drugs and vitamins, and their doses, for use in a single daily pill (the Polypill) to achieve a large effect in preventing cardiovascular disease with minimal adverse effects. The strategy was to simultaneously reduce four cardiovascular risk factors (low density lipoprotein (LDL) cholesterol, blood pressure, serum homocysteine and platelet function) regardless of pre-treatment levels.

What was the nature of the evidence?

The research used existing evidence from published meta-analyses of randomised controlled trials (RCTs) and cohort studies. In addition, a meta-analysis of 15 RCTs of low dose aspirin (50-125 mg/day) was carried out.

What interventions were examined in the research?

The authors identified categories of drugs or vitamins used to modify LDL cholesterol, blood pressure, homocysteine and platelet function and summarised selected previous research on these drugs.

For modifying LDL cholesterol, lowering blood pressure and reducing serum homocysteine, the authors drew conclusions from three of their earlier meta-analyses of statins, blood pressure lowering drugs and folic acid.

The authors performed a meta-analysis of 15 RCTs identified from three electronic databases and two previous meta-analyses on the effects of low dose aspirin (an antiplatelet agent) on the reduction in IHD events and stroke and its adverse effects.

The long term effect of a specified absolute reduction in LDL cholesterol, blood pressure and homocysteine expressed as the proportional reduction in the incidence of ischaemic heart disease (IHD) events and stroke was taken from four published sources that were based on systematic reviews of cohort studies, three of which were by the authors.

The authors calculated the combined effect of changing the four risk factors by multiplying the relative risks associated with each. A Markov model was used to calculate the years of life gained without a heart attack or stroke if people without a previous cardiovascular event used the Polypill from age 55.

What were the findings?

The authors report a 61% reduction in risk of an IHD event and a 17% reduction in risk of stroke with the use of statins to modify LDL cholesterol, a 46% reduction in risk of an IHD event and a 63% reduction in risk of stroke with three classes of drug to reduce blood pressure, a 16% reduction in risk of an IHD event and a 24% reduction in risk of stroke with folic acid for reducing serum homocysteine. The meta-analysis of low-dose aspirin indicated a 32% reduction in risk of an IHD event and a 16% reduction in risk of stroke.

The authors calculated the combined effect of the agents to have an 88% reduction in risk of an IHD event and an 80% reduction in risk of stroke. The authors stated that about a third of people taking the Polypill from age 55 would benefit, each gaining an average of 11-12 years free from an IHD event or stroke.

If the three classes of blood pressure lowering drugs with the lowest prevalence of adverse effects were used in a Polypill formulation, 8% of people taking the drug would be expected to have symptoms attributable to one or more of the six components of the pill, mostly due to aspirin. If the three least expensive blood pressure lowering drugs were used, about 15% of people taking the pill would be expected to have symptoms. Aspirin had the most serious adverse effects of all the components, mainly due to haemorrhage.

What were the authors' conclusions?

The authors conclude that the Polypill strategy could largely prevent heart attacks and stroke if taken by everyone aged 55 and older and everyone with existing cardiovascular disease. It would be acceptably safe and with widespread use would have a greater impact on the prevention of disease in the Western world than any other single intervention.

How reliable are the conclusions?

The authors' conclusions are premature since this proposed combined drug for the prevention of cardiovascular disease has not yet been clinically tested. Polypill would need to be rigorously evaluated in an RCT before conclusions could be drawn about its safety and effectiveness, particularly since the interactions of combining these drugs have not been investigated. It is likely that the Polypill would be controversial as it is a preventative drug which would be used to treat people with no diagnosed disease.

Systematic Reviews

Information staff at CRD searched for systematic reviews relevant to this topic. Systematic reviews are valuable sources of evidence as they locate, appraise and synthesize all available evidence on a particular topic.

There were no related systematic reviews identified on the Cochrane Database of Systematic Reviews or on the Database of Abstracts of Reviews of Effects (DARE).

This report has been prepared for the National electronic Library for Health by the NHS Centre for Reviews and Dissemination, based at the University of York

References and Resources

1. Med diet cuts heart deaths by one third (contains story on Polypill). Daily Mail, 27 June 2003, p15.
2. Combination pill 'could cut heart attacks and strokes'. Financial Times, 27 June 2003, p3.
3. Miracle pill adds 10 years to your life. Daily Express, 27 June 2003, p1, p6.
4. £1 'superpill' could cut heart attacks by 80%. The Times, 27 June 2003, p1.
5. Longer life pill. The Star, 27 June 2003, p24.
6. Once-a-day pill 'cuts heart attacks by 80%'. The Guardian, 27 June 2003, p1.
7. Super pill for over-55s could cut coronaries and strokes. Daily Telegraph, 27 June 2003, p9.
8. The superpill. Daily Mirror, 27 June 2003, p11.
9. For only 60p a day, the 'magic bullet' that could prevent heart attacks and strokes. The Independent, 27 June 2003, p3.
10. Take a pill and live 12 years longer. The Sun, 27 June 2003, p8.
11. Wald NJ, Law MR. A strategy to reduce cardiovascular disease by more than 80%. BMJ 2003;326:1419-1424.
12. Rodgers A. A cure for cardiovascular disease? (editorial). BMJ 2003;326:1407-1408.

Consumer Information

C.1 Search Strategy

D.1 Internet Searches

Department of Health
http://www.dh.gov.uk/Home/fs/en
Found nothing of relevance in Publications Library, Policy and Guidelines or Consultation sections.

Final Health and Care 3 Year Work Plan
http://www.statistics.gov.uk/nsworkprogramme/downloads/04g_Health&Care.pdf

National Statistics Online
http://www.statistics.gov.uk/
Searched section about National Statistics and the Office for National Statistics. This includes sections on key documents and consultations as well as links to information about the Statistics Commission, Statistics Users' Council and Health Statistics User Group (HSUG).

Browsed through minutes of meetings, consultation documents, newsletters and key documents. Found nothing of direct relevance, but some documents that may be of interest generally.

National Statistics Work Programme 2003/04 to 2005/06
http://www.statistics.gov.uk/NSWorkProgramme/downloads/NSWorkProg2003-06.pdf

Radical Statistics Group
http://www.radstats.org.uk/
Searched the Health section, as well as publications, news and conference proceedings. Found nothing of relevance.

Royal Statistical Society
http://www.rss.org.uk/
Has a medical section, but there was nothing on it.

The UK Parliament
http://www.parliament.uk/index.cfm
Searched the Health Select Committee reports, publications, oral and uncorrected evidence. Also searched the Commons Hansard. A number of references were found in responses to Health Committee reports (e.g. response to report on provision of maternity services is critical of statistics use), and a number of Parliamentary Questions were also found using statistics on topics such as waiting lists, nursing homes, asthma, alcohol, smoking, ambulance services, bed blocking etc. This gave some indication of the topics of interest to MPs and the areas where policy might be influenced.

User Consultation
http://www.statistics.gov.uk/about/Consultation_by_theme/downloads/he_user_consultation_health.pdf

Statistics Green Paper (February 1998) - Statistics: a matter of trust
http://www.archive.official-documents.co.uk/document/ons/govstat/report.htm

Statistics White Paper - Building trust in statistics (October 1999)
http://www.statistics.gov.uk/about_ns/downloads/WhitePaperText2.pdf

Appendix E: Categories of Organisations Using Health Statistics

E.1 Organisations using Health Statistics

UK Parliament / Devolved Parliaments and Assemblies

Government Departments:

• H M Treasury;
  
• Office of the Deputy Prime Minister.

Local Government:

• Economic Development / Regeneration;
  
• Social Services;
  
• Education.

Research Units / Organisations:

• Commercial;
  
• Academic.

Media:

• TV / Radio;
  
• Newspapers.

Professional Interest Groups:

• Royal Colleges;
  
• Trade Unions;
  
• Education and Training (e.g. BMA/BDA/RCN);
  
• Regulatory bodies (e.g. Health and Safety Executive).

Special Interest Groups:

• Voluntary organisations;
  
• Patient Groups;
  
• Local Pressure / Public Groups.

Appendix F: HSUG Members' Comments

The Health Statistics User Group was established by the Statistics User Council, the Society for Social Medicine, and the Royal Statistics Society, to 'represent all users of health and health services statistics'. Its activities are:

'aimed at maintaining and improving data quality, data access, and use of health and health services statistics.

The Group provides a regular forum where members can discuss relevant issues. Results of discussions and other health information developments are disseminated via seminars, conferences, evening meetings, or via the HSUG newsletter.

The membership of the HSUG covers the main user groups, including academics, NHS organisations, Public Health, ONS, DH, NHSIA and UK Data Archive.

G.1 Interview Questions

H.1 Methodology Used to Compile list of Questionnaire Recipients from the Directory of NHS Management (Binley's)

Strategic Health Authorities

Primary Care Trusts

Special Health Authorities

N I Management Executive

NI Health & Social Services Boards

Scotland - Special Health Boards & Others

Scotland UHBs

Wales - Regions

Wales - Local Health Boards

HAZ

Community Health Councils

Healthcare Interest Groups

Procedure carried out in the event of unrecognised e-mail addresses

If an e-mail was returned to YHEC due to mail management systems' failure to recognise the address, then, wherever a postal address was documented in Binley's, the intended recipients were sent the questionnaire and accompanying letter using surface mail.

H.2 Letter Accompanying Questionnaire

Dear

York Health Economics Consortium (YHEC), a research company at the University of York, has been contracted by the Statistics Commission to undertake a review of the use of health statistics in the UK. As part of this evaluation, a questionnaire (enclosed) is being distributed to users of official health statistics to ascertain what types of statistics are being used, and to identify particular concerns of users. The questionnaire responses will be used to identify issues and develop recommendations for the Statistics Commission. General information on the purpose and completion of the questionnaire is also attached.

Users' views are vital to this research. Consequently, your views, based on your experience as a user of health statistics, are very important to us.

We would be grateful if you could please complete and return this questionnaire by Monday, 30th April 2004 in the freepost envelope provided. If you are not the appropriate person to complete this questionnaire, we would appreciate it if you could please forward it to a colleague within your organisation who has experience in using official health statistics.

We are using several different means of identifying and contacting potential users of health statistics. Therefore, we would like to apologise if we have already contacted you about completing this questionnaire. If you have completed and returned this questionnaire in response to our previous correspondence, please ignore this letter.

If you would like additional information or have any questions, please do not hesitate to contact me by telephone (01904 433620) or by email (yhec@york.ac.uk). We greatly appreciate your cooperation and thank you for your interest and for taking the time to complete the survey questions.

Yours sincerely




JACQUELINE O'REILLY
Consultant

H.3 Questionnaire: Review of Use of Health Statistics
User Questionnaire

General Information for Respondents

This questionnaire has been designed by York Health Economics Consortium (YHEC), in conjunction with the Statistics Commission, to obtain information on the use of health statistics. Relevant users were identified from voluntary organisations, lobby groups, local and central government departments and agencies, the NHS, the academic and research community, and other entities.

2. It would be appreciated if you could please provide your views, as a user, about the health statistics you have used. The questionnaire consists of two sections. Section 1 investigates which particular health areas are of interest to you and why. Section 2 examines issues relating to quality, accessibility and accuracy.

3. If you are not the appropriate person to complete this questionnaire, we would appreciate it if you could please forward it to a colleague within your organisation who has experience in using official health statistics.

4. This questionnaire survey will be used to inform the Statistics Commission of users' concerns. It builds on work recently undertaken by the Commission (see http://www.statscom.org.uk/media_pdfs/reports/Health Seminar Final Report.pdf)

5. The responses will be aggregated and presented in summary form. The names of individuals who have completed this questionnaire will not be passed to the Statistics Commission.

6. A glossary of the terms used in this questionnaire is reported overleaf.

7. We would appreciate it if you could please complete and return the attached questionnaire by Friday, 30th April 2004 in the FREEPOST envelope attached or post it to: York Health Economics Consortium FREEPOST YO405 University of York Market Square, Vanbrugh Way YORK YO10 5ZZ You do not need a stamp.

8. If you would like additional information or have any questions, please do not hesitate to contact Jacqueline O'Reilly by telephone (01904 433620) or by email (yhec@york.ac.uk).

We very much value your opinion.

Review of Health Statistcs: User Questionnaire

Section 1: Health Statistics and their uses

1.1 How often do you use health statistics in the areas listed below? (Please tick one box for each area.)

1.2 What is the main source of health statistics that you use? (Please tick all that apply.)

1.3 To what country do the statistics that you use refer? (Please tick all that apply.) E

1.4 For what purposes you use these health statistics in the areas you identified in Question 1? (Please tick all that apply.)

1.5 To whom do you disseminate this information containing the health statistics? (Please tick all that apply.)

Section 2: Information Concerning Quality Aspects of Health Statistics

2.1 To what extent do you agree or disagree with each of the following statements? (Please tick one box only for each statement.)

2.2 Are there any specific health statistics that you would like to be made available?

If YES, please comment below:

2.3 Does your use of health statistics influence policy, either directly or indirectly?

If YES, please explain how below:

Additional comments, including areas where improvement is possible (please specify the particular health statistics to which your comments refer in the area below).

Background Information

Please return this questionnaire by Friday, 30th April 2004 in the FREEPOST envelope. Thank you for your assistance in completing this questionnaire.

Enc.

Appendix H4: Questionnaire

H.4 Analysis of Questionnaire Responses

H.4.1 Alphabetical List Of 'Other' Data Sources Used By Questionnaire Respondents

H.4.2 Areas Where More Statistics are Needed

When asked specifically to detail health statistics that the recipient would like to be made available the following areas were mentioned:

Note: The frequency is 1 unless stated otherwise.

H.4.3 Responses from Respondents who Felt that their Use of Health Statistics Influenced Policy

Table H.1: Policy

Table H.2: Services

Table H.3: Finance

Table H.3: Finance

Table H.4: Healthcare

Table H.5: Report

Table H.6: Research and publications

Table H.7: Other

Table H.8: Responses from respondents who felt that their use of health statistics did not influence policy

Table H.9: Additional comments