All producers of statistics take the goal of high quality in official statistics seriously, but there is still room for improvement. There is a need for all producers to ensure access to written documentation of guidelines, procedures, methods and quality. It is recommended that efforts are undertaken to improve the procedures for releasing preliminary figures. More producers must find good solutions for complying with the provisions on confidentiality in the Statistics Act, and they should take the initiative to engage in more regular contact with users.
The quality assurance system must be further developed and supplemented. Established methods must be reviewed, simplified and applied to all official statistics. New methods, such as measurement of quality in real time, should be developed and made available for the production of official statistics.
Information sources
This initial report is based on information from various sources.
The most important source is the quality evaluation, which is a new instrument (Chapter 3). The basis for the quality evaluation is the quality requirements in the Statistics Act and quality principles in the European Statistics Code of Practice (Eurostat 2017). All producers of official statistics, including the three statistical divisions in Statistics Norway, were included in the quality evaluation. The data were collected in the period from September 2021 to January 2022. The results from the quality evaluation will be published as an interim report in the summer of 2022 (SSB 2022, in progress).
In November 2021, a peer review was undertaken (Chapter 4), commissioned by Eurostat. It was limited to Norway’s contributions to the European Statistical System. Although the peer review does not apply to all official statistics, the assessments and recommendations are of interest and included in this report.
Many producers use data from administrative information systems as a source for official statistics. Since 2012, Statistics Norway has been engaged in a standardised and formalised cooperation on quality with, inter alia, owners of administrative information systems (Chapter 5). Whether other producers of official statistics have similar arrangements with data owners is unclear. The report describes the status of Statistics Norway’s cooperation with owners of administrative information systems.
Section 5 of the Statistics Act and the European Statistics Code of Practice specify relevance as a quality requirement. Statistics that fail to meet the users’ needs are not relevant. User orientation is therefore crucial for ensuring the quality of statistics and has a key position in the quality assurance system. As of today, there is no systematic records on user orientation efforts by producers other than Statistics Norway. The efforts and status described in this report therefore apply primarily to Statistics Norway. The assessments and recommendations with regard to user orientation nevertheless apply to all producers.
Since 2011, Statistics Norway has conducted quality reviews as a key tool in its internal quality assurance work. A quality review investigates whether statistical output or a statistical domain comply with the quality requirements in the Statistics Act and the European Statistics Code of Practice (Chapter 7). It involves a review of a statistical production process to identify its strengths and weaknesses, and the results include a list of recommended improvements. The chapter on quality reviews is based on experience from Statistics Norway’s internal quality assurance work. The assessments and recommendations apply to all official statistics.
This report is based on the information and quality assurance systems that are available at the time of reporting. In the assessments and recommendations, all production of official statistics are to the greatest extent possible viewed as a whole
Recommendations
In light of the data retrieved from the aforementioned information sources at the end of 2021, the items below are recommended for improvement.
The producers of official statistics should improve their written documentation of guidelines, procedures, methods and quality. Written documentation will help ensure that more quality principles in the Statistics Act and the European Statistics Code of Practice are complied with. Work to prepare written documentation was initiated by a number of producers during the collection of data for the quality evaluation, but some work still remains to be done in this area.
Practices vary considerably between the producers of official statistics when it comes to preliminary statistics. There are various rationales for releasing preliminary statistics, as well as varying practices and opinions regarding what constitutes preliminary statistics. Users may find this confusing, but this assumption has not been investigated in detail. It is recommended that efforts are undertaken to raise awareness about and improve the procedures for releasing preliminary figures. One question that several producers could ask themselves is whether it is necessary to release preliminary statistics.
The producers take the legal provisions on statistical confidentiality seriously, but some of them are uncertain about how the law should be interpreted and how good solutions to ensure statistical confidentiality can be found. This challenge should be given extra high priority, since shortcomings in this area could undermine public confidence in official statistics.
Statistics Norway has agreements on the supply of data and cooperation on quality in administrative information systems. It is recommended that more authorities that produce official statistics sign such agreements. This could also be a relevant internal measure in an enterprise where one department owns a data source and another department uses it to produce official statistics.
Most producers meet the quality requirement that the statistics should be relevant, but the quality evaluation indicates that there might be a need to raise awareness in this area. Producers should maintain regular contact with users.
Methods for measuring quality
This quality report is based on the quality evaluation and information on quality from previously established methods. The established methods are recognised, but are not used for all official statistics at the time of reporting. The memo ‘System for quality assurance of official statistics’ (SSB 2021a) alerted Statistics Norway to the fact that there is a need for more information on various aspects of quality in all official statistics. Although the quality assurance work has progressed since 2021, more information is still needed.
There is a need to collect and map out more of the information held by producers. One of the objectives for the quality assurance work is to obtain more knowledge about the quality of official statistics by improving the knowledge base for developing the system going forward. Established methods should be refined and supplemented.
All producers of official statistics must undertake quality reviews. Self-assessments based on the quality evaluation form can be adapted to function as a self-assessment of statistical processes and output. It is recommended that a systematic procedure is established for the selection and prioritisation of statistics for quality reviews among all producers. The combination of thorough quality reviews of selected statistics and self-assessments of the total production of statistics will provide a good basis for the annual quality report to the Ministry of Finance.
Efforts have been initiated to establish quality indicators in Statistics Norway’s production process. It is recommended that a set of common indicators is devised and implemented for use by all producers.
The producers will be well served by having tools and guidelines etc. that can help them comply with the quality requirements in the Statistics Act and the European Statistics Code of Practice. Having all producers engaged in quality assurance work will be a step forward. There is a need for arenas where the producers can collaborate on specific measures and engage in closer cooperation. This can be achieved by expanding the methodology network and strengthening the quality aspect. Participation in the methodology network is currently voluntary. The need for a more binding quality cooperation between the producers should be investigated.
Better documentation of quality and increasing the visibility of the quality assurance work support the goal of transparency and verifiability and will make it easier to follow up quality enhancement measures. One recommendation from the peer review was to investigate measures to document the quality system and make it more visible.
Division of labour in the quality assurance work
This initial report on the quality of official statistics contains a number of recommendations for measures that can help enhance the quality of official statistics. The recommendations can be divided into two main groups. The first of these consists of specific recommendations, primarily from the quality evaluation, but the report also contains some specific recommendations from the other information sources. The second group consists of recommendations for developing the system for quality assurance of official statistics.
The responsibility for implementing measures based on the recommendations in the first group lies with the individual producers. Under Section 6 of the Statistics Act, Statistics Norway has a special responsibility for the coordination of official statistics and reporting the quality of official statistics. This comes on top of the role of producer of official statistics. It is presumed that Statistics Norway carries out this coordination and reporting in collaboration with the Committee for Official Statistics. The outline of the division of labour is as follows:
Statistics Norway
- Plan, coordinate and harmonise the quality assurance work on official statistics
- Prepare an annual public report for the Ministry of Finance on the quality of official statistics
- Provide the data for the quality report
- Lead quality evaluations and quality reviews
- Develop indicators for quality in official statistics
- Chair the methodology network
All producers of official statistics
- Follow up recommendations from the quality evaluation and other quality reporting
- Perform independent cost-benefit analyses of the recommendations and establish priorities based on these assessments
- Participate in quality evaluations and quality reviews
- Participate actively in the methodology network