Namur, July 12, 2021

Challenges such as the imminent strategic choices posed by the European structural funds, the Recovery programme underway within the Government of Wallonia, questions on the interest in and the value of installing 5G, and whether it is even necessary, along with issues surrounding the implementation of a guaranteed universal income, and other energy, climate and environmental issues, raise the question of the impact of the decisions made by both public and private operators [1].

In their recent work The Politics Industry, while analysing the shortcomings and failure of American democracy and the possibilities for reconstructing it, Katherine M. Gehl and Michael E. Porter call for policy innovation. Katherine Gehl, founder of the Institute for Political Innovation, argues that laboratories of democracy have a role to play in the transformations within the political and social system itself to help governments achieve their objectives and, above all, to achieve the results their citizens deserve [2]. Although the authors, who are immersed in the business and entrepreneurship culture, focus primarily on democratic engineering in order to restore its negative effects on economic competitiveness, the issue of prior, objective analysis or assessment of the impacts that political decisions can have on society and its economy is not high on their agenda. In the absence of this type of approach, we believe that criticising policymaking and its lack of rationality – along with demonstrating the absence of general interest and common good – appears futile.

The weakening of a strong impact analysis probably contributed to Philippe Zittoun’s description, based on the work of the celebrated economists, sociologists and political scientists Herbert Simon (1916-2001) and Charles Lindblom (1917-2018), of complex cognitive tinkering. In this tinkering process, the necessary rational links between problem, objective, solution, tools, values and causes are absent [3]. Ignorance, intuitions, ideology and inertia combine to give us answers that look plausible, promise much, and predictably betray us, write the recent winners of the Nobel Prize for Economics, Abhijit Banerjee and Esther Duflo [4].

Dreamstime – Dzmitry Skazau

1. What is policy impact prior analysis?

The purpose of impact analysis is to establish a comparison between what has happened or will happen after the implementation of the measure or programme and what would have happened if the measure or programme had not been implemented. This comparison can be referred to as the programme impact [5].

Policy impact prior analysis can help to refine decisions before they are implemented and to comprehend their potential effects in different economic environments. The impact assessment provides a framework for understanding whether the beneficiaries do actually benefit from the programme, rather than from other factors or actors. A combination of qualitative and quantitative methods is useful to give an overview of the programme impact. There are two types of impact analysis: ex ante and ex post. An ex-ante impact analysis attempts to measure the expected impacts of future programmes and policies, taking into account the current situation of a target area, and may involve simulations based on assumptions relating to the functioning of the economy. Ex ante analyses are usually based on structural models of the economic environment facing the potential participants. The underlying assumptions for the structural models involve identifying the main economic actors in the development of the programme and the links between the actors and the different markets to determine the results of the programme. These models can predict the programme impacts [6].

In April 2016, in their common desire for Better Regulation, the European Parliament, the Council of the European Union and the European Commission decided to increase and strengthen impact assessments [7] as tools for improving the quality of EU legislation, in addition to consulting with citizens and stakeholders and assessing the existing legislation. In the view of these three institutions, impact assessments should map out alternative solutions and, where possible, potential short and long-term costs and benefits, assessing the economic, environmental and social impacts in an integrated and balanced way and using both qualitative and quantitative analyses. These assessments must respect the principles of subsidiarity and proportionality, as well as fundamental rights. They must also consider the impact of the various options in terms of competitiveness, administrative burdens, the effect on SMEs, digital aspects and other elements linked to territorial impact. Impact assessments should also be based on data that is accurate, objective, and complete [8].

In recent years, the European Commission has gone to great lengths to update its technical governance tools in its efforts to achieve better regulation. This concept means designing EU policies and laws so that they achieve their objectives at the lowest possible cost. For the Commission, better regulation does not involve regulating or deregulating, but rather adopting a way of working which ensures that policy decisions are taken openly and transparently, are guided by the best factual data available, and are supported by stakeholder participation. Impact assessment (or impact analysis) is an important element of this approach to policy issues, as are foresight (or forward-looking) tools, and tools used for stakeholder consultation and participation, planning, implementation, assessment, monitoring etc., which are part of the public or collective policy cycle, and even, by extension, the business policy cycle [9].

Better regulation covers the entire political cycle, from policy conception and preparation, to adoption, implementation, application (including monitoring and enforcement [10]), assessment and revision of measures. For each phase of the cycle, a number of principles, objectives, tools and procedures for improving regulation are used to build capacity for achieving the best possible strategy.

Although impact assessment is not a new tool, since it was theorised extensively in the 1980s and 1990s [11], its role in the process has been strengthened considerably by the European Commission, to the extent that, in our view, it is now of central importance. Even its content has been broadened. The Better Regulations Guidelines of 2017 highlight this transparency and draw a distinction with assessment practices: in an impact assessment process, the term impact describes all the changes which are expected to happen due to the implementation and application of a given policy option/intervention. Such impacts may occur over different timescales, affect different actors and be relevant at different scales (local, regional, national and EU).  In an evaluation context, impact refers to the changes associated with a particular intervention which occur over the longer term [12]. The Guidelines glossary also states that impact assessment is an integrated process for assessing and comparing the merits of a range of public or collective policy options developed to solve a clearly defined problem. Impact assessment is only an aid to policymaking / decision-making and not a substitute for it [13].

Thus, impact assessments refer to the ex-ante assessment carried out during the policy formulation phase of the policy cycle.

This process consists in gathering and analysing evidence to support policy development. It confirms the existence of a problem to be solved, establishes the objectives, identifies its underlying causes, analyses whether a public action is necessary, and assesses the advantages and disadvantages of the available solutions [14].

The Commission’s impact assessment system follows an integrated approach which assesses the environmental, social and economic impacts of a range of policy options, thereby incorporating sustainability into the drafting of EU policies. The impact reports formatted by the Commission also include the impacts on SMEs and on European competitiveness and a detailed description of the consultation strategy and the results achieved [15].

2. Complex, public-interest processes that make democracy more transparent

In a parliamentary context, impact studies designed as ex-ante assessments of legislation satisfy, firstly, an ambition to overhaul policy practices, secondly, an open government challenge to make public debate more transparent, and, thirdly, a desire for efficiency in the transformation of public and collective action, since assessment means better action. Generating knowledge on the objectives, the context, the resources, the expected results and the effects of the proposed policies means giving both parliamentarians and citizens the means to assess the consequences of the recommended measures. It also means supporting public decision-making by plainly revealing the budgetary impacts of the decisions policymakers want to make. These advantages are undoubtedly ways to revitalise our democracies [16].

Used for prior assessment of legislation, impact assessment aims to analyse all the behaviors and situations that present a direct or indirect causal link with the legislation being examined, to identify the unforeseen effects, the adverse effects [17]. It involves identifying the genuine changes expected in society which could be directly associated with the prescriptive (legislative or regulatory) measures implemented by the actors involved in the policy [18]. It is therefore understandable that questions relating to concerns such as the impact of technological choices on health or the extent to which the legislation is consistent with climate and sustainable development objectives are essential questions posed in impact studies [19].

Measuring the impact is therefore the key challenge of the assessment, but it is also the hardest issue to tackle from a methodological point of view [20]. As indicated in the Morel-L’Huissier-Petit report submitted to the French National Assembly in 2018, assessing the mobilisation of resources and the control of public expenditure when implementing legislation or a policy is the driving force for more effective public action which is able to innovate and evolve its management methods in order to adapt positively to the paradox of modern public action: how to do better with less, against a backdrop of cutting public expenditure, rising democratic demands and Public Service expectations, and accelerating economic and social trends [21]. This report also recommends expanding impact studies to cover tabled legislative proposals and substantial amendments in order to supplement the content, review the impact studies already accompanying the legislative proposals, develop robust impact and cost simulators and use them regularly, and, lastly, organise discussions within committees and at public hearings dedicated to assessing impact studies [22].

Concerning the low-carbon strategy, France’s High Council on Climate indicated, in December 2019, that, with regard to environmental and particularly climate assessment, the existing impact studies have not achieved their potential: they cover only a small portion of the legislation adopted (legislative proposals of parliamentary origin and amendments are not included), they are rarely used, and they are often incomplete [23].

However, these assessment works of the High Council on Climate are very interesting from a methodological perspective. When supplemented, impact prior analyses can be considered to follow a seven-stage process, guided by a compass as shown below.

Overall, it might be argued that the impact of a policy is all its effect on real-world conditions, including: 1. impact on the target situation or group, 2. impact on situations or groups other than the target (spill over effects), 3. impact on future as well as immediate conditions, 4. direct costs, in terms of resources devoted to the program, 5. indirect costs, including loss of opportunities to do other things. All the benefits and costs, both immediate and future, must be measured in both symbolic and tangible terms and be explained with concrete equivalences [24].

The main purpose of any ex-ante assessment is without doubt to clarify the political objectives from the outset, for example before voting on a law, and to help define or eliminate any incompatibilities within or between the general objectives and the operational objectives [25]. The fundamental problem seems to be that the impacts of changes brought about by public policies are often minor, or even marginal, compared with those caused by external social and economic developments. It then becomes hard to get the message across [26]. That is why demonstrating a significant public policy impact often means having to deal with a major programme, or series of programmes. The measures must be properly conceived, properly financed and made sustainable over time [27]. These measures can be discussed with stakeholders or even with citizens, as was the case with the measures in the independence insurance bill debated at the citizens’ panel on ageing, organised by the Parliament of Wallonia in 2017 and 2018 [28].

More than simply a judgment, impact assessment is a learning approach whereby lessons can be learned from the policy or action being assessed, and the content improved as a result. Any assessment requires collaboration and dialogue between its key participants, namely the representatives, assessors, beneficiaries of the policies, programmes, projects or functions, and stakeholders, in other words the individuals or bodies that have an interest in both the policy or programme being assessed and the results of the assessment. Assessment in this sense is merely a process in which the actors themselves adopt the thinking on the practices and the results of the subject being assessed [29]. The methods may be many and varied, but the key points are probably the ethics of the assessment and some essential quality criteria: a high-quality model, a large amount of robust data, meeting expectations, and genuine consideration of the common good [30].

 3. Interests and obstacles for a strategic intelligence tool

Impact prior analysis is one of the strategic policy intelligence tools promoted by the European Commission. It also respects the following principles:

– principle of participation: foresight, evaluation or Technology Assessment exercises take care of the diversity of perspectives of actors in order not to maintain one unequivocal ‘truth’ about a given innovation policy theme;

– principle of objectivisation: strategic intelligence supports more ‘objective’ formulation of diverging perceptions by offering appropriate indicators, analyses and information processing mechanism;

– principle of mediation and alignment: strategic intelligence facilitates mutual learning about the perspectives of different actors and their backgrounds, which supports the finding of consensus;

– principle of decision support: strategic intelligence processes facilitate political decisions and support their successful subsequent implementation  [31].

An impact assessment can therefore be broken down into traditional cost-benefit measures and measures relating to areas such as sustainable development, environment, technological innovation and social impact. The Sustainability Impact Assessment has been developed by the European Commission and includes a detailed analysis of the potential economic, social, human and environmental impacts of ongoing commercial negotiations. These assessments are an opportunity for stakeholders from the EU and the partner countries to share their points of view with the negotiators [32].

In recent decades, the literature on policy assessment has increased substantially and new methodologies have been developed to identify the causal effects of policies [33]. In addition, the openness approaches pursued by governments and parliaments are introducing democratic innovation aspects which need to be taken into account. Although the quality of the impact analysis methods, particularly environmental (air, water, ecological systems, socio-economic systems, etc.), has been improved and diversified considerably since the beginning of the 2000s, especially through the works of Christopher Wood [34] and Peter Morris and Riki Therivel [35], it must be acknowledged that, in practice, these processes are rarely applied and that, often, the public authorities prefer not to activate them. However, major clients such as the European Commission and the OECD are becoming increasingly demanding in this area in terms of assessment and climate/energy indicators. This is also a real opportunity to create closer links between impact assessments and public inquiries.

Beyond the technical sphere of civil servants and experts, many elected representatives tend to perceive policy impact prior analysis as an additional layer on top of the decision-making process – which generates a degree of indifference – rather than a beneficial layer which represents real added value for stakeholders.

We also know that, when taken to the extreme, impact assessment is a tool that can hinder or even prevent legislative and programme-based action. The Anglo-Saxons have an extreme vision of efficiency, even going as far as the concept – assumed – of a regulatory guillotine [36]. This fairly radical approach may involve two paths: one in which, faced with the proliferation of ex ante assessment procedures, the political system risks rigidity, the other in which, for fear of generating additional prescriptive complexity, the elected representatives avoid all legislative change. The OECD is interested in this aspect [37].

In this way, prior policy impact assessment could open a lively debate on legislative relevance. Something that is always healthy, particularly in parliamentary settings.

Philippe Destatte

@PhD2050

[1] I would like to thank Sarah Bodart, analyst and economist at The Destree Institute’s Wallonia Policy Lab, for her advice and suggestions for finalising this paper.

[2] Katherine M. GEHL & Michaël E. PORTER, The Politics Industry, How Political Innovation Can Break Partisan Gridlock and Save our Democracy, p. 179, Boston, Harvard Business Review Press, 2020.

[3] Philippe ZITTOUN, La fabrique politique des politiques publiques, p. 146, Paris, Presses de Sciences-Po, 2013. – Charles E. LINDBLOM, The Policy-Making Process, Prentice-Hall, 1968.

[4] Abhijit BV. BANERJEE and Esther DUFLO, Économie utile pour des temps difficiles, p. 439-440, Paris, Seuil, 2020. – See also Esther DUFLO, Rachel GLENNESTER and Michael KREMER, Using Randomization in Development Economics Research: A Toolkit in T. Paul SCHULZ and John STRAUSS ed., Handbook of Development Economics, vol. 4, p. 3895–3962, Amsterdam, North-Holland, 2008.

[5] Lawrence B. MOHR, Impact Analysis for Program Evaluation, p. 2-3, Chicago, The Dorsey Press, 1988.

[6] Shahidur R. KHANDKER, Gayatri B. KOOLWAL, Hussain A. SAMAD, Handbook on Impact Evaluation: Quantitative Methods and Practices, p. 19-20, Washington, World Bank, 2010.

https://openknowledge.worldbank.org/handle/10986/2693.

[7] The OECD defines impact as the positive or negative, primary and secondary long-term effects produced by an intervention, directly or indirectly, intended or unintended.  Niels DABELSTEIN dir., Glossaire des principaux termes relatifs à l’évaluation et à la gestion axées sur les résultats, p. 22, Paris, OECD, 2002. http://www.oecd.org/dataoecd/29/21/2754804.pdf

See also the EVALSED glossary: Nick BOZEAT (GHK) & Elliot STERN (Tavistock Institute) dir., EVALSED, The Resource for the Evaluation of Socio Economic Development, Sept. 2013: Impact: The change that can be credibly attributed to an intervention. Same as “effect” of intervention or “contribution to change”. – A consequence affecting direct beneficiaries following the end of their participation in an intervention or after the completion of public facilities, or else an indirect consequence affecting other beneficiaries who may be winners or losers. Certain impacts (specific impacts) can be observed among direct beneficiaries after a few months and others only in the longer term (e.g. the monitoring of assisted firms). In the field of development support, these longer-term impacts are usually referred to as sustainable results. Some impacts appear indirectly (e.g. turnover generated for the suppliers of assisted firms). Others can be observed at the macro-economic or macro-social level (e.g. improvement of the image of the assisted region); these are global impacts. Evaluation is frequently used to examine one or more intermediate impacts, between specific and global impacts. Impacts may be positive or negative, expected or unexpected. – Philippe DESTATTE, Evaluation of Foresight: how to take long-term impact into consideration? For-learn Mutual Learning Workshop, Evaluation of Foresight, Seville, IPTS-DG RTD, December 13-14, 2007. – Gustavo FAHRENKROG e.a., RTD Evaluation Tool Box: Assessing the Socio-economic Impact of RTD Policies. IPTS Technical Report Series. Seville, 2002.

[8] Better Regulation, Interinstitutional agreement between the European Parliament, the Council of the European Union and the European Commission, Brussels, 13 April 2016.

[9] Better Regulation Guidelines, Commission Staff Working Document, p. 5sv, 7 July 2017 (SWD (2017) 350.

[10] Application means the daily application of the requirements of the legislation after it has entered into force. EU regulations are applicable from their effective date, while rules set out in EU directives will apply only from the effective date of the national legislation that transposes the EU directive into national law. Application covers transposition and implementation. Better Regulation Guidelines…, p. 88.

[11] For example: Saul PLEETER ed., Economic Impact Analysis: Methodology and Application, Boston – The Hague – London, Martinus Nijhoff, 1980.

[12] Better regulation guidelines, p. 89, Brussels, EC, 2017.

[13] Ibidem.

[14] Szvetlana ACS, Nicole OSTLAENDER, Giulia LISTORTI, Jiri HRADEC, Matthew HARDY, Paul SMITS, Leen HORDIJK, Modelling for EU Policy support: Impact Assessments, Luxembourg, Publications Office of the European Union, 2019.

[15] Better Regulation Guidelines…, p. 13.

[16] Pierre MOREL-L’HUISSIER and Valérie PETIT, Rapport d’information par le Comité d’Évaluation et de Contrôle des politiques publiques sur l’évaluation des dispositifs d’évaluation des politiques publiques, p. 7-24,  Paris, National Assembly, 15 March 2018.

[17] Geneviève CEREXHE, L’évaluation des lois, in Christian DE VISSCHER and Frédéric VARONE ed., Évaluer les politiques publiques, Regards croisés sur la Belgique, p. 117, Louvain-la-Neuve, Bruylant-Academia, 2001.

[18] In simple terms, a successful impact assessment aims to establish the situation that society would have experienced in the absence of the policy being assessed. By comparing this fictional, also called counterfactual, situation to the situation actually observed, a causal relationship can be deduced between the public intervention and an indicator deemed relevant (health, employment, education, etc.). Rozenn DESPLATZ and Marc FERRACCI, Comment évaluer les politiques publiques ? Un guide à l’usage des décideurs et praticiens, p. 5, Paris, France Stratégie, September 2016.

Cliquer pour accéder à guide_methodologique_20160906web.pdf

See also: Stéphane PAUL, Hélène MILET and Elise CROVELLA, L’évaluation des politiques publiques, Comprendre et pratiquer, Paris, Presses de l’EHESP, 2016.

[19] This extension can also be found in the AFIGESE definition: Impact: social, economic and environmental consequence(s) attributable to a public intervention. Marie-Claude MALHOMME e.a., Glossaire de l’Évaluation, p. 77, Paris, AFIGESE- Caisse d’Épargne, 2000.

[20] Jean-Pierre BATTERTI, Marianne BONDAZ and Martine MARIGEAUD e.a., Cadrage méthodologique de l’évaluation des politiques publiques partenariales : guide, Inspection générale de l’Administration, Inspection générale des Finances, Inspection générale des Affaires sociales, December 2012

http://www.ladocumentationfrancaise.fr/rapports-publics/124000683-guide-cadrage-methodologique-de-l-evaluation-des-politiques-publiques-partenariales

[21] Pierre MOREL-L’HUISSIER and Valérie PETIT, Rapport d’information…

[22] Pierre MOREL-L’HUISSIER and Valérie PETIT, Rapport d’information... p.11-13

[23] Where there is evidence that some provisions of a law have a potentially significant effect on the low-carbon trajectory, whether positive or negative, the text initiator decides to steer the text towards a detailed impact study relating to the national low-carbon strategy (SNBC). This detailed study is the subject of a detailed public opinion on its quality, produced by an independent authority with the capacity to do so. This process must be concluded before the legal text is tabled in Parliament. It is suggested that Parliament should expand detailed impact studies relating to the low-carbon strategy to cover legislative proposals. Évaluer les lois en cohérence avec les ambitions, p. 5-6, Paris, High Council on Climate, December 2019.

[24] Thomas R. DYE, Understanding Public Policy, p. 313, Upper Saddle River (New Jersey), Prentice Hall, 2002. The impact of a policy is all its effect on real-world conditions, including : impact on the target situation or group, impact on situations or groups other than the target (spillover effects), impact on future as well as immediate conditions, direct costs, in terms of resources devoted to the program, indirect costs, including loss of opportunities to do other things. All the benefits and costs, both immediate and future, must be measured in both symbolic and tangible effects. – See also: Shahidur R. KHANDKER, S.R., Gayatri B. KOOLWAL, & Hussain A. SAMAD, Handbook on Impact Evaluation, Quantitative methods and practices, Washington D.C, World Bank, 2010.

[25] Paul CAIRNEY, Understanding Public Policy, Theories and Issues, p. 39, London, Palgrave-MacMillan, 2012.

[26] Karel VAN DEN BOSCH & Bea CANTILLON, Policy Impact, in Michaël MORAN, Martin REIN & Robert E. GOODIN, The Oxford Handbook of Public Policy, p. 296-318, p. 314, Oxford, Oxford University Press, 2006.

[27] Th. R. DYE, op. cit., p. 315.

[28] Ph. DESTATTE, Que s’est-il passé au Parlement de Wallonie le 12 mai 201 ?7 Blog PhD2050, Namur, 17 June 2017, https://phd2050.wordpress.com/2017/06/17/panel2/

[29] Philippe DESTATTE and Philippe DURANCE dir., Les mots-clefs de la prospective territoriale, p. 23-24, Paris, La Documentation française, 2009.

Cliquer pour accéder à philippe-destatte_philippe-durance_mots-cles_prospective_documentation-francaise_2008.pdf

[30] Jean-Claude BARBIER, A propos de trois critères de qualité des évaluations: le modèle, la réponse aux attentes, l’intérêt général, dans Ph. DESTATTE, Évaluation, prospective, développement régional, p. 71sv, Charleroi, Institut Destrée, 2001.

[31] Alexander TÜBKE, Ken DUCATEL, James P. GAVIGAN, Pietro MONCADA-PATERNO-CASTELLO ed., Strategic Policy Intelligence: Current Trends, the State of the Play and perspectives, S&T Intelligence for Policy-Making Processes, IPTS, Seville, Dec. 2001.

[32] Sustainability Impact Assessment (SIA) https://ec.europa.eu/trade/policy/policy-making/analysis/policy-evaluation/sustainability-impact-assessments/index_en.htm

[33] Massimo LOI and Margarida RODRIGUES, A note on the impact evaluation of public policies: the counterfactual analysis, JRC Scientific & Policy Report, Brussels, European Commission, Joint Research Center, 2012. (Report EU 25519 EN).

[34] Christopher WOOD, Environmental Impact Assessment, A Comparative Review, Harlow, Pearson Education, 2003. (1st ed. 1993).

[35] Peter MORRIS & Riki THERIVEL, Methods of Environmental Impact Assessment, London – New York, Spon Press, 2001.

[36] Thanks to Michaël Van Cutsem for this remark. http://regulatoryreform.com/regulatory-guillotine/

[37] La réforme de la réglementation dans les pays du Moyen-Orient et d’Afrique du Nord, Paris, OCDE, 2013.