Defining reforms

For the purposes of this index, a reform (anti-reform) is considered to be a normative act which changes the behavior of economic agents and results in more (less) efficient use of resources. For example, normative changes narrowing the scope for corruption, establishing market prices for certain goods, curtailing cross-subsidies, fostering competition, etc. are considered to be reforms. On the other hand, normative changes creating new opportunities for corruption, tightening regulations for certain goods, increasing cross-subsidies, harming competition, etc. are considered to be anti-reforms.

Evaluation procedure

The evaluation period is equal to two weeks (except holiday-heavy periods such as around New Year’s and Christmas or extraordinary events such as natural disaster or war). Reforms and anti-reforms are chosen by the iMoRe project manager and two editors from the Reform Index Editorial Board. They use the information from the official internet pages of the Verkhovna Rada (Parliament), President of Ukraine, Cabinet of Ministers, National Bank and other governmental institutions. During selection, the components and directions of the Index, which are affected by each normative act, are also determined.

Normative acts which came into force during the evaluation period or which have a set enactment date are evaluated.International agreements are also evaluated in case they have been signed by the Government or President and ratified by Parliament (if ratification is required). Media rumors, verbal statements by officials, blog posts, drafts of normative acts are not evaluated. All normative acts are evaluated following their publication. Introductions and cancellations of temporary anti-crisis measures, such as those concerning currency exchange and capital movements, are not evaluated. Consistent with the audit results from 01.09.2016, the Index does not evaluate temporary normative acts with a regulation validity period of less than one year and experiments or pilot projects that restrict the participation of economic agents (see the audit results here).

The Reform Index does not evaluate the quality of implementation or enforcement of normative acts.

After normative acts have been selected and classified, the project manager compiles a questionnaire listing the selected normative acts, each supplied with a short description and link to its official text. The manager then sends the questionnaire to Reform index experts and partner organizations for evaluation.

The evaluation is done on a «-5» to «+5» scale.

Experts evaluate the impact of each normative act along one or several directions defined in the questionnaire. Their evaluation results reflect a given normative act’s impact on a certain area (direction) of reforms (negative impact has a minus sign, and positive impact has a plus sign). A grade of +5/-5 for an event means that this event radically changes the rules of the game or significantly affects the behavior of many economic agents (in a positive or negative way, respectively).

After the normative acts have been assessed, experts evaluate the general reform progress for each of the five Index components (also on a -5 to +5 scale) in case there are other reforms in the respective focus areas. The Reform Index is calculated based on the evaluation of its components. While evaluating reform progress, experts take into account quantity and quality of the normative acts included in the given component. However, there is no clearly defined rule or formula linking normative acts to the component’s evaluation. Thus, the Reform Index is a subjective index, reflecting its experts’ perception of normative reform progress. If the experts did not give an assessment of the direction, it is calculated based on the sum of the event assessments given by the expert (if the expert did not give any assessment in a certain direction, then it is not calculated).

See here for detailed rules of evaluation (in Ukrainian).

Calculation method

Evaluation of normative act

Since one normative act may influence several directions, it may be included into several Index components. The evaluation of a normative act within a certain component is defined as the median of experts’ scores.

The total evaluation of a normative act is equal to the sum of evaluations on all components to which this act was assigned. Therefore, normative acts presented in several Index components have a higher evaluation than those with a single presence. Furthermore, a normative act which influences several components may have an evaluation greater than +/-5. This approach was adopted in order to highlight complex reforms that change regulations in several areas of economic activity.

Component evaluation

Experts evaluate the total reform progress in one or several components, taking into account the normative acts included there. As mentioned above, there is no clearly defined rule on how the evaluations of a certain component of the index are linked to normative act evaluations. The evaluation of the total reform progress in a certain area is at the discretion of experts. The final reform evaluation for each component is defined as the median score across all experts who evaluate the component.

Index calculation

The value of iMoRe for a given period is defined as the mean score of evaluations across all five components, thus it is based solely on the evaluation of the components and not on the evaluations of normative acts or reform directions. The IMoRe Editorial Board considers reform progress to be satisfactory if Index value being greater than +2.0 points during the evaluation period.

Calculation example

Suppose that three normative acts have been adopted during a given evaluation period:

A law setting a framework for restructuring state-owned coal mines, thereby potentially reducing subsidies from the state budget, was adopted and signed. The Editorial Board established that the law influences the efficiency of public expenditures (code 204) and energy independence (code 501), hence it was included into two Index components: I2. Public finance and I5. Energy sector.
A regulatory act mandating the detailed disclosure of state budget expenditures. The Editorial Board established that this normative act would encourage transparency of public finances and therefore influence efficiency of public expenditures (code 204). The regulatory act was included into I2. Public finance component.
A regulation to provide free heat and water meters for households earning less than the subsistence level. In the Editorial Board’s opinion, this decision would influence energy independence (code 501) and the state budget (code 299, other in public finances). Thus, it was included into two components: I2. Public finance and I5. Energy sector.

Let us assume that the normative acts and respective Index components received the following grades:

	Expert 1	Expert 2	Expert 3
Normative act 1	3	4	3,5
Normative act 2	2	1	2
Normative act 3	-1	-1,5	-2
І2. Public Finance	2	2,5	3

	Expert 1	Expert 2
Normative act 1	4	5
Normative act 3	1	1,5
І5. Energy	3,5	4

The evaluation of normative acts proceeded in two stages. First, evaluations within applicable Index components were made:

	Expert 1	Expert 2	Expert 3	Evaluation on the component I2. Public Finance
Normative act 1	3	4	3,5	3,5= median (3;4;3,5)
Normative act 2	2	1	2	2,0 = median (2;1;2)
Normative act 3	-1	-1,5	-2	-1,5 = median (-1; -1,5; -2)

	Expert 1	Expert 2	Expert 3	Evaluation on the component I5. Energy
Normative act 1	4	5		4,5 = median (4;5)
Normative act 3	1	2		1,5 = median (1;2)

In the second stage, the total score for each normative act was computed. Normative act 1 received 8.0 points (3.5 on Public Finance and 4.5 on Energy Sector). Normative act 2 got 2.0 points ( Public Finance only). Normative act 3 received 0.0 points (-1.5 on Public Finance and +1.5 on Energy Sector).

The value of the Index was calculated as follows:

	Expert 1	Expert 2	Expert 3	Evaluation of the component
Governance				0
Public Finance	2	2,5	3	2,5=median (2; 2,5; 3)
Monetary system				0
Business environment				0
Energy	3	4		3,5 = median (3; 4)
Human capital				0
Index Value				0,9=(0+2,5+0+0+3,5+0)/6

Evaluation of reform directions (complementary)

As mentioned above, each of the six Reform Index components spans one or more directions. For a more comprehensive analysis of reform progress complete, it is expedient to study the dynamics of directions separately. This step is complementary and is used in analytics and infographics.

A direction’s score is equal to the sum of scores of normative acts assigned to that direction. In case of a situation when a normative act influences several directions within one component, experts only evaluate this normative act for the component as a whole, not for the separate underlying directions. In such cases, the score of a given normative act is divided by the number of directions in the component in question.

For instance, in June 2016, the Government approved new regulations on benefit payments to temporarily displaced persons. The Editorial Board established that this regulation influenced two directions: social safety net and labor market (code 202), and efficiency of public expenditures (code 204), both of which are included into component I2. Public finance. Experts’ evaluation of this normative act for the Public finance component yielded a score of 1.0. Hence, the relevant component directions, 202 and 204, received 0.5pt each.

Annual audit

The IМоRе Editorial Board makes every effort to maintain the high quality of the Index. For this purpose, the iMoRe Editorial Board performs an annual audit of the regulations (normative acts) which were included into the Index in order to: a) identify the regulations that were either included erroneously (i.e. temporary regulations or measures that cannot be considered reforms) or overlooked and did not make it into the Index (this may be the case due to the vague definition of the term “reform”); b) improve the Index structure in terms of components and directions; and c) improve the calculation methodology.

During the audit, it may be decided to introduce a new reform direction. In this case, all previously evaluated normative acts are analyzed to determine whether they influence the new direction.

If a normative act is removed or transferred from an existing to a new direction, iMoRe and its components should be recalculated — the evaluation of the direction is calculated by the sum of events in it with a restriction — it cannot acquire values greater / less than the range [-5; 5].

Download audits summary (in Ukrainian)

Download tables with changes after the audits (in Ukrainian)

Path of Reforms

Reform Index Experts

Related Articles