Evaluating Anti-Drug Policies: From Feeling the Impact to Realizing Change

On June 1, 2018, the Ministry of Justice released a press release stating that the "Satisfaction Survey on Government Anti-Drug Actions in 2018" showed 68% of respondents affirmed the government's various efforts over the past year in combating, preventing, resisting, and treating drugs. The attached polling report indicated that the policy with the highest public satisfaction was the "legislative direction to increase penalties for drug trafficking and confiscate proceeds from drug trafficking," at 87.6%. Conversely, the policy with the lowest satisfaction was the "effectiveness of establishing a national drug database to assist in drug enforcement," at 42.1%. This poll and press release came about because one of the cross-agency KPIs for major policies under the National Development Plan established by the National Development Council (NDC) in 2018 was the "anti-drug strategy." Its KPI was defined as "public satisfaction with the government's anti-drug actions at 50% or higher," with the measurement standard being "a satisfaction survey conducted via polling by the NDC."

Using public satisfaction to evaluate the effectiveness of anti-drug strategies is a poor approach, and unfortunately, the relevant ministries do not seem to realize this. This article will first explain the importance of drug policy evaluation, discuss the problems with using public satisfaction to assess anti-drug strategies, and recommend alternative practices.

Why Drug Policy Evaluation Matters

In 2017, the European Union published a drug policy evaluation guide titled Evaluating drug policy: a seven-step guide to support the commissioning and managing of evaluations, which points out that policy evaluation is vital for decision-making because it helps ensure that policies and related programs achieve their intended effects, use resources efficiently, and avoid unintended negative consequences.

For drug policies, appropriate policy evaluation methods are particularly crucial because "drugs" easily evoke fear in our society. "Highlight" policies formulated in response to fear may tend to create a short-term public feeling that "the government is taking action" and "we can feel at ease," but they do not necessarily solve the problem and may even generate numerous long-term side effects. A common practice in Taiwan is to increase the criminal penalties for drug-related offenses without adequate supporting measures. Under conditions of overcrowded prisons and insufficient treatment resources, arresting more people and locking them up for longer periods only leads to foreseeable recidivism when they are released and unable to reintegrate into society.

Problems with Using Public Satisfaction to Evaluate Anti-Drug Strategy Effectiveness

r

To explain the problems of using "public satisfaction surveys" to evaluate the effectiveness of anti-drug strategies, we must first clarify what an "indicator" is. An indicator is a quantitative or qualitative element or variable used to measure what you intend to measure.

To "measure what you want to measure," you must first define "what you want to measure" before considering "what indicators to use for measurement." In the context of drug policy, you must first decide whether the broad direction you wish to measure is "how satisfied people are with anti-drug policies" or "what impact a specific policy has had on the drug abuse situation." This decision then dictates the indicators you choose. If the former is chosen, setting the indicator as "public satisfaction" and the measurement method as a "public opinion poll" makes logical sense. However, if the latter is chosen, it becomes absurd, because people being "satisfied" with a specific policy does not necessarily equate to the actual impact that specific policy has on society.

The Taiwanese government has repeatedly emphasized that drug abuse is rampant in Taiwan, and that the government intends to improve and resolve this issue. Therefore, logically, the indicator should measure "what impact a specific policy has had on the drug situation." For example, Premier William Lai (Lai Ch'in-te) instructed at the Anti-Drug Board meeting on May 23, 2018, that all ministries should make reducing the population of first-time drug addicts the primary goal of anti-drug work. Setting aside whether this goal is appropriate ^[1], the indicator it should adopt is at least the "incidence rate of first-time drug use," rather than "public satisfaction with the government's anti-drug actions."

The "public satisfaction with government anti-drug actions" poll report itself demonstrates why it is unsuitable as a KPI. The policy with the highest public satisfaction in the survey report was the "legislative direction to increase penalties for drug trafficking and confiscate proceeds from drug trafficking." Yet, can anyone tell us what concrete impact this policy has actually had on society? In 2015, Taiwan raised the minimum statutory penalty for manufacturing, selling, and transporting Category 3 drugs from 5 years to 7 years (the maximum statutory penalty remained unchanged at an upper limit of 15 years' imprisonment). Did the government evaluate how many people were actually deterred from drug trafficking by this legal amendment? What impact did locking up different types of drug traffickers for longer periods have on their recidivism? When impact evaluations are completely ignored, using public satisfaction to claim the effectiveness of a specific policy is nothing short of self-deception.

Furthermore, the policy with the lowest public satisfaction was the "effectiveness of establishing a national drug database to assist in drug enforcement." To understand the effectiveness of establishing a national drug database in assisting drug enforcement, shouldn't one look at how many (major) drug cases were successfully cracked after the database was established? If satisfaction is truly to be assessed, it should involve understanding to what extent law enforcement personnel feel the database has assisted them in their work. The lowest public satisfaction does not mean the database's effectiveness in assisting drug enforcement is poor; it likely only means the public does not quite understand what the database is.

The mild consequence of improper policy evaluation is "failing to measure what truly needs to be measured," while the severe consequence is guiding governance in the wrong direction. The Minister of Justice himself stated that "citizens in our country are superstitious about heavy penalties." If opinion polls continue to be used to evaluate the effectiveness of anti-drug policies, it will likely remain difficult to break this superstition.

Recommended Practices

Regarding how drug policy evaluation should be conducted, this article does not believe that directly prescribing recommended KPIs is a good approach. The true way to solve the problem is to build the right conditions for policy evaluation, allowing good evaluations to grow naturally. Below, referencing the EU drug policy guidelines, this article suggests how the Taiwanese government can build the conditions to develop policy evaluations. Readers interested in understanding how EU drug policy evaluations are practically conducted can also refer to the EU drug policy evaluation webpage, which lists the EU's past drug strategies, plans, and evaluations. This provides insight into how the policy evaluations recommended in the guide have evolved step-by-step and offers a vision of what a well-executed policy evaluation report in Taiwan could look like.

1. Laying the Foundation

   Good policy evaluations require certain basic conditions:

   Commitment from leaders. Government evaluations require the investment of resources, time, money, and personnel. Moreover, evaluation recommendations must be fed back into the next steps of action to be meaningful. Therefore, a deep commitment to this work from policy-making officials within the government is crucial.

   Stakeholder participation. Government evaluations require the participation of stakeholders because they can help obtain data, ensure practical approaches, and ensure evaluations cover important dimensions of the policy. Furthermore, a steering group comprising stakeholders should be formed to provide guidance throughout the evaluation process. Members of the steering group should include: decision-makers from relevant government departments, civil servants responsible for helping implement decisions, target groups affected by the policy (e.g., relevant medical professionals, police, social workers, and service-user groups), experts familiar with drug policies and evaluation methodologies, and civil society organizations.

   Judging from Taiwan's establishment of the NDC and its inclusion of setting KPIs as part of national development, we can see that the government is gradually attaching importance to policy evaluation, though it is not yet sufficiently mature. In the pursuit of meeting KPIs, many civil servants are also running themselves ragged. Perhaps within the NDC, we need leaders to begin understanding how good policy evaluations are conducted and whether they have the genuine willingness to promote this area.

   Moreover, looking at the NDC's meeting minutes regarding the formulation of the anti-drug strategy KPI, we currently see only representatives from relevant government departments attending and discussing the matter, which has created several problems. Based on the meeting minutes, two forces are tugging at the formulation of the anti-drug strategy KPI: one is the meeting chairperson, who believes that rather than setting KPIs that are easily achievable, units often fail to set truly important ones. Because units only set achievable goals rather than ones that resonate with the public, the public is dissatisfied with government actions. The other force consists of representatives from relevant government departments, some of whom believe public satisfaction is not a good indicator, while others worry that setting public satisfaction as a KPI is beyond their control and might not be successfully met. The Minister of the NDC wants to break away from the past practice of setting only easily attainable KPI targets, and this effort is commendable; however, directly pivoting to public satisfaction is not a sound approach. The key to deepening discussions probably still lies in incorporating stakeholders into policy evaluation. Given the composition of the NDC, it may not be entirely suitable to list stakeholders as committee members; however, is it possible for the NDC to authorize the establishment of a working group that includes stakeholders to participate in the planning of policy evaluations and continuously monitor their progress?

   Finally, there is a problem specific to Taiwan that impacts whether working groups can operate smoothly: government agencies sometimes draft resolutions before meetings even take place, a habit that easily leads to the hollowization of meetings. This phenomenon is evident in the NDC's meeting minutes. The chairperson had essentially predetermined that public satisfaction would be used as the anti-drug strategy KPI, leaving little room for substantive exchange of opinions and mutual persuasion. Chairpersons and staff certainly can and must prepare countermeasures for discussion in advance. To truly do this right, two things must be done: first, pre-meetings where staff invite participants to provide input and help organize it before the meeting starts. Second, outlining different options and explaining the pros and cons of each for participants to reference, rather than presenting only a single option. These preparatory works must also ensure the chairperson is fully briefed, so that their job is to use pre-meeting preparations to turn on-site discussions into efficient communication.

2. Establishing an Evaluation Team

   Policy evaluation requires a professional team; it cannot be accomplished simply by operational units working hard on paperwork. Having operational units explain how much work they have done themselves and evaluate their own effectiveness is currently the norm in Taiwan. However, policy evaluation involves evaluation design, which further entails research questions, research methodologies, continuous data collection, and data analysis. This cannot be managed as an add-on task by operational units, let alone expected to be impartial when evaluating their own performance.

   Establishing an evaluation team requires deciding whether to build an internal team or hire an external team. The advantage of an internal team is a better understanding of the policy context and easier access to data, whereas the advantage of an external team is greater independence. Given Taiwan's current situation, it is recommended to prioritize an internal team, incorporate external experts as needed, and publish the evaluation reports. If even greater independence is required, commissioned research projects can be organized irregularly and contracted out to external teams.

   Prioritizing an internal team is recommended because of the inherent convenience in internal government communication and data acquisition. Establishing such a team also represents an attempt to gradually separate evaluation work from operational units from within, with the evaluation team taking over. The evaluation team must communicate with operational units regarding what data should be provided. Operational units will still have a certain amount of paperwork and self-evaluation, but data output will be conducted in a more meaningful way. Furthermore, this requires the government to cultivate professional evaluation talent over the long term, interact continuously with relevant operational units, and build an internal evaluation culture. If the internal team lacks certain talents, recruitment can be extended outward. The evaluation reports of the internal team should be made public for external scrutiny, using transparency to compel the team to remain impartial.

   Since the Ministry of Justice remains the lead agency for drug policy, it is recommended that policy evaluation be undertaken by the Research Center for Criminal Policy and Corrections of the Academy for the Judiciary, Ministry of Justice. However, cross-ministerial cooperation is essential because drug policy involves multiple ministries—the Ministry of Health and Welfare, the Ministry of Labor, the Ministry of Education, and the Ministry of the Interior are all relevant, and the data needed for evaluation may be scattered across different agencies. Initiating this work may initially require a significant effort just to ask different ministries to provide data, but as systems are gradually established, it will also become easier for external evaluation teams to conduct assessments.

3. Determining Evaluation Types, Scope, and Designing the Evaluation

   The policy evaluation steering group mentioned earlier must determine the type and scope of the evaluation, while the evaluation team collaborates with the steering group to flesh out the evaluation design.

   Policy evaluations are divided into ex-ante, mid-term, and ex-post evaluations, and their scope can cover overall policies or individual elements within a policy. A typical ex-ante evaluation is the human rights impact assessment conducted when bills are submitted to the Legislative Yuan. Mid-term evaluations are more about understanding whether gaps exist between policy formulation and implementation, as well as conducting a preliminary review of the effects produced by governance. Ex-post evaluations primarily examine whether a policy has achieved its originally set goals, its expected or unexpected effects, and provide subsequent governance recommendations. The difference between an overall policy evaluation and an individual element evaluation is that the former explores relationships between different policy elements—though it is difficult to conduct in-depth due to its broad scope—while the latter deeply examines specific policy elements for a more thorough analysis.

   Judging from the NDC's current design of the anti-drug strategy KPI, it exhibits characteristics of both mid-term and ex-post evaluations, with the scope encompassing the overall policy, resembling the concept of the public periodically grading the government's governance performance.

   If the NDC primarily wishes to evaluate annual government governance performance and break away from the practice where government departments set their own easily achievable KPIs, it is recommended to start by conceptualizing an ex-post evaluation. To conceptualize an ex-post evaluation, discussions should begin with which drug problems are deemed extremely serious and in need of improvement. Starting from phenomena that must be improved, one can then determine how to measure these phenomena and understand how governance actions alter them. At this stage, the evaluation team must operationalize this into research questions, research methods, and data collection. In short, policy evaluation is split into two stages: "conceptualizing what needs to be changed" and "conducting the evaluation design."

   A crucial thinking tool for conceptualizing ex-post evaluations is "result chains," sometimes referred to as "cause-and-effect chains" or "logic models." For those interested in the application of result chains in Taiwan's drug policy analysis and the resulting outcome evaluations derived from them, please refer to the article titled "The New Generation Anti-Drug Strategy—An Aspiration Yet to Be Realized"; this article will not repeat those details.

   Finally, it must be reiterated that implementing an evaluation design is meaningful only if time and money are spent to establish monitoring data; it cannot be resolved simply by commissioning an opinion poll. Regarding overall policy, if the government is deeply concerned about juvenile drug abuse, a long-term social survey on adolescent substance abuse may be necessary. Although such social surveys suffer from underreporting issues, they can be used to monitor long-term trends in drug abuse. While they may not establish causality when applied to overall drug policy evaluations (due to the multitude of macro-policy variables making control difficult), they can monitor trend changes alongside simultaneous policy actions, suggesting areas where policies may need adjustment. Beyond overall policy, regarding individual policy elements, it is recommended to prioritize establishing monitoring data related to the treatment outcomes of drug offenders, as the data scope in this area is limited and relatively easy to obtain, and the government inherently has a responsibility to provide appropriate treatment assistance for their reintegration into society.

Conclusion—From Feeling the Impact to Realizing Change

Taiwan's use of "public satisfaction with government anti-drug actions at 50% or higher" as the anti-drug strategy KPI is a flawed approach because it measures whether the public "feels the impact," rather than measuring what has actually been "changed" by anti-drug actions. Viewed within Taiwan's local context, using public satisfaction to evaluate the effectiveness of drug policies is not overly surprising; historically speaking, it can be described as a product of Taiwan's democratic transition—namely, various political forces gradually growing accustomed to appealing to "public opinion" and treating it as a matter of course. Amid this taken-for-granted assumption, several problems have emerged, one of which is overlooking the fact that public "perceived impact" and actual "change" are two entirely different things, yet they lie worlds apart in policy evaluation.

This article argues that changing inappropriate policy evaluation practices must begin with cultivating the soil from which policy evaluations grow. Therefore, it offers several recommendations for soil improvement, including the establishment of policy evaluation steering groups and evaluation teams, as well as considerations on how to plan and design policy evaluations.

In improving the soil for policy evaluations, the government may need to try breaking away from the mindset of "highlight policies"—that is, taking actions that instantly make the public feel the impact, win applause, and translate into votes. Such a change is not easy, but it is worth trying for friends both inside and outside the government who care about Taiwan's future. Inside the government, a few leaders and departments can gradually foster an evaluation culture, while voices outside the government should cease overly demanding that policies merely make the public "feel" something, advocating instead for genuine change.

※This article was published in the PTS News Issues Center on June 5, 2018

---

Notes

1. Reducing the population of first-time drug addicts is not necessarily a good policy goal because it fails to grasp the core of drug harm. Not all psychoactive substances classified as drugs carry the same level of harm, nor do all people who use drugs become addicted. Furthermore, placing excessive emphasis on reducing the population of first-time addicts tends to overlook the harms generated by the policy itself. To reduce the number of first-time addicts, the most intuitive approach is to hunt down drug users, isolate them, and monitor them to prevent them from "contaminating" others who have not been exposed to drugs. However, this simultaneously brings about severe stigmatization, making it extremely difficult for them to survive in society. This practice of manufacturing stigma and social exclusion can also be seen in the "An-Chu Project" (Safe Residence Project). Refer to the An-Chu Project poster, "Join Hands to Eradicate the Drug Menace" (asking citizens to treat neighbors involved with drugs in the community as a public hazard and call them out), as well as news reports on the An-Chu Project, "Drive Drug Bugs Out of the Community! National Prosecutors and Police Observe Kaohsiung's 'Community Anti-Drug Network'".
2. Human rights impact assessments for bills in Taiwan are poorly executed and have degenerated into mere formalities. This issue is pointed out in the Legislative Yuan's special research report titled "A Discussion on the Effectiveness of Taiwan's Promotion of Gender and Human Rights Impact Assessment Mechanisms".