Judicial Statistics: Why Do People Distrust the Judiciary?

On March 8, 2017, at the second meeting of the 3rd Division of the National Conference on Judicial Reform, then-Academy for the Judiciary President Tsai Pi-Yu mentioned in her presentation: "The 2016 survey report on the general public's perception of the judiciary... in fact, looking at the statistics (Figure 1 below), the factors currently affecting public trust in the judiciary are overwhelmingly influenced by television, the internet, and newspapers. As for personal firsthand experiences influencing public perceptions of the judiciary, it is actually only 3.2%. This is because the public (72.7%, Figure 2 below) believes talk show pundits' rhetoric that 'if you have money you live, if you are poor you die.

Click image to enlarge

Click image to enlarge

Clearly, the Academy for the Judiciary probably does not teach statistics, which led President Tsai to misinterpret the results shown by the statistical data without conducting any analysis.

Let us start from the beginning. When reviewing a statistical report, the first things to note must be its sampling method and questionnaire design. According to page 9 of this report:

This survey targeted citizens aged 20 and over residing in 20 counties and cities across the Taiwan region. Excluding non-human factors (busy phone lines, unanswered calls, faulty phone lines, disconnected numbers, fax machines, non-residential phone lines, lack of qualified respondents, and full quotas), a total of 41,006 individuals were contacted, of whom 34,251 refused to be interviewed, 1,490 agreed to an interview but did not complete it, and 5,265 valid questionnaires were ultimately completed. The success rate of the interviews was 12.8%.

In current telephone polling in Taiwan, a refusal rate generally falling between 20% and 30% is common and often questioned for lacking credibility. Yet this survey, commissioned by the Judicial Yuan and conducted by Evidence Polls Co., Ltd., had an astonishing refusal rate of 83.5%! Although the sample representativeness[2] is nearly identical to the population in all aspects except for education level (see chart below), one cannot help but question whether the reliability of this survey is sufficient.

However, let us temporarily set aside the issue of reliability and examine Ts'ai Pi-yü's deduction! If her deduction is broken down using the legal profession's favorite syllogism, it is as follows:

1. Major Premise: The public's primary channels for obtaining judicial news or information are television, the internet, and newspapers.
2. Minor Premise: The public believes talk show pundits' rhetoric that "if you have money you live, if you are poor you die."
3. Conclusion: "The media indeed dominates the degree of public trust in the judiciary" (Ts'ai Pi-yü, Presentation Page 22).

Anyone with a discerning eye knows this is a logical fallacy. In statistics and social sciences, we must constantly bear in mind that "correlation does not equal causation." Let us first test whether there is any correlation between the two!

First, as mentioned at the outset, how a questionnaire is designed is crucial! As seen in Figure 1 below, the phrasing for the question "Through which channels do you obtain judicial news or information?" is "multiple-choice, unprompted," meaning respondents simply answered with what came to mind. Consequently, we cannot determine which channel was the primary one among those who reported multiple channels. Furthermore, Questions 3-8 (see Figure 2 below) first mention television pundits and then ask whether respondents agree that "if you have money you live, if you are poor you die," which carries the suspicion of leading questions. The designers of this questionnaire clearly intended from the start to link the media with judicial trust; therefore, we cannot help but question the validity of this questionnaire.

Let us first look at the public's own views on the media. Based on the responses to Questions 3-7 (see chart below), 40.85% of respondents felt there was no impact, showing that Ts'ai Pi-yü's view ("The media indeed dominates the degree of public trust in the judiciary") is not shared by everyone.

Next, let us analyze the viewing habits of television media among those who believe there is an impact. I first divided all samples into three categories based on Question 20, as shown in the chart below:

Subsequently, by cross-analyzing the two, we can obtain the following chart:

We can see a startling result: among those who only watch television, the proportion of people who believe television media has a positive impact on perceptions of the judiciary is actually higher compared to other groups! We could even say that television media has enhanced the public's perception of the judiciary! At this point, the conclusion of the Academy for the Judiciary has already been debunked.

However, we cannot ignore that a higher proportion of people still believe the media has a negative impact on perceptions of the judiciary. Therefore, I combined questions 3-1 to 3-5 (as shown below) to create a so-called "Judicial Trust Index," where a lower score represents higher trust in the judiciary, and conversely, a higher score represents lower trust.

After calculation, the average score of the Judicial Trust Index is 17.1, which, when translated into text description, falls roughly between "average" and "somewhat distrustful."

Meanwhile, to verify whether the proposition that "among the factors affecting public trust in the judiciary, television media accounts for the overwhelming majority of the influence" holds true, I designed a model incorporating gender, age, education level, actual judicial literacy level^[3], degree of understanding the judiciary through television media, and whether one has ever been to court. Furthermore, to compare the influence of each variable, I standardized all variables beforehand, with the results shown in the chart below:

Let us explain this table step by step. First, looking at the P>|t| column, we can see that the P-value for education is large, indicating that differences in "education level" have very little relationship with "whether the public trusts the judiciary." Conversely, however, this also means that public distrust of the judiciary is a shared phenomenon across all education levels. Apart from education level, differences in all other variables can be considered correlated with "whether the public trusts the judiciary." But which factor is the most correlated? We can look at the Coef. column: in terms of absolute value, the larger the number, the stronger the correlation. In descending order, they are: age (older people are more distrustful of the judiciary), actual judicial literacy level (higher literacy leads to greater trust), and whether one has been to court (those who have been to court are more distrustful than those who have not). Ironically, "the degree of understanding the judiciary through television media" shows the least correlation.

We must candidly admit, however, that the R-squared^[4] in the top right corner is only 0.03, meaning this is not a model with strong explanatory power. In other words, there are other reasons explaining why the public trusts/distrusts the judiciary that I did not take into account. But at the very least, we can confirm that "public trust or distrust of the judiciary" is by no means a simple issue, nor can it be resolved by simply shifting all the blame onto the media. (The chart below is from Ts'ai Pi-yü's presentation)