Better outlier detection procedure

[memoryfull]

RFSD companion paper writes:

<...> we engaged in manual review of top-20 firms in terms of revenue or total assets within each 2-digit industry (excluding financial firms), firms with largest year-on-year changes in key financials, and firms with imputed statements and largest revenues. Our review has identified 436 firms that filed 1,130 anomalous statements in 2011–2023. The judgement was made based on the audit opinions, financials of known industry leaders, firm websites, or public information regarding the firms suspected of reporting anomalous values. We recommend RFSD users to exclude those companies from consideration and do so in our external validation.

The existing procedure to find firms with anomalous filings is manual by design. We have experimented with various rule-based procedures to detect outliers, comparing year-on-year changes in financials within firms or between-firm financials within industry. These experiments have yielded mixed results due to a large false positive rate. We would welcome any discussion of a reliable automated outlier detection procedure for the RFSD.

Background: https://t.me/rumka_ipp/792

[memoryfull]

The published version of the RFSD companion paper now engages with classical outlier detection algorithms in addition to manual classification:

Our manual review has produced an expert-made list of anomalous filing. Instead, one might consider classical or machine learning-based algorithms for outlier detection to increase the objectivity of the results. Hawkins defines an outlier as an observation that deviates so significantly from the rest of the data that it raises suspicion it was produced by a different data generating process. We applied five well-established classical outlier detection algorithms to the RFSD data, considering each second-digit industry and year as a separate data set. Such an industry-year stratification was necessary to account for high between-industry variation in financials (e.g., petrochemical producers will report much larger financials than furniture manufacturers), inflation, and temporal improvements in filing quality. As in our manual review, we considered Hawkins outliers only in terms of revenue and total assets, searching for anomalies in 19,377,043 firm-year observations of 3,844,541 firms reporting both variables in 2011–2023. We then compared the manual outlier classification with the outlierness scores of five statistical algorithms by calculating the Receiver Operating Characteristic Area Under the Curve (ROC AUC) for each year and averaging it over the period. As most algorithms require the neighbourhood size K to be specified, we used the Natural Neighbour algorithm to determine the optimal K for each industry-year before applying the outlier detection algorithms. All calculations were performed using the DDOutlier package in R. The results are in Table 1. We immediately see that most of the algorithms are in relatively high agreement with manual classification, judging by the AUC in the 0.8–0.9 range, with HiOut reporting the highest conformance. However, the key challenge in relying on statistical outlier detection is that one still needs to specify a threshold of outlierness score to treat an observation as an outlier. This problem is highlighted in columns 2–8 of Table 1. If we set the score threshold at the minimum value of the score among the manually identified outliers, each algorithm will classify 1.2–1.4 million firms as outliers each year, which is unrealistic. If the equate the threshold to the median score among the manually identified outliers, we will classify 1.5–3.6 thousand firms as outliers by any algorithm, except for KDEOS. These results emphasize that while the statistical outlier detection algorithms are producing objective results, threshold selection is still an expert-driven process requiring supervised data. Furthermore, the experts may better identify Hawkins outliers since they rely on external data such as audit opinions in their manual review. For these reasons we elect to rely on manually defined outliers in our coding.

I will leave this issue open for now in case anyone wants to contribute.