📈 economics

Expected versus Unexpected Non-Competition: A Corruption-Risk Approach to Public Procurement in Spain

This paper introduces a machine learning-based framework using Spanish public procurement data to distinguish between expected and unexpected non-competition, thereby creating a refined, false-positive-reducing proxy for corruption risk that accounts for structural market and institutional factors.

Original authors: Iván Pastor-Sanz, Félix J. López Iturriaga

Published 2026-10-19

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Iván Pastor-Sanz, Félix J. López Iturriaga

Original paper licensed under CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Public procurement is the process by which governments buy the goods, services, and infrastructure needed to run a country, from building roads to supplying hospitals. Because these contracts involve vast sums of money and complex rules, they are a frequent target for corruption. One of the most common warning signs that something is wrong is a lack of competition. When a government asks for bids on a project, a healthy market usually produces several companies competing for the work. If only one company shows up, or if the bidding process is kept secret, it often suggests that the contract was rigged to favor a specific supplier. For years, auditors and watchdogs have relied on simple lists of these warning signs, known as red flags, to spot suspicious deals. However, these lists have a major flaw: they cannot tell the difference between a contract that is suspiciously uncompetitive and one that is simply uncompetitive because of legitimate reasons, such as a specialized market where few companies exist or a complex project that scares off bidders. This confusion leads to many false alarms, wasting time and resources on contracts that are actually clean.

To solve this problem, researchers Iván Pastor-Sanz and Félix J. López-Iturriaga turned to a massive collection of data from Spain, analyzing 393,631 public tenders awarded between 2018 and 2024. Instead of using a static checklist, they built a computer model that learns from the history of these contracts to understand what a "normal" level of competition looks like for any given situation. The model takes into account details available before a contract is awarded, such as the type of work being done, the specific rules used to select the winner, and the region where the project is located. By studying these factors, the model calculates the expected probability that a tender will attract fewer than three bids. It then compares this expectation to what actually happened. If a contract receives very few bids when the model predicted it should have attracted many, that gap is flagged as an unexpected anomaly. This approach allows officials to separate the noise of normal market conditions from the signal of potential corruption.

The researchers found that the most important factors determining whether a contract gets many bids or just one are not the amount of money involved, but rather the type of project and the rules used to run the bidding process. The specific sector of the economy, such as construction or IT services, explained nearly half of the variation in competition levels. The type of procedure used to award the contract was the second most powerful predictor. This confirms a long-held view in political science that the amount of freedom or discretion given to government officials in designing the bidding process directly shapes the opportunity for favoritism. Contracts awarded through procedures that allow officials to negotiate directly with suppliers without public advertising were far more likely to have low competition, but the model showed that this was often an expected outcome of the rules, not necessarily proof of a crime.

The true value of this work lies in its ability to filter out the false alarms. By focusing only on the contracts where the actual competition was significantly lower than what the model expected, the researchers identified a tiny, highly suspicious group of tenders. Out of nearly 400,000 contracts, the model flagged just 2,195 cases, or about 0.56 percent, as unexpectedly uncompetitive. These are the deals that deserve the closest scrutiny because they deviate sharply from the norm for their specific type and location. The study showed that these flagged cases strongly correlated with other established measures of corruption risk, suggesting the model is correctly identifying genuine integrity issues rather than just random noise. This method offers a way for oversight bodies to move from reacting to every low-bid contract to proactively targeting the specific few where the risk is highest, ensuring that limited audit resources are spent where they can do the most good.

The findings suggest that while low competition is a useful indicator, it is not a perfect one on its own. The study does not prove that every flagged contract is corrupt, but it provides a much sharper tool for finding them. By distinguishing between the expected lack of competition caused by market realities and the unexpected lack of competition that hints at foul play, this approach helps governments manage the massive flow of public funds with greater precision. As Spain continues to receive billions of euros in recovery funds from the European Union, the ability to monitor these contracts effectively is more critical than ever. The research demonstrates that using data to understand the context of a deal is far more powerful than simply counting the number of bidders, offering a clearer path toward transparency and accountability in public spending.

Technical Summary: Expected versus Unexpected Non-Competition in Spanish Public Procurement

Problem Statement
Public procurement in Spain, particularly given its role as a major recipient of Next Generation EU funds, faces significant corruption risks often proxied by limited competition (e.g., single or below-threshold bidding). However, traditional "red-flag" indicators suffer from a critical measurement problem: they conflate benign low competition—driven by legitimate market structures, sectoral characteristics, or procedural constraints—with suspicious low competition indicative of rent-seeking or favouritism. Static thresholds generate high rates of false positives, failing to distinguish between expected non-competition (structural) and unexpected non-competition (potentially corrupt). This paper addresses the need for a more refined, ex-ante measure of integrity risk that can separate predictable market outcomes from anomalies warranting audit scrutiny.

Methodology
The study utilizes a comprehensive dataset of 393,631 awarded public tenders in Spain between 2018 and 2024, sourced from OpenTender and the Spanish national procurement platform. The data adheres to the Open Contracting Data Standard and includes pre-award characteristics such as contract type, Common Procurement Vocabulary (CPV) sectors, estimated values, procedure types, and geographical locations.

Target Variable: A binary indicator, LowCompetition, defined as receiving fewer than three bids.
Feature Engineering: Thirteen explanatory variables were constructed exclusively from pre-award information to prevent information leakage. Continuous and categorical predictors underwent Weight of Evidence (WoE) encoding to handle missing values, enforce monotonicity, and capture non-linear relationships while preserving interpretability.
Modeling Approach: A Gradient Boosting Machine (GBM) was selected following comparative analysis with logistic regression, Random Forests, and decision trees. The model was optimized via randomized grid search (focusing on AUC) with hyperparameters including boosting iterations (50–300), learning rate (0.01–0.1), and tree depth (3–5).
Validation Strategy: The dataset was partitioned temporally: data prior to July 2024 was split 70/30 for training and testing, while data from July 2024 onwards served as an out-of-sample validation set to ensure temporal stability and simulate real-world ex-ante risk assessment.

Key Contributions

Decomposition of Non-Competition: The paper introduces a model-based counterfactual that decomposes realized non-competition into an "expected" component (driven by structural and procedural factors) and an "unexpected" residual component. This residual serves as a refined proxy for integrity risk, reducing false positives inherent in static red-flag systems.
Political Economy Insight: By identifying the dominant determinants of competition, the study provides empirical evidence supporting the political-economy view that administrative discretion and market structure, rather than contract value alone, shape the scope for favouritism.
Contextual Extension: The research extends existing literature, which is largely based on Italian, French, or German datasets, to the Spanish context—a critical case study due to the volume of EU recovery funds involved.
Operationalizing Machine Learning: The study positions machine learning not merely as a predictive tool but as a measurement device for the political economy of government contracting, offering a transferable framework for risk-based auditing.

Results

Predictive Performance: The GBM model achieved an Area Under the Curve (AUC) of 0.778 on the test set and 0.727 on the temporal validation set, demonstrating robust discriminative power and temporal stability.
Feature Importance: The analysis reveals that the procurement sector (accounting for nearly 50% of explanatory power) and procedure type (19.6%) are the dominant predictors of competition levels. Notably, contract value ranked only eighth (1.9%), challenging the assumption that financial magnitude is the primary driver of competition risk.
External Validity: The model's predicted probability of low competition showed strong negative correlations with OpenTender's integrity scores (approx. -0.52) and single-bid indicators, confirming that the model captures meaningful dimensions of procedural risk.
Anomaly Detection: By defining "unexpected low competition" as tenders with a predicted probability of low competition below a threshold (e.g., $\tau = 0.30$ ) that nonetheless received fewer than three bids, the model identified 2,195 cases (0.56% of the total sample). These anomalies were concentrated in generally competitive procedure types (e.g., open procedures), whereas structurally restrictive procedures showed few anomalies, validating the model's ability to isolate deviations from expected norms.

Significance and Claims
The paper claims that its approach offers a significant advancement over traditional compliance monitoring by shifting from reactive, ex-post checks to proactive, risk-based oversight. By isolating the "unexpected" component of non-competition, the model allows oversight bodies to prioritize a small, manageable subset of tenders for deep-dive audits, thereby optimizing limited investigative resources. The authors assert that this method does not replace human judgment but provides a structured, interpretable basis for prioritization, helping to distinguish between legitimate market conditions and potential irregularities. The study concludes that interpretable predictive tools can effectively complement existing red-flag systems to enhance the efficiency and targeting of public procurement audits, particularly in high-stakes environments like EU-funded projects.

Technical Summary: Expected versus Unexpected Non-Competition in Spanish Public Procurement

More like this