Join Order Optimization with Learned Uncertainty Estimates for Robust Execution under Cardinality Misestimation in E-Commerce Marketplace Analytics

• Karim Nabil

  Department of Computer Science, Nile Valley Institute of Technology, Gamal Abdel Nasser Street, Beni Suef 62511, Egypt

  Author

• Mostafa Saleh

  Department of Information Systems, Delta Academy of Computing, Port Said Street, Tanta 31512, Egypt

  Author

E-commerce marketplace analytics increasingly relies on complex relational queries that join high-volume event logs, transactional facts, and slowly changing dimensions to derive timely measures of demand, conversion, inventory health, and seller performance. These workloads are executed under rapidly shifting data distributions driven by promotions, seasonality, catalog churn, and heterogeneous seller behavior, which makes cardinality estimation particularly error-prone. Join order optimization is therefore exposed to systematic risk: a plan that is optimal under point estimates can become brittle when intermediate result sizes deviate by orders of magnitude, triggering memory spills, skew amplification, and poor parallelization decisions. This paper studies join order optimization augmented with learned uncertainty estimates, targeting robust execution when cardinalities are misestimated. We propose a probabilistic formulation in which learned models predict not only expected selectivities and join cardinalities but also calibrated uncertainty, represented as distributions over latent factors that capture predicate correlation, skew, and time drift. The optimizer then searches for join orders that minimize a risk-sensitive objective combining expected cost with tail-risk penalties, while preserving the ability to adapt at runtime using progressive feedback from partial execution. The resulting approach integrates uncertainty propagation through join trees, robust cost modeling for hash and sort-merge joins under memory constraints, and lightweight plan switching based on statistically grounded triggers. We focus on marketplace analytics patterns and show, through controlled experimental design, how uncertainty-aware optimization reduces severe regressions under misestimation, improves plan stability, and yields predictable latency distributions without requiring manual hints or conservative heuristics that degrade average performance.

• PDF