基于混合专家的海量行业数据质量智能检测方法研究

Research on Intelligent Quality Detection Method for Massive Industry Data Based on Mixture of Experts

  • 摘要: 当前,海量数据已成为装备制造企业转型升级的核心数字底座。然而,由于其存在多源异构、物理关联、时空耦合等特性,传统数据检测方法难以识别其隐蔽的逻辑冲突与系统级异常。为解决复杂数据场景下的质量检测难题,提出基于混合专家的海量行业数据质量智能检测方法,构建包括回归分析、聚类分析及语义规范的专家子库,并结合Top-k稀疏激活路由策略与负载均衡损失函数设计基于门控网络的自适应路由机制。实验结果表明,该方法能够将混合数据检测的F1分数由0.320 0提升至0.705 0,路由命中率Hit@1达0.973 7,克服了单一检测模型在处理混合字段时的性能局限性。

     

    Abstract: Currently, massive data has become the core digital foundation for equipment manufacturing enterprises.Due to its characteristics such as multi-source heterogeneity, physical correlation, and spatiotemporal coupling, traditional data detection methods struggle to identify hidden logical conflicts and system-level anomalies.To address the quality detection challenges in complex data scenarios, driven by multi-source data, an intelligent detection method for massive industry data quality based on mixture of experts is proposed.An expert sub-library including regression analysis, clustering analysis, and semantic norms is constructed, and a self-adaptive routing mechanism based on gated networks is designed by combining the Top-k sparse activation routing strategy and the load balancing loss function.Experimental results show that this method has increased the F1 score of mixed data detection from 0.320 0 to 0.705 0, with a routing hit rate of Hit@1 reaching 0.973 7, and overcomes the performance limitations of a single detection model when processing mixed fields.

     

/

返回文章
返回