信用风险数据处理_datanalysis-credit-risk

以下为本文档的中文说明该技能是信用风险数据清洗与变量筛选管线用于贷前建模前的数据预处理。主要功能是处理原始信用数据涵盖数据质量评估、缺失值分析和变量选择。使用场景包括处理需要质量评估的原始信用数据需要进行缺失率分析的数据集在建模前需要进行变量筛选的信用风险分析项目。完整的处理流程包括11个独立步骤获取数据——加载和格式化原始数据机构样本分析——统计各机构的样本量和坏样本率分离OOS数据——从建模样本中分离出样本外数据过滤异常月份——移除坏样本数或总样本数不足的月份计算缺失率——计算每个特征的整体和机构级别缺失率删除高缺失率变量——移除缺失率过高的特征过滤低IV变量——移除信息值低于阈值的变量过滤高PSI变量——移除群体稳定性指标过高的变量空值重要性去噪——通过空值重要性分析去除噪声特征移除高相关性变量——删除高度相关的冗余变量生成清洗报告——输出完整的数据清洗过程报告。Data Cleaning and Variable ScreeningQuick Start# Run the complete data cleaning pipelinepython.github/skills/datanalysis-credit-risk/scripts/example.pyComplete Process DescriptionThe data cleaning pipeline consists of the following 11 steps, each executed independently without deleting the original data:Get Data- Load and format raw dataOrganization Sample Analysis- Statistics of sample count and bad sample rate for each organizationSeparate OOS Data- Separate out-of-sample (OOS) samples from modeling samplesFilter Abnormal Months- Remove months with insufficient bad sample count or total sample countCalculate Missing Rate- Calculate overall and organization-level missing rates for each featureDrop High Missing Rate Features- Remove features with overall missing rate exceeding thresholdDrop Low IV Features- Remove features with overall IV too low or IV too low in too many organizationsDrop High PSI Features- Remove features with unstable PSINull Importance Denoising- Remove noise features using label permutation methodDrop High Correlation Features- Remove high correlation features based on original gainExport Report- Generate Excel report containing details and statistics of all stepsCore FunctionsFunctionPurposeModuleget_dataset()Load and format datareferences.funcorg_analysis()Organization sample analysisreferences.funcmissing_check()Calculate missing ratereferences.funcdrop_abnormal_ym()Filter abnormal monthsreferences.analysisdrop_highmiss_features()Drop high missing rate featuresreferences.analysisdrop_lowiv_features()Drop low IV featuresreferences.analysisdrop_highpsi_features()Drop high PSI featuresreferences.analysisdrop_highnoise_features()Null Importance denoisingreferences.analysisdrop_highcorr_features()Drop high correlation featuresreferences.analysisiv_distribution_by_org()IV distribution statisticsreferences.analysispsi_distribution_by_org()PSI distribution statisticsreferences.analysisvalue_ratio_distribution_by_org()Value ratio distribution statisticsreferences.analysisexport_cleaning_report()Export cleaning reportreferences.analysisParameter DescriptionData Loading ParametersDATA_PATH: Data file path (best are parquet format)DATE_COL: Date column nameY_COL: Label column nameORG_COL: Organization column nameKEY_COLS: Primary key column name listOOS Organization ConfigurationOOS_ORGS: Out-of-sample organization listAbnormal Month Filtering Parametersmin_ym_bad_sample: Minimum bad sample count per month (default 10)min_ym_sample: Minimum total sample count per month (default 500)Missing Rate Parametersmissing_ratio: Overall missing rate threshold (default 0.6)IV Parametersoverall_iv_threshold: Overall IV threshold (default 0.1)org_iv_threshold: Single organization IV threshold (default 0.1)max_org_threshold: Maximum tolerated low IV organization count (default 2)PSI Parameterspsi_threshold: PSI threshold (default 0.1)max_months_ratio: Maximum unstable month ratio (default 1/3)max_orgs: Maximum unstable organization count (default 6)Null Importance Parametersn_estimators: Number of trees (default 100)max_depth: Maximum tree depth (default 5)gain_threshold: Gain difference threshold (default 50)High Correlation Parametersmax_corr: Correlation threshold (default 0.9)top_n_keep: Keep top N features by original gain ranking (default 20)Output ReportThe generated Excel report contains the following sheets:汇总- Summary information of all steps, including operation results and conditions机构样本统计- Sample count and bad sample rate for each organization分离OOS数据- OOS sample and modeling sample countsStep4-异常月份处理- Abnormal months that were removed缺失率明细- Overall and organization-level missing rates for each featureStep5-有值率分布统计- Distribution of features in different value ratio rangesStep6-高缺失率处理- High missing rate features that were removedStep7-IV明细- IV values of each feature in each organization and overallStep7-IV处理- Features that do not meet IV conditions and low IV organizationsStep7-IV分布统计- Distribution of features in different IV rangesStep8-PSI明细- PSI values of each feature in each organization each monthStep8-PSI处理- Features that do not meet PSI conditions and unstable organizationsStep8-PSI分布统计- Distribution of features in different PSI rangesStep9-null importance处理- Noise features that were removedStep10-高相关性剔除- High correlation features that were removedFeaturesInteractive Input: Parameters can be input before each step execution, with default values supportedIndependent Execution: Each step is executed independently without deleting original data, facilitating comparative analysisComplete Report: Generate complete Excel report containing details, statistics, and distributionsMulti-process Support: IV and PSI calculations support multi-process accelerationOrganization-level Analysis: Support organization-level statistics and modeling/OOS distinction