Nemotron-CLIMB FastText Classifiers API完全指南：参数说明与调用示例 [特殊字符]

Nemotron-CLIMB FastText Classifiers API完全指南参数说明与调用示例 【免费下载链接】nemotron-climb-fasttext-classifiers项目地址: https://ai.gitcode.com/hf_mirrors/nvidia/nemotron-climb-fasttext-classifiersNemotron-CLIMB FastText Classifiers是NVIDIA开发的五个轻量级CPU文本分类器专门用于大规模语言模型训练数据质量控制。这些fastText分类器通过知识蒸馏技术从Nemotron-4-340B-Instruct模型生成标签能够高效评估网页文档在五个质量维度上的得分为LLM训练数据筛选提供自动化解决方案。 核心功能与模型概述Nemotron-CLIMB FastText Classifiers包含五个独立的二进制分类模型每个模型专注于一个特定的质量维度文本质量分类器(best_model_quality.bin) - 评估文本的整体质量广告内容分类器(best_model_advertisement.bin) - 检测文档中的广告和促销内容信息价值分类器(best_model_informational_value.bin) - 衡量文档的信息丰富度文化价值分类器(best_model_cultural_value.bin) - 评估文档的文化意义教育价值分类器(best_model_educational_value.bin) - 判断文档的教育价值每个模型文件大小约为6.8GB采用fastText监督分类框架基于300维词嵌入和n-gram特征表示。 快速安装与配置环境要求Python 3.7fastText库标准x86_64或ARM CPU硬件无需GPU安装步骤# 克隆仓库 git clone https://gitcode.com/hf_mirrors/nvidia/nemotron-climb-fasttext-classifiers # 安装fastText pip install fasttext下载模型文件项目包含五个预训练模型文件best_model_quality.binbest_model_advertisement.binbest_model_informational_value.binbest_model_cultural_value.binbest_model_educational_value.bin API参数详解基础参数说明输入参数text(字符串): 待分类的文本内容UTF-8编码k(整数, 可选): 返回的top-k预测结果数量默认1threshold(浮点数, 可选): 置信度阈值默认0.0输出格式每个分类器输出0-5的Likert量表评分对应标签格式__label__0- 最低质量__label__1- 低质量__label__2- 中等质量__label__3- 良好质量__label__4- 高质量__label__5- 最高质量模型初始化参数import fasttext # 加载质量分类器 quality_model fasttext.load_model(best_model_quality.bin) # 加载广告内容分类器 advertisement_model fasttext.load_model(best_model_advertisement.bin) # 加载信息价值分类器 informational_model fasttext.load_model(best_model_informational_value.bin) # 加载文化价值分类器 cultural_model fasttext.load_model(best_model_cultural_value.bin) # 加载教育价值分类器 educational_model fasttext.load_model(best_model_educational_value.bin) 完整调用示例示例1单文本分类import fasttext # 初始化所有分类器 models { quality: fasttext.load_model(best_model_quality.bin), advertisement: fasttext.load_model(best_model_advertisement.bin), informational: fasttext.load_model(best_model_informational_value.bin), cultural: fasttext.load_model(best_model_cultural_value.bin), educational: fasttext.load_model(best_model_educational_value.bin) } # 待分类文本 sample_text Artificial intelligence is transforming various industries by enabling automated decision-making and data analysis. This technology has the potential to revolutionize healthcare, finance, and education sectors. # 执行分类 for dimension, model in models.items(): predictions model.predict(sample_text, k3) labels predictions[0] probabilities predictions[1] print(f{dimension.capitalize()} Score:) for label, prob in zip(labels, probabilities): score int(label.replace(__label__, )) print(f Score {score}: {prob:.4f} confidence) print()示例2批量文档处理import fasttext from typing import List, Dict class NemotronClassifierPipeline: def __init__(self, model_dir: str .): 初始化分类器流水线 self.models { quality: fasttext.load_model(f{model_dir}/best_model_quality.bin), advertisement: fasttext.load_model(f{model_dir}/best_model_advertisement.bin), informational: fasttext.load_model(f{model_dir}/best_model_informational_value.bin), cultural: fasttext.load_model(f{model_dir}/best_model_cultural_value.bin), educational: fasttext.load_model(f{model_dir}/best_model_educational_value.bin) } def classify_document(self, text: str) - Dict[str, Dict]: 对单个文档进行全方位分类 results {} for dimension, model in self.models.items(): predictions model.predict(text, k1) label predictions[0][0] confidence predictions[1][0] score int(label.replace(__label__, )) results[dimension] { score: score, confidence: float(confidence), label: label } return results def batch_classify(self, documents: List[str]) - List[Dict[str, Dict]]: 批量分类文档 return [self.classify_document(doc) for doc in documents] # 使用示例 pipeline NemotronClassifierPipeline() documents [ This is a technical article about machine learning algorithms., Buy now! Limited time offer! Get 50% discount!, The history of ancient civilizations provides valuable cultural insights. ] results pipeline.batch_classify(documents) for i, (doc, result) in enumerate(zip(documents, results)): print(fDocument {i1}:) print(f Preview: {doc[:50]}...) print(f Quality Score: {result[quality][score]} (confidence: {result[quality][confidence]:.3f})) print(f Advertisement Score: {result[advertisement][score]}) print(f Informational Score: {result[informational][score]}) print(f Cultural Score: {result[cultural][score]}) print(f Educational Score: {result[educational][score]}) print()示例3数据筛选流水线import fasttext import pandas as pd from tqdm import tqdm class DataQualityFilter: def __init__(self, threshold_config: Dict[str, int] None): 初始化数据质量过滤器 if threshold_config is None: threshold_config { quality: 3, # 质量分数阈值 advertisement: 2, # 广告分数阈值越低越好 informational: 3, # 信息价值阈值 cultural: 2, # 文化价值阈值 educational: 2 # 教育价值阈值 } self.thresholds threshold_config self.models { quality: fasttext.load_model(best_model_quality.bin), advertisement: fasttext.load_model(best_model_advertisement.bin), informational: fasttext.load_model(best_model_informational_value.bin), cultural: fasttext.load_model(best_model_cultural_value.bin), educational: fasttext.load_model(best_model_educational_value.bin) } def filter_document(self, text: str) - bool: 判断文档是否通过所有质量检查 scores {} for dimension, model in self.models.items(): predictions model.predict(text, k1) label predictions[0][0] score int(label.replace(__label__, )) scores[dimension] score # 应用过滤规则 if scores[quality] self.thresholds[quality]: return False if scores[advertisement] self.thresholds[advertisement]: return False if scores[informational] self.thresholds[informational]: return False if scores[cultural] self.thresholds[cultural]: return False if scores[educational] self.thresholds[educational]: return False return True def process_dataset(self, texts: List[str]) - pd.DataFrame: 处理整个数据集并返回详细结果 results [] for text in tqdm(texts, descProcessing documents): scores {} for dimension, model in self.models.items(): predictions model.predict(text, k1) label predictions[0][0] score int(label.replace(__label__, )) scores[f{dimension}_score] score scores[passed_filter] self.filter_document(text) scores[text_preview] text[:100] ... if len(text) 100 else text results.append(scores) return pd.DataFrame(results) # 使用示例 filter_pipeline DataQualityFilter() # 假设有一个文档列表 documents [...] # 你的文档列表 # 处理并过滤 results_df filter_pipeline.process_dataset(documents) # 查看过滤结果 print(fTotal documents: {len(documents)}) print(fPassed filter: {results_df[passed_filter].sum()}) print(fFilter rate: {results_df[passed_filter].mean():.2%}) # 保存结果 results_df.to_csv(filtered_results.csv, indexFalse)⚙️ 高级配置与优化性能优化技巧批量处理一次性处理多个文档以提高效率内存管理每个模型约6.8GB确保有足够内存多进程处理对于大规模数据集使用Python多进程from concurrent.futures import ProcessPoolExecutor import fasttext def classify_batch(texts: List[str], model_path: str) - List[int]: 批量分类函数适合多进程 model fasttext.load_model(model_path) scores [] for text in texts: predictions model.predict(text, k1) label predictions[0][0] score int(label.replace(__label__, )) scores.append(score) return scores # 并行处理示例 def parallel_classification(documents: List[str], num_workers: int 4): 并行分类文档 chunk_size len(documents) // num_workers chunks [documents[i:ichunk_size] for i in range(0, len(documents), chunk_size)] with ProcessPoolExecutor(max_workersnum_workers) as executor: futures [] for chunk in chunks: future executor.submit(classify_batch, chunk, best_model_quality.bin) futures.append(future) results [] for future in futures: results.extend(future.result()) return results自定义阈值配置class CustomClassifierConfig: def __init__(self): self.config { quality: { model_path: best_model_quality.bin, threshold: 3, weight: 1.0 }, advertisement: { model_path: best_model_advertisement.bin, threshold: 2, # 广告分数越低越好 weight: 0.8 }, informational: { model_path: best_model_informational_value.bin, threshold: 3, weight: 0.9 }, cultural: { model_path: best_model_cultural_value.bin, threshold: 2, weight: 0.7 }, educational: { model_path: best_model_educational_value.bin, threshold: 2, weight: 0.8 } } def get_weighted_score(self, text: str) - float: 计算加权综合分数 total_score 0 total_weight 0 for dimension, config in self.config.items(): model fasttext.load_model(config[model_path]) predictions model.predict(text, k1) label predictions[0][0] raw_score int(label.replace(__label__, )) # 标准化到0-1范围 normalized_score raw_score / 5.0 weighted_score normalized_score * config[weight] total_score weighted_score total_weight config[weight] return total_score / total_weight if total_weight 0 else 0 应用场景与最佳实践场景1LLM训练数据预处理def preprocess_training_data(raw_documents: List[str], min_quality: int 3, max_advertisement: int 2) - List[str]: 预处理LLM训练数据 filter_pipeline DataQualityFilter({ quality: min_quality, advertisement: max_advertisement, informational: 2, cultural: 1, educational: 1 }) filtered_docs [] for doc in tqdm(raw_documents): if filter_pipeline.filter_document(doc): filtered_docs.append(doc) return filtered_docs场景2内容质量监控class ContentQualityMonitor: def __init__(self): self.models { quality: fasttext.load_model(best_model_quality.bin), advertisement: fasttext.load_model(best_model_advertisement.bin) } def monitor_stream(self, text_stream, window_size: int 100): 实时监控文本流质量 quality_scores [] ad_scores [] for text in text_stream: # 质量评分 q_pred self.models[quality].predict(text, k1) q_score int(q_pred[0][0].replace(__label__, )) quality_scores.append(q_score) # 广告评分 ad_pred self.models[advertisement].predict(text, k1) ad_score int(ad_pred[0][0].replace(__label__, )) ad_scores.append(ad_score) # 滑动窗口分析 if len(quality_scores) window_size: quality_scores.pop(0) ad_scores.pop(0) avg_quality sum(quality_scores) / len(quality_scores) avg_ad sum(ad_scores) / len(ad_scores) yield { current_quality: q_score, current_ad_score: ad_score, avg_quality: avg_quality, avg_ad_score: avg_ad, alert: q_score 2 or ad_score 3 } 性能指标与评估模型性能特征推理速度CPU上每秒处理数千个文档内存占用每个模型约6.8GB准确率基于100,000个测试文档评估F1分数各维度分类性能优异评估示例def evaluate_classifier_performance(test_data: List[Tuple[str, int]], model_path: str) - Dict[str, float]: 评估分类器性能 model fasttext.load_model(model_path) predictions [] true_labels [] for text, true_label in test_data: pred model.predict(text, k1) pred_label int(pred[0][0].replace(__label__, )) predictions.append(pred_label) true_labels.append(true_label) # 计算指标 from sklearn.metrics import accuracy_score, precision_score, recall_score, f1_score return { accuracy: accuracy_score(true_labels, predictions), precision: precision_score(true_labels, predictions, averageweighted), recall: recall_score(true_labels, predictions, averageweighted), f1: f1_score(true_labels, predictions, averageweighted) } 故障排除与常见问题常见问题解决内存不足错误# 解决方案分批处理 def process_in_batches(documents: List[str], batch_size: int 1000): for i in range(0, len(documents), batch_size): batch documents[i:ibatch_size] # 处理批次 yield process_batch(batch)模型加载失败# 检查文件路径和权限 import os model_path best_model_quality.bin if not os.path.exists(model_path): raise FileNotFoundError(fModel file not found: {model_path}) if os.path.getsize(model_path) 6_800_000_000: # 约6.8GB print(Warning: Model file size seems smaller than expected)文本编码问题# 确保UTF-8编码 def ensure_utf8(text: str) - str: if isinstance(text, bytes): return text.decode(utf-8, errorsignore) return text 总结与下一步Nemotron-CLIMB FastText Classifiers为大规模语言模型训练数据质量控制提供了高效的解决方案。通过五个专业分类器的组合使用您可以自动化数据筛选快速过滤低质量训练数据多维质量评估从五个维度全面评估文档质量高性能处理CPU上实现高速批量处理灵活配置根据需求调整过滤阈值下一步建议尝试不同的阈值组合找到适合您数据集的配置结合其他预处理步骤去重、标准化等监控分类器在不同领域文本上的表现考虑微调模型以适应特定领域需求通过合理使用这些分类器您可以显著提升LLM训练数据的质量从而获得更好的模型性能。【免费下载链接】nemotron-climb-fasttext-classifiers项目地址: https://ai.gitcode.com/hf_mirrors/nvidia/nemotron-climb-fasttext-classifiers创作声明：本文部分内容由AI辅助生成（AIGC），仅供参考