用 Instructor 与多模态 Gemini 从旅行视频中提取结构化推荐【免费下载链接】instructorstructured outputs for llms项目地址: https://gitcode.com/GitHub_Trending/in/instructor本篇文章演示如何在 instructor 框架中使用 Google Gemini 的多模态能力将一段旅行视频直接交给 Gemini配合 Pydantic 响应模型一次性提取出推荐了哪些目的地、每个地点在哪里、为什么值得去等结构化信息。读完本文你将掌握视频上传、多模态消息构造、嵌套响应模型定义与结果解析的完整流程并能进一步延伸出时间戳、说话人分离等进阶方案。背景为什么用多模态模型做结构化提取传统的信息抽取以文本为输入而旅行、纪录片、访谈类内容的大量信息恰恰藏在画面与语音里。Gemini 等原生多模态模型可以直接接收视频文件理解其中的口播、画面与字幕再由 instructor 将模型的自由输出收敛为 Pydantic 模型约束的结构化数据形成一条视频 → 结构化推荐清单的自动化流水线。本案例原始发布于 docs/blog/posts/multimodal-gemini.md以一个日本飞驒高山旅行视频为输入最终提取出包含飞驒和牛、宫川早市、白川乡等在内的完整目的地清单并被应用在影视从业者的多模态检索场景中。环境准备安装带有 Gemini 支持的 instructor 扩展包即可一条命令搞定pip install instructor[google-genai]该扩展包会拉取 Google 官方的google-genaiSDK。仓库中 docs/integrations/google.md 明确指出google/model前缀使用当前的google-genaiSDK推荐而vertexai/model与gemini/model为旧前缀应迁移到google/model。定义数据模型我们先用 Pydantic 定义两个模型单个目的地TouristDestination与整体推荐Recommendations。注意Recommendations中的chain_of_thought字段让模型先输出推理过程再给出结论能显著提升提取质量from pydantic import BaseModel class TouristDestination(BaseModel): name: str description: str location: str class Recommendations(BaseModel): chain_of_thought: str description: str destinations: list[TouristDestination]这里的嵌套结构list[TouristDestination]会被 instructor 自动转换为 Gemini 可理解的 JSON Schema从源码看模型 schema 正是通过response_model.model_json_schema()获取的见 instructor/v2/providers/gemini/utils.py。初始化 Gemini 客户端使用 instructor 的from_provider统一入口初始化客户端client instructor.from_provider(google/gemini-2.5-flash)from_provider会自动根据提供商与模型能力选择合适的模式。从源码实现看Gemini 支持两类模式见 instructor/v2/providers/gemini/handlers.pyinstructor.Mode.TOOLS走 Gemini 的工具调用function calling接口是默认模式instructor.Mode.JSON即源码中的MD_JSON走 JSON Schema 输出模式。两种模式都内置了重试reask机制当返回结果未通过 Pydantic 校验时会把错误信息回灌给模型让它修正——TOOLS 模式通过FunctionResponse携带校验错误见 handlers.pyJSON 模式则直接在后续轮次中追加根据错误修正 JSON的提示见 handlers.py。需要说明的是当前仓库将instructor/providers/gemini标记为兼容层instructor/providers/gemini/init.py实际实现统一收口到instructor/v2/providers/gemini因此无论走哪条路径行为一致。上传并处理视频Gemini 的视频输入需要先通过 Files API 上传上传后会返回一个文件对象可直接作为普通消息内容传入file genai.upload_file(./takayama.mp4)随后在消息中把文字指令与文件对象一起交给模型resp client.create( messages[ { role: user, content: [What places do they recommend in this video?, file], } ], response_modelRecommendations, ) print(resp)消息的content是一个列表可以自由混排普通文本与文件对象。这种列表式内容在 docs/examples/multi_modal_gemini.md 中也有印证列表可以同时容纳普通用户消息与genai.upload_file返回的文件对象非常灵活。对于视频等大文件务必使用upload_fileFiles API而不是内联字节。文档中记录了内联方式的上限报错400 Request payload size exceeds the limit: 20971520 bytes即约 20MB 的请求体上限超过后必须改用 Files API 上传见 docs/examples/multi_modal_gemini.md。解析原始输出视频分析完成后模型会返回完整的结构化结果。以下是本案例的原始输出未做任何手工整理可以看到chain_of_thought、description与destinations三个字段都被完整填充Recommendations( chain_of_thoughtThe video recommends visiting Takayama city, in the Hida Region, Gifu Prefecture. The video suggests visiting the Miyagawa Morning Market, to try the Sarubobo good luck charms, and to enjoy the cookie cup espresso, made by Koma Coffee. Then, the video suggests visiting a traditional Japanese Cafe, called Kissako Katsure, and try their matcha and sweets. Afterwards, the video suggests to visit the Sanmachi Historic District, where you can find local crafts and delicious foods. The video recommends trying Hida Wagyu beef, at the Kin no Kotte Ushi shop, or to have a sit-down meal at the Kitchen Hida. Finally, the video recommends visiting Shirakawa-go, a World Heritage Site in Gifu Prefecture., descriptionThis video recommends a number of places to visit in Takayama city, in the Hida Region, Gifu Prefecture. It shows some of the local street food and highlights some of the unique shops and restaurants in the area., destinations[ TouristDestination( nameTakayama, descriptionTakayama is a city at the base of the Japan Alps, located in the Hida Region of Gifu., locationHida Region, Gifu Prefecture ), TouristDestination( nameMiyagawa Morning Market, descriptionThe Miyagawa Morning Market, or the Miyagawa Asai-chi in Japanese, is a market that has existed officially since the Edo Period, more than 100 years ago. Its open every single day, rain or shine, from 7am to noon., locationHida Takayama ), TouristDestination( nameNakaya - Handmade Hida Sarubobo, descriptionThe Nakaya shop sells handcrafted Sarubobo good luck charms., locationHida Takayama ), TouristDestination( nameKoma Coffee, descriptionKoma Coffee is a shop that has been in business for about 50 or 60 years, and they serve coffee in a cookie cup. Theyve been serving coffee for about 10 years., locationHida Takayama ), TouristDestination( nameKissako Katsure, descriptionKissako Katsure is a traditional Japanese style cafe, called Kissako, and the name means would you like to have some tea. They have a variety of teas and sweets., locationHida Takayama ), TouristDestination( nameSanmachi Historic District, descriptionSanmachi Dori is a Historic Merchant District in Takayama, all of the buildings here have been preserved to look as they did in the Edo Period., locationHida Takayama ), TouristDestination( nameSuwa Orchard, descriptionThe Suwa Orchard has been in business for more than 50 years., locationHida Takayama ), TouristDestination( nameKitchen HIDA, descriptionKitchen HIDA is a restaurant with a 50 year history, known for their Hida Beef dishes and for using a lot of local ingredients., locationHida Takayama ), TouristDestination( nameKin no Kotte Ushi, descriptionKin no Kotte Ushi is a shop known for selling Beef Sushi, especially Hida Wagyu Beef Sushi. Their sushi is medium rare., locationHida Takayama ), TouristDestination( nameShirakawa-go, descriptionShirakawa-go is a World Heritage Site in Gifu Prefecture., locationGifu Prefecture ) ] )整理后视频中提到的 10 个推荐点如下高山Takayama核心目的地位于岐阜县飞驒地区、日本阿尔卑斯山脚下。宫川早市Miyagawa Morning Market自江户时代起延续百年以上的市集每天 7 点到正午营业。Nakaya 手工飞驒猴宝宝店售卖手工制作的 Sarubobo 护身符。Koma Coffee经营约 50–60 年的老店以饼干杯咖啡闻名。Kissako Katsure日式传统茶屋供应多种茶饮与和果子。三町古街Sanmachi Historic District完整保留江户时代风貌的商街。Suwa 果园经营超过 50 年的果园。Kitchen HIDA50 年历史餐厅主打飞驒牛肉与本地食材。Kin no Kotte Ushi以飞驒和牛寿司三分熟闻名的店铺。白川乡Shirakawa-go岐阜县境内的世界文化遗产。这些结果本身就是一份可入库、可检索的目的地数据表——这正是多模态输入 → 结构化输出的典型收益。局限与挑战当前方案展示了多模态 AI 分析视频的强大能力但也存在几个值得正视的问题缺乏时间信息目前只能提取整体推荐无法得知某个推荐出现在视频的哪个时刻难以把推荐与具体片段关联起来。缺少说话人分离模型不会区分视频中的不同说话人在多主持、多受访者的视频中无法定位谁在推荐什么。内容密度压力更长或更复杂的视频可能超出模型的注意力上限导致漏提或提取精度下降。未来探索方向针对上述局限可以从以下方向扩展系统能力时间戳提取让模型为每个推荐输出时间戳。将时间格式约束为Literal类型有助于后续解析在 docs/integrations/google.md 中也提到 Gemini 对 Union 类型支持有限用Literal替代是推荐做法class TimestampedRecommendation(BaseModel): timestamp: str timestamp_format: Literal[HH:MM, HH:MM:SS] # Helps with parsing recommendation: str class EnhancedRecommendations(BaseModel): destinations: list[TouristDestination] timestamped_mentions: list[TimestampedRecommendation]说话人分离接入说话人识别把推荐归因到具体个人对多主持或访谈类视频尤其有价值。分段分析把长视频切成小段分别分析再聚合去重从而保持精度、覆盖全部信息将视频拆分为更小的片段逐段独立分析汇总并去重各段结果。多语言支持扩展模型对不同语言的识别能力准确捕获各文化语境下的特色推荐。视觉元素分析即使音频中未提及也让模型识别并描述画面中出现的地标、食物或活动。情感分析引入情感判断评估说话人对某个推荐的热衷或保留态度。进阶阅读Multimodal Conceptsinstructor 统一的 Image / Audio / PDF 多模态接口支持from_url、from_gs_url、from_path、from_base64、autodetect等一致的加载方式以及PDFWithGenaiFile、PDFWithCacheControl等特化对象。Google Integration完整的 Gemini 接入指南涵盖generation_configtemperature、max_tokens、top_p、top_k、安全设置、流式输出与已知限制。OpenAI Multimodal对比不同提供商的多模态实现方式。Anthropic Structured Output了解替代提供商的结构化输出方案。Chat with PDFs using GeminiGemini 处理 PDF 文档的实操案例。【免费下载链接】instructorstructured outputs for llms项目地址: https://gitcode.com/GitHub_Trending/in/instructor创作声明：本文部分内容由AI辅助生成（AIGC），仅供参考
用 Instructor 与多模态 Gemini 从旅行视频中提取结构化推荐
用 Instructor 与多模态 Gemini 从旅行视频中提取结构化推荐【免费下载链接】instructorstructured outputs for llms项目地址: https://gitcode.com/GitHub_Trending/in/instructor本篇文章演示如何在 instructor 框架中使用 Google Gemini 的多模态能力将一段旅行视频直接交给 Gemini配合 Pydantic 响应模型一次性提取出推荐了哪些目的地、每个地点在哪里、为什么值得去等结构化信息。读完本文你将掌握视频上传、多模态消息构造、嵌套响应模型定义与结果解析的完整流程并能进一步延伸出时间戳、说话人分离等进阶方案。背景为什么用多模态模型做结构化提取传统的信息抽取以文本为输入而旅行、纪录片、访谈类内容的大量信息恰恰藏在画面与语音里。Gemini 等原生多模态模型可以直接接收视频文件理解其中的口播、画面与字幕再由 instructor 将模型的自由输出收敛为 Pydantic 模型约束的结构化数据形成一条视频 → 结构化推荐清单的自动化流水线。本案例原始发布于 docs/blog/posts/multimodal-gemini.md以一个日本飞驒高山旅行视频为输入最终提取出包含飞驒和牛、宫川早市、白川乡等在内的完整目的地清单并被应用在影视从业者的多模态检索场景中。环境准备安装带有 Gemini 支持的 instructor 扩展包即可一条命令搞定pip install instructor[google-genai]该扩展包会拉取 Google 官方的google-genaiSDK。仓库中 docs/integrations/google.md 明确指出google/model前缀使用当前的google-genaiSDK推荐而vertexai/model与gemini/model为旧前缀应迁移到google/model。定义数据模型我们先用 Pydantic 定义两个模型单个目的地TouristDestination与整体推荐Recommendations。注意Recommendations中的chain_of_thought字段让模型先输出推理过程再给出结论能显著提升提取质量from pydantic import BaseModel class TouristDestination(BaseModel): name: str description: str location: str class Recommendations(BaseModel): chain_of_thought: str description: str destinations: list[TouristDestination]这里的嵌套结构list[TouristDestination]会被 instructor 自动转换为 Gemini 可理解的 JSON Schema从源码看模型 schema 正是通过response_model.model_json_schema()获取的见 instructor/v2/providers/gemini/utils.py。初始化 Gemini 客户端使用 instructor 的from_provider统一入口初始化客户端client instructor.from_provider(google/gemini-2.5-flash)from_provider会自动根据提供商与模型能力选择合适的模式。从源码实现看Gemini 支持两类模式见 instructor/v2/providers/gemini/handlers.pyinstructor.Mode.TOOLS走 Gemini 的工具调用function calling接口是默认模式instructor.Mode.JSON即源码中的MD_JSON走 JSON Schema 输出模式。两种模式都内置了重试reask机制当返回结果未通过 Pydantic 校验时会把错误信息回灌给模型让它修正——TOOLS 模式通过FunctionResponse携带校验错误见 handlers.pyJSON 模式则直接在后续轮次中追加根据错误修正 JSON的提示见 handlers.py。需要说明的是当前仓库将instructor/providers/gemini标记为兼容层instructor/providers/gemini/init.py实际实现统一收口到instructor/v2/providers/gemini因此无论走哪条路径行为一致。上传并处理视频Gemini 的视频输入需要先通过 Files API 上传上传后会返回一个文件对象可直接作为普通消息内容传入file genai.upload_file(./takayama.mp4)随后在消息中把文字指令与文件对象一起交给模型resp client.create( messages[ { role: user, content: [What places do they recommend in this video?, file], } ], response_modelRecommendations, ) print(resp)消息的content是一个列表可以自由混排普通文本与文件对象。这种列表式内容在 docs/examples/multi_modal_gemini.md 中也有印证列表可以同时容纳普通用户消息与genai.upload_file返回的文件对象非常灵活。对于视频等大文件务必使用upload_fileFiles API而不是内联字节。文档中记录了内联方式的上限报错400 Request payload size exceeds the limit: 20971520 bytes即约 20MB 的请求体上限超过后必须改用 Files API 上传见 docs/examples/multi_modal_gemini.md。解析原始输出视频分析完成后模型会返回完整的结构化结果。以下是本案例的原始输出未做任何手工整理可以看到chain_of_thought、description与destinations三个字段都被完整填充Recommendations( chain_of_thoughtThe video recommends visiting Takayama city, in the Hida Region, Gifu Prefecture. The video suggests visiting the Miyagawa Morning Market, to try the Sarubobo good luck charms, and to enjoy the cookie cup espresso, made by Koma Coffee. Then, the video suggests visiting a traditional Japanese Cafe, called Kissako Katsure, and try their matcha and sweets. Afterwards, the video suggests to visit the Sanmachi Historic District, where you can find local crafts and delicious foods. The video recommends trying Hida Wagyu beef, at the Kin no Kotte Ushi shop, or to have a sit-down meal at the Kitchen Hida. Finally, the video recommends visiting Shirakawa-go, a World Heritage Site in Gifu Prefecture., descriptionThis video recommends a number of places to visit in Takayama city, in the Hida Region, Gifu Prefecture. It shows some of the local street food and highlights some of the unique shops and restaurants in the area., destinations[ TouristDestination( nameTakayama, descriptionTakayama is a city at the base of the Japan Alps, located in the Hida Region of Gifu., locationHida Region, Gifu Prefecture ), TouristDestination( nameMiyagawa Morning Market, descriptionThe Miyagawa Morning Market, or the Miyagawa Asai-chi in Japanese, is a market that has existed officially since the Edo Period, more than 100 years ago. Its open every single day, rain or shine, from 7am to noon., locationHida Takayama ), TouristDestination( nameNakaya - Handmade Hida Sarubobo, descriptionThe Nakaya shop sells handcrafted Sarubobo good luck charms., locationHida Takayama ), TouristDestination( nameKoma Coffee, descriptionKoma Coffee is a shop that has been in business for about 50 or 60 years, and they serve coffee in a cookie cup. Theyve been serving coffee for about 10 years., locationHida Takayama ), TouristDestination( nameKissako Katsure, descriptionKissako Katsure is a traditional Japanese style cafe, called Kissako, and the name means would you like to have some tea. They have a variety of teas and sweets., locationHida Takayama ), TouristDestination( nameSanmachi Historic District, descriptionSanmachi Dori is a Historic Merchant District in Takayama, all of the buildings here have been preserved to look as they did in the Edo Period., locationHida Takayama ), TouristDestination( nameSuwa Orchard, descriptionThe Suwa Orchard has been in business for more than 50 years., locationHida Takayama ), TouristDestination( nameKitchen HIDA, descriptionKitchen HIDA is a restaurant with a 50 year history, known for their Hida Beef dishes and for using a lot of local ingredients., locationHida Takayama ), TouristDestination( nameKin no Kotte Ushi, descriptionKin no Kotte Ushi is a shop known for selling Beef Sushi, especially Hida Wagyu Beef Sushi. Their sushi is medium rare., locationHida Takayama ), TouristDestination( nameShirakawa-go, descriptionShirakawa-go is a World Heritage Site in Gifu Prefecture., locationGifu Prefecture ) ] )整理后视频中提到的 10 个推荐点如下高山Takayama核心目的地位于岐阜县飞驒地区、日本阿尔卑斯山脚下。宫川早市Miyagawa Morning Market自江户时代起延续百年以上的市集每天 7 点到正午营业。Nakaya 手工飞驒猴宝宝店售卖手工制作的 Sarubobo 护身符。Koma Coffee经营约 50–60 年的老店以饼干杯咖啡闻名。Kissako Katsure日式传统茶屋供应多种茶饮与和果子。三町古街Sanmachi Historic District完整保留江户时代风貌的商街。Suwa 果园经营超过 50 年的果园。Kitchen HIDA50 年历史餐厅主打飞驒牛肉与本地食材。Kin no Kotte Ushi以飞驒和牛寿司三分熟闻名的店铺。白川乡Shirakawa-go岐阜县境内的世界文化遗产。这些结果本身就是一份可入库、可检索的目的地数据表——这正是多模态输入 → 结构化输出的典型收益。局限与挑战当前方案展示了多模态 AI 分析视频的强大能力但也存在几个值得正视的问题缺乏时间信息目前只能提取整体推荐无法得知某个推荐出现在视频的哪个时刻难以把推荐与具体片段关联起来。缺少说话人分离模型不会区分视频中的不同说话人在多主持、多受访者的视频中无法定位谁在推荐什么。内容密度压力更长或更复杂的视频可能超出模型的注意力上限导致漏提或提取精度下降。未来探索方向针对上述局限可以从以下方向扩展系统能力时间戳提取让模型为每个推荐输出时间戳。将时间格式约束为Literal类型有助于后续解析在 docs/integrations/google.md 中也提到 Gemini 对 Union 类型支持有限用Literal替代是推荐做法class TimestampedRecommendation(BaseModel): timestamp: str timestamp_format: Literal[HH:MM, HH:MM:SS] # Helps with parsing recommendation: str class EnhancedRecommendations(BaseModel): destinations: list[TouristDestination] timestamped_mentions: list[TimestampedRecommendation]说话人分离接入说话人识别把推荐归因到具体个人对多主持或访谈类视频尤其有价值。分段分析把长视频切成小段分别分析再聚合去重从而保持精度、覆盖全部信息将视频拆分为更小的片段逐段独立分析汇总并去重各段结果。多语言支持扩展模型对不同语言的识别能力准确捕获各文化语境下的特色推荐。视觉元素分析即使音频中未提及也让模型识别并描述画面中出现的地标、食物或活动。情感分析引入情感判断评估说话人对某个推荐的热衷或保留态度。进阶阅读Multimodal Conceptsinstructor 统一的 Image / Audio / PDF 多模态接口支持from_url、from_gs_url、from_path、from_base64、autodetect等一致的加载方式以及PDFWithGenaiFile、PDFWithCacheControl等特化对象。Google Integration完整的 Gemini 接入指南涵盖generation_configtemperature、max_tokens、top_p、top_k、安全设置、流式输出与已知限制。OpenAI Multimodal对比不同提供商的多模态实现方式。Anthropic Structured Output了解替代提供商的结构化输出方案。Chat with PDFs using GeminiGemini 处理 PDF 文档的实操案例。【免费下载链接】instructorstructured outputs for llms项目地址: https://gitcode.com/GitHub_Trending/in/instructor创作声明：本文部分内容由AI辅助生成（AIGC），仅供参考