Sapiens2 视觉骨干与人体任务模型在 Transformers 中的完整使用指南【免费下载链接】transformers Transformers: the model-definition framework for state-of-the-art machine learning models in text, vision, audio, and multimodal models, for both inference and training.项目地址: https://gitcode.com/GitHub_Trending/tra/transformers导读Sapiens2 是 Meta 提出的一族以人为中心human-centric的高分辨率视觉 Transformer面向姿态估计、人体部位分割、表面法线估计、点图pointmap估计等密集预测任务。本文基于 sapiens2 官方文档并结合仓库内的 configuration_sapiens2.py、modeling_sapiens2.py、image_processing_sapiens2.py 三个核心源文件逐项讲解模型架构设计、AutoModel/AutoBackbone/各任务头Pose/Seg/Normal/Pointmap/Matting的推理代码、后处理管线以及配置参数语义。读完本文你将能独立加载官方 checkpoint、完成七类典型调用并理解其后处理与损失计算原理。模型概览Sapiens2 是什么Sapiens2 模型出自论文Sapiens2Rawal Khirodkar, He Wen, Julieta Martinez, Yuan Dong, Zhaoen Su, Shunsuke Saito。它是在约 10 亿张经过人工筛选标注的高质量人体图像上预训练的人体视觉模型家族将掩码图像重建与自蒸馏对比目标结合同时学习低层与语义特征。按论文摘要其参数规模从 0.4B 覆盖到 5B以原生 1K 分辨率训练并提供用于扩展空间推理的分层 4K 变体相较前代在姿态估计4 mAP、身体部位分割24.3 mIoU、法线估计误差降低 45.6%等指标上均有显著提升并扩展到 pointmap 与 albedo 估计等新任务。需要强调的是上述改进数值均来自论文摘要表述属于模型自证的结果在仓库证据层面你可以直接验证的是模型于 2026-06-03 被贡献进 Transformers模型文档 记录了 HF papers 发布于 2026-04-23官方 checkpoint 均以facebook/sapiens2-*命名存放于 Hub本仓库的sapiens2模型代码位于 src/transformers/models/sapiens2由其modular_sapiens2.py模块化源生成modeling_sapiens2.py、configuration_sapiens2.py、image_processing_sapiens2.py三个落地文件。使用要点TipsSapiens2 使用旋转位置编码RoPE支持任意输入分辨率默认图像处理器会把图像缩放到 1024×768高×宽。注意力采用分组查询注意力GQA用于中间层首尾各 8 层使用完整多头注意力。默认 8 个register tokens可降低 patch token 上的高范数伪影产生更干净的注意力图利于密集预测任务。架构设计从源码读懂四个关键设计理解 Sapiens2Config 中的默认值与 modeling_sapiens2.py 的实现是后续调参与二次开发的前提。1. 输入嵌入CLS Register 卷积 PatchSapiens2Embeddings 将输入序列构造为[CLS] register_tokens patch_embeddingscls_token作为全局表征register_tokens默认 8 个作为可学习寄存器承载冗余信息patch 投影是一个 kernel 与 stride 均等于patch_size默认 16的 Conv2d。此外当config.use_mask_tokenTrue时还会加入 mask token配合bool_masked_pos供掩码图像建模MIM预训练使用——注意这是预训练专用开关绝大多数发布的 checkpoint 没有该权重模型在加载时会忽略缺失的mask_token。2. 二维连续坐标 RoPE动态适配任意分辨率与固定绝对位置编码不同Sapiens2RopePositionEmbedding 先按输入的实际分辨率动态计算每个 patch 中心的二维归一化坐标[-1, 1]见get_patches_center_coordinates结果带 lru_cache再结合inv_freq基频由rope_theta控制生成 cos/sin。训练期还支持坐标增强pos_embed_shift/pos_embed_jitter/pos_embed_rescale其中pos_embed_rescale默认 2.0这正是模型训练时带随机尺度、推理时可处理任意分辨率的机制来源。RoPE 只施加到 patch token 上CLS 与 register 前缀 token 不参与旋转见apply_rotary_pos_emb。3. 每层可配置的注意力头数GQA ↔ MHASapiens2Config通过num_key_value_heads_per_layer精细控制每一层 KV 头数等于num_attention_heads即完整多头注意力更小值即分组查询注意力。若未显式给出__post_init__会自动填充——首num_first_full_attention_layers默认 8与末num_last_full_attention_layers默认 8层使用全部注意力头其余层使用num_key_value_attention_heads默认 8。use_qk_normTrue时还会对 Q/K 施加 RMSNormeps 默认 1e-6再进入 RoPE。模型类通过_supports_sdpa / _supports_flash_attn / _supports_flex_attn声明同时支持 SDPA、Flash Attention 与 Flex Attention 后端可用attn_implementation参数按硬件选择。此外 LayerScalelayerscale_value与 SiLU 门控 MLPuse_gated_mlpTruehidden act 为 silu共同构成了每个 Transformer 层。4. 统一解码头Sapiens2Head与子配置Sapiens2HeadConfig所有密集预测任务都复用 Sapiens2Head 作为上采样解码器其结构参数收敛在 Sapiens2HeadConfig 中通过head_config子配置注入sub_configs {head_config: Sapiens2HeadConfig}。主要包括三组upsample_*逐级上采样块的输出通道数首个块输入为hidden_sizekernel 默认 4也可改用 pixel-shuffle 上采样use_pixel_shuffleconv_*上采样后的精修卷积层kernel 默认 1scale_*用于 pointmap 的焦距尺度分支stride-2 卷积 MLP 预测标量其 MLP 输入维度scale_final_input_size会在配置初始化时依据image_size与patch_size自动推算_init_scale_final_input_size。任务头与输出结构一览任务模型类输出字段说明Sapiens2ForPoseEstimationloss/heatmaps308 个关键点高斯热图支持flip_pairs与可见性加权 MSESapiens2ForSemanticSegmentationloss/logits29 类人体部位分割 logits交叉熵 semantic_loss_ignore_index(255)Sapiens2ForNormalEstimationloss/normals原始未归一化 XYZ 法线图Sapiens2ForPointmapEstimationloss/pointmaps/scales相机空间逐像素 XYZ 坐标与焦距比例Sapiens2ForImageMattingloss/alphas/foregroundssigmoid 激活的 alpha 与前/背景分离结果Sapiens2Backbonefeature_maps/cls_tokens多阶段空间特征 各阶段 CLS自动映射关系可在 modeling_auto.py 中确认sapiens2同时注册了AutoModelForSemanticSegmentation / AutoModelForImageMatting / AutoModelForNormalEstimation / AutoModelForPointmapEstimation / AutoModelForPoseEstimation等入口。快速上手checkpoint 与基础环境所有示例均以官方 0.4B checkpoint 为例命名规律为facebook/sapiens2-{pretrain|seg|pose|normal|pointmap|matting}-{0.4b|1b|...}。模型与图像处理器通过Auto*接口加载device_mapauto便于异构设备推理。代码中的load_image同时支持本地图片路径与网络图片 URL 输入本地路径建议使用绝对路径例如load_image(/path/to/your_image.jpg)。若运行姿态估计相关后处理还需安装opencv-pythonpip install opencv-python。1. AutoModel获取整图嵌入CLS tokenimport torch from transformers import AutoImageProcessor, AutoModel from transformers.image_utils import load_image image load_image(http://images.cocodataset.org/val2017/000000004016.jpg) # 也支持本地路径 image_processor AutoImageProcessor.from_pretrained(facebook/sapiens2-pretrain-0.4b) model AutoModel.from_pretrained(facebook/sapiens2-pretrain-0.4b, device_mapauto) inputs image_processor(imagesimage, return_tensorspt).to(model.device) with torch.inference_mode(): outputs model(**inputs) # outputs.pooler_output is the CLS token (whole-image embedding) cls_token outputs.pooler_output print(CLS token shape:, cls_token.shape) # [1, 1024]从 Sapiens2Model.forward 的实现看pooler_output即最终sequence_output[:, 0, :]——取经过层归一化的序列中第一个位置CLS token维度等于hidden_size(1024)。2. AutoBackbone直接取空间特征图Sapiens2Backbone 会把 patch token 重新整形回空间维度并将 CLS token 直接挂在输出对象上import torch from transformers import AutoBackbone, AutoImageProcessor from transformers.image_utils import load_image image load_image(http://images.cocodataset.org/val2017/000000004016.jpg) image_processor AutoImageProcessor.from_pretrained(facebook/sapiens2-pretrain-0.4b) model AutoBackbone.from_pretrained(facebook/sapiens2-pretrain-0.4b, device_mapauto) inputs image_processor(imagesimage, return_tensorspt).to(model.device) with torch.inference_mode(): outputs model(**inputs, return_class_tokenTrue) # Patch tokens shaped (batch, height, width, channels) patch_features outputs.feature_maps[0] cls_token outputs.cls_tokens[0] print(CLS token shape:, cls_token.shape) # [1, 1024] print(Patch features shape:, patch_features.shape) # [1, 64, 48, 1024]代码层面Backbone 会去掉每阶段序列中的前缀 tokennum_prefix 1 num_register_tokens后按num_patches_h × num_patches_w重塑并转置到(B, C, H, W)布局。是否整形由reshape_hidden_states默认 True控制输出前还可按normalize_backbone_outputs默认 True对特征施加 RMSNormstage_names [stem, stage1, ..., stageN]可用于选择输出阶段。3. 表面法线估计Normal Estimationimport torch from transformers import AutoImageProcessor, AutoModelForNormalEstimation from transformers.image_utils import load_image image load_image(http://images.cocodataset.org/val2017/000000004016.jpg) image_processor AutoImageProcessor.from_pretrained(facebook/sapiens2-normal-0.4b) model AutoModelForNormalEstimation.from_pretrained(facebook/sapiens2-normal-0.4b, device_mapauto) inputs image_processor(image, return_tensorspt).to(model.device) with torch.inference_mode(): outputs model(**inputs) # outputs.normals shape: (batch_size, 3, height, width) — raw, unnormalized XYZ normals print(Normals shape:, outputs.normals.shape) # [1, 3, 1024, 768] # Remove preprocessing padding, resize to original size, and L2-normalize to unit vectors in [-1, 1] original_size (image.height, image.width) result image_processor.post_process_normal_estimation( outputs, source_sizes[original_size], target_sizes[original_size] ) normals result[0][normals] print(Normals shape:, normals.shape) # [3, original_height, original_width]可视化片段法线值域[-1,1]→ RGB[0,255]并用分割结果去背景# Convert L2-normalized normals in [-1, 1] to RGB in [0, 255] normals_rgb ((normals 1.0) / 2.0 * 255.0).clamp(0, 255).to(torch.uint8) # Apply background removal using the segmentation model output. # segmentation is the output of post_process_semantic_segmentation — a (H, W) tensor # of per-pixel class IDs, where class 0 is background. background_mask segmentation 0 normals_rgb[:, background_mask] 0 print(Normals RGB shape:, normals_rgb.shape) # [3, original_height, original_width]模型输出的normals是未经归一化的原始 XYZ 向量归一化在训练中以监督信号形式存在因此文档示例通过post_process_normal_estimation统一完成去 padding、缩放回原尺寸、L2 归一化到单位向量三步target_sizes传入原图尺寸即可。4. 点图估计Pointmap Estimationimport torch from transformers import AutoImageProcessor, AutoModelForPointmapEstimation from transformers.image_utils import load_image image load_image(http://images.cocodataset.org/val2017/000000004016.jpg) image_processor AutoImageProcessor.from_pretrained(facebook/sapiens2-pointmap-0.4b) model AutoModelForPointmapEstimation.from_pretrained(facebook/sapiens2-pointmap-0.4b, device_mapauto) inputs image_processor(image, return_tensorspt).to(model.device) with torch.inference_mode(): outputs model(**inputs) # outputs.pointmaps shape: (batch_size, 3, height, width) — raw XYZ in canonical camera space print(Pointmaps shape:, outputs.pointmaps.shape) # [1, 3, 1024, 768] # Remove preprocessing padding, resize to original size, and apply focal-length scale original_size (image.height, image.width) result image_processor.post_process_pointmap_estimation( outputs, source_sizes[original_size], target_sizes[original_size] ) pointmap result[0][pointmap] print(Pointmap shape:, pointmap.shape) # [3, original_height, original_width]可视化片段用逆深度 turbo 色带渲染 pointmapimport matplotlib.pyplot as plt # segmentation is the output of post_process_semantic_segmentation — a (H, W) tensor # of per-pixel class IDs, where class 0 is background. foreground_mask segmentation ! 0 depth pointmap[2] # Z channel: depth in camera space, shape (H, W) pointmap_rgb torch.zeros(3, *depth.shape, dtypetorch.uint8) foreground_depth depth[foreground_mask] if foreground_depth.numel() 0: depth_low, depth_high torch.quantile(foreground_depth, torch.tensor([0.01, 0.99])) inverse_depth 1.0 / foreground_depth.clamp(min1e-6) inverse_depth_low 1.0 / depth_high.clamp(min1e-6) inverse_depth_high 1.0 / depth_low.clamp(min1e-6) inverse_depth_normalized ((inverse_depth - inverse_depth_low) / (inverse_depth_high - inverse_depth_low 1e-8)).clamp(0, 1) turbo plt.get_cmap(turbo) foreground_colors torch.from_numpy(turbo(inverse_depth_normalized.cpu().numpy())[..., :3] * 255).to(torch.uint8) # (N, 3) pointmap_rgb[:, foreground_mask] foreground_colors.T print(Pointmap RGB shape:, pointmap_rgb.shape) # [3, original_height, original_width]pointmap 输出包含人体在规范相机空间中的逐像素三维坐标。模型配置为 pointmap 任务额外构造了Sapiens2PointmapScaleHead对应head_config中的scale_*配置组以回归规范焦距/真实焦距比例因此post_process_pointmap_estimation会把该尺度应用到预测坐标上得到更接近真实尺度的三维结构。5. 姿态估计单人框关键点检测import torch from transformers import AutoImageProcessor, AutoModelForPoseEstimation from transformers.image_utils import load_image image load_image(http://images.cocodataset.org/val2017/000000004016.jpg) image_processor AutoImageProcessor.from_pretrained(facebook/sapiens2-pose-0.4b) model AutoModelForPoseEstimation.from_pretrained(facebook/sapiens2-pose-0.4b, device_mapauto) # Provide bounding boxes in COCO format (x, y, width, height) for each person boxes [[[270.8, 0.6, 294.1, 379.5]]] inputs image_processor(image, boxesboxes, return_tensorspt).to(model.device) with torch.inference_mode(): outputs model(**inputs) # outputs.heatmaps shape: (num_persons, num_keypoints, heatmap_height, heatmap_width) print(Heatmaps shape:, outputs.heatmaps.shape) # [1, 308, 256, 192] # Decode heatmaps to image-space keypoint coordinates results image_processor.post_process_pose_estimation(outputs, boxesboxes)[0] keypoints results[0][keypoints] # (num_keypoints, 2) — x/y in image coordinates scores results[0][scores] # (num_keypoints,) — per-keypoint confidence print(Keypoints shape:, keypoints.shape)姿态推理的预处理是检测框驱动的从 image_processing_sapiens2.py 可见boxesCOCO 格式 x/y/w/h会先经boxes_to_crop_params计算裁剪中心与尺度默认外扩 padding1.25并按目标长宽比校正再经crop_and_resize完成仿射等效裁剪缩放等价于原版 Sapiens2 代码库中 rotation0 的 cv2 仿射 warp缩小用双线性、放大用双三次。返回的 heatmaps 是每人每关键点的高斯热图0.4B 姿态模型为 308 关键点。6. 姿态估计增强水平翻转测试时增强翻转增强TTA通过平均原图与镜像图的预测来提升关键点精度。将[left_keypoint, right_keypoint]配对张量flip_pairs传给第二次前向模型会先把热图翻回原方向再返回内部调用flip_back因此两份输出可直接平均import torch from transformers import AutoImageProcessor, AutoModelForPoseEstimation from transformers.image_utils import load_image image load_image(http://images.cocodataset.org/val2017/000000004016.jpg) image_processor AutoImageProcessor.from_pretrained(facebook/sapiens2-pose-0.4b) model AutoModelForPoseEstimation.from_pretrained(facebook/sapiens2-pose-0.4b, device_mapauto) boxes [[[270.8, 0.6, 294.1, 379.5]]] inputs image_processor(image, boxesboxes, return_tensorspt).to(model.device) pixel_values inputs[pixel_values] flip_pairs torch.tensor(model.config.flip_pairs, devicemodel.device) with torch.inference_mode(): outputs model(pixel_values) outputs_flipped model(pixel_values.flip(-1), flip_pairsflip_pairs) results image_processor.post_process_pose_estimation(outputs, outputs_flippedoutputs_flipped, boxesboxes)[0] keypoints results[0][keypoints] scores results[0][scores]flip_pairs存放于config形如[[左耳, 右耳], ...]。参考 flip_back 的实现可知镜像热图会按左右对交换关键点通道再沿宽轴翻转回原方向若为回归型 target 还需把 offset 通道取负。把两次前向的结果同时交给post_process_pose_estimation后处理内部完成平均后再解码。7. 姿态估计训练带可见性权重的掩码 MSE将labelsGT 热图与可选的label_weights传给模型即可在 forward 中直接得到损失方便接入Trainer微调import torch from transformers import AutoImageProcessor, AutoModelForPoseEstimation from transformers.image_utils import load_image image load_image(http://images.cocodataset.org/val2017/000000004016.jpg) image_processor AutoImageProcessor.from_pretrained(facebook/sapiens2-pose-0.4b) model AutoModelForPoseEstimation.from_pretrained(facebook/sapiens2-pose-0.4b, device_mapauto) # Provide bounding boxes in COCO format (x, y, width, height) for each person boxes [[[270.8, 0.6, 294.1, 379.5]]] inputs image_processor(image, boxesboxes, return_tensorspt).to(model.device) # Create dummy labels (heatmaps) and visibility weights to simulate ground truth # 1.0 for visible keypoints, 0.0 for occluded/invisible keypoints batch_size, num_keypoints 1, 308 heatmap_height, heatmap_width 1024, 768 labels torch.randn(batch_size, num_keypoints, heatmap_height, heatmap_width, devicemodel.device) label_weights torch.ones(batch_size, num_keypoints, 1, 1, devicemodel.device) # Forward pass with loss calculation outputs model(**inputs, labelslabels, label_weightslabel_weights) print(Loss:, outputs.loss.item())从 Sapiens2ForPoseEstimation.forward 可见损失即F.mse_loss(heatmaps, labels, weightlabel_weights)label_weights形状(B, K, 1, 1)或与热图同尺寸均可对遮挡/不可见关键点置 0 即可屏蔽其梯度贡献。其余各任务分割、法线、pointmap、matting也都支持传入labels返回loss字段。8. 语义分割人体部位解析import torch from transformers import AutoImageProcessor, AutoModelForSemanticSegmentation from transformers.image_utils import load_image image load_image(http://images.cocodataset.org/val2017/000000004016.jpg) image_processor AutoImageProcessor.from_pretrained(facebook/sapiens2-seg-0.4b) model AutoModelForSemanticSegmentation.from_pretrained(facebook/sapiens2-seg-0.4b, device_mapauto) inputs image_processor(image, return_tensorspt).to(model.device) with torch.inference_mode(): outputs model(**inputs) # outputs.logits shape: (batch_size, num_labels, height, width) print(Logits shape:, outputs.logits.shape) # [1, 29, 1024, 768] # Get per-pixel class predictions, optionally resized to the original image size original_size (image.height, image.width) segmentation image_processor.post_process_semantic_segmentation( outputs, target_sizes[original_size] )[0] print(Segmentation map shape:, segmentation.shape) # [original_height, original_width]0.4B 分割 checkpoint 输出 29 类人体部位 logitsnum_labels29即前面法线/pointmap 可视化示例中所使用的背景掩码来源class 0 为背景。该模型还支持在预处理时同时传入segmentation_maps配合do_reduce_labels处理 ADE20k 等数据集的背景标签移位训练损失使用带semantic_loss_ignore_index默认 255的交叉熵。9. 图像抠像Mattingimport torch from transformers import AutoImageProcessor, AutoModelForImageMatting from transformers.image_utils import load_image image load_image(http://images.cocodataset.org/val2017/000000004016.jpg) image_processor AutoImageProcessor.from_pretrained(facebook/sapiens2-matting-1b) model AutoModelForImageMatting.from_pretrained(facebook/sapiens2-matting-1b, device_mapauto) inputs image_processor(image, return_tensorspt).to(model.device) with torch.inference_mode(): outputs model(**inputs) # outputs.foregrounds: (1, 3, H, W), outputs.alphas: (1, 1, H, W) — both in [0, 1] original_size (image.height, image.width) # Pass an optional background to composite the foreground over it. # A (3, 1, 1) tensor broadcasts as a uniform color; PIL images and numpy arrays are also accepted. background torch.tensor([0, 177, 64], dtypetorch.uint8).view(3, 1, 1) # chroma green in RGB result image_processor.post_process_image_matting( outputs, target_sizes[original_size], backgroundsbackground )[0] print(Alpha shape:, result[alpha].shape) # [1, original_height, original_width] print(Foreground shape:, result[foreground].shape) # [3, original_height, original_width] print(Composite shape:, result[composite].shape) # [3, original_height, original_width] — uint8 [0, 255]Matting 任务输出 sigmoid 激活、值域[0,1]的 alpha 与预乘前景Sapiens2ImageMattingOutput中的alphas/foregrounds。post_process_image_matting负责把结果缩放回目标尺寸并按公式composite foreground * (1 - alpha) * background合成抠像预览图——背景既可以是(3,1,1)纯色张量也接受 PIL 图像或 numpy 数组。ImageProcessor 后处理能力速查Sapiens2ImageProcessor 的默认配置为size {height: 1024, width: 768}、do_resizeTrue均值方差采用 ImageNet 默认值。除了各示例中用到的方法预处理还支持segmentation_maps语义分割标注与boxes姿态检测框输入。任务级后处理方法如下后处理方法对应任务主要行为post_process_semantic_segmentation分割argmax 得到逐像素类别可选target_sizes回缩原尺寸post_process_pose_estimation姿态解码热图为坐标kernel_size默认 11 的高斯模糊去偏 阈值支持 TTA 平均返回{keypoints, scores}post_process_normal_estimation法线去 padding、重采样、L2 归一化到[-1,1]post_process_pointmap_estimation点图去 padding、重采样、施加焦距尺度post_process_image_matting抠像拆分alpha/foreground可选背景合成composite其中姿态解码属于 DARK无偏数据处理风格先对热图做保留峰值的gaussian_blur_preserve_max模糊再基于泰勒展开做亚像素偏移修正post_dark_unbiased_data_processing因此post_process_pose_estimation返回的关键点坐标是浮点精度的图像空间坐标。配置速查关键参数一览下表汇总 Sapiens2Config 的高频参数model_type sapiens2并集成BackboneConfigMixin可通过out_features/out_indices选择骨干阶段参数默认值语义hidden_size/intermediate_size1024 / 4096隐藏维度与 MLP 中间维度num_hidden_layers/num_attention_heads24 / 16Transformer 层数与注意力头数patch_size/image_size16 / 224patch 尺寸与预训练图像尺寸num_register_tokens8register token 数量rope_theta100.0RoPE 基频use_gated_mlp/hidden_actTrue /siluSwiGLU 门控 MLPnum_key_value_attention_heads8GQA 层 KV 头数num_first/last_full_attention_layers8 / 8首尾使用完整注意力的层数use_qk_norm/rms_norm_epsTrue / 1e-6QK 归一化与 RMSNorm epslayerscale_value1.0LayerScale 初值pos_embed_rescale2.0训练期 RoPE 坐标随机缩放幅度semantic_loss_ignore_index255分割损失的忽略标签flip_pairsNone姿态 TTA 的左右关键点配对head_configNone解码头子配置Sapiens2HeadConfig小结Sapiens2 在 Transformers 中的集成是编码器 统一解码头 任务后处理的清晰范式主干支持任意分辨率推理、register tokens 与 QK-norm解码头由 Sapiens2HeadConfig 声明式配置五个任务通过各自的AutoModelFor*入口开箱即用法线、pointmap 等原始输出再经 ImageProcessor 的后处理方法完成坐标/通道语义的还原。你可以基于本文示例直接替换 checkpoint 名进行实验也可以从 modular_sapiens2.py 出发了解官方模型是如何以模块化方式维护与生成代码的。该模型代码版权归属 Meta Platforms, Inc. 与 HuggingFace Inc.遵循 Sapiens2 License使用时请留意对应许可条款。【免费下载链接】transformers Transformers: the model-definition framework for state-of-the-art machine learning models in text, vision, audio, and multimodal models, for both inference and training.项目地址: https://gitcode.com/GitHub_Trending/tra/transformers创作声明：本文部分内容由AI辅助生成（AIGC），仅供参考