数据目录数据治理数据血缘后端前端数据工程数据集成【免费下载链接】datahubThe Context Platform for your Data and AI Stack项目地址https://gitcode.com/GitHub_Trending/da/datahub点击查看免费下载本文以 DataHub 元数据模型文档《OwnershipType》为核心系统讲解ownershipType实体如何定义数据资产所有者的角色与职责如技术负责人、业务负责人、数据管家等并深入仓库源码PDL 模型、OwnershipTypeService、GraphQL Resolver、Python SDK 示例剖析其 URN 标识、aspect 结构、内置与自定义类型的差异、增删改查全流程及向后兼容策略。读完本文你将能够在自己的 DataHub 实例中创建自定义所有权类型、将其分配给数据集并正确管理其生命周期。一、什么是 OwnershipType在 DataHub 中ownershipType实体代表一种自定义的所有权类别custom ownership category。所有权类型定义了用户或用户组对数据资产可以承担的角色与职责。DataHub 内置了Technical Owner技术负责人、Business Owner业务负责人、Data Steward数据管家等所有权类型同时允许组织根据自身的治理模型和组织结构创建自定义的所有权类型从而将谁拥有这份数据这一元数据从固定枚举升级为可扩展的一等实体。从代码结构上看该实体由三部分构成实体 Key aspectOwnershipTypeKey.pdlaspect 名ownershipTypeKey实体信息 aspectOwnershipTypeInfo.pdlaspect 名ownershipTypeInfo历史遗留的枚举定义OwnershipType.pdl二、Identity如何唯一标识一个所有权类型OwnershipType实体由单一字段唯一标识id所有权类型的唯一标识字符串。对于自定义类型通常是一个 UUID对于内置类型则是带系统前缀的标识符。URN 结构遵循模式urn:li:ownershipType:id典型示例内置类型urn:li:ownershipType:__system__technical_owner自定义类型urn:li:ownershipType:8b3d78d1-a9d9-4f79-a948-10c52e3e8f9e系统自动生成的 UUID具名自定义类型urn:li:ownershipType:data_quality_lead组织自定义的人类可读 id在模型层OwnershipTypeKey记录了 id 与显示名的设计区别id 是用于数据所有权类型名的唯一标识如 Business Owner、Data Steward、Technical Owner应当与用于显示的 name 字段区分开见 OwnershipTypeKey.pdl。而实体 URN 由EntityKeyUtils.convertEntityKeyToUrn(key, Constants.OWNERSHIP_TYPE_ENTITY_NAME)根据 Key 生成见 OwnershipTypeService.java。三、核心能力ownershipTypeInfo 与 Status 管理3.1 核心信息ownershipTypeInfoownershipTypeInfoaspect 承载所有权类型的关键元数据字段定义见 OwnershipTypeInfo.pdl字段类型说明namestring必填所有权类型的显示名称如 Data Quality Lead、Compliance Officerdescriptionoptional string该所有权类型职责与范围的详细说明createdAuditStamp记录创建时间与创建者的审计戳lastModifiedAuditStamp记录最近一次修改时间与操作者的审计戳值得注意的是name和审计字段在 PDL 中都带有Searchable注解name使用WORD_GRAM字段类型enableAutocomplete: trueboostScore: 10.0即支持搜索与自动补全且权重较高created.time被索引为createdAtDATETIMEcreated.actor被索引为createdByURNlastModified.time被索引为lastModifiedAtDATETIMElastModified.actor被索引为lastModifiedByURN。这意味着所有权类型天然具备可搜索、可排序、可按创建人过滤的索引能力无需额外配置。3.2 内置类型 vs 自定义类型DataHub 随系统自动创建四个内置所有权类型类型 id说明__system__technical_owner参与资产的生产、维护或分发__system__business_owner与资产相关的主要利益相关者或领域专家__system__data_steward参与资产的治理__system__none未指定所有权类型内置类型具有以下约束这些规则在 OwnershipTypeService.java 中以isSystemOwnershipType方法硬编码判定即URN 的 id 以__system__前缀开头内置类型 id 以__system__前缀开头不能硬删除hard-delete只能通过statusaspect 软删除自定义类型没有此前缀限制可以被彻底删除。3.3 在所有权分配中的使用所有权类型通过数据资产ownershipaspect 中Owner记录的typeUrn字段被引用从而在资产所有者与其具体角色之间建立关联Owner { owner: urn:li:corpuser:jdoe typeUrn: urn:li:ownershipType:data_quality_lead type: CUSTOM // deprecated field, maintained for backwards compatibility }Ownershipaspect见 Ownership.pdl还维护了一个ownerTypes映射该映射将所有者按其所有权类型 URN 分组通过 mutation hook 自动填充ownerTypes: optional map[string, array[Urn]] {}该字段在 PDL 中以MAP_ARRAY类型建立搜索索引queryByDefault: false可用于按所有权类型聚合与检索资产的所有者。此外Ownership还包含lastModified审计戳默认值为time: 0、actor: urn:li:corpuser:unknown其中 time 为 0 表示缺失数据。3.4 Status 管理statusaspect 控制所有权类型是激活还是软删除状态removed当设为 true 时该所有权类型被视为已删除但其引用被保留。内置所有权类型只能软删除status.removed true而自定义类型可以从系统中完全移除。四、代码实战创建、分配与查询自定义所有权类型4.1 使用 Python SDK 创建自定义所有权类型示例脚本 ownership_type_create_custom.py 演示了完整的创建流程它依次发送两条 MCPMetadataChangeProposal先发送 Key aspect再发送 Info aspect。import os import time from datahub.emitter.mce_builder import make_user_urn from datahub.emitter.mcp import MetadataChangeProposalWrapper from datahub.emitter.rest_emitter import DatahubRestEmitter from datahub.metadata.schema_classes import ( AuditStampClass, OwnershipTypeInfoClass, OwnershipTypeKeyClass, ) emitter DatahubRestEmitter( gms_serveros.getenv(DATAHUB_GMS_URL, http://localhost:8080), tokenos.getenv(DATAHUB_GMS_TOKEN), ) ownership_type_id data_quality_lead ownership_type_urn furn:li:ownershipType:{ownership_type_id} current_timestamp int(time.time() * 1000) actor_urn make_user_urn(datahub) # Emit the key aspect ownership_type_key OwnershipTypeKeyClass(idownership_type_id) emitter.emit_mcp( MetadataChangeProposalWrapper( entityUrnownership_type_urn, aspectownership_type_key, ) ) # Emit the info aspect ownership_type_info OwnershipTypeInfoClass( nameData Quality Lead, descriptionResponsible for ensuring data quality standards and monitoring data quality metrics, createdAuditStampClass(timecurrent_timestamp, actoractor_urn), lastModifiedAuditStampClass(timecurrent_timestamp, actoractor_urn), ) emitter.emit_mcp( MetadataChangeProposalWrapper( entityUrnownership_type_urn, aspectownership_type_info, ) ) print(fCreated custom ownership type: {ownership_type_urn})关键点解读Emitter 配置DatahubRestEmitter默认指向http://localhost:8080GMS可通过环境变量DATAHUB_GMS_URL和DATAHUB_GMS_TOKEN覆盖便于接入带认证的部署环境两步写入先写 Key aspect 确立实体身份再写ownershipTypeInfoaspect 补充展示信息两步都基于同一 URNurn:li:ownershipType:data_quality_lead审计戳AuditStamp的time为毫秒级时间戳int(time.time() * 1000)actor使用make_user_urn(datahub)构造。4.2 为数据集分配带自定义所有权类型的所有者示例脚本 dataset_add_owner_custom_type.py 演示了如何为数据集添加带自定义所有权类型的所有者from datahub.emitter.mce_builder import ( make_dataset_urn, make_ownership_type_urn, make_user_urn, ) from datahub.emitter.mcp import MetadataChangeProposalWrapper from datahub.emitter.rest_emitter import DatahubRestEmitter from datahub.ingestion.graph.client import DataHubGraph, DataHubGraphConfig from datahub.metadata.schema_classes import ( OwnerClass, OwnershipClass, OwnershipTypeClass, ) # Create DataHub client graph DataHubGraph(DataHubGraphConfig(serverhttp://localhost:8080)) emitter DatahubRestEmitter(http://localhost:8080) # Create dataset URN dataset_urn make_dataset_urn(platformsnowflake, nameanalytics.users, envPROD) # Create custom ownership type URN # This should reference a previously created custom ownership type custom_ownership_type_urn make_ownership_type_urn(data_quality_lead) # Create an owner with the custom ownership type owner OwnerClass( ownermake_user_urn(jdoe), typeOwnershipTypeClass.CUSTOM, # Use CUSTOM enum for custom types typeUrncustom_ownership_type_urn, # Reference the custom ownership type entity ) # Get existing ownership or create new try: existing_ownership graph.get_aspect(dataset_urn, OwnershipClass) if existing_ownership: # Add to existing owners existing_ownership.owners.append(owner) ownership existing_ownership else: # Create new ownership aspect ownership OwnershipClass(owners[owner]) except Exception: # Create new ownership aspect if retrieval fails ownership OwnershipClass(owners[owner]) # Emit the ownership aspect mcp MetadataChangeProposalWrapper( entityUrnstr(dataset_urn), aspectownership, ) emitter.emit_mcp(mcp) print( fAdded owner {owner.owner} with custom ownership type {custom_ownership_type_urn} ) print(fto dataset {dataset_urn})要点解读make_ownership_type_urn(data_quality_lead)构造urn:li:ownershipType:data_quality_lead该自定义类型必须已提前创建OwnerClass中type字段使用OwnershipTypeClass.CUSTOM对应枚举中的CUSTOM值typeUrn指向自定义所有权类型实体脚本同时演示了DataHubGraph.get_aspect读取已有Ownershipaspect 并追加 owner 的读-改-写模式避免覆盖原有所有者列表。4.3 列出所有所有权类型GraphQL示例脚本 ownership_type_list.py 通过 GraphQL 查询列出全部所有权类型from datahub.ingestion.graph.client import DataHubGraph, DataHubGraphConfig # Create DataHub client graph DataHubGraph(DataHubGraphConfig(serverhttp://localhost:8080)) # Search for all ownership type entities # Note: The GraphQL API provides a listOwnershipTypes query, but we can also # use the search API to find all ownership types search_query query listOwnershipTypes($input: ListOwnershipTypesInput!) { listOwnershipTypes(input: $input) { start count total ownershipTypes { urn type info { name description } } } } variables { input: { start: 0, count: 100, # Adjust as needed } } # Execute the GraphQL query result graph.execute_graphql(querysearch_query, variablesvariables) # Process and display the results if result and listOwnershipTypes in result: ownership_types result[listOwnershipTypes][ownershipTypes] total result[listOwnershipTypes][total] print(fFound {total} ownership types:) print(- * 80) for ownership_type in ownership_types: urn ownership_type[urn] name ownership_type[info][name] description ownership_type[info].get(description, No description) print(fURN: {urn}) print(fName: {name}) print(fDescription: {description}) print(- * 80) else: print(No ownership types found or query failed)listOwnershipTypes查询支持start/count分页参数返回total总数与ownershipTypes列表每条包含urn、type与info { name, description }。4.4 通过 REST API 查询所有权类型使用entitiesV2接口按 URN 获取某个所有权类型的完整信息curl http://localhost:8080/entitiesV2/urn%3Ali%3AownershipType%3A__system__technical_owner响应包含ownershipTypeInfo与status两个 aspect{ urn: urn:li:ownershipType:__system__technical_owner, aspects: { ownershipTypeInfo: { value: { name: Technical Owner, description: Involved in the production, maintenance, or distribution of the asset(s)., created: { time: 1234567890000, actor: urn:li:corpuser:datahub }, lastModified: { time: 1234567890000, actor: urn:li:corpuser:datahub } } } } }注意 URN 中的:需要 URL 编码为%3A。后端读取时通过getOwnershipTypeEntityResponse同时拉取ownershipTypeInfo与status两个 aspect见 OwnershipTypeService.java。五、集成点Owner Aspect、GraphQL 与鉴权5.1 与 Owner aspect 的关系ownershipType实体与绝大多数数据资产数据集、仪表盘、图表等上的ownershipaspect 存在主关联关系。Owner记录中的typeUrn字段引用一个 ownershipType 实体对应的Relationship声明如下Relationship { name: ownershipType, entityTypes: [ ownershipType ] } typeUrn: optional Urn该关系使得系统可以在搜索与发现search and discovery中按所有权类型过滤资产在整个组织中按角色对所有者进行分组追踪每个资产上谁承担了什么职责。5.2 GraphQL API 集成GraphQL API 通过以下 Resolver 暴露所有权类型的完整 CRUD 能力全部位于 datahub-graphql-core/.../resolvers/ownership/Resolver功能CreateOwnershipTypeResolver.java创建新的自定义所有权类型UpdateOwnershipTypeResolver.java修改已有所有权类型DeleteOwnershipTypeResolver.java删除所有权类型对系统类型执行软删除ListOwnershipTypesResolver.java返回所有可用所有权类型GraphQL 实体类型为CUSTOM_OWNERSHIP_TYPE映射到OwnershipTypeEntityGraphQL 类型。从 CreateOwnershipTypeResolver.java 可以看到创建成功后 Resolver 会构建OwnershipTypeEntity.builder().setType(EntityType.CUSTOM_OWNERSHIP_TYPE)返回给前端。前端侧数据资产侧边栏的 OwnershipTypesSelect.tsx 与权限策略表单中的 OwnershipTypesSelect.tsx 均消费这些 GraphQL 查询。5.3 鉴权Authorization管理所有权类型需要特定权限创建、更新或删除所有权类型需要canManageOwnershipTypes权限该权限通常仅授予平台管理员与治理团队。这一点在 Resolver 层有强制校验CreateOwnershipTypeResolver在执行业务逻辑前先调用AuthorizationUtils.canManageOwnershipTypes(context)不满足时直接抛出AuthorizationException(Unauthorized to perform this action. Please contact your DataHub administrator.)见 CreateOwnershipTypeResolver.java。值得注意的是OwnershipTypeService本身不做鉴权其 Javadoc 明确说明 no Authorization is performed within the service见 OwnershipTypeService.java鉴权职责完全由上层调用方GraphQL Resolver 等承担。六、常见使用模式根据文档与源码以下场景最适合引入自定义所有权类型组织特定角色定义与你的组织结构匹配的所有权类型如 Product Manager、Data Engineer、Analytics Lead合规角色为监管合规创建类型如 Privacy Officer、Compliance Reviewer、Audit Contact生命周期角色在数据生命周期各阶段追踪不同职责如 Data Producer、Data Consumer、Data Custodian领域特定角色为特定领域建立所有权类型如 Marketing Data Owner、Finance Data Steward。七、注意事项与异常行为7.1 向后兼容type 与 typeUrn 并存Owner记录同时包含已弃用的type字段OwnershipType枚举与较新的typeUrn字段指向 ownershipType 实体的 URN。枚举字段仅为向后兼容而保留新实现不应再使用。当使用自定义所有权类型时枚举字段被设置为CUSTOM。枚举定义见 OwnershipType.pdl。7.2 系统类型删除限制内置所有权类型id 以__system__开头的类型无法被完全删除。OwnershipTypeService.deleteOwnershipType()的行为如下见 OwnershipTypeService.java系统类型软删除即写入statusaspect 并设置removed true自定义类型硬删除即调用entityClient.deleteEntity整体移除实体若deleteReferences为 true还会进一步调用deleteEntityReferences清理引用。这保证了即使系统类型被停用对其的引用依然有效。对应的测试覆盖见 OwnershipTypeServiceTest.java。7.3 从枚举到实体的迁移历史上所有权类型被定义为一个固定枚举OwnershipType.pdl。ownershipType实体的引入在保持兼容性的同时带来了可扩展性。已弃用的枚举值DEVELOPER、DATAOWNER、DELEGATE、PRODUCER、CONSUMER、STAKEHOLDER应迁移到对应的系统所有权类型TECHNICAL_OWNER、BUSINESS_OWNER、DATA_STEWARD。从 PDL 注释中可以确认映射关系DEVELOPER/DATAOWNER/DELEGATE/PRODUCER应使用TECHNICAL_OWNERCONSUMER使用TECHNICAL_OWNER或BUSINESS_OWNERSTAKEHOLDER使用TECHNICAL_OWNER、BUSINESS_OWNER或DATA_STEWARD。7.4 ID 生成规则创建自定义所有权类型时注意以下 ID 约束系统会自动为id字段生成 UUIDOwnershipTypeService.createOwnershipType中使用UUID.randomUUID().toString()见 OwnershipTypeService.java组织也可以使用人类可读的 ID如data_quality_lead以便管理ID 不得包含保留字符且必须是 URL 安全的__system__前缀为内置类型保留自定义类型不得使用。八、源码延伸测试与前端消费如果你想进一步深入该实体的实现细节以下仓库路径可作为继续探索的入口后端服务与 CRUD 实现OwnershipTypeService.java、OwnershipTypeServiceFactory.javaGraphQL 类型映射OwnershipType.java、OwnershipTypeMapper.javaResolver 单元测试CreateOwnershipTypeResolverTest.java、DeleteOwnershipTypeResolverTest.java、ListOwnershipTypesResolverTest.java、UpdateOwnershipTypeResolverTest.java前端消费所有权类型选择器 OwnershipTypesSelect.tsx、useOwnershipTypes.ts 及其测试 useOwnershipTypes.test.ts。九、总结ownershipType实体是 DataHub 所有权模型从固定枚举迈向可扩展实体体系的关键一步。通过urn:li:ownershipType:id的唯一标识、ownershipTypeInfo与status两个核心 aspect、以及 Python SDK / REST / GraphQL 三类 API 的完整支持组织可以灵活地构建贴合自身治理结构的角色体系同时保持对历史枚举数据的向后兼容。建议新项目一律使用typeUrn引用所有权类型并将内置的__system__类型视为只读治理基础仅在必要时通过软删除停用。赞分享数据目录数据治理数据血缘后端前端数据工程数据集成【免费下载链接】datahubThe Context Platform for your Data and AI Stack项目地址https://gitcode.com/GitHub_Trending/da/datahub点击查看免费下载相关推荐DataHub 自定义 Ownership Types从零定义所有权类型到为实体分配 Owner 的完整实践DataHub 自定义 Ownership Types从零定义所有权类型到为实体分配 Owner 的完整实践 本文基于 DataHub 官方文档 docs/o数据目录数据治理数据血缘后端前端数据工程数据集成DataHub v0.2.8 版本解析自定义所有权类型、Data Products 与 Tableau Chrome 扩展全面指南DataHub v0.2.8 版本解析自定义所有权类型、Data Products 与 Tableau Chrome 扩展全面指南 本文基于 DataHub数据目录数据治理数据血缘后端前端数据工程数据集成树莓派GPU编程终极入门PyVideoCore让QPU开发不再复杂树莓派GPU编程终极入门PyVideoCore让QPU开发不再复杂 想要解锁树莓派的GPU计算能力吗PyVideoCore为你提供了一个简单而强大的Pyth上一篇如何快速掌握正则表达式Regulex可视化工具让学习效率提升300%下一篇终极指南如何使用 qrcode.react 在 React 中轻松生成二维码创作声明：本文部分内容由AI辅助生成（AIGC），仅供参考