# 数据采集

来源快照：`result_data_new_260518.json`

- 来源 ID：`1243415516585656329`
- 模块：文件设置
- 目录：采集设置
- space ID：`1160193825049370641`
- 原文链接：未提供
- 产品基线：未提供

## 正文

# **一、功能背景及价值用途**

有客户希望通过sql日志或者业务截图能够识别出来第三方数据库中相关表结构后与E10数据库表进行表字段映射，并接入ETL流程的能力实现数据和相关附件的采集

# **二、功能概述**

2.1 【文件采集设置】菜单更名为【采集设置】，【文件采集设置】角色权限更名为【采集设置】

2.2 新增页签为【数据采集】、【文件采集】，文件采集为原本功能

2.3 进入【数据采集】tab页，默认为空，给出提示【您暂时还没有采集任务信息，快新建一个试试吧~】

2.4 点击右上角的【新建】或者页面中的【新建采集任务】，打开【新建采集任务】窗口

2.5 受原本角色权限【采集设置】控制，无此权限的用户在后台文件设置中不显示此菜单

2.6 需采购千里聆ocr识别和档案管理专项

# **三、使用说明**

权限：角色权限-文件设置-采集设置

![](https://weapp.eteams.cn/papi/file/preview?fileId=1243417109822996481&type=redirect&customParam=%7B%22isExtDist%22%3Atrue%2C%22distributeId%22%3A%22%22%2C%22externalId%22%3A%22%22%2C%22appId%22%3A%22%22%7D)

## **3.1 新建连接器**

步骤：后台管理中心-平台引擎-ESB中心-连接器-新建连接器-创建具体连接

![](https://weapp.eteams.cn/papi/file/preview?fileId=1243417109902278659&type=redirect&customParam=%7B%22isExtDist%22%3Atrue%2C%22distributeId%22%3A%22%22%2C%22externalId%22%3A%22%22%2C%22appId%22%3A%22%22%7D)

## **3.2 导入档案专项应用包**

![](https://weapp.eteams.cn/papi/file/preview?fileId=1243417105611915269&type=redirect&customParam=%7B%22isExtDist%22%3Atrue%2C%22distributeId%22%3A%22%22%2C%22externalId%22%3A%22%22%2C%22appId%22%3A%22%22%7D)

## **3.3 配置采集任务**

入口：文件设置-采集设置-数据采集

![](https://weapp.eteams.cn/papi/file/preview?fileId=1243417109902278657&type=redirect&customParam=%7B%22isExtDist%22%3Atrue%2C%22distributeId%22%3A%22%22%2C%22externalId%22%3A%22%22%2C%22appId%22%3A%22%22%7D)

【采集任务名称】：必填

【数据库连接】：必填，配置数据库连接信息

![](https://weapp.eteams.cn/papi/file/preview?fileId=1243417122779619331&type=redirect&customParam=%7B%22isExtDist%22%3Atrue%2C%22distributeId%22%3A%22%22%2C%22externalId%22%3A%22%22%2C%22appId%22%3A%22%22%7D)

【采集数据映射】：自动映射、手动映射

![](https://weapp.eteams.cn/papi/file/preview?fileId=1243417122803957761&type=redirect&customParam=%7B%22isExtDist%22%3Atrue%2C%22distributeId%22%3A%22%22%2C%22externalId%22%3A%22%22%2C%22appId%22%3A%22%22%7D)

自动映射：通过用户上传的sql日志或者业务截图识别出来第三方表结构后再与E10数据库表进行映射设置

![](https://weapp.eteams.cn/papi/file/preview?fileId=1243417118563926017&type=redirect&customParam=%7B%22isExtDist%22%3Atrue%2C%22distributeId%22%3A%22%22%2C%22externalId%22%3A%22%22%2C%22appId%22%3A%22%22%7D)

![](https://weapp.eteams.cn/papi/file/preview?fileId=1243417127043604483&type=redirect&customParam=%7B%22isExtDist%22%3Atrue%2C%22distributeId%22%3A%22%22%2C%22externalId%22%3A%22%22%2C%22appId%22%3A%22%22%7D)

分析节点：勾选分析节点的字段可用于字段映射中的源表字段展示

![](https://weapp.eteams.cn/papi/file/preview?fileId=1243417127043604484&type=redirect&customParam=%7B%22isExtDist%22%3Atrue%2C%22distributeId%22%3A%22%22%2C%22externalId%22%3A%22%22%2C%22appId%22%3A%22%22%7D)

选择所属系统和E10数据表，再进行字段关系映射

![](https://weapp.eteams.cn/papi/file/preview?fileId=1243417114104971270&type=redirect&customParam=%7B%22isExtDist%22%3Atrue%2C%22distributeId%22%3A%22%22%2C%22externalId%22%3A%22%22%2C%22appId%22%3A%22%22%7D)

![](https://weapp.eteams.cn/papi/file/preview?fileId=1243417118563926020&type=redirect&customParam=%7B%22isExtDist%22%3Atrue%2C%22distributeId%22%3A%22%22%2C%22externalId%22%3A%22%22%2C%22appId%22%3A%22%22%7D)

选择全量采集/增量采集，点击生成数据映射关系

![](https://weapp.eteams.cn/papi/file/preview?fileId=1243417122775007240&type=redirect&customParam=%7B%22isExtDist%22%3Atrue%2C%22distributeId%22%3A%22%22%2C%22externalId%22%3A%22%22%2C%22appId%22%3A%22%22%7D)

![](https://weapp.eteams.cn/papi/file/preview?fileId=1243417109911494660&type=redirect&customParam=%7B%22isExtDist%22%3Atrue%2C%22distributeId%22%3A%22%22%2C%22externalId%22%3A%22%22%2C%22appId%22%3A%22%22%7D)

此时，后台数据加工会生成etl数据

![](https://weapp.eteams.cn/papi/file/preview?fileId=1243417114273570821&type=redirect&customParam=%7B%22isExtDist%22%3Atrue%2C%22distributeId%22%3A%22%22%2C%22externalId%22%3A%22%22%2C%22appId%22%3A%22%22%7D)

手动映射：直接跳转ETL流程设置

![](https://weapp.eteams.cn/papi/file/preview?fileId=1243417118462058503&type=redirect&customParam=%7B%22isExtDist%22%3Atrue%2C%22distributeId%22%3A%22%22%2C%22externalId%22%3A%22%22%2C%22appId%22%3A%22%22%7D)

 

【附件采集设置】：根据【采集数据映射】返回是否有附件字段联动显示；若是数据映射中有附件字段，则展示【附件采集设置】字段，且必填，配置附件所在的源存储路径，用于采集附件数据

【采集源数据】：选择【存储连接】，取附件所在的源存储

![](https://weapp.eteams.cn/papi/file/preview?fileId=1243417122775007239&type=redirect&customParam=%7B%22isExtDist%22%3Atrue%2C%22distributeId%22%3A%22%22%2C%22externalId%22%3A%22%22%2C%22appId%22%3A%22%22%7D)

【采集文件路径】不必填，不配置默认采集存储上的全部数据，配置路径只采集指定路径下的数据

【采集形式】：全量采集、增量采集

全量采集：一次性采集全部目标数据

增量采集：只采集自上次采集后新增的数据 

## **3.4 采集效果**

手动采集：采集任务启用，自动开始采集，点击【点击查看】按钮，可查看数据采集明细和文件采集明细

![](https://weapp.eteams.cn/papi/file/preview?fileId=1243417118458658818&type=redirect&customParam=%7B%22isExtDist%22%3Atrue%2C%22distributeId%22%3A%22%22%2C%22externalId%22%3A%22%22%2C%22appId%22%3A%22%22%7D)

定时采集：采集任务启用,根据设置的时间定时同步，后台esb动作流新增对应的定时任务

![](https://weapp.eteams.cn/papi/file/preview?fileId=1243417118462058500&type=redirect&customParam=%7B%22isExtDist%22%3Atrue%2C%22distributeId%22%3A%22%22%2C%22externalId%22%3A%22%22%2C%22appId%22%3A%22%22%7D)

![](https://weapp.eteams.cn/papi/file/preview?fileId=1243417109846958083&type=redirect&customParam=%7B%22isExtDist%22%3Atrue%2C%22distributeId%22%3A%22%22%2C%22externalId%22%3A%22%22%2C%22appId%22%3A%22%22%7D)

同时也可在数据加工查看对应的数据，采集进度明细查看同手动采集

![](https://weapp.eteams.cn/papi/file/preview?fileId=1243417122775007234&type=redirect&customParam=%7B%22isExtDist%22%3Atrue%2C%22distributeId%22%3A%22%22%2C%22externalId%22%3A%22%22%2C%22appId%22%3A%22%22%7D)

数据采集完成之后，可在档案专项查看采集到的数据

![](https://weapp.eteams.cn/papi/file/preview?fileId=1243417114273570817&type=redirect&customParam=%7B%22isExtDist%22%3Atrue%2C%22distributeId%22%3A%22%22%2C%22externalId%22%3A%22%22%2C%22appId%22%3A%22%22%7D)

![](https://weapp.eteams.cn/papi/file/preview?fileId=1243417114099982338&type=redirect&customParam=%7B%22isExtDist%22%3Atrue%2C%22distributeId%22%3A%22%22%2C%22externalId%22%3A%22%22%2C%22appId%22%3A%22%22%7D)![](https://weapp.eteams.cn/papi/file/preview?fileId=1243417105652654088&type=redirect&customParam=%7B%22isExtDist%22%3Atrue%2C%22distributeId%22%3A%22%22%2C%22externalId%22%3A%22%22%2C%22appId%22%3A%22%22%7D)
