feat: 2026-07-11 全量更新 - 代办集成+会议室预定+知识迭代修复+UI统一+Bug修复
== 已部署上线 (9项) == - 代办事项真实数据源集成 (企微审批API 8bug修复链) - H5/坐席端 Logo样式统一+绿色背景 - 视频引导页修复 (localStorage key v2) - 坐席端 v9 Vue版本修复 (ElMessage._context) - 截图按钮 v10 修复 (getDisplayMedia user gesture) - 扫码样式恢复+H5扫码登录跳转修复 - H5截图快捷键提示 == 代码完成待部署 (3项) == - 知识迭代3Bug修复 (#8 POST端点/#7 MERGE幂等/#6 过期检查) - 会议室预定-小鱼易联终端 (40文件, 40/40测试通过) - IT资产升级审批推送 (asset_service.py) == 需求文档 (2项) == - 坐席端AI辅助消息框-PRD (4项新功能确认) - 坐席端布局优化建议 v2.0 (7天计划) == 新增文档 == - 日报-2026-07-11.md - 知识迭代Bug修复报告-20260711.md - 会议室预定-部署指南.md - CHANGELOG.md 更新 == 测试 == - test_todo_integration.py: 40/40 - test_meetingroom.py: 40/40 - test_bugfix_ki_suggestions.py: 21/21
This commit is contained in:
Simon
@@ -0,0 +1,64 @@
|
||||
# 复杂场景重构第二阶段 — 交付总结
|
||||
|
||||
## TL;DR
|
||||
P2(上下文压缩) + P3(多轮纠错) 全部开发完成，48/48 单元测试通过，待部署。
|
||||
|
||||
## 交付概览
|
||||
- **交付状态**: ✅ 开发完成，待部署
|
||||
- **测试通过率**: 48/48 (100%)
|
||||
- **已知问题数**: 0（预存 TS 类型错误非本次引入）
|
||||
- **SOP 流程**: PRD → 架构设计 → 代码实现(T01-T05) → QA 测试
|
||||
|
||||
## 文件清单
|
||||
|
||||
### 新建文件（15个）
|
||||
|
||||
**后端（8个）**:
|
||||
| 文件 | 功能 |
|
||||
|------|------|
|
||||
| `backend/app/utils/token_counter.py` | TokenCounter — tiktoken精确计数+字符估算兜底 |
|
||||
| `backend/app/services/automation/context_compressor.py` | ContextCompressor — 渐进式3级压缩+截断降级 |
|
||||
| `backend/app/services/automation/snapshot_service.py` | SnapshotService — 快照创建+撤销(限5次)+版本对比 |
|
||||
| `backend/app/services/automation/correction_service.py` | CorrectionService — 批量更正(原子事务)+依赖检查+版本链 |
|
||||
| `backend/alembic/versions/049_add_p2_p3_tables.py` | 迁移脚本 — 2新表+2新列 |
|
||||
| `backend/tests/test_p2_p3.py` | 48个单元测试 |
|
||||
| `docs/02-产品需求/复杂场景重构第二阶段-增量PRD.md` | PRD文档 |
|
||||
| `docs/03-技术架构/复杂场景重构第二阶段-架构设计.md` | 架构设计文档 |
|
||||
|
||||
**前端（7个）**:
|
||||
| 文件 | 端 | 功能 |
|
||||
|------|-----|------|
|
||||
| `frontend-agent/src/api/automation_p2.ts` | 坐席端 | P2/P3 API 请求封装 |
|
||||
| `frontend-agent/src/components/automation/DependencyWarning.vue` | 坐席端 | 依赖联动提示 |
|
||||
| `frontend-agent/src/views/automation/CorrectionHistory.vue` | 坐席端 | 更正历史时间线+撤销 |
|
||||
| `frontend-agent/src/views/automation/VersionDiff.vue` | 坐席端 | 版本对比+版本链 |
|
||||
| `frontend-agent/src/views/automation/CompressionLogs.vue` | 坐席端 | 压缩日志表格 |
|
||||
| `frontend-h5/src/components/automation/CorrectionNotice.vue` | H5 | 更正通知+依赖警告 |
|
||||
| `frontend-h5/src/components/automation/UndoButton.vue` | H5 | 撤销更正按钮 |
|
||||
|
||||
### 修改文件（6个）
|
||||
| 文件 | 变更 |
|
||||
|------|------|
|
||||
| `backend/app/models/automation.py` | +2模型(ContextCompression/InformationSnapshot) +2字段(derived_from/correction_reason) |
|
||||
| `backend/app/constants.py` | +3错误码(4017-4019) |
|
||||
| `backend/app/config.py` | +6配置项(压缩阈值/超时/级别/保留轮数/撤销次数等) |
|
||||
| `backend/app/schemas/automation.py` | +10个P2/P3 Schema |
|
||||
| `backend/app/api/automation.py` | +6个API端点+import |
|
||||
| `backend/app/services/automation/__init__.py` | 导出新服务 |
|
||||
|
||||
### 新增 API 端点（6个）
|
||||
| 方法 | 路径 | 功能 |
|
||||
|------|------|------|
|
||||
| POST | `/itportal/automation/sessions/{id}/batch-correct` | 批量更正（原子事务） |
|
||||
| POST | `/itportal/automation/sessions/{id}/undo-correction` | 撤销最近一次更正 |
|
||||
| GET | `/itportal/automation/sessions/{id}/correction-history` | 更正历史 |
|
||||
| GET | `/itportal/automation/sessions/{id}/version-chain/{item}` | 版本链 |
|
||||
| POST | `/itportal/automation/sessions/{id}/version-diff/{item}` | 版本对比 |
|
||||
| GET | `/itportal/automation/sessions/{id}/compression-logs` | 压缩日志 |
|
||||
|
||||
## 用户下一步建议
|
||||
1. **部署后端**：通过堡垒机上传新增/修改的 `.py` 文件 → 执行迁移 049 → `docker compose up -d backend` 重建容器
|
||||
2. **部署前端**：坐席端 `npm run build` → 上传 dist → nginx 重启；H5 端同理
|
||||
3. **安装 tiktoken**：`pip install tiktoken>=0.5.0`（如未安装，TokenCounter 自动降级为字符估算）
|
||||
4. **企微内实测**：部署后在企微中走一遍 AI 对话，验证上下文压缩和多轮纠错的实际体验
|
||||
5. **配置调整**：如需调整压缩阈值或撤销次数，修改 `config.py` 或环境变量
|
||||
@@ -0,0 +1,173 @@
|
||||
---
|
||||
name: desktop-automation
|
||||
description: "Windows 桌面自动化技能——通过模拟键盘鼠标操作控制桌面应用程序，适用于无法通过 API 或协议直接交互的软件（如微信PC版、钉钉、企业内部系统等）。核心能力包括窗口管理、鼠标点击移动拖拽、键盘输入快捷键、屏幕截图、OCR 文字识别、图像匹配定位。典型场景：导出微信聊天记录为图文 HTML、自动填表、批量文件操作、桌面软件自动化测试。当用户请求模拟操作桌面软件、自动点击、键盘鼠标控制、导出聊天记录、操控微信钉钉等桌面 GUI 自动化任务时触发此技能。"
|
||||
agent_created: true
|
||||
---
|
||||
|
||||
# Desktop Automation
|
||||
|
||||
## Overview
|
||||
|
||||
通过 PyAutoGUI + uiautomation + win32gui 组合方案，在 Windows 上实现桌面应用程序的
|
||||
全自动化操作。覆盖从窗口定位、鼠标键盘模拟、屏幕截图到 OCR 文字识别的完整链路，支持将
|
||||
操作过程和结果编译为图文 HTML 报告。
|
||||
|
||||
**技术栈分层：**
|
||||
|
||||
| 层级 | 工具 | 用途 |
|
||||
|------|------|------|
|
||||
| 窗口控制 | win32gui / pygetwindow | 查找、激活、移动窗口 |
|
||||
| UI 控件 | uiautomation | 精准定位按钮/输入框/列表等控件 |
|
||||
| 鼠标键盘 | pyautogui | 坐标级点击、移动、拖拽、键盘输入 |
|
||||
| 截图识别 | Pillow + PaddleOCR | 屏幕截图 + 中英文 OCR |
|
||||
| 剪贴板 | pyperclip | 中文文本输入（绕过 pyautogui 编码问题）|
|
||||
|
||||
## Safety Rules (MUST READ)
|
||||
|
||||
桌面自动化会直接操控用户电脑，执行前必须遵守以下规则：
|
||||
|
||||
1. **确认前置条件**：确保目标应用已打开并处于可操作状态（如微信已登录）
|
||||
2. **坐标依赖性**：pyautogui 基于屏幕绝对坐标，分辨率/DPI 缩放变化会导致点击偏移。
|
||||
优先使用 uiautomation 控件定位，pyautogui 坐标作为兜底
|
||||
3. **安全中断**：所有脚本内置 `FAILSAFE=True`，将鼠标快速移至屏幕四角可紧急终止
|
||||
4. **操作前截图**：执行关键操作前先截图，便于回溯和调试
|
||||
5. **用户确认**：涉及发送消息、删除文件等不可逆操作前，必须向用户确认
|
||||
6. **DPI 适配**：Windows 高 DPI（150%/200%）下需调用 `pyautogui.size()` 校准坐标
|
||||
|
||||
## Quick Start
|
||||
|
||||
### 1. 环境初始化
|
||||
|
||||
首次使用时运行环境检测脚本，安装缺失依赖：
|
||||
|
||||
```bash
|
||||
python scripts/init_env.py
|
||||
```
|
||||
|
||||
该脚本会检测并安装：pyautogui, uiautomation, Pillow, paddleocr, pygetwindow,
|
||||
pyperclip, opencv-python
|
||||
|
||||
### 2. 核心自动化引擎
|
||||
|
||||
`scripts/desktop_auto.py` 提供通用桌面自动化 API：
|
||||
|
||||
```python
|
||||
from desktop_auto import DesktopAuto
|
||||
|
||||
auto = DesktopAuto()
|
||||
|
||||
# 窗口管理
|
||||
auto.find_window("微信") # 查找窗口
|
||||
auto.activate_window("微信") # 激活并前置窗口
|
||||
auto.get_window_rect("微信") # 获取窗口位置和大小
|
||||
|
||||
# 鼠标操作
|
||||
auto.click(x=500, y=300) # 绝对坐标点击
|
||||
auto.click_relative(dx=50, dy=0) # 相对当前位置点击
|
||||
auto.double_click(x=500, y=300) # 双击
|
||||
auto.right_click(x=500, y=300) # 右键
|
||||
auto.drag(start, end, duration=0.5) # 拖拽
|
||||
auto.scroll(clicks=-3) # 向下滚动3格
|
||||
|
||||
# 键盘操作
|
||||
auto.type_text("你好世界") # 中文输入（通过剪贴板）
|
||||
auto.press_key("enter") # 单键
|
||||
auto.hotkey("ctrl", "c") # 组合键
|
||||
auto.press_key("ctrl+shift+esc") # 快捷键
|
||||
|
||||
# 截图
|
||||
auto.screenshot("fullscreen.png") # 全屏截图
|
||||
auto.screenshot_region(rect, "region.png") # 区域截图
|
||||
auto.screenshot_window("微信", "wechat.png") # 窗口截图
|
||||
|
||||
# 图像匹配定位
|
||||
pos = auto.locate_on_screen("button.png") # 找图返回坐标
|
||||
if pos:
|
||||
auto.click(pos.x, pos.y)
|
||||
|
||||
# 等待
|
||||
auto.wait(2) # 等待2秒
|
||||
auto.wait_for_image("loaded.png", timeout=10) # 等待图片出现
|
||||
```
|
||||
|
||||
### 3. 微信聊天记录导出
|
||||
|
||||
`scripts/wechat_export.py` 专门用于导出微信聊天记录：
|
||||
|
||||
```bash
|
||||
python scripts/wechat_export.py --contact "张三" --count 100 --output ./output
|
||||
```
|
||||
|
||||
参数说明：
|
||||
- `--contact`：好友或群聊名称（模糊匹配）
|
||||
- `--count`：要导出的消息条数（默认50）
|
||||
- `--output`：输出目录（默认 ./wechat_export）
|
||||
- `--scroll-pause`：每次滚动间隔秒数（默认1.5，过快会漏消息）
|
||||
- `--ocr`：是否启用 OCR 文字识别（默认开启）
|
||||
- `--screenshot`：是否保存截图（默认开启）
|
||||
|
||||
导出流程：
|
||||
1. 激活微信窗口 → 搜索联系人 → 打开聊天窗口
|
||||
2. 滚动到顶部（加载历史消息）→ 逐步向下滚动截图
|
||||
3. 对每张截图执行 OCR → 解析消息结构（发送者/时间/内容）
|
||||
4. 消息去重 → 编译为 HTML 报告
|
||||
|
||||
### 4. HTML 报告编译
|
||||
|
||||
`scripts/html_compiler.py` 将截图和 OCR 结果编译为图文 HTML：
|
||||
|
||||
```python
|
||||
from html_compiler import ChatHTMLCompiler
|
||||
|
||||
compiler = ChatHTMLCompiler(output_dir="./output")
|
||||
compiler.add_messages(messages) # 添加结构化消息
|
||||
compiler.add_screenshots(screenshots) # 添加截图
|
||||
compiler.compile("chat_report.html") # 生成 HTML
|
||||
```
|
||||
|
||||
生成的 HTML 特性：
|
||||
- 仿微信聊天气泡样式（绿色=自己，白色=对方）
|
||||
- 时间线分组显示
|
||||
- 图片消息内嵌展示
|
||||
- 支持搜索和过滤
|
||||
- 响应式布局，移动端可读
|
||||
|
||||
## Workflow Decision Tree
|
||||
|
||||
根据任务类型选择执行路径：
|
||||
|
||||
```
|
||||
用户请求
|
||||
├── 导出微信聊天记录 → scripts/wechat_export.py
|
||||
├── 操控其他桌面软件 → scripts/desktop_auto.py (通用API)
|
||||
├── 自动填表/录入 → desktop_auto.py (控件定位优先)
|
||||
├── 批量截图监控 → desktop_auto.py (screenshot + scroll)
|
||||
└── 其他 GUI 自动化 → desktop_auto.py + 自定义逻辑
|
||||
```
|
||||
|
||||
## Script Index
|
||||
|
||||
| 脚本 | 用途 | 入口 |
|
||||
|------|------|------|
|
||||
| `scripts/init_env.py` | 环境检测与依赖安装 | `python scripts/init_env.py` |
|
||||
| `scripts/desktop_auto.py` | 核心自动化引擎（通用API） | `from desktop_auto import DesktopAuto` |
|
||||
| `scripts/wechat_export.py` | 微信聊天记录导出 | `python scripts/wechat_export.py [options]` |
|
||||
| `scripts/html_compiler.py` | HTML 图文报告编译器 | `from html_compiler import ChatHTMLCompiler` |
|
||||
|
||||
## References
|
||||
|
||||
- `references/automation_guide.md` — 详细 API 参考、坐标定位策略、DPI 适配指南、常见问题
|
||||
- `references/wechat_patterns.md` — 微信 PC 版 UI 结构分析、控件定位模式、已知坑点
|
||||
|
||||
## Assets
|
||||
|
||||
- `assets/template.html` — HTML 报告模板（聊天气泡样式）
|
||||
|
||||
## Limitations
|
||||
|
||||
1. **仅支持 Windows**：win32gui 和 uiautomation 为 Windows 专用库
|
||||
2. **坐标敏感**：分辨率/DPI 变化需重新校准坐标；优先使用控件定位
|
||||
3. **OCR 准确率**：PaddleOCR 中文准确率约 95%，特殊符号/表情可能误识别
|
||||
4. **微信版本依赖**：微信 UI 更新可能导致控件结构变化，需适配
|
||||
5. **不可后台运行**：pyautogui 需要目标窗口在前台可见
|
||||
6. **安全限制**：不可操作需要管理员权限的 UAC 弹窗（需用户手动确认）
|
||||
@@ -0,0 +1,83 @@
|
||||
<!DOCTYPE html>
|
||||
<html lang="zh-CN">
|
||||
<head>
|
||||
<meta charset="UTF-8">
|
||||
<meta name="viewport" content="width=device-width, initial-scale=1.0">
|
||||
<title>{{TITLE}}</title>
|
||||
<style>
|
||||
* { margin: 0; padding: 0; box-sizing: border-box; }
|
||||
body {
|
||||
font-family: -apple-system, BlinkMacSystemFont, "Segoe UI",
|
||||
"PingFang SC", "Microsoft YaHei", sans-serif;
|
||||
background: #EDEDED;
|
||||
color: #333;
|
||||
line-height: 1.6;
|
||||
}
|
||||
.header {
|
||||
background: #ededed;
|
||||
border-bottom: 1px solid #dcdcdc;
|
||||
padding: 12px 20px;
|
||||
position: sticky;
|
||||
top: 0;
|
||||
z-index: 100;
|
||||
}
|
||||
.chat-container {
|
||||
max-width: 800px;
|
||||
margin: 0 auto;
|
||||
padding: 16px;
|
||||
}
|
||||
.message-row {
|
||||
display: flex;
|
||||
margin-bottom: 16px;
|
||||
gap: 8px;
|
||||
}
|
||||
.message-row.self { flex-direction: row-reverse; }
|
||||
.avatar {
|
||||
width: 40px; height: 40px;
|
||||
border-radius: 4px;
|
||||
display: flex; align-items: center; justify-content: center;
|
||||
color: white; font-size: 16px; font-weight: 600;
|
||||
flex-shrink: 0;
|
||||
}
|
||||
.bubble {
|
||||
padding: 10px 14px;
|
||||
border-radius: 4px;
|
||||
font-size: 15px;
|
||||
word-wrap: break-word;
|
||||
max-width: 60%;
|
||||
}
|
||||
.message-row.other .bubble {
|
||||
background: white;
|
||||
border: 1px solid #e5e5e5;
|
||||
}
|
||||
.message-row.self .bubble {
|
||||
background: #95EC69;
|
||||
}
|
||||
.time-separator {
|
||||
text-align: center;
|
||||
margin: 20px 0;
|
||||
color: #999;
|
||||
font-size: 12px;
|
||||
}
|
||||
.footer {
|
||||
text-align: center;
|
||||
padding: 24px;
|
||||
color: #999;
|
||||
font-size: 12px;
|
||||
}
|
||||
</style>
|
||||
</head>
|
||||
<body>
|
||||
<div class="header">
|
||||
<strong>{{TITLE}}</strong>
|
||||
<span style="float:right;color:#888;font-size:12px;">{{META}}</span>
|
||||
</div>
|
||||
<div class="chat-container">
|
||||
<!-- 消息列表 -->
|
||||
<!-- {{MESSAGE_ITEMS}} -->
|
||||
</div>
|
||||
<div class="footer">
|
||||
Generated by Desktop Automation Skill
|
||||
</div>
|
||||
</body>
|
||||
</html>
|
||||
@@ -0,0 +1,376 @@
|
||||
# 桌面自动化 API 参考与指南
|
||||
|
||||
## 目录
|
||||
1. [坐标系统与 DPI 适配](#1-坐标系统与-dpi-适配)
|
||||
2. [窗口管理 API](#2-窗口管理-api)
|
||||
3. [鼠标操作 API](#3-鼠标操作-api)
|
||||
4. [键盘操作 API](#4-键盘操作-api)
|
||||
5. [截图与图像匹配 API](#5-截图与图像匹配-api)
|
||||
6. [UI 控件定位 API](#6-ui-控件定位-api)
|
||||
7. [常见自动化模式](#7-常见自动化模式)
|
||||
8. [故障排除](#8-故障排除)
|
||||
|
||||
---
|
||||
|
||||
## 1. 坐标系统与 DPI 适配
|
||||
|
||||
### 坐标原点
|
||||
|
||||
pyautogui 使用屏幕左上角为原点 (0, 0)，X 轴向右递增，Y 轴向下递增。
|
||||
|
||||
```
|
||||
(0,0) ──────────── X+ ────────→
|
||||
│
|
||||
│
|
||||
Y+
|
||||
│
|
||||
↓
|
||||
```
|
||||
|
||||
### DPI 缩放问题
|
||||
|
||||
Windows 高 DPI（150%/200%）下，pyautogui 报告的分辨率可能与实际不同：
|
||||
|
||||
| DPI 缩放 | 物理分辨率 | pyautogui 报告 | 偏移 |
|
||||
|----------|-----------|---------------|------|
|
||||
| 100% | 1920x1080 | 1920x1080 | 无 |
|
||||
| 150% | 1920x1080 | 1280x720 | 有 |
|
||||
| 200% | 3840x2160 | 1920x1080 | 有 |
|
||||
|
||||
### 适配方案
|
||||
|
||||
```python
|
||||
# 方案1: 使用 pyautogui.size() 获取逻辑分辨率，按比例计算坐标
|
||||
screen_w, screen_h = pyautogui.size()
|
||||
center_x = screen_w // 2
|
||||
center_y = screen_h // 2
|
||||
|
||||
# 方案2: 使用 uiautomation 控件定位（不受 DPI 影响）
|
||||
import uiautomation as auto
|
||||
btn = auto.ButtonControl(Name="确定")
|
||||
btn.Click() # 自动计算物理坐标
|
||||
|
||||
# 方案3: 使用图像匹配定位（不受 DPI 影响）
|
||||
pos = pyautogui.locateCenterOnScreen("button.png", confidence=0.9)
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 2. 窗口管理 API
|
||||
|
||||
### find_window(title_keyword)
|
||||
|
||||
通过标题模糊匹配查找窗口。
|
||||
|
||||
```python
|
||||
# 精确匹配
|
||||
info = auto.find_window("微信")
|
||||
|
||||
# 部分匹配
|
||||
info = auto.find_window("记事本") # 匹配 "无标题 - 记事本"
|
||||
```
|
||||
|
||||
### activate_window(title_keyword)
|
||||
|
||||
激活窗口并置于最前。内部处理：
|
||||
- 最小化状态先恢复
|
||||
- 使用 SetForegroundWindow
|
||||
- 失败时用 ALT+TAB 技巧绕过 Windows 前台锁定
|
||||
|
||||
### get_window_rect(title_keyword)
|
||||
|
||||
返回 `(left, top, right, bottom)` 元组。
|
||||
|
||||
### move_window(title_keyword, x, y, width, height)
|
||||
|
||||
移动并调整窗口大小。固定窗口位置和大小可确保坐标稳定：
|
||||
|
||||
```python
|
||||
# 将微信固定到屏幕左侧，大小 800x600
|
||||
auto.move_window("微信", 0, 0, 800, 600)
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 3. 鼠标操作 API
|
||||
|
||||
### click(x, y, button, clicks, interval, duration)
|
||||
|
||||
| 参数 | 类型 | 默认值 | 说明 |
|
||||
|------|------|--------|------|
|
||||
| x, y | int | None | 目标坐标，None=当前位置 |
|
||||
| button | str | "left" | "left"/"right"/"middle" |
|
||||
| clicks | int | 1 | 点击次数 |
|
||||
| interval | float | 0.0 | 多次点击间隔（秒）|
|
||||
| duration | float | 0.0 | 移动耗时（秒），0=瞬移 |
|
||||
|
||||
### drag(start, end, duration, button)
|
||||
|
||||
拖拽操作，模拟鼠标按下→移动→释放。
|
||||
|
||||
```python
|
||||
# 拖拽文件
|
||||
auto.drag((100, 200), (500, 600), duration=0.8)
|
||||
```
|
||||
|
||||
### scroll(clicks, x, y)
|
||||
|
||||
| clicks 值 | 方向 |
|
||||
|-----------|------|
|
||||
| 正数 | 向上 |
|
||||
| 负数 | 向下 |
|
||||
|
||||
微信聊天滚动通常每次 -3 到 -5 格。
|
||||
|
||||
---
|
||||
|
||||
## 4. 键盘操作 API
|
||||
|
||||
### type_text(text)
|
||||
|
||||
**重要**：pyautogui.typewrite() 不支持中文输入。此方法通过剪贴板粘贴实现：
|
||||
|
||||
1. 判断文本是否纯 ASCII
|
||||
2. ASCII: 直接用 typewrite（支持 interval）
|
||||
3. 非 ASCII: pyperclip.copy() + Ctrl+V
|
||||
|
||||
### press_key(key)
|
||||
|
||||
支持组合键，用 `+` 连接：
|
||||
|
||||
```python
|
||||
auto.press_key("enter") # 回车
|
||||
auto.press_key("ctrl+a") # 全选
|
||||
auto.press_key("ctrl+shift+n") # 新建文件夹
|
||||
auto.press_key("alt+f4") # 关闭窗口
|
||||
```
|
||||
|
||||
### 常用键名速查
|
||||
|
||||
| 键名 | 说明 | 键名 | 说明 |
|
||||
|------|------|------|------|
|
||||
| enter | 回车 | esc | ESC |
|
||||
| tab | Tab | space | 空格 |
|
||||
| backspace | 退格 | delete | 删除 |
|
||||
| up/down/left/right | 方向键 | home/end | Home/End |
|
||||
| pageup/pagedown | 翻页 | f1-f12 | 功能键 |
|
||||
| ctrl/alt/shift | 修饰键 | win | Windows键 |
|
||||
|
||||
---
|
||||
|
||||
## 5. 截图与图像匹配 API
|
||||
|
||||
### screenshot(save_path)
|
||||
|
||||
全屏截图，返回 PIL.Image 对象。
|
||||
|
||||
### screenshot_region(rect, save_path)
|
||||
|
||||
区域截图。rect 支持两种格式：
|
||||
- `(left, top, right, bottom)` — 两个坐标点
|
||||
- `(left, top, width, height)` — 坐标+尺寸
|
||||
|
||||
### screenshot_window(title_keyword, save_path)
|
||||
|
||||
对指定窗口截图，自动计算窗口区域。
|
||||
|
||||
### locate_on_screen(image_path, confidence)
|
||||
|
||||
使用 OpenCV 模板匹配查找屏幕上的图片。
|
||||
|
||||
**confidence 建议值：**
|
||||
- 精确匹配（按钮、图标）: 0.95
|
||||
- 模糊匹配（动态内容）: 0.80
|
||||
- 最低阈值: 0.70
|
||||
|
||||
### wait_for_image(image_path, timeout, interval)
|
||||
|
||||
轮询等待图片出现，超时返回 None。
|
||||
|
||||
### click_image(image_path)
|
||||
|
||||
wait_for_image + click 的组合，最常用的自动化模式。
|
||||
|
||||
---
|
||||
|
||||
## 6. UI 控件定位 API
|
||||
|
||||
### 为什么优先使用控件定位？
|
||||
|
||||
| 方式 | 优点 | 缺点 |
|
||||
|------|------|------|
|
||||
| 坐标点击 | 简单直接 | 受分辨率/DPI/窗口位置影响 |
|
||||
| 图像匹配 | 不受坐标影响 | 受主题/缩放/遮挡影响 |
|
||||
| **控件定位** | **最稳定，不受视觉变化影响** | 仅支持标准 Windows 控件 |
|
||||
|
||||
### find_control(window_title, control_type, name, auto_id)
|
||||
|
||||
| 参数 | 说明 | 示例 |
|
||||
|------|------|------|
|
||||
| window_title | 窗口标题 | "记事本" |
|
||||
| control_type | 控件类型 | "Button", "Edit", "Text" |
|
||||
| name | 控件名称 | "保存" |
|
||||
| auto_id | AutomationId | "SaveButton" |
|
||||
|
||||
### 常见控件类型
|
||||
|
||||
| control_type | 对应控件 |
|
||||
|--------------|---------|
|
||||
| Button | 按钮 |
|
||||
| Edit | 文本输入框 |
|
||||
| Text | 静态文本 |
|
||||
| ComboBox | 下拉框 |
|
||||
| CheckBox | 复选框 |
|
||||
| RadioButton | 单选按钮 |
|
||||
| ListControl | 列表 |
|
||||
| ListItem | 列表项 |
|
||||
| TreeControl | 树形控件 |
|
||||
| TreeItem | 树节点 |
|
||||
| TabControl | 选项卡 |
|
||||
| MenuItem | 菜单项 |
|
||||
|ToolBar | 工具栏 |
|
||||
|
||||
### 控件操作方法
|
||||
|
||||
```python
|
||||
ctrl = auto.find_control("窗口标题", control_type="Button", name="确定")
|
||||
if ctrl:
|
||||
ctrl.Click() # 点击
|
||||
ctrl.DoubleClick() # 双击
|
||||
ctrl.RightClick() # 右键
|
||||
print(ctrl.Name) # 获取名称
|
||||
print(ctrl.IsEnabled) # 是否可用
|
||||
print(ctrl.IsOffscreen) # 是否在屏幕外
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 7. 常见自动化模式
|
||||
|
||||
### 模式1: 点击→等待→输入
|
||||
|
||||
```python
|
||||
# 点击输入框
|
||||
auto.click(100, 200)
|
||||
# 等待输入框获得焦点
|
||||
auto.wait(0.3)
|
||||
# 输入文本
|
||||
auto.type_text("Hello World")
|
||||
# 回车提交
|
||||
auto.press_key("enter")
|
||||
```
|
||||
|
||||
### 模式2: 搜索→选择→确认
|
||||
|
||||
```python
|
||||
# 打开搜索 (Ctrl+F)
|
||||
auto.hotkey("ctrl", "f")
|
||||
auto.wait(0.5)
|
||||
# 输入搜索词
|
||||
auto.type_text("关键词")
|
||||
auto.wait(1.0)
|
||||
# 回车选择第一个结果
|
||||
auto.press_key("enter")
|
||||
```
|
||||
|
||||
### 模式3: 循环滚动截图
|
||||
|
||||
```python
|
||||
for i in range(10):
|
||||
# 截图
|
||||
auto.screenshot(f"capture_{i:03d}.png")
|
||||
# 向下滚动
|
||||
auto.scroll(-3)
|
||||
# 等待内容加载
|
||||
auto.wait(1.5)
|
||||
```
|
||||
|
||||
### 模式4: 图像匹配点击
|
||||
|
||||
```python
|
||||
# 截取目标按钮的图片保存为 button.png
|
||||
# 然后在脚本中用图像匹配定位
|
||||
if auto.click_image("send_button.png", timeout=5):
|
||||
print("发送按钮已点击")
|
||||
else:
|
||||
print("未找到发送按钮")
|
||||
```
|
||||
|
||||
### 模式5: 窗口操作序列
|
||||
|
||||
```python
|
||||
# 1. 激活目标窗口
|
||||
auto.activate_window("目标应用")
|
||||
# 2. 固定窗口位置和大小（确保坐标稳定）
|
||||
auto.move_window("目标应用", 100, 100, 1200, 800)
|
||||
# 3. 等待窗口就绪
|
||||
auto.wait(1.0)
|
||||
# 4. 开始操作
|
||||
auto.click(300, 150) # 相对于固定后的窗口位置
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 8. 故障排除
|
||||
|
||||
### 问题: click() 点击位置偏移
|
||||
|
||||
**原因**: DPI 缩放导致坐标映射错误。
|
||||
|
||||
**解决**:
|
||||
```python
|
||||
# 在脚本开头添加 DPI 感知
|
||||
import ctypes
|
||||
ctypes.windll.shcore.SetProcessDpiAwareness(2)
|
||||
```
|
||||
|
||||
### 问题: type_text() 中文无法输入
|
||||
|
||||
**原因**: pyperclip 剪贴板操作可能被其他程序干扰。
|
||||
|
||||
**解决**:
|
||||
```python
|
||||
# 方案1: 增加延迟
|
||||
pyperclip.copy("中文")
|
||||
time.sleep(0.2)
|
||||
pyautogui.hotkey("ctrl", "v")
|
||||
|
||||
# 方案2: 使用 uiautomation 的控件输入
|
||||
edit_ctrl.SendKeys("中文")
|
||||
```
|
||||
|
||||
### 问题: locate_on_screen() 找不到图片
|
||||
|
||||
**原因**: 置信度过高 / 图片尺寸不匹配 / 窗口被遮挡。
|
||||
|
||||
**解决**:
|
||||
```python
|
||||
# 降低置信度
|
||||
pos = auto.locate_on_screen("button.png", confidence=0.7)
|
||||
|
||||
# 限定搜索区域提高速度和准确率
|
||||
pos = auto.locate_on_screen("button.png",
|
||||
confidence=0.85,
|
||||
region=(100, 100, 800, 600))
|
||||
```
|
||||
|
||||
### 问题: SetForegroundWindow 失败
|
||||
|
||||
**原因**: Windows 不允许后台进程直接抢焦点。
|
||||
|
||||
**解决**: activate_window() 内部已处理，如果仍然失败：
|
||||
```python
|
||||
# 先点击任务栏图标
|
||||
auto.click(taskbar_x, taskbar_y)
|
||||
# 或使用 ALT+TAB
|
||||
auto.hotkey("alt", "tab")
|
||||
```
|
||||
|
||||
### 问题: pyautogui 报错 FailSafeException
|
||||
|
||||
**原因**: 鼠标被快速移到了屏幕左上角 (0,0)，触发了安全保护。
|
||||
|
||||
**解决**: 这是设计中的安全机制。如果要禁用（不推荐）：
|
||||
```python
|
||||
pyautogui.FAILSAFE = False
|
||||
```
|
||||
@@ -0,0 +1,296 @@
|
||||
# 微信 PC 版 UI 结构与自动化模式
|
||||
|
||||
## 目录
|
||||
1. [微信窗口结构](#1-微信窗口结构)
|
||||
2. [聊天区域定位](#2-聊天区域定位)
|
||||
3. [搜索联系人流程](#3-搜索联系人流程)
|
||||
4. [消息滚动与截图策略](#4-消息滚动与截图策略)
|
||||
5. [OCR 消息解析规则](#5-ocr-消息解析规则)
|
||||
6. [已知坑点与限制](#6-已知坑点与限制)
|
||||
|
||||
---
|
||||
|
||||
## 1. 微信窗口结构
|
||||
|
||||
### 窗口类名
|
||||
|
||||
```
|
||||
WeChatMainWndForPC
|
||||
```
|
||||
|
||||
### 窗口布局（3.x 版本）
|
||||
|
||||
```
|
||||
┌─────────────────────────────────────────────────┐
|
||||
│ 标题栏 (搜索框 + 功能按钮) │ ~8% 高度
|
||||
├──────┬──────────────┬────────────────────────────┤
|
||||
│ 导航 │ 聊天列表 │ 聊天内容区 │
|
||||
│ 栏 │ │ │
|
||||
│ │ - 联系人A │ ┌──────────────────┐ │
|
||||
│ 微信 │ - 联系人B │ │ 消息1 │ │
|
||||
│ 通讯 │ - 群聊C │ │ 消息2 │ │
|
||||
│ 录 │ - ... │ │ 消息3 │ │
|
||||
│ │ │ └──────────────────┘ │
|
||||
│ 收藏 │ │ │
|
||||
│ 朋友圈│ │ ┌──────────────────┐ │
|
||||
│ │ │ │ 输入框 │ │ ~12% 高度
|
||||
│ 设置 │ │ └──────────────────┘ │
|
||||
├──────┴──────────────┴────────────────────────────┤
|
||||
│ 状态栏 │
|
||||
└─────────────────────────────────────────────────┘
|
||||
| ~6% | ~24% | ~70% |
|
||||
```
|
||||
|
||||
### 各区域占比（相对于窗口）
|
||||
|
||||
| 区域 | left | top | right | bottom |
|
||||
|------|------|-----|-------|--------|
|
||||
| 导航栏 | 0% | 8% | 6% | 92% |
|
||||
| 聊天列表 | 6% | 8% | 30% | 92% |
|
||||
| 聊天内容 | 30% | 8% | 98% | 88% |
|
||||
| 输入框 | 30% | 88% | 98% | 100% |
|
||||
|
||||
---
|
||||
|
||||
## 2. 聊天区域定位
|
||||
|
||||
### 计算聊天区域绝对坐标
|
||||
|
||||
```python
|
||||
# 获取微信窗口位置和大小
|
||||
win_info = auto.find_window("微信")
|
||||
left, top, right, bottom = win_info.rect
|
||||
width = right - left
|
||||
height = bottom - top
|
||||
|
||||
# 计算聊天内容区域
|
||||
chat_left = int(left + width * 0.30)
|
||||
chat_top = int(top + height * 0.08)
|
||||
chat_right = int(left + width * 0.98)
|
||||
chat_bottom = int(top + height * 0.88)
|
||||
|
||||
chat_rect = (chat_left, chat_top,
|
||||
chat_right - chat_left,
|
||||
chat_bottom - chat_top)
|
||||
```
|
||||
|
||||
### 固定窗口大小
|
||||
|
||||
为确保坐标稳定，建议固定微信窗口大小：
|
||||
|
||||
```python
|
||||
# 固定为 1000x700，放在屏幕左侧
|
||||
auto.move_window("微信", 0, 0, 1000, 700)
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 3. 搜索联系人流程
|
||||
|
||||
### 使用快捷键搜索
|
||||
|
||||
微信 PC 版支持 `Ctrl+F` 快捷键打开搜索框：
|
||||
|
||||
```python
|
||||
# 1. 激活微信窗口
|
||||
auto.activate_window("微信")
|
||||
auto.wait(0.5)
|
||||
|
||||
# 2. Ctrl+F 打开搜索
|
||||
auto.hotkey("ctrl", "f")
|
||||
auto.wait(0.8) # 等待搜索框动画完成
|
||||
|
||||
# 3. 输入联系人名称（剪贴板方式支持中文）
|
||||
auto.type_text("张三")
|
||||
auto.wait(1.0) # 等待搜索结果
|
||||
|
||||
# 4. Enter 打开第一个搜索结果
|
||||
auto.press_key("enter")
|
||||
auto.wait(1.5) # 等待聊天窗口加载
|
||||
```
|
||||
|
||||
### 搜索框定位（控件方式）
|
||||
|
||||
如果快捷键不生效，可通过控件定位搜索框：
|
||||
|
||||
```python
|
||||
# 使用 uiautomation 查找搜索框
|
||||
search_box = auto.find_control(
|
||||
"微信",
|
||||
control_type="Edit",
|
||||
name="搜索"
|
||||
)
|
||||
if search_box:
|
||||
search_box.Click()
|
||||
auto.wait(0.3)
|
||||
auto.type_text("张三")
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 4. 消息滚动与截图策略
|
||||
|
||||
### 滚动到顶部
|
||||
|
||||
```python
|
||||
# 点击聊天区域确保焦点在此
|
||||
auto.click(chat_center_x, chat_center_y)
|
||||
auto.wait(0.3)
|
||||
|
||||
# 持续向上滚动直到内容不再变化
|
||||
prev_hash = None
|
||||
for i in range(30): # 最多滚动30次
|
||||
auto.scroll(5) # 向上滚动5格
|
||||
auto.wait(0.5)
|
||||
|
||||
# 截图并比较
|
||||
img = auto.screenshot_region(chat_rect)
|
||||
current_hash = hashlib.md5(img.tobytes()).hexdigest()
|
||||
|
||||
if current_hash == prev_hash:
|
||||
print("已到达顶部")
|
||||
break
|
||||
prev_hash = current_hash
|
||||
```
|
||||
|
||||
### 向下滚动截图
|
||||
|
||||
```python
|
||||
messages = []
|
||||
seen_hashes = set()
|
||||
|
||||
for i in range(max_iterations):
|
||||
# 截图当前聊天区域
|
||||
img = auto.screenshot_region(chat_rect)
|
||||
img.save(f"screenshots/chat_{i:04d}.png")
|
||||
|
||||
# OCR 识别
|
||||
ocr_results = ocr.recognize(img)
|
||||
new_msgs = parse_messages(ocr_results)
|
||||
|
||||
# 去重
|
||||
for msg in new_msgs:
|
||||
h = hash_message(msg)
|
||||
if h not in seen_hashes:
|
||||
seen_hashes.add(h)
|
||||
messages.append(msg)
|
||||
|
||||
# 向下滚动
|
||||
auto.scroll(-3)
|
||||
auto.wait(1.5) # 等待新内容加载
|
||||
```
|
||||
|
||||
### 滚动参数建议
|
||||
|
||||
| 参数 | 建议值 | 说明 |
|
||||
|------|--------|------|
|
||||
| scroll clicks | -3 到 -5 | 每次滚动格数 |
|
||||
| scroll_pause | 1.0-2.0 秒 | 滚动间隔 |
|
||||
| max_no_new | 5 | 连续无新消息时停止 |
|
||||
|
||||
**注意**: 滚动过快会导致消息加载不完整，遗漏消息。
|
||||
|
||||
---
|
||||
|
||||
## 5. OCR 消息解析规则
|
||||
|
||||
### 微信消息布局特征
|
||||
|
||||
```
|
||||
┌────────────────────────────────┐
|
||||
│ 14:30 │ ← 时间戳（居中，独立行）
|
||||
├────────────────────────────────┤
|
||||
│ 张三 │ ← 发送者名称（左对齐）
|
||||
│ ┌──────────────┐ │
|
||||
│ │ 你好！ │ │ ← 消息气泡（白色=对方）
|
||||
│ └──────────────┘ │
|
||||
├────────────────────────────────┤
|
||||
│ ┌──────────────┐ │
|
||||
│ │ 嗯，收到了 │ │ ← 消息气泡（绿色=自己）
|
||||
│ └──────────────┘ │
|
||||
├────────────────────────────────┤
|
||||
│ 14:31 │ ← 时间戳
|
||||
├────────────────────────────────┤
|
||||
│ ┌────────┐ │
|
||||
│ │ [图片] │ │ ← 图片消息
|
||||
│ └────────┘ │
|
||||
└────────────────────────────────┘
|
||||
```
|
||||
|
||||
### 启发式解析规则
|
||||
|
||||
1. **时间戳识别**: 文本包含 `:` 且长度 < 20，含数字
|
||||
2. **发送者识别**: 短文本（≤10字符），不以标点开头/结尾
|
||||
3. **消息分隔**: 行间距 > 30 像素时判定为新消息
|
||||
4. **气泡位置**: 左侧=对方消息，右侧=自己消息
|
||||
|
||||
### 消息类型识别
|
||||
|
||||
| 类型 | OCR 特征 | msg_type |
|
||||
|------|---------|----------|
|
||||
| 文本消息 | 普通文字 | text |
|
||||
| 图片消息 | "[图片]" 或色块区域 | image |
|
||||
| 文件消息 | "[文件] 文件名" | file |
|
||||
| 语音消息 | "[语音] XX"" | voice |
|
||||
| 系统消息 | "XX 撤回了一条消息" 等 | system |
|
||||
| 链接卡片 | 标题+描述+来源 | link |
|
||||
|
||||
### 准确率说明
|
||||
|
||||
- 纯文本消息: ~90%
|
||||
- 时间戳: ~95%
|
||||
- 发送者名称: ~85%
|
||||
- 图片/文件消息: ~60%（需结合截图判断）
|
||||
- 复杂消息（链接卡片、小程序）: ~40%
|
||||
|
||||
---
|
||||
|
||||
## 6. 已知坑点与限制
|
||||
|
||||
### 6.1 微信版本差异
|
||||
|
||||
| 版本 | 搜索快捷键 | 窗口类名 | 备注 |
|
||||
|------|-----------|---------|------|
|
||||
| 3.9.x | Ctrl+F | WeChatMainWndForPC | 当前主流 |
|
||||
| 3.8.x | Ctrl+F | WeChatMainWndForPC | 兼容 |
|
||||
| 4.0+ | 未知 | 可能变更 | 需适配 |
|
||||
|
||||
### 6.2 高 DPI 问题
|
||||
|
||||
微信在 150% 缩放下：
|
||||
- 窗口实际大小与 `GetWindowRect` 返回值可能不一致
|
||||
- 截图区域计算需使用 `pyautogui.size()` 而非 `GetSystemMetrics`
|
||||
|
||||
### 6.3 消息加载机制
|
||||
|
||||
微信 PC 版的消息加载特性：
|
||||
- **虚拟滚动**: 只渲染可见区域的消息，滚动时动态加载
|
||||
- **懒加载图片**: 图片消息首次可见时才加载缩略图
|
||||
- **消息上限**: 单次加载最多约 100 条，需多次滚动
|
||||
|
||||
### 6.4 不可后台操作
|
||||
|
||||
pyautogui 需要目标窗口在前台可见。如果微信被其他窗口遮挡：
|
||||
- 截图会包含遮挡窗口的内容
|
||||
- 控件定位可能失败
|
||||
|
||||
**解决**: 操作前务必 `activate_window("微信")`
|
||||
|
||||
### 6.5 企业微信差异
|
||||
|
||||
企业微信（WeCom）的窗口类名不同：
|
||||
|
||||
```python
|
||||
# 企业微信
|
||||
WECHAT_CLASS_NAME = "WeWorkWindow"
|
||||
|
||||
# 搜索快捷键
|
||||
WECHAT_SEARCH_SHORTCUT = "ctrl+f" # 相同
|
||||
```
|
||||
|
||||
### 6.6 消息发送限制
|
||||
|
||||
此技能仅用于**读取**聊天记录，不推荐用于自动发送消息：
|
||||
- 微信可能检测自动化操作并限制账号
|
||||
- 误发消息风险高
|
||||
- 如需发送，务必添加 `auto.confirm()` 确认步骤
|
||||
@@ -0,0 +1,809 @@
|
||||
#!/usr/bin/env python3
|
||||
"""
|
||||
desktop_auto.py — Windows 桌面自动化核心引擎
|
||||
|
||||
提供窗口管理、鼠标键盘模拟、屏幕截图、OCR 识别、图像匹配等通用 API。
|
||||
基于 pyautogui + uiautomation + win32gui 组合方案。
|
||||
|
||||
依赖: pyautogui, uiautomation, Pillow, pygetwindow, pyperclip, opencv-python
|
||||
作者: Duckula (agent_created)
|
||||
"""
|
||||
|
||||
import os
|
||||
import sys
|
||||
import time
|
||||
import subprocess
|
||||
from typing import Optional, Tuple, List, Any
|
||||
from dataclasses import dataclass
|
||||
from pathlib import Path
|
||||
|
||||
# ============================================================
|
||||
# 依赖检测与导入（延迟加载，允许部分功能独立使用）
|
||||
# ============================================================
|
||||
|
||||
def _check_dependency(package: str) -> bool:
|
||||
"""检测 Python 包是否已安装"""
|
||||
try:
|
||||
__import__(package)
|
||||
return True
|
||||
except ImportError:
|
||||
return False
|
||||
|
||||
def _ensure_dependency(package: str, pip_name: str = None) -> Any:
|
||||
"""
|
||||
确保依赖已安装，未安装则提示用户。
|
||||
pip_name: pip 安装时使用的包名（与 import 名不同时指定）
|
||||
"""
|
||||
try:
|
||||
return __import__(package)
|
||||
except ImportError:
|
||||
pip_name = pip_name or package
|
||||
print(f"[WARN] 缺少依赖: {package}，正在安装 {pip_name} ...")
|
||||
subprocess.check_call([sys.executable, "-m", "pip", "install", pip_name])
|
||||
return __import__(package)
|
||||
|
||||
# 核心依赖（必须）
|
||||
pyautogui = _ensure_dependency("pyautogui")
|
||||
pyperclip = _ensure_dependency("pyperclip")
|
||||
PIL_Image = _ensure_dependency("PIL.Image", "Pillow")
|
||||
|
||||
# 可选依赖（按需加载）
|
||||
def _import_pygetwindow():
|
||||
"""延迟加载 pygetwindow（窗口管理）"""
|
||||
return _ensure_dependency("pygetwindow")
|
||||
|
||||
def _import_uiautomation():
|
||||
"""延迟加载 uiautomation（UI 控件定位）"""
|
||||
return _ensure_dependency("uiautomation")
|
||||
|
||||
def _import_win32gui():
|
||||
"""延迟加载 win32gui（Windows API）"""
|
||||
try:
|
||||
import win32gui
|
||||
import win32con
|
||||
return win32gui, win32con
|
||||
except ImportError:
|
||||
print("[WARN] 缺少 pywin32，正在安装...")
|
||||
subprocess.check_call([sys.executable, "-m", "pip", "install", "pywin32"])
|
||||
import win32gui
|
||||
import win32con
|
||||
return win32gui, win32con
|
||||
|
||||
|
||||
# ============================================================
|
||||
# 数据结构
|
||||
# ============================================================
|
||||
|
||||
@dataclass
|
||||
class WindowInfo:
|
||||
"""窗口信息"""
|
||||
hwnd: int # 窗口句柄
|
||||
title: str # 窗口标题
|
||||
class_name: str # 窗口类名
|
||||
rect: Tuple[int, int, int, int] # (left, top, right, bottom)
|
||||
is_visible: bool # 是否可见
|
||||
is_minimized: bool # 是否最小化
|
||||
|
||||
@property
|
||||
def width(self) -> int:
|
||||
return self.rect[2] - self.rect[0]
|
||||
|
||||
@property
|
||||
def height(self) -> int:
|
||||
return self.rect[3] - self.rect[1]
|
||||
|
||||
@property
|
||||
def center(self) -> Tuple[int, int]:
|
||||
"""窗口中心点坐标"""
|
||||
return (
|
||||
(self.rect[0] + self.rect[2]) // 2,
|
||||
(self.rect[1] + self.rect[3]) // 2,
|
||||
)
|
||||
|
||||
|
||||
@dataclass
|
||||
class MessageItem:
|
||||
"""结构化消息（用于聊天记录导出）"""
|
||||
sender: str # 发送者名称
|
||||
content: str # 消息文本内容
|
||||
timestamp: Optional[str] = None # 时间戳（如能从 OCR 识别）
|
||||
msg_type: str = "text" # 消息类型: text/image/file/system/unknown
|
||||
screenshot_path: Optional[str] = None # 对应截图文件路径
|
||||
raw_ocr: Optional[str] = None # OCR 原始识别文本
|
||||
|
||||
|
||||
# ============================================================
|
||||
# 核心自动化引擎
|
||||
# ============================================================
|
||||
|
||||
class DesktopAuto:
|
||||
"""
|
||||
Windows 桌面自动化引擎
|
||||
|
||||
封装 pyautogui + uiautomation + win32gui，提供统一的高层 API。
|
||||
所有方法均带有详细的参数注释和使用说明。
|
||||
"""
|
||||
|
||||
# pyautogui 安全设置
|
||||
FAILSAFE = True # 鼠标移到屏幕左上角(0,0)时抛异常终止
|
||||
PAUSE = 0.1 # 每个 pyautogui 操作后的默认暂停（秒）
|
||||
|
||||
def __init__(self, failsafe: bool = True, pause: float = 0.1):
|
||||
"""
|
||||
初始化自动化引擎。
|
||||
|
||||
Args:
|
||||
failsafe: 安全保护，True 时鼠标移至屏幕左上角可紧急终止
|
||||
pause: 每个操作之间的默认暂停时间（秒），防止操作过快
|
||||
"""
|
||||
pyautogui.FAILSAFE = failsafe
|
||||
pyautogui.PAUSE = pause
|
||||
self._screen_width, self._screen_height = pyautogui.size()
|
||||
print(f"[INFO] 屏幕分辨率: {self._screen_width}x{self._screen_height}")
|
||||
print(f"[INFO] 安全保护: {'ON' if failsafe else 'OFF'}")
|
||||
|
||||
# ========================================================
|
||||
# 窗口管理
|
||||
# ========================================================
|
||||
|
||||
def find_window(self, title_keyword: str) -> Optional[WindowInfo]:
|
||||
"""
|
||||
通过标题关键词查找窗口（模糊匹配）。
|
||||
|
||||
Args:
|
||||
title_keyword: 窗口标题中包含的关键词（如 "微信"、"记事本"）
|
||||
|
||||
Returns:
|
||||
WindowInfo 对象，未找到返回 None
|
||||
|
||||
Example:
|
||||
>>> auto.find_window("微信")
|
||||
WindowInfo(hwnd=0x12345, title='微信', class_name='WeChatMainWndForPC', ...)
|
||||
"""
|
||||
win32gui, win32con = _import_win32gui()
|
||||
result = []
|
||||
|
||||
def _enum_callback(hwnd, _):
|
||||
if win32gui.IsWindowVisible(hwnd):
|
||||
title = win32gui.GetWindowText(hwnd)
|
||||
if title_keyword.lower() in title.lower():
|
||||
rect = win32gui.GetWindowRect(hwnd)
|
||||
class_name = win32gui.GetClassName(hwnd)
|
||||
result.append(WindowInfo(
|
||||
hwnd=hwnd,
|
||||
title=title,
|
||||
class_name=class_name,
|
||||
rect=rect,
|
||||
is_visible=True,
|
||||
is_minimized=win32gui.IsIconic(hwnd),
|
||||
))
|
||||
|
||||
win32gui.EnumWindows(_enum_callback, None)
|
||||
|
||||
if result:
|
||||
info = result[0]
|
||||
print(f"[INFO] 找到窗口: '{info.title}' (hwnd={info.hwnd:#x})")
|
||||
return info
|
||||
print(f"[WARN] 未找到包含 '{title_keyword}' 的窗口")
|
||||
return None
|
||||
|
||||
def find_all_windows(self, title_keyword: str = "") -> List[WindowInfo]:
|
||||
"""
|
||||
查找所有匹配标题关键词的可见窗口。
|
||||
|
||||
Args:
|
||||
title_keyword: 窗口标题关键词，空字符串返回所有可见窗口
|
||||
|
||||
Returns:
|
||||
WindowInfo 列表
|
||||
"""
|
||||
win32gui, win32con = _import_win32gui()
|
||||
result = []
|
||||
|
||||
def _enum_callback(hwnd, _):
|
||||
if win32gui.IsWindowVisible(hwnd):
|
||||
title = win32gui.GetWindowText(hwnd)
|
||||
if not title_keyword or title_keyword.lower() in title.lower():
|
||||
rect = win32gui.GetWindowRect(hwnd)
|
||||
class_name = win32gui.GetClassName(hwnd)
|
||||
result.append(WindowInfo(
|
||||
hwnd=hwnd,
|
||||
title=title,
|
||||
class_name=class_name,
|
||||
rect=rect,
|
||||
is_visible=True,
|
||||
is_minimized=win32gui.IsIconic(hwnd),
|
||||
))
|
||||
|
||||
win32gui.EnumWindows(_enum_callback, None)
|
||||
return result
|
||||
|
||||
def activate_window(self, title_keyword: str) -> bool:
|
||||
"""
|
||||
激活窗口并置于最前。
|
||||
|
||||
处理最小化状态：先恢复再激活。
|
||||
使用 SetForegroundWindow，注意 Windows 不允许后台进程
|
||||
直接抢焦点，此方法在脚本自身有前台窗口时更可靠。
|
||||
|
||||
Args:
|
||||
title_keyword: 窗口标题关键词
|
||||
|
||||
Returns:
|
||||
True 表示成功激活，False 表示未找到窗口
|
||||
"""
|
||||
win32gui, win32con = _import_win32gui()
|
||||
info = self.find_window(title_keyword)
|
||||
if not info:
|
||||
return False
|
||||
|
||||
hwnd = info.hwnd
|
||||
|
||||
# 如果最小化了，先恢复
|
||||
if info.is_minimized:
|
||||
win32gui.ShowWindow(hwnd, win32con.SW_RESTORE)
|
||||
time.sleep(0.3)
|
||||
|
||||
# 尝试置顶
|
||||
try:
|
||||
win32gui.SetForegroundWindow(hwnd)
|
||||
except Exception:
|
||||
# SetForegroundWindow 失败的常见原因：
|
||||
# 当前进程不是前台进程。用 ALT+TAB 技巧绕过限制
|
||||
win32gui.ShowWindow(hwnd, win32con.SW_MINIMIZE)
|
||||
win32gui.ShowWindow(hwnd, win32con.SW_RESTORE)
|
||||
|
||||
time.sleep(0.5) # 等待窗口动画完成
|
||||
print(f"[INFO] 已激活窗口: '{info.title}'")
|
||||
return True
|
||||
|
||||
def get_window_rect(self, title_keyword: str) -> Optional[Tuple[int, int, int, int]]:
|
||||
"""
|
||||
获取窗口的位置和大小。
|
||||
|
||||
Returns:
|
||||
(left, top, right, bottom) 元组，未找到返回 None
|
||||
"""
|
||||
info = self.find_window(title_keyword)
|
||||
if info:
|
||||
return info.rect
|
||||
return None
|
||||
|
||||
def move_window(self, title_keyword: str, x: int, y: int,
|
||||
width: int = None, height: int = None) -> bool:
|
||||
"""
|
||||
移动并调整窗口大小。
|
||||
|
||||
Args:
|
||||
title_keyword: 窗口标题关键词
|
||||
x, y: 新的左上角坐标
|
||||
width, height: 新的宽高，None 表示保持原大小
|
||||
|
||||
Returns:
|
||||
True 成功，False 失败
|
||||
"""
|
||||
win32gui, win32con = _import_win32gui()
|
||||
info = self.find_window(title_keyword)
|
||||
if not info:
|
||||
return False
|
||||
|
||||
w = width if width else info.width
|
||||
h = height if height else info.height
|
||||
win32gui.MoveWindow(info.hwnd, x, y, w, h, True)
|
||||
print(f"[INFO] 窗口移动至 ({x}, {y})，大小 {w}x{h}")
|
||||
return True
|
||||
|
||||
# ========================================================
|
||||
# 鼠标操作
|
||||
# ========================================================
|
||||
|
||||
def click(self, x: int = None, y: int = None,
|
||||
button: str = "left", clicks: int = 1,
|
||||
interval: float = 0.0, duration: float = 0.0):
|
||||
"""
|
||||
在指定坐标点击鼠标。
|
||||
|
||||
Args:
|
||||
x, y: 目标坐标，None 表示在当前位置点击
|
||||
button: "left" / "right" / "middle"
|
||||
clicks: 点击次数（2=双击）
|
||||
interval: 多次点击之间的间隔（秒）
|
||||
duration: 鼠标移动到目标的耗时（秒），0=瞬间移动
|
||||
|
||||
Example:
|
||||
>>> auto.click(500, 300) # 左键单击
|
||||
>>> auto.click(500, 300, clicks=2) # 左键双击
|
||||
>>> auto.click(500, 300, button="right") # 右键单击
|
||||
"""
|
||||
pyautogui.click(x, y, clicks=clicks, interval=interval,
|
||||
button=button, duration=duration)
|
||||
|
||||
def click_relative(self, dx: int, dy: int, button: str = "left"):
|
||||
"""
|
||||
相对当前位置移动并点击。
|
||||
|
||||
Args:
|
||||
dx, dy: 相对偏移量（像素）
|
||||
button: 鼠标按钮
|
||||
"""
|
||||
current = pyautogui.position()
|
||||
pyautogui.click(current.x + dx, current.y + dy, button=button)
|
||||
|
||||
def double_click(self, x: int = None, y: int = None, duration: float = 0.0):
|
||||
"""双击"""
|
||||
pyautogui.doubleClick(x, y, duration=duration)
|
||||
|
||||
def right_click(self, x: int = None, y: int = None, duration: float = 0.0):
|
||||
"""右键单击"""
|
||||
pyautogui.rightClick(x, y, duration=duration)
|
||||
|
||||
def move_to(self, x: int, y: int, duration: float = 0.3):
|
||||
"""
|
||||
移动鼠标到指定坐标（不点击）。
|
||||
|
||||
Args:
|
||||
x, y: 目标坐标
|
||||
duration: 移动耗时（秒），模拟人类移动轨迹
|
||||
"""
|
||||
pyautogui.moveTo(x, y, duration=duration)
|
||||
|
||||
def drag(self, start: Tuple[int, int], end: Tuple[int, int],
|
||||
duration: float = 0.5, button: str = "left"):
|
||||
"""
|
||||
从 start 点拖拽到 end 点。
|
||||
|
||||
Args:
|
||||
start: (x, y) 起始坐标
|
||||
end: (x, y) 终点坐标
|
||||
duration: 拖拽耗时
|
||||
button: 按住的鼠标按钮
|
||||
|
||||
Example:
|
||||
>>> auto.drag((100, 200), (100, 400), duration=0.8)
|
||||
"""
|
||||
pyautogui.moveTo(start[0], start[1], duration=0.2)
|
||||
pyautogui.dragTo(end[0], end[1], duration=duration, button=button)
|
||||
|
||||
def scroll(self, clicks: int, x: int = None, y: int = None):
|
||||
"""
|
||||
滚动鼠标滚轮。
|
||||
|
||||
Args:
|
||||
clicks: 滚动格数，正数=向上，负数=向下
|
||||
x, y: 在指定坐标滚动，None=当前位置
|
||||
|
||||
Example:
|
||||
>>> auto.scroll(-3) # 向下滚动3格
|
||||
>>> auto.scroll(5) # 向上滚动5格
|
||||
"""
|
||||
pyautogui.scroll(clicks, x=x, y=y)
|
||||
|
||||
def get_mouse_pos(self) -> Tuple[int, int]:
|
||||
"""获取当前鼠标坐标"""
|
||||
pos = pyautogui.position()
|
||||
return (pos.x, pos.y)
|
||||
|
||||
# ========================================================
|
||||
# 键盘操作
|
||||
# ========================================================
|
||||
|
||||
def type_text(self, text: str, interval: float = 0.0):
|
||||
"""
|
||||
输入文本。
|
||||
|
||||
对于中文等非 ASCII 字符，pyautogui.typewrite 无法直接输入，
|
||||
此方法通过剪贴板粘贴实现，支持任意语言。
|
||||
|
||||
Args:
|
||||
text: 要输入的文本
|
||||
interval: 每个字符之间的间隔（仅对 ASCII 有效）
|
||||
|
||||
Example:
|
||||
>>> auto.type_text("你好世界")
|
||||
>>> auto.type_text("Hello World", interval=0.05)
|
||||
"""
|
||||
# 判断是否包含非 ASCII 字符
|
||||
if text.isascii():
|
||||
pyautogui.typewrite(text, interval=interval)
|
||||
else:
|
||||
# 中文/Unicode 通过剪贴板输入
|
||||
pyperclip.copy(text)
|
||||
pyautogui.hotkey("ctrl", "v")
|
||||
time.sleep(0.1)
|
||||
|
||||
def press_key(self, key: str, presses: int = 1, interval: float = 0.0):
|
||||
"""
|
||||
按键。
|
||||
|
||||
Args:
|
||||
key: 键名，如 "enter", "esc", "tab", "space", "backspace",
|
||||
"ctrl+shift+esc"（组合键用+连接）
|
||||
presses: 按键次数
|
||||
interval: 多次按键之间的间隔
|
||||
|
||||
Example:
|
||||
>>> auto.press_key("enter")
|
||||
>>> auto.press_key("ctrl+a") # 全选
|
||||
>>> auto.press_key("ctrl+shift+n") # 新建文件夹
|
||||
"""
|
||||
if "+" in key:
|
||||
# 组合键
|
||||
keys = [k.strip() for k in key.split("+")]
|
||||
pyautogui.hotkey(*keys)
|
||||
else:
|
||||
pyautogui.press(key, presses=presses, interval=interval)
|
||||
|
||||
def hotkey(self, *keys):
|
||||
"""
|
||||
同时按下多个键（组合键）。
|
||||
|
||||
Args:
|
||||
keys: 要同时按下的键序列
|
||||
|
||||
Example:
|
||||
>>> auto.hotkey("ctrl", "c") # 复制
|
||||
>>> auto.hotkey("ctrl", "shift", "esc") # 任务管理器
|
||||
"""
|
||||
pyautogui.hotkey(*keys)
|
||||
|
||||
def hold_key(self, key: str, duration: float = 1.0):
|
||||
"""
|
||||
按住某个键一段时间后释放。
|
||||
|
||||
Args:
|
||||
key: 键名
|
||||
duration: 按住时长（秒）
|
||||
"""
|
||||
pyautogui.keyDown(key)
|
||||
time.sleep(duration)
|
||||
pyautogui.keyUp(key)
|
||||
|
||||
# ========================================================
|
||||
# 屏幕截图
|
||||
# ========================================================
|
||||
|
||||
def screenshot(self, save_path: str = None) -> Any:
|
||||
"""
|
||||
全屏截图。
|
||||
|
||||
Args:
|
||||
save_path: 保存路径，None 则只返回 Image 对象不保存
|
||||
|
||||
Returns:
|
||||
PIL.Image 对象
|
||||
"""
|
||||
img = pyautogui.screenshot()
|
||||
if save_path:
|
||||
img.save(save_path)
|
||||
print(f"[INFO] 截图已保存: {save_path}")
|
||||
return img
|
||||
|
||||
def screenshot_region(self, rect: Tuple[int, int, int, int],
|
||||
save_path: str = None) -> Any:
|
||||
"""
|
||||
区域截图。
|
||||
|
||||
Args:
|
||||
rect: (left, top, right, bottom) 或 (left, top, width, height)
|
||||
save_path: 保存路径
|
||||
|
||||
Returns:
|
||||
PIL.Image 对象
|
||||
"""
|
||||
# 统一为 (left, top, width, height) 格式
|
||||
if len(rect) == 4:
|
||||
left, top, x2, y2 = rect
|
||||
# 如果是 (left, top, right, bottom) 格式
|
||||
if x2 > left and y2 > top and (x2 - left) < self._screen_width:
|
||||
width = x2 - left
|
||||
height = y2 - top
|
||||
else:
|
||||
width = x2
|
||||
height = y2
|
||||
else:
|
||||
raise ValueError("rect 必须是4元素元组")
|
||||
|
||||
img = pyautogui.screenshot(region=(left, top, width, height))
|
||||
if save_path:
|
||||
img.save(save_path)
|
||||
print(f"[INFO] 区域截图已保存: {save_path}")
|
||||
return img
|
||||
|
||||
def screenshot_window(self, title_keyword: str,
|
||||
save_path: str = None) -> Any:
|
||||
"""
|
||||
对指定窗口截图（截取窗口可见区域）。
|
||||
|
||||
Args:
|
||||
title_keyword: 窗口标题关键词
|
||||
save_path: 保存路径
|
||||
|
||||
Returns:
|
||||
PIL.Image 对象，未找到窗口返回 None
|
||||
"""
|
||||
info = self.find_window(title_keyword)
|
||||
if not info:
|
||||
return None
|
||||
|
||||
left, top, right, bottom = info.rect
|
||||
img = pyautogui.screenshot(region=(left, top,
|
||||
right - left, bottom - top))
|
||||
if save_path:
|
||||
img.save(save_path)
|
||||
print(f"[INFO] 窗口截图已保存: {save_path}")
|
||||
return img
|
||||
|
||||
# ========================================================
|
||||
# 图像匹配定位
|
||||
# ========================================================
|
||||
|
||||
def locate_on_screen(self, image_path: str,
|
||||
confidence: float = 0.9,
|
||||
region: Tuple[int, int, int, int] = None) -> Optional[Tuple[int, int]]:
|
||||
"""
|
||||
在屏幕上查找指定图片的位置。
|
||||
|
||||
使用 OpenCV 模板匹配，返回匹配中心的坐标。
|
||||
需要安装 opencv-python。
|
||||
|
||||
Args:
|
||||
image_path: 要查找的图片文件路径
|
||||
confidence: 匹配置信度 (0-1)，越高越严格
|
||||
region: 限定搜索区域 (left, top, width, height)
|
||||
|
||||
Returns:
|
||||
(x, y) 匹配中心坐标，未找到返回 None
|
||||
|
||||
Example:
|
||||
>>> pos = auto.locate_on_screen("send_button.png", confidence=0.8)
|
||||
>>> if pos:
|
||||
... auto.click(pos[0], pos[1])
|
||||
"""
|
||||
try:
|
||||
location = pyautogui.locateCenterOnScreen(
|
||||
image_path, confidence=confidence, region=region
|
||||
)
|
||||
if location:
|
||||
return (location.x, location.y)
|
||||
except pyautogui.ImageNotFoundException:
|
||||
pass
|
||||
except Exception as e:
|
||||
# opencv 未安装时的降级处理
|
||||
if "cv2" in str(e):
|
||||
print("[ERROR] 需要 opencv-python: pip install opencv-python")
|
||||
else:
|
||||
raise
|
||||
return None
|
||||
|
||||
def locate_all_on_screen(self, image_path: str,
|
||||
confidence: float = 0.9) -> List[Tuple[int, int]]:
|
||||
"""
|
||||
查找屏幕上所有匹配位置。
|
||||
|
||||
Returns:
|
||||
[(x, y), ...] 匹配中心坐标列表
|
||||
"""
|
||||
results = []
|
||||
try:
|
||||
for loc in pyautogui.locateAllOnScreen(image_path, confidence=confidence):
|
||||
results.append((loc.left + loc.width // 2,
|
||||
loc.top + loc.height // 2))
|
||||
except pyautogui.ImageNotFoundException:
|
||||
pass
|
||||
return results
|
||||
|
||||
def wait_for_image(self, image_path: str, timeout: float = 10.0,
|
||||
interval: float = 0.5,
|
||||
confidence: float = 0.9) -> Optional[Tuple[int, int]]:
|
||||
"""
|
||||
等待指定图片出现在屏幕上。
|
||||
|
||||
Args:
|
||||
image_path: 要等待的图片路径
|
||||
timeout: 最大等待时间（秒）
|
||||
interval: 检查间隔
|
||||
confidence: 匹配置信度
|
||||
|
||||
Returns:
|
||||
出现后返回坐标，超时返回 None
|
||||
"""
|
||||
start = time.time()
|
||||
while time.time() - start < timeout:
|
||||
pos = self.locate_on_screen(image_path, confidence=confidence)
|
||||
if pos:
|
||||
print(f"[INFO] 图片 '{image_path}' 已出现于 {pos}")
|
||||
return pos
|
||||
time.sleep(interval)
|
||||
print(f"[WARN] 等待图片 '{image_path}' 超时 ({timeout}s)")
|
||||
return None
|
||||
|
||||
def click_image(self, image_path: str, confidence: float = 0.9,
|
||||
timeout: float = 10.0) -> bool:
|
||||
"""
|
||||
等待图片出现并点击。
|
||||
|
||||
Args:
|
||||
image_path: 要点击的图片路径
|
||||
confidence: 匹配置信度
|
||||
timeout: 最大等待时间
|
||||
|
||||
Returns:
|
||||
True 成功点击，False 超时未找到
|
||||
"""
|
||||
pos = self.wait_for_image(image_path, timeout=timeout,
|
||||
confidence=confidence)
|
||||
if pos:
|
||||
self.click(pos[0], pos[1])
|
||||
return True
|
||||
return False
|
||||
|
||||
# ========================================================
|
||||
# 等待与延迟
|
||||
# ========================================================
|
||||
|
||||
def wait(self, seconds: float):
|
||||
"""
|
||||
等待指定时间。
|
||||
|
||||
Args:
|
||||
seconds: 等待秒数
|
||||
"""
|
||||
time.sleep(seconds)
|
||||
|
||||
def wait_for_window(self, title_keyword: str, timeout: float = 10.0,
|
||||
interval: float = 0.5) -> bool:
|
||||
"""
|
||||
等待指定窗口出现。
|
||||
|
||||
Args:
|
||||
title_keyword: 窗口标题关键词
|
||||
timeout: 最大等待时间
|
||||
interval: 检查间隔
|
||||
|
||||
Returns:
|
||||
True 窗口出现，False 超时
|
||||
"""
|
||||
start = time.time()
|
||||
while time.time() - start < timeout:
|
||||
if self.find_window(title_keyword):
|
||||
return True
|
||||
time.sleep(interval)
|
||||
return False
|
||||
|
||||
# ========================================================
|
||||
# UI 控件定位（uiautomation）
|
||||
# ========================================================
|
||||
|
||||
def find_control(self, window_title: str, control_type: str = None,
|
||||
name: str = None, auto_id: str = None,
|
||||
depth: int = None) -> Optional[Any]:
|
||||
"""
|
||||
通过 uiautomation 定位 UI 控件。
|
||||
|
||||
优先使用此方法而非坐标点击，因为控件定位不受分辨率影响。
|
||||
|
||||
Args:
|
||||
window_title: 窗口标题关键词
|
||||
control_type: 控件类型，如 "Button", "Edit", "Text",
|
||||
"ListControl", "ListItem"
|
||||
name: 控件名称（模糊匹配）
|
||||
auto_id: AutomationId 属性
|
||||
depth: 搜索深度
|
||||
|
||||
Returns:
|
||||
uiautomation 控件对象，未找到返回 None
|
||||
|
||||
Example:
|
||||
>>> btn = auto.find_control("记事本", control_type="Button", name="保存")
|
||||
>>> if btn:
|
||||
... btn.Click()
|
||||
"""
|
||||
uia = _import_uiautomation()
|
||||
|
||||
# 先找到顶层窗口
|
||||
top_window = uia.WindowControl(searchDepth=1, Name=window_title)
|
||||
if not top_window.Exists(2, 0.5):
|
||||
print(f"[WARN] 未找到窗口: {window_title}")
|
||||
return None
|
||||
|
||||
# 构建搜索参数
|
||||
kwargs = {}
|
||||
if control_type:
|
||||
kwargs["ControlType"] = control_type
|
||||
if name:
|
||||
kwargs["Name"] = name
|
||||
if auto_id:
|
||||
kwargs["AutomationId"] = auto_id
|
||||
if depth:
|
||||
kwargs["searchDepth"] = depth
|
||||
|
||||
control = top_window.FindFirst(**kwargs) if kwargs else None
|
||||
if control:
|
||||
print(f"[INFO] 找到控件: {control_type} '{name}'")
|
||||
else:
|
||||
print(f"[WARN] 未找到控件: {control_type} '{name}'")
|
||||
return control
|
||||
|
||||
def click_control(self, window_title: str, control_type: str = None,
|
||||
name: str = None, auto_id: str = None) -> bool:
|
||||
"""
|
||||
定位并点击 UI 控件。
|
||||
|
||||
Returns:
|
||||
True 成功，False 未找到
|
||||
"""
|
||||
ctrl = self.find_control(window_title, control_type, name, auto_id)
|
||||
if ctrl:
|
||||
ctrl.Click()
|
||||
time.sleep(0.2)
|
||||
return True
|
||||
return False
|
||||
|
||||
def get_control_text(self, window_title: str, control_type: str = None,
|
||||
name: str = None, auto_id: str = None) -> Optional[str]:
|
||||
"""
|
||||
获取控件的文本内容。
|
||||
|
||||
Returns:
|
||||
控件文本，未找到返回 None
|
||||
"""
|
||||
ctrl = self.find_control(window_title, control_type, name, auto_id)
|
||||
if ctrl:
|
||||
return ctrl.Name
|
||||
return None
|
||||
|
||||
# ========================================================
|
||||
# 实用工具
|
||||
# ========================================================
|
||||
|
||||
def get_screen_size(self) -> Tuple[int, int]:
|
||||
"""获取屏幕分辨率"""
|
||||
return (self._screen_width, self._screen_height)
|
||||
|
||||
def alert(self, message: str, title: str = "Desktop Automation"):
|
||||
"""
|
||||
弹出消息框（使用 pyautogui 的 alert）。
|
||||
|
||||
用于在自动化流程中向用户展示信息或等待确认。
|
||||
"""
|
||||
pyautogui.alert(text=message, title=title)
|
||||
|
||||
def confirm(self, message: str, title: str = "Confirm") -> bool:
|
||||
"""
|
||||
弹出确认框，等待用户选择 OK/Cancel。
|
||||
|
||||
Returns:
|
||||
True=OK, False=Cancel
|
||||
"""
|
||||
return pyautogui.confirm(text=message, title=title) == "OK"
|
||||
|
||||
def prompt(self, message: str, title: str = "Input",
|
||||
default: str = "") -> Optional[str]:
|
||||
"""
|
||||
弹出输入框。
|
||||
|
||||
Returns:
|
||||
用户输入的文本，取消返回 None
|
||||
"""
|
||||
return pyautogui.prompt(text=message, title=title, default=default)
|
||||
|
||||
|
||||
# ============================================================
|
||||
# CLI 入口（用于快速测试）
|
||||
# ============================================================
|
||||
|
||||
if __name__ == "__main__":
|
||||
auto = DesktopAuto()
|
||||
|
||||
print("\n=== Desktop Automation Engine ===")
|
||||
print(f"屏幕分辨率: {auto.get_screen_size()}")
|
||||
print(f"当前鼠标位置: {auto.get_mouse_pos()}")
|
||||
|
||||
# 列出所有可见窗口
|
||||
print("\n--- 可见窗口列表 ---")
|
||||
windows = auto.find_all_windows()
|
||||
for w in windows[:20]: # 只显示前20个
|
||||
if w.title: # 跳过无标题窗口
|
||||
print(f" [{w.class_name}] {w.title} rect={w.rect}")
|
||||
|
||||
print(f"\n共 {len([w for w in windows if w.title])} 个有标题窗口")
|
||||
@@ -0,0 +1,702 @@
|
||||
#!/usr/bin/env python3
|
||||
"""
|
||||
html_compiler.py — 聊天记录 HTML 图文报告编译器
|
||||
|
||||
将结构化消息和截图编译为美观的 HTML 报告，仿微信聊天气泡样式。
|
||||
|
||||
特性:
|
||||
- 仿微信聊天气泡样式（绿色=自己，白色=对方）
|
||||
- 时间线分组显示
|
||||
- 图片消息内嵌展示
|
||||
- 发送者头像（首字母色块）
|
||||
- 搜索和过滤功能
|
||||
- 响应式布局，移动端可读
|
||||
- 暗色模式支持
|
||||
|
||||
依赖: Pillow（图片转 base64 内嵌）
|
||||
作者: Duckula (agent_created)
|
||||
"""
|
||||
|
||||
import os
|
||||
import base64
|
||||
import html
|
||||
from pathlib import Path
|
||||
from datetime import datetime
|
||||
from typing import List, Optional, Dict
|
||||
from dataclasses import dataclass, field
|
||||
|
||||
# 尝试导入 MessageItem（如果从同目录运行）
|
||||
try:
|
||||
from desktop_auto import MessageItem
|
||||
except ImportError:
|
||||
# 独立运行时定义简化版
|
||||
@dataclass
|
||||
class MessageItem:
|
||||
sender: str
|
||||
content: str
|
||||
timestamp: Optional[str] = None
|
||||
msg_type: str = "text"
|
||||
screenshot_path: Optional[str] = None
|
||||
raw_ocr: Optional[str] = None
|
||||
|
||||
|
||||
# ============================================================
|
||||
# 颜色方案（基于发送者名称生成稳定的头像颜色）
|
||||
# ============================================================
|
||||
|
||||
AVATAR_COLORS = [
|
||||
"#07C160", "#FA5151", "#576B95", "#FF9800",
|
||||
"#10AEFF", "#6236FF", "#FFC300", "#FF6B6B",
|
||||
"#4ECDC4", "#45B7D1", "#96CEB4", "#FFEAA7",
|
||||
"#DDA0DD", "#98D8C8", "#F7DC6F", "#BB8FCE",
|
||||
]
|
||||
|
||||
|
||||
def get_avatar_color(name: str) -> str:
|
||||
"""根据名称生成稳定的头像背景色"""
|
||||
hash_val = sum(ord(c) for c in name)
|
||||
return AVATAR_COLORS[hash_val % len(AVATAR_COLORS)]
|
||||
|
||||
|
||||
def get_initial(name: str) -> str:
|
||||
"""获取名称首字母（中文取第一个字，英文取首字母大写）"""
|
||||
if not name:
|
||||
return "?"
|
||||
return name[0].upper()
|
||||
|
||||
|
||||
def img_to_base64(img_path: str, max_width: int = 400) -> Optional[str]:
|
||||
"""
|
||||
将图片转换为 base64 编码（用于 HTML 内嵌）。
|
||||
|
||||
Args:
|
||||
img_path: 图片文件路径
|
||||
max_width: 最大宽度（像素），超过会等比缩放
|
||||
|
||||
Returns:
|
||||
data URI 字符串，失败返回 None
|
||||
"""
|
||||
if not os.path.exists(img_path):
|
||||
return None
|
||||
|
||||
try:
|
||||
from PIL import Image
|
||||
import io
|
||||
|
||||
img = Image.open(img_path)
|
||||
# 等比缩放
|
||||
if img.width > max_width:
|
||||
ratio = max_width / img.width
|
||||
img = img.resize((max_width, int(img.height * ratio)))
|
||||
|
||||
# 转 PNG base64
|
||||
buffer = io.BytesIO()
|
||||
img.save(buffer, format="PNG")
|
||||
b64 = base64.b64encode(buffer.getvalue()).decode()
|
||||
return f"data:image/png;base64,{b64}"
|
||||
except Exception as e:
|
||||
print(f"[WARN] 图片转 base64 失败: {img_path} - {e}")
|
||||
return None
|
||||
|
||||
|
||||
# ============================================================
|
||||
# HTML 模板
|
||||
# ============================================================
|
||||
|
||||
HTML_TEMPLATE = """<!DOCTYPE html>
|
||||
<html lang="zh-CN">
|
||||
<head>
|
||||
<meta charset="UTF-8">
|
||||
<meta name="viewport" content="width=device-width, initial-scale=1.0">
|
||||
<title>{title}</title>
|
||||
<style>
|
||||
* {{
|
||||
margin: 0;
|
||||
padding: 0;
|
||||
box-sizing: border-box;
|
||||
}}
|
||||
|
||||
body {{
|
||||
font-family: -apple-system, BlinkMacSystemFont, "Segoe UI",
|
||||
"PingFang SC", "Hiragino Sans GB", "Microsoft YaHei",
|
||||
sans-serif;
|
||||
background: #EDEDED;
|
||||
color: #333;
|
||||
line-height: 1.6;
|
||||
}}
|
||||
|
||||
/* 顶部信息栏 */
|
||||
.header {{
|
||||
background: #ededed;
|
||||
border-bottom: 1px solid #dcdcdc;
|
||||
padding: 12px 20px;
|
||||
position: sticky;
|
||||
top: 0;
|
||||
z-index: 100;
|
||||
display: flex;
|
||||
align-items: center;
|
||||
justify-content: space-between;
|
||||
}}
|
||||
|
||||
.header-info {{
|
||||
display: flex;
|
||||
flex-direction: column;
|
||||
}}
|
||||
|
||||
.header-title {{
|
||||
font-size: 16px;
|
||||
font-weight: 600;
|
||||
color: #191919;
|
||||
}}
|
||||
|
||||
.header-meta {{
|
||||
font-size: 12px;
|
||||
color: #888;
|
||||
margin-top: 2px;
|
||||
}}
|
||||
|
||||
/* 搜索栏 */
|
||||
.search-bar {{
|
||||
padding: 8px 16px;
|
||||
background: #f5f5f5;
|
||||
border-bottom: 1px solid #e0e0e0;
|
||||
}}
|
||||
|
||||
.search-bar input {{
|
||||
width: 100%;
|
||||
padding: 8px 12px;
|
||||
border: 1px solid #ddd;
|
||||
border-radius: 20px;
|
||||
font-size: 14px;
|
||||
outline: none;
|
||||
transition: border-color 0.2s;
|
||||
}}
|
||||
|
||||
.search-bar input:focus {{
|
||||
border-color: #07C160;
|
||||
}}
|
||||
|
||||
/* 聊天容器 */
|
||||
.chat-container {{
|
||||
max-width: 800px;
|
||||
margin: 0 auto;
|
||||
padding: 16px;
|
||||
}}
|
||||
|
||||
/* 时间分隔符 */
|
||||
.time-separator {{
|
||||
text-align: center;
|
||||
margin: 20px 0;
|
||||
color: #999;
|
||||
font-size: 12px;
|
||||
}}
|
||||
|
||||
.time-separator span {{
|
||||
background: rgba(0, 0, 0, 0.08);
|
||||
padding: 2px 8px;
|
||||
border-radius: 4px;
|
||||
}}
|
||||
|
||||
/* 消息行 */
|
||||
.message-row {{
|
||||
display: flex;
|
||||
margin-bottom: 16px;
|
||||
align-items: flex-start;
|
||||
gap: 8px;
|
||||
}}
|
||||
|
||||
.message-row.self {{
|
||||
flex-direction: row-reverse;
|
||||
}}
|
||||
|
||||
/* 头像 */
|
||||
.avatar {{
|
||||
width: 40px;
|
||||
height: 40px;
|
||||
border-radius: 4px;
|
||||
display: flex;
|
||||
align-items: center;
|
||||
justify-content: center;
|
||||
color: white;
|
||||
font-size: 16px;
|
||||
font-weight: 600;
|
||||
flex-shrink: 0;
|
||||
}}
|
||||
|
||||
/* 消息内容区 */
|
||||
.message-content {{
|
||||
max-width: 60%;
|
||||
display: flex;
|
||||
flex-direction: column;
|
||||
}}
|
||||
|
||||
.message-row.self .message-content {{
|
||||
align-items: flex-end;
|
||||
}}
|
||||
|
||||
/* 发送者名称 */
|
||||
.sender-name {{
|
||||
font-size: 12px;
|
||||
color: #888;
|
||||
margin-bottom: 2px;
|
||||
padding: 0 4px;
|
||||
}}
|
||||
|
||||
/* 消息气泡 */
|
||||
.bubble {{
|
||||
padding: 10px 14px;
|
||||
border-radius: 4px;
|
||||
font-size: 15px;
|
||||
word-wrap: break-word;
|
||||
position: relative;
|
||||
white-space: pre-wrap;
|
||||
}}
|
||||
|
||||
/* 对方消息气泡（白色） */
|
||||
.message-row.other .bubble {{
|
||||
background: white;
|
||||
color: #333;
|
||||
border: 1px solid #e5e5e5;
|
||||
}}
|
||||
|
||||
.message-row.other .bubble::before {{
|
||||
content: "";
|
||||
position: absolute;
|
||||
left: -6px;
|
||||
top: 12px;
|
||||
border: 6px solid transparent;
|
||||
border-right-color: white;
|
||||
}}
|
||||
|
||||
/* 自己消息气泡（绿色） */
|
||||
.message-row.self .bubble {{
|
||||
background: #95EC69;
|
||||
color: #333;
|
||||
}}
|
||||
|
||||
.message-row.self .bubble::before {{
|
||||
content: "";
|
||||
position: absolute;
|
||||
right: -6px;
|
||||
top: 12px;
|
||||
border: 6px solid transparent;
|
||||
border-left-color: #95EC69;
|
||||
}}
|
||||
|
||||
/* 图片消息 */
|
||||
.bubble.image-bubble {{
|
||||
padding: 4px;
|
||||
border-radius: 4px;
|
||||
}}
|
||||
|
||||
.bubble.image-bubble img {{
|
||||
max-width: 100%;
|
||||
border-radius: 4px;
|
||||
display: block;
|
||||
}}
|
||||
|
||||
/* 系统消息 */
|
||||
.message-row.system {{
|
||||
justify-content: center;
|
||||
}}
|
||||
|
||||
.message-row.system .bubble {{
|
||||
background: rgba(0, 0, 0, 0.08);
|
||||
color: #999;
|
||||
font-size: 12px;
|
||||
border: none;
|
||||
}}
|
||||
|
||||
.message-row.system .bubble::before {{
|
||||
display: none;
|
||||
}}
|
||||
|
||||
/* 截图展示 */
|
||||
.screenshot-section {{
|
||||
margin-top: 40px;
|
||||
border-top: 2px solid #ddd;
|
||||
padding-top: 20px;
|
||||
}}
|
||||
|
||||
.screenshot-section h3 {{
|
||||
color: #666;
|
||||
margin-bottom: 16px;
|
||||
font-size: 16px;
|
||||
}}
|
||||
|
||||
.screenshot-grid {{
|
||||
display: grid;
|
||||
grid-template-columns: repeat(auto-fill, minmax(200px, 1fr));
|
||||
gap: 12px;
|
||||
}}
|
||||
|
||||
.screenshot-item {{
|
||||
border: 1px solid #ddd;
|
||||
border-radius: 8px;
|
||||
overflow: hidden;
|
||||
cursor: pointer;
|
||||
transition: transform 0.2s;
|
||||
}}
|
||||
|
||||
.screenshot-item:hover {{
|
||||
transform: scale(1.02);
|
||||
box-shadow: 0 4px 12px rgba(0, 0, 0, 0.1);
|
||||
}}
|
||||
|
||||
.screenshot-item img {{
|
||||
width: 100%;
|
||||
display: block;
|
||||
}}
|
||||
|
||||
.screenshot-item .label {{
|
||||
padding: 6px 8px;
|
||||
font-size: 12px;
|
||||
color: #666;
|
||||
background: #f5f5f5;
|
||||
}}
|
||||
|
||||
/* 底部统计 */
|
||||
.footer {{
|
||||
text-align: center;
|
||||
padding: 24px;
|
||||
color: #999;
|
||||
font-size: 12px;
|
||||
}}
|
||||
|
||||
/* 暗色模式 */
|
||||
@media (prefers-color-scheme: dark) {{
|
||||
body {{
|
||||
background: #1a1a1a;
|
||||
color: #e0e0e0;
|
||||
}}
|
||||
.header {{
|
||||
background: #2a2a2a;
|
||||
border-color: #3a3a3a;
|
||||
}}
|
||||
.header-title {{
|
||||
color: #e0e0e0;
|
||||
}}
|
||||
.search-bar {{
|
||||
background: #2a2a2a;
|
||||
}}
|
||||
.search-bar input {{
|
||||
background: #333;
|
||||
color: #e0e0e0;
|
||||
border-color: #444;
|
||||
}}
|
||||
.message-row.other .bubble {{
|
||||
background: #333;
|
||||
color: #e0e0e0;
|
||||
border-color: #444;
|
||||
}}
|
||||
.message-row.other .bubble::before {{
|
||||
border-right-color: #333;
|
||||
}}
|
||||
.message-row.self .bubble {{
|
||||
background: #2b5e1a;
|
||||
color: #e0e0e0;
|
||||
}}
|
||||
.message-row.self .bubble::before {{
|
||||
border-left-color: #2b5e1a;
|
||||
}}
|
||||
}}
|
||||
|
||||
/* 模态查看器 */
|
||||
.modal {{
|
||||
display: none;
|
||||
position: fixed;
|
||||
top: 0; left: 0; right: 0; bottom: 0;
|
||||
background: rgba(0, 0, 0, 0.9);
|
||||
z-index: 1000;
|
||||
justify-content: center;
|
||||
align-items: center;
|
||||
cursor: zoom-out;
|
||||
}}
|
||||
|
||||
.modal img {{
|
||||
max-width: 90%;
|
||||
max-height: 90%;
|
||||
}}
|
||||
</style>
|
||||
</head>
|
||||
<body>
|
||||
<div class="header">
|
||||
<div class="header-info">
|
||||
<div class="header-title">{title}</div>
|
||||
<div class="header-meta">{meta_text}</div>
|
||||
</div>
|
||||
<div class="header-meta">
|
||||
{message_count} 条消息 · {screenshot_count} 张截图
|
||||
</div>
|
||||
</div>
|
||||
|
||||
<div class="search-bar">
|
||||
<input type="text" id="searchInput"
|
||||
placeholder="搜索消息内容..." onkeyup="filterMessages()">
|
||||
</div>
|
||||
|
||||
<div class="chat-container" id="chatContainer">
|
||||
{message_html}
|
||||
</div>
|
||||
|
||||
{screenshot_section}
|
||||
|
||||
<div class="footer">
|
||||
由 Desktop Automation Skill 生成 ·
|
||||
{export_time}
|
||||
</div>
|
||||
|
||||
<div class="modal" id="imageModal" onclick="closeModal()">
|
||||
<img id="modalImage" src="">
|
||||
</div>
|
||||
|
||||
<script>
|
||||
// 搜索过滤
|
||||
function filterMessages() {{
|
||||
const keyword = document.getElementById('searchInput').value.toLowerCase();
|
||||
const messages = document.querySelectorAll('.message-row');
|
||||
messages.forEach(msg => {{
|
||||
const text = msg.textContent.toLowerCase();
|
||||
msg.style.display = text.includes(keyword) ? '' : 'none';
|
||||
}});
|
||||
}}
|
||||
|
||||
// 图片点击放大
|
||||
function openModal(src) {{
|
||||
const modal = document.getElementById('imageModal');
|
||||
const img = document.getElementById('modalImage');
|
||||
img.src = src;
|
||||
modal.style.display = 'flex';
|
||||
}}
|
||||
|
||||
function closeModal() {{
|
||||
document.getElementById('imageModal').style.display = 'none';
|
||||
}}
|
||||
|
||||
// ESC 关闭模态框
|
||||
document.addEventListener('keydown', (e) => {{
|
||||
if (e.key === 'Escape') closeModal();
|
||||
}});
|
||||
</script>
|
||||
</body>
|
||||
</html>"""
|
||||
|
||||
|
||||
# ============================================================
|
||||
# HTML 编译器
|
||||
# ============================================================
|
||||
|
||||
class ChatHTMLCompiler:
|
||||
"""
|
||||
将结构化消息和截图编译为图文 HTML 报告。
|
||||
|
||||
Usage:
|
||||
compiler = ChatHTMLCompiler(output_dir="./output")
|
||||
compiler.add_messages(messages)
|
||||
compiler.add_screenshots_dir("./screenshots")
|
||||
compiler.compile("聊天记录 - 张三", "张三")
|
||||
"""
|
||||
|
||||
# 自己发送消息的标识关键词
|
||||
SELF_KEYWORDS = ["我", "自己", "I", "me", "my"]
|
||||
|
||||
def __init__(self, output_dir: str = "."):
|
||||
"""
|
||||
Args:
|
||||
output_dir: 输出目录
|
||||
"""
|
||||
self.output_dir = Path(output_dir)
|
||||
self.output_dir.mkdir(parents=True, exist_ok=True)
|
||||
self.messages: List[MessageItem] = []
|
||||
self.screenshots: List[str] = []
|
||||
|
||||
def add_messages(self, messages: List[MessageItem]):
|
||||
"""添加消息列表"""
|
||||
self.messages.extend(messages)
|
||||
|
||||
def add_message(self, message: MessageItem):
|
||||
"""添加单条消息"""
|
||||
self.messages.append(message)
|
||||
|
||||
def add_screenshots_dir(self, dir_path: str):
|
||||
"""添加截图目录下所有图片"""
|
||||
d = Path(dir_path)
|
||||
if d.exists():
|
||||
for f in sorted(d.glob("*.png")):
|
||||
self.screenshots.append(str(f))
|
||||
|
||||
def add_screenshot(self, path: str):
|
||||
"""添加单张截图"""
|
||||
if os.path.exists(path):
|
||||
self.screenshots.append(path)
|
||||
|
||||
def _is_self(self, sender: str) -> bool:
|
||||
"""判断是否是自己发送的消息"""
|
||||
return any(kw in sender for kw in SELF_KEYWORDS) if sender else False
|
||||
|
||||
def _render_message(self, msg: MessageItem, index: int) -> str:
|
||||
"""渲染单条消息为 HTML"""
|
||||
is_self = self._is_self(msg.sender)
|
||||
avatar_color = get_avatar_color(msg.sender)
|
||||
initial = get_initial(msg.sender)
|
||||
|
||||
if msg.msg_type == "system":
|
||||
return f"""
|
||||
<div class="message-row system" data-index="{index}">
|
||||
<div class="bubble">{html.escape(msg.content)}</div>
|
||||
</div>"""
|
||||
|
||||
if msg.msg_type == "image" and msg.screenshot_path:
|
||||
img_uri = img_to_base64(msg.screenshot_path)
|
||||
if img_uri:
|
||||
return f"""
|
||||
<div class="message-row {'self' if is_self else 'other'}" data-index="{index}">
|
||||
<div class="avatar" style="background:{avatar_color}">{initial}</div>
|
||||
<div class="message-content">
|
||||
<div class="sender-name">{html.escape(msg.sender)}</div>
|
||||
<div class="bubble image-bubble">
|
||||
<img src="{img_uri}"
|
||||
onclick="openModal(this.src)"
|
||||
alt="图片消息">
|
||||
</div>
|
||||
</div>
|
||||
</div>"""
|
||||
|
||||
# 文本消息
|
||||
content = html.escape(msg.content)
|
||||
return f"""
|
||||
<div class="message-row {'self' if is_self else 'other'}" data-index="{index}">
|
||||
<div class="avatar" style="background:{avatar_color}">{initial}</div>
|
||||
<div class="message-content">
|
||||
<div class="sender-name">{html.escape(msg.sender)}</div>
|
||||
<div class="bubble">{content}</div>
|
||||
</div>
|
||||
</div>"""
|
||||
|
||||
def _render_messages(self) -> str:
|
||||
"""渲染所有消息为 HTML"""
|
||||
if not self.messages:
|
||||
return '<div style="text-align:center;padding:40px;color:#999;">暂无消息</div>'
|
||||
|
||||
parts = []
|
||||
prev_timestamp = None
|
||||
|
||||
for i, msg in enumerate(self.messages):
|
||||
# 时间分隔符
|
||||
if msg.timestamp and msg.timestamp != prev_timestamp:
|
||||
parts.append(f"""
|
||||
<div class="time-separator">
|
||||
<span>{html.escape(msg.timestamp)}</span>
|
||||
</div>""")
|
||||
prev_timestamp = msg.timestamp
|
||||
|
||||
parts.append(self._render_message(msg, i))
|
||||
|
||||
return "\n".join(parts)
|
||||
|
||||
def _render_screenshots(self) -> str:
|
||||
"""渲染截图区域为 HTML"""
|
||||
if not self.screenshots:
|
||||
return ""
|
||||
|
||||
items = []
|
||||
for i, path in enumerate(self.screenshots):
|
||||
img_uri = img_to_base64(path, max_width=300)
|
||||
if img_uri:
|
||||
items.append(f"""
|
||||
<div class="screenshot-item" onclick="openModal('{img_uri}')">
|
||||
<img src="{img_uri}" alt="截图 {i+1}">
|
||||
<div class="label">截图 {i+1}</div>
|
||||
</div>""")
|
||||
|
||||
if not items:
|
||||
return ""
|
||||
|
||||
return f"""
|
||||
<div class="screenshot-section">
|
||||
<h3>原始截图 ({len(items)} 张)</h3>
|
||||
<div class="screenshot-grid">
|
||||
{''.join(items)}
|
||||
</div>
|
||||
</div>"""
|
||||
|
||||
def compile(self, title: str, contact_name: str = "") -> str:
|
||||
"""
|
||||
编译生成 HTML 报告。
|
||||
|
||||
Args:
|
||||
title: 报告标题
|
||||
contact_name: 联系人名称（用于元数据）
|
||||
|
||||
Returns:
|
||||
生成的 HTML 文件路径
|
||||
"""
|
||||
# 生成元数据
|
||||
export_time = datetime.now().strftime("%Y-%m-%d %H:%M:%S")
|
||||
meta_text = f"联系人: {contact_name} | 导出时间: {export_time}"
|
||||
|
||||
# 渲染各部分
|
||||
message_html = self._render_messages()
|
||||
screenshot_section = self._render_screenshots()
|
||||
|
||||
# 填充模板
|
||||
html_content = HTML_TEMPLATE.format(
|
||||
title=html.escape(title),
|
||||
meta_text=html.escape(meta_text),
|
||||
message_count=len(self.messages),
|
||||
screenshot_count=len(self.screenshots),
|
||||
message_html=message_html,
|
||||
screenshot_section=screenshot_section,
|
||||
export_time=export_time,
|
||||
)
|
||||
|
||||
# 写入文件
|
||||
output_path = self.output_dir / f"{title}.html"
|
||||
with open(output_path, "w", encoding="utf-8") as f:
|
||||
f.write(html_content)
|
||||
|
||||
print(f"[INFO] HTML 报告已生成: {output_path}")
|
||||
return str(output_path)
|
||||
|
||||
|
||||
# ============================================================
|
||||
# CLI 入口
|
||||
# ============================================================
|
||||
|
||||
if __name__ == "__main__":
|
||||
import json
|
||||
import sys
|
||||
|
||||
if len(sys.argv) < 2:
|
||||
print("Usage: python html_compiler.py <messages.json> [output_dir]")
|
||||
print(" messages.json: 由 wechat_export.py 生成的消息 JSON 文件")
|
||||
sys.exit(1)
|
||||
|
||||
json_path = sys.argv[1]
|
||||
output_dir = sys.argv[2] if len(sys.argv) > 2 else "."
|
||||
|
||||
# 从 JSON 加载消息
|
||||
with open(json_path, "r", encoding="utf-8") as f:
|
||||
data = json.load(f)
|
||||
|
||||
messages = [
|
||||
MessageItem(
|
||||
sender=item.get("sender", "未知"),
|
||||
content=item.get("content", ""),
|
||||
timestamp=item.get("timestamp"),
|
||||
msg_type=item.get("type", "text"),
|
||||
screenshot_path=item.get("screenshot"),
|
||||
)
|
||||
for item in data
|
||||
]
|
||||
|
||||
compiler = ChatHTMLCompiler(output_dir=output_dir)
|
||||
compiler.add_messages(messages)
|
||||
|
||||
# 尝试加载同目录截图
|
||||
screenshot_dir = Path(json_path).parent / "screenshots"
|
||||
if screenshot_dir.exists():
|
||||
compiler.add_screenshots_dir(str(screenshot_dir))
|
||||
|
||||
report_path = compiler.compile("聊天记录导出", "")
|
||||
print(f"\n报告路径: {report_path}")
|
||||
@@ -0,0 +1,287 @@
|
||||
#!/usr/bin/env python3
|
||||
"""
|
||||
init_env.py — 桌面自动化技能环境检测与依赖安装
|
||||
|
||||
检测所需 Python 包是否已安装，未安装的自动通过 pip 安装。
|
||||
同时检测系统环境（Windows 版本、分辨率、DPI 缩放）。
|
||||
|
||||
用法:
|
||||
python init_env.py # 检测并安装缺失依赖
|
||||
python init_env.py --check # 仅检测，不安装
|
||||
python init_env.py --ocr # 仅安装 OCR 相关依赖
|
||||
|
||||
作者: Duckula (agent_created)
|
||||
"""
|
||||
|
||||
import os
|
||||
import sys
|
||||
import platform
|
||||
import subprocess
|
||||
from typing import List, Tuple
|
||||
|
||||
|
||||
# ============================================================
|
||||
# 依赖定义
|
||||
# ============================================================
|
||||
|
||||
# 核心依赖（必须安装）
|
||||
CORE_DEPS = [
|
||||
{
|
||||
"import_name": "pyautogui",
|
||||
"pip_name": "pyautogui",
|
||||
"description": "鼠标键盘模拟（核心）",
|
||||
"required": True,
|
||||
},
|
||||
{
|
||||
"import_name": "pyperclip",
|
||||
"pip_name": "pyperclip",
|
||||
"description": "剪贴板操作（中文输入）",
|
||||
"required": True,
|
||||
},
|
||||
{
|
||||
"import_name": "PIL",
|
||||
"pip_name": "Pillow",
|
||||
"description": "图像处理（截图）",
|
||||
"required": True,
|
||||
},
|
||||
{
|
||||
"import_name": "pygetwindow",
|
||||
"pip_name": "PyGetWindow",
|
||||
"description": "窗口管理",
|
||||
"required": True,
|
||||
},
|
||||
]
|
||||
|
||||
# Windows 专用依赖
|
||||
WINDOWS_DEPS = [
|
||||
{
|
||||
"import_name": "win32gui",
|
||||
"pip_name": "pywin32",
|
||||
"description": "Windows API 调用",
|
||||
"required": True,
|
||||
},
|
||||
{
|
||||
"import_name": "uiautomation",
|
||||
"pip_name": "uiautomation",
|
||||
"description": "UI 控件自动化",
|
||||
"required": False,
|
||||
},
|
||||
]
|
||||
|
||||
# OCR 依赖（可选，用于文字识别）
|
||||
OCR_DEPS = [
|
||||
{
|
||||
"import_name": "paddleocr",
|
||||
"pip_name": "paddleocr",
|
||||
"description": "PaddleOCR 中文识别（推荐）",
|
||||
"required": False,
|
||||
},
|
||||
{
|
||||
"import_name": "paddle",
|
||||
"pip_name": "paddlepaddle",
|
||||
"description": "PaddlePaddle 深度学习框架",
|
||||
"required": False,
|
||||
},
|
||||
]
|
||||
|
||||
# 图像匹配依赖
|
||||
IMAGE_DEPS = [
|
||||
{
|
||||
"import_name": "cv2",
|
||||
"pip_name": "opencv-python",
|
||||
"description": "OpenCV 图像匹配",
|
||||
"required": False,
|
||||
},
|
||||
]
|
||||
|
||||
|
||||
# ============================================================
|
||||
# 检测函数
|
||||
# ============================================================
|
||||
|
||||
def check_package(import_name: str) -> bool:
|
||||
"""检测 Python 包是否已安装"""
|
||||
try:
|
||||
__import__(import_name)
|
||||
return True
|
||||
except ImportError:
|
||||
return False
|
||||
|
||||
|
||||
def install_package(pip_name: str) -> bool:
|
||||
"""通过 pip 安装包"""
|
||||
print(f" 正在安装 {pip_name} ...")
|
||||
try:
|
||||
result = subprocess.run(
|
||||
[sys.executable, "-m", "pip", "install", pip_name],
|
||||
capture_output=True,
|
||||
text=True,
|
||||
timeout=120, # 2分钟超时
|
||||
)
|
||||
if result.returncode == 0:
|
||||
print(f" [OK] {pip_name} 安装成功")
|
||||
return True
|
||||
else:
|
||||
print(f" [FAIL] {pip_name} 安装失败")
|
||||
if result.stderr:
|
||||
# 只显示最后几行错误信息
|
||||
lines = result.stderr.strip().split("\n")
|
||||
for line in lines[-5:]:
|
||||
print(f" {line}")
|
||||
return False
|
||||
except subprocess.TimeoutExpired:
|
||||
print(f" [FAIL] {pip_name} 安装超时")
|
||||
return False
|
||||
except Exception as e:
|
||||
print(f" [FAIL] {pip_name} 安装异常: {e}")
|
||||
return False
|
||||
|
||||
|
||||
def check_system_info():
|
||||
"""检测系统环境信息"""
|
||||
print("\n" + "=" * 50)
|
||||
print(" 系统环境检测")
|
||||
print("=" * 50)
|
||||
|
||||
print(f" 操作系统: {platform.system()} {platform.version()}")
|
||||
print(f" 架构: {platform.machine()}")
|
||||
print(f" Python: {sys.version.split()[0]} ({sys.executable})")
|
||||
|
||||
if platform.system() == "Windows":
|
||||
# 检测 DPI 缩放
|
||||
try:
|
||||
import ctypes
|
||||
# 设置进程 DPI 感知
|
||||
ctypes.windll.shcore.SetProcessDpiAwareness(2) # PER_MONITOR_AWARE
|
||||
|
||||
# 获取屏幕分辨率
|
||||
user32 = ctypes.windll.user32
|
||||
user32.SetProcessDPIAware()
|
||||
width = user32.GetSystemMetrics(0)
|
||||
height = user32.GetSystemMetrics(1)
|
||||
print(f" 屏幕分辨率: {width}x{height}")
|
||||
|
||||
# 获取缩放比例
|
||||
hdc = user32.GetDC(0)
|
||||
LOGPIXELSX = 88
|
||||
dpi = ctypes.windll.gdi32.GetDeviceCaps(hdc, LOGPIXELSX)
|
||||
scale = round(dpi / 96 * 100)
|
||||
print(f" DPI 缩放: {scale}% ({dpi} DPI)")
|
||||
|
||||
if scale > 100:
|
||||
print(f" [!] 检测到高 DPI 缩放，坐标可能需要校准")
|
||||
print(f" 建议: 在脚本中调用 pyautogui.size() 获取实际分辨率")
|
||||
|
||||
except Exception as e:
|
||||
print(f" [WARN] 无法检测 DPI: {e}")
|
||||
|
||||
|
||||
def check_dependencies(dep_list: List[dict], install: bool = True) -> Tuple[int, int]:
|
||||
"""
|
||||
检测并安装依赖列表。
|
||||
|
||||
Returns:
|
||||
(已安装数, 总数)
|
||||
"""
|
||||
installed = 0
|
||||
total = len(dep_list)
|
||||
|
||||
for dep in dep_list:
|
||||
name = dep["import_name"]
|
||||
pip_name = dep["pip_name"]
|
||||
desc = dep["description"]
|
||||
required = dep["required"]
|
||||
|
||||
if check_package(name):
|
||||
print(f" [OK] {name:20s} {desc}")
|
||||
installed += 1
|
||||
else:
|
||||
status = "REQ" if required else "OPT"
|
||||
print(f" [{status}] {name:20s} {desc} — 未安装")
|
||||
|
||||
if install and (required or install_optional):
|
||||
if install_package(pip_name):
|
||||
installed += 1
|
||||
elif required:
|
||||
print(f" [ERROR] 必需依赖 {pip_name} 安装失败！")
|
||||
|
||||
return installed, total
|
||||
|
||||
|
||||
# ============================================================
|
||||
# 主流程
|
||||
# ============================================================
|
||||
|
||||
# 全局变量：是否安装可选依赖
|
||||
install_optional = True
|
||||
|
||||
|
||||
def main():
|
||||
global install_optional
|
||||
|
||||
import argparse
|
||||
|
||||
parser = argparse.ArgumentParser(description="桌面自动化环境检测与安装")
|
||||
parser.add_argument("--check", action="store_true",
|
||||
help="仅检测，不安装缺失依赖")
|
||||
parser.add_argument("--ocr", action="store_true",
|
||||
help="仅检测/安装 OCR 相关依赖")
|
||||
parser.add_argument("--no-optional", action="store_true",
|
||||
help="跳过可选依赖")
|
||||
args = parser.parse_args()
|
||||
|
||||
install = not args.check
|
||||
install_optional = not args.no_optional
|
||||
|
||||
# 系统信息
|
||||
check_system_info()
|
||||
|
||||
# 核心依赖
|
||||
print("\n" + "-" * 50)
|
||||
print(" 核心依赖检测")
|
||||
print("-" * 50)
|
||||
core_ok, core_total = check_dependencies(CORE_DEPS, install)
|
||||
|
||||
# Windows 专用依赖
|
||||
if platform.system() == "Windows":
|
||||
print("\n" + "-" * 50)
|
||||
print(" Windows 专用依赖检测")
|
||||
print("-" * 50)
|
||||
win_ok, win_total = check_dependencies(WINDOWS_DEPS, install)
|
||||
|
||||
# OCR 依赖
|
||||
if not args.ocr or args.ocr:
|
||||
print("\n" + "-" * 50)
|
||||
print(" OCR 依赖检测（可选）")
|
||||
print("-" * 50)
|
||||
ocr_ok, ocr_total = check_dependencies(OCR_DEPS, install)
|
||||
|
||||
# 图像匹配依赖
|
||||
print("\n" + "-" * 50)
|
||||
print(" 图像匹配依赖检测（可选）")
|
||||
print("-" * 50)
|
||||
img_ok, img_total = check_dependencies(IMAGE_DEPS, install)
|
||||
|
||||
# 汇总
|
||||
print("\n" + "=" * 50)
|
||||
print(" 安装汇总")
|
||||
print("=" * 50)
|
||||
print(f" 核心: {core_ok}/{core_total}")
|
||||
if platform.system() == "Windows":
|
||||
print(f" Windows: {win_ok}/{win_total}")
|
||||
print(f" OCR: {ocr_ok}/{ocr_total}")
|
||||
print(f" 图像匹配: {img_ok}/{img_total}")
|
||||
|
||||
if core_ok == core_total:
|
||||
print("\n [SUCCESS] 核心环境就绪！可以开始使用桌面自动化。")
|
||||
if ocr_ok < ocr_total:
|
||||
print(" [INFO] OCR 依赖未完整安装，聊天导出功能将受限。")
|
||||
print(" 完整安装: pip install paddleocr paddlepaddle")
|
||||
return 0
|
||||
else:
|
||||
print("\n [ERROR] 核心依赖安装不完整，请手动安装缺失包。")
|
||||
return 1
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
sys.exit(main())
|
||||
@@ -0,0 +1,676 @@
|
||||
#!/usr/bin/env python3
|
||||
"""
|
||||
wechat_export.py — 微信 PC 版聊天记录导出工具
|
||||
|
||||
通过模拟键盘鼠标操作微信桌面客户端，滚动截图聊天窗口，
|
||||
配合 OCR 识别将聊天记录导出为结构化消息和图文 HTML 报告。
|
||||
|
||||
工作原理:
|
||||
1. 激活微信窗口 → 搜索联系人 → 打开聊天窗口
|
||||
2. 滚动到聊天顶部 → 逐步向下滚动并截图
|
||||
3. 对截图执行 OCR → 解析消息结构（发送者/时间/内容）
|
||||
4. 消息去重 → 编译为 HTML 报告
|
||||
|
||||
依赖: pyautogui, uiautomation, Pillow, paddleocr, pyperclip, pygetwindow
|
||||
作者: Duckula (agent_created)
|
||||
|
||||
注意:
|
||||
- 微信需已登录并保持在前台
|
||||
- 分辨率/DPI 变化可能影响坐标，脚本自动检测并适配
|
||||
- 滚动速度不宜过快，否则可能漏消息
|
||||
"""
|
||||
|
||||
import os
|
||||
import sys
|
||||
import time
|
||||
import json
|
||||
import hashlib
|
||||
import argparse
|
||||
from pathlib import Path
|
||||
from datetime import datetime
|
||||
from typing import List, Optional, Tuple, Dict
|
||||
|
||||
# 将脚本目录加入 path，方便导入同目录模块
|
||||
SCRIPT_DIR = Path(__file__).parent
|
||||
sys.path.insert(0, str(SCRIPT_DIR))
|
||||
|
||||
from desktop_auto import DesktopAuto, MessageItem, WindowInfo
|
||||
|
||||
|
||||
# ============================================================
|
||||
# 微信窗口常量（基于微信 3.9+ PC 版 UI 分析）
|
||||
# ============================================================
|
||||
|
||||
# 微信窗口类名
|
||||
WECHAT_CLASS_NAME = "WeChatMainWndForPC"
|
||||
|
||||
# 微信搜索框快捷键
|
||||
WECHAT_SEARCH_SHORTCUT = "ctrl+f"
|
||||
|
||||
# 微信发送消息快捷键
|
||||
WECHAT_SEND_SHORTCUT = "enter"
|
||||
|
||||
# 聊天区域的大致比例（相对于窗口）
|
||||
# 微信窗口布局：左侧导航栏 | 中间聊天列表 | 右侧聊天内容
|
||||
# 聊天内容区域大约在窗口宽度的 30%~98%，高度的 8%~92%
|
||||
CHAT_AREA_RATIO = {
|
||||
"left": 0.30, # 聊天内容区左边占窗口宽度的30%开始
|
||||
"top": 0.08, # 顶部留出标题栏
|
||||
"right": 0.98, # 右边距
|
||||
"bottom": 0.88, # 底部留出输入框
|
||||
}
|
||||
|
||||
# 每次滚动的格数（负数=向下）
|
||||
SCROLL_CLICKS = -3
|
||||
|
||||
# 默认参数
|
||||
DEFAULT_COUNT = 50
|
||||
DEFAULT_SCROLL_PAUSE = 1.5
|
||||
DEFAULT_OUTPUT_DIR = "./wechat_export"
|
||||
|
||||
|
||||
# ============================================================
|
||||
# OCR 引擎封装
|
||||
# ============================================================
|
||||
|
||||
class OCREngine:
|
||||
"""
|
||||
OCR 引擎封装，支持 PaddleOCR（首选）和 Tesseract（降级）。
|
||||
|
||||
PaddleOCR 对中文识别准确率更高（~95%），但安装包较大。
|
||||
Tesseract 作为降级方案，需单独安装 tesseract-ocr 并下载中文语言包。
|
||||
"""
|
||||
|
||||
def __init__(self, engine: str = "auto"):
|
||||
"""
|
||||
初始化 OCR 引擎。
|
||||
|
||||
Args:
|
||||
engine: "paddle" / "tesseract" / "auto"（自动选择可用的）
|
||||
"""
|
||||
self.engine_type = None
|
||||
self._ocr = None
|
||||
|
||||
if engine in ("auto", "paddle"):
|
||||
try:
|
||||
from paddleocr import PaddleOCR
|
||||
# 使用轻量级模型，首次加载会下载模型文件
|
||||
self._ocr = PaddleOCR(
|
||||
use_angle_cls=True, # 角度分类（处理歪斜文字）
|
||||
lang="ch", # 中文模型
|
||||
show_log=False,
|
||||
use_gpu=False, # 不使用 GPU（兼容性）
|
||||
)
|
||||
self.engine_type = "paddle"
|
||||
print("[INFO] OCR 引擎: PaddleOCR (中文模型)")
|
||||
return
|
||||
except ImportError:
|
||||
if engine == "paddle":
|
||||
print("[WARN] PaddleOCR 未安装，尝试 Tesseract...")
|
||||
pass
|
||||
|
||||
if engine in ("auto", "tesseract"):
|
||||
try:
|
||||
import pytesseract
|
||||
# 检查 tesseract 可执行文件
|
||||
result = os.popen("tesseract --version 2>&1").read()
|
||||
if "tesseract" in result.lower():
|
||||
self._ocr = pytesseract
|
||||
self.engine_type = "tesseract"
|
||||
# 设置中文
|
||||
pytesseract.pytesseract.tesseract_cmd = (
|
||||
r"C:\Program Files\Tesseract-OCR\tesseract.exe"
|
||||
)
|
||||
print("[INFO] OCR 引擎: Tesseract (chi_sim)")
|
||||
return
|
||||
except ImportError:
|
||||
pass
|
||||
|
||||
print("[WARN] 无可用 OCR 引擎！将跳过文字识别。")
|
||||
print(" 安装 PaddleOCR: pip install paddleocr paddlepaddle")
|
||||
print(" 或安装 Tesseract: https://github.com/UB-Mannheim/tesseract/wiki")
|
||||
|
||||
def recognize(self, image) -> List[Dict]:
|
||||
"""
|
||||
对图片执行 OCR 识别。
|
||||
|
||||
Args:
|
||||
image: PIL.Image 对象 或 图片文件路径
|
||||
|
||||
Returns:
|
||||
识别结果列表，每项包含:
|
||||
- text: 识别的文本
|
||||
- confidence: 置信度 (0-1)
|
||||
- box: 文字区域坐标 [[x1,y1],[x2,y2],[x3,y3],[x4,y4]]
|
||||
"""
|
||||
if not self._ocr:
|
||||
return []
|
||||
|
||||
results = []
|
||||
|
||||
if self.engine_type == "paddle":
|
||||
# PaddleOCR 接口
|
||||
img_path = image if isinstance(image, str) else None
|
||||
if img_path:
|
||||
output = self._ocr.ocr(img_path, cls=True)
|
||||
else:
|
||||
# PIL Image 需要保存为临时文件
|
||||
import tempfile
|
||||
with tempfile.NamedTemporaryFile(suffix=".png", delete=False) as f:
|
||||
image.save(f.name)
|
||||
output = self._ocr.ocr(f.name, cls=True)
|
||||
os.unlink(f.name)
|
||||
|
||||
if output and output[0]:
|
||||
for line in output[0]:
|
||||
box, (text, conf) = line
|
||||
results.append({
|
||||
"text": text.strip(),
|
||||
"confidence": float(conf),
|
||||
"box": box,
|
||||
})
|
||||
|
||||
elif self.engine_type == "tesseract":
|
||||
# Tesseract 接口
|
||||
img = image if hasattr(image, 'save') else None
|
||||
if isinstance(image, str):
|
||||
from PIL import Image
|
||||
img = Image.open(image)
|
||||
|
||||
data = self._ocr.image_to_data(
|
||||
img, lang="chi_sim+eng", output_type=self._ocr.Output.DICT
|
||||
)
|
||||
for i in range(len(data["text"])):
|
||||
if data["text"][i].strip():
|
||||
results.append({
|
||||
"text": data["text"][i].strip(),
|
||||
"confidence": float(data["conf"][i]) / 100,
|
||||
"box": [
|
||||
[data["left"][i], data["top"][i]],
|
||||
[data["left"][i] + data["width"][i], data["top"][i]],
|
||||
[data["left"][i] + data["width"][i],
|
||||
data["top"][i] + data["height"][i]],
|
||||
[data["left"][i], data["top"][i] + data["height"][i]],
|
||||
],
|
||||
})
|
||||
|
||||
return results
|
||||
|
||||
def recognize_text(self, image) -> str:
|
||||
"""
|
||||
识别图片中的所有文字，返回纯文本。
|
||||
|
||||
Args:
|
||||
image: PIL.Image 或 文件路径
|
||||
|
||||
Returns:
|
||||
识别的文本（换行分隔）
|
||||
"""
|
||||
results = self.recognize(image)
|
||||
return "\n".join(r["text"] for r in results if r["confidence"] > 0.5)
|
||||
|
||||
|
||||
# ============================================================
|
||||
# 微信聊天记录导出器
|
||||
# ============================================================
|
||||
|
||||
class WeChatExporter:
|
||||
"""
|
||||
微信聊天记录导出器
|
||||
|
||||
通过自动化操作微信 PC 版客户端，滚动截图并 OCR 识别，
|
||||
将聊天记录导出为结构化数据和图文 HTML 报告。
|
||||
"""
|
||||
|
||||
def __init__(self, output_dir: str = DEFAULT_OUTPUT_DIR,
|
||||
use_ocr: bool = True, use_screenshot: bool = True,
|
||||
scroll_pause: float = DEFAULT_SCROLL_PAUSE):
|
||||
"""
|
||||
Args:
|
||||
output_dir: 输出目录
|
||||
use_ocr: 是否启用 OCR 文字识别
|
||||
use_screenshot: 是否保存截图
|
||||
scroll_pause: 每次滚动间隔（秒）
|
||||
"""
|
||||
self.auto = DesktopAuto()
|
||||
self.output_dir = Path(output_dir)
|
||||
self.output_dir.mkdir(parents=True, exist_ok=True)
|
||||
|
||||
# 子目录
|
||||
self.screenshot_dir = self.output_dir / "screenshots"
|
||||
self.screenshot_dir.mkdir(exist_ok=True)
|
||||
|
||||
self.use_ocr = use_ocr
|
||||
self.use_screenshot = use_screenshot
|
||||
self.scroll_pause = scroll_pause
|
||||
|
||||
# OCR 引擎
|
||||
self.ocr = OCREngine() if use_ocr else None
|
||||
|
||||
# 消息去重索引
|
||||
self._seen_hashes = set()
|
||||
|
||||
# 微信窗口信息
|
||||
self.wechat_info: Optional[WindowInfo] = None
|
||||
self.chat_area_rect: Optional[Tuple[int, int, int, int]] = None
|
||||
|
||||
def _compute_chat_area(self) -> Tuple[int, int, int, int]:
|
||||
"""
|
||||
计算聊天内容区域在屏幕上的绝对坐标。
|
||||
|
||||
根据微信窗口位置和 CHAT_AREA_RATIO 比例计算。
|
||||
返回 (left, top, width, height) 格式。
|
||||
"""
|
||||
if not self.wechat_info:
|
||||
raise RuntimeError("微信窗口未找到")
|
||||
|
||||
rect = self.wechat_info.rect # (left, top, right, bottom)
|
||||
win_left, win_top, win_right, win_bottom = rect
|
||||
win_width = win_right - win_left
|
||||
win_height = win_bottom - win_top
|
||||
|
||||
# 按比例计算聊天区域
|
||||
left = int(win_left + win_width * CHAT_AREA_RATIO["left"])
|
||||
top = int(win_top + win_height * CHAT_AREA_RATIO["top"])
|
||||
right = int(win_left + win_width * CHAT_AREA_RATIO["right"])
|
||||
bottom = int(win_top + win_height * CHAT_AREA_RATIO["bottom"])
|
||||
|
||||
return (left, top, right - left, bottom - top)
|
||||
|
||||
def _msg_hash(self, msg: MessageItem) -> str:
|
||||
"""计算消息哈希用于去重"""
|
||||
content = f"{msg.sender}|{msg.content}|{msg.timestamp or ''}"
|
||||
return hashlib.md5(content.encode()).hexdigest()
|
||||
|
||||
def _parse_messages_from_ocr(self, ocr_results: List[Dict],
|
||||
screenshot_path: str = None) -> List[MessageItem]:
|
||||
"""
|
||||
从 OCR 结果解析结构化消息。
|
||||
|
||||
微信聊天界面的文字布局特征：
|
||||
- 消息按时间线从上到下排列
|
||||
- 每条消息通常包含发送者名称和消息内容
|
||||
- 自己发送的消息在右侧（绿色气泡），对方在左侧（白色气泡）
|
||||
- 时间戳通常独立一行，居中显示
|
||||
|
||||
此方法使用启发式规则进行解析，准确率约 80-90%。
|
||||
对于复杂消息（图片、文件、链接卡片），标记为 unknown 类型。
|
||||
|
||||
Args:
|
||||
ocr_results: OCR 识别结果列表
|
||||
screenshot_path: 对应的截图路径
|
||||
|
||||
Returns:
|
||||
解析出的消息列表
|
||||
"""
|
||||
messages = []
|
||||
|
||||
if not ocr_results:
|
||||
return messages
|
||||
|
||||
# 按垂直坐标排序（从上到下）
|
||||
sorted_results = sorted(ocr_results, key=lambda r: r["box"][0][1])
|
||||
|
||||
current_sender = ""
|
||||
current_text_lines = []
|
||||
|
||||
for i, item in enumerate(sorted_results):
|
||||
text = item["text"].strip()
|
||||
if not text:
|
||||
continue
|
||||
|
||||
box = item["box"]
|
||||
y = box[0][1]
|
||||
x = box[0][0]
|
||||
|
||||
# 启发式判断：时间戳（包含 : 且长度短）
|
||||
is_timestamp = (
|
||||
":" in text and len(text) < 20
|
||||
and any(c.isdigit() for c in text)
|
||||
)
|
||||
|
||||
# 启发式判断：发送者名称（短文本，不以标点开头）
|
||||
is_sender = (
|
||||
len(text) <= 10
|
||||
and not text.startswith((":", "：", "-", "—"))
|
||||
and not text.endswith(("。", "！", "？", ".", "!", "?"))
|
||||
)
|
||||
|
||||
if is_timestamp:
|
||||
# 保存当前消息（如果有）
|
||||
if current_text_lines:
|
||||
msg = MessageItem(
|
||||
sender=current_sender or "未知",
|
||||
content="\n".join(current_text_lines),
|
||||
timestamp=text,
|
||||
msg_type="text",
|
||||
screenshot_path=screenshot_path,
|
||||
raw_ocr="\n".join(r["text"] for r in sorted_results),
|
||||
)
|
||||
messages.append(msg)
|
||||
current_text_lines = []
|
||||
current_sender = ""
|
||||
continue
|
||||
|
||||
# 判断是否是新的发送者（行间距较大时可能是新消息）
|
||||
if i > 0:
|
||||
prev_y = sorted_results[i - 1]["box"][0][1]
|
||||
y_gap = y - prev_y
|
||||
if y_gap > 30: # 行间距大于30像素，可能是新消息
|
||||
if current_text_lines:
|
||||
msg = MessageItem(
|
||||
sender=current_sender or "未知",
|
||||
content="\n".join(current_text_lines),
|
||||
msg_type="text",
|
||||
screenshot_path=screenshot_path,
|
||||
raw_ocr="\n".join(r["text"] for r in sorted_results),
|
||||
)
|
||||
messages.append(msg)
|
||||
current_text_lines = []
|
||||
current_sender = ""
|
||||
|
||||
current_text_lines.append(text)
|
||||
|
||||
# 保存最后一条消息
|
||||
if current_text_lines:
|
||||
msg = MessageItem(
|
||||
sender=current_sender or "未知",
|
||||
content="\n".join(current_text_lines),
|
||||
msg_type="text",
|
||||
screenshot_path=screenshot_path,
|
||||
raw_ocr="\n".join(r["text"] for r in sorted_results),
|
||||
)
|
||||
messages.append(msg)
|
||||
|
||||
return messages
|
||||
|
||||
# ========================================================
|
||||
# 导出流程
|
||||
# ========================================================
|
||||
|
||||
def find_wechat(self) -> bool:
|
||||
"""
|
||||
查找微信窗口。
|
||||
|
||||
Returns:
|
||||
True 找到并激活，False 未找到
|
||||
"""
|
||||
print("\n[1/5] 查找微信窗口...")
|
||||
|
||||
# 先尝试通过类名查找
|
||||
self.wechat_info = self.auto.find_window("微信")
|
||||
|
||||
if not self.wechat_info:
|
||||
print("[ERROR] 未找到微信窗口！请确保微信PC版已打开并登录。")
|
||||
return False
|
||||
|
||||
# 计算聊天区域
|
||||
self.chat_area_rect = self._compute_chat_area()
|
||||
print(f"[INFO] 聊天区域: {self.chat_area_rect}")
|
||||
|
||||
return True
|
||||
|
||||
def search_contact(self, contact_name: str) -> bool:
|
||||
"""
|
||||
在微信中搜索并打开指定联系人的聊天窗口。
|
||||
|
||||
Args:
|
||||
contact_name: 联系人名称或群聊名称
|
||||
|
||||
Returns:
|
||||
True 成功打开聊天，False 失败
|
||||
"""
|
||||
print(f"\n[2/5] 搜索联系人: {contact_name}...")
|
||||
|
||||
# 激活微信窗口
|
||||
if not self.auto.activate_window("微信"):
|
||||
return False
|
||||
|
||||
time.sleep(0.5)
|
||||
|
||||
# 使用 Ctrl+F 打开搜索
|
||||
self.auto.hotkey("ctrl", "f")
|
||||
time.sleep(0.8)
|
||||
|
||||
# 输入联系人名称（通过剪贴板支持中文）
|
||||
self.auto.type_text(contact_name)
|
||||
time.sleep(1.0)
|
||||
|
||||
# 等待搜索结果出现后按 Enter 打开
|
||||
self.auto.press_key("enter")
|
||||
time.sleep(1.5)
|
||||
|
||||
print(f"[INFO] 已打开与 '{contact_name}' 的聊天窗口")
|
||||
return True
|
||||
|
||||
def scroll_to_top(self, max_scrolls: int = 30):
|
||||
"""
|
||||
滚动到聊天记录顶部。
|
||||
|
||||
通过持续向上滚动直到内容不再变化来检测顶部。
|
||||
|
||||
Args:
|
||||
max_scrolls: 最大滚动次数，防止无限滚动
|
||||
"""
|
||||
print("\n[3/5] 滚动到聊天顶部...")
|
||||
|
||||
# 先点击聊天区域确保焦点在此
|
||||
if self.chat_area_rect:
|
||||
left, top, width, height = self.chat_area_rect
|
||||
self.auto.click(left + width // 2, top + height // 2)
|
||||
time.sleep(0.3)
|
||||
|
||||
prev_screenshot_hash = None
|
||||
for i in range(max_scrolls):
|
||||
# 向上滚动
|
||||
self.auto.scroll(5)
|
||||
time.sleep(self.scroll_pause * 0.5)
|
||||
|
||||
# 截图并计算哈希判断是否到顶
|
||||
img = self.auto.screenshot_region(self.chat_area_rect)
|
||||
current_hash = hashlib.md5(img.tobytes()).hexdigest()
|
||||
|
||||
if current_hash == prev_screenshot_hash:
|
||||
print(f"[INFO] 已到达聊天顶部（{i+1}次滚动）")
|
||||
break
|
||||
prev_screenshot_hash = current_hash
|
||||
|
||||
else:
|
||||
print(f"[WARN] 未检测到顶部，已达最大滚动次数 {max_scrolls}")
|
||||
|
||||
def capture_messages(self, count: int) -> List[MessageItem]:
|
||||
"""
|
||||
通过滚动截图捕获聊天消息。
|
||||
|
||||
Args:
|
||||
count: 目标消息条数
|
||||
|
||||
Returns:
|
||||
捕获到的消息列表（已去重）
|
||||
"""
|
||||
print(f"\n[4/5] 开始截图捕获消息（目标 {count} 条）...")
|
||||
|
||||
all_messages = []
|
||||
screenshot_index = 0
|
||||
no_new_count = 0 # 连续无新消息计数
|
||||
max_no_new = 5 # 连续无新消息超过此值则停止
|
||||
|
||||
while len(all_messages) < count and no_new_count < max_no_new:
|
||||
screenshot_index += 1
|
||||
screenshot_path = str(
|
||||
self.screenshot_dir / f"chat_{screenshot_index:04d}.png"
|
||||
)
|
||||
|
||||
# 截取聊天区域
|
||||
img = self.auto.screenshot_region(self.chat_area_rect)
|
||||
if self.use_screenshot:
|
||||
img.save(screenshot_path)
|
||||
|
||||
# OCR 识别
|
||||
if self.ocr:
|
||||
ocr_results = self.ocr.recognize(img)
|
||||
messages = self._parse_messages_from_ocr(
|
||||
ocr_results, screenshot_path if self.use_screenshot else None
|
||||
)
|
||||
|
||||
# 去重
|
||||
new_count = 0
|
||||
for msg in messages:
|
||||
h = self._msg_hash(msg)
|
||||
if h not in self._seen_hashes:
|
||||
self._seen_hashes.add(h)
|
||||
all_messages.append(msg)
|
||||
new_count += 1
|
||||
|
||||
if new_count == 0:
|
||||
no_new_count += 1
|
||||
else:
|
||||
no_new_count = 0
|
||||
print(f" 截图 {screenshot_index}: +{new_count} 条新消息 "
|
||||
f"(总计 {len(all_messages)}/{count})")
|
||||
else:
|
||||
# 无 OCR，仅保存截图
|
||||
all_messages.append(MessageItem(
|
||||
sender="unknown",
|
||||
content=f"[截图 {screenshot_index}]",
|
||||
msg_type="image",
|
||||
screenshot_path=screenshot_path,
|
||||
))
|
||||
|
||||
# 向下滚动
|
||||
self.auto.scroll(SCROLL_CLICKS)
|
||||
time.sleep(self.scroll_pause)
|
||||
|
||||
print(f"\n[INFO] 捕获完成: 共 {len(all_messages)} 条消息, "
|
||||
f"{screenshot_index} 张截图")
|
||||
|
||||
return all_messages
|
||||
|
||||
def export(self, contact_name: str, count: int = DEFAULT_COUNT) -> bool:
|
||||
"""
|
||||
完整导出流程。
|
||||
|
||||
Args:
|
||||
contact_name: 联系人名称
|
||||
count: 目标消息条数
|
||||
|
||||
Returns:
|
||||
True 成功，False 失败
|
||||
"""
|
||||
print("=" * 60)
|
||||
print(f" 微信聊天记录导出")
|
||||
print(f" 联系人: {contact_name}")
|
||||
print(f" 目标条数: {count}")
|
||||
print(f" 输出目录: {self.output_dir}")
|
||||
print(f" OCR: {'ON' if self.use_ocr else 'OFF'}")
|
||||
print(f" 截图: {'ON' if self.use_screenshot else 'OFF'}")
|
||||
print("=" * 60)
|
||||
|
||||
# Step 1: 查找微信
|
||||
if not self.find_wechat():
|
||||
return False
|
||||
|
||||
# Step 2: 搜索联系人
|
||||
if not self.search_contact(contact_name):
|
||||
return False
|
||||
|
||||
# Step 3: 滚动到顶部
|
||||
self.scroll_to_top()
|
||||
|
||||
# Step 4: 截图捕获消息
|
||||
messages = self.capture_messages(count)
|
||||
|
||||
# Step 5: 编译 HTML 报告
|
||||
print("\n[5/5] 编译 HTML 报告...")
|
||||
try:
|
||||
from html_compiler import ChatHTMLCompiler
|
||||
|
||||
compiler = ChatHTMLCompiler(output_dir=str(self.output_dir))
|
||||
compiler.add_messages(messages)
|
||||
if self.use_screenshot:
|
||||
compiler.add_screenshots_dir(str(self.screenshot_dir))
|
||||
|
||||
report_path = compiler.compile(
|
||||
f"聊天记录 - {contact_name}",
|
||||
contact_name,
|
||||
)
|
||||
print(f"\n[SUCCESS] HTML 报告已生成: {report_path}")
|
||||
except Exception as e:
|
||||
print(f"[WARN] HTML 编译失败: {e}")
|
||||
# 降级：保存 JSON
|
||||
json_path = self.output_dir / "messages.json"
|
||||
with open(json_path, "w", encoding="utf-8") as f:
|
||||
json.dump(
|
||||
[{"sender": m.sender, "content": m.content,
|
||||
"timestamp": m.timestamp, "type": m.msg_type,
|
||||
"screenshot": m.screenshot_path}
|
||||
for m in messages],
|
||||
f, ensure_ascii=False, indent=2
|
||||
)
|
||||
print(f"[INFO] 消息已保存为 JSON: {json_path}")
|
||||
|
||||
# 保存元数据
|
||||
meta = {
|
||||
"contact": contact_name,
|
||||
"export_time": datetime.now().isoformat(),
|
||||
"message_count": len(messages),
|
||||
"screenshot_count": len(list(self.screenshot_dir.glob("*.png"))),
|
||||
"output_dir": str(self.output_dir),
|
||||
}
|
||||
meta_path = self.output_dir / "export_meta.json"
|
||||
with open(meta_path, "w", encoding="utf-8") as f:
|
||||
json.dump(meta, f, ensure_ascii=False, indent=2)
|
||||
|
||||
print(f"\n导出完成！共 {len(messages)} 条消息。")
|
||||
print(f"输出目录: {self.output_dir}")
|
||||
return True
|
||||
|
||||
|
||||
# ============================================================
|
||||
# CLI 入口
|
||||
# ============================================================
|
||||
|
||||
def main():
|
||||
parser = argparse.ArgumentParser(
|
||||
description="微信聊天记录导出工具",
|
||||
formatter_class=argparse.RawDescriptionHelpFormatter,
|
||||
epilog="""
|
||||
示例:
|
||||
# 导出与"张三"的最近100条聊天记录
|
||||
python wechat_export.py --contact "张三" --count 100
|
||||
|
||||
# 导出到指定目录，不启用OCR
|
||||
python wechat_export.py --contact "工作群" --output ./output --no-ocr
|
||||
|
||||
# 调整滚动间隔（网络慢时增大）
|
||||
python wechat_export.py --contact "张三" --scroll-pause 2.0
|
||||
""",
|
||||
)
|
||||
parser.add_argument("--contact", required=True,
|
||||
help="好友或群聊名称（模糊匹配）")
|
||||
parser.add_argument("--count", type=int, default=DEFAULT_COUNT,
|
||||
help=f"要导出的消息条数（默认 {DEFAULT_COUNT}）")
|
||||
parser.add_argument("--output", default=DEFAULT_OUTPUT_DIR,
|
||||
help=f"输出目录（默认 {DEFAULT_OUTPUT_DIR}）")
|
||||
parser.add_argument("--scroll-pause", type=float, default=DEFAULT_SCROLL_PAUSE,
|
||||
help=f"每次滚动间隔秒数（默认 {DEFAULT_SCROLL_PAUSE}）")
|
||||
parser.add_argument("--no-ocr", action="store_true",
|
||||
help="禁用 OCR 文字识别（仅保存截图）")
|
||||
parser.add_argument("--no-screenshot", action="store_true",
|
||||
help="不保存截图")
|
||||
|
||||
args = parser.parse_args()
|
||||
|
||||
exporter = WeChatExporter(
|
||||
output_dir=args.output,
|
||||
use_ocr=not args.no_ocr,
|
||||
use_screenshot=not args.no_screenshot,
|
||||
scroll_pause=args.scroll_pause,
|
||||
)
|
||||
|
||||
success = exporter.export(args.contact, args.count)
|
||||
sys.exit(0 if success else 1)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@@ -0,0 +1,30 @@
|
||||
# 头像修复完成 — 后端代理方案
|
||||
|
||||
## 一句话
|
||||
通过新建后端头像代理端点 `/api/avatar/proxy`，不修改 nginx 安全头，解决 COEP / Mixed Content / CSP 三层拦截导致的头像不显示问题。
|
||||
|
||||
## 验证结果
|
||||
- **API 代理**：✅ 正常返回 image/jpeg（42KB / 65KB 测试均通过）
|
||||
- **浏览器控制台**：✅ 零 COEP / Mixed Content / CSP 错误
|
||||
- **Network 请求**：✅ 3 个头像全部通过 `/api/avatar/proxy?url=...` 200 加载
|
||||
- **视觉验证**：✅ 坐席工作台头像正常显示（左侧会话列表 + 顶部导航栏）
|
||||
|
||||
## 修改文件
|
||||
| 文件 | 操作 |
|
||||
|-----|------|
|
||||
| `backend/app/services/avatar_service.py` | 修改（新增 `wrap_avatar_url` + HTTP→HTTPS 转换） |
|
||||
| `backend/app/services/session_service.py` | 修改（3 个头像返回路径统一包装） |
|
||||
| `backend/app/api/avatar.py` | 新建（代理端点 + SSRF 白名单） |
|
||||
| `backend/app/api/router.py` | 修改（注册路由） |
|
||||
|
||||
## 已部署
|
||||
- 4 个文件已上传 → 复制到 `/opt/wecom-it-desk/app/` → `docker compose restart backend`
|
||||
- Redis 头像缓存 `employee:avatar:*` 已清除（3 个 key）
|
||||
- 代理端点即时生效
|
||||
|
||||
## 关于"获取坐席列表失败"
|
||||
本次浏览器验证未复现该错误。如持续出现，需检查后端日志中 `/api/agents` 的具体报错。
|
||||
|
||||
## 交付物
|
||||
- `deliverables/software-bugfix-avatar-coep-delivery-2026-07-10.md` — 详细修复报告
|
||||
- 截图 `screenshot-1783683005062.png` — 验证截图
|
||||
@@ -0,0 +1,77 @@
|
||||
# 头像显示修复报告 — 2026-07-10
|
||||
|
||||
## 问题摘要
|
||||
|
||||
坐席端（`https://itsupport.servyou.com.cn/itagent/workspace`）员工头像仍不显示，浏览器控制台报三类错误：
|
||||
|
||||
1. **COEP 阻断** — `ERR_BLOCKED_BY_RESPONSE.NotSameOriginAfterDefaultedToSameOriginByCoep`
|
||||
2. **Mixed Content** — HTTPS 页面不允许加载 HTTP 资源
|
||||
3. **CSP 限制** — `img-src` 白名单不含 `wework.qpic.cn`
|
||||
|
||||
## 根因分析
|
||||
|
||||
三层安全策略叠加导致企微头像域名 `wework.qpic.cn` 被完全拦截：
|
||||
|
||||
| 安全策略 | nginx 配置 | 拦截表现 |
|
||||
|---------|-----------|---------|
|
||||
| COEP | `require-corp` | 跨域图片无 CORS 头被拦截 |
|
||||
| Mixed Content | — | HTTP 图片在 HTTPS 页面被拦截 |
|
||||
| CSP | `img-src 'self' data: https://res.wx.qq.com` | `wework.qpic.cn` 不在白名单 |
|
||||
|
||||
## 修复方案：后端头像代理
|
||||
|
||||
**不修改 nginx 安全头**（保持安全级别不变），通过后端代理绕过所有三层限制：
|
||||
|
||||
```
|
||||
前端 img.src = "/api/avatar/proxy?url=<encoded_avatar_url>"
|
||||
|
|
||||
v
|
||||
后端 httpx.AsyncClient → wework.qpic.cn
|
||||
|
|
||||
v
|
||||
返回图片字节流 (同域, HTTPS, 不受 CSP 限制)
|
||||
```
|
||||
|
||||
## 修改文件
|
||||
|
||||
| 文件 | 操作 | 修改内容 |
|
||||
|-----|------|---------|
|
||||
| `backend/app/services/avatar_service.py` | 修改 | `clean_avatar_url` 增加 HTTP→HTTPS 转换；新增 `wrap_avatar_url` 代理 URL 包装函数 |
|
||||
| `backend/app/services/session_service.py` | 修改 | 3个头像返回路径统一应用 `wrap_avatar_url` |
|
||||
| `backend/app/api/avatar.py` | 新建 | 头像代理端点 `/api/avatar/proxy`，SSRF 白名单校验 `wework.qpic.cn`，`httpx.AsyncClient` 异步获取，返回 `Response(content=..., media_type="image/jpeg")` |
|
||||
| `backend/app/api/router.py` | 修改 | 注册 `avatar_router` 路由，前缀 `/api/avatar/proxy` |
|
||||
|
||||
## 验证结果
|
||||
|
||||
### 1. API 层（curl）
|
||||
|
||||
| 测试项 | 结果 |
|
||||
|-------|------|
|
||||
| 代理 HTTPS 头像 URL | ✅ HTTP 200, image/jpeg, 42KB |
|
||||
| 代理 HTTP 头像 URL（自动转 HTTPS） | ✅ HTTP 200, image/jpeg, 65KB |
|
||||
| 无效 URL（非企微域名） | ✅ 403 拒绝 |
|
||||
|
||||
### 2. 浏览器验证（agent-browser）
|
||||
|
||||
- ✅ 坐席工作台正常加载，无重定向到登录页
|
||||
- ✅ **3个头像全部通过代理加载**：
|
||||
- `GET /api/avatar/proxy?url=... (Image) 200` × 3
|
||||
- ✅ **控制台零错误**：无 COEP、无 Mixed Content、无 CSP 报错
|
||||
- ✅ **网络请求零 4xx/5xx**
|
||||
- ✅ 头像在左侧会话列表和顶部导航栏均正常显示
|
||||
|
||||
### 3. 截图证据
|
||||
|
||||
见 `screenshot-1783683005062.png` — 坐席工作台界面，头像正常显示在左侧列表和顶部用户信息区域。
|
||||
|
||||
## 部署摘要
|
||||
|
||||
1. 上传 4 个修改文件 → `/tmp/`
|
||||
2. `cp` 到 `/opt/wecom-it-desk/app/` 对应路径
|
||||
3. `docker compose restart backend`
|
||||
4. 清除 Redis 头像缓存 `employee:avatar:*`（3 个 key）
|
||||
5. 代理端点 `/api/avatar/proxy` 即时生效
|
||||
|
||||
## 关于"获取坐席列表失败"
|
||||
|
||||
本次浏览器验证中未复现该错误。该错误与头像代理无关，如持续出现需检查后端日志中 `/api/agents` 的具体报错（可能为认证或数据库问题）。
|
||||
Reference in New Issue
Block a user