DAY015 – 上传目录被普通文件占用

故障场景

故障现象：首页正常，API health 与订单接口返回 500，日志提示上传目录不可用。

服务器状态：

server2：状态 INJECTED；开始时间 2026-07-27 05:00:11 +0800；影响范围 演练项目受影响，SSH 不受影响，非演练业务不应受影响。

排障过程

1、看项目结构文档

- nginx 程序：/usr/sbin/nginx
- systemd 服务：nginx.service
- nginx 配置：/etc/nginx/sites-available/hermes-day015.conf
- nginx 启用配置：/etc/nginx/sites-enabled/hermes-day015.conf
- nginx 日志：/var/log/nginx/hermes-day015-access.log、/var/log/nginx/hermes-day015-error.log
- 应用目录：/opt/hermes-drills/day015/
- API 程序：/opt/hermes-drills/day015/app/api.py
- API 启动脚本：/opt/hermes-drills/day015/start-app.sh
- API 日志：/opt/hermes-drills/day015/logs/api.log
- 可执行场景套件：/home/monika/hermes-ops-drills/standard-scenario-suite.md

## 端口
- nginx：18080
- API：127.0.0.1:18081

2、看本地 curl 返回情况

monika@VM-0-3-ubuntu:~/hermes-ops-drills/DAY015$ curl -i 127.0.0.1:18080/health
HTTP/1.1 200 OK
Server: nginx/1.24.0 (Ubuntu)
Date: Mon, 27 Jul 2026 16:33:37 GMT
Content-Type: application/octet-stream
Content-Length: 9
Connection: keep-alive

nginx ok
monika@VM-0-3-ubuntu:~/hermes-ops-drills/DAY015$ curl -i 127.0.0.1:18080/api/health
HTTP/1.1 500 Internal Server Error
Server: nginx/1.24.0 (Ubuntu)
Date: Mon, 27 Jul 2026 16:33:43 GMT
Content-Type: application/json; charset=utf-8
Content-Length: 54
Connection: keep-alive

{"status": "error", "error": "upload dir unavailable"}
monika@VM-0-3-ubuntu:~/hermes-ops-drills/DAY015$ curl -i 127.0.0.1:18080/api/orders
HTTP/1.1 500 Internal Server Error
Server: nginx/1.24.0 (Ubuntu)
Date: Mon, 27 Jul 2026 16:33:51 GMT
Content-Type: application/json; charset=utf-8
Content-Length: 54
Connection: keep-alive

{"status": "error", "error": "upload dir unavailable"}

首页与 nginx health 正常，但 API health 与订单接口都返回 500，报错都是 upload dir unavailable，问题出在 API 侧的上传目录上。

3、看后端日志

monika@VM-0-3-ubuntu:~/hermes-ops-drills/DAY015$ tail -n 50 /opt/hermes-drills/day015/logs/api.log
2026-07-27 17:17:35 level=ERROR path=/health error=upload dir unavailable
2026-07-27 17:17:35 127.0.0.1 GET /health 500
2026-07-27 17:17:35 level=ERROR path=/orders error=upload dir unavailable
2026-07-27 17:17:35 127.0.0.1 GET /orders 500

日志显示也是“上传目录不可用”。

4、查上传目录相关配置

monika@VM-0-3-ubuntu:~/hermes-ops-drills/DAY015$ grep -C 5 -n -R "upload" /opt/hermes-drills/day015/
api.py-16: data = {}
api.py-17: for line in CONFIG.read_text().splitlines():
api.py-18: if "=" in line and not line.strip().startswith("#"):
api.py-19: key, value = line.split("=", 1)
api.py-20: data[key.strip()] = value.strip()
api.py-21: upload = (BASE / data.get("UPLOAD_DIR", "../data/uploads")).resolve()
api.py-22: if not upload.exists() or not upload.is_dir():
api.py-23: raise RuntimeError("upload dir unavailable")
api.py-24: probe = upload / ".write-test"
api.py-25: probe.write_text(str(time.time()))
api.py-26: probe.unlink(missing_ok=True)
api.py-27: return data
api.py-29: class Handler(BaseHTTPRequestHandler):
api_config.env-1: APP_NAME=huadong-erp-day015-api
api_config.env-2: UPLOAD_DIR=../data/uploads
api_config.env-3: ORDERS_DB=../data/orders.db
api_config.env-4: SLOW_ORDERS_SECONDS=0

代码逻辑是“当上传目录不存在或者非目录”时返回 upload dir unavailable；环境变量里上传目录是 ../data/uploads。顺着路径线索查看上传目录具体情况。

5、查看上传目录，发现 uploads 变成文件了

monika@VM-0-3-ubuntu:/opt/hermes-drills/day015/data$ ls -lh
total 16K
-rw-rw-r-- 1 monika monika 12K Jul 27 05:00 orders.db
-rw-rw-r-- 1 monika monika 27 Jul 27 05:00 uploads

进来一看，uploads 变成文件了，ls -lh 返回结果最前面不是 d 而是 -。太荒唐了，听 AI 说脚本不够健壮或者写错就会发生这种事情。

在修改为目录前，还需要一些步骤：先查看文件类型，再确认有没有应用正在使用此 uploads 文件。

monika@VM-0-3-ubuntu:/opt/hermes-drills/day015/data$ file uploads
uploads: ASCII text
monika@VM-0-3-ubuntu:/opt/hermes-drills/day015/data$ sudo lsof uploads

lsof 没有返回结果，确认没有正在使用它的进程。

6、记录、备份，再重建目录

确认没有正在使用它的进程后，先记录此文件的修改时间；开了日志审计也应该记录相关这文件的操作，好知道这文件是谁的脚本什么时候生成的；再备份此文件。我没开日志审计，就只用 stat 记录时间信息了。

monika@VM-0-3-ubuntu:/opt/hermes-drills/day015/data$ stat uploads
  File: uploads
  Size: 27         Blocks: 8          IO Block: 4096   regular file
Device: 252,0    Inode: 393473     Links: 1
Access: 2026-07-27 05:00:11.000000000 +0800
Modify: 2026-07-27 05:00:11.000000000 +0800
Change: 2026-07-27 05:00:11.000000000 +0800
 Birth: -
monika@VM-0-3-ubuntu:/opt/hermes-drills/day015/data$ stat uploads > $(date +%F)_uploads_errorfile
monika@VM-0-3-ubuntu:/opt/hermes-drills/day015/data$ ls -lh
total 20K
-rw-rw-r-- 1 monika monika 375 Jul 28 01:04 2026-07-28_uploads_errorfile
-rw-rw-r-- 1 monika monika 12K Jul 27 05:00 orders.db
-rw-rw-r-- 1 monika monika 27 Jul 27 05:00 uploads
monika@VM-0-3-ubuntu:/opt/hermes-drills/day015/data$ cp -a uploads uploads_errorfile.bak
monika@VM-0-3-ubuntu:/opt/hermes-drills/day015/data$ rm uploads
monika@VM-0-3-ubuntu:/opt/hermes-drills/day015/data$ mkdir -p uploads
monika@VM-0-3-ubuntu:/opt/hermes-drills/day015/data$ ls -lh
total 24K
-rw-rw-r-- 1 monika monika 375 Jul 28 01:04 2026-07-28_uploads_errorfile
-rw-rw-r-- 1 monika monika 12K Jul 27 05:00 orders.db
drwxrwxr-x 2 monika monika 4.0K Jul 28 01:12 uploads
-rw-rw-r-- 1 monika monika 27 Jul 27 05:00 uploads_errorfile.bak
monika@VM-0-3-ubuntu:/opt/hermes-drills/day015/data$ sudo chmod 754 uploads

7、最后检查业务恢复状态

monika@VM-0-3-ubuntu:/opt/hermes-drills/day015/data$ curl -i 127.0.0.1:18080/api/orders
HTTP/1.1 200 OK
Server: nginx/1.24.0 (Ubuntu)
Date: Mon, 27 Jul 2026 17:14:22 GMT
Content-Type: application/json; charset=utf-8
Content-Length: 87
Connection: keep-alive

{"orders": [{"id": "SO-001", "status": "paid"}, {"id": "SO-002", "status": "pending"}]}
monika@VM-0-3-ubuntu:/opt/hermes-drills/day015/data$ curl -i 127.0.0.1:18080/api/health
HTTP/1.1 200 OK
Server: nginx/1.24.0 (Ubuntu)
Date: Mon, 27 Jul 2026 17:14:31 GMT
Content-Type: application/json; charset=utf-8
Content-Length: 49
Connection: keep-alive

{"status": "ok", "app": "huadong-erp-day015-api"}

内网公网都正常，业务恢复。

原文首发于个人博客：liujixiang.de5.net