Commit graph

20 commits

Author SHA1 Message Date
b4a2f303ac ci: add mypy job + pre-commit hook, tighten type annotations
- .github/workflows/ci.yml: new 'types' job runs mypy src/; docker now needs lint+types+test
- .pre-commit-config.yaml: add mirrors-mypy hook on src/ with types-requests, types-PyYAML
- pyproject.toml: add per-module overrides ignoring legacy modules (main, downloader, reddit_client, database, extractors.{reddit,gfycat}) while keeping src/web/ strict
- src/web/deps.py: type DOWNLOADS_DIR explicitly as Path
- src/web/config_manager.py: type guards on get_subreddits/users/blacklist returns
- src/web/routers/favorites.py: dict[str, object] for result, avoid in-place + on object
- src/web/routers/media.py: dict[str, object] for delete result
- src/web/rate_limit.py: move Awaitable/Callable under TYPE_CHECKING
- ruff format + auto-fix on src/ + tests/
2026-05-17 11:58:40 +01:00
34c2b16e19 test: cover routers, auth, rate limit, config_manager, imgur extractor
- tests/test_auth.py: enable/disable via env, valid/invalid creds
- tests/test_rate_limit.py: bucket math + window expiry
- tests/test_router_media.py: path traversal (file/thumb), paginated subreddits/authors, blacklist preview shape, DELETE flow, cleanup-by-type 400 paths
- tests/test_router_stats.py: cache TTL, .json exclusion, missing dir
- tests/test_router_favorites.py: paginated favorites/authors, no N+1 (query count), favorites_only filter
- tests/test_router_config.py: CRUD subreddits/users/blacklist
- tests/test_config_manager.py: case-insensitive dedupe, blacklist removes from collection, domain normalization
- tests/test_extractors_imgur.py: i.imgur direct, gifv->mp4, albums/galleries -> None

src/database.py: resolve RMC_DB_PATH lazily in Database() so tests can monkeypatch the env per case (no more module-import capture)

Result: 135 passed (was 77). Coverage 55% global; 100% on auth/imgur, 96% rate_limit, 84% stats router, 80% config_manager.
2026-05-17 11:54:40 +01:00
5f9c6023c1 perf: async I/O, caching, pagination, frontend fluidity, optional auth, rate limits
Backend:
- src/web/routers/stats.py: filesystem scan via asyncio.to_thread with 30s TTL cache
- src/web/routers/media.py: async thumbnail generation with per-file lock + semaphore(2)
- src/database.py: consolidate get_authors_with_stats (was N+1 favorites lookup) into a single query with EXISTS subquery
- src/web/routers/{media,favorites}.py: optional pagination for subreddits/authors/blacklist-preview/favorites-authors
- src/web/rate_limit.py (new): per-IP+path token bucket dependency
- Rate limits applied to /api/stats, /api/stats/enhanced (30/min), cleanup-blacklist, cleanup-by-type (5/min)
- src/web/auth.py (new): optional HTTP Basic auth, activated when RMC_AUTH_USER and RMC_AUTH_PASS are both set
- src/web/app.py: mount StaticFiles at /static, register auth dependency app-wide

Frontend:
- src/web/static/js/api.js (new): fetchJSON/debounce/throttle helpers on window.api
- index.html: debounced gallery filter onchange handlers (200ms)
- index.html: window eviction at 500 items in galleryItems (keeps DOM lean during long sessions)
- index.html: pause status polling on visibilitychange (no fetches when tab is hidden)
- index.html: loadStats() now throttled to 2s, with force=true used after delete/cleanup
- index.html: parallel batch DELETE with 5 workers + progress UI
- index.html: adapt to new response shape from paginated subreddits/authors endpoints
2026-05-17 11:50:29 +01:00
0402aeb06e fix(security): block path traversal in file/thumb endpoints
- Add _safe_resolve helper validating that resolved path stays within DOWNLOADS_DIR
- Apply to get_media_file and get_video_thumbnail
- Reject null bytes and overlong filenames
- Replace bare except in magic-byte fallback with OSError + logging
2026-05-17 11:43:21 +01:00
986f1dfef4 chore: checkpoint baseline (routers, tests, pyproject) 2026-05-17 11:40:35 +01:00
a58498a315
Update README.md
typos from en_us to en_uk
2026-02-11 19:59:41 +00:00
23e841e8d3 Update user-agent string to match project name 2025-09-20 14:30:00 -03:00
614da5b3c5 Add project documentation 2025-09-20 12:15:00 -03:00
99ef75703c Add web frontend SPA with dark theme dashboard and gallery 2025-09-18 16:00:00 -03:00
45ab33797c Add FastAPI web backend with REST API and scheduler 2025-09-16 10:30:00 -03:00
c0165f363d Add YAML configuration manager for web interface 2025-09-14 14:00:00 -03:00
c85bd56489 Add shell scripts and systemd service for deployment 2025-09-11 11:45:00 -03:00
b9bb2006ed Implement main collector entry point with filtering pipeline 2025-09-09 15:30:00 -03:00
1484c1f135 Add media URL extractors for Reddit, Imgur, and Gfycat 2025-09-07 10:00:00 -03:00
4e6a186484 Implement download manager with retry logic and hash deduplication 2025-09-05 13:30:00 -03:00
e110967094 Add Immich JSON sidecar generator for metadata integration 2025-09-03 09:20:00 -03:00
6bff87c98c Add SQLite database layer for post tracking and deduplication 2025-09-01 16:45:00 -03:00
ab751f469f Implement Reddit public JSON API client with rate limiting 2025-08-30 11:00:00 -03:00
d8c4f91d68 Add configuration schema and YAML loader 2025-08-28 14:15:00 -03:00
2cc5f01c52 Initial project structure and dependencies 2025-08-26 10:30:00 -03:00