diff --git a/README.md b/README.md
index f43895d..03871fc 100644
--- a/README.md
+++ b/README.md
@@ -1,3 +1,165 @@
-# compare_filesize_drupal6
+# Drupal 6 – duże obrazy, lokalizacja tematów i bezpieczna optymalizacja
+Rozbudowana wersja `compare_filesize_drupal6`. Działa **poza Drupalem**, w Pythonie 3.9+; potrzebuje dostępu do MySQL/MariaDB i katalogu plików.
+## Co robi
+
+- Wyszukuje największe pliki i największe rozdzielczości (JPG, PNG, GIF – GIF wyłącznie do analizy).
+- Wskazuje, **w jakim temacie, poście lub komentarzu** pojawia się obraz. Raport pokazuje `nid`, `cid`, tytuł, typ zawartości, URL do wpisu, URL do edycji i źródło powiązania.
+- Szuka połączeń przez `upload.fid` i pola `*_fid` w tabelach CCK/ImageField/FileField; obsługuje aktualne rewizje (`vid`) i opcjonalne tabele załączników komentarzy (`cid` + `fid`).
+- Szuka obrazów w `node_revisions.body` i `node_revisions.teaser`, `comments.comment`, opcjonalnie też w tekstowych polach CCK. Rozróżnia **pewne** dopasowania identyfikatora `fid` lub pełnej ścieżki od **możliwych** trafień po samej nazwie pliku.
+- Generuje link do oryginalnego pliku (jeżeli publiczny) i pomocniczy link do Drupal Search (jeżeli moduł Search jest aktywny).
+- Generuje CSV (polskie kolumny, Excel, UTF-8 BOM) i JSON.
+- Symuluje dokładną optymalizację w **dry-run**, bez zapisu źródłowego pliku i bazy.
+- Przy `--apply` zachowuje oryginał w oddzielnym katalogu, przygotowuje nowy obraz do atomowej podmiany, zapisuje `files.filesize`, a przy błędzie próbuje odtworzyć oryginał i wartości bazy.
+- Wykrywa duplikaty SHA-256 i pliki na dysku, których nie ma w analizowanym zestawieniu z bazy.
+
+## Instalacja
+
+```bash
+unzip compare_filesize_drupal6-extended.zip
+cd compare_filesize_drupal6
+python3 -m venv .venv
+source .venv/bin/activate
+pip install -r requirements.txt
+cp config.example.py config.py
+```
+
+W `config.py` ustaw:
+
+```python
+DB_CONFIG = {
+ 'host': 'localhost',
+ 'port': 3306,
+ 'database': 'drupal6_db',
+ 'user': 'drupal_user',
+ 'password': 'twoje_haslo',
+ 'charset': 'utf8',
+}
+FILES_BASE_PATH = '/var/www/drupal/sites/default/files'
+PUBLIC_FILES_PATH = 'sites/default/files'
+DRUPAL_ROOT = '/var/www/drupal'
+DRUPAL_BASE_URL = 'https://forum.example.org'
+DB_TABLE_PREFIX = ''
+```
+
+**Ważne:** `FILES_BASE_PATH` to lokalizacja rzeczywistych plików, a `PUBLIC_FILES_PATH` to fragment zapisany zwykle w `files.filepath`. Jeśli pliki są np. w `/var/www/drupal/files`, zmień obie wartości. Można też podać je w CLI: `--public-files-path`, `--drupal-root`. Gdy Drupal pracuje bez clean URLs, dodaj `--no-clean-urls`.
+
+## 1. Znajdź tematy i komentarze z największymi obrazami
+
+```bash
+python main.py --find-usage --top 50 --report-csv najwieksze-z-linkami.csv
+```
+
+Przykładowe wiersze (fikcyjne):
+
+```text
+ 1. 18.42 MB 4200x5900 fid=1234 scans/mapa.jpg
+ -> Archiwalne skany (forum, nid=812, pewne)
+ LINK: https://forum.example.org/node/812
+ EDYCJA: https://forum.example.org/node/812/edit
+ SKAD: upload.fid, tresc wpisu (sciezka)
+ -> Zdjęcie w komentarzu (forum, nid=920, cid=42, pewne)
+ LINK: https://forum.example.org/node/920#comment-42
+```
+
+CSV zawiera jedną linię dla każdego znalezionego miejsca. Gdy powiązania brak, w CSV zostaje plik, jego `fid`, rozmiar i propozycja wyszukiwania nazwą w Drupal Search.
+
+### Tylko skany lub wybrane pliki
+
+```bash
+python main.py --find-usage --top 100 --path-contains skan --report-csv skany.csv
+python main.py --find-usage --fid 1234 --report-csv jeden-obraz.csv
+python main.py --find-usage --min-size-mb 5 --usage-scope all --report-csv wieksze-niz-5mb.csv
+```
+
+`--usage-scope top` szuka tylko w `--top N` największych obrazach; `optimizable` – w obrazach przekraczających limity; `all` – w całym filtrowanym zestawieniu. Przy bardzo dużych bazach `all` może oznaczać długi skan treści. `--no-text-scan` wyłącza szukanie w treściach i zostawia szybsze powiązania `fid`. `--scan-cck-text` rozszerza skan na tekstowe pola CCK.
+
+## 2. Dokładny dry-run optymalizacji
+
+```bash
+python main.py \
+ --find-usage --usage-scope optimizable \
+ --optimize --dry-run --optimize-top 20 \
+ --max-width 2400 --max-height 3200 --max-file-mb 2 \
+ --report-csv plan.csv --report-json plan.json
+```
+
+Dostaniesz rzeczywisty przewidywany rozmiar po rekompresji (test wykonany na pliku tymczasowym). **Nie zmieni to oryginałów ani SQL.** `--top` dotyczy wyświetlania; `--optimize-top` ogranicza liczbę obrazów poddawanych testowi/zmniejszaniu.
+
+## 3. Zmniejsz np. pierwsze 10 obrazów i aktualizuj bazę
+
+**Najpierw zrób pełen backup bazy danych i katalogu plików** oraz przetestuj na kopii. Następnie:
+
+```bash
+python main.py \
+ --optimize --apply \
+ --optimize-top 10 --max-changes 10 \
+ --backup-dir /srv/backups/drupal6-images \
+ --max-width 2400 --max-height 3200 --max-file-mb 2 \
+ --report-csv wykonane.csv --report-json wykonane.json
+```
+
+Każda udana zmiana:
+
+1. Zapisuje oryginał w osobnym katalogu (automatycznie tworzony podfolder uruchomienia).
+2. Zastępuje obraz tylko jeżeli nowy plik jest mniejszy o co najmniej `--min-gain-percent` (domyślnie 5%).
+3. Aktualizuje `files.filesize` dla wszystkich `fid` wskazujących na tę samą ścieżkę.
+4. W przypadku błędu SQL próbuje odtworzyć oryginalny plik i rozmiar w DB (także przy MyISAM – kompensacja aktualizacji).
+
+`--apply` jest wymagane do każdego zapisu. Samo `--optimize` lub `--update-db` nie wykonuje zmian. **Realna optymalizacja zawsze aktualizuje rozmiar w DB**, także bez `--update-db`.
+
+Jeżeli chcesz wyłącznie zsynchronizować rozmiary plików w istniejącym `files`:
+
+```bash
+python main.py --update-db --dry-run
+python main.py --update-db --apply
+```
+
+`--update-db` w połączeniu z `--optimize --apply` dodatkowo koryguje rozmiary tych plików, które *nie zostały* zoptymalizowane.
+
+## Zasady jakości i bezpieczeństwa
+
+- **JPEG:** zachowuje rozszerzenie i ścieżkę, skaluje proporcjonalnie filtrem LANCZOS, zapisuje z wysoką jakością 92/89/86/85 (`--jpeg-quality`, `--min-jpeg-quality`). Nie schodzi poniżej skonfigurowanej jakości, nawet gdy cel 2 MB nie zostanie osiągnięty. JPEG jest formatem stratnym – każda ponowna kompresja może zmienić piksele.
+- **PNG:** zachowuje format oraz kanał przezroczystości, używa kompresji **bezstratnej**. Nie konwertuje PNG do JPEG i nie zmienia rozszerzeń, bo mogłoby to popsuć stare linki w Drupalu. Duży PNG ze skanem może pozostać duży, jeśli kompresja bezstratna nic nie da. Konwersja do JPEG wymagałaby osobnej, kontrolowanej migracji i aktualizacji odnośników.
+- **Skan tekstowy:** domyślny limit **2400 × 3200 px** zamiast starego 1920 × 1080. Przed decyzją sprawdź czytelność drobnego druku w próbce i w razie potrzeby zwiększ limity lub pomiń plik.
+- **GIF/animacje, nieobsługiwane formaty, niezgodności formatu i rozszerzenia:** pomijane przy zapisie. Symlinki i hardlinki też są pomijane.
+- Kopia oryginału wymagana jest przy `--optimize --apply`, a lokalizacja kopii **musi być poza** `FILES_BASE_PATH` i poza katalogiem WWW Drupala (jeśli ustawiono `DRUPAL_ROOT`).
+- Weryfikacja czasu modyfikacji i inode chroni przed zmianą pliku w trakcie skanowania; zalecane jest wykonywanie pracy w oknie serwisowym.
+- Zachowuje ICC, DPI, EXIF (z prawidłową obsługą orientacji) i tekstowe metadane PNG w obsługiwanych formatach, ale nie gwarantuje zachowania każdego niestandardowego segmentu metadanych.
+- Aktualizowany jest tylko `files.filesize` – nie są zmieniane ścieżki, `fid`, treści postów ani załączniki. Zmiana dużego obrazka może wymagać ręcznego odświeżenia wygenerowanych miniatur przez ImageCache.
+- Kopia wykonana przez samo narzędzie obejmuje **obraz**, nie całą bazę. MyISAM nie zapewnia transakcji; procedura przywracania jest best-effort. Pełny backup DB i plików przed zmianami jest obowiązkowy.
+
+## Pozostałe opcje
+
+```bash
+python main.py --find-duplicates
+python main.py --show-deleted --exclude-folders imagecache styles
+python main.py --find-usage --no-clean-urls --site-url https://example.org/drupal
+python main.py --help
+```
+
+Uwaga do `--show-deleted`: pokazuje pliki na dysku **bez pasującego wpisu `files` w analizowanym zestawieniu**, nie zaś potwierdzenie, że plik naprawdę nie występuje w treści. Przy filtrach (`--path-contains`, `--min-size-mb`, `--fid`) plik odfiltrowany może zostać pokazany jako „spoza zestawienia”. Nic nie jest usuwane.
+
+### Ograniczenia wyszukiwania
+
+- Powiązania po `fid` są najbardziej jednoznaczne; wyszukiwanie po samej nazwie pliku bywa niejednoznaczne – w raporcie `pewnosc=mozliwe`.
+- Tekst przeszukiwany jest w **aktualnych rewizjach** węzłów i komentarzach. Nie szuka w archiwalnych rewizjach ani w niestandardowych zewnętrznych modułach przechowywania treści bez włączonego skanu CCK.
+- Link do nieopublikowanego wpisu lub komentarza może wymagać konta z prawami administracyjnymi. Raport to zaznacza.
+- **Drupal 6 stronicuje komentarze**: `node/NID#comment-CID` może nie przewinąć do komentarza z dalszej strony. Jeżeli masz zainstalowany moduł [Permalink dla Drupal 6](https://www.drupal.org/project/permalink), dodaj `--comment-permalinks`, aby generować `comment/CID#comment-CID`, które kieruje na właściwą stronę komentarzy. Bez tego modułu nie używaj tej flagi. Niezależnie od typu linku raport pokazuje `nid` i `cid`.
+- Nie ma gwarancji odnalezienia obrazów w niestandardowych relacjach tworzonych przez własne moduły, w CSS i w HTML poza bazą.
+- W przypadku prywatnego systemu plików adres bezpośredni do pliku może nie działać; odnośnik do tematu nadal powinien wskazać miejsce.
+
+## Testy
+
+```bash
+python -m unittest discover -s tests -v
+```
+
+Testy z lokalną, symulowaną bazą weryfikują powiązania Upload/CCK, obecne rewizje, komentarze, brak zmian w dry-run, aktualizację `files.filesize`, backup, przywracanie po błędzie SQL, format PNG i ścieżki katalogów. **Nie zastępują próby na kopii konkretnej instalacji Drupal 6**.
+
+### Źródła struktury bazy Drupal 6
+
+- [Oficjalny migrator Drupal 6 Upload](https://api.drupal.org/api/drupal/core%21modules%21file%21src%21Plugin%21migrate%21source%21d6%21Upload.php/8.9.x) – `upload.fid`, `upload.nid`, `upload.vid`.
+- [Oficjalny migrator komentarzy D6](https://api.drupal.org/api/drupal/core%21modules%21comment%21src%21Plugin%21migrate%21source%21d6%21Comment.php/class/Comment/main) – tabela `comments`, `cid`, `nid`, status komentarza.
+- [CCK 6](https://api.druphelp.com/api/cck/content.module/6) – `content_type_*`, `content_field_*`.
diff --git a/config.example.py b/config.example.py
index ac57154..00e7446 100644
--- a/config.example.py
+++ b/config.example.py
@@ -1,10 +1,19 @@
-# config.py
-
+# Skopiuj do config.py i dostosuj. Nigdy nie dodawaj config.py do Git.
DB_CONFIG = {
'host': 'localhost',
+ 'port': 3306,
'database': 'drupal6_db',
'user': 'drupal_user',
'password': 'secure_password',
+ # Stary MySQL/Drupal 6 czesto wymaga utf8 zamiast utf8mb4.
+ 'charset': 'utf8',
}
-FILES_BASE_PATH = '/ścieżka/do/drupala/sites/default/files/'
+# Katalog, w ktorym fizycznie znajduja sie obrazy (moze byc inny w Drupal 6).
+FILES_BASE_PATH = '/var/www/drupal/sites/default/files'
+# Publiczny fragment sciezki do files/ zapisywany w tabeli files.filepath:
+PUBLIC_FILES_PATH = 'sites/default/files'
+# Katalog projektu Drupal: opcjonalny, pomaga dla sciezek pelnych.
+DRUPAL_ROOT = '/var/www/drupal'
+DRUPAL_BASE_URL = 'https://forum.example.org'
+DB_TABLE_PREFIX = '' # np. 'drupal_'
diff --git a/drupal_lookup.py b/drupal_lookup.py
new file mode 100644
index 0000000..585527a
--- /dev/null
+++ b/drupal_lookup.py
@@ -0,0 +1,325 @@
+"""Wyszukiwanie powiazan plikow z wpisami, tematami forum i komentarzami Drupal 6.
+
+Powiazania 'bezposrednie' to identyfikatory fid (Upload, CCK/FileField/ImageField).
+Powiazania z tekstu to poszlaki: pelna sciezka (mocniejsze) lub sama nazwa.
+"""
+from __future__ import annotations
+
+import html
+import os
+import re
+from collections import defaultdict
+from urllib.parse import quote
+
+
+IDENT = re.compile(r'^[A-Za-z_][A-Za-z0-9_]*$')
+
+
+def sql_ident(value):
+ if not IDENT.fullmatch(value):
+ raise ValueError(f'Niedozwolona nazwa tabeli/kolumny: {value!r}')
+ return f'`{value}`'
+
+
+def chunks(seq, size=250):
+ seq = list(seq)
+ for start in range(0, len(seq), size):
+ yield seq[start:start + size]
+
+
+def node_url(base_url, nid, cid=None, *, clean_urls=True, comment_permalinks=False):
+ if cid is not None and comment_permalinks:
+ base = (base_url.rstrip('/') + '/') if base_url else ''
+ if clean_urls:
+ return f'{base}comment/{int(cid)}#comment-{int(cid)}'
+ return f'{base}?q=comment/{int(cid)}#comment-{int(cid)}'
+ if not base_url:
+ result = f'node/{int(nid)}'
+ elif clean_urls:
+ result = f'{base_url.rstrip("/")}/node/{int(nid)}'
+ else:
+ result = f'{base_url.rstrip("/")}/?q=node/{int(nid)}'
+ if cid is not None:
+ result += f'#comment-{int(cid)}'
+ return result
+
+
+def search_url(base_url, filename, *, clean_urls=True):
+ if not base_url:
+ return ''
+ term = quote(filename, safe='')
+ if clean_urls:
+ return f'{base_url.rstrip("/")}/search/node/{term}'
+ return f'{base_url.rstrip("/")}/?q=search/node/{term}'
+
+
+class ReferenceMatcher:
+ """Jedno przejscie przez tresc na liste interesujacych plikow.
+
+ Dopasowanie tylko nazwy pliku ma pewnosc 'mozliwe', nigdy 'pewne'.
+ """
+
+ def __init__(self, images, basename_match=True):
+ variants = defaultdict(list)
+ for image in images:
+ fid = int(image['fid'])
+ file_path = (image.get('db_filepath') or '').replace('\\', '/').lstrip('/')
+ relative = (image.get('rel_path') or '').replace('\\', '/').lstrip('/')
+ for path in (file_path, relative):
+ if path and '/' in path:
+ for variant in (path, quote(path, safe='/')):
+ variants[variant.casefold()].append((fid, 'sciezka'))
+ name = os.path.basename(relative or file_path)
+ if basename_match and name and len(name) >= 5:
+ for variant in (name, quote(name, safe='')):
+ variants[variant.casefold()].append((fid, 'nazwa'))
+ self.variants = dict(variants)
+ # Najpierw dluzsze wzorce, np. 'uploads/scan.jpg' przed 'scan.jpg'.
+ self.pattern = None
+ if variants:
+ ordered = sorted(variants, key=lambda x: (-len(x), x))
+ self.pattern = re.compile('|'.join(re.escape(x) for x in ordered), re.IGNORECASE)
+
+ def match(self, text):
+ if not self.pattern or not text:
+ return {}
+ text = html.unescape(str(text))
+ found = {}
+ for occurrence in self.pattern.finditer(text):
+ word = occurrence.group(0).casefold()
+ for fid, kind in self.variants.get(word, []):
+ if kind == 'nazwa':
+ before = text[occurrence.start() - 1] if occurrence.start() else ''
+ after = text[occurrence.end()] if occurrence.end() < len(text) else ''
+ if before and (before.isalnum() or before in '._-'):
+ continue
+ if after and (after.isalnum() or after in '._-'):
+ continue
+ # Preferuj trafienie przez pelna sciezke od samej nazwy.
+ if fid not in found or (found[fid] == 'nazwa' and kind == 'sciezka'):
+ found[fid] = kind
+ return found
+
+
+class DrupalReferenceLookup:
+ def __init__(self, connection, *, table_prefix='', base_url='', clean_urls=True, comment_permalinks=False):
+ self.connection = connection
+ self.prefix = table_prefix
+ self.base_url = base_url
+ self.clean_urls = clean_urls
+ self.comment_permalinks = comment_permalinks
+ if table_prefix and not IDENT.fullmatch(table_prefix):
+ raise ValueError('Nieprawidlowy prefix tabel')
+ self.tables = set()
+ self.columns = {}
+
+ def table(self, name):
+ return f'{self.prefix}{name}'
+
+ def inspect_schema(self):
+ with self.connection.cursor() as cursor:
+ cursor.execute('SHOW TABLES')
+ self.tables = {str(next(iter(row.values()))) for row in cursor.fetchall()}
+ if self.table('node') not in self.tables:
+ raise RuntimeError('Nie znaleziono tabeli node; sprawdz --table-prefix')
+ for table in sorted(self.tables):
+ if self.prefix and not table.startswith(self.prefix):
+ continue
+ if not IDENT.fullmatch(table):
+ continue
+ cursor.execute(f'SHOW COLUMNS FROM {sql_ident(table)}')
+ self.columns[table] = {row['Field']: row['Type'].lower() for row in cursor.fetchall()}
+
+ def _add(self, refs, fid, row, source, confidence='pewne', cid=None):
+ fid = int(fid)
+ nid = int(row['nid'])
+ cid = int(cid) if cid is not None else None
+ key = (nid, cid)
+ if key not in refs[fid]:
+ refs[fid][key] = {
+ 'nid': nid,
+ 'cid': cid,
+ 'title': row.get('title') or '(bez tytulu)',
+ 'node_type': row.get('node_type') or '',
+ 'published': int(row.get('node_status', 1)) == 1,
+ 'comment_published': (int(row['comment_status']) == 0 if row.get('comment_status') is not None else None),
+ 'url': node_url(self.base_url, nid, cid, clean_urls=self.clean_urls,
+ comment_permalinks=self.comment_permalinks),
+ 'edit_url': node_url(self.base_url, nid, clean_urls=self.clean_urls) + '/edit' if self.clean_urls else (
+ f'{self.base_url.rstrip("/")}/?q=node/{nid}/edit' if self.base_url else f'node/{nid}/edit'),
+ 'sources': [],
+ 'confidence': confidence,
+ }
+ ref = refs[fid][key]
+ if source not in ref['sources']:
+ ref['sources'].append(source)
+ if confidence == 'pewne':
+ ref['confidence'] = 'pewne'
+
+ def _direct_relations(self, refs, fids):
+ node = sql_ident(self.table('node'))
+ # Upload + kazda tabela z nid i fid lub XXX_fid (CCK, filefield...).
+ for table, cols in sorted(self.columns.items()):
+ if not table.startswith(self.prefix):
+ continue
+ if 'nid' not in cols or table == self.table('node'):
+ continue
+ fid_cols = [col for col in cols if col == 'fid' or col.endswith('_fid')]
+ if not fid_cols:
+ continue
+ if not (table == self.table('upload') or table.startswith(self.table('content_')) or 'fid' in cols or any(col.startswith('field_') for col in fid_cols)):
+ continue
+ try:
+ with self.connection.cursor() as cursor:
+ for fid_column in fid_cols:
+ revision_join = ' AND n.vid = t.vid' if 'vid' in cols else ''
+ for batch in chunks(fids):
+ placeholders = ','.join(['%s'] * len(batch))
+ query = (
+ f'SELECT n.nid AS nid, n.title AS title, n.type AS node_type, '
+ f'n.status AS node_status, t.{sql_ident(fid_column)} AS fid, '
+ f'{"t.cid" if "cid" in cols else "NULL"} AS cid '
+ f'FROM {sql_ident(table)} t JOIN {node} n ON n.nid = t.nid'
+ f'{revision_join} WHERE t.{sql_ident(fid_column)} IN ({placeholders})'
+ )
+ cursor.execute(query, tuple(batch))
+ for row in cursor.fetchall():
+ if row.get('fid') is not None:
+ source = f'{table}.{fid_column}'
+ self._add(refs, row['fid'], row, source, cid=row.get('cid'))
+ except Exception as exc:
+ print(f'[WARN] Nie udalo sie odczytac powiazan {table}: {exc}')
+
+ def _comment_direct_relations(self, refs, fids):
+ # W niektorych instalacjach zalaczniki do komentarzy maja tylko cid+fid.
+ comments = self.table('comments')
+ if comments not in self.columns:
+ return
+ for table, cols in sorted(self.columns.items()):
+ if table in (comments, self.table('node')) or 'cid' not in cols or 'nid' in cols:
+ continue
+ fid_cols = [col for col in cols if col == 'fid' or col.endswith('_fid')]
+ if not fid_cols:
+ continue
+ # Nie interpretuj losowych tabel jako zalacznikow do komentarzy.
+ if 'comment' not in table:
+ continue
+ try:
+ with self.connection.cursor() as cursor:
+ for fid_column in fid_cols:
+ for batch in chunks(fids):
+ placeholders = ','.join(['%s'] * len(batch))
+ query = (
+ f'SELECT n.nid AS nid, n.title AS title, n.type AS node_type, '
+ f'n.status AS node_status, c.status AS comment_status, '
+ f'c.cid AS cid, t.{sql_ident(fid_column)} AS fid '
+ f'FROM {sql_ident(table)} t '
+ f'JOIN {sql_ident(comments)} c ON c.cid = t.cid '
+ f'JOIN {sql_ident(self.table("node"))} n ON n.nid = c.nid '
+ f'WHERE t.{sql_ident(fid_column)} IN ({placeholders})'
+ )
+ cursor.execute(query, tuple(batch))
+ for row in cursor.fetchall():
+ if row.get('fid') is not None:
+ self._add(refs, row['fid'], row, f'{table}.{fid_column}', cid=row['cid'])
+ except Exception as exc:
+ print(f'[WARN] Nie udalo sie odczytac zalacznikow do komentarzy {table}: {exc}')
+
+ def _stream_query(self, query):
+ # Kursor strumieniowy nie pobiera do RAM calego body z wielkiej bazy.
+ import pymysql
+ cursor = self.connection.cursor(pymysql.cursors.SSDictCursor)
+ try:
+ cursor.execute(query)
+ while True:
+ batch = cursor.fetchmany(200)
+ if not batch:
+ break
+ for row in batch:
+ yield row
+ finally:
+ cursor.close()
+
+ def _scan_body(self, refs, matcher):
+ revision = self.table('node_revisions')
+ if revision not in self.columns or 'body' not in self.columns[revision]:
+ print('[WARN] Brak node_revisions.body - pomijam tekst wpisow')
+ return
+ teaser_field = ('r.teaser AS teaser' if 'teaser' in self.columns[revision]
+ else 'NULL AS teaser')
+ query = (
+ f'SELECT n.nid, n.title, n.type AS node_type, n.status AS node_status, '
+ f'r.body AS body, {teaser_field} FROM {sql_ident(self.table("node"))} n '
+ f'JOIN {sql_ident(revision)} r ON r.vid = n.vid AND r.nid = n.nid '
+ f'WHERE (r.body IS NOT NULL AND r.body <> \'\')'
+ + (" OR (r.teaser IS NOT NULL AND r.teaser <> '')" if 'teaser' in self.columns[revision] else '')
+ )
+ for row in self._stream_query(query):
+ for field, source in (('body', 'tresc wpisu'), ('teaser', 'zajawka wpisu')):
+ for fid, kind in matcher.match(row.get(field)).items():
+ confidence = 'pewne' if kind == 'sciezka' else 'mozliwe'
+ self._add(refs, fid, row, f'{source} ({kind})', confidence)
+
+ def _scan_comments(self, refs, matcher):
+ table = self.table('comments')
+ if table not in self.columns or 'comment' not in self.columns[table]:
+ return
+ subject_field = ('c.subject AS comment_subject' if 'subject' in self.columns[table]
+ else 'NULL AS comment_subject')
+ query = (
+ f'SELECT n.nid, n.title, n.type AS node_type, n.status AS node_status, '
+ f'c.cid, c.status AS comment_status, c.comment, {subject_field} '
+ f'FROM {sql_ident(table)} c '
+ f'JOIN {sql_ident(self.table("node"))} n ON n.nid = c.nid '
+ )
+ for row in self._stream_query(query):
+ for field, source in (('comment', 'komentarz'), ('comment_subject', 'tytul komentarza')):
+ for fid, kind in matcher.match(row.get(field)).items():
+ self._add(refs, fid, row, f'{source} ({kind})',
+ 'pewne' if kind == 'sciezka' else 'mozliwe', row['cid'])
+
+ def _scan_cck_text(self, refs, matcher):
+ # Opcjonalne: tekst moze byc w polach niestandardowych CCK.
+ for table, cols in sorted(self.columns.items()):
+ if not table.startswith(self.table('content_')) or 'nid' not in cols:
+ continue
+ columns = [col for col, typ in cols.items() if (
+ 'text' in typ or 'varchar' in typ
+ ) and col not in ('nid', 'vid')]
+ if not columns:
+ continue
+ revision_join = ' AND n.vid = t.vid' if 'vid' in cols else ''
+ for col in columns:
+ query = (
+ f'SELECT n.nid, n.title, n.type AS node_type, n.status AS node_status, '
+ f't.{sql_ident(col)} AS value FROM {sql_ident(table)} t '
+ f'JOIN {sql_ident(self.table("node"))} n ON n.nid = t.nid'
+ f'{revision_join} WHERE t.{sql_ident(col)} IS NOT NULL'
+ )
+ try:
+ for row in self._stream_query(query):
+ for fid, kind in matcher.match(row.get('value')).items():
+ self._add(refs, fid, row, f'{table}.{col} ({kind})', 'pewne' if kind == 'sciezka' else 'mozliwe')
+ except Exception as exc:
+ print(f'[WARN] Nie udalo sie przeskanowac pola {table}.{col}: {exc}')
+
+ def lookup(self, images, *, scan_text=True, scan_cck_text=False, basename_match=True):
+ if not images:
+ return {}
+ self.inspect_schema()
+ fids = sorted({int(img['fid']) for img in images})
+ refs = defaultdict(dict)
+ self._direct_relations(refs, fids)
+ self._comment_direct_relations(refs, fids)
+ if scan_text:
+ matcher = ReferenceMatcher(images, basename_match=basename_match)
+ if matcher.pattern:
+ print('[INFO] Szukam odnosnikow w aktualnych tresciach wpisow i komentarzy...')
+ self._scan_body(refs, matcher)
+ self._scan_comments(refs, matcher)
+ if scan_cck_text:
+ self._scan_cck_text(refs, matcher)
+ return {
+ fid: sorted(per_location.values(), key=lambda ref: (ref['nid'], ref['cid'] or 0))
+ for fid, per_location in refs.items()
+ }
diff --git a/image_optimizer.py b/image_optimizer.py
new file mode 100644
index 0000000..b616528
--- /dev/null
+++ b/image_optimizer.py
@@ -0,0 +1,153 @@
+"""Zachowawcza, bezstratna dla PNG / wysokiej jakosci dla JPEG, optymalizacja.
+
+Ten modul NIGDY nie nadpisuje pliku zrodlowego: tworzy jedynie propozycje
+w pliku tymczasowym. main.py odpowiada za backup, podmiane i transakcje SQL.
+"""
+from __future__ import annotations
+
+import os
+import tempfile
+from pathlib import Path
+
+from PIL import Image, ImageOps, PngImagePlugin
+
+
+class ImageOptimizationError(Exception):
+ pass
+
+
+def _image_save_kwargs(source, image_format):
+ kwargs = {}
+ if source.info.get('icc_profile'):
+ kwargs['icc_profile'] = source.info['icc_profile']
+ if source.info.get('dpi'):
+ kwargs['dpi'] = source.info['dpi']
+ if image_format == 'PNG':
+ pnginfo = PngImagePlugin.PngInfo()
+ text_chunks = getattr(source, 'text', {}) or {}
+ for key, value in text_chunks.items():
+ if isinstance(key, str) and isinstance(value, str):
+ pnginfo.add_text(key, value)
+ if text_chunks:
+ kwargs['pnginfo'] = pnginfo
+ return kwargs
+
+
+def create_optimized_candidate(
+ path,
+ *,
+ max_width=2400,
+ max_height=3200,
+ max_bytes=2 * 1024 * 1024,
+ jpeg_quality=92,
+ min_jpeg_quality=85,
+ min_gain_percent=5.0,
+ temp_dir=None,
+):
+ """Zwraca {status,reason,before,after,width,height,quality,temp_path}.
+
+ Status: ready, skip, error. Wlasnosc pliku temp_path przekazywana wywolujacemu:
+ wywolujacy MUSI usunac plik, takze przy bledzie.
+ """
+ orig_size = os.path.getsize(path)
+ result = {
+ 'status': 'skip', 'reason': '', 'before': orig_size,
+ 'after': orig_size, 'width': 0, 'height': 0, 'quality': None,
+ 'temp_path': None,
+ }
+ temp_path = None
+ try:
+ with Image.open(path) as source:
+ fmt = source.format
+ ext = Path(path).suffix.lower()
+ if fmt not in ('JPEG', 'PNG'):
+ result['reason'] = f'format {fmt or "nieznany"} nie jest bezpiecznie obslugiwany'
+ return result
+ if fmt == 'JPEG' and ext not in ('.jpg', '.jpeg'):
+ result['reason'] = 'format i rozszerzenie pliku nie sa zgodne'
+ return result
+ if fmt == 'PNG' and ext != '.png':
+ result['reason'] = 'format i rozszerzenie pliku nie sa zgodne'
+ return result
+ if getattr(source, 'is_animated', False) and getattr(source, 'n_frames', 1) > 1:
+ result['reason'] = 'animacja (nie modyfikujemy)'
+ return result
+
+ original_width, original_height = source.size
+ result['width'], result['height'] = original_width, original_height
+ if original_width <= max_width and original_height <= max_height and orig_size <= max_bytes:
+ result['reason'] = 'obraz juz spelnia limity'
+ return result
+
+ # EXIF Orientation ma znaczenie zwlaszcza dla pionowych skanow.
+ working = ImageOps.exif_transpose(source)
+ if (fmt == 'PNG' and working.mode == 'P' and
+ (working.width > max_width or working.height > max_height)):
+ # Skalowanie obrazu paletowego przed konwersja daje nearest-neighbor.
+ # Konwertuj najpierw, aby LANCZOS pracowal na pelnych kolorach.
+ working = working.convert('RGBA' if 'transparency' in working.info else 'RGB')
+ working.thumbnail((max_width, max_height), Image.Resampling.LANCZOS)
+ result['width'], result['height'] = working.size
+ kwargs = _image_save_kwargs(source, fmt)
+
+ fd, temp_path = tempfile.mkstemp(suffix=ext, prefix='.d6-image-', dir=temp_dir)
+ os.close(fd)
+
+ if fmt == 'JPEG':
+ if working.mode not in ('RGB', 'L', 'CMYK'):
+ if working.mode == 'RGBA' or 'transparency' in working.info:
+ result['reason'] = 'JPEG z kanalem alfa (nie modyfikujemy)'
+ return result
+ working = working.convert('RGB')
+ # Pobieraj EXIF juz po prawidlowej korekcji orientacji.
+ try:
+ exif = working.getexif()
+ if exif:
+ kwargs['exif'] = exif.tobytes()
+ except (ValueError, TypeError, OSError):
+ # Uszkodzone EXIF nie powinno blokowac zapisu obrazu.
+ pass
+ qualities = list(range(jpeg_quality, min_jpeg_quality - 1, -3))
+ if min_jpeg_quality not in qualities:
+ qualities.append(min_jpeg_quality)
+ best_quality = None
+ best_size = orig_size
+ for quality in qualities:
+ working.save(
+ temp_path, format='JPEG', optimize=True,
+ progressive=True, subsampling=0, quality=quality, **kwargs,
+ )
+ size = os.path.getsize(temp_path)
+ if size < best_size:
+ best_quality, best_size = quality, size
+ if size <= max_bytes and 100.0 * (orig_size - size) / max(1, orig_size) >= min_gain_percent:
+ break
+ # Jezeli nie udalo sie zejsc pod limit, zastosuj najmniejsza
+ # akceptowalna jakosc. Plik wynikowy bedzie ostatnim kandydatem.
+ if best_quality != quality:
+ working.save(
+ temp_path, format='JPEG', optimize=True,
+ progressive=True, subsampling=0, quality=best_quality or min_jpeg_quality,
+ **kwargs,
+ )
+ result['quality'] = best_quality
+ else:
+ working.save(temp_path, format='PNG', optimize=True, compress_level=9, **kwargs)
+
+ result['after'] = os.path.getsize(temp_path)
+ gain_percent = 100.0 * (orig_size - result['after']) / max(1, orig_size)
+ if result['after'] >= orig_size or gain_percent < min_gain_percent:
+ result['reason'] = f'za mala oszczednosc ({gain_percent:.1f}% < {min_gain_percent:.1f}%)'
+ return result
+ result['status'] = 'ready'
+ result['reason'] = 'osiagnieto limit' if result['after'] <= max_bytes else 'zmniejszono; nadal powyzej limitu'
+ result['temp_path'] = temp_path
+ temp_path = None # przekazujemy plik wywolujacemu
+ return result
+ except Exception as exc:
+ result['status'] = 'error'
+ result['reason'] = f'{type(exc).__name__}: {exc}'
+ return result
+ finally:
+ if temp_path and os.path.exists(temp_path):
+ os.unlink(temp_path)
diff --git a/main.py b/main.py
index 8540790..6e7c337 100644
--- a/main.py
+++ b/main.py
@@ -1,362 +1,776 @@
-import os
+#!/usr/bin/env python3
+"""Audyt, wyszukiwanie wpisow i bezpieczna optymalizacja obrazow Drupal 6.
+
+Tryb domyslny jest wylacznie do odczytu. Zapisy sa mozliwe tylko z --apply.
+"""
+from __future__ import annotations
+
import argparse
-import pymysql
+import csv
import hashlib
+import importlib
+import json
+import os
+import posixpath
+import re
+import shutil
+import stat
+import sys
+import tempfile
+from collections import defaultdict
+from datetime import datetime, timezone
+from pathlib import Path
+from urllib.parse import quote
+
from PIL import Image
-from config import DB_CONFIG, FILES_BASE_PATH
-MAX_WIDTH = 1920
-MAX_HEIGHT = 1080
-MAX_FILESIZE_BYTES = 1 * 1024 * 1024 # 1MB
+from drupal_lookup import DrupalReferenceLookup, node_url, search_url
+from image_optimizer import create_optimized_candidate
-def get_connection():
- return pymysql.connect(
- host=DB_CONFIG['host'],
- user=DB_CONFIG['user'],
- password=DB_CONFIG['password'],
- database=DB_CONFIG['database'],
- charset='utf8mb4',
- cursorclass=pymysql.cursors.DictCursor
+
+def build_parser():
+ parser = argparse.ArgumentParser(
+ description='Drupal 6: duze pliki, linki do tematow/komentarzy, bezpieczna optymalizacja',
+ formatter_class=argparse.ArgumentDefaultsHelpFormatter,
)
+ analysis = parser.add_argument_group('Analiza i filtrowanie')
+ analysis.add_argument('--top', type=int, default=20, help='Ile najwiekszych plikow wyswietlic')
+ analysis.add_argument('--extensions', default='jpg,jpeg,png,gif', help='Rozszerzenia do analizy')
+ analysis.add_argument('--exclude-folders', nargs='*', default=[], help='Podfoldery pomijane w files/')
+ analysis.add_argument('--path-contains', default='', help='Filtr fragmentu sciezki, np. skan')
+ analysis.add_argument('--fid', type=int, action='append', help='Analizuj tylko podane fid (mozna powtorzyc)')
+ analysis.add_argument('--min-size-mb', type=float, default=0, help='Pokaz pliki od wskazanej liczby MB')
+ analysis.add_argument('--find-duplicates', action='store_true', help='Sprawdz duplikaty po SHA-256')
+ analysis.add_argument('--show-deleted', action='store_true', help='Pokaz pliki na dysku spoza tabeli files')
+ analysis.add_argument('--debug', action='store_true', help='Szczegolowy log')
-def get_image_resolution(path):
+ lookup = parser.add_argument_group('Wskazanie tematu / posta / komentarza')
+ lookup.add_argument('--find-usage', '--where-used', dest='find_usage', action='store_true',
+ help='Szukaj powiazan plikow z wezłami i komentarzami')
+ lookup.add_argument('--usage-scope', choices=('top', 'optimizable', 'all'), default='top',
+ help='W ktorych plikach szukac powiazan')
+ lookup.add_argument('--site-url', help='Adres Drupal, np. https://forum.example.org')
+ lookup.add_argument('--no-clean-urls', action='store_true', help='Linki w formacie ?q=node/ID')
+ lookup.add_argument('--comment-permalinks', action='store_true',
+ help='Linki comment/CID (TYLKO z zainstalowanym backportem Permalink w Drupal 6)')
+ lookup.add_argument('--no-text-scan', action='store_true', help='Nie szukaj w tresci wpisow/komentarzy')
+ lookup.add_argument('--scan-cck-text', action='store_true', help='Szukaj takze w polach tekstowych CCK (wolniejsze)')
+ lookup.add_argument('--no-basename-match', action='store_true', help='W tekscie wymagaj sciezki, nie samej nazwy')
+ lookup.add_argument('--table-prefix', help='Prefix tabel Drupala, np. drupal_')
+ lookup.add_argument('--public-files-path', help='Sciezka publiczna w Drupal, np. sites/default/files')
+ lookup.add_argument('--drupal-root', help='Katalog glowny instalacji Drupal (pomaga rozwiazac sciezki)')
+
+ change = parser.add_argument_group('Optymalizacja i kontrola zapisow')
+ change.add_argument('--optimize', action='store_true', help='Zaplanuj optymalizacje lub wykonaj ja z --apply')
+ change.add_argument('--update-db', action='store_true', help='Wyrownaj pole files.filesize (z --apply)')
+ mode = change.add_mutually_exclusive_group()
+ mode.add_argument('--apply', action='store_true', help='Zezwol na realny zapis obrazow/DB')
+ mode.add_argument('--dry-run', action='store_true', help='Jawny tryb podgladu (domyslny)')
+ change.add_argument('--backup-dir', help='Katalog dla kopii oryginalow (wymagany z --optimize --apply)')
+ change.add_argument('--max-changes', type=int, default=0, help='Maksymalna liczba realnie optymalizowanych plikow (0=bez limitu)')
+ change.add_argument('--optimize-top', type=int, default=0, help='Optymalizuj najwyzej N najwiekszych kandydatow (0=wszystkie)')
+ change.add_argument('--max-width', type=int, default=2400, help='Maksymalna szerokosc po optymalizacji')
+ change.add_argument('--max-height', type=int, default=3200, help='Maksymalna wysokosc po optymalizacji')
+ change.add_argument('--max-file-mb', type=float, default=2.0, help='Docelowy limit MB; nie kosztem duzej utraty jakosci')
+ change.add_argument('--jpeg-quality', type=int, default=92, help='Poczatkowa jakosc JPEG')
+ change.add_argument('--min-jpeg-quality', type=int, default=85, help='Minimalna dopuszczalna jakosc JPEG')
+ change.add_argument('--min-gain-percent', type=float, default=5.0, help='Minimalne zmniejszenie objetosci w procentach')
+
+ report = parser.add_argument_group('Raporty')
+ report.add_argument('--report-csv', help='CSV z plikami i linkami (UTF-8 BOM, otwiera sie w Excelu)')
+ report.add_argument('--report-json', help='Raport JSON z lokalizacjami, rozmiarami i statusami')
+ return parser
+
+
+def validate_args(args, parser):
+ if args.top <= 0 or args.min_size_mb < 0 or args.max_file_mb <= 0:
+ parser.error('--top > 0, --min-size-mb >= 0 i --max-file-mb > 0 sa wymagane')
+ if min(args.max_width, args.max_height) <= 0:
+ parser.error('--max-width i --max-height musza byc dodatnie')
+ if not (1 <= args.min_jpeg_quality <= args.jpeg_quality <= 95):
+ parser.error('Jakosc JPEG: 1 <= --min-jpeg-quality <= --jpeg-quality <= 95')
+ if not (0 <= args.min_gain_percent <= 100):
+ parser.error('--min-gain-percent musi byc w przedziale 0..100')
+ if args.max_changes < 0 or args.optimize_top < 0:
+ parser.error('--max-changes i --optimize-top nie moga byc ujemne')
+ if args.apply and not (args.optimize or args.update_db):
+ parser.error('--apply wymaga --optimize lub --update-db')
+ if args.apply and args.optimize and not args.backup_dir:
+ parser.error('Dla --optimize --apply podaj --backup-dir poza folderem z plikami')
+ if args.scan_cck_text and args.no_text_scan:
+ parser.error('--scan-cck-text wyklucza sie z --no-text-scan')
+ if args.site_url and not re.match(r'^https?://[^/]+', args.site_url, flags=re.I):
+ parser.error('--site-url musi byc adresem https://... lub http://...')
+ if args.table_prefix and not re.fullmatch(r'[A-Za-z_][A-Za-z0-9_]*', args.table_prefix):
+ parser.error('Niedozwolony prefix tabel')
+
+
+def load_config():
try:
- with Image.open(path) as img:
- return img.size # (width, height)
+ config = importlib.import_module('config')
+ except ModuleNotFoundError as exc:
+ if exc.name == 'config':
+ raise RuntimeError('Brak config.py - skopiuj config.example.py do config.py i ustaw dane') from exc
+ raise
+ if not hasattr(config, 'DB_CONFIG') or not hasattr(config, 'FILES_BASE_PATH'):
+ raise RuntimeError('config.py musi definiowac DB_CONFIG oraz FILES_BASE_PATH')
+ return config
+
+
+def get_connection(config):
+ import pymysql
+ db = dict(config.DB_CONFIG)
+ connection_kwargs = {
+ 'host': db['host'],
+ 'user': db['user'],
+ 'password': db['password'],
+ 'database': db['database'],
+ 'charset': db.get('charset', 'utf8'),
+ 'port': int(db.get('port', 3306)),
+ 'cursorclass': pymysql.cursors.DictCursor,
+ 'autocommit': False,
+ 'connect_timeout': 15,
+ }
+ for optional in ('unix_socket', 'ssl'):
+ if optional in db:
+ connection_kwargs[optional] = db[optional]
+ return pymysql.connect(**connection_kwargs)
+
+
+def _within(path, root):
+ path, root = os.path.realpath(path), os.path.realpath(root)
+ try:
+ return os.path.commonpath([path, root]) == root
+ except ValueError:
+ return False
+
+
+def resolve_filepath(db_filepath, base_path, public_path='sites/default/files', drupal_root=None):
+ """Rozpoznaj Drupal filepath: relative/to/files albo sites/default/files/x.jpg.
+
+ NIGDY nie zezwalaj na wyjscie poza FILES_BASE_PATH (takze przez symlink).
+ Zwraca (full_path, relative_to_files) lub (None, None).
+ """
+ if not isinstance(db_filepath, str) or '\x00' in db_filepath or '://' in db_filepath:
+ return None, None
+ db_filepath = db_filepath.replace('\\', '/').strip()
+ base = os.path.realpath(base_path)
+ public = (public_path or '').replace('\\', '/').strip('/')
+ if not db_filepath:
+ return None, None
+
+ candidates = []
+ if os.path.isabs(db_filepath):
+ if _within(db_filepath, base):
+ candidates.append(db_filepath)
+ elif not (public and db_filepath.lstrip('/').startswith(public + '/')) and not (
+ db_filepath.lstrip('/').startswith(os.path.basename(base) + '/')
+ ):
+ # /sites/default/files/x jest sciezka URL; /etc/passwd nie.
+ return None, None
+
+ relative = db_filepath.lstrip('/')
+ if '..' in relative.split('/'):
+ return None, None
+ normalized = posixpath.normpath(relative)
+ if normalized in ('', '.', '..'):
+ return None, None
+ if public and normalized.startswith(public + '/'):
+ candidates.append(os.path.join(base, normalized[len(public) + 1:]))
+ # Czeste przy przenoszeniu Drupala: config podaje katalog "files",
+ # a DB przechowuje "files/name.jpg".
+ base_folder = os.path.basename(base)
+ if normalized.startswith(base_folder + '/'):
+ candidates.append(os.path.join(base, normalized[len(base_folder) + 1:]))
+ if drupal_root:
+ candidates.append(os.path.join(drupal_root, normalized))
+ candidates.append(os.path.join(base, normalized))
+
+ # Zachowuj oryginalna sciezke: realpath ukrylby symlink i mogl ominac
+ # zabezpieczenie blokujace jego modyfikacje. _within sprawdza realpath.
+ allowed = [os.path.abspath(c) for c in candidates if _within(c, base)]
+ if not allowed:
+ return None, None
+ found = next((path for path in allowed if os.path.isfile(path)), allowed[0])
+ return found, os.path.relpath(found, base).replace(os.sep, '/')
+
+
+def sha256_file(path):
+ digest = hashlib.sha256()
+ with open(path, 'rb') as src:
+ for chunk in iter(lambda: src.read(1024 * 1024), b''):
+ digest.update(chunk)
+ return digest.hexdigest()
+
+
+def file_resolution(path):
+ try:
+ with Image.open(path) as im:
+ return im.size
except Exception:
return (0, 0)
-def hash_file(path):
- hasher = hashlib.md5()
- try:
- with open(path, 'rb') as f:
- for chunk in iter(lambda: f.read(8192), b""):
- hasher.update(chunk)
- return hasher.hexdigest()
- except Exception:
- return None
-def resize_image(path):
- try:
- with Image.open(path) as img:
- width, height = img.size
- if width <= MAX_WIDTH and height <= MAX_HEIGHT and os.path.getsize(path) <= MAX_FILESIZE_BYTES:
- return False # Nie trzeba zmieniać
-
- # Oblicz skalę proporcjonalnie, by zmieścić w max rozmiar i rozdzielczość
- width_ratio = MAX_WIDTH / width
- height_ratio = MAX_HEIGHT / height
- scale_ratio = min(width_ratio, height_ratio, 1)
-
- new_width = int(width * scale_ratio)
- new_height = int(height * scale_ratio)
-
- img = img.resize((new_width, new_height), Image.LANCZOS)
- # Zapisz nadpisując, jakość 85%
- img.save(path, optimize=True, quality=85)
+def _excluded(rel_path, folders):
+ rel_path = rel_path.replace('\\', '/').lstrip('/')
+ for folder in folders:
+ item = folder.replace('\\', '/').strip('/')
+ if item and (rel_path == item or rel_path.startswith(item + '/')):
return True
- except Exception as e:
- print(f"[ERROR] Nie udało się zmniejszyć {path}: {e}")
- return False
-
-def fetch_used_filepaths(cursor):
- """
- Pobiera unikalne ścieżki plików (filepath) ze wszystkich tabel, które mają kolumnę 'filepath'.
- Zwraca set ścieżek względnych (z '/' zamiast '\\').
- """
- # Pobierz listę tabel w bazie
- cursor.execute("SHOW TABLES")
- tables = [row[f'Tables_in_{DB_CONFIG["database"]}'] for row in cursor.fetchall()]
-
- filepaths = set()
-
- for table in tables:
- # Sprawdź kolumny danej tabeli
- cursor.execute(f"SHOW COLUMNS FROM `{table}` LIKE 'filepath'")
- if cursor.rowcount == 1:
- # Tabela ma kolumnę 'filepath' — pobierz jej wartości
- try:
- cursor.execute(f"SELECT filepath FROM `{table}`")
- rows = cursor.fetchall()
- for row in rows:
- path = row['filepath'].replace('\\', '/')
- filepaths.add(path)
- except Exception as e:
- print(f"[WARN] Nie udało się pobrać danych z tabeli '{table}': {e}")
- # Możesz tu też zdecydować, czy chcesz przerwać, czy tylko ostrzec
-
- return filepaths
+ return False
-def file_exists_on_disk(base_path, rel_path):
- # Normalizuj ścieżkę z bazy
- rel_path_norm = rel_path.lstrip("/\\") # usuń początkowe ukośniki
+def collect_images(connection, config, args):
+ base = os.path.realpath(config.FILES_BASE_PATH)
+ table_prefix = args.table_prefix if args.table_prefix is not None else getattr(config, 'DB_TABLE_PREFIX', '')
+ if table_prefix and not re.fullmatch(r'[A-Za-z_][A-Za-z0-9_]*', table_prefix):
+ raise ValueError('Nieprawidlowy prefix tabel w config.py')
+ tables_files = f'`{table_prefix}files`'
+ with connection.cursor() as cursor:
+ cursor.execute(f'SELECT fid, filepath, filesize FROM {tables_files}')
+ file_rows = cursor.fetchall()
- full_path = os.path.normpath(os.path.join(base_path, rel_path_norm))
-
- # Debug print, żeby zweryfikować jak łączy się ścieżka
- # print(f"Sprawdzam plik: {full_path}")
-
- return os.path.isfile(full_path), full_path
-
-def compare_files(
- debug=False,
- to_optimize=False,
- update_db=False,
- dry_run=False,
- extensions='jpg,jpeg,png,gif',
- exclude_folders=None,
- find_duplicates=False,
- show_deleted=False,
- top=20,
-):
- allowed_exts = tuple(f".{ext.strip().lower()}" for ext in extensions.split(','))
- exclude_folders = exclude_folders or []
- exclude_folders = [folder.rstrip('/\\') for folder in exclude_folders]
-
- connection = get_connection()
- cursor = connection.cursor()
-
- # Pobierz pliki z bazy
- cursor.execute("SELECT fid, filepath, filesize FROM files")
- files = cursor.fetchall()
-
- # Zbiór ścieżek plików używanych w systemie (dla duplikatów i wykrywania osieroconych)
- used_filepaths = fetch_used_filepaths(cursor)
-
- stats = {
- 'total': 0,
- 'missing': 0,
- 'mismatched': 0,
- 'saved_bytes_potential': 0,
- 'largest': [],
- 'highest_res': [],
- 'to_optimize': [],
- 'duplicates': [],
- 'deleted': [],
- }
-
- images_info = []
- hash_map = {}
-
- # Przetwarzanie plików z bazy
- for file in files:
- rel_path = file['filepath'].replace('\\', '/')
- # Sprawdź wykluczenia folderów
- if any(rel_path.startswith(excl + '/') or rel_path == excl for excl in exclude_folders):
- if debug:
- print(f"[POMINIĘTO] {rel_path} (folder wykluczony)")
+ public = args.public_files_path if args.public_files_path is not None else getattr(config, 'PUBLIC_FILES_PATH', 'sites/default/files')
+ root = args.drupal_root if args.drupal_root is not None else getattr(config, 'DRUPAL_ROOT', None)
+ extensions = {'.' + x.strip().lower().lstrip('.') for x in args.extensions.split(',') if x.strip()}
+ groups = {}
+ missing = []
+ for row in file_rows:
+ fid = int(row['fid'])
+ if args.fid and fid not in args.fid:
continue
-
- if not rel_path.lower().endswith(allowed_exts):
+ db_path = str(row['filepath'] or '')
+ if not db_path.lower().endswith(tuple(extensions)):
+ continue
+ if args.path_contains and args.path_contains.casefold() not in db_path.casefold():
+ continue
+ full_path, rel_path = resolve_filepath(db_path, base, public, root)
+ if not full_path:
+ missing.append({'fid': fid, 'db_filepath': db_path, 'reason': 'niebezpieczna lub nierozpoznana sciezka'})
+ continue
+ if _excluded(rel_path, args.exclude_folders):
continue
-
- fid = file['fid']
- db_size = file['filesize']
- full_path = os.path.join(FILES_BASE_PATH, rel_path)
-
- stats['total'] += 1
-
if not os.path.isfile(full_path):
- stats['missing'] += 1
- if debug:
- print(f"[BRAK] {rel_path}")
- print(f" → {full_path}")
+ missing.append({'fid': fid, 'db_filepath': db_path, 'path': full_path, 'reason': 'brak na dysku'})
+ continue
+ stat_result = os.stat(full_path)
+ if stat_result.st_size < args.min_size_mb * 1024 * 1024:
+ continue
+ if full_path not in groups:
+ width, height = file_resolution(full_path)
+ groups[full_path] = {
+ 'path': full_path,
+ 'rel_path': rel_path,
+ 'actual_size': stat_result.st_size,
+ 'before_size': stat_result.st_size,
+ 'original_ino': stat_result.st_ino,
+ 'original_mtime_ns': stat_result.st_mtime_ns,
+ 'after_size': stat_result.st_size,
+ 'width': width, 'height': height,
+ 'db_rows': [], 'fids': [], 'references': [],
+ 'optimize_status': 'not_requested', 'optimization_reason': '',
+ 'quality': None,
+ }
+ img = groups[full_path]
+ img['db_rows'].append({'fid': fid, 'db_filepath': db_path, 'filesize': int(row['filesize'] or 0)})
+ img['fids'].append(fid)
+
+ images = list(groups.values())
+ for image in images:
+ image['fid'] = image['fids'][0]
+ image['db_filepath'] = image['db_rows'][0]['db_filepath']
+ image['db_size'] = image['db_rows'][0]['filesize']
+ images.sort(key=lambda img: img['before_size'], reverse=True)
+ return images, missing, table_prefix
+
+
+def is_candidate(image, args):
+ return (
+ image['width'] > args.max_width or
+ image['height'] > args.max_height or
+ image['before_size'] > args.max_file_mb * 1024 * 1024
+ )
+
+
+def _update_sizes(connection, table_prefix, image, filesize):
+ table = f'`{table_prefix}files`'
+ for db_row in image['db_rows']:
+ fid = db_row['fid']
+ db_path = db_row['db_filepath']
+ with connection.cursor() as cursor:
+ cursor.execute(
+ f'UPDATE {table} SET filesize = %s WHERE fid = %s AND filepath = %s',
+ (int(filesize), fid, db_path),
+ )
+ # Dla rowcount=0 sprawdzamy, czy row nadal istnieje i ma oczekiwany rozmiar.
+ if cursor.rowcount == 0:
+ cursor.execute(f'SELECT filesize, filepath FROM {table} WHERE fid = %s', (fid,))
+ row = cursor.fetchone()
+ if not row or row['filepath'] != db_path or int(row['filesize']) != int(filesize):
+ raise RuntimeError(f'Nie udalo sie zsynchronizowac files.fid={fid}')
+
+
+def _restore_database_sizes(connection, table_prefix, image):
+ # Kompensacja takze dla starego MySQL/MyISAM (ROLLBACK moze nie dzialac).
+ table = f'`{table_prefix}files`'
+ with connection.cursor() as cursor:
+ for row in image['db_rows']:
+ cursor.execute(
+ f'UPDATE {table} SET filesize = %s WHERE fid = %s AND filepath = %s',
+ (row['filesize'], row['fid'], row['db_filepath'])
+ )
+ connection.commit()
+
+
+def _atomic_restore(backup_file, destination):
+ fd, restore_tmp = tempfile.mkstemp(prefix='.d6-restore-', dir=os.path.dirname(destination))
+ os.close(fd)
+ try:
+ shutil.copy2(backup_file, restore_tmp)
+ os.replace(restore_tmp, destination)
+ finally:
+ if os.path.exists(restore_tmp):
+ os.unlink(restore_tmp)
+
+
+def _safe_backup_dir(requested, base, drupal_root=None):
+ backup_root = os.path.realpath(os.path.abspath(requested))
+ if _within(backup_root, base):
+ raise ValueError('Backup nie moze znajdowac sie wewnatrz FILES_BASE_PATH')
+ if drupal_root and _within(backup_root, drupal_root):
+ raise ValueError('Backup nie moze znajdowac sie wewnatrz katalogu WWW Drupala')
+ os.makedirs(backup_root, exist_ok=True)
+ run_id = datetime.now(timezone.utc).strftime('%Y%m%dT%H%M%S%fZ') + f'-{os.getpid()}'
+ run_dir = os.path.join(backup_root, run_id)
+ os.mkdir(run_dir)
+ return run_dir
+
+
+def apply_candidate(connection, table_prefix, image, candidate, *, backup_run_dir, base_dir):
+ """Backup + atomowe podmienienie pliku + SQL (rollback i restore przy bledzie)."""
+ source = image['path']
+ tmp = candidate['temp_path']
+ if not tmp:
+ raise RuntimeError('Brak pliku kandydata')
+ stat_before = os.stat(source)
+ if os.path.islink(source) or stat_before.st_nlink != 1:
+ raise RuntimeError('Plik jest dowiazaniem symbolicznym/twardym; pomijam')
+ if (stat_before.st_size != image['before_size'] or
+ stat_before.st_ino != image['original_ino'] or
+ stat_before.st_mtime_ns != image['original_mtime_ns']):
+ raise RuntimeError('Plik zostal zmieniony od poczatku skanowania')
+
+ # Kopia w zewnetrznym folderze z zachowaniem struktury i uprawnien.
+ relative_path = os.path.relpath(source, base_dir)
+ backup_file = os.path.join(backup_run_dir, relative_path)
+ os.makedirs(os.path.dirname(backup_file), exist_ok=True)
+ if os.path.lexists(backup_file):
+ raise RuntimeError(f'Kopia zapasowa juz istnieje: {backup_file}')
+ shutil.copy2(source, backup_file)
+ if os.path.getsize(backup_file) != stat_before.st_size:
+ raise RuntimeError('Nieudane kopiowanie backupu (niezgodny rozmiar)')
+
+ # Zachowanie uprawnien, czasu oraz wlasciciela jesli jest mozliwosc.
+ shutil.copystat(source, tmp)
+ if hasattr(os, 'chown'):
+ try:
+ os.chown(tmp, stat_before.st_uid, stat_before.st_gid)
+ except PermissionError:
+ # Uruchomienie z uprawnieniami wlasciciela daje i tak ten sam uid.
+ if os.stat(tmp).st_uid != stat_before.st_uid:
+ raise
+
+ stat_now = os.stat(source)
+ if (stat_now.st_ino, stat_now.st_size, stat_now.st_mtime_ns) != (
+ stat_before.st_ino, stat_before.st_size, stat_before.st_mtime_ns
+ ):
+ raise RuntimeError('Plik zostal zmodyfikowany rownolegle; przerwano zapis')
+
+ os.replace(tmp, source) # w obrebie tego samego filesystemu
+ candidate['temp_path'] = None
+ try:
+ _update_sizes(connection, table_prefix, image, candidate['after'])
+ connection.commit()
+ except Exception as exc:
+ connection.rollback()
+ try:
+ _atomic_restore(backup_file, source)
+ except Exception as restore_error:
+ raise RuntimeError(
+ f'Blad DB ({exc}); AUTO-RESTORE NIEUDANY ({restore_error}), kopia: {backup_file}'
+ ) from exc
+ try:
+ _restore_database_sizes(connection, table_prefix, image)
+ except Exception as restore_db_error:
+ raise RuntimeError(
+ f'Plik przywrocony z backupu, ale nie udalo sie odtworzyc rozmiarow w DB: '
+ f'{restore_db_error}; kopia: {backup_file}'
+ ) from exc
+ raise RuntimeError(f'Blad DB, przywrocono plik i rozmiary w bazie: {exc}') from exc
+ return backup_file
+
+
+def process_changes(connection, config, args, images, table_prefix):
+ summary = {'candidates': 0, 'planned': 0, 'optimized': 0, 'saved_bytes': 0,
+ 'db_mismatches': 0, 'db_updated': 0, 'errors': 0, 'backups_dir': ''}
+ base = os.path.realpath(config.FILES_BASE_PATH)
+ if args.apply and args.optimize:
+ root = args.drupal_root if args.drupal_root is not None else getattr(config, 'DRUPAL_ROOT', None)
+ summary['backups_dir'] = _safe_backup_dir(args.backup_dir, base, root)
+ print(f'[BACKUP] Kopie oryginalow: {summary["backups_dir"]}')
+
+ candidates = [img for img in images if is_candidate(img, args)]
+ summary['candidates'] = len(candidates)
+ if args.optimize_top:
+ candidates = candidates[:args.optimize_top]
+ candidate_ids = {img['path'] for img in candidates}
+ max_bytes = int(args.max_file_mb * 1024 * 1024)
+
+ for image in images:
+ mismatched_rows = [db for db in image['db_rows'] if db['filesize'] != image['before_size']]
+ summary['db_mismatches'] += len(mismatched_rows)
+ if not args.optimize or image['path'] not in candidate_ids:
+ if args.update_db and mismatched_rows:
+ if args.apply:
+ try:
+ _update_sizes(connection, table_prefix, image, image['before_size'])
+ connection.commit()
+ summary['db_updated'] += len(mismatched_rows)
+ print(f'[DB-UPDATE] fid={image["fids"]} -> {image["before_size"]} B')
+ except Exception as exc:
+ connection.rollback()
+ try:
+ _restore_database_sizes(connection, table_prefix, image)
+ except Exception as restore_error:
+ print(f'[CRITICAL] Nie udalo sie przywrocic metadanych DB: {restore_error}')
+ summary['errors'] += 1
+ print(f'[ERROR DB] {image["rel_path"]}: {exc}')
+ else:
+ print(f'[DB-PLAN] fid={image["fids"]}: dysk {image["before_size"]} B, DB: {[r["filesize"] for r in image["db_rows"]]}')
continue
- actual_size = os.path.getsize(full_path)
- width, height = get_image_resolution(full_path)
- file_hash = hash_file(full_path)
-
- images_info.append({
- 'fid': fid,
- 'path': full_path,
- 'rel_path': rel_path,
- 'db_size': db_size,
- 'actual_size': actual_size,
- 'width': width,
- 'height': height,
- 'hash': file_hash,
- })
-
- # Rozmiar różniący się
- if actual_size != db_size:
- stats['mismatched'] += 1
- diff = db_size - actual_size
- if diff > 0:
- stats['saved_bytes_potential'] += diff
-
- if debug:
- print(f"[ROZMIAR] {rel_path}")
- print(f" → DB: {db_size} B, Dysk: {actual_size} B ({(db_size - actual_size)/1024:.1f} KB różnicy)")
-
- if update_db and not dry_run:
- update_sql = "UPDATE files SET filesize = %s WHERE fid = %s"
- cursor.execute(update_sql, (actual_size, fid))
- connection.commit()
- elif update_db and dry_run:
- delta_kb = (actual_size - db_size) / 1024
- print(f"[DRY-RUN] {rel_path}")
- print(f" DB: {db_size} B | FS: {actual_size} B | Δ: {delta_kb:+.1f} KB")
-
- # Optymalizacja (np. zmniejszanie obrazów)
- if to_optimize:
- if (width > MAX_WIDTH or height > MAX_HEIGHT) or actual_size > MAX_FILESIZE_BYTES:
- stats['to_optimize'].append({
- 'path': full_path,
- 'size': actual_size,
- 'res': f"{width}x{height}"
- })
- if update_db and not dry_run:
- resized = resize_image(full_path)
- if resized:
- new_size = os.path.getsize(full_path)
- # Aktualizuj bazę po zmianie
- update_sql = "UPDATE files SET filesize = %s WHERE fid = %s"
- cursor.execute(update_sql, (new_size, fid))
+ if os.path.islink(image['path']) or os.stat(image['path']).st_nlink != 1:
+ image['optimize_status'] = 'skip'
+ image['optimization_reason'] = 'dowiazanie symboliczne/twarde (pomijam)'
+ print(f'[SKIP] {image["rel_path"]}: {image["optimization_reason"]}')
+ continue
+ if args.apply and args.max_changes and summary['optimized'] >= args.max_changes:
+ image['optimize_status'] = 'limit'
+ image['optimization_reason'] = 'osiagnieto limit --max-changes'
+ continue
+ # W dry-run obliczamy prawdziwy rozmiar po zapisie na pliku tymczasowym.
+ result = create_optimized_candidate(
+ image['path'], max_width=args.max_width, max_height=args.max_height,
+ max_bytes=max_bytes, jpeg_quality=args.jpeg_quality,
+ min_jpeg_quality=args.min_jpeg_quality,
+ min_gain_percent=args.min_gain_percent,
+ temp_dir=os.path.dirname(image['path']) if args.apply else None,
+ )
+ image['optimization_reason'] = result['reason']
+ image['optimize_status'] = result['status']
+ image['quality'] = result.get('quality')
+ image['preview_size'] = result['after']
+ if result['status'] == 'error':
+ summary['errors'] += 1
+ print(f'[ERROR] {image["rel_path"]}: {result["reason"]}')
+ continue
+ if result['status'] == 'skip':
+ if args.debug:
+ print(f'[SKIP] {image["rel_path"]}: {result["reason"]}')
+ # Opcjonalny sync DB dla plikow, ktorych optymalizacja sie nie udala.
+ if args.update_db and mismatched_rows:
+ if args.apply:
+ try:
+ _update_sizes(connection, table_prefix, image, image['before_size'])
connection.commit()
- if debug:
- print(f"[ZMIENIONO] {rel_path}: zmniejszono do {new_size} B")
+ summary['db_updated'] += len(mismatched_rows)
+ except Exception as exc:
+ connection.rollback()
+ try:
+ _restore_database_sizes(connection, table_prefix, image)
+ except Exception as restore_error:
+ print(f'[CRITICAL] Nie udalo sie przywrocic metadanych DB: {restore_error}')
+ summary['errors'] += 1
+ print(f'[ERROR DB] {image["rel_path"]}: {exc}')
+ else:
+ print(f'[DB-PLAN] fid={image["fids"]}: {image["before_size"]} B')
+ continue
+ summary['planned'] += 1
+ gain = result['before'] - result['after']
+ desc = (
+ f'{image["rel_path"]} | {result["before"] / 1024**2:.2f} -> '
+ f'{result["after"] / 1024**2:.2f} MB | '
+ f'{result["width"]}x{result["height"]} | -{gain / 1024**2:.2f} MB'
+ )
+ try:
+ if args.apply:
+ backup_file = apply_candidate(connection, table_prefix, image, result,
+ backup_run_dir=summary['backups_dir'], base_dir=base)
+ image['backup_file'] = backup_file
+ image['after_size'] = result['after']
+ image['optimize_status'] = 'done'
+ summary['optimized'] += 1
+ summary['saved_bytes'] += gain
+ summary['db_updated'] += len(image['db_rows'])
+ print(f'[ZMIENIONO] {desc} | backup: {backup_file}')
+ else:
+ image['optimize_status'] = 'planned'
+ image['after_size'] = result['after']
+ summary['saved_bytes'] += gain
+ print(f'[DRY-RUN] {desc} (bez zapisu, DB bez zmian)')
+ except Exception as exc:
+ image['optimize_status'] = 'error'
+ image['optimization_reason'] = str(exc)
+ summary['errors'] += 1
+ print(f'[ERROR] {image["rel_path"]}: {exc}')
+ finally:
+ if result.get('temp_path') and os.path.isfile(result['temp_path']):
+ os.unlink(result['temp_path'])
+ summary['apply'] = bool(args.apply)
+ return summary
- elif update_db and dry_run:
- print(f"[DRY-RUN] Zmniejszyłbym {rel_path}")
- # Mapowanie hashy do detekcji duplikatów
- if find_duplicates and file_hash:
- if file_hash not in hash_map:
- hash_map[file_hash] = []
- hash_map[file_hash].append({
- 'fid': fid,
- 'rel_path': rel_path,
- 'path': full_path,
- 'db_size': db_size,
- 'actual_size': actual_size,
- 'width': width,
- 'height': height,
- })
+def find_references(connection, args, config, images, table_prefix):
+ if not args.find_usage:
+ return
+ if args.usage_scope == 'top':
+ selected = images[:args.top]
+ elif args.usage_scope == 'optimizable':
+ selected = [image for image in images if is_candidate(image, args)]
+ else:
+ selected = images
- # Największe pliki
- stats['largest'] = sorted(images_info, key=lambda x: x['actual_size'], reverse=True)[:top]
+ print(f'[INFO] Szukam tematu/posta dla {len(selected)} plikow (scope={args.usage_scope})')
+ if len(selected) > 1000:
+ print('[WARN] Duza liczba plikow do wyszukania; skan tekstu moze trwac dlugo i zuzyc duzo RAM')
+ lookup_files = []
+ for image in selected:
+ for db in image['db_rows']:
+ lookup_files.append({'fid': db['fid'], 'db_filepath': db['db_filepath'], 'rel_path': image['rel_path']})
+ if not lookup_files:
+ return
+ base_url = args.site_url if args.site_url is not None else getattr(config, 'DRUPAL_BASE_URL', '')
+ finder = DrupalReferenceLookup(connection, table_prefix=table_prefix,
+ base_url=base_url, clean_urls=not args.no_clean_urls,
+ comment_permalinks=args.comment_permalinks)
+ found = finder.lookup(lookup_files, scan_text=not args.no_text_scan,
+ scan_cck_text=args.scan_cck_text, basename_match=not args.no_basename_match)
+ for image in selected:
+ image['usage_checked'] = True
+ # Laczymy referencje do tego samego pliku z kilku rekordow fid.
+ combined = {}
+ for fid in image['fids']:
+ for ref in found.get(fid, []):
+ key = (ref['nid'], ref['cid'])
+ if key not in combined:
+ combined[key] = dict(ref)
+ combined[key]['sources'] = list(ref['sources'])
+ combined[key]['fids'] = [fid]
+ else:
+ if fid not in combined[key]['fids']:
+ combined[key]['fids'].append(fid)
+ for source in ref['sources']:
+ if source not in combined[key]['sources']:
+ combined[key]['sources'].append(source)
+ if ref['confidence'] == 'pewne':
+ combined[key]['confidence'] = 'pewne'
+ image['references'] = sorted(combined.values(), key=lambda r: (r['nid'], r['cid'] or 0))
+ return
- # Najwyższa rozdzielczość
- stats['highest_res'] = sorted(images_info, key=lambda x: x['width'] * x['height'], reverse=True)[:top]
- # Szukanie duplikatów
- if find_duplicates:
- duplicates = []
- for file_hash, files_list in hash_map.items():
- if len(files_list) > 1:
- # Sprawdź które pliki są używane w systemie
- used = []
- orphaned = []
- for f in files_list:
- if f['rel_path'] in used_filepaths:
- used.append(f)
- else:
- orphaned.append(f)
-
- duplicates.append({
- 'hash': file_hash,
- 'used': used,
- 'orphaned': orphaned,
- })
- stats['duplicates'] = duplicates
-
- # Pokazywanie plików na dysku, których brak w bazie (usunięte w bazie)
- if show_deleted:
- disk_files = []
- for root, _, files in os.walk(FILES_BASE_PATH):
- # Sprawdź czy folder jest wykluczony
- rel_root = os.path.relpath(root, FILES_BASE_PATH).replace('\\','/')
- if any(rel_root.startswith(excl) for excl in exclude_folders):
+def disk_only_files(images, config, args):
+ base = os.path.realpath(config.FILES_BASE_PATH)
+ known = {img['rel_path'] for img in images}
+ allowed = {'.' + x.strip().lower().lstrip('.') for x in args.extensions.split(',') if x.strip()}
+ seen = []
+ for root, directories, files in os.walk(base, followlinks=False):
+ directories[:] = [folder for folder in directories if not _excluded(
+ os.path.relpath(os.path.join(root, folder), base).replace(os.sep, '/'), args.exclude_folders
+ )]
+ for filename in files:
+ if os.path.splitext(filename)[1].lower() not in allowed:
continue
+ full_path = os.path.join(root, filename)
+ if not _within(full_path, base):
+ continue
+ rel = os.path.relpath(full_path, base).replace(os.sep, '/')
+ if rel not in known and not _excluded(rel, args.exclude_folders):
+ seen.append(rel)
+ return seen
- for f in files:
- if not f.lower().endswith(allowed_exts):
- continue
- full_path = os.path.join(root, f)
- rel_path = os.path.relpath(full_path, FILES_BASE_PATH).replace('\\', '/')
- if rel_path not in used_filepaths:
- disk_files.append(rel_path)
- stats['deleted'] = disk_files
- connection.close()
- return stats
+def image_url(base_url, public_files_path, rel_path):
+ if not base_url or not public_files_path:
+ return ''
+ return (f'{base_url.rstrip("/")}/' +
+ quote(public_files_path.strip('/') + '/' + rel_path.lstrip('/'), safe='/'))
-def print_summary(stats, to_optimize=False, find_duplicates=False, show_deleted=False, top=20):
- print(f"\n📊 Podsumowanie analizy:")
- print(f" 🔢 Liczba plików: {stats['total']}")
- print(f" ❌ Brakujące pliki: {stats['missing']}")
- print(f" 🛠 Rozmiary różniące się: {stats['mismatched']}")
- print(f" 💾 Potencjalne oszczędności po zmniejszeniu: {stats['saved_bytes_potential'] / (1024**2):.2f} MB")
- print(f"\n🖼 Największe obrazy (top {top}):")
- for img in stats['largest']:
- size_kb = img['actual_size'] / 1024
- size_mb = size_kb / 1024
- print(f" - {img['path']} ({size_kb:.1f} KB / {size_mb:.2f} MB)")
+def display_summary(images, missing, process_stats, args, config, *, deleted=None, duplicates=None):
+ print('\n=== PODSUMOWANIE ===')
+ print(f'Znalezione pliki: {len(images)} | Nieodnalezione: {len(missing)} | '
+ f'Niezgodne rozmiary DB: {process_stats["db_mismatches"]}')
+ print(f'Kandydaci do zmniejszenia: {process_stats["candidates"]} | '
+ f'Zmniejszone: {process_stats["optimized"]} | Bledy: {process_stats["errors"]}')
+ prefix = 'Przewidywane' if not args.apply else 'Uzyskane'
+ print(f'{prefix} oszczednosci: {process_stats["saved_bytes"] / 1024**2:.2f} MB')
+ if process_stats['backups_dir']:
+ print(f'Kopie zapasowe: {process_stats["backups_dir"]}')
+ print('\n=== NAJWIEKSZE OBRAZY ===')
+ base_url = args.site_url if args.site_url is not None else getattr(config, 'DRUPAL_BASE_URL', '')
+ for rank, img in enumerate(images[:args.top], 1):
+ print(f'{rank:3d}. {img["before_size"] / 1024**2:8.2f} MB '
+ f'{img["width"]:5d}x{img["height"]:<5d} '
+ f'fid={",".join(map(str, img["fids"]))} {img["rel_path"]}')
+ if base_url:
+ public_path = (args.public_files_path if args.public_files_path is not None
+ else getattr(config, 'PUBLIC_FILES_PATH', 'sites/default/files'))
+ print(f' PLIK (jesli publiczny): {image_url(base_url, public_path, img["rel_path"])}')
+ if args.find_usage and img.get('usage_checked'):
+ if img['references']:
+ for ref in img['references']:
+ label = f'{ref["title"]} ({ref["node_type"]}, nid={ref["nid"]}'
+ if ref['cid']:
+ label += f', cid={ref["cid"]}'
+ label += f', {ref["confidence"]})'
+ if not ref['published']:
+ label += ' [WPIS NIEOPUBLIKOWANY]'
+ if ref.get('comment_published') is False:
+ label += ' [KOMENTARZ NIEOPUBLIKOWANY]'
+ print(f' -> {label}')
+ print(f' LINK: {ref["url"]}')
+ print(f' EDYCJA: {ref["edit_url"]}')
+ print(f' SKAD: {", ".join(ref["sources"])}')
+ else:
+ print(' -> [BRAK POWIAZANIA] Nie znaleziono w obslugiwanych polach')
+ name = os.path.basename(img['rel_path'])
+ maybe_search_url = search_url(base_url, name, clean_urls=not args.no_clean_urls)
+ if maybe_search_url:
+ print(f' SZUKAJ (Drupal Search, jesli aktywny): {maybe_search_url}')
+ if args.find_usage and args.usage_scope != 'top':
+ shown = {img['path'] for img in images[:args.top]}
+ additional = [img for img in images if img['path'] not in shown and img['references']]
+ if additional:
+ print(f'\n=== POWIAZANIA POZA TOP {args.top} ===')
+ for img in additional:
+ print(f'{img["rel_path"]} ({img["before_size"] / 1024**2:.2f} MB)')
+ for ref in img['references']:
+ print(f' -> {ref["title"]}: {ref["url"]} [{ref["confidence"]}]')
+ print(f'\n=== NAJWYZSZA ROZDZIELCZOSC (TOP {args.top}) ===')
+ for img in sorted(images, key=lambda f: f['width'] * f['height'], reverse=True)[:args.top]:
+ print(f'{img["width"]:5d}x{img["height"]:<5d} {img["before_size"] / 1024**2:8.2f} MB {img["rel_path"]}')
+ if missing:
+ print(f'\n=== PLIKI W BAZIE BEZ PLIKU NA DYSKU (pierwsze 30) ===')
+ for item in missing[:30]:
+ print(f'fid={item["fid"]} {item["db_filepath"]}: {item["reason"]}')
+ if duplicates:
+ print(f'\n=== DUPLIKATY SHA-256: {len(duplicates)} grup ===')
+ for d in duplicates:
+ print(f'Hash {d["hash"][:16]}... | {d["size"] / 1024**2:.2f} MB')
+ for filepath in d['paths']:
+ print(f' {filepath}')
+ if deleted is not None:
+ print(f'\n=== NA DYSKU, ALE NIE W ZESTAWIENIU: {len(deleted)} (pierwsze 60) ===')
+ for rel in deleted[:60]:
+ print(f' {rel}')
+ if args.fid or args.path_contains or args.min_size_mb:
+ print(' [UWAGA] Przy aktywnych filtrach lista zawiera takze pliki odfiltrowane.')
+ if not args.apply:
+ print('\n[DRY-RUN] Nie zmieniono plikow ani bazy. Zapis wymaga --apply.')
- print(f"\n📐 Najwyższe rozdzielczości (top {top}):")
- for img in stats['highest_res']:
- print(f" - {img['path']} ({img['width']}x{img['height']})")
- if to_optimize and stats['to_optimize']:
- print(f"\n⚡ Pliki do optymalizacji ({len(stats['to_optimize'])}):")
- for opt in stats['to_optimize']:
- size_kb = opt['size'] / 1024
- print(f" - {opt['path']} ({opt['res']}, {size_kb:.1f} KB)")
+def duplicate_groups(images):
+ by_hash = defaultdict(list)
+ for image in images:
+ digest = sha256_file(image['path'])
+ by_hash[digest].append(image)
+ return [{'hash': digest, 'size': items[0]['before_size'],
+ 'paths': [img['rel_path'] for img in items]}
+ for digest, items in by_hash.items() if len(items) > 1]
- if find_duplicates and stats['duplicates']:
- print(f"\n🔍 Duplikaty ({len(stats['duplicates'])} grup):")
- for group in stats['duplicates']:
- print(f" Hash: {group['hash']}")
- if group['used']:
- print(" ▶ Używane w systemie:")
- for f in group['used']:
- print(f" - {f['rel_path']}")
- if group['orphaned']:
- print(" ⚠️ Osierocone (nieużywane):")
- for f in group['orphaned']:
- print(f" - {f['rel_path']}")
- if show_deleted and stats['deleted']:
- print(f"\n❌ Pliki na dysku nieznalezione w bazie (usunięte?):")
- for f in stats['deleted']:
- print(f" - {f}")
+def write_csv(csv_path, images, config, args):
+ fieldnames = [
+ 'fid', 'filepath_db', 'filepath', 'rozmiar_przed_B', 'rozmiar_po_lub_plan_B',
+ 'rozmiar_db_B', 'szerokosc', 'wysokosc', 'status_optymalizacji',
+ 'powod', 'nid', 'typ_wpisu', 'tytul', 'cid', 'link_do_wpisu',
+ 'link_edycji', 'link_do_pliku', 'zrodlo_powiazania', 'pewnosc', 'opublikowany', 'komentarz_opublikowany', 'szukaj_w_drupal',
+ ]
+ base_url = args.site_url if args.site_url is not None else getattr(config, 'DRUPAL_BASE_URL', '')
+ public = (args.public_files_path if args.public_files_path is not None
+ else getattr(config, 'PUBLIC_FILES_PATH', 'sites/default/files'))
+ with open(csv_path, 'w', encoding='utf-8-sig', newline='') as target:
+ writer = csv.DictWriter(target, fieldnames=fieldnames, delimiter=';')
+ writer.writeheader()
+ for image in images:
+ for db_row in image['db_rows']:
+ refs = [ref for ref in image['references']
+ if not ref.get('fids') or db_row['fid'] in ref['fids']] or [None]
+ for ref in refs:
+ row = {
+ 'fid': db_row['fid'], 'filepath_db': db_row['db_filepath'],
+ 'filepath': image['rel_path'], 'rozmiar_przed_B': image['before_size'],
+ 'rozmiar_po_lub_plan_B': image['after_size'],
+ 'rozmiar_db_B': db_row['filesize'], 'szerokosc': image['width'],
+ 'wysokosc': image['height'], 'status_optymalizacji': image['optimize_status'],
+ 'powod': image['optimization_reason'],
+ 'link_do_pliku': image_url(base_url, public, image['rel_path']),
+ 'szukaj_w_drupal': search_url(base_url, os.path.basename(image['rel_path']),
+ clean_urls=not args.no_clean_urls),
+ }
+ if ref:
+ row.update({
+ 'nid': ref['nid'], 'typ_wpisu': ref['node_type'], 'tytul': ref['title'],
+ 'cid': ref['cid'] or '', 'link_do_wpisu': ref['url'],
+ 'link_edycji': ref['edit_url'], 'zrodlo_powiazania': ', '.join(ref['sources']),
+ 'pewnosc': ref['confidence'], 'opublikowany': 'tak' if ref['published'] else 'nie',
+ 'komentarz_opublikowany': ('tak' if ref['comment_published'] else 'nie')
+ if ref.get('comment_published') is not None else '',
+ })
+ writer.writerow(row)
-def main():
- parser = argparse.ArgumentParser(description="Analiza plików w Drupal 6")
- parser.add_argument('--debug', action='store_true', help='Tryb debugowania')
- parser.add_argument('--update-db', action='store_true', help='Aktualizuj rozmiary plików w bazie')
- parser.add_argument('--dry-run', action='store_true', help='Symuluj zmiany bez zapisywania')
- parser.add_argument('--extensions', default='jpg,jpeg,png,gif', help='Rozszerzenia plików do analizy')
- parser.add_argument('--exclude-folders', nargs='*', default=[], help='Foldery do wykluczenia (względem files/)')
- parser.add_argument('--find-duplicates', action='store_true', help='Znajdź i pokaż duplikaty')
- parser.add_argument('--show-deleted', action='store_true', help='Pokaż pliki na dysku usunięte z bazy')
- parser.add_argument('--optimize', action='store_true', help='Zmniejsz obrazy powyżej limitów')
- parser.add_argument('--top', type=int, default=20, help='Ilość największych i najwyższych do pokazania')
- args = parser.parse_args()
+def write_json(path, images, missing, stats, duplicates=None, deleted=None):
+ output = {
+ 'mode': 'apply' if stats.get('apply') else 'dry_run_or_read_only',
+ 'stats': stats, 'missing': missing,
+ 'images': images, 'duplicate_groups': duplicates or [], 'disk_only': deleted or [],
+ }
+ with open(path, 'w', encoding='utf-8') as target:
+ json.dump(output, target, ensure_ascii=False, indent=2)
- stats = compare_files(
- debug=args.debug,
- to_optimize=args.optimize,
- update_db=args.update_db,
- dry_run=args.dry_run,
- extensions=args.extensions,
- exclude_folders=args.exclude_folders,
- find_duplicates=args.find_duplicates,
- show_deleted=args.show_deleted,
- top=args.top,
- )
- print_summary(
- stats,
- to_optimize=args.optimize,
- find_duplicates=args.find_duplicates,
- show_deleted=args.show_deleted,
- top=args.top
- )
+def main(argv=None):
+ parser = build_parser()
+ args = parser.parse_args(argv)
+ validate_args(args, parser)
+ try:
+ config = load_config()
+ if not os.path.isdir(config.FILES_BASE_PATH):
+ raise RuntimeError(f'Brak katalogu FILES_BASE_PATH: {config.FILES_BASE_PATH}')
+ with get_connection(config) as connection:
+ images, missing, prefix = collect_images(connection, config, args)
+ print(f'[INFO] Zidentyfikowano {len(images)} obrazow w bazie; '
+ f'{len(missing)} wpisow bez pliku. Tryb: {"APPLY" if args.apply else "DRY-RUN"}.')
+ stats = process_changes(connection, config, args, images, prefix)
+ find_references(connection, args, config, images, prefix)
+ duplicates = duplicate_groups(images) if args.find_duplicates else None
+ deleted = disk_only_files(images, config, args) if args.show_deleted else None
+ display_summary(images, missing, stats, args, config, deleted=deleted, duplicates=duplicates)
+ if args.report_csv:
+ write_csv(args.report_csv, images, config, args)
+ print(f'[RAPORT] CSV: {args.report_csv}')
+ if args.report_json:
+ write_json(args.report_json, images, missing, stats, duplicates=duplicates, deleted=deleted)
+ print(f'[RAPORT] JSON: {args.report_json}')
+ return 1 if stats['errors'] else 0
+ except Exception as exc:
+ print(f'[ERROR] {type(exc).__name__}: {exc}', file=sys.stderr)
+ if args.debug:
+ import traceback
+ traceback.print_exc()
+ return 2
+
if __name__ == '__main__':
- main()
+ sys.exit(main())
diff --git a/requirements.txt b/requirements.txt
index 48f10fb..c6190f3 100644
--- a/requirements.txt
+++ b/requirements.txt
@@ -1,2 +1,2 @@
-pymysql
-Pillow
+PyMySQL>=1.0,<2
+Pillow>=9.1
diff --git a/tests/test_drupal6.py b/tests/test_drupal6.py
new file mode 100644
index 0000000..c3067f8
--- /dev/null
+++ b/tests/test_drupal6.py
@@ -0,0 +1,379 @@
+import io
+import os
+import re
+import sys
+import tempfile
+import types
+import unittest
+from contextlib import redirect_stdout
+from pathlib import Path
+from unittest.mock import patch
+
+from PIL import Image
+
+sys.path.insert(0, str(Path(__file__).resolve().parents[1]))
+import main
+from drupal_lookup import DrupalReferenceLookup, ReferenceMatcher, node_url, search_url
+from image_optimizer import create_optimized_candidate
+
+
+class FakeCursor:
+ def __init__(self, conn):
+ self.conn = conn
+ self.results = []
+ self.index = 0
+ self.rowcount = 0
+
+ def __enter__(self):
+ return self
+
+ def __exit__(self, *_):
+ return False
+
+ def close(self):
+ pass
+
+ def execute(self, sql, params=None):
+ params = params or ()
+ self.index = 0
+ self.rowcount = 0
+ self.results = []
+ if sql == 'SHOW TABLES':
+ self.results = [{'Tables_in_test': table} for table in self.conn.schemas]
+ elif sql.startswith('SHOW COLUMNS FROM'):
+ table = re.search(r'`([^`]+)`', sql).group(1)
+ self.results = [{'Field': name, 'Type': data_type} for name, data_type in self.conn.schemas[table].items()]
+ elif sql.startswith('SELECT fid, filepath, filesize FROM'):
+ self.results = [dict(row) for row in self.conn.files]
+ elif sql.startswith('SELECT filesize, filepath FROM'):
+ self.results = [dict(row) for row in self.conn.files if row['fid'] == params[0]]
+ elif sql.startswith('UPDATE `files` SET filesize'):
+ if self.conn.fail_updates:
+ self.conn.fail_updates -= 1
+ raise RuntimeError('Simulated DB error')
+ size, fid, path = params
+ for row in self.conn.files:
+ if row['fid'] == fid and row['filepath'] == path:
+ if row['filesize'] != size:
+ row['filesize'] = size
+ self.rowcount = 1
+ break
+ return
+ elif 'FROM `upload` t JOIN' in sql or 'FROM `content_field_scan` t JOIN' in sql:
+ table = 'upload' if 'FROM `upload`' in sql else 'content_field_scan'
+ field = 'fid' if table == 'upload' else 'field_scan_fid'
+ for entry in self.conn.relations.get(table, []):
+ node = self.conn.nodes.get(entry['nid'])
+ if not node or (entry.get('vid') and node['vid'] != entry['vid']):
+ continue
+ if entry[field] in params:
+ self.results.append({
+ 'nid': node['nid'], 'title': node['title'],
+ 'node_type': node['type'], 'node_status': node['status'],
+ 'fid': entry[field], 'cid': entry.get('cid'),
+ })
+ elif 'FROM `comment_upload` t' in sql:
+ for entry in self.conn.relations.get('comment_upload', []):
+ comment = next(c for c in self.conn.comments if c['cid'] == entry['cid'])
+ node = self.conn.nodes[comment['nid']]
+ if entry['fid'] in params:
+ self.results.append({'nid': node['nid'], 'title': node['title'],
+ 'node_type': node['type'], 'node_status': node['status'],
+ 'comment_status': comment['status'],
+ 'cid': comment['cid'], 'fid': entry['fid']})
+ elif 'r.body AS body' in sql:
+ for node in self.conn.nodes.values():
+ for revision in self.conn.revisions:
+ if revision['nid'] == node['nid'] and revision['vid'] == node['vid']:
+ self.results.append({
+ 'nid': node['nid'], 'title': node['title'], 'node_type': node['type'],
+ 'node_status': node['status'], 'body': revision['body'],
+ })
+ elif 'c.cid, c.status AS comment_status' in sql:
+ for comment in self.conn.comments:
+ node = self.conn.nodes[comment['nid']]
+ self.results.append({
+ 'nid': node['nid'], 'title': node['title'], 'node_type': node['type'],
+ 'node_status': node['status'], 'cid': comment['cid'],
+ 'comment_status': comment['status'], 'comment': comment['comment'],
+ })
+ else:
+ raise AssertionError(f'Unexpected fake SQL: {sql} / {params}')
+ self.rowcount = len(self.results)
+
+ def fetchall(self):
+ return list(self.results)
+
+ def fetchone(self):
+ return self.results[0] if self.results else None
+
+ def fetchmany(self, count):
+ result = self.results[self.index:self.index + count]
+ self.index += len(result)
+ return result
+
+
+class FakeConnection:
+ def __init__(self):
+ self.schemas = {
+ 'files': {'fid': 'int', 'filepath': 'varchar(255)', 'filesize': 'int'},
+ 'node': {'nid': 'int', 'vid': 'int', 'title': 'varchar(255)', 'type': 'varchar(32)', 'status': 'tinyint'},
+ 'node_revisions': {'nid': 'int', 'vid': 'int', 'body': 'longtext'},
+ 'upload': {'fid': 'int', 'nid': 'int', 'vid': 'int'},
+ 'content_field_scan': {'field_scan_fid': 'int', 'nid': 'int', 'vid': 'int'},
+ 'comments': {'cid': 'int', 'nid': 'int', 'comment': 'longtext', 'status': 'tinyint'},
+ 'comment_upload': {'cid': 'int', 'fid': 'int'},
+ }
+ self.nodes = {
+ 11: {'nid': 11, 'vid': 101, 'title': 'Stary skan mapy', 'type': 'forum', 'status': 1},
+ 12: {'nid': 12, 'vid': 102, 'title': 'Zdjecia z wakacji', 'type': 'story', 'status': 1},
+ 13: {'nid': 13, 'vid': 103, 'title': 'Odpowiedz na forum', 'type': 'forum', 'status': 0},
+ }
+ self.relations = {
+ 'upload': [{'nid': 11, 'vid': 101, 'fid': 7}, {'nid': 13, 'vid': 999, 'fid': 7}],
+ 'content_field_scan': [{'nid': 12, 'vid': 102, 'field_scan_fid': 8}],
+ 'comment_upload': [{'cid': 55, 'fid': 8}],
+ }
+ self.revisions = [
+ {'nid': 11, 'vid': 101, 'body': '
'},
+ {'nid': 12, 'vid': 102, 'body': 'Wspomnienia.
'},
+ {'nid': 13, 'vid': 103, 'body': 'Zobacz mapa.jpg tutaj.'},
+ {'nid': 13, 'vid': 999, 'body': 'NIEAKTUALNE /sites/default/files/scans/mapa.jpg'},
+ ]
+ self.comments = [
+ {'cid': 55, 'nid': 12, 'comment': 'Znalazlem /sites/default/files/scans/mapa.jpg', 'status': 1},
+ ]
+ self.files = []
+ self.fail_updates = 0
+ self.commits = 0
+ self.rollbacks = 0
+
+ def __enter__(self):
+ return self
+
+ def __exit__(self, *_):
+ return False
+
+ def cursor(self, *_):
+ return FakeCursor(self)
+
+ def commit(self):
+ self.commits += 1
+
+ def rollback(self):
+ self.rollbacks += 1
+
+
+class TestLinks(unittest.TestCase):
+ def test_links(self):
+ self.assertEqual(node_url('https://example.org/forum/', 42), 'https://example.org/forum/node/42')
+ self.assertEqual(node_url('https://example.org/forum', 42, 66), 'https://example.org/forum/node/42#comment-66')
+ self.assertEqual(node_url('https://example.org/forum', 42, clean_urls=False), 'https://example.org/forum/?q=node/42')
+ self.assertIn('skan%20mapa.jpg', search_url('https://example.org', 'skan mapa.jpg'))
+ self.assertEqual(node_url('https://example.org', 42, 66, comment_permalinks=True),
+ 'https://example.org/comment/66#comment-66')
+
+ def test_matcher_precision(self):
+ matcher = ReferenceMatcher([
+ {'fid': 7, 'db_filepath': 'sites/default/files/scans/mapa.jpg', 'rel_path': 'scans/mapa.jpg'},
+ ])
+ self.assertEqual(matcher.match('')[7], 'sciezka')
+ self.assertEqual(matcher.match('mapa.jpg')[7], 'nazwa')
+ self.assertFalse(matcher.match('supermapa.jpg'))
+ self.assertFalse(ReferenceMatcher([
+ {'fid': 7, 'db_filepath': 'sites/default/files/scans/mapa.jpg', 'rel_path': 'scans/mapa.jpg'}
+ ], basename_match=False).match('mapa.jpg'))
+
+ def test_schema_references_current_only_and_comment_status(self):
+ conn = FakeConnection()
+ lookup = DrupalReferenceLookup(conn, base_url='https://forum.example.org')
+ files = [
+ {'fid': 7, 'db_filepath': 'sites/default/files/scans/mapa.jpg', 'rel_path': 'scans/mapa.jpg'},
+ {'fid': 8, 'db_filepath': 'sites/default/files/inne.jpg', 'rel_path': 'inne.jpg'},
+ ]
+ fake_pymysql = types.SimpleNamespace(cursors=types.SimpleNamespace(SSDictCursor=object))
+ with patch.dict(sys.modules, {'pymysql': fake_pymysql}):
+ with redirect_stdout(io.StringIO()):
+ result = lookup.lookup(files)
+ self.assertEqual({x['nid'] for x in result[7]}, {11, 12, 13})
+ self.assertEqual({x['nid'] for x in result[8]}, {12})
+ attached_to_comment = [x for x in result[8] if x['cid'] == 55][0]
+ self.assertIn('comment_upload.fid', attached_to_comment['sources'])
+ direct = [x for x in result[7] if x['nid'] == 11][0]
+ self.assertIn('upload.fid', direct['sources'])
+ self.assertIn('tresc wpisu (sciezka)', direct['sources'])
+ self.assertEqual(direct['confidence'], 'pewne')
+ self.assertTrue(direct['url'].endswith('/node/11'))
+ comment = [x for x in result[7] if x['cid'] == 55][0]
+ self.assertFalse(comment['comment_published'])
+ self.assertEqual(comment['url'], 'https://forum.example.org/node/12#comment-55')
+ possible = [x for x in result[7] if x['nid'] == 13][0]
+ self.assertFalse(possible['published'])
+ self.assertEqual(possible['confidence'], 'mozliwe')
+ self.assertNotIn('upload.fid', possible['sources']) # nieaktualna rewizja
+
+
+class TestFiles(unittest.TestCase):
+ def setUp(self):
+ self.workdir = tempfile.TemporaryDirectory()
+ self.addCleanup(self.workdir.cleanup)
+ self.root = Path(self.workdir.name)
+ self.site_root = self.root / 'drupal'
+ self.folder = self.site_root / 'sites' / 'default' / 'files'
+ (self.folder / 'scans').mkdir(parents=True)
+ self.image = self.folder / 'scans' / 'mapa.jpg'
+ src = Image.effect_noise((950, 1250), 115).convert('RGB')
+ src.save(self.image, 'JPEG', quality=99)
+ self.conn = FakeConnection()
+ self.conn.files = [{
+ 'fid': 7, 'filepath': 'sites/default/files/scans/mapa.jpg', 'filesize': 12345,
+ }]
+ self.cfg = types.SimpleNamespace(
+ FILES_BASE_PATH=str(self.folder), PUBLIC_FILES_PATH='sites/default/files',
+ DRUPAL_ROOT=str(self.site_root), DRUPAL_BASE_URL='https://forum.example.org',
+ )
+
+ def args(self, *overrides):
+ return main.build_parser().parse_args(list(overrides))
+
+ def test_resolve_prevent_traversal(self):
+ valid, rel = main.resolve_filepath('sites/default/files/scans/mapa.jpg', str(self.folder))
+ self.assertEqual(valid, str(self.image))
+ self.assertEqual(rel, 'scans/mapa.jpg')
+ bad, _ = main.resolve_filepath('../../etc/shadow', str(self.folder))
+ self.assertIsNone(bad)
+ outside, _ = main.resolve_filepath('/etc/passwd', str(self.folder))
+ self.assertIsNone(outside)
+
+ def test_backup_must_not_be_in_webroot(self):
+ with self.assertRaises(ValueError):
+ main._safe_backup_dir(str(self.site_root / 'backup'),
+ str(self.folder), str(self.site_root))
+
+ def test_dry_run_no_changes_and_preview(self):
+ args = self.args('--optimize', '--update-db', '--dry-run', '--max-width', '600',
+ '--max-height', '800', '--max-file-mb', '0.5')
+ images, missing, prefix = main.collect_images(self.conn, self.cfg, args)
+ self.assertFalse(missing)
+ before = self.image.read_bytes()
+ with redirect_stdout(io.StringIO()):
+ stats = main.process_changes(self.conn, self.cfg, args, images, prefix)
+ self.assertEqual(self.image.read_bytes(), before)
+ self.assertEqual(self.conn.files[0]['filesize'], 12345)
+ self.assertEqual(stats['optimized'], 0)
+ self.assertEqual(stats['planned'], 1)
+ self.assertEqual(images[0]['optimize_status'], 'planned')
+ self.assertLess(images[0]['after_size'], images[0]['before_size'])
+
+ def test_apply_backup_db_and_restore_on_sql_failure(self):
+ args = self.args('--optimize', '--apply', '--max-width', '600', '--max-height', '800',
+ '--max-file-mb', '0.5', '--backup-dir', str(self.root / 'backups'))
+ before = self.image.read_bytes()
+ images, _, prefix = main.collect_images(self.conn, self.cfg, args)
+ with redirect_stdout(io.StringIO()):
+ stats = main.process_changes(self.conn, self.cfg, args, images, prefix)
+ self.assertEqual(stats['optimized'], 1)
+ self.assertEqual(stats['errors'], 0)
+ self.assertLess(self.image.stat().st_size, len(before))
+ self.assertEqual(self.conn.files[0]['filesize'], self.image.stat().st_size)
+ self.assertEqual(Path(images[0]['backup_file']).read_bytes(), before)
+ with Image.open(self.image) as img:
+ self.assertLessEqual(img.width, 600)
+ self.assertLessEqual(img.height, 800)
+
+ # Ponownie uruchomienie z bladem w DB: plik i metadata wracaja.
+ self.conn.fail_updates = 1
+ self.conn.files[0]['filesize'] = 12345
+ second_before = self.image.read_bytes()
+ args2 = self.args('--optimize', '--apply', '--max-width', '400', '--max-height', '550',
+ '--max-file-mb', '0.3', '--backup-dir', str(self.root / 'backups'))
+ images2, _, prefix = main.collect_images(self.conn, self.cfg, args2)
+ with redirect_stdout(io.StringIO()):
+ stats2 = main.process_changes(self.conn, self.cfg, args2, images2, prefix)
+ self.assertEqual(stats2['errors'], 1)
+ self.assertEqual(self.image.read_bytes(), second_before)
+ self.assertEqual(self.conn.files[0]['filesize'], 12345)
+
+ def test_symlink_is_not_modified(self):
+ link = self.folder / 'scans' / 'alias.jpg'
+ link.symlink_to(self.image)
+ self.conn.files[0]['filepath'] = 'sites/default/files/scans/alias.jpg'
+ args = self.args('--optimize', '--dry-run', '--max-width', '500')
+ images, _, prefix = main.collect_images(self.conn, self.cfg, args)
+ self.assertEqual(images[0]['path'], str(link))
+ with redirect_stdout(io.StringIO()):
+ stats = main.process_changes(self.conn, self.cfg, args, images, prefix)
+ self.assertEqual(images[0]['optimize_status'], 'skip')
+ self.assertEqual(stats['planned'], 0)
+ self.assertTrue(link.is_symlink())
+
+ def test_exif_orientation_after_optimization(self):
+ original = self.folder / 'scan-orientation.jpg'
+ img = Image.new('RGB', (240, 120), 'orange')
+ exif = Image.Exif()
+ exif[274] = 6
+ img.save(original, 'JPEG', quality=100, exif=exif)
+ optimized = create_optimized_candidate(str(original), max_width=150, max_height=200,
+ max_bytes=100, min_gain_percent=0)
+ self.assertEqual(optimized['status'], 'ready', optimized['reason'])
+ try:
+ with Image.open(optimized['temp_path']) as out:
+ self.assertEqual(out.size, (100, 200))
+ self.assertNotEqual(out.getexif().get(274), 6)
+ finally:
+ os.unlink(optimized['temp_path'])
+
+ def test_png_with_alpha_preserved(self):
+ filename = self.folder / 'scans' / 'alpha.png'
+ img = Image.new('RGBA', (300, 900), (10, 100, 50, 130))
+ img.save(filename)
+ result = create_optimized_candidate(str(filename), max_width=200, max_height=500,
+ max_bytes=200, min_gain_percent=0)
+ self.assertEqual(result['status'], 'ready', result['reason'])
+ try:
+ with Image.open(result['temp_path']) as output:
+ self.assertEqual(output.mode, 'RGBA')
+ self.assertEqual(output.size, (167, 500))
+ self.assertEqual(output.getpixel((1, 1))[3], 130)
+ finally:
+ os.unlink(result['temp_path'])
+
+ def test_no_optimization_for_small_or_gif(self):
+ tiny = self.folder / 'tiny.png'
+ Image.new('RGB', (30, 30), 'red').save(tiny)
+ self.assertEqual(create_optimized_candidate(str(tiny))['status'], 'skip')
+ gif = self.folder / 'animated.gif'
+ Image.new('RGB', (300, 500), 'red').save(gif)
+ self.assertEqual(create_optimized_candidate(str(gif), max_width=100)['status'], 'skip')
+
+ def test_cli_end_to_end_read_only_generates_reports(self):
+ csv_path = self.root / 'end.csv'
+ json_path = self.root / 'end.json'
+ with patch.object(main, 'load_config', return_value=self.cfg), \
+ patch.object(main, 'get_connection', return_value=self.conn), \
+ redirect_stdout(io.StringIO()):
+ status = main.main(['--find-usage', '--no-text-scan', '--top', '10',
+ '--report-csv', str(csv_path),
+ '--report-json', str(json_path)])
+ self.assertEqual(status, 0)
+ self.assertIn('https://forum.example.org/node/11', csv_path.read_text(encoding='utf-8-sig'))
+ self.assertIn('"images"', json_path.read_text(encoding='utf-8'))
+ self.assertEqual(self.conn.files[0]['filesize'], 12345)
+
+ def test_csv_has_direct_link(self):
+ args = self.args('--find-usage', '--site-url', 'https://forum.example.org', '--report-csv', str(self.root / 'out.csv'))
+ images, _, prefix = main.collect_images(self.conn, self.cfg, args)
+ images[0]['references'] = [
+ {'nid': 11, 'cid': None, 'title': 'Przypiety temat', 'node_type': 'forum',
+ 'published': True, 'comment_published': None, 'sources': ['upload.fid'],
+ 'confidence': 'pewne', 'url': 'https://forum.example.org/node/11',
+ 'edit_url': 'https://forum.example.org/node/11/edit'}
+ ]
+ main.write_csv(args.report_csv, images, self.cfg, args)
+ text = Path(args.report_csv).read_text(encoding='utf-8-sig')
+ self.assertIn('https://forum.example.org/node/11', text)
+ self.assertIn('Przypiety temat', text)
+
+
+if __name__ == '__main__':
+ unittest.main()