Initial commit
This commit is contained in:
45
README.md
Normal file
45
README.md
Normal file
@@ -0,0 +1,45 @@
|
|||||||
|
# RAG-Crawler
|
||||||
|
|
||||||
|
### Projekt-Zusammenfassung
|
||||||
|
|
||||||
|
Der RAG-Crawler ist ein asynchroner, konfigurierbarer Web-Crawler, der entwickelt wurde, um Inhalte von Webseiten systematisch zu extrahieren und für die spätere Verwendung in RAG-Systemen (Retrieval-Augmented Generation) zu speichern. Er ist modular aufgebaut und bietet erweiterte Funktionen wie Zustandsverwaltung zum Pausieren und Fortsetzen, Duplikaterkennung zur Vermeidung redundanter Daten und inkrementelles Crawling, um nur geänderte Inhalte erneut zu verarbeiten. Die Konfiguration ermöglicht eine feingranulare Steuerung des Crawling-Verhaltens, des Loggings und der Datenbereinigung.
|
||||||
|
|
||||||
|
### Feature-Liste
|
||||||
|
|
||||||
|
* **Asynchrones Crawling:** Nutzt `aiohttp` für performante, nicht-blockierende HTTP-Anfragen.
|
||||||
|
* **Zustandsverwaltung:** Speichert den Fortschritt (URL-Warteschlange und besuchte Links) in einer Datei, um den Crawling-Prozess unterbrechen und später fortsetzen zu können.
|
||||||
|
* **Duplikaterkennung:** Verwendet SimHash, um semantisch ähnliche Inhalte zu identifizieren und das Speichern von Duplikaten zu verhindern.
|
||||||
|
* **Inkrementelles Crawling:** Führt eine SQLite-Datenbank mit den Hashes der Seiteninhalte, um nur Seiten zu verarbeiten, deren Inhalt sich seit dem letzten Crawl geändert hat.
|
||||||
|
* **Konfigurierbare Crawling-Modi:** Unterstützt verschiedene Modi pro Start-URL:
|
||||||
|
* `single_page`: Crawlt nur die angegebene Start-URL.
|
||||||
|
* `domain_wide`: Crawlt die gesamte Domain, ausgehend von der Start-URL.
|
||||||
|
* `path_limited`: Beschränkt das Crawling auf einen bestimmten URL-Pfad.
|
||||||
|
* **HTML-Bereinigung:** Entfernt vor der Inhaltsextraktion unerwünschte HTML-Tags (z.B. Header, Footer) und Textmuster (z.B. "Datenschutz"), um die Datenqualität zu verbessern.
|
||||||
|
* **`robots.txt`-Unterstützung:** Respektiert die in der `robots.txt` einer Webseite definierten Crawling-Regeln.
|
||||||
|
* **Flexibles Logging:** Unterstützt das Logging in die Konsole, rotierende Log-Dateien sowie in SQLite- und MySQL-Datenbanken.
|
||||||
|
* **Statistik-Tracking:** Erfasst detaillierte Statistiken für jeden Crawling-Vorgang (Dauer, besuchte Seiten, gespeicherte Datenmenge, Fehler).
|
||||||
|
* **URL-Normalisierung:** Bereinigt URLs, indem Fragmente und gängige Tracking-Parameter entfernt werden, um die Effizienz der Duplikaterkennung zu erhöhen.
|
||||||
|
|
||||||
|
### Konfigurations-Tabelle (`config.json`)
|
||||||
|
|
||||||
|
| Parameter | Beschreibung | Standardwert |
|
||||||
|
| :--- | :--- | :--- |
|
||||||
|
| `PAGE_LIMIT` | Die maximale Anzahl von Seiten, die pro Domain gecrawlt werden. | `400` |
|
||||||
|
| `CRAWL_DELAY` | Die Verzögerung in Sekunden zwischen Anfragen an dieselbe Domain. | `2` |
|
||||||
|
| `USER_AGENT` | Der User-Agent-String, der für HTTP-Anfragen verwendet wird. | `"Mozilla/5.0 ..."` |
|
||||||
|
| `PATH_STRICT` | (Legacy) Wenn `true`, werden nur URLs gecrawlt, die mit einer der `start_urls` beginnen. | `false` |
|
||||||
|
| `BLOCKED_PATTERNS` | URL-Muster, die vom Crawler ignoriert werden sollen. | `["/logout", "/auth", ...]` |
|
||||||
|
| `MAX_RETRIES` | Maximale Anzahl von Wiederholungen für fehlgeschlagene HTTP-Anfragen. | `3` |
|
||||||
|
| `RETRY_DELAY_BASE` | Basisverzögerung in Sekunden für exponentielles Backoff bei Wiederholungen. | `5` |
|
||||||
|
| `MIN_CONTENT_LENGTH` | Minimale Länge des extrahierten Textes (in Zeichen), damit eine Seite gespeichert wird. | `500` |
|
||||||
|
| `OUTPUT_DIR` | Das Verzeichnis, in dem die gecrawlten Daten als JSON-Dateien gespeichert werden. | `"/app/output"` |
|
||||||
|
| `log` | Konfiguration für das Logging-System (Handler, Dateipfade, DB-Zugangsdaten). | `{...}` |
|
||||||
|
| `html_cleaner` | Konfiguration zur Bereinigung von HTML (zu entfernende Tags und Textmuster). | `{...}` |
|
||||||
|
| `priority_patterns` | URL-Pfade, die in der Crawling-Warteschlange priorisiert werden. | `["/docs/", "/wiki/", ...]` |
|
||||||
|
| `duplicate_detection` | Aktiviert/deaktiviert die Duplikaterkennung und setzt den Ähnlichkeitsschwellenwert. | `{ "enable": true, ... }` |
|
||||||
|
| `incremental_crawling` | Aktiviert/deaktiviert das inkrementelle Crawling und gibt den Pfad zur Hash-Datenbank an. | `{ "enable": true, ... }` |
|
||||||
|
| `state_management` | Aktiviert/deaktiviert die Zustandsverwaltung und gibt den Namen der Zustandsdatei an. | `{ "enable": true, ... }` |
|
||||||
|
|
||||||
|
### Zweck der `url_list.json`
|
||||||
|
|
||||||
|
Die Datei `url_list.json` dient als Eingabe für den Crawler und definiert die zu verarbeitenden "Quellen". Sie ist eine Liste von Objekten, wobei jedes Objekt eine Quelle mit einer oder mehreren `start_urls` und einem spezifischen `crawl_mode` (z.B. `single_page` oder `domain_wide`) repräsentiert. Dies ermöglicht es, mehrere Webseiten mit unterschiedlichen Crawling-Strategien in einem einzigen Durchlauf zu verarbeiten.
|
||||||
96
config.json
Normal file
96
config.json
Normal file
@@ -0,0 +1,96 @@
|
|||||||
|
{
|
||||||
|
"_documentation": "Allgemeine Konfigurationseinstellungen für den Web-Crawler.",
|
||||||
|
"PAGE_LIMIT": {
|
||||||
|
"value": 400,
|
||||||
|
"description": "Die maximale Anzahl von Seiten, die pro Domain gecrawlt werden."
|
||||||
|
},
|
||||||
|
"CRAWL_DELAY": {
|
||||||
|
"value": 2,
|
||||||
|
"description": "Die Verzögerung in Sekunden zwischen aufeinanderfolgenden Anfragen an dieselbe Domain."
|
||||||
|
},
|
||||||
|
"USER_AGENT": {
|
||||||
|
"value": "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:109.0) Gecko/20100101 Firefox/115.0",
|
||||||
|
"description": "Der User-Agent-String, der für HTTP-Anfragen verwendet wird."
|
||||||
|
},
|
||||||
|
"PATH_STRICT": {
|
||||||
|
"value": false,
|
||||||
|
"description": "Wenn true, werden nur URLs gecrawlt, die mit einem der start_urls beginnen."
|
||||||
|
},
|
||||||
|
"BLOCKED_PATTERNS": {
|
||||||
|
"value": [
|
||||||
|
"/logout",
|
||||||
|
"/auth",
|
||||||
|
"?session="
|
||||||
|
],
|
||||||
|
"description": "URL-Muster, die vom Crawler blockiert werden sollen."
|
||||||
|
},
|
||||||
|
"MAX_RETRIES": {
|
||||||
|
"value": 3,
|
||||||
|
"description": "Die maximale Anzahl von Wiederholungen für fehlgeschlagene HTTP-Anfragen."
|
||||||
|
},
|
||||||
|
"RETRY_DELAY_BASE": {
|
||||||
|
"value": 5,
|
||||||
|
"description": "Die Basisverzögerung in Sekunden für exponentielles Backoff bei Wiederholungen."
|
||||||
|
},
|
||||||
|
"MIN_CONTENT_LENGTH": {
|
||||||
|
"value": 500,
|
||||||
|
"description": "Die minimale Länge des Inhalts (in Zeichen), die eine Seite haben muss, um gespeichert zu werden."
|
||||||
|
},
|
||||||
|
"OUTPUT_DIR": {
|
||||||
|
"value": "/app/output",
|
||||||
|
"description": "Das Verzeichnis, in dem die gecrawlten Daten gespeichert werden."
|
||||||
|
},
|
||||||
|
"log": {
|
||||||
|
"value": {
|
||||||
|
"handlers": ["file", "console", "sqlite"],
|
||||||
|
"file": {
|
||||||
|
"log_file": "crawler.log",
|
||||||
|
"max_size_mb": 12,
|
||||||
|
"keep_last": 7
|
||||||
|
},
|
||||||
|
"sqlite": {
|
||||||
|
"db_file": "crawler_logs.db"
|
||||||
|
},
|
||||||
|
"mysql": {
|
||||||
|
"host": "localhost",
|
||||||
|
"user": "user",
|
||||||
|
"password": "password",
|
||||||
|
"database": "crawler_logs",
|
||||||
|
"log_stats": true
|
||||||
|
}
|
||||||
|
},
|
||||||
|
"description": "Konfiguration für das Logging-System."
|
||||||
|
},
|
||||||
|
"html_cleaner": {
|
||||||
|
"value": {
|
||||||
|
"remove_tags": ["header", "nav", "aside", "footer"],
|
||||||
|
"remove_patterns": ["Datenschutz", "Kontakt", "Newsletter", "Zuletzt aktualisiert"]
|
||||||
|
},
|
||||||
|
"description": "Konfiguration zur Bereinigung von HTML-Inhalten."
|
||||||
|
},
|
||||||
|
"priority_patterns": {
|
||||||
|
"value": ["/docs/", "/wiki/", "/handbuch/", "/kapitel/"],
|
||||||
|
"description": "URL-Pfade, die beim Crawling priorisiert werden sollen."
|
||||||
|
},
|
||||||
|
"duplicate_detection": {
|
||||||
|
"value": {
|
||||||
|
"enable": true,
|
||||||
|
"similarity_threshold": 95
|
||||||
|
},
|
||||||
|
"description": "Konfiguration für die Duplikat-Erkennung mittels SimHash."
|
||||||
|
},
|
||||||
|
"incremental_crawling": {
|
||||||
|
"value": {
|
||||||
|
"enable": true,
|
||||||
|
"db_file": "crawled_hashes.db"
|
||||||
|
},
|
||||||
|
"description": "Konfiguration für das inkrementelle Crawling."
|
||||||
|
},
|
||||||
|
"state_management": {
|
||||||
|
"value": {
|
||||||
|
"enable": true,
|
||||||
|
"state_file": "crawler_state.json"
|
||||||
|
},
|
||||||
|
"description": "Konfiguration für die Speicherung des Crawler-Zustands zum Fortsetzen."
|
||||||
|
}
|
||||||
|
}
|
||||||
BIN
crawled_hashes.db
Normal file
BIN
crawled_hashes.db
Normal file
Binary file not shown.
683
crawler.log
Normal file
683
crawler.log
Normal file
@@ -0,0 +1,683 @@
|
|||||||
|
2025-11-12 04:04:00,437 - INFO - Logging-System erfolgreich initialisiert.
|
||||||
|
2025-11-12 04:04:00,448 - INFO - Processing source: https://sqlite.org/docs.html
|
||||||
|
2025-11-12 04:04:00,454 - INFO - Starte Quelle: https://sqlite.org/docs.html (Modus: single_page)
|
||||||
|
2025-11-12 04:04:00,456 - INFO - Crawle: https://sqlite.org/docs.html
|
||||||
|
2025-11-12 04:04:00,459 - INFO - Lade robots.txt von: https://sqlite.org/robots.txt
|
||||||
|
2025-11-12 04:04:01,621 - ERROR - Allgemeiner Fehler beim Crawlen von https://sqlite.org/docs.html: [Errno 13] Permission denied: '/app'
|
||||||
|
2025-11-12 04:04:03,627 - INFO - Quelle https://sqlite.org/docs.html erfolgreich gecrawlt.
|
||||||
|
2025-11-12 04:04:03,634 - INFO - Lösche Zustandsdatei.
|
||||||
|
2025-11-12 04:04:03,640 - INFO - Datenbankverbindung für Hashes geschlossen.
|
||||||
|
2025-11-12 04:04:03,646 - INFO - Finished source: https://sqlite.org/docs.html
|
||||||
|
2025-11-12 04:04:03,656 - INFO - Processing source: https://www.gesetze-im-internet.de/ao_1977/BJNR006130976.html
|
||||||
|
2025-11-12 04:04:03,662 - INFO - Starte Quelle: https://www.gesetze-im-internet.de/ao_1977/BJNR006130976.html (Modus: single_page)
|
||||||
|
2025-11-12 04:04:03,667 - INFO - Crawle: https://www.gesetze-im-internet.de/ao_1977/BJNR006130976.html
|
||||||
|
2025-11-12 04:04:03,671 - INFO - Lade robots.txt von: https://www.gesetze-im-internet.de/robots.txt
|
||||||
|
2025-11-12 04:04:04,001 - ERROR - Allgemeiner Fehler beim Crawlen von https://www.gesetze-im-internet.de/ao_1977/BJNR006130976.html: 'utf-8' codec can't decode byte 0xfc in position 945: invalid start byte
|
||||||
|
2025-11-12 04:04:06,006 - INFO - Quelle https://www.gesetze-im-internet.de/ao_1977/BJNR006130976.html erfolgreich gecrawlt.
|
||||||
|
2025-11-12 04:04:06,013 - INFO - Lösche Zustandsdatei.
|
||||||
|
2025-11-12 04:04:06,022 - INFO - Datenbankverbindung für Hashes geschlossen.
|
||||||
|
2025-11-12 04:04:06,027 - INFO - Finished source: https://www.gesetze-im-internet.de/ao_1977/BJNR006130976.html
|
||||||
|
2025-11-12 04:04:06,035 - INFO - Processing source: https://www.gesetze-im-internet.de/gmbhg/BJNR004770892.html
|
||||||
|
2025-11-12 04:04:06,044 - INFO - Starte Quelle: https://www.gesetze-im-internet.de/gmbhg/BJNR004770892.html (Modus: single_page)
|
||||||
|
2025-11-12 04:04:06,052 - INFO - Crawle: https://www.gesetze-im-internet.de/gmbhg/BJNR004770892.html
|
||||||
|
2025-11-12 04:04:06,111 - ERROR - Allgemeiner Fehler beim Crawlen von https://www.gesetze-im-internet.de/gmbhg/BJNR004770892.html: 'utf-8' codec can't decode byte 0xe4 in position 324: invalid continuation byte
|
||||||
|
2025-11-12 04:04:08,117 - INFO - Quelle https://www.gesetze-im-internet.de/gmbhg/BJNR004770892.html erfolgreich gecrawlt.
|
||||||
|
2025-11-12 04:04:08,122 - INFO - Lösche Zustandsdatei.
|
||||||
|
2025-11-12 04:04:08,131 - INFO - Datenbankverbindung für Hashes geschlossen.
|
||||||
|
2025-11-12 04:04:08,138 - INFO - Finished source: https://www.gesetze-im-internet.de/gmbhg/BJNR004770892.html
|
||||||
|
2025-11-12 04:04:08,142 - INFO - Processing source: https://www.gesetze-im-internet.de/pangv_2022/BJNR492110021.html
|
||||||
|
2025-11-12 04:04:08,148 - INFO - Starte Quelle: https://www.gesetze-im-internet.de/pangv_2022/BJNR492110021.html (Modus: single_page)
|
||||||
|
2025-11-12 04:04:08,153 - INFO - Crawle: https://www.gesetze-im-internet.de/pangv_2022/BJNR492110021.html
|
||||||
|
2025-11-12 04:04:08,203 - ERROR - Allgemeiner Fehler beim Crawlen von https://www.gesetze-im-internet.de/pangv_2022/BJNR492110021.html: 'utf-8' codec can't decode byte 0xfc in position 956: invalid start byte
|
||||||
|
2025-11-12 04:04:10,207 - INFO - Quelle https://www.gesetze-im-internet.de/pangv_2022/BJNR492110021.html erfolgreich gecrawlt.
|
||||||
|
2025-11-12 04:04:10,212 - INFO - Lösche Zustandsdatei.
|
||||||
|
2025-11-12 04:04:10,218 - INFO - Datenbankverbindung für Hashes geschlossen.
|
||||||
|
2025-11-12 04:04:10,224 - INFO - Finished source: https://www.gesetze-im-internet.de/pangv_2022/BJNR492110021.html
|
||||||
|
2025-11-12 04:04:10,233 - INFO - Processing source: https://www.gesetze-im-internet.de/ustg_1980/BJNR119530979.html
|
||||||
|
2025-11-12 04:04:10,240 - INFO - Starte Quelle: https://www.gesetze-im-internet.de/ustg_1980/BJNR119530979.html (Modus: single_page)
|
||||||
|
2025-11-12 04:04:10,247 - INFO - Crawle: https://www.gesetze-im-internet.de/ustg_1980/BJNR119530979.html
|
||||||
|
2025-11-12 04:04:10,370 - ERROR - Allgemeiner Fehler beim Crawlen von https://www.gesetze-im-internet.de/ustg_1980/BJNR119530979.html: 'utf-8' codec can't decode byte 0xfc in position 951: invalid start byte
|
||||||
|
2025-11-12 04:04:12,374 - INFO - Quelle https://www.gesetze-im-internet.de/ustg_1980/BJNR119530979.html erfolgreich gecrawlt.
|
||||||
|
2025-11-12 04:04:12,381 - INFO - Lösche Zustandsdatei.
|
||||||
|
2025-11-12 04:04:12,388 - INFO - Datenbankverbindung für Hashes geschlossen.
|
||||||
|
2025-11-12 04:04:12,393 - INFO - Finished source: https://www.gesetze-im-internet.de/ustg_1980/BJNR119530979.html
|
||||||
|
2025-11-12 04:04:12,398 - INFO - Processing source: https://www.gesetze-im-internet.de/uwg_2004/BJNR141400004.html
|
||||||
|
2025-11-12 04:04:12,406 - INFO - Starte Quelle: https://www.gesetze-im-internet.de/uwg_2004/BJNR141400004.html (Modus: single_page)
|
||||||
|
2025-11-12 04:04:12,410 - INFO - Crawle: https://www.gesetze-im-internet.de/uwg_2004/BJNR141400004.html
|
||||||
|
2025-11-12 04:04:12,460 - ERROR - Allgemeiner Fehler beim Crawlen von https://www.gesetze-im-internet.de/uwg_2004/BJNR141400004.html: 'utf-8' codec can't decode byte 0xfc in position 970: invalid start byte
|
||||||
|
2025-11-12 04:04:14,464 - INFO - Quelle https://www.gesetze-im-internet.de/uwg_2004/BJNR141400004.html erfolgreich gecrawlt.
|
||||||
|
2025-11-12 04:04:14,473 - INFO - Lösche Zustandsdatei.
|
||||||
|
2025-11-12 04:04:14,479 - INFO - Datenbankverbindung für Hashes geschlossen.
|
||||||
|
2025-11-12 04:04:14,487 - INFO - Finished source: https://www.gesetze-im-internet.de/uwg_2004/BJNR141400004.html
|
||||||
|
2025-11-12 04:04:14,491 - INFO - Processing source: https://www.gesetze-im-internet.de/hgb/BJNR002190897.html
|
||||||
|
2025-11-12 04:04:14,497 - INFO - Starte Quelle: https://www.gesetze-im-internet.de/hgb/BJNR002190897.html (Modus: single_page)
|
||||||
|
2025-11-12 04:04:14,501 - INFO - Crawle: https://www.gesetze-im-internet.de/hgb/BJNR002190897.html
|
||||||
|
2025-11-12 04:04:14,709 - ERROR - Allgemeiner Fehler beim Crawlen von https://www.gesetze-im-internet.de/hgb/BJNR002190897.html: 'utf-8' codec can't decode byte 0xfc in position 949: invalid start byte
|
||||||
|
2025-11-12 04:04:16,724 - INFO - Quelle https://www.gesetze-im-internet.de/hgb/BJNR002190897.html erfolgreich gecrawlt.
|
||||||
|
2025-11-12 04:04:16,729 - INFO - Lösche Zustandsdatei.
|
||||||
|
2025-11-12 04:04:16,739 - INFO - Datenbankverbindung für Hashes geschlossen.
|
||||||
|
2025-11-12 04:04:16,747 - INFO - Finished source: https://www.gesetze-im-internet.de/hgb/BJNR002190897.html
|
||||||
|
2025-11-12 04:04:16,753 - INFO - Processing source: https://www.gesetze-im-internet.de/bgb/BJNR001950896.html
|
||||||
|
2025-11-12 04:04:16,762 - INFO - Starte Quelle: https://www.gesetze-im-internet.de/bgb/BJNR001950896.html (Modus: single_page)
|
||||||
|
2025-11-12 04:04:16,768 - INFO - Crawle: https://www.gesetze-im-internet.de/bgb/BJNR001950896.html
|
||||||
|
2025-11-12 04:04:17,342 - ERROR - Allgemeiner Fehler beim Crawlen von https://www.gesetze-im-internet.de/bgb/BJNR001950896.html: 'utf-8' codec can't decode byte 0xfc in position 276: invalid start byte
|
||||||
|
2025-11-12 04:04:19,347 - INFO - Quelle https://www.gesetze-im-internet.de/bgb/BJNR001950896.html erfolgreich gecrawlt.
|
||||||
|
2025-11-12 04:04:19,352 - INFO - Lösche Zustandsdatei.
|
||||||
|
2025-11-12 04:04:19,358 - INFO - Datenbankverbindung für Hashes geschlossen.
|
||||||
|
2025-11-12 04:04:19,366 - INFO - Finished source: https://www.gesetze-im-internet.de/bgb/BJNR001950896.html
|
||||||
|
2025-11-12 04:04:19,379 - INFO - Processing source: https://www.gesetze-im-internet.de/fahrschausbo_2012/BJNR131800012.html
|
||||||
|
2025-11-12 04:04:19,385 - INFO - Starte Quelle: https://www.gesetze-im-internet.de/fahrschausbo_2012/BJNR131800012.html (Modus: single_page)
|
||||||
|
2025-11-12 04:04:19,393 - INFO - Crawle: https://www.gesetze-im-internet.de/fahrschausbo_2012/BJNR131800012.html
|
||||||
|
2025-11-12 04:04:19,449 - ERROR - Allgemeiner Fehler beim Crawlen von https://www.gesetze-im-internet.de/fahrschausbo_2012/BJNR131800012.html: 'utf-8' codec can't decode byte 0xfc in position 296: invalid start byte
|
||||||
|
2025-11-12 04:04:21,457 - INFO - Quelle https://www.gesetze-im-internet.de/fahrschausbo_2012/BJNR131800012.html erfolgreich gecrawlt.
|
||||||
|
2025-11-12 04:04:21,463 - INFO - Lösche Zustandsdatei.
|
||||||
|
2025-11-12 04:04:21,470 - INFO - Datenbankverbindung für Hashes geschlossen.
|
||||||
|
2025-11-12 04:04:21,473 - INFO - Finished source: https://www.gesetze-im-internet.de/fahrschausbo_2012/BJNR131800012.html
|
||||||
|
2025-11-12 04:04:21,476 - INFO - Processing source: https://www.gesetze-im-internet.de/fahrlpr_fv/BJNR004200018.html
|
||||||
|
2025-11-12 04:04:21,481 - INFO - Starte Quelle: https://www.gesetze-im-internet.de/fahrlpr_fv/BJNR004200018.html (Modus: single_page)
|
||||||
|
2025-11-12 04:04:21,484 - INFO - Crawle: https://www.gesetze-im-internet.de/fahrlpr_fv/BJNR004200018.html
|
||||||
|
2025-11-12 04:04:21,509 - ERROR - Allgemeiner Fehler beim Crawlen von https://www.gesetze-im-internet.de/fahrlpr_fv/BJNR004200018.html: 'utf-8' codec can't decode byte 0xfc in position 276: invalid start byte
|
||||||
|
2025-11-12 04:04:23,514 - INFO - Quelle https://www.gesetze-im-internet.de/fahrlpr_fv/BJNR004200018.html erfolgreich gecrawlt.
|
||||||
|
2025-11-12 04:04:23,520 - INFO - Lösche Zustandsdatei.
|
||||||
|
2025-11-12 04:04:23,523 - INFO - Datenbankverbindung für Hashes geschlossen.
|
||||||
|
2025-11-12 04:04:23,528 - INFO - Finished source: https://www.gesetze-im-internet.de/fahrlpr_fv/BJNR004200018.html
|
||||||
|
2025-11-12 04:04:23,535 - INFO - Processing source: https://www.gesetze-im-internet.de/fahrlg_2018/BJNR216210017.html
|
||||||
|
2025-11-12 04:04:23,540 - INFO - Starte Quelle: https://www.gesetze-im-internet.de/fahrlg_2018/BJNR216210017.html (Modus: single_page)
|
||||||
|
2025-11-12 04:04:23,544 - INFO - Crawle: https://www.gesetze-im-internet.de/fahrlg_2018/BJNR216210017.html
|
||||||
|
2025-11-12 04:04:23,615 - ERROR - Allgemeiner Fehler beim Crawlen von https://www.gesetze-im-internet.de/fahrlg_2018/BJNR216210017.html: 'utf-8' codec can't decode byte 0xfc in position 285: invalid start byte
|
||||||
|
2025-11-12 04:04:25,620 - INFO - Quelle https://www.gesetze-im-internet.de/fahrlg_2018/BJNR216210017.html erfolgreich gecrawlt.
|
||||||
|
2025-11-12 04:04:25,627 - INFO - Lösche Zustandsdatei.
|
||||||
|
2025-11-12 04:04:25,633 - INFO - Datenbankverbindung für Hashes geschlossen.
|
||||||
|
2025-11-12 04:04:25,640 - INFO - Finished source: https://www.gesetze-im-internet.de/fahrlg_2018/BJNR216210017.html
|
||||||
|
2025-11-12 04:04:25,644 - INFO - Processing source: https://www.gesetze-im-internet.de/fahrlausbv/BJNR001500018.html
|
||||||
|
2025-11-12 04:04:25,651 - INFO - Starte Quelle: https://www.gesetze-im-internet.de/fahrlausbv/BJNR001500018.html (Modus: single_page)
|
||||||
|
2025-11-12 04:04:25,656 - INFO - Crawle: https://www.gesetze-im-internet.de/fahrlausbv/BJNR001500018.html
|
||||||
|
2025-11-12 04:04:25,727 - ERROR - Allgemeiner Fehler beim Crawlen von https://www.gesetze-im-internet.de/fahrlausbv/BJNR001500018.html: 'utf-8' codec can't decode byte 0xfc in position 971: invalid start byte
|
||||||
|
2025-11-12 04:04:27,732 - INFO - Quelle https://www.gesetze-im-internet.de/fahrlausbv/BJNR001500018.html erfolgreich gecrawlt.
|
||||||
|
2025-11-12 04:04:27,738 - INFO - Lösche Zustandsdatei.
|
||||||
|
2025-11-12 04:04:27,744 - INFO - Datenbankverbindung für Hashes geschlossen.
|
||||||
|
2025-11-12 04:04:27,748 - INFO - Finished source: https://www.gesetze-im-internet.de/fahrlausbv/BJNR001500018.html
|
||||||
|
2025-11-12 04:04:27,754 - INFO - Processing source: https://www.gesetze-im-internet.de/pbefg/BJNR002410961.html
|
||||||
|
2025-11-12 04:04:27,757 - INFO - Starte Quelle: https://www.gesetze-im-internet.de/pbefg/BJNR002410961.html (Modus: single_page)
|
||||||
|
2025-11-12 04:04:27,761 - INFO - Crawle: https://www.gesetze-im-internet.de/pbefg/BJNR002410961.html
|
||||||
|
2025-11-12 04:04:27,841 - ERROR - Allgemeiner Fehler beim Crawlen von https://www.gesetze-im-internet.de/pbefg/BJNR002410961.html: 'utf-8' codec can't decode byte 0xf6 in position 288: invalid start byte
|
||||||
|
2025-11-12 04:04:29,849 - INFO - Quelle https://www.gesetze-im-internet.de/pbefg/BJNR002410961.html erfolgreich gecrawlt.
|
||||||
|
2025-11-12 04:04:29,853 - INFO - Lösche Zustandsdatei.
|
||||||
|
2025-11-12 04:04:29,857 - INFO - Datenbankverbindung für Hashes geschlossen.
|
||||||
|
2025-11-12 04:04:29,861 - INFO - Finished source: https://www.gesetze-im-internet.de/pbefg/BJNR002410961.html
|
||||||
|
2025-11-12 04:04:29,865 - INFO - Processing source: https://www.gesetze-im-internet.de/g_kg_1998/BJNR148510998.html
|
||||||
|
2025-11-12 04:04:29,870 - INFO - Starte Quelle: https://www.gesetze-im-internet.de/g_kg_1998/BJNR148510998.html (Modus: single_page)
|
||||||
|
2025-11-12 04:04:29,873 - INFO - Crawle: https://www.gesetze-im-internet.de/g_kg_1998/BJNR148510998.html
|
||||||
|
2025-11-12 04:04:29,920 - ERROR - Allgemeiner Fehler beim Crawlen von https://www.gesetze-im-internet.de/g_kg_1998/BJNR148510998.html: 'utf-8' codec can't decode byte 0xfc in position 270: invalid start byte
|
||||||
|
2025-11-12 04:04:31,926 - INFO - Quelle https://www.gesetze-im-internet.de/g_kg_1998/BJNR148510998.html erfolgreich gecrawlt.
|
||||||
|
2025-11-12 04:04:31,932 - INFO - Lösche Zustandsdatei.
|
||||||
|
2025-11-12 04:04:31,937 - INFO - Datenbankverbindung für Hashes geschlossen.
|
||||||
|
2025-11-12 04:04:31,940 - INFO - Finished source: https://www.gesetze-im-internet.de/g_kg_1998/BJNR148510998.html
|
||||||
|
2025-11-12 04:04:31,943 - INFO - Processing source: https://www.gesetze-im-internet.de/fev_2010/BJNR198000010.html
|
||||||
|
2025-11-12 04:04:31,948 - INFO - Starte Quelle: https://www.gesetze-im-internet.de/fev_2010/BJNR198000010.html (Modus: single_page)
|
||||||
|
2025-11-12 04:04:31,952 - INFO - Crawle: https://www.gesetze-im-internet.de/fev_2010/BJNR198000010.html
|
||||||
|
2025-11-12 04:04:32,057 - ERROR - Allgemeiner Fehler beim Crawlen von https://www.gesetze-im-internet.de/fev_2010/BJNR198000010.html: 'utf-8' codec can't decode byte 0xfc in position 286: invalid start byte
|
||||||
|
2025-11-12 04:04:34,062 - INFO - Quelle https://www.gesetze-im-internet.de/fev_2010/BJNR198000010.html erfolgreich gecrawlt.
|
||||||
|
2025-11-12 04:04:34,069 - INFO - Lösche Zustandsdatei.
|
||||||
|
2025-11-12 04:04:34,077 - INFO - Datenbankverbindung für Hashes geschlossen.
|
||||||
|
2025-11-12 04:04:34,084 - INFO - Finished source: https://www.gesetze-im-internet.de/fev_2010/BJNR198000010.html
|
||||||
|
2025-11-12 04:04:34,091 - INFO - Processing source: https://www.gesetze-im-internet.de/stvzo_2012/BJNR067910012.html
|
||||||
|
2025-11-12 04:04:34,103 - INFO - Starte Quelle: https://www.gesetze-im-internet.de/stvzo_2012/BJNR067910012.html (Modus: single_page)
|
||||||
|
2025-11-12 04:04:34,110 - INFO - Crawle: https://www.gesetze-im-internet.de/stvzo_2012/BJNR067910012.html
|
||||||
|
2025-11-12 04:04:34,415 - ERROR - Allgemeiner Fehler beim Crawlen von https://www.gesetze-im-internet.de/stvzo_2012/BJNR067910012.html: 'utf-8' codec can't decode byte 0xdf in position 281: invalid continuation byte
|
||||||
|
2025-11-12 04:04:36,421 - INFO - Quelle https://www.gesetze-im-internet.de/stvzo_2012/BJNR067910012.html erfolgreich gecrawlt.
|
||||||
|
2025-11-12 04:04:36,425 - INFO - Lösche Zustandsdatei.
|
||||||
|
2025-11-12 04:04:36,428 - INFO - Datenbankverbindung für Hashes geschlossen.
|
||||||
|
2025-11-12 04:04:36,432 - INFO - Finished source: https://www.gesetze-im-internet.de/stvzo_2012/BJNR067910012.html
|
||||||
|
2025-11-12 04:04:36,435 - INFO - Processing source: https://www.gesetze-im-internet.de/stvg/BJNR004370909.html
|
||||||
|
2025-11-12 04:04:36,439 - INFO - Starte Quelle: https://www.gesetze-im-internet.de/stvg/BJNR004370909.html (Modus: single_page)
|
||||||
|
2025-11-12 04:04:36,441 - INFO - Crawle: https://www.gesetze-im-internet.de/stvg/BJNR004370909.html
|
||||||
|
2025-11-12 04:04:36,549 - ERROR - Allgemeiner Fehler beim Crawlen von https://www.gesetze-im-internet.de/stvg/BJNR004370909.html: 'utf-8' codec can't decode byte 0xdf in position 280: invalid continuation byte
|
||||||
|
2025-11-12 04:04:38,553 - INFO - Quelle https://www.gesetze-im-internet.de/stvg/BJNR004370909.html erfolgreich gecrawlt.
|
||||||
|
2025-11-12 04:04:38,557 - INFO - Lösche Zustandsdatei.
|
||||||
|
2025-11-12 04:04:38,560 - INFO - Datenbankverbindung für Hashes geschlossen.
|
||||||
|
2025-11-12 04:04:38,565 - INFO - Finished source: https://www.gesetze-im-internet.de/stvg/BJNR004370909.html
|
||||||
|
2025-11-12 04:04:38,569 - INFO - Processing source: https://www.gesetze-im-internet.de/stvoausnv_8/BJNR113000998.html
|
||||||
|
2025-11-12 04:04:38,572 - INFO - Starte Quelle: https://www.gesetze-im-internet.de/stvoausnv_8/BJNR113000998.html (Modus: single_page)
|
||||||
|
2025-11-12 04:04:38,574 - INFO - Crawle: https://www.gesetze-im-internet.de/stvoausnv_8/BJNR113000998.html
|
||||||
|
2025-11-12 04:04:38,602 - ERROR - Allgemeiner Fehler beim Crawlen von https://www.gesetze-im-internet.de/stvoausnv_8/BJNR113000998.html: 'utf-8' codec can't decode byte 0xfc in position 300: invalid start byte
|
||||||
|
2025-11-12 04:04:40,607 - INFO - Quelle https://www.gesetze-im-internet.de/stvoausnv_8/BJNR113000998.html erfolgreich gecrawlt.
|
||||||
|
2025-11-12 04:04:40,611 - INFO - Lösche Zustandsdatei.
|
||||||
|
2025-11-12 04:04:40,615 - INFO - Datenbankverbindung für Hashes geschlossen.
|
||||||
|
2025-11-12 04:04:40,618 - INFO - Finished source: https://www.gesetze-im-internet.de/stvoausnv_8/BJNR113000998.html
|
||||||
|
2025-11-12 04:04:40,622 - INFO - Processing source: https://www.gesetze-im-internet.de/stvoausnv_5/BJNR062300994.html
|
||||||
|
2025-11-12 04:04:40,625 - INFO - Starte Quelle: https://www.gesetze-im-internet.de/stvoausnv_5/BJNR062300994.html (Modus: single_page)
|
||||||
|
2025-11-12 04:04:40,628 - INFO - Crawle: https://www.gesetze-im-internet.de/stvoausnv_5/BJNR062300994.html
|
||||||
|
2025-11-12 04:04:40,655 - ERROR - Allgemeiner Fehler beim Crawlen von https://www.gesetze-im-internet.de/stvoausnv_5/BJNR062300994.html: 'utf-8' codec can't decode byte 0xfc in position 284: invalid start byte
|
||||||
|
2025-11-12 04:04:42,659 - INFO - Quelle https://www.gesetze-im-internet.de/stvoausnv_5/BJNR062300994.html erfolgreich gecrawlt.
|
||||||
|
2025-11-12 04:04:42,664 - INFO - Lösche Zustandsdatei.
|
||||||
|
2025-11-12 04:04:42,668 - INFO - Datenbankverbindung für Hashes geschlossen.
|
||||||
|
2025-11-12 04:04:42,671 - INFO - Finished source: https://www.gesetze-im-internet.de/stvoausnv_5/BJNR062300994.html
|
||||||
|
2025-11-12 04:04:42,673 - INFO - Processing source: https://www.gesetze-im-internet.de/stvoausnv_9/BJNR317100998.html
|
||||||
|
2025-11-12 04:04:42,676 - INFO - Starte Quelle: https://www.gesetze-im-internet.de/stvoausnv_9/BJNR317100998.html (Modus: single_page)
|
||||||
|
2025-11-12 04:04:42,679 - INFO - Crawle: https://www.gesetze-im-internet.de/stvoausnv_9/BJNR317100998.html
|
||||||
|
2025-11-12 04:04:42,706 - ERROR - Allgemeiner Fehler beim Crawlen von https://www.gesetze-im-internet.de/stvoausnv_9/BJNR317100998.html: 'utf-8' codec can't decode byte 0xfc in position 301: invalid start byte
|
||||||
|
2025-11-12 04:04:44,711 - INFO - Quelle https://www.gesetze-im-internet.de/stvoausnv_9/BJNR317100998.html erfolgreich gecrawlt.
|
||||||
|
2025-11-12 04:04:44,713 - INFO - Lösche Zustandsdatei.
|
||||||
|
2025-11-12 04:04:44,715 - INFO - Datenbankverbindung für Hashes geschlossen.
|
||||||
|
2025-11-12 04:04:44,717 - INFO - Finished source: https://www.gesetze-im-internet.de/stvoausnv_9/BJNR317100998.html
|
||||||
|
2025-11-12 04:04:44,719 - INFO - Processing source: https://www.gesetze-im-internet.de/stvoausnv_12/BJNR086600005.html
|
||||||
|
2025-11-12 04:04:44,721 - INFO - Starte Quelle: https://www.gesetze-im-internet.de/stvoausnv_12/BJNR086600005.html (Modus: single_page)
|
||||||
|
2025-11-12 04:04:44,723 - INFO - Crawle: https://www.gesetze-im-internet.de/stvoausnv_12/BJNR086600005.html
|
||||||
|
2025-11-12 04:04:44,747 - ERROR - Allgemeiner Fehler beim Crawlen von https://www.gesetze-im-internet.de/stvoausnv_12/BJNR086600005.html: 'utf-8' codec can't decode byte 0xf6 in position 286: invalid start byte
|
||||||
|
2025-11-12 04:04:46,751 - INFO - Quelle https://www.gesetze-im-internet.de/stvoausnv_12/BJNR086600005.html erfolgreich gecrawlt.
|
||||||
|
2025-11-12 04:04:46,753 - INFO - Lösche Zustandsdatei.
|
||||||
|
2025-11-12 04:04:46,755 - INFO - Datenbankverbindung für Hashes geschlossen.
|
||||||
|
2025-11-12 04:04:46,757 - INFO - Finished source: https://www.gesetze-im-internet.de/stvoausnv_12/BJNR086600005.html
|
||||||
|
2025-11-12 04:04:46,758 - INFO - Processing source: https://www.gesetze-im-internet.de/stvoausnv_4/BJNR011240992.html
|
||||||
|
2025-11-12 04:04:46,761 - INFO - Starte Quelle: https://www.gesetze-im-internet.de/stvoausnv_4/BJNR011240992.html (Modus: single_page)
|
||||||
|
2025-11-12 04:04:46,763 - INFO - Crawle: https://www.gesetze-im-internet.de/stvoausnv_4/BJNR011240992.html
|
||||||
|
2025-11-12 04:04:46,789 - ERROR - Allgemeiner Fehler beim Crawlen von https://www.gesetze-im-internet.de/stvoausnv_4/BJNR011240992.html: 'utf-8' codec can't decode byte 0xfc in position 301: invalid start byte
|
||||||
|
2025-11-12 04:04:48,796 - INFO - Quelle https://www.gesetze-im-internet.de/stvoausnv_4/BJNR011240992.html erfolgreich gecrawlt.
|
||||||
|
2025-11-12 04:04:48,817 - INFO - Lösche Zustandsdatei.
|
||||||
|
2025-11-12 04:04:48,839 - INFO - Datenbankverbindung für Hashes geschlossen.
|
||||||
|
2025-11-12 04:04:48,850 - INFO - Finished source: https://www.gesetze-im-internet.de/stvoausnv_4/BJNR011240992.html
|
||||||
|
2025-11-12 04:04:48,868 - INFO - Processing source: https://www.gesetze-im-internet.de/stvo_2013/BJNR036710013.html
|
||||||
|
2025-11-12 04:04:48,878 - INFO - Starte Quelle: https://www.gesetze-im-internet.de/stvo_2013/BJNR036710013.html (Modus: single_page)
|
||||||
|
2025-11-12 04:04:48,891 - INFO - Crawle: https://www.gesetze-im-internet.de/stvo_2013/BJNR036710013.html
|
||||||
|
2025-11-12 04:04:49,014 - ERROR - Allgemeiner Fehler beim Crawlen von https://www.gesetze-im-internet.de/stvo_2013/BJNR036710013.html: 'utf-8' codec can't decode byte 0xdf in position 280: invalid continuation byte
|
||||||
|
2025-11-12 04:04:51,026 - INFO - Quelle https://www.gesetze-im-internet.de/stvo_2013/BJNR036710013.html erfolgreich gecrawlt.
|
||||||
|
2025-11-12 04:04:51,035 - INFO - Lösche Zustandsdatei.
|
||||||
|
2025-11-12 04:04:51,042 - INFO - Datenbankverbindung für Hashes geschlossen.
|
||||||
|
2025-11-12 04:04:51,045 - INFO - Finished source: https://www.gesetze-im-internet.de/stvo_2013/BJNR036710013.html
|
||||||
|
2025-11-12 04:04:51,056 - INFO - Processing source: https://eur-lex.europa.eu/legal-content/DE/TXT/HTML/?uri=CELEX:32016R0679
|
||||||
|
2025-11-12 04:04:51,065 - INFO - Starte Quelle: https://eur-lex.europa.eu/legal-content/DE/TXT/HTML/?uri=CELEX:32016R0679 (Modus: single_page)
|
||||||
|
2025-11-12 04:04:51,071 - INFO - Crawle: https://eur-lex.europa.eu/legal-content/DE/TXT/HTML/?uri=CELEX%3A32016R0679
|
||||||
|
2025-11-12 04:04:51,075 - INFO - Lade robots.txt von: https://eur-lex.europa.eu/robots.txt
|
||||||
|
2025-11-12 04:04:51,208 - WARNING - Fehler beim Abruf von https://eur-lex.europa.eu/robots.txt: Status 202
|
||||||
|
2025-11-12 04:04:51,210 - WARNING - robots.txt verbietet: https://eur-lex.europa.eu/legal-content/DE/TXT/HTML/?uri=CELEX%3A32016R0679
|
||||||
|
2025-11-12 04:04:53,216 - INFO - Quelle https://eur-lex.europa.eu/legal-content/DE/TXT/HTML/?uri=CELEX%3A32016R0679 erfolgreich gecrawlt.
|
||||||
|
2025-11-12 04:04:53,222 - INFO - Lösche Zustandsdatei.
|
||||||
|
2025-11-12 04:04:53,225 - INFO - Datenbankverbindung für Hashes geschlossen.
|
||||||
|
2025-11-12 04:04:53,228 - INFO - Finished source: https://eur-lex.europa.eu/legal-content/DE/TXT/HTML/?uri=CELEX:32016R0679
|
||||||
|
2025-11-12 04:04:53,230 - INFO - Processing source: https://www.gnu.org/software/bash/manual/bash.html
|
||||||
|
2025-11-12 04:04:53,233 - INFO - Starte Quelle: https://www.gnu.org/software/bash/manual/bash.html (Modus: single_page)
|
||||||
|
2025-11-12 04:04:53,235 - INFO - Crawle: https://www.gnu.org/software/bash/manual/bash.html
|
||||||
|
2025-11-12 04:04:53,238 - INFO - Lade robots.txt von: https://www.gnu.org/robots.txt
|
||||||
|
2025-11-12 04:04:58,062 - ERROR - Allgemeiner Fehler beim Crawlen von https://www.gnu.org/software/bash/manual/bash.html: [Errno 13] Permission denied: '/app'
|
||||||
|
2025-11-12 04:05:00,067 - INFO - Quelle https://www.gnu.org/software/bash/manual/bash.html erfolgreich gecrawlt.
|
||||||
|
2025-11-12 04:05:00,073 - INFO - Lösche Zustandsdatei.
|
||||||
|
2025-11-12 04:05:00,076 - INFO - Datenbankverbindung für Hashes geschlossen.
|
||||||
|
2025-11-12 04:05:00,079 - INFO - Finished source: https://www.gnu.org/software/bash/manual/bash.html
|
||||||
|
2025-11-12 04:05:00,083 - INFO - Processing source: https://www.json.org/json-en.html
|
||||||
|
2025-11-12 04:05:00,086 - INFO - Starte Quelle: https://www.json.org/json-en.html (Modus: single_page)
|
||||||
|
2025-11-12 04:05:00,089 - INFO - Crawle: https://www.json.org/json-en.html
|
||||||
|
2025-11-12 04:05:00,091 - INFO - Lade robots.txt von: https://www.json.org/robots.txt
|
||||||
|
2025-11-12 04:05:00,632 - WARNING - robots.txt verbietet: https://www.json.org/json-en.html
|
||||||
|
2025-11-12 04:05:02,637 - INFO - Quelle https://www.json.org/json-en.html erfolgreich gecrawlt.
|
||||||
|
2025-11-12 04:05:02,641 - INFO - Lösche Zustandsdatei.
|
||||||
|
2025-11-12 04:05:02,643 - INFO - Datenbankverbindung für Hashes geschlossen.
|
||||||
|
2025-11-12 04:05:02,646 - INFO - Finished source: https://www.json.org/json-en.html
|
||||||
|
2025-11-12 04:05:02,649 - INFO - Processing source: https://www.bussgeldkatalog.org/
|
||||||
|
2025-11-12 04:05:02,652 - INFO - Starte Quelle: https://www.bussgeldkatalog.org/ (Modus: domain_wide)
|
||||||
|
2025-11-12 04:05:02,655 - INFO - Crawle: https://www.bussgeldkatalog.org/
|
||||||
|
2025-11-12 04:05:02,657 - INFO - Lade robots.txt von: https://www.bussgeldkatalog.org/robots.txt
|
||||||
|
2025-11-12 04:05:03,014 - ERROR - Allgemeiner Fehler beim Crawlen von https://www.bussgeldkatalog.org/: [Errno 13] Permission denied: '/app'
|
||||||
|
2025-11-12 04:05:05,017 - INFO - Quelle https://www.bussgeldkatalog.org/ erfolgreich gecrawlt.
|
||||||
|
2025-11-12 04:05:05,022 - INFO - Lösche Zustandsdatei.
|
||||||
|
2025-11-12 04:05:05,025 - INFO - Datenbankverbindung für Hashes geschlossen.
|
||||||
|
2025-11-12 04:05:05,027 - INFO - Finished source: https://www.bussgeldkatalog.org/
|
||||||
|
2025-11-12 04:05:05,030 - INFO - Processing source: https://docs.budibase.com/
|
||||||
|
2025-11-12 04:05:05,032 - INFO - Starte Quelle: https://docs.budibase.com/ (Modus: domain_wide)
|
||||||
|
2025-11-12 04:05:05,034 - INFO - Crawle: https://docs.budibase.com/
|
||||||
|
2025-11-12 04:05:05,037 - INFO - Lade robots.txt von: https://docs.budibase.com/robots.txt
|
||||||
|
2025-11-12 04:05:06,423 - ERROR - Allgemeiner Fehler beim Crawlen von https://docs.budibase.com/: [Errno 13] Permission denied: '/app'
|
||||||
|
2025-11-12 04:05:08,427 - INFO - Quelle https://docs.budibase.com/ erfolgreich gecrawlt.
|
||||||
|
2025-11-12 04:05:08,431 - INFO - Lösche Zustandsdatei.
|
||||||
|
2025-11-12 04:05:08,435 - INFO - Datenbankverbindung für Hashes geschlossen.
|
||||||
|
2025-11-12 04:05:08,438 - INFO - Finished source: https://docs.budibase.com/
|
||||||
|
2025-11-12 04:05:08,441 - INFO - Processing source: https://shadowhelix.de/
|
||||||
|
2025-11-12 04:05:08,444 - INFO - Starte Quelle: https://shadowhelix.de/ (Modus: domain_wide)
|
||||||
|
2025-11-12 04:05:08,447 - INFO - Crawle: https://shadowhelix.de/
|
||||||
|
2025-11-12 04:05:08,449 - INFO - Lade robots.txt von: https://shadowhelix.de/robots.txt
|
||||||
|
2025-11-12 04:05:08,802 - WARNING - robots.txt verbietet: https://shadowhelix.de/
|
||||||
|
2025-11-12 04:05:10,806 - INFO - Quelle https://shadowhelix.de/ erfolgreich gecrawlt.
|
||||||
|
2025-11-12 04:05:10,811 - INFO - Lösche Zustandsdatei.
|
||||||
|
2025-11-12 04:05:10,814 - INFO - Datenbankverbindung für Hashes geschlossen.
|
||||||
|
2025-11-12 04:05:10,817 - INFO - Finished source: https://shadowhelix.de/
|
||||||
|
2025-11-12 04:05:10,819 - INFO - Processing source: https://markdown.de/
|
||||||
|
2025-11-12 04:05:10,823 - INFO - Starte Quelle: https://markdown.de/ (Modus: domain_wide)
|
||||||
|
2025-11-12 04:05:10,825 - INFO - Crawle: https://markdown.de/
|
||||||
|
2025-11-12 04:05:10,827 - INFO - Lade robots.txt von: https://markdown.de/robots.txt
|
||||||
|
2025-11-12 04:05:10,973 - WARNING - robots.txt verbietet: https://markdown.de/
|
||||||
|
2025-11-12 04:05:12,976 - INFO - Quelle https://markdown.de/ erfolgreich gecrawlt.
|
||||||
|
2025-11-12 04:05:12,981 - INFO - Lösche Zustandsdatei.
|
||||||
|
2025-11-12 04:05:12,986 - INFO - Datenbankverbindung für Hashes geschlossen.
|
||||||
|
2025-11-12 04:05:12,988 - INFO - Finished source: https://markdown.de/
|
||||||
|
2025-11-12 04:05:12,991 - INFO - Processing source: https://mariadb.com/docs/
|
||||||
|
2025-11-12 04:05:12,994 - INFO - Starte Quelle: https://mariadb.com/docs/ (Modus: domain_wide)
|
||||||
|
2025-11-12 04:05:12,997 - INFO - Crawle: https://mariadb.com/docs/
|
||||||
|
2025-11-12 04:05:13,000 - INFO - Lade robots.txt von: https://mariadb.com/robots.txt
|
||||||
|
2025-11-12 04:05:13,884 - ERROR - Allgemeiner Fehler beim Crawlen von https://mariadb.com/docs/: [Errno 13] Permission denied: '/app'
|
||||||
|
2025-11-12 04:05:15,889 - INFO - Quelle https://mariadb.com/docs/ erfolgreich gecrawlt.
|
||||||
|
2025-11-12 04:05:15,893 - INFO - Lösche Zustandsdatei.
|
||||||
|
2025-11-12 04:05:15,895 - INFO - Datenbankverbindung für Hashes geschlossen.
|
||||||
|
2025-11-12 04:05:15,898 - INFO - Finished source: https://mariadb.com/docs/
|
||||||
|
2025-11-12 04:05:15,901 - INFO - Processing source: https://html.spec.whatwg.org/
|
||||||
|
2025-11-12 04:05:15,904 - INFO - Starte Quelle: https://html.spec.whatwg.org/ (Modus: domain_wide)
|
||||||
|
2025-11-12 04:05:15,906 - INFO - Crawle: https://html.spec.whatwg.org/
|
||||||
|
2025-11-12 04:05:15,908 - INFO - Lade robots.txt von: https://html.spec.whatwg.org/robots.txt
|
||||||
|
2025-11-12 04:07:39,943 - ERROR - Allgemeiner Fehler beim Crawlen von https://html.spec.whatwg.org/: [Errno 13] Permission denied: '/app'
|
||||||
|
2025-11-12 04:07:41,948 - INFO - Quelle https://html.spec.whatwg.org/ erfolgreich gecrawlt.
|
||||||
|
2025-11-12 04:07:41,954 - INFO - Lösche Zustandsdatei.
|
||||||
|
2025-11-12 04:07:41,956 - INFO - Datenbankverbindung für Hashes geschlossen.
|
||||||
|
2025-11-12 04:07:41,971 - INFO - Finished source: https://html.spec.whatwg.org/
|
||||||
|
2025-11-12 04:07:41,973 - INFO - Processing source: https://restfulapi.net/
|
||||||
|
2025-11-12 04:07:41,975 - INFO - Starte Quelle: https://restfulapi.net/ (Modus: domain_wide)
|
||||||
|
2025-11-12 04:07:41,977 - INFO - Crawle: https://restfulapi.net/
|
||||||
|
2025-11-12 04:07:41,979 - INFO - Lade robots.txt von: https://restfulapi.net/robots.txt
|
||||||
|
2025-11-12 04:07:42,652 - ERROR - Allgemeiner Fehler beim Crawlen von https://restfulapi.net/: [Errno 13] Permission denied: '/app'
|
||||||
|
2025-11-12 04:07:44,656 - INFO - Quelle https://restfulapi.net/ erfolgreich gecrawlt.
|
||||||
|
2025-11-12 04:07:44,661 - INFO - Lösche Zustandsdatei.
|
||||||
|
2025-11-12 04:07:44,667 - INFO - Datenbankverbindung für Hashes geschlossen.
|
||||||
|
2025-11-12 04:07:44,670 - INFO - Finished source: https://restfulapi.net/
|
||||||
|
2025-11-12 04:07:44,672 - INFO - Processing source: https://react.dev/
|
||||||
|
2025-11-12 04:07:44,676 - INFO - Starte Quelle: https://react.dev/ (Modus: domain_wide)
|
||||||
|
2025-11-12 04:07:44,679 - INFO - Crawle: https://react.dev/
|
||||||
|
2025-11-12 04:07:44,682 - INFO - Lade robots.txt von: https://react.dev/robots.txt
|
||||||
|
2025-11-12 04:07:45,182 - ERROR - Allgemeiner Fehler beim Crawlen von https://react.dev/: [Errno 13] Permission denied: '/app'
|
||||||
|
2025-11-12 04:07:47,186 - INFO - Quelle https://react.dev/ erfolgreich gecrawlt.
|
||||||
|
2025-11-12 04:07:47,190 - INFO - Lösche Zustandsdatei.
|
||||||
|
2025-11-12 04:07:47,193 - INFO - Datenbankverbindung für Hashes geschlossen.
|
||||||
|
2025-11-12 04:07:47,196 - INFO - Finished source: https://react.dev/
|
||||||
|
2025-11-12 04:07:47,199 - INFO - Processing source: https://expressjs.com/
|
||||||
|
2025-11-12 04:07:47,202 - INFO - Starte Quelle: https://expressjs.com/ (Modus: domain_wide)
|
||||||
|
2025-11-12 04:07:47,204 - INFO - Crawle: https://expressjs.com/
|
||||||
|
2025-11-12 04:07:47,206 - INFO - Lade robots.txt von: https://expressjs.com/robots.txt
|
||||||
|
2025-11-12 04:07:47,452 - WARNING - robots.txt verbietet: https://expressjs.com/
|
||||||
|
2025-11-12 04:07:49,456 - INFO - Quelle https://expressjs.com/ erfolgreich gecrawlt.
|
||||||
|
2025-11-12 04:07:49,460 - INFO - Lösche Zustandsdatei.
|
||||||
|
2025-11-12 04:07:49,464 - INFO - Datenbankverbindung für Hashes geschlossen.
|
||||||
|
2025-11-12 04:07:49,469 - INFO - Finished source: https://expressjs.com/
|
||||||
|
2025-11-12 04:07:49,472 - INFO - Processing source: https://docs.docker.com/
|
||||||
|
2025-11-12 04:07:49,475 - INFO - Starte Quelle: https://docs.docker.com/ (Modus: domain_wide)
|
||||||
|
2025-11-12 04:07:49,477 - INFO - Crawle: https://docs.docker.com/
|
||||||
|
2025-11-12 04:07:49,481 - INFO - Lade robots.txt von: https://docs.docker.com/robots.txt
|
||||||
|
2025-11-12 04:07:49,763 - ERROR - Allgemeiner Fehler beim Crawlen von https://docs.docker.com/: [Errno 13] Permission denied: '/app'
|
||||||
|
2025-11-12 04:07:51,767 - INFO - Quelle https://docs.docker.com/ erfolgreich gecrawlt.
|
||||||
|
2025-11-12 04:07:51,771 - INFO - Lösche Zustandsdatei.
|
||||||
|
2025-11-12 04:07:51,774 - INFO - Datenbankverbindung für Hashes geschlossen.
|
||||||
|
2025-11-12 04:07:51,776 - INFO - Finished source: https://docs.docker.com/
|
||||||
|
2025-11-12 04:07:51,779 - INFO - Processing source: https://docs.openwebui.com/
|
||||||
|
2025-11-12 04:07:51,782 - INFO - Starte Quelle: https://docs.openwebui.com/ (Modus: domain_wide)
|
||||||
|
2025-11-12 04:07:51,785 - INFO - Crawle: https://docs.openwebui.com/
|
||||||
|
2025-11-12 04:07:51,787 - INFO - Lade robots.txt von: https://docs.openwebui.com/robots.txt
|
||||||
|
2025-11-12 04:07:52,030 - WARNING - robots.txt verbietet: https://docs.openwebui.com/
|
||||||
|
2025-11-12 04:07:54,035 - INFO - Quelle https://docs.openwebui.com/ erfolgreich gecrawlt.
|
||||||
|
2025-11-12 04:07:54,044 - INFO - Lösche Zustandsdatei.
|
||||||
|
2025-11-12 04:07:54,047 - INFO - Datenbankverbindung für Hashes geschlossen.
|
||||||
|
2025-11-12 04:07:54,050 - INFO - Finished source: https://docs.openwebui.com/
|
||||||
|
2025-11-12 04:07:54,052 - INFO - Processing source: https://www.php.net/manual/en/
|
||||||
|
2025-11-12 04:07:54,055 - INFO - Starte Quelle: https://www.php.net/manual/en/ (Modus: domain_wide)
|
||||||
|
2025-11-12 04:07:54,057 - INFO - Crawle: https://www.php.net/manual/en/
|
||||||
|
2025-11-12 04:07:54,060 - INFO - Lade robots.txt von: https://www.php.net/robots.txt
|
||||||
|
2025-11-12 04:07:54,648 - WARNING - discarding data: None
|
||||||
|
2025-11-12 04:07:54,679 - ERROR - Allgemeiner Fehler beim Crawlen von https://www.php.net/manual/en/: [Errno 13] Permission denied: '/app'
|
||||||
|
2025-11-12 04:07:56,683 - INFO - Quelle https://www.php.net/manual/en/ erfolgreich gecrawlt.
|
||||||
|
2025-11-12 04:07:56,689 - INFO - Lösche Zustandsdatei.
|
||||||
|
2025-11-12 04:07:56,693 - INFO - Datenbankverbindung für Hashes geschlossen.
|
||||||
|
2025-11-12 04:07:56,696 - INFO - Finished source: https://www.php.net/manual/en/
|
||||||
|
2025-11-12 04:07:56,700 - INFO - Processing source: https://www.postgresql.org/docs/
|
||||||
|
2025-11-12 04:07:56,703 - INFO - Starte Quelle: https://www.postgresql.org/docs/ (Modus: domain_wide)
|
||||||
|
2025-11-12 04:07:56,706 - INFO - Crawle: https://www.postgresql.org/docs/
|
||||||
|
2025-11-12 04:07:56,709 - INFO - Lade robots.txt von: https://www.postgresql.org/robots.txt
|
||||||
|
2025-11-12 04:07:57,042 - ERROR - Allgemeiner Fehler beim Crawlen von https://www.postgresql.org/docs/: [Errno 13] Permission denied: '/app'
|
||||||
|
2025-11-12 04:07:59,047 - INFO - Quelle https://www.postgresql.org/docs/ erfolgreich gecrawlt.
|
||||||
|
2025-11-12 04:07:59,052 - INFO - Lösche Zustandsdatei.
|
||||||
|
2025-11-12 04:07:59,055 - INFO - Datenbankverbindung für Hashes geschlossen.
|
||||||
|
2025-11-12 04:07:59,058 - INFO - Finished source: https://www.postgresql.org/docs/
|
||||||
|
2025-11-12 04:07:59,060 - INFO - Processing source: https://nginx.org/en/docs/
|
||||||
|
2025-11-12 04:07:59,064 - INFO - Starte Quelle: https://nginx.org/en/docs/ (Modus: domain_wide)
|
||||||
|
2025-11-12 04:07:59,066 - INFO - Crawle: https://nginx.org/en/docs/
|
||||||
|
2025-11-12 04:07:59,069 - INFO - Lade robots.txt von: https://nginx.org/robots.txt
|
||||||
|
2025-11-12 04:07:59,282 - ERROR - Allgemeiner Fehler beim Crawlen von https://nginx.org/en/docs/: [Errno 13] Permission denied: '/app'
|
||||||
|
2025-11-12 04:08:01,287 - INFO - Quelle https://nginx.org/en/docs/ erfolgreich gecrawlt.
|
||||||
|
2025-11-12 04:08:01,291 - INFO - Lösche Zustandsdatei.
|
||||||
|
2025-11-12 04:08:01,294 - INFO - Datenbankverbindung für Hashes geschlossen.
|
||||||
|
2025-11-12 04:08:01,298 - INFO - Finished source: https://nginx.org/en/docs/
|
||||||
|
2025-11-12 04:08:01,301 - INFO - Processing source: https://docs.nginx.com/nginx/admin-guide/
|
||||||
|
2025-11-12 04:08:01,304 - INFO - Starte Quelle: https://docs.nginx.com/nginx/admin-guide/ (Modus: domain_wide)
|
||||||
|
2025-11-12 04:08:01,306 - INFO - Crawle: https://docs.nginx.com/nginx/admin-guide/
|
||||||
|
2025-11-12 04:08:01,309 - INFO - Lade robots.txt von: https://docs.nginx.com/robots.txt
|
||||||
|
2025-11-12 04:08:02,393 - ERROR - Allgemeiner Fehler beim Crawlen von https://docs.nginx.com/nginx/admin-guide/: [Errno 13] Permission denied: '/app'
|
||||||
|
2025-11-12 04:08:04,399 - INFO - Quelle https://docs.nginx.com/nginx/admin-guide/ erfolgreich gecrawlt.
|
||||||
|
2025-11-12 04:08:04,403 - INFO - Lösche Zustandsdatei.
|
||||||
|
2025-11-12 04:08:04,406 - INFO - Datenbankverbindung für Hashes geschlossen.
|
||||||
|
2025-11-12 04:08:04,409 - INFO - Finished source: https://docs.nginx.com/nginx/admin-guide/
|
||||||
|
2025-11-12 04:08:04,411 - INFO - Processing source: https://wordpress.org/documentation/
|
||||||
|
2025-11-12 04:08:04,415 - INFO - Starte Quelle: https://wordpress.org/documentation/ (Modus: domain_wide)
|
||||||
|
2025-11-12 04:08:04,418 - INFO - Crawle: https://wordpress.org/documentation/
|
||||||
|
2025-11-12 04:08:04,420 - INFO - Lade robots.txt von: https://wordpress.org/robots.txt
|
||||||
|
2025-11-12 04:08:05,248 - ERROR - Allgemeiner Fehler beim Crawlen von https://wordpress.org/documentation/: [Errno 13] Permission denied: '/app'
|
||||||
|
2025-11-12 04:08:07,252 - INFO - Quelle https://wordpress.org/documentation/ erfolgreich gecrawlt.
|
||||||
|
2025-11-12 04:08:07,256 - INFO - Lösche Zustandsdatei.
|
||||||
|
2025-11-12 04:08:07,259 - INFO - Datenbankverbindung für Hashes geschlossen.
|
||||||
|
2025-11-12 04:08:07,262 - INFO - Finished source: https://wordpress.org/documentation/
|
||||||
|
2025-11-12 04:08:07,265 - INFO - Processing source: https://github.com/fail2ban/
|
||||||
|
2025-11-12 04:08:07,268 - INFO - Starte Quelle: https://github.com/fail2ban/ (Modus: domain_wide)
|
||||||
|
2025-11-12 04:08:07,271 - INFO - Crawle: https://github.com/fail2ban/
|
||||||
|
2025-11-12 04:08:07,273 - INFO - Lade robots.txt von: https://github.com/robots.txt
|
||||||
|
2025-11-12 04:08:08,043 - ERROR - Allgemeiner Fehler beim Crawlen von https://github.com/fail2ban/: [Errno 13] Permission denied: '/app'
|
||||||
|
2025-11-12 04:08:10,047 - INFO - Quelle https://github.com/fail2ban/ erfolgreich gecrawlt.
|
||||||
|
2025-11-12 04:08:10,052 - INFO - Lösche Zustandsdatei.
|
||||||
|
2025-11-12 04:08:10,056 - INFO - Datenbankverbindung für Hashes geschlossen.
|
||||||
|
2025-11-12 04:08:10,058 - INFO - Finished source: https://github.com/fail2ban/
|
||||||
|
2025-11-12 04:08:10,061 - INFO - Processing source: httpshttps://grafana.com/docs/
|
||||||
|
2025-11-12 04:08:10,064 - INFO - Starte Quelle: httpshttps://grafana.com/docs/ (Modus: domain_wide)
|
||||||
|
2025-11-12 04:08:10,067 - INFO - Crawle: httpshttps://grafana.com/docs/
|
||||||
|
2025-11-12 04:08:10,070 - INFO - Lade robots.txt von: /robots.txt
|
||||||
|
2025-11-12 04:08:10,072 - ERROR - Fehler beim Abruf von /robots.txt: /robots.txt
|
||||||
|
2025-11-12 04:08:10,074 - WARNING - robots.txt verbietet: httpshttps://grafana.com/docs/
|
||||||
|
2025-11-12 04:08:12,078 - INFO - Quelle httpshttps://grafana.com/docs/ erfolgreich gecrawlt.
|
||||||
|
2025-11-12 04:08:12,084 - INFO - Lösche Zustandsdatei.
|
||||||
|
2025-11-12 04:08:12,088 - INFO - Datenbankverbindung für Hashes geschlossen.
|
||||||
|
2025-11-12 04:08:12,091 - INFO - Finished source: httpshttps://grafana.com/docs/
|
||||||
|
2025-11-12 04:08:12,094 - INFO - Processing source: https://nodejs.org/docs/latest/
|
||||||
|
2025-11-12 04:08:12,102 - INFO - Starte Quelle: https://nodejs.org/docs/latest/ (Modus: domain_wide)
|
||||||
|
2025-11-12 04:08:12,104 - INFO - Crawle: https://nodejs.org/docs/latest/
|
||||||
|
2025-11-12 04:08:12,107 - INFO - Lade robots.txt von: https://nodejs.org/robots.txt
|
||||||
|
2025-11-12 04:08:12,220 - WARNING - robots.txt verbietet: https://nodejs.org/docs/latest/
|
||||||
|
2025-11-12 04:08:14,224 - INFO - Quelle https://nodejs.org/docs/latest/ erfolgreich gecrawlt.
|
||||||
|
2025-11-12 04:08:14,230 - INFO - Lösche Zustandsdatei.
|
||||||
|
2025-11-12 04:08:14,234 - INFO - Datenbankverbindung für Hashes geschlossen.
|
||||||
|
2025-11-12 04:08:14,238 - INFO - Finished source: https://nodejs.org/docs/latest/
|
||||||
|
2025-11-12 04:08:14,240 - INFO - Processing source: https://www.virtualmin.com/docs/
|
||||||
|
2025-11-12 04:08:14,243 - INFO - Starte Quelle: https://www.virtualmin.com/docs/ (Modus: domain_wide)
|
||||||
|
2025-11-12 04:08:14,246 - INFO - Crawle: https://www.virtualmin.com/docs/
|
||||||
|
2025-11-12 04:08:14,249 - INFO - Lade robots.txt von: https://www.virtualmin.com/robots.txt
|
||||||
|
2025-11-12 04:08:15,381 - ERROR - Allgemeiner Fehler beim Crawlen von https://www.virtualmin.com/docs/: [Errno 13] Permission denied: '/app'
|
||||||
|
2025-11-12 04:08:17,385 - INFO - Quelle https://www.virtualmin.com/docs/ erfolgreich gecrawlt.
|
||||||
|
2025-11-12 04:08:17,388 - INFO - Lösche Zustandsdatei.
|
||||||
|
2025-11-12 04:08:17,392 - INFO - Datenbankverbindung für Hashes geschlossen.
|
||||||
|
2025-11-12 04:08:17,394 - INFO - Finished source: https://www.virtualmin.com/docs/
|
||||||
|
2025-11-12 04:08:17,398 - INFO - Processing source: https://developer.mozilla.org/en-US/docs/Web/
|
||||||
|
2025-11-12 04:08:17,403 - INFO - Starte Quelle: https://developer.mozilla.org/en-US/docs/Web/ (Modus: domain_wide)
|
||||||
|
2025-11-12 04:08:17,406 - INFO - Crawle: https://developer.mozilla.org/en-US/docs/Web/
|
||||||
|
2025-11-12 04:08:17,409 - INFO - Lade robots.txt von: https://developer.mozilla.org/robots.txt
|
||||||
|
2025-11-12 04:08:18,351 - ERROR - Allgemeiner Fehler beim Crawlen von https://developer.mozilla.org/en-US/docs/Web/: [Errno 13] Permission denied: '/app'
|
||||||
|
2025-11-12 04:08:20,355 - INFO - Quelle https://developer.mozilla.org/en-US/docs/Web/ erfolgreich gecrawlt.
|
||||||
|
2025-11-12 04:08:20,359 - INFO - Lösche Zustandsdatei.
|
||||||
|
2025-11-12 04:08:20,362 - INFO - Datenbankverbindung für Hashes geschlossen.
|
||||||
|
2025-11-12 04:08:20,365 - INFO - Finished source: https://developer.mozilla.org/en-US/docs/Web/
|
||||||
|
2025-11-12 04:08:20,368 - INFO - Processing source: https://owasp.org/www-project-top-ten/
|
||||||
|
2025-11-12 04:08:20,371 - INFO - Starte Quelle: https://owasp.org/www-project-top-ten/ (Modus: domain_wide)
|
||||||
|
2025-11-12 04:08:20,373 - INFO - Crawle: https://owasp.org/www-project-top-ten/
|
||||||
|
2025-11-12 04:08:20,375 - INFO - Lade robots.txt von: https://owasp.org/robots.txt
|
||||||
|
2025-11-12 04:08:20,707 - ERROR - Allgemeiner Fehler beim Crawlen von https://owasp.org/www-project-top-ten/: [Errno 13] Permission denied: '/app'
|
||||||
|
2025-11-12 04:08:22,711 - INFO - Quelle https://owasp.org/www-project-top-ten/ erfolgreich gecrawlt.
|
||||||
|
2025-11-12 04:08:22,717 - INFO - Lösche Zustandsdatei.
|
||||||
|
2025-11-12 04:08:22,721 - INFO - Datenbankverbindung für Hashes geschlossen.
|
||||||
|
2025-11-12 04:08:22,724 - INFO - Finished source: https://owasp.org/www-project-top-ten/
|
||||||
|
2025-11-12 04:08:22,726 - INFO - Processing source: https://cheatsheetseries.owasp.org/
|
||||||
|
2025-11-12 04:08:22,730 - INFO - Starte Quelle: https://cheatsheetseries.owasp.org/ (Modus: domain_wide)
|
||||||
|
2025-11-12 04:08:22,733 - INFO - Crawle: https://cheatsheetseries.owasp.org/
|
||||||
|
2025-11-12 04:08:22,736 - INFO - Lade robots.txt von: https://cheatsheetseries.owasp.org/robots.txt
|
||||||
|
2025-11-12 04:08:23,037 - INFO - Inhalt verworfen (Länge: 489 < 500): https://cheatsheetseries.owasp.org/
|
||||||
|
2025-11-12 04:08:25,060 - INFO - Crawle: https://cheatsheetseries.owasp.org/Glossary.html
|
||||||
|
2025-11-12 04:08:25,317 - ERROR - Allgemeiner Fehler beim Crawlen von https://cheatsheetseries.owasp.org/Glossary.html: [Errno 13] Permission denied: '/app'
|
||||||
|
2025-11-12 04:08:27,321 - INFO - Crawle: https://cheatsheetseries.owasp.org/IndexASVS.html
|
||||||
|
2025-11-12 04:08:27,635 - ERROR - Allgemeiner Fehler beim Crawlen von https://cheatsheetseries.owasp.org/IndexASVS.html: [Errno 13] Permission denied: '/app'
|
||||||
|
2025-11-12 04:08:29,639 - INFO - Crawle: https://cheatsheetseries.owasp.org/IndexMASVS.html
|
||||||
|
2025-11-12 04:08:29,845 - ERROR - Allgemeiner Fehler beim Crawlen von https://cheatsheetseries.owasp.org/IndexMASVS.html: [Errno 13] Permission denied: '/app'
|
||||||
|
2025-11-12 04:08:31,850 - INFO - Crawle: https://cheatsheetseries.owasp.org/IndexProactiveControls.html
|
||||||
|
2025-11-12 04:08:32,060 - ERROR - Allgemeiner Fehler beim Crawlen von https://cheatsheetseries.owasp.org/IndexProactiveControls.html: [Errno 13] Permission denied: '/app'
|
||||||
|
2025-11-12 04:08:34,064 - INFO - Crawle: https://cheatsheetseries.owasp.org/IndexTopTen.html
|
||||||
|
2025-11-12 04:08:34,266 - ERROR - Allgemeiner Fehler beim Crawlen von https://cheatsheetseries.owasp.org/IndexTopTen.html: [Errno 13] Permission denied: '/app'
|
||||||
|
2025-11-12 04:08:36,270 - INFO - Crawle: https://cheatsheetseries.owasp.org/News.xml
|
||||||
|
2025-11-12 04:08:36,320 - ERROR - parsed tree length: 1, wrong data type or not valid HTML
|
||||||
|
2025-11-12 04:08:36,323 - ERROR - empty HTML tree: None
|
||||||
|
2025-11-12 04:08:36,325 - WARNING - discarding data: None
|
||||||
|
2025-11-12 04:08:36,367 - ERROR - Allgemeiner Fehler beim Crawlen von https://cheatsheetseries.owasp.org/News.xml: [Errno 13] Permission denied: '/app'
|
||||||
|
2025-11-12 04:08:38,371 - INFO - Crawle: https://cheatsheetseries.owasp.org/bundle.zip
|
||||||
|
2025-11-12 04:08:41,239 - ERROR - Allgemeiner Fehler beim Crawlen von https://cheatsheetseries.owasp.org/bundle.zip: 'utf-8' codec can't decode byte 0x97 in position 11: invalid start byte
|
||||||
|
2025-11-12 04:08:43,245 - INFO - Crawle: https://cheatsheetseries.owasp.org/cheatsheets/AJAX_Security_Cheat_Sheet.html
|
||||||
|
2025-11-12 04:08:43,518 - ERROR - Allgemeiner Fehler beim Crawlen von https://cheatsheetseries.owasp.org/cheatsheets/AJAX_Security_Cheat_Sheet.html: [Errno 13] Permission denied: '/app'
|
||||||
|
2025-11-12 04:08:45,521 - INFO - Crawle: https://cheatsheetseries.owasp.org/cheatsheets/Abuse_Case_Cheat_Sheet.html
|
||||||
|
2025-11-12 04:08:45,833 - ERROR - Allgemeiner Fehler beim Crawlen von https://cheatsheetseries.owasp.org/cheatsheets/Abuse_Case_Cheat_Sheet.html: [Errno 13] Permission denied: '/app'
|
||||||
|
2025-11-12 04:08:47,837 - INFO - Crawle: https://cheatsheetseries.owasp.org/cheatsheets/Access_Control_Cheat_Sheet.html
|
||||||
|
2025-11-12 04:08:48,020 - INFO - Inhalt verworfen (Länge: 204 < 500): https://cheatsheetseries.owasp.org/cheatsheets/Access_Control_Cheat_Sheet.html
|
||||||
|
2025-11-12 04:08:50,047 - INFO - Crawle: https://cheatsheetseries.owasp.org/cheatsheets/Attack_Surface_Analysis_Cheat_Sheet.html
|
||||||
|
2025-11-12 04:08:50,286 - ERROR - Allgemeiner Fehler beim Crawlen von https://cheatsheetseries.owasp.org/cheatsheets/Attack_Surface_Analysis_Cheat_Sheet.html: [Errno 13] Permission denied: '/app'
|
||||||
|
2025-11-12 04:08:52,290 - INFO - Crawle: https://cheatsheetseries.owasp.org/cheatsheets/Authentication_Cheat_Sheet.html
|
||||||
|
2025-11-12 04:08:52,638 - ERROR - Allgemeiner Fehler beim Crawlen von https://cheatsheetseries.owasp.org/cheatsheets/Authentication_Cheat_Sheet.html: [Errno 13] Permission denied: '/app'
|
||||||
|
2025-11-12 04:08:54,643 - INFO - Crawle: https://cheatsheetseries.owasp.org/cheatsheets/Authorization_Cheat_Sheet.html
|
||||||
|
2025-11-12 04:08:54,926 - ERROR - Allgemeiner Fehler beim Crawlen von https://cheatsheetseries.owasp.org/cheatsheets/Authorization_Cheat_Sheet.html: [Errno 13] Permission denied: '/app'
|
||||||
|
2025-11-12 04:08:56,930 - INFO - Crawle: https://cheatsheetseries.owasp.org/cheatsheets/Authorization_Testing_Automation_Cheat_Sheet.html
|
||||||
|
2025-11-12 04:08:57,541 - ERROR - Allgemeiner Fehler beim Crawlen von https://cheatsheetseries.owasp.org/cheatsheets/Authorization_Testing_Automation_Cheat_Sheet.html: [Errno 13] Permission denied: '/app'
|
||||||
|
2025-11-12 04:08:59,545 - INFO - Crawle: https://cheatsheetseries.owasp.org/cheatsheets/Automotive_Security.html
|
||||||
|
2025-11-12 04:08:59,752 - ERROR - Allgemeiner Fehler beim Crawlen von https://cheatsheetseries.owasp.org/cheatsheets/Automotive_Security.html: [Errno 13] Permission denied: '/app'
|
||||||
|
2025-11-12 04:09:01,756 - INFO - Crawle: https://cheatsheetseries.owasp.org/cheatsheets/Bean_Validation_Cheat_Sheet.html
|
||||||
|
2025-11-12 04:09:02,186 - ERROR - Allgemeiner Fehler beim Crawlen von https://cheatsheetseries.owasp.org/cheatsheets/Bean_Validation_Cheat_Sheet.html: [Errno 13] Permission denied: '/app'
|
||||||
|
2025-11-12 04:09:04,190 - INFO - Crawle: https://cheatsheetseries.owasp.org/cheatsheets/Browser_Extension_Vulnerabilities_Cheat_Sheet.html
|
||||||
|
2025-11-12 04:09:05,015 - ERROR - Allgemeiner Fehler beim Crawlen von https://cheatsheetseries.owasp.org/cheatsheets/Browser_Extension_Vulnerabilities_Cheat_Sheet.html: [Errno 13] Permission denied: '/app'
|
||||||
|
2025-11-12 04:09:07,019 - INFO - Crawle: https://cheatsheetseries.owasp.org/cheatsheets/C-Based_Toolchain_Hardening_Cheat_Sheet.html
|
||||||
|
2025-11-12 04:09:07,592 - ERROR - Allgemeiner Fehler beim Crawlen von https://cheatsheetseries.owasp.org/cheatsheets/C-Based_Toolchain_Hardening_Cheat_Sheet.html: [Errno 13] Permission denied: '/app'
|
||||||
|
2025-11-12 04:09:09,597 - INFO - Crawle: https://cheatsheetseries.owasp.org/cheatsheets/CI_CD_Security_Cheat_Sheet.html
|
||||||
|
2025-11-12 04:09:09,874 - ERROR - Allgemeiner Fehler beim Crawlen von https://cheatsheetseries.owasp.org/cheatsheets/CI_CD_Security_Cheat_Sheet.html: [Errno 13] Permission denied: '/app'
|
||||||
|
2025-11-12 04:09:11,877 - INFO - Crawle: https://cheatsheetseries.owasp.org/cheatsheets/Choosing_and_Using_Security_Questions_Cheat_Sheet.html
|
||||||
|
2025-11-12 04:09:12,113 - ERROR - Allgemeiner Fehler beim Crawlen von https://cheatsheetseries.owasp.org/cheatsheets/Choosing_and_Using_Security_Questions_Cheat_Sheet.html: [Errno 13] Permission denied: '/app'
|
||||||
|
2025-11-12 04:09:14,118 - INFO - Crawle: https://cheatsheetseries.owasp.org/cheatsheets/Clickjacking_Defense_Cheat_Sheet.html
|
||||||
|
2025-11-12 04:09:14,422 - ERROR - Allgemeiner Fehler beim Crawlen von https://cheatsheetseries.owasp.org/cheatsheets/Clickjacking_Defense_Cheat_Sheet.html: [Errno 13] Permission denied: '/app'
|
||||||
|
2025-11-12 04:09:16,426 - INFO - Crawle: https://cheatsheetseries.owasp.org/cheatsheets/Content_Security_Policy_Cheat_Sheet.html
|
||||||
|
2025-11-12 04:09:16,747 - ERROR - Allgemeiner Fehler beim Crawlen von https://cheatsheetseries.owasp.org/cheatsheets/Content_Security_Policy_Cheat_Sheet.html: [Errno 13] Permission denied: '/app'
|
||||||
|
2025-11-12 04:09:18,752 - INFO - Crawle: https://cheatsheetseries.owasp.org/cheatsheets/Cookie_Theft_Mitigation_Cheat_Sheet.html
|
||||||
|
2025-11-12 04:09:19,033 - ERROR - Allgemeiner Fehler beim Crawlen von https://cheatsheetseries.owasp.org/cheatsheets/Cookie_Theft_Mitigation_Cheat_Sheet.html: [Errno 13] Permission denied: '/app'
|
||||||
|
2025-11-12 04:09:21,037 - INFO - Crawle: https://cheatsheetseries.owasp.org/cheatsheets/Credential_Stuffing_Prevention_Cheat_Sheet.html
|
||||||
|
2025-11-12 04:09:21,269 - ERROR - Allgemeiner Fehler beim Crawlen von https://cheatsheetseries.owasp.org/cheatsheets/Credential_Stuffing_Prevention_Cheat_Sheet.html: [Errno 13] Permission denied: '/app'
|
||||||
|
2025-11-12 04:09:23,273 - INFO - Crawle: https://cheatsheetseries.owasp.org/cheatsheets/Cross-Site_Request_Forgery_Prevention_Cheat_Sheet.html
|
||||||
|
2025-11-12 04:09:24,119 - ERROR - Allgemeiner Fehler beim Crawlen von https://cheatsheetseries.owasp.org/cheatsheets/Cross-Site_Request_Forgery_Prevention_Cheat_Sheet.html: [Errno 13] Permission denied: '/app'
|
||||||
|
2025-11-12 04:09:26,128 - INFO - Crawle: https://cheatsheetseries.owasp.org/cheatsheets/Cross_Site_Scripting_Prevention_Cheat_Sheet.html
|
||||||
|
2025-11-12 04:09:26,485 - ERROR - Allgemeiner Fehler beim Crawlen von https://cheatsheetseries.owasp.org/cheatsheets/Cross_Site_Scripting_Prevention_Cheat_Sheet.html: [Errno 13] Permission denied: '/app'
|
||||||
|
2025-11-12 04:09:28,488 - INFO - Crawle: https://cheatsheetseries.owasp.org/cheatsheets/Cryptographic_Storage_Cheat_Sheet.html
|
||||||
|
2025-11-12 04:09:28,823 - ERROR - Allgemeiner Fehler beim Crawlen von https://cheatsheetseries.owasp.org/cheatsheets/Cryptographic_Storage_Cheat_Sheet.html: [Errno 13] Permission denied: '/app'
|
||||||
|
2025-11-12 04:09:30,827 - INFO - Crawle: https://cheatsheetseries.owasp.org/cheatsheets/DOM_Clobbering_Prevention_Cheat_Sheet.html
|
||||||
|
2025-11-12 04:09:31,123 - ERROR - Allgemeiner Fehler beim Crawlen von https://cheatsheetseries.owasp.org/cheatsheets/DOM_Clobbering_Prevention_Cheat_Sheet.html: [Errno 13] Permission denied: '/app'
|
||||||
|
2025-11-12 04:09:33,127 - INFO - Crawle: https://cheatsheetseries.owasp.org/cheatsheets/DOM_based_XSS_Prevention_Cheat_Sheet.html
|
||||||
|
2025-11-12 04:09:33,649 - ERROR - Allgemeiner Fehler beim Crawlen von https://cheatsheetseries.owasp.org/cheatsheets/DOM_based_XSS_Prevention_Cheat_Sheet.html: [Errno 13] Permission denied: '/app'
|
||||||
|
2025-11-12 04:09:35,654 - INFO - Crawle: https://cheatsheetseries.owasp.org/cheatsheets/Database_Security_Cheat_Sheet.html
|
||||||
|
2025-11-12 04:09:35,875 - ERROR - Allgemeiner Fehler beim Crawlen von https://cheatsheetseries.owasp.org/cheatsheets/Database_Security_Cheat_Sheet.html: [Errno 13] Permission denied: '/app'
|
||||||
|
2025-11-12 04:09:37,879 - INFO - Crawle: https://cheatsheetseries.owasp.org/cheatsheets/Denial_of_Service_Cheat_Sheet.html
|
||||||
|
2025-11-12 04:09:38,130 - ERROR - Allgemeiner Fehler beim Crawlen von https://cheatsheetseries.owasp.org/cheatsheets/Denial_of_Service_Cheat_Sheet.html: [Errno 13] Permission denied: '/app'
|
||||||
|
2025-11-12 04:09:40,135 - INFO - Crawle: https://cheatsheetseries.owasp.org/cheatsheets/Dependency_Graph_SBOM_Cheat_Sheet.html
|
||||||
|
2025-11-12 04:09:40,401 - ERROR - Allgemeiner Fehler beim Crawlen von https://cheatsheetseries.owasp.org/cheatsheets/Dependency_Graph_SBOM_Cheat_Sheet.html: [Errno 13] Permission denied: '/app'
|
||||||
|
2025-11-12 04:09:42,404 - INFO - Crawle: https://cheatsheetseries.owasp.org/cheatsheets/Deserialization_Cheat_Sheet.html
|
||||||
|
2025-11-12 04:09:42,762 - ERROR - Allgemeiner Fehler beim Crawlen von https://cheatsheetseries.owasp.org/cheatsheets/Deserialization_Cheat_Sheet.html: [Errno 13] Permission denied: '/app'
|
||||||
|
2025-11-12 04:09:44,767 - INFO - Crawle: https://cheatsheetseries.owasp.org/cheatsheets/Django_REST_Framework_Cheat_Sheet.html
|
||||||
|
2025-11-12 04:09:45,055 - ERROR - Allgemeiner Fehler beim Crawlen von https://cheatsheetseries.owasp.org/cheatsheets/Django_REST_Framework_Cheat_Sheet.html: [Errno 13] Permission denied: '/app'
|
||||||
|
2025-11-12 04:09:47,059 - INFO - Crawle: https://cheatsheetseries.owasp.org/cheatsheets/Django_Security_Cheat_Sheet.html
|
||||||
|
2025-11-12 04:09:47,329 - ERROR - Allgemeiner Fehler beim Crawlen von https://cheatsheetseries.owasp.org/cheatsheets/Django_Security_Cheat_Sheet.html: [Errno 13] Permission denied: '/app'
|
||||||
|
2025-11-12 04:09:49,335 - INFO - Crawle: https://cheatsheetseries.owasp.org/cheatsheets/Docker_Security_Cheat_Sheet.html
|
||||||
|
2025-11-12 04:09:49,709 - ERROR - Allgemeiner Fehler beim Crawlen von https://cheatsheetseries.owasp.org/cheatsheets/Docker_Security_Cheat_Sheet.html: [Errno 13] Permission denied: '/app'
|
||||||
|
2025-11-12 04:09:51,713 - INFO - Crawle: https://cheatsheetseries.owasp.org/cheatsheets/DotNet_Security_Cheat_Sheet.html
|
||||||
|
2025-11-12 04:09:52,690 - ERROR - Allgemeiner Fehler beim Crawlen von https://cheatsheetseries.owasp.org/cheatsheets/DotNet_Security_Cheat_Sheet.html: [Errno 13] Permission denied: '/app'
|
||||||
|
2025-11-12 04:09:54,695 - INFO - Crawle: https://cheatsheetseries.owasp.org/cheatsheets/Drone_Security_Cheat_Sheet.html
|
||||||
|
2025-11-12 04:09:54,986 - ERROR - Allgemeiner Fehler beim Crawlen von https://cheatsheetseries.owasp.org/cheatsheets/Drone_Security_Cheat_Sheet.html: [Errno 13] Permission denied: '/app'
|
||||||
|
2025-11-12 04:09:56,990 - INFO - Crawle: https://cheatsheetseries.owasp.org/cheatsheets/Error_Handling_Cheat_Sheet.html
|
||||||
|
2025-11-12 04:09:57,368 - ERROR - Allgemeiner Fehler beim Crawlen von https://cheatsheetseries.owasp.org/cheatsheets/Error_Handling_Cheat_Sheet.html: [Errno 13] Permission denied: '/app'
|
||||||
|
2025-11-12 04:09:59,372 - INFO - Crawle: https://cheatsheetseries.owasp.org/cheatsheets/File_Upload_Cheat_Sheet.html
|
||||||
|
2025-11-12 04:09:59,612 - ERROR - Allgemeiner Fehler beim Crawlen von https://cheatsheetseries.owasp.org/cheatsheets/File_Upload_Cheat_Sheet.html: [Errno 13] Permission denied: '/app'
|
||||||
|
2025-11-12 04:10:01,617 - INFO - Crawle: https://cheatsheetseries.owasp.org/cheatsheets/Forgot_Password_Cheat_Sheet.html
|
||||||
|
2025-11-12 04:10:01,842 - ERROR - Allgemeiner Fehler beim Crawlen von https://cheatsheetseries.owasp.org/cheatsheets/Forgot_Password_Cheat_Sheet.html: [Errno 13] Permission denied: '/app'
|
||||||
|
2025-11-12 04:10:03,847 - INFO - Crawle: https://cheatsheetseries.owasp.org/cheatsheets/GraphQL_Cheat_Sheet.html
|
||||||
|
2025-11-12 04:10:04,227 - ERROR - Allgemeiner Fehler beim Crawlen von https://cheatsheetseries.owasp.org/cheatsheets/GraphQL_Cheat_Sheet.html: [Errno 13] Permission denied: '/app'
|
||||||
|
2025-11-12 04:10:06,231 - INFO - Crawle: https://cheatsheetseries.owasp.org/cheatsheets/HTML5_Security_Cheat_Sheet.html
|
||||||
|
2025-11-12 04:10:06,561 - ERROR - Allgemeiner Fehler beim Crawlen von https://cheatsheetseries.owasp.org/cheatsheets/HTML5_Security_Cheat_Sheet.html: [Errno 13] Permission denied: '/app'
|
||||||
|
2025-11-12 04:10:08,567 - INFO - Crawle: https://cheatsheetseries.owasp.org/cheatsheets/HTTP_Headers_Cheat_Sheet.html
|
||||||
|
2025-11-12 04:10:08,868 - ERROR - Allgemeiner Fehler beim Crawlen von https://cheatsheetseries.owasp.org/cheatsheets/HTTP_Headers_Cheat_Sheet.html: [Errno 13] Permission denied: '/app'
|
||||||
|
2025-11-12 04:10:10,872 - INFO - Crawle: https://cheatsheetseries.owasp.org/cheatsheets/HTTP_Strict_Transport_Security_Cheat_Sheet.html
|
||||||
|
2025-11-12 04:10:11,064 - ERROR - Allgemeiner Fehler beim Crawlen von https://cheatsheetseries.owasp.org/cheatsheets/HTTP_Strict_Transport_Security_Cheat_Sheet.html: [Errno 13] Permission denied: '/app'
|
||||||
|
2025-11-12 04:10:13,067 - INFO - Crawle: https://cheatsheetseries.owasp.org/cheatsheets/Infrastructure_as_Code_Security_Cheat_Sheet.html
|
||||||
|
2025-11-12 04:10:13,296 - ERROR - Allgemeiner Fehler beim Crawlen von https://cheatsheetseries.owasp.org/cheatsheets/Infrastructure_as_Code_Security_Cheat_Sheet.html: [Errno 13] Permission denied: '/app'
|
||||||
|
2025-11-12 04:10:15,301 - INFO - Crawle: https://cheatsheetseries.owasp.org/cheatsheets/Injection_Prevention_Cheat_Sheet.html
|
||||||
|
2025-11-12 04:10:15,672 - ERROR - Allgemeiner Fehler beim Crawlen von https://cheatsheetseries.owasp.org/cheatsheets/Injection_Prevention_Cheat_Sheet.html: [Errno 13] Permission denied: '/app'
|
||||||
|
2025-11-12 04:10:17,676 - INFO - Crawle: https://cheatsheetseries.owasp.org/cheatsheets/Injection_Prevention_in_Java_Cheat_Sheet.html
|
||||||
|
2025-11-12 04:10:17,872 - INFO - Inhalt verworfen (Länge: 193 < 500): https://cheatsheetseries.owasp.org/cheatsheets/Injection_Prevention_in_Java_Cheat_Sheet.html
|
||||||
|
2025-11-12 04:10:19,894 - INFO - Crawle: https://cheatsheetseries.owasp.org/cheatsheets/Input_Validation_Cheat_Sheet.html
|
||||||
|
2025-11-12 04:10:20,194 - ERROR - Allgemeiner Fehler beim Crawlen von https://cheatsheetseries.owasp.org/cheatsheets/Input_Validation_Cheat_Sheet.html: [Errno 13] Permission denied: '/app'
|
||||||
|
2025-11-12 04:10:22,199 - INFO - Crawle: https://cheatsheetseries.owasp.org/cheatsheets/Insecure_Direct_Object_Reference_Prevention_Cheat_Sheet.html
|
||||||
|
2025-11-12 04:10:22,389 - ERROR - Allgemeiner Fehler beim Crawlen von https://cheatsheetseries.owasp.org/cheatsheets/Insecure_Direct_Object_Reference_Prevention_Cheat_Sheet.html: [Errno 13] Permission denied: '/app'
|
||||||
|
2025-11-12 04:10:24,394 - INFO - Crawle: https://cheatsheetseries.owasp.org/cheatsheets/JAAS_Cheat_Sheet.html
|
||||||
|
2025-11-12 04:10:24,689 - ERROR - Allgemeiner Fehler beim Crawlen von https://cheatsheetseries.owasp.org/cheatsheets/JAAS_Cheat_Sheet.html: [Errno 13] Permission denied: '/app'
|
||||||
|
2025-11-12 04:10:26,693 - INFO - Crawle: https://cheatsheetseries.owasp.org/cheatsheets/JSON_Web_Token_for_Java_Cheat_Sheet.html
|
||||||
|
2025-11-12 04:10:27,362 - ERROR - Allgemeiner Fehler beim Crawlen von https://cheatsheetseries.owasp.org/cheatsheets/JSON_Web_Token_for_Java_Cheat_Sheet.html: [Errno 13] Permission denied: '/app'
|
||||||
|
2025-11-12 04:10:29,366 - INFO - Crawle: https://cheatsheetseries.owasp.org/cheatsheets/Java_Security_Cheat_Sheet.html
|
||||||
|
2025-11-12 04:10:30,414 - ERROR - Allgemeiner Fehler beim Crawlen von https://cheatsheetseries.owasp.org/cheatsheets/Java_Security_Cheat_Sheet.html: [Errno 13] Permission denied: '/app'
|
||||||
|
2025-11-12 04:10:32,418 - INFO - Crawle: https://cheatsheetseries.owasp.org/cheatsheets/Key_Management_Cheat_Sheet.html
|
||||||
|
2025-11-12 04:10:32,699 - ERROR - Allgemeiner Fehler beim Crawlen von https://cheatsheetseries.owasp.org/cheatsheets/Key_Management_Cheat_Sheet.html: [Errno 13] Permission denied: '/app'
|
||||||
|
2025-11-12 04:10:34,703 - INFO - Crawle: https://cheatsheetseries.owasp.org/cheatsheets/Kubernetes_Security_Cheat_Sheet.html
|
||||||
|
2025-11-12 04:10:35,296 - ERROR - Allgemeiner Fehler beim Crawlen von https://cheatsheetseries.owasp.org/cheatsheets/Kubernetes_Security_Cheat_Sheet.html: [Errno 13] Permission denied: '/app'
|
||||||
|
2025-11-12 04:10:37,302 - INFO - Crawle: https://cheatsheetseries.owasp.org/cheatsheets/LDAP_Injection_Prevention_Cheat_Sheet.html
|
||||||
|
2025-11-12 04:10:37,557 - ERROR - Allgemeiner Fehler beim Crawlen von https://cheatsheetseries.owasp.org/cheatsheets/LDAP_Injection_Prevention_Cheat_Sheet.html: [Errno 13] Permission denied: '/app'
|
||||||
|
2025-11-12 04:10:39,561 - INFO - Crawle: https://cheatsheetseries.owasp.org/cheatsheets/LLM_Prompt_Injection_Prevention_Cheat_Sheet.html
|
||||||
|
2025-11-12 04:10:40,044 - ERROR - Allgemeiner Fehler beim Crawlen von https://cheatsheetseries.owasp.org/cheatsheets/LLM_Prompt_Injection_Prevention_Cheat_Sheet.html: [Errno 13] Permission denied: '/app'
|
||||||
|
2025-11-12 04:10:42,048 - INFO - Crawle: https://cheatsheetseries.owasp.org/cheatsheets/Laravel_Cheat_Sheet.html
|
||||||
|
2025-11-12 04:10:42,404 - ERROR - Allgemeiner Fehler beim Crawlen von https://cheatsheetseries.owasp.org/cheatsheets/Laravel_Cheat_Sheet.html: [Errno 13] Permission denied: '/app'
|
||||||
|
2025-11-12 04:10:44,408 - INFO - Crawle: https://cheatsheetseries.owasp.org/cheatsheets/Legacy_Application_Management_Cheat_Sheet.html
|
||||||
|
2025-11-12 04:10:44,642 - ERROR - Allgemeiner Fehler beim Crawlen von https://cheatsheetseries.owasp.org/cheatsheets/Legacy_Application_Management_Cheat_Sheet.html: [Errno 13] Permission denied: '/app'
|
||||||
|
2025-11-12 04:10:46,645 - INFO - Crawle: https://cheatsheetseries.owasp.org/cheatsheets/Logging_Cheat_Sheet.html
|
||||||
|
2025-11-12 04:10:46,975 - ERROR - Allgemeiner Fehler beim Crawlen von https://cheatsheetseries.owasp.org/cheatsheets/Logging_Cheat_Sheet.html: [Errno 13] Permission denied: '/app'
|
||||||
|
2025-11-12 04:10:48,979 - INFO - Crawle: https://cheatsheetseries.owasp.org/cheatsheets/Logging_Vocabulary_Cheat_Sheet.html
|
||||||
|
2025-11-12 04:10:49,371 - ERROR - Allgemeiner Fehler beim Crawlen von https://cheatsheetseries.owasp.org/cheatsheets/Logging_Vocabulary_Cheat_Sheet.html: [Errno 13] Permission denied: '/app'
|
||||||
|
2025-11-12 04:10:51,375 - INFO - Crawle: https://cheatsheetseries.owasp.org/cheatsheets/Mass_Assignment_Cheat_Sheet.html
|
||||||
|
2025-11-12 04:10:51,714 - ERROR - Allgemeiner Fehler beim Crawlen von https://cheatsheetseries.owasp.org/cheatsheets/Mass_Assignment_Cheat_Sheet.html: [Errno 13] Permission denied: '/app'
|
||||||
|
2025-11-12 04:10:53,718 - INFO - Crawle: https://cheatsheetseries.owasp.org/cheatsheets/Microservices_Security_Cheat_Sheet.html
|
||||||
|
2025-11-12 04:10:53,987 - ERROR - Allgemeiner Fehler beim Crawlen von https://cheatsheetseries.owasp.org/cheatsheets/Microservices_Security_Cheat_Sheet.html: [Errno 13] Permission denied: '/app'
|
||||||
|
2025-11-12 04:10:55,991 - INFO - Crawle: https://cheatsheetseries.owasp.org/cheatsheets/Microservices_based_Security_Arch_Doc_Cheat_Sheet.html
|
||||||
|
2025-11-12 04:10:56,310 - ERROR - Allgemeiner Fehler beim Crawlen von https://cheatsheetseries.owasp.org/cheatsheets/Microservices_based_Security_Arch_Doc_Cheat_Sheet.html: [Errno 13] Permission denied: '/app'
|
||||||
|
2025-11-12 04:10:58,314 - INFO - Crawle: https://cheatsheetseries.owasp.org/cheatsheets/Mobile_Application_Security_Cheat_Sheet.html
|
||||||
|
2025-11-12 04:10:58,616 - ERROR - Allgemeiner Fehler beim Crawlen von https://cheatsheetseries.owasp.org/cheatsheets/Mobile_Application_Security_Cheat_Sheet.html: [Errno 13] Permission denied: '/app'
|
||||||
|
2025-11-12 04:11:00,620 - INFO - Crawle: https://cheatsheetseries.owasp.org/cheatsheets/Multifactor_Authentication_Cheat_Sheet.html
|
||||||
|
2025-11-12 04:11:00,990 - ERROR - Allgemeiner Fehler beim Crawlen von https://cheatsheetseries.owasp.org/cheatsheets/Multifactor_Authentication_Cheat_Sheet.html: [Errno 13] Permission denied: '/app'
|
||||||
|
2025-11-12 04:11:02,995 - INFO - Crawle: https://cheatsheetseries.owasp.org/cheatsheets/NPM_Security_Cheat_Sheet.html
|
||||||
|
2025-11-12 04:11:03,310 - ERROR - Allgemeiner Fehler beim Crawlen von https://cheatsheetseries.owasp.org/cheatsheets/NPM_Security_Cheat_Sheet.html: [Errno 13] Permission denied: '/app'
|
||||||
|
2025-11-12 04:11:05,313 - INFO - Crawle: https://cheatsheetseries.owasp.org/cheatsheets/Network_Segmentation_Cheat_Sheet.html
|
||||||
|
2025-11-12 04:11:05,546 - ERROR - Allgemeiner Fehler beim Crawlen von https://cheatsheetseries.owasp.org/cheatsheets/Network_Segmentation_Cheat_Sheet.html: [Errno 13] Permission denied: '/app'
|
||||||
|
2025-11-12 04:11:07,550 - INFO - Crawle: https://cheatsheetseries.owasp.org/cheatsheets/NoSQL_Security_Cheat_Sheet.html
|
||||||
|
2025-11-12 04:11:07,807 - ERROR - Allgemeiner Fehler beim Crawlen von https://cheatsheetseries.owasp.org/cheatsheets/NoSQL_Security_Cheat_Sheet.html: [Errno 13] Permission denied: '/app'
|
||||||
|
2025-11-12 04:11:09,811 - INFO - Crawle: https://cheatsheetseries.owasp.org/cheatsheets/NodeJS_Docker_Cheat_Sheet.html
|
||||||
|
2025-11-12 04:11:10,134 - ERROR - Allgemeiner Fehler beim Crawlen von https://cheatsheetseries.owasp.org/cheatsheets/NodeJS_Docker_Cheat_Sheet.html: [Errno 13] Permission denied: '/app'
|
||||||
|
2025-11-12 04:11:12,139 - INFO - Crawle: https://cheatsheetseries.owasp.org/cheatsheets/Nodejs_Security_Cheat_Sheet.html
|
||||||
|
2025-11-12 04:11:12,772 - ERROR - Allgemeiner Fehler beim Crawlen von https://cheatsheetseries.owasp.org/cheatsheets/Nodejs_Security_Cheat_Sheet.html: [Errno 13] Permission denied: '/app'
|
||||||
|
2025-11-12 04:11:14,777 - INFO - Crawle: https://cheatsheetseries.owasp.org/cheatsheets/OAuth2_Cheat_Sheet.html
|
||||||
|
2025-11-12 04:11:15,002 - ERROR - Allgemeiner Fehler beim Crawlen von https://cheatsheetseries.owasp.org/cheatsheets/OAuth2_Cheat_Sheet.html: [Errno 13] Permission denied: '/app'
|
||||||
|
2025-11-12 04:11:17,006 - INFO - Crawle: https://cheatsheetseries.owasp.org/cheatsheets/OS_Command_Injection_Defense_Cheat_Sheet.html
|
||||||
|
2025-11-12 04:11:17,307 - ERROR - Allgemeiner Fehler beim Crawlen von https://cheatsheetseries.owasp.org/cheatsheets/OS_Command_Injection_Defense_Cheat_Sheet.html: [Errno 13] Permission denied: '/app'
|
||||||
|
2025-11-12 04:11:19,311 - INFO - Crawle: https://cheatsheetseries.owasp.org/cheatsheets/PHP_Configuration_Cheat_Sheet.html
|
||||||
|
2025-11-12 04:11:19,522 - ERROR - Allgemeiner Fehler beim Crawlen von https://cheatsheetseries.owasp.org/cheatsheets/PHP_Configuration_Cheat_Sheet.html: [Errno 13] Permission denied: '/app'
|
||||||
|
2025-11-12 04:11:21,525 - INFO - Crawle: https://cheatsheetseries.owasp.org/cheatsheets/Password_Storage_Cheat_Sheet.html
|
||||||
|
2025-11-12 04:11:21,791 - ERROR - Allgemeiner Fehler beim Crawlen von https://cheatsheetseries.owasp.org/cheatsheets/Password_Storage_Cheat_Sheet.html: [Errno 13] Permission denied: '/app'
|
||||||
|
2025-11-12 04:11:23,795 - INFO - Crawle: https://cheatsheetseries.owasp.org/cheatsheets/Pinning_Cheat_Sheet.html
|
||||||
|
2025-11-12 04:11:24,054 - ERROR - Allgemeiner Fehler beim Crawlen von https://cheatsheetseries.owasp.org/cheatsheets/Pinning_Cheat_Sheet.html: [Errno 13] Permission denied: '/app'
|
||||||
|
2025-11-12 04:11:26,057 - INFO - Crawle: https://cheatsheetseries.owasp.org/cheatsheets/Prototype_Pollution_Prevention_Cheat_Sheet.html
|
||||||
|
2025-11-12 04:11:26,259 - ERROR - Allgemeiner Fehler beim Crawlen von https://cheatsheetseries.owasp.org/cheatsheets/Prototype_Pollution_Prevention_Cheat_Sheet.html: [Errno 13] Permission denied: '/app'
|
||||||
|
2025-11-12 04:11:28,264 - INFO - Crawle: https://cheatsheetseries.owasp.org/cheatsheets/Query_Parameterization_Cheat_Sheet.html
|
||||||
|
2025-11-12 04:11:28,597 - ERROR - Allgemeiner Fehler beim Crawlen von https://cheatsheetseries.owasp.org/cheatsheets/Query_Parameterization_Cheat_Sheet.html: [Errno 13] Permission denied: '/app'
|
||||||
|
2025-11-12 04:11:30,601 - INFO - Crawle: https://cheatsheetseries.owasp.org/cheatsheets/REST_Assessment_Cheat_Sheet.html
|
||||||
|
2025-11-12 04:11:30,864 - ERROR - Allgemeiner Fehler beim Crawlen von https://cheatsheetseries.owasp.org/cheatsheets/REST_Assessment_Cheat_Sheet.html: [Errno 13] Permission denied: '/app'
|
||||||
|
2025-11-12 04:11:32,868 - INFO - Crawle: https://cheatsheetseries.owasp.org/cheatsheets/REST_Security_Cheat_Sheet.html
|
||||||
|
2025-11-12 04:11:33,164 - ERROR - Allgemeiner Fehler beim Crawlen von https://cheatsheetseries.owasp.org/cheatsheets/REST_Security_Cheat_Sheet.html: [Errno 13] Permission denied: '/app'
|
||||||
|
2025-11-12 04:11:35,169 - INFO - Crawle: https://cheatsheetseries.owasp.org/cheatsheets/Ruby_on_Rails_Cheat_Sheet.html
|
||||||
|
2025-11-12 04:11:35,564 - ERROR - Allgemeiner Fehler beim Crawlen von https://cheatsheetseries.owasp.org/cheatsheets/Ruby_on_Rails_Cheat_Sheet.html: [Errno 13] Permission denied: '/app'
|
||||||
|
2025-11-12 04:11:37,568 - INFO - Crawle: https://cheatsheetseries.owasp.org/cheatsheets/SAML_Security_Cheat_Sheet.html
|
||||||
|
2025-11-12 04:11:37,897 - ERROR - Allgemeiner Fehler beim Crawlen von https://cheatsheetseries.owasp.org/cheatsheets/SAML_Security_Cheat_Sheet.html: [Errno 13] Permission denied: '/app'
|
||||||
|
2025-11-12 04:11:39,902 - INFO - Crawle: https://cheatsheetseries.owasp.org/cheatsheets/SQL_Injection_Prevention_Cheat_Sheet.html
|
||||||
|
2025-11-12 04:11:40,243 - ERROR - Allgemeiner Fehler beim Crawlen von https://cheatsheetseries.owasp.org/cheatsheets/SQL_Injection_Prevention_Cheat_Sheet.html: [Errno 13] Permission denied: '/app'
|
||||||
|
2025-11-12 04:11:42,247 - INFO - Crawle: https://cheatsheetseries.owasp.org/cheatsheets/Secrets_Management_Cheat_Sheet.html
|
||||||
|
2025-11-12 04:11:42,741 - ERROR - Allgemeiner Fehler beim Crawlen von https://cheatsheetseries.owasp.org/cheatsheets/Secrets_Management_Cheat_Sheet.html: [Errno 13] Permission denied: '/app'
|
||||||
|
2025-11-12 04:11:44,745 - INFO - Crawle: https://cheatsheetseries.owasp.org/cheatsheets/Secure_AI_Model_Ops_Cheat_Sheet.html
|
||||||
|
2025-11-12 04:11:44,968 - ERROR - Allgemeiner Fehler beim Crawlen von https://cheatsheetseries.owasp.org/cheatsheets/Secure_AI_Model_Ops_Cheat_Sheet.html: [Errno 13] Permission denied: '/app'
|
||||||
|
2025-11-12 04:11:46,973 - INFO - Crawle: https://cheatsheetseries.owasp.org/cheatsheets/Secure_Cloud_Architecture_Cheat_Sheet.html
|
||||||
|
2025-11-12 04:11:47,339 - ERROR - Allgemeiner Fehler beim Crawlen von https://cheatsheetseries.owasp.org/cheatsheets/Secure_Cloud_Architecture_Cheat_Sheet.html: [Errno 13] Permission denied: '/app'
|
||||||
|
2025-11-12 04:11:49,344 - INFO - Crawle: https://cheatsheetseries.owasp.org/cheatsheets/Secure_Code_Review_Cheat_Sheet.html
|
||||||
|
2025-11-12 04:11:49,657 - ERROR - Allgemeiner Fehler beim Crawlen von https://cheatsheetseries.owasp.org/cheatsheets/Secure_Code_Review_Cheat_Sheet.html: [Errno 13] Permission denied: '/app'
|
||||||
|
2025-11-12 04:11:51,661 - INFO - Crawle: https://cheatsheetseries.owasp.org/cheatsheets/Secure_Product_Design_Cheat_Sheet.html
|
||||||
|
2025-11-12 04:11:51,885 - ERROR - Allgemeiner Fehler beim Crawlen von https://cheatsheetseries.owasp.org/cheatsheets/Secure_Product_Design_Cheat_Sheet.html: [Errno 13] Permission denied: '/app'
|
||||||
|
2025-11-12 04:11:53,888 - INFO - Crawle: https://cheatsheetseries.owasp.org/cheatsheets/Securing_Cascading_Style_Sheets_Cheat_Sheet.html
|
||||||
|
2025-11-12 04:11:54,109 - ERROR - Allgemeiner Fehler beim Crawlen von https://cheatsheetseries.owasp.org/cheatsheets/Securing_Cascading_Style_Sheets_Cheat_Sheet.html: [Errno 13] Permission denied: '/app'
|
||||||
|
2025-11-12 04:11:56,112 - INFO - Crawle: https://cheatsheetseries.owasp.org/cheatsheets/Server_Side_Request_Forgery_Prevention_Cheat_Sheet.html
|
||||||
|
2025-11-12 04:11:56,474 - ERROR - Allgemeiner Fehler beim Crawlen von https://cheatsheetseries.owasp.org/cheatsheets/Server_Side_Request_Forgery_Prevention_Cheat_Sheet.html: [Errno 13] Permission denied: '/app'
|
||||||
|
2025-11-12 04:11:58,479 - INFO - Crawle: https://cheatsheetseries.owasp.org/cheatsheets/Serverless_FaaS_Security_Cheat_Sheet.html
|
||||||
|
2025-11-12 04:11:58,754 - ERROR - Allgemeiner Fehler beim Crawlen von https://cheatsheetseries.owasp.org/cheatsheets/Serverless_FaaS_Security_Cheat_Sheet.html: [Errno 13] Permission denied: '/app'
|
||||||
|
2025-11-12 04:12:00,759 - INFO - Crawle: https://cheatsheetseries.owasp.org/cheatsheets/Session_Management_Cheat_Sheet.html
|
||||||
|
2025-11-12 04:12:01,144 - ERROR - Allgemeiner Fehler beim Crawlen von https://cheatsheetseries.owasp.org/cheatsheets/Session_Management_Cheat_Sheet.html: [Errno 13] Permission denied: '/app'
|
||||||
|
2025-11-12 04:12:03,148 - INFO - Crawle: https://cheatsheetseries.owasp.org/cheatsheets/Software_Supply_Chain_Security_Cheat_Sheet.html
|
||||||
|
2025-11-12 04:12:03,434 - ERROR - Allgemeiner Fehler beim Crawlen von https://cheatsheetseries.owasp.org/cheatsheets/Software_Supply_Chain_Security_Cheat_Sheet.html: [Errno 13] Permission denied: '/app'
|
||||||
|
2025-11-12 04:12:05,438 - INFO - Crawle: https://cheatsheetseries.owasp.org/cheatsheets/Symfony_Cheat_Sheet.html
|
||||||
|
2025-11-12 04:12:05,844 - ERROR - Allgemeiner Fehler beim Crawlen von https://cheatsheetseries.owasp.org/cheatsheets/Symfony_Cheat_Sheet.html: [Errno 13] Permission denied: '/app'
|
||||||
|
2025-11-12 04:12:07,847 - INFO - Crawle: https://cheatsheetseries.owasp.org/cheatsheets/TLS_Cipher_String_Cheat_Sheet.html
|
||||||
|
2025-11-12 04:12:08,018 - INFO - Inhalt verworfen (Länge: 227 < 500): https://cheatsheetseries.owasp.org/cheatsheets/TLS_Cipher_String_Cheat_Sheet.html
|
||||||
|
2025-11-12 04:12:10,041 - INFO - Crawle: https://cheatsheetseries.owasp.org/cheatsheets/Third_Party_Javascript_Management_Cheat_Sheet.html
|
||||||
|
2025-11-12 04:12:10,384 - ERROR - Allgemeiner Fehler beim Crawlen von https://cheatsheetseries.owasp.org/cheatsheets/Third_Party_Javascript_Management_Cheat_Sheet.html: [Errno 13] Permission denied: '/app'
|
||||||
|
2025-11-12 04:12:12,394 - INFO - Crawle: https://cheatsheetseries.owasp.org/cheatsheets/Third_Party_Payment_Gateway_Integration.html
|
||||||
|
2025-11-12 04:12:12,597 - ERROR - Allgemeiner Fehler beim Crawlen von https://cheatsheetseries.owasp.org/cheatsheets/Third_Party_Payment_Gateway_Integration.html: [Errno 13] Permission denied: '/app'
|
||||||
|
2025-11-12 04:12:14,601 - INFO - Crawle: https://cheatsheetseries.owasp.org/cheatsheets/Threat_Modeling_Cheat_Sheet.html
|
||||||
|
2025-11-12 04:12:14,858 - ERROR - Allgemeiner Fehler beim Crawlen von https://cheatsheetseries.owasp.org/cheatsheets/Threat_Modeling_Cheat_Sheet.html: [Errno 13] Permission denied: '/app'
|
||||||
|
2025-11-12 04:12:16,861 - INFO - Crawle: https://cheatsheetseries.owasp.org/cheatsheets/Transaction_Authorization_Cheat_Sheet.html
|
||||||
|
2025-11-12 04:12:17,132 - ERROR - Allgemeiner Fehler beim Crawlen von https://cheatsheetseries.owasp.org/cheatsheets/Transaction_Authorization_Cheat_Sheet.html: [Errno 13] Permission denied: '/app'
|
||||||
|
2025-11-12 04:12:19,135 - INFO - Crawle: https://cheatsheetseries.owasp.org/cheatsheets/Transport_Layer_Protection_Cheat_Sheet.html
|
||||||
|
2025-11-12 04:12:19,325 - INFO - Inhalt verworfen (Länge: 254 < 500): https://cheatsheetseries.owasp.org/cheatsheets/Transport_Layer_Protection_Cheat_Sheet.html
|
||||||
|
2025-11-12 04:12:21,351 - INFO - Crawle: https://cheatsheetseries.owasp.org/cheatsheets/Transport_Layer_Security_Cheat_Sheet.html
|
||||||
|
2025-11-12 04:12:21,635 - ERROR - Allgemeiner Fehler beim Crawlen von https://cheatsheetseries.owasp.org/cheatsheets/Transport_Layer_Security_Cheat_Sheet.html: [Errno 13] Permission denied: '/app'
|
||||||
|
2025-11-12 04:12:23,640 - INFO - Crawle: https://cheatsheetseries.owasp.org/cheatsheets/Unvalidated_Redirects_and_Forwards_Cheat_Sheet.html
|
||||||
|
2025-11-12 04:12:23,923 - ERROR - Allgemeiner Fehler beim Crawlen von https://cheatsheetseries.owasp.org/cheatsheets/Unvalidated_Redirects_and_Forwards_Cheat_Sheet.html: [Errno 13] Permission denied: '/app'
|
||||||
|
2025-11-12 04:12:25,926 - INFO - Crawle: https://cheatsheetseries.owasp.org/cheatsheets/User_Privacy_Protection_Cheat_Sheet.html
|
||||||
|
2025-11-12 04:12:26,193 - ERROR - Allgemeiner Fehler beim Crawlen von https://cheatsheetseries.owasp.org/cheatsheets/User_Privacy_Protection_Cheat_Sheet.html: [Errno 13] Permission denied: '/app'
|
||||||
|
2025-11-12 04:12:28,198 - INFO - Crawle: https://cheatsheetseries.owasp.org/cheatsheets/Virtual_Patching_Cheat_Sheet.html
|
||||||
|
2025-11-12 04:12:28,478 - ERROR - Allgemeiner Fehler beim Crawlen von https://cheatsheetseries.owasp.org/cheatsheets/Virtual_Patching_Cheat_Sheet.html: [Errno 13] Permission denied: '/app'
|
||||||
|
2025-11-12 04:12:30,482 - INFO - Crawle: https://cheatsheetseries.owasp.org/cheatsheets/Vulnerability_Disclosure_Cheat_Sheet.html
|
||||||
|
2025-11-12 04:12:30,775 - ERROR - Allgemeiner Fehler beim Crawlen von https://cheatsheetseries.owasp.org/cheatsheets/Vulnerability_Disclosure_Cheat_Sheet.html: [Errno 13] Permission denied: '/app'
|
||||||
|
2025-11-12 04:12:32,779 - INFO - Crawle: https://cheatsheetseries.owasp.org/cheatsheets/Vulnerable_Dependency_Management_Cheat_Sheet.html
|
||||||
|
2025-11-12 04:12:33,055 - ERROR - Allgemeiner Fehler beim Crawlen von https://cheatsheetseries.owasp.org/cheatsheets/Vulnerable_Dependency_Management_Cheat_Sheet.html: [Errno 13] Permission denied: '/app'
|
||||||
|
2025-11-12 04:12:35,060 - INFO - Crawle: https://cheatsheetseries.owasp.org/cheatsheets/WebSocket_Security_Cheat_Sheet.html
|
||||||
|
2025-11-12 04:12:35,379 - ERROR - Allgemeiner Fehler beim Crawlen von https://cheatsheetseries.owasp.org/cheatsheets/WebSocket_Security_Cheat_Sheet.html: [Errno 13] Permission denied: '/app'
|
||||||
|
2025-11-12 04:12:37,384 - INFO - Crawle: https://cheatsheetseries.owasp.org/cheatsheets/Web_Service_Security_Cheat_Sheet.html
|
||||||
|
2025-11-12 04:12:37,619 - ERROR - Allgemeiner Fehler beim Crawlen von https://cheatsheetseries.owasp.org/cheatsheets/Web_Service_Security_Cheat_Sheet.html: [Errno 13] Permission denied: '/app'
|
||||||
|
2025-11-12 04:12:39,624 - INFO - Crawle: https://cheatsheetseries.owasp.org/cheatsheets/XML_External_Entity_Prevention_Cheat_Sheet.html
|
||||||
|
2025-11-12 04:12:40,267 - ERROR - Allgemeiner Fehler beim Crawlen von https://cheatsheetseries.owasp.org/cheatsheets/XML_External_Entity_Prevention_Cheat_Sheet.html: [Errno 13] Permission denied: '/app'
|
||||||
|
2025-11-12 04:12:42,271 - INFO - Crawle: https://cheatsheetseries.owasp.org/cheatsheets/XML_Security_Cheat_Sheet.html
|
||||||
|
2025-11-12 04:12:42,925 - ERROR - Allgemeiner Fehler beim Crawlen von https://cheatsheetseries.owasp.org/cheatsheets/XML_Security_Cheat_Sheet.html: [Errno 13] Permission denied: '/app'
|
||||||
|
2025-11-12 04:12:44,929 - INFO - Crawle: https://cheatsheetseries.owasp.org/cheatsheets/XSS_Filter_Evasion_Cheat_Sheet.html
|
||||||
|
2025-11-12 04:12:45,676 - ERROR - Allgemeiner Fehler beim Crawlen von https://cheatsheetseries.owasp.org/cheatsheets/XSS_Filter_Evasion_Cheat_Sheet.html: [Errno 13] Permission denied: '/app'
|
||||||
|
2025-11-12 04:12:47,680 - INFO - Crawle: https://cheatsheetseries.owasp.org/cheatsheets/XS_Leaks_Cheat_Sheet.html
|
||||||
|
2025-11-12 04:12:48,039 - ERROR - Allgemeiner Fehler beim Crawlen von https://cheatsheetseries.owasp.org/cheatsheets/XS_Leaks_Cheat_Sheet.html: [Errno 13] Permission denied: '/app'
|
||||||
|
2025-11-12 04:12:50,043 - INFO - Crawle: https://cheatsheetseries.owasp.org/cheatsheets/Zero_Trust_Architecture_Cheat_Sheet.html
|
||||||
|
2025-11-12 04:12:50,384 - ERROR - Allgemeiner Fehler beim Crawlen von https://cheatsheetseries.owasp.org/cheatsheets/Zero_Trust_Architecture_Cheat_Sheet.html: [Errno 13] Permission denied: '/app'
|
||||||
|
2025-11-12 04:12:52,388 - INFO - Crawle: https://cheatsheetseries.owasp.org/cheatsheets/gRPC_Security_Cheat_Sheet.html
|
||||||
|
2025-11-12 04:12:52,896 - ERROR - Allgemeiner Fehler beim Crawlen von https://cheatsheetseries.owasp.org/cheatsheets/gRPC_Security_Cheat_Sheet.html: [Errno 13] Permission denied: '/app'
|
||||||
|
2025-11-12 04:12:54,901 - INFO - Crawle: https://cheatsheetseries.owasp.org/index.html
|
||||||
|
2025-11-12 04:12:55,080 - INFO - Inhalt verworfen (Länge: 489 < 500): https://cheatsheetseries.owasp.org/index.html
|
||||||
|
2025-11-12 04:12:57,105 - INFO - Quelle https://cheatsheetseries.owasp.org/ erfolgreich gecrawlt.
|
||||||
|
2025-11-12 04:12:57,111 - INFO - Lösche Zustandsdatei.
|
||||||
|
2025-11-12 04:12:57,114 - INFO - Datenbankverbindung für Hashes geschlossen.
|
||||||
|
2025-11-12 04:12:57,118 - INFO - Finished source: https://cheatsheetseries.owasp.org/
|
||||||
|
2025-11-12 04:12:57,121 - INFO - Processing source: https://nodered.org/docs/
|
||||||
|
2025-11-12 04:12:57,125 - INFO - Starte Quelle: https://nodered.org/docs/ (Modus: domain_wide)
|
||||||
|
2025-11-12 04:12:57,127 - INFO - Crawle: https://nodered.org/docs/
|
||||||
|
2025-11-12 04:12:57,130 - INFO - Lade robots.txt von: https://nodered.org/robots.txt
|
||||||
|
2025-11-12 04:12:57,345 - WARNING - robots.txt verbietet: https://nodered.org/docs/
|
||||||
|
2025-11-12 04:12:59,350 - INFO - Quelle https://nodered.org/docs/ erfolgreich gecrawlt.
|
||||||
|
2025-11-12 04:12:59,354 - INFO - Lösche Zustandsdatei.
|
||||||
|
2025-11-12 04:12:59,357 - INFO - Datenbankverbindung für Hashes geschlossen.
|
||||||
|
2025-11-12 04:12:59,359 - INFO - Finished source: https://nodered.org/docs/
|
||||||
|
2025-11-12 04:12:59,361 - INFO - Processing source: https://www.home-assistant.io/docs/
|
||||||
|
2025-11-12 04:12:59,364 - INFO - Starte Quelle: https://www.home-assistant.io/docs/ (Modus: domain_wide)
|
||||||
|
2025-11-12 04:12:59,367 - INFO - Crawle: https://www.home-assistant.io/docs/
|
||||||
|
2025-11-12 04:12:59,370 - INFO - Lade robots.txt von: https://www.home-assistant.io/robots.txt
|
||||||
|
2025-11-12 04:12:59,775 - ERROR - Allgemeiner Fehler beim Crawlen von https://www.home-assistant.io/docs/: [Errno 13] Permission denied: '/app'
|
||||||
|
2025-11-12 04:13:01,779 - INFO - Quelle https://www.home-assistant.io/docs/ erfolgreich gecrawlt.
|
||||||
|
2025-11-12 04:13:01,783 - INFO - Lösche Zustandsdatei.
|
||||||
|
2025-11-12 04:13:01,787 - INFO - Datenbankverbindung für Hashes geschlossen.
|
||||||
|
2025-11-12 04:13:01,790 - INFO - Finished source: https://www.home-assistant.io/docs/
|
||||||
|
2025-11-12 04:13:01,794 - INFO - All crawling tasks finished.
|
||||||
|
2025-11-12 04:13:01,796 - INFO - --- Gesamte Crawling-Statistik ---
|
||||||
|
2025-11-12 04:13:01,799 - INFO - Gesamtdauer: 540.93 Sekunden
|
||||||
|
2025-11-12 04:13:01,801 - INFO - Besuchte Seiten insgesamt: 166
|
||||||
|
2025-11-12 04:13:01,804 - INFO - Gespeicherte Seiten insgesamt: 0
|
||||||
|
2025-11-12 04:13:01,806 - INFO - Fehler insgesamt: 151
|
||||||
|
2025-11-12 04:13:01,808 - INFO - Gesamtdatenvolumen: 0.00 MB
|
||||||
|
2025-11-12 04:13:01,811 - INFO - ------------------------------------
|
||||||
BIN
crawler_logs.db
Normal file
BIN
crawler_logs.db
Normal file
Binary file not shown.
72
git_sync.sh
Executable file
72
git_sync.sh
Executable file
@@ -0,0 +1,72 @@
|
|||||||
|
#!/bin/bash
|
||||||
|
|
||||||
|
# =====================================================
|
||||||
|
# Git Projekt Management Script
|
||||||
|
# =====================================================
|
||||||
|
# Funktionen:
|
||||||
|
# 1) Erstmaliges Hochladen (Repo initialisieren + push)
|
||||||
|
# 2) Änderungen hochladen (commit + push)
|
||||||
|
# 3) Projekt vom Server herunterladen (clone)
|
||||||
|
# 4) Benutzername für Git-Server speichern
|
||||||
|
# =====================================================
|
||||||
|
|
||||||
|
# === Konfiguration ===
|
||||||
|
REMOTE_URL="http://192.168.19.10:3020/madgerm/web-crawler.git"
|
||||||
|
# Wenn dein Login eine E-Mail ist, benutze %40 statt @ im REMOTE_URL:
|
||||||
|
# Beispiel: http://madgerm%40msn.com@192.168.19.10:3020/madgerm/MiniMal.git
|
||||||
|
|
||||||
|
echo "---------------------------------------------"
|
||||||
|
echo " GIT PROJECT MANAGER"
|
||||||
|
echo "---------------------------------------------"
|
||||||
|
echo "1) Projekt erstmalig hochladen"
|
||||||
|
echo "2) Änderungen hochladen (Standard)"
|
||||||
|
echo "3) Projekt vom Server herunterladen"
|
||||||
|
echo "4) Git-Login (Benutzername) speichern"
|
||||||
|
echo
|
||||||
|
read -p "Wähle eine Option [1-4, Standard=2]: " choice
|
||||||
|
|
||||||
|
# Wenn der Benutzer einfach Enter drückt, wird 2 gesetzt
|
||||||
|
choice=${choice:-2}
|
||||||
|
|
||||||
|
case $choice in
|
||||||
|
1)
|
||||||
|
echo ">>> Projekt wird initialisiert und hochgeladen..."
|
||||||
|
git init
|
||||||
|
git branch -M main
|
||||||
|
git add .
|
||||||
|
git commit -m "Initial commit"
|
||||||
|
git remote add origin "$REMOTE_URL"
|
||||||
|
git push -u origin main
|
||||||
|
;;
|
||||||
|
2)
|
||||||
|
echo ">>> Änderungen werden zum Server hochgeladen..."
|
||||||
|
git add .
|
||||||
|
git commit -m "Auto sync: $(date '+%Y-%m-%d %H:%M:%S')"
|
||||||
|
git push origin main
|
||||||
|
;;
|
||||||
|
3)
|
||||||
|
echo ">>> Projekt wird vom Server heruntergeladen..."
|
||||||
|
read -p "Zielordner (Standard: aktuelles Verzeichnis): " TARGET_DIR
|
||||||
|
TARGET_DIR=${TARGET_DIR:-.}
|
||||||
|
git clone "$REMOTE_URL" "$TARGET_DIR"
|
||||||
|
;;
|
||||||
|
4)
|
||||||
|
echo ">>> Git-Benutzernamen speichern..."
|
||||||
|
read -p "Gib deinen Git-Login (z.B. madgerm@msn.com) ein: " GITUSER
|
||||||
|
if [ -n "$GITUSER" ]; then
|
||||||
|
git config --global user.name "$GITUSER"
|
||||||
|
git config --global credential.username "$GITUSER"
|
||||||
|
echo "✅ Benutzername gespeichert:"
|
||||||
|
echo " user.name = $(git config --global user.name)"
|
||||||
|
echo " credential.username = $(git config --global credential.username)"
|
||||||
|
echo
|
||||||
|
echo "💡 Hinweis: Wenn du dein Passwort speichern möchtest:"
|
||||||
|
echo " git config --global credential.helper store"
|
||||||
|
else
|
||||||
|
echo "⚠️ Kein Benutzername eingegeben. Vorgang abgebrochen."
|
||||||
|
fi
|
||||||
|
;;
|
||||||
|
*)
|
||||||
|
echo "Ungültige Eingabe. Abbruch."
|
||||||
|
;;
|
||||||
|
esac
|
||||||
BIN
src/__pycache__/crawler_core.cpython-312.pyc
Normal file
BIN
src/__pycache__/crawler_core.cpython-312.pyc
Normal file
Binary file not shown.
BIN
src/__pycache__/db_logger.cpython-312.pyc
Normal file
BIN
src/__pycache__/db_logger.cpython-312.pyc
Normal file
Binary file not shown.
BIN
src/__pycache__/duplicate_detector.cpython-312.pyc
Normal file
BIN
src/__pycache__/duplicate_detector.cpython-312.pyc
Normal file
Binary file not shown.
BIN
src/__pycache__/hash_manager.cpython-312.pyc
Normal file
BIN
src/__pycache__/hash_manager.cpython-312.pyc
Normal file
Binary file not shown.
BIN
src/__pycache__/html_cleaner.cpython-312.pyc
Normal file
BIN
src/__pycache__/html_cleaner.cpython-312.pyc
Normal file
Binary file not shown.
BIN
src/__pycache__/logger_setup.cpython-312.pyc
Normal file
BIN
src/__pycache__/logger_setup.cpython-312.pyc
Normal file
Binary file not shown.
BIN
src/__pycache__/state_manager.cpython-312.pyc
Normal file
BIN
src/__pycache__/state_manager.cpython-312.pyc
Normal file
Binary file not shown.
BIN
src/__pycache__/stats_manager.cpython-312.pyc
Normal file
BIN
src/__pycache__/stats_manager.cpython-312.pyc
Normal file
Binary file not shown.
BIN
src/__pycache__/storage.cpython-312.pyc
Normal file
BIN
src/__pycache__/storage.cpython-312.pyc
Normal file
Binary file not shown.
BIN
src/__pycache__/url_utils.cpython-312.pyc
Normal file
BIN
src/__pycache__/url_utils.cpython-312.pyc
Normal file
Binary file not shown.
277
src/crawler_core.py
Normal file
277
src/crawler_core.py
Normal file
@@ -0,0 +1,277 @@
|
|||||||
|
import trafilatura, asyncio
|
||||||
|
from asyncio import PriorityQueue
|
||||||
|
import aiohttp
|
||||||
|
import logging
|
||||||
|
from urllib.parse import urljoin, urlparse
|
||||||
|
from urllib import robotparser
|
||||||
|
from bs4 import BeautifulSoup
|
||||||
|
|
||||||
|
from .url_utils import normalize_url
|
||||||
|
from .storage import save
|
||||||
|
from .html_cleaner import clean_html
|
||||||
|
from .duplicate_detector import DuplicateDetector
|
||||||
|
from .hash_manager import HashManager
|
||||||
|
from .state_manager import StateManager
|
||||||
|
from .stats_manager import CrawlStats
|
||||||
|
|
||||||
|
try:
|
||||||
|
from simhash import Simhash
|
||||||
|
except ImportError:
|
||||||
|
Simhash = None
|
||||||
|
|
||||||
|
|
||||||
|
class RobotsTxtChecker:
|
||||||
|
def __init__(self, user_agent):
|
||||||
|
self.user_agent = user_agent
|
||||||
|
self.parsers = {} # Cache: netloc -> robotparser.RobotFileParser
|
||||||
|
|
||||||
|
async def _get_parser(self, session, url):
|
||||||
|
parsed_url = urlparse(url)
|
||||||
|
netloc = parsed_url.netloc
|
||||||
|
|
||||||
|
if netloc not in self.parsers:
|
||||||
|
robots_url = urljoin(url, "/robots.txt")
|
||||||
|
rp = robotparser.RobotFileParser()
|
||||||
|
rp.set_url(robots_url)
|
||||||
|
|
||||||
|
logging.info(f"Lade robots.txt von: {robots_url}")
|
||||||
|
|
||||||
|
try:
|
||||||
|
# Asynchroner Abruf der robots.txt mit aiohttp
|
||||||
|
async with session.get(robots_url, timeout=aiohttp.ClientTimeout(total=10)) as response:
|
||||||
|
if response.status == 200:
|
||||||
|
text = await response.text()
|
||||||
|
rp.parse(text.splitlines())
|
||||||
|
elif response.status == 404:
|
||||||
|
# Keine robots.txt gefunden, alles erlaubt
|
||||||
|
pass
|
||||||
|
else:
|
||||||
|
logging.warning(f"Fehler beim Abruf von {robots_url}: Status {response.status}")
|
||||||
|
except aiohttp.ClientError as e:
|
||||||
|
logging.error(f"Fehler beim Abruf von {robots_url}: {e}")
|
||||||
|
# Bei Fehler oder Timeout wird der Parser ohne Regeln gecacht (implizit alles erlaubt)
|
||||||
|
|
||||||
|
self.parsers[netloc] = rp
|
||||||
|
|
||||||
|
return self.parsers[netloc]
|
||||||
|
|
||||||
|
async def is_allowed(self, session, url):
|
||||||
|
rp = await self._get_parser(session, url)
|
||||||
|
return rp.can_fetch(self.user_agent, url)
|
||||||
|
|
||||||
|
# Instanziierung des Checkers
|
||||||
|
robots_checker = None
|
||||||
|
|
||||||
|
|
||||||
|
def extract_text(html):
|
||||||
|
text = trafilatura.extract(html)
|
||||||
|
|
||||||
|
# fallback bei Gesetzen / technischen Dokus
|
||||||
|
if not text or len(text) < 400:
|
||||||
|
soup = BeautifulSoup(html, "lxml")
|
||||||
|
text = soup.get_text(" ", strip=True)
|
||||||
|
|
||||||
|
if not text or len(text) < 100:
|
||||||
|
return None
|
||||||
|
|
||||||
|
return text
|
||||||
|
|
||||||
|
|
||||||
|
def extract_title(html):
|
||||||
|
"""Extrahiert den Titel aus dem HTML-Inhalt."""
|
||||||
|
soup = BeautifulSoup(html, "lxml")
|
||||||
|
title_tag = soup.find("title")
|
||||||
|
if title_tag:
|
||||||
|
return title_tag.string.strip()
|
||||||
|
return "Kein Titel gefunden"
|
||||||
|
|
||||||
|
|
||||||
|
def allowed_link(base_url_normalized, base_path, next_url, crawl_mode, blocked_patterns, path_strict):
|
||||||
|
# Normalisiere die URL für Deduplizierung und Fragment-/Tracking-Entfernung
|
||||||
|
normalized_url = normalize_url(next_url)
|
||||||
|
|
||||||
|
if any(pat in normalized_url for pat in blocked_patterns):
|
||||||
|
return False
|
||||||
|
|
||||||
|
if crawl_mode == "single_page":
|
||||||
|
return False
|
||||||
|
|
||||||
|
# Domain-Limit
|
||||||
|
# base_url_normalized ist die normalisierte Start-URL der Quelle
|
||||||
|
if urlparse(normalized_url).netloc != urlparse(base_url_normalized).netloc:
|
||||||
|
return False
|
||||||
|
|
||||||
|
# Pfadbegrenzung
|
||||||
|
if crawl_mode == "path_limited":
|
||||||
|
# Strikte Pfadbegrenzung: Der Pfad der gefundenen URL muss mit dem Basis-Pfad beginnen.
|
||||||
|
# Wir verwenden urlparse, um den Pfad der gefundenen URL zu erhalten.
|
||||||
|
next_path = urlparse(normalized_url).path
|
||||||
|
if not next_path.startswith(base_path):
|
||||||
|
return False
|
||||||
|
|
||||||
|
# Legacy path_strict (falls noch verwendet, obwohl path_limited dies ersetzen sollte)
|
||||||
|
if path_strict and not normalized_url.startswith(base_url_normalized):
|
||||||
|
return False
|
||||||
|
|
||||||
|
return normalized_url # Gibt die normalisierte URL zurück, wenn erlaubt
|
||||||
|
|
||||||
|
|
||||||
|
async def crawl_source(session, source, config):
|
||||||
|
stats = CrawlStats()
|
||||||
|
global robots_checker
|
||||||
|
if not robots_checker:
|
||||||
|
user_agent = config.get("USER_AGENT", {}).get("value")
|
||||||
|
robots_checker = RobotsTxtChecker(user_agent)
|
||||||
|
|
||||||
|
start_urls = source["start_urls"]
|
||||||
|
crawl_mode = source.get("crawl_mode", "domain_wide")
|
||||||
|
blocked_patterns = source.get("blocked_patterns", [])
|
||||||
|
path_strict = source.get("path_strict", False)
|
||||||
|
|
||||||
|
# Lade Konfigurationswerte
|
||||||
|
page_limit = config.get("PAGE_LIMIT", {}).get("value", 200)
|
||||||
|
crawl_delay = config.get("CRAWL_DELAY", {}).get("value", 1)
|
||||||
|
min_content_length = config.get("MIN_CONTENT_LENGTH", {}).get("value", 500)
|
||||||
|
priority_patterns = config.get("priority_patterns", {}).get("value", [])
|
||||||
|
duplicate_detection_config = config.get("duplicate_detection", {}).get("value", {})
|
||||||
|
incremental_crawling_config = config.get("incremental_crawling", {}).get("value", {})
|
||||||
|
state_management_config = config.get("state_management", {}).get("value", {})
|
||||||
|
output_dir = config.get("OUTPUT_DIR", {}).get("value")
|
||||||
|
|
||||||
|
# Initialisiere DuplicateDetector, falls aktiviert
|
||||||
|
duplicate_detector = None
|
||||||
|
if duplicate_detection_config.get("enable"):
|
||||||
|
duplicate_detector = DuplicateDetector(
|
||||||
|
similarity_threshold=duplicate_detection_config.get("similarity_threshold", 95)
|
||||||
|
)
|
||||||
|
|
||||||
|
# Initialisiere HashManager, falls aktiviert
|
||||||
|
hash_manager = None
|
||||||
|
if incremental_crawling_config.get("enable"):
|
||||||
|
db_file = incremental_crawling_config.get("db_file", "crawled_hashes.db")
|
||||||
|
hash_manager = HashManager(db_file)
|
||||||
|
|
||||||
|
# --- Zustandsverwaltung ---
|
||||||
|
state_manager = None
|
||||||
|
queue = PriorityQueue()
|
||||||
|
visited = set()
|
||||||
|
normalized_start_urls = [normalize_url(u) for u in start_urls]
|
||||||
|
base_url_normalized = normalized_start_urls[0]
|
||||||
|
|
||||||
|
if state_management_config.get("enable"):
|
||||||
|
state_file_template = state_management_config.get("state_file", "crawler_state.json")
|
||||||
|
source_netloc = urlparse(base_url_normalized).netloc
|
||||||
|
state_file = state_file_template.replace(".json", f"_{source_netloc}.json")
|
||||||
|
state_manager = StateManager(state_file)
|
||||||
|
|
||||||
|
loaded_state = state_manager.load_state()
|
||||||
|
if loaded_state:
|
||||||
|
logging.info(f"Lade Zustand aus {state_file}")
|
||||||
|
queue, visited = loaded_state
|
||||||
|
else:
|
||||||
|
for u in normalized_start_urls:
|
||||||
|
await queue.put((0, u))
|
||||||
|
else:
|
||||||
|
for u in normalized_start_urls:
|
||||||
|
await queue.put((0, u))
|
||||||
|
# -------------------------
|
||||||
|
|
||||||
|
# Extrahiere den Basis-Pfad für path_limited Modus
|
||||||
|
base_path = urlparse(base_url_normalized).path
|
||||||
|
|
||||||
|
logging.info(f"Starte Quelle: {start_urls[0]} (Modus: {crawl_mode})")
|
||||||
|
|
||||||
|
try:
|
||||||
|
while not queue.empty() and len(visited) < page_limit:
|
||||||
|
priority, url = await queue.get() # Entpacke Priorität und URL
|
||||||
|
|
||||||
|
if url in visited:
|
||||||
|
continue
|
||||||
|
visited.add(url)
|
||||||
|
stats.total_visited += 1
|
||||||
|
|
||||||
|
logging.info(f"Crawle: {url}")
|
||||||
|
|
||||||
|
# Asynchrone robots.txt Prüfung
|
||||||
|
if not await robots_checker.is_allowed(session, url):
|
||||||
|
logging.warning(f"robots.txt verbietet: {url}")
|
||||||
|
await asyncio.sleep(crawl_delay) # Respektiere den Delay, auch wenn übersprungen wird
|
||||||
|
continue
|
||||||
|
|
||||||
|
try:
|
||||||
|
# Asynchroner Abruf mit aiohttp
|
||||||
|
async with session.get(url, timeout=aiohttp.ClientTimeout(total=20)) as response:
|
||||||
|
response.raise_for_status() # Wirft HTTPStatusError für 4xx/5xx
|
||||||
|
html = await response.text()
|
||||||
|
|
||||||
|
cleaned_html = clean_html(html, config)
|
||||||
|
text = extract_text(cleaned_html)
|
||||||
|
|
||||||
|
if text and len(text) >= min_content_length:
|
||||||
|
# Simhash-Berechnung, falls für inkrementelles Crawling benötigt
|
||||||
|
new_simhash_value = None
|
||||||
|
if hash_manager and Simhash:
|
||||||
|
new_simhash_value = Simhash(text).value
|
||||||
|
|
||||||
|
# Inkrementelles Crawling: Prüfen, ob der Inhalt unverändert ist
|
||||||
|
if hash_manager and new_simhash_value is not None:
|
||||||
|
if hash_manager.is_unchanged(url, new_simhash_value):
|
||||||
|
logging.info(f"Inhalt unverändert, übersprungen: {url}")
|
||||||
|
continue
|
||||||
|
|
||||||
|
# Duplikat-Erkennung
|
||||||
|
if duplicate_detector and duplicate_detector.is_duplicate(text):
|
||||||
|
logging.info(f"Duplikat übersprungen: {url}")
|
||||||
|
else:
|
||||||
|
# Speichern und Hashes aktualisieren
|
||||||
|
title = extract_title(cleaned_html)
|
||||||
|
save(url, title, text, output_dir)
|
||||||
|
stats.total_saved += 1
|
||||||
|
stats.total_data_volume += len(text.encode('utf-8'))
|
||||||
|
if duplicate_detector:
|
||||||
|
duplicate_detector.add_hash(text)
|
||||||
|
if hash_manager and new_simhash_value is not None:
|
||||||
|
hash_manager.update_hash(url, new_simhash_value)
|
||||||
|
else:
|
||||||
|
logging.info(f"Inhalt verworfen (Länge: {len(text) if text else 0} < {min_content_length}): {url}")
|
||||||
|
|
||||||
|
if crawl_mode != "single_page":
|
||||||
|
soup = BeautifulSoup(html, "lxml")
|
||||||
|
for link in soup.find_all("a", href=True):
|
||||||
|
next_url = urljoin(url, link["href"])
|
||||||
|
|
||||||
|
# allowed_link gibt False oder die normalisierte URL zurück
|
||||||
|
normalized_next_url = allowed_link(base_url_normalized, base_path, next_url, crawl_mode, blocked_patterns, path_strict)
|
||||||
|
|
||||||
|
if normalized_next_url and normalized_next_url not in visited:
|
||||||
|
# Priorität basierend auf Mustern bestimmen
|
||||||
|
prio = 0 if any(pat in normalized_next_url for pat in priority_patterns) else 1
|
||||||
|
await queue.put((prio, normalized_next_url))
|
||||||
|
|
||||||
|
except aiohttp.ClientError as e:
|
||||||
|
logging.error(f"aiohttp Fehler beim Abruf von {url}: {e}")
|
||||||
|
stats.errors += 1
|
||||||
|
except Exception as e:
|
||||||
|
logging.error(f"Allgemeiner Fehler beim Crawlen von {url}: {e}")
|
||||||
|
stats.errors += 1
|
||||||
|
|
||||||
|
await asyncio.sleep(crawl_delay)
|
||||||
|
|
||||||
|
except (KeyboardInterrupt, Exception):
|
||||||
|
if state_manager:
|
||||||
|
logging.warning("Prozess unterbrochen. Speichere aktuellen Zustand...")
|
||||||
|
state_manager.save_state(queue, visited)
|
||||||
|
logging.info("Zustand gespeichert.")
|
||||||
|
raise
|
||||||
|
else:
|
||||||
|
# This block executes when the try block completes without an exception.
|
||||||
|
logging.info(f"Quelle {base_url_normalized} erfolgreich gecrawlt.")
|
||||||
|
if state_manager:
|
||||||
|
logging.info("Lösche Zustandsdatei.")
|
||||||
|
state_manager.delete_state()
|
||||||
|
# Schließe die Datenbankverbindung des HashManagers
|
||||||
|
if hash_manager:
|
||||||
|
hash_manager.close()
|
||||||
|
|
||||||
|
stats.finish()
|
||||||
|
return stats
|
||||||
169
src/db_logger.py
Normal file
169
src/db_logger.py
Normal file
@@ -0,0 +1,169 @@
|
|||||||
|
import logging
|
||||||
|
import sqlite3
|
||||||
|
import warnings
|
||||||
|
|
||||||
|
# Platzhalter für mysql.connector, um Importfehler zu vermeiden, wenn es nicht installiert ist
|
||||||
|
try:
|
||||||
|
import mysql.connector
|
||||||
|
MYSQL_AVAILABLE = True
|
||||||
|
except ImportError:
|
||||||
|
MYSQL_AVAILABLE = False
|
||||||
|
warnings.warn("Das 'mysql-connector-python' Paket ist nicht installiert. Der MySQLHandler ist nicht verfügbar.")
|
||||||
|
|
||||||
|
class SQLiteHandler(logging.Handler):
|
||||||
|
"""Ein logging.Handler, der Log-Einträge in eine SQLite-Datenbank schreibt."""
|
||||||
|
|
||||||
|
def __init__(self, db_file):
|
||||||
|
super().__init__()
|
||||||
|
self.db_file = db_file
|
||||||
|
self._conn = None
|
||||||
|
self._cursor = None
|
||||||
|
self._db_initialized = False
|
||||||
|
|
||||||
|
def _init_db(self):
|
||||||
|
"""Initialisiert die Datenbankverbindung und erstellt die Tabelle, falls sie nicht existiert."""
|
||||||
|
try:
|
||||||
|
self._conn = sqlite3.connect(self.db_file)
|
||||||
|
self._cursor = self._conn.cursor()
|
||||||
|
self._cursor.execute('''
|
||||||
|
CREATE TABLE IF NOT EXISTS logs (
|
||||||
|
timestamp DATETIME DEFAULT CURRENT_TIMESTAMP,
|
||||||
|
level VARCHAR(10),
|
||||||
|
message TEXT
|
||||||
|
)
|
||||||
|
''')
|
||||||
|
self._conn.commit()
|
||||||
|
self._db_initialized = True
|
||||||
|
except sqlite3.Error as e:
|
||||||
|
warnings.warn(f"Fehler beim Initialisieren der SQLite-Datenbank: {e}")
|
||||||
|
self._conn = None
|
||||||
|
|
||||||
|
def emit(self, record):
|
||||||
|
"""Schreibt einen Log-Eintrag in die Datenbank."""
|
||||||
|
if not self._db_initialized:
|
||||||
|
self._init_db()
|
||||||
|
|
||||||
|
if not self._conn:
|
||||||
|
return
|
||||||
|
|
||||||
|
try:
|
||||||
|
log_entry = (record.levelname, self.format(record))
|
||||||
|
self._cursor.execute('INSERT INTO logs (level, message) VALUES (?, ?)', log_entry)
|
||||||
|
self._conn.commit()
|
||||||
|
except sqlite3.Error as e:
|
||||||
|
warnings.warn(f"Fehler beim Schreiben des Logs in die SQLite-Datenbank: {e}")
|
||||||
|
|
||||||
|
def close(self):
|
||||||
|
"""Schließt die Datenbankverbindung."""
|
||||||
|
if self._conn:
|
||||||
|
self._conn.close()
|
||||||
|
super().close()
|
||||||
|
|
||||||
|
class MySQLHandler(logging.Handler):
|
||||||
|
"""Ein logging.Handler, der Log-Einträge in eine MySQL-Datenbank schreibt."""
|
||||||
|
|
||||||
|
def __init__(self, host, user, password, database):
|
||||||
|
super().__init__()
|
||||||
|
if not MYSQL_AVAILABLE:
|
||||||
|
self._conn = None
|
||||||
|
return
|
||||||
|
|
||||||
|
self.db_config = {
|
||||||
|
'host': host,
|
||||||
|
'user': user,
|
||||||
|
'password': password,
|
||||||
|
'database': database
|
||||||
|
}
|
||||||
|
self._conn = None
|
||||||
|
self._cursor = None
|
||||||
|
self._db_initialized = False
|
||||||
|
|
||||||
|
def _init_db(self):
|
||||||
|
"""Initialisiert die Datenbankverbindung und erstellt die Tabelle, falls sie nicht existiert."""
|
||||||
|
try:
|
||||||
|
self._conn = mysql.connector.connect(**self.db_config)
|
||||||
|
self._cursor = self._conn.cursor()
|
||||||
|
self._cursor.execute('''
|
||||||
|
CREATE TABLE IF NOT EXISTS logs (
|
||||||
|
id INT AUTO_INCREMENT PRIMARY KEY,
|
||||||
|
timestamp DATETIME DEFAULT CURRENT_TIMESTAMP,
|
||||||
|
level VARCHAR(10),
|
||||||
|
message TEXT
|
||||||
|
)
|
||||||
|
''')
|
||||||
|
self._conn.commit()
|
||||||
|
self._db_initialized = True
|
||||||
|
except mysql.connector.Error as e:
|
||||||
|
warnings.warn(f"Fehler beim Initialisieren der MySQL-Datenbank: {e}")
|
||||||
|
self._conn = None
|
||||||
|
|
||||||
|
def emit(self, record):
|
||||||
|
"""Schreibt einen Log-Eintrag in die Datenbank."""
|
||||||
|
if not MYSQL_AVAILABLE or (not self._db_initialized and not self._conn):
|
||||||
|
self._init_db()
|
||||||
|
|
||||||
|
if not self._conn:
|
||||||
|
return
|
||||||
|
|
||||||
|
try:
|
||||||
|
log_entry = (record.levelname, self.format(record))
|
||||||
|
self._cursor.execute('INSERT INTO logs (level, message) VALUES (%s, %s)', log_entry)
|
||||||
|
self._conn.commit()
|
||||||
|
except mysql.connector.Error as e:
|
||||||
|
warnings.warn(f"Fehler beim Schreiben des Logs in die MySQL-Datenbank: {e}")
|
||||||
|
|
||||||
|
def close(self):
|
||||||
|
"""Schließt die Datenbankverbindung."""
|
||||||
|
if self._conn:
|
||||||
|
self._conn.close()
|
||||||
|
super().close()
|
||||||
|
|
||||||
|
def log_stats_to_mysql(stats, source_url, db_config):
|
||||||
|
"""Schreibt Crawl-Statistiken in eine separate MySQL-Tabelle."""
|
||||||
|
if not MYSQL_AVAILABLE:
|
||||||
|
warnings.warn("MySQL-Connector nicht verfügbar. Statistiken können nicht geloggt werden.")
|
||||||
|
return
|
||||||
|
|
||||||
|
conn = None
|
||||||
|
try:
|
||||||
|
conn = mysql.connector.connect(**db_config)
|
||||||
|
cursor = conn.cursor()
|
||||||
|
|
||||||
|
# Tabelle erstellen, falls sie nicht existiert
|
||||||
|
cursor.execute('''
|
||||||
|
CREATE TABLE IF NOT EXISTS crawl_stats (
|
||||||
|
id INT AUTO_INCREMENT PRIMARY KEY,
|
||||||
|
source_url VARCHAR(255),
|
||||||
|
duration_seconds FLOAT,
|
||||||
|
total_visited INT,
|
||||||
|
total_saved INT,
|
||||||
|
total_data_volume_bytes BIGINT,
|
||||||
|
errors INT,
|
||||||
|
timestamp DATETIME DEFAULT CURRENT_TIMESTAMP
|
||||||
|
)
|
||||||
|
''')
|
||||||
|
|
||||||
|
# Statistiken einfügen
|
||||||
|
query = (
|
||||||
|
"INSERT INTO crawl_stats (source_url, duration_seconds, total_visited, "
|
||||||
|
"total_saved, total_data_volume_bytes, errors) "
|
||||||
|
"VALUES (%s, %s, %s, %s, %s, %s)"
|
||||||
|
)
|
||||||
|
values = (
|
||||||
|
source_url,
|
||||||
|
stats.duration,
|
||||||
|
stats.total_visited,
|
||||||
|
stats.total_saved,
|
||||||
|
stats.total_data_volume,
|
||||||
|
stats.errors
|
||||||
|
)
|
||||||
|
cursor.execute(query, values)
|
||||||
|
conn.commit()
|
||||||
|
logging.info(f"Statistiken für {source_url} erfolgreich in MySQL geloggt.")
|
||||||
|
|
||||||
|
except mysql.connector.Error as e:
|
||||||
|
warnings.warn(f"Fehler beim Schreiben der Statistiken in die MySQL-Datenbank: {e}")
|
||||||
|
finally:
|
||||||
|
if conn and conn.is_connected():
|
||||||
|
cursor.close()
|
||||||
|
conn.close()
|
||||||
62
src/duplicate_detector.py
Normal file
62
src/duplicate_detector.py
Normal file
@@ -0,0 +1,62 @@
|
|||||||
|
import logging
|
||||||
|
|
||||||
|
try:
|
||||||
|
from simhash import Simhash
|
||||||
|
except ImportError:
|
||||||
|
Simhash = None
|
||||||
|
logging.warning("Die 'simhash'-Bibliothek wurde nicht gefunden. Die Duplikat-Erkennung ist deaktiviert.")
|
||||||
|
|
||||||
|
class DuplicateDetector:
|
||||||
|
"""
|
||||||
|
Erkennt und verwaltet Textduplikate mittels SimHash.
|
||||||
|
"""
|
||||||
|
def __init__(self, similarity_threshold=95):
|
||||||
|
"""
|
||||||
|
Initialisiert den DuplicateDetector.
|
||||||
|
|
||||||
|
Args:
|
||||||
|
similarity_threshold (int): Der prozentuale Schwellenwert für die Ähnlichkeit,
|
||||||
|
ab dem zwei Texte als Duplikate gelten.
|
||||||
|
"""
|
||||||
|
if Simhash is None:
|
||||||
|
self.enabled = False
|
||||||
|
return
|
||||||
|
|
||||||
|
self.enabled = True
|
||||||
|
self.hashes = set()
|
||||||
|
self.similarity_threshold = similarity_threshold
|
||||||
|
|
||||||
|
def is_duplicate(self, text):
|
||||||
|
"""
|
||||||
|
Überprüft, ob ein Text ein Duplikat eines bereits gesehenen Textes ist.
|
||||||
|
|
||||||
|
Args:
|
||||||
|
text (str): Der zu überprüfende Text.
|
||||||
|
|
||||||
|
Returns:
|
||||||
|
bool: True, wenn der Text ein Duplikat ist, andernfalls False.
|
||||||
|
"""
|
||||||
|
if not self.enabled:
|
||||||
|
return False
|
||||||
|
|
||||||
|
new_hash = Simhash(text)
|
||||||
|
|
||||||
|
for existing_hash in self.hashes:
|
||||||
|
similarity = (1 - (new_hash.distance(existing_hash) / 64.0)) * 100
|
||||||
|
if similarity >= self.similarity_threshold:
|
||||||
|
logging.info(f"Duplikat gefunden mit einer Ähnlichkeit von {similarity:.2f}%.")
|
||||||
|
return True
|
||||||
|
return False
|
||||||
|
|
||||||
|
def add_hash(self, text):
|
||||||
|
"""
|
||||||
|
Fügt den SimHash des Textes zur Menge der bekannten Hashes hinzu.
|
||||||
|
|
||||||
|
Args:
|
||||||
|
text (str): Der Text, dessen Hash hinzugefügt werden soll.
|
||||||
|
"""
|
||||||
|
if not self.enabled:
|
||||||
|
return
|
||||||
|
|
||||||
|
new_hash = Simhash(text)
|
||||||
|
self.hashes.add(new_hash)
|
||||||
88
src/hash_manager.py
Normal file
88
src/hash_manager.py
Normal file
@@ -0,0 +1,88 @@
|
|||||||
|
import sqlite3
|
||||||
|
import logging
|
||||||
|
from datetime import datetime
|
||||||
|
|
||||||
|
logger = logging.getLogger(__name__)
|
||||||
|
|
||||||
|
class HashManager:
|
||||||
|
"""Verwaltet die Speicherung und das Laden von Hashes für das inkrementelle Crawling via SQLite."""
|
||||||
|
|
||||||
|
def __init__(self, db_path):
|
||||||
|
"""
|
||||||
|
Initialisiert den HashManager und die Datenbankverbindung.
|
||||||
|
|
||||||
|
Args:
|
||||||
|
db_path (str): Der Pfad zur SQLite-Datenbankdatei.
|
||||||
|
"""
|
||||||
|
self.db_path = db_path
|
||||||
|
self.conn = None
|
||||||
|
try:
|
||||||
|
self.conn = sqlite3.connect(self.db_path)
|
||||||
|
self._init_db()
|
||||||
|
except sqlite3.Error as e:
|
||||||
|
logger.error(f"Datenbankfehler beim Verbinden mit {self.db_path}: {e}")
|
||||||
|
raise
|
||||||
|
|
||||||
|
def _init_db(self):
|
||||||
|
"""Erstellt die 'hashes'-Tabelle, falls sie nicht existiert."""
|
||||||
|
try:
|
||||||
|
with self.conn:
|
||||||
|
self.conn.execute("""
|
||||||
|
CREATE TABLE IF NOT EXISTS hashes (
|
||||||
|
url TEXT PRIMARY KEY,
|
||||||
|
simhash TEXT NOT NULL,
|
||||||
|
timestamp TEXT NOT NULL
|
||||||
|
)
|
||||||
|
""")
|
||||||
|
except sqlite3.Error as e:
|
||||||
|
logger.error(f"Fehler beim Initialisieren der Datenbanktabelle: {e}")
|
||||||
|
|
||||||
|
def is_unchanged(self, url, new_simhash):
|
||||||
|
"""
|
||||||
|
Prüft, ob sich der Inhalt einer URL geändert hat, indem der SimHash in der DB verglichen wird.
|
||||||
|
|
||||||
|
Args:
|
||||||
|
url (str): Die zu prüfende URL.
|
||||||
|
new_simhash (int): Der neue SimHash des Inhalts.
|
||||||
|
|
||||||
|
Returns:
|
||||||
|
bool: True, wenn die URL bekannt und der SimHash unverändert ist, sonst False.
|
||||||
|
"""
|
||||||
|
if not self.conn:
|
||||||
|
return False
|
||||||
|
try:
|
||||||
|
cursor = self.conn.cursor()
|
||||||
|
cursor.execute("SELECT simhash FROM hashes WHERE url = ?", (url,))
|
||||||
|
result = cursor.fetchone()
|
||||||
|
if result and result[0] == str(new_simhash):
|
||||||
|
return True
|
||||||
|
except sqlite3.Error as e:
|
||||||
|
logger.error(f"Fehler beim Prüfen des Hashes für URL {url}: {e}")
|
||||||
|
return False
|
||||||
|
|
||||||
|
def update_hash(self, url, simhash):
|
||||||
|
"""
|
||||||
|
Aktualisiert den Hash und den Zeitstempel für eine gegebene URL in der Datenbank.
|
||||||
|
|
||||||
|
Args:
|
||||||
|
url (str): Die URL, deren Hash aktualisiert wird.
|
||||||
|
simhash (int): Der neue SimHash des Inhalts.
|
||||||
|
"""
|
||||||
|
if not self.conn:
|
||||||
|
return
|
||||||
|
timestamp = datetime.utcnow().isoformat()
|
||||||
|
try:
|
||||||
|
with self.conn:
|
||||||
|
self.conn.execute("""
|
||||||
|
INSERT OR REPLACE INTO hashes (url, simhash, timestamp)
|
||||||
|
VALUES (?, ?, ?)
|
||||||
|
""", (url, str(simhash), timestamp))
|
||||||
|
logger.debug(f"Hash für URL {url} in der Datenbank aktualisiert.")
|
||||||
|
except sqlite3.Error as e:
|
||||||
|
logger.error(f"Fehler beim Aktualisieren des Hashes für URL {url}: {e}")
|
||||||
|
|
||||||
|
def close(self):
|
||||||
|
"""Schließt die Datenbankverbindung."""
|
||||||
|
if self.conn:
|
||||||
|
self.conn.close()
|
||||||
|
logger.info("Datenbankverbindung für Hashes geschlossen.")
|
||||||
40
src/html_cleaner.py
Normal file
40
src/html_cleaner.py
Normal file
@@ -0,0 +1,40 @@
|
|||||||
|
import re
|
||||||
|
from bs4 import BeautifulSoup, Comment
|
||||||
|
|
||||||
|
def clean_html(html_content: str, config: dict) -> str:
|
||||||
|
"""
|
||||||
|
Bereinigt den HTML-Inhalt, indem unerwünschte Tags und Textmuster entfernt werden.
|
||||||
|
|
||||||
|
Args:
|
||||||
|
html_content: Der rohe HTML-Inhalt als String.
|
||||||
|
config: Das Konfigurationsobjekt.
|
||||||
|
|
||||||
|
Returns:
|
||||||
|
Der bereinigte HTML-Inhalt als String.
|
||||||
|
"""
|
||||||
|
cleaner_config = config.get('html_cleaner', {}).get('value', {})
|
||||||
|
remove_tags = cleaner_config.get('remove_tags', [])
|
||||||
|
remove_patterns = cleaner_config.get('remove_patterns', [])
|
||||||
|
|
||||||
|
if not html_content:
|
||||||
|
return ""
|
||||||
|
|
||||||
|
soup = BeautifulSoup(html_content, 'html.parser')
|
||||||
|
|
||||||
|
# 1. Entferne unerwünschte Tags
|
||||||
|
for tag_name in remove_tags:
|
||||||
|
for tag in soup.find_all(tag_name):
|
||||||
|
tag.decompose()
|
||||||
|
|
||||||
|
# 2. Entferne unerwünschte Textmuster aus dem verbleibenden Inhalt
|
||||||
|
if remove_patterns:
|
||||||
|
combined_pattern = "|".join(remove_patterns)
|
||||||
|
|
||||||
|
for element in soup.find_all(string=True):
|
||||||
|
if element.parent.name in ['style', 'script'] or isinstance(element, Comment):
|
||||||
|
continue
|
||||||
|
|
||||||
|
new_string = re.sub(combined_pattern, '', str(element), flags=re.IGNORECASE)
|
||||||
|
element.replace_with(new_string)
|
||||||
|
|
||||||
|
return str(soup)
|
||||||
73
src/logger_setup.py
Normal file
73
src/logger_setup.py
Normal file
@@ -0,0 +1,73 @@
|
|||||||
|
import logging
|
||||||
|
import sys
|
||||||
|
from logging.handlers import RotatingFileHandler
|
||||||
|
from src.db_logger import SQLiteHandler, MySQLHandler
|
||||||
|
def setup_logging(config):
|
||||||
|
"""
|
||||||
|
Konfiguriert das Logging-System basierend auf der Konfigurationsdatei.
|
||||||
|
"""
|
||||||
|
log_config = config.get('log', {}).get('value', {})
|
||||||
|
|
||||||
|
# Deaktivieren, wenn keine Handler konfiguriert sind
|
||||||
|
if not log_config or not log_config.get('handlers'):
|
||||||
|
logging.disable(logging.CRITICAL)
|
||||||
|
return
|
||||||
|
|
||||||
|
# Root-Logger konfigurieren
|
||||||
|
logger = logging.getLogger()
|
||||||
|
if logger.hasHandlers():
|
||||||
|
logger.handlers.clear()
|
||||||
|
logger.setLevel(logging.INFO)
|
||||||
|
|
||||||
|
# Formatter erstellen
|
||||||
|
formatter = logging.Formatter('%(asctime)s - %(levelname)s - %(message)s')
|
||||||
|
|
||||||
|
handlers_to_add = log_config.get('handlers', [])
|
||||||
|
|
||||||
|
# File-Handler dynamisch erstellen
|
||||||
|
if 'file' in handlers_to_add:
|
||||||
|
file_conf = log_config.get('file', {})
|
||||||
|
log_file = file_conf.get('log_file', 'crawler.log')
|
||||||
|
max_size_mb = file_conf.get('max_size_mb', 10)
|
||||||
|
keep_last = file_conf.get('keep_last', 5)
|
||||||
|
max_bytes = max_size_mb * 1024 * 1024
|
||||||
|
|
||||||
|
file_handler = RotatingFileHandler(
|
||||||
|
log_file, maxBytes=max_bytes, backupCount=keep_last
|
||||||
|
)
|
||||||
|
file_handler.setFormatter(formatter)
|
||||||
|
logger.addHandler(file_handler)
|
||||||
|
|
||||||
|
# Console-Handler dynamisch erstellen
|
||||||
|
if 'console' in handlers_to_add:
|
||||||
|
stream_handler = logging.StreamHandler(sys.stdout)
|
||||||
|
stream_handler.setFormatter(formatter)
|
||||||
|
logger.addHandler(stream_handler)
|
||||||
|
|
||||||
|
# SQLite-Handler dynamisch erstellen
|
||||||
|
if 'sqlite' in handlers_to_add:
|
||||||
|
sqlite_conf = log_config.get('sqlite', {})
|
||||||
|
db_file = sqlite_conf.get('db_file')
|
||||||
|
if db_file:
|
||||||
|
sqlite_handler = SQLiteHandler(db_file=db_file)
|
||||||
|
sqlite_handler.setFormatter(formatter)
|
||||||
|
logger.addHandler(sqlite_handler)
|
||||||
|
|
||||||
|
# MySQL-Handler dynamisch erstellen
|
||||||
|
if 'mysql' in handlers_to_add:
|
||||||
|
mysql_conf = log_config.get('mysql', {})
|
||||||
|
# Nur initialisieren, wenn alle Konfigurationsschlüssel vorhanden sind
|
||||||
|
if all(k in mysql_conf for k in ['host', 'user', 'password', 'database']):
|
||||||
|
mysql_handler = MySQLHandler(
|
||||||
|
host=mysql_conf['host'],
|
||||||
|
user=mysql_conf['user'],
|
||||||
|
password=mysql_conf['password'],
|
||||||
|
database=mysql_conf['database']
|
||||||
|
)
|
||||||
|
mysql_handler.setFormatter(formatter)
|
||||||
|
logger.addHandler(mysql_handler)
|
||||||
|
|
||||||
|
if logger.hasHandlers():
|
||||||
|
logging.info("Logging-System erfolgreich initialisiert.")
|
||||||
|
else:
|
||||||
|
logging.disable(logging.CRITICAL)
|
||||||
59
src/state_manager.py
Normal file
59
src/state_manager.py
Normal file
@@ -0,0 +1,59 @@
|
|||||||
|
import json
|
||||||
|
import os
|
||||||
|
from queue import PriorityQueue
|
||||||
|
from typing import Set, Tuple, List, Optional
|
||||||
|
|
||||||
|
class StateManager:
|
||||||
|
"""Manages the state of the crawler for pausing and resuming."""
|
||||||
|
|
||||||
|
def __init__(self, state_file: str):
|
||||||
|
"""
|
||||||
|
Initializes the StateManager.
|
||||||
|
|
||||||
|
Args:
|
||||||
|
state_file: The path to the file where the state will be saved.
|
||||||
|
"""
|
||||||
|
self.state_file = state_file
|
||||||
|
|
||||||
|
def save_state(self, queue: PriorityQueue, visited: Set[str]):
|
||||||
|
"""
|
||||||
|
Saves the current state of the queue and visited set to a file.
|
||||||
|
|
||||||
|
Args:
|
||||||
|
queue: The PriorityQueue to save.
|
||||||
|
visited: The set of visited URLs to save.
|
||||||
|
"""
|
||||||
|
state = {
|
||||||
|
"queue": list(queue.queue),
|
||||||
|
"visited": list(visited)
|
||||||
|
}
|
||||||
|
with open(self.state_file, 'w') as f:
|
||||||
|
json.dump(state, f, indent=4)
|
||||||
|
|
||||||
|
def load_state(self) -> Optional[Tuple[PriorityQueue, Set[str]]]:
|
||||||
|
"""
|
||||||
|
Loads the state from the state file if it exists.
|
||||||
|
|
||||||
|
Returns:
|
||||||
|
A tuple containing the PriorityQueue and the set of visited URLs,
|
||||||
|
or None if no state file is found.
|
||||||
|
"""
|
||||||
|
if not os.path.exists(self.state_file):
|
||||||
|
return None
|
||||||
|
|
||||||
|
with open(self.state_file, 'r') as f:
|
||||||
|
state = json.load(f)
|
||||||
|
|
||||||
|
queue = PriorityQueue()
|
||||||
|
for item in state["queue"]:
|
||||||
|
# The items are stored as lists in JSON, convert them back to tuples
|
||||||
|
queue.put(tuple(item))
|
||||||
|
|
||||||
|
visited = set(state["visited"])
|
||||||
|
|
||||||
|
return queue, visited
|
||||||
|
|
||||||
|
def delete_state(self):
|
||||||
|
"""Deletes the state file if it exists."""
|
||||||
|
if os.path.exists(self.state_file):
|
||||||
|
os.remove(self.state_file)
|
||||||
19
src/stats_manager.py
Normal file
19
src/stats_manager.py
Normal file
@@ -0,0 +1,19 @@
|
|||||||
|
import time
|
||||||
|
|
||||||
|
class CrawlStats:
|
||||||
|
def __init__(self):
|
||||||
|
self.start_time = time.time()
|
||||||
|
self.end_time = None
|
||||||
|
self.total_visited = 0
|
||||||
|
self.total_saved = 0
|
||||||
|
self.total_data_volume = 0
|
||||||
|
self.errors = 0
|
||||||
|
|
||||||
|
def finish(self):
|
||||||
|
self.end_time = time.time()
|
||||||
|
|
||||||
|
@property
|
||||||
|
def duration(self):
|
||||||
|
if self.end_time:
|
||||||
|
return self.end_time - self.start_time
|
||||||
|
return time.time() - self.start_time
|
||||||
70
src/storage.py
Normal file
70
src/storage.py
Normal file
@@ -0,0 +1,70 @@
|
|||||||
|
import hashlib
|
||||||
|
import json
|
||||||
|
import logging
|
||||||
|
from datetime import datetime
|
||||||
|
from pathlib import Path
|
||||||
|
from urllib.parse import urlparse
|
||||||
|
|
||||||
|
def get_safe_path(url):
|
||||||
|
"""
|
||||||
|
Generiert einen sicheren, hierarchischen Pfad basierend auf der URL.
|
||||||
|
Ersetzt ungültige Dateisystemzeichen und verwendet die Domain/den Pfad.
|
||||||
|
"""
|
||||||
|
parsed = urlparse(url)
|
||||||
|
netloc = parsed.netloc
|
||||||
|
path = parsed.path.strip('/')
|
||||||
|
|
||||||
|
# Ersetze ungültige Zeichen im Pfad (z.B. Query-Trenner) durch Unterstriche
|
||||||
|
# Da die URL bereits normalisiert ist, sollte die Query leer sein, aber wir sichern ab.
|
||||||
|
safe_path = path.replace(':', '_').replace('*', '_').replace('"', '_').replace('<', '_').replace('>', '_').replace('|', '_').replace('?', '_').replace('&', '_')
|
||||||
|
|
||||||
|
# Füge die Domain hinzu
|
||||||
|
full_path = Path(netloc) / safe_path
|
||||||
|
|
||||||
|
# Wenn der Pfad leer ist (z.B. bei der Root-URL), verwenden wir 'index'
|
||||||
|
if not full_path.name:
|
||||||
|
full_path = full_path / "index"
|
||||||
|
|
||||||
|
# Füge die Dateiendung hinzu (z.B. .json, da wir später JSON speichern)
|
||||||
|
return full_path.with_suffix(".json")
|
||||||
|
|
||||||
|
def save(url, title, content, output_dir):
|
||||||
|
"""
|
||||||
|
Speichert die extrahierten Daten (URL, Titel, Zeitstempel, Inhalt) als JSON-Objekt.
|
||||||
|
"""
|
||||||
|
|
||||||
|
# 1. Generiere den Zeitstempel im ISO 8601 Format
|
||||||
|
timestamp = datetime.now().isoformat()
|
||||||
|
|
||||||
|
# 2. Erstelle das JSON-Datenobjekt
|
||||||
|
data = {
|
||||||
|
"url": url,
|
||||||
|
"title": title,
|
||||||
|
"timestamp": timestamp,
|
||||||
|
"content": content
|
||||||
|
}
|
||||||
|
|
||||||
|
# 3. Generiere den hierarchischen Pfad (für Browsability)
|
||||||
|
relative_path = get_safe_path(url)
|
||||||
|
|
||||||
|
# 4. Kombiniere Output-Dir und relativen Pfad
|
||||||
|
output_path = Path(output_dir)
|
||||||
|
fname = output_path / relative_path
|
||||||
|
|
||||||
|
# Stelle sicher, dass das Verzeichnis existiert
|
||||||
|
fname.parent.mkdir(parents=True, exist_ok=True)
|
||||||
|
|
||||||
|
# Speichere nur, wenn die Datei nicht existiert
|
||||||
|
if not fname.exists():
|
||||||
|
# Speichere das JSON-Objekt
|
||||||
|
try:
|
||||||
|
json_content = json.dumps(data, ensure_ascii=False, indent=4)
|
||||||
|
fname.write_text(json_content, encoding="utf-8")
|
||||||
|
logging.info(f"Gespeichert als {fname.relative_to(output_path)}")
|
||||||
|
except Exception as e:
|
||||||
|
logging.error(f"Fehler beim Speichern von JSON für {url}: {e}")
|
||||||
|
else:
|
||||||
|
# Wenn die Datei existiert, überspringen wir das Speichern.
|
||||||
|
logging.info(f"Datei existiert bereits: {fname.relative_to(output_path)}")
|
||||||
|
|
||||||
|
return fname
|
||||||
30
src/url_utils.py
Normal file
30
src/url_utils.py
Normal file
@@ -0,0 +1,30 @@
|
|||||||
|
from urllib.parse import urlparse, parse_qs, urlunparse, urlencode
|
||||||
|
|
||||||
|
# Liste gängiger Tracking-Parameter, die entfernt werden sollen
|
||||||
|
TRACKING_PARAMS = ['utm_source', 'utm_medium', 'utm_campaign', 'utm_term', 'utm_content', 'session', 'ref', 'gclid', 'fbclid']
|
||||||
|
|
||||||
|
def normalize_url(url):
|
||||||
|
"""
|
||||||
|
Normalisiert eine URL, indem Fragmente entfernt und gängige Tracking-Parameter
|
||||||
|
aus der Query-Komponente entfernt werden.
|
||||||
|
"""
|
||||||
|
parsed = urlparse(url)
|
||||||
|
|
||||||
|
# 1. Fragment entfernen
|
||||||
|
# 2. Query-Parameter bereinigen
|
||||||
|
|
||||||
|
query_params = parse_qs(parsed.query)
|
||||||
|
|
||||||
|
# Tracking-Parameter entfernen
|
||||||
|
cleaned_params = {k: v for k, v in query_params.items() if k.lower() not in TRACKING_PARAMS}
|
||||||
|
|
||||||
|
# Query neu zusammensetzen
|
||||||
|
# Da parse_qs Listen von Werten zurückgibt, müssen wir sie für urlunparse/urlencode
|
||||||
|
# in das Standardformat zurückführen (z.B. k=v1&k=v2)
|
||||||
|
|
||||||
|
cleaned_query = urlencode(cleaned_params, doseq=True)
|
||||||
|
|
||||||
|
# urlunparse erwartet eine Liste von 6 Elementen: scheme, netloc, path, params, query, fragment
|
||||||
|
normalized_url = urlunparse(parsed._replace(query=cleaned_query, fragment=''))
|
||||||
|
|
||||||
|
return normalized_url
|
||||||
32
src/web_server.py
Normal file
32
src/web_server.py
Normal file
@@ -0,0 +1,32 @@
|
|||||||
|
import http.server
|
||||||
|
import socketserver
|
||||||
|
import os
|
||||||
|
|
||||||
|
class ReadmeHandler(http.server.SimpleHTTPRequestHandler):
|
||||||
|
def do_GET(self):
|
||||||
|
readme_path = os.path.join(os.path.dirname(__file__), '..', 'README.md')
|
||||||
|
try:
|
||||||
|
with open(readme_path, 'r', encoding='utf-8') as f:
|
||||||
|
content = f.read()
|
||||||
|
self.send_response(200)
|
||||||
|
self.send_header('Content-type', 'text/plain; charset=utf-8')
|
||||||
|
self.end_headers()
|
||||||
|
self.wfile.write(content.encode('utf-8'))
|
||||||
|
except FileNotFoundError:
|
||||||
|
self.send_error(404, 'File Not Found: README.md')
|
||||||
|
|
||||||
|
def start_web_server():
|
||||||
|
PORT = 8000
|
||||||
|
httpd = None
|
||||||
|
try:
|
||||||
|
handler = ReadmeHandler
|
||||||
|
httpd = socketserver.TCPServer(("", PORT), handler)
|
||||||
|
print(f"Serving on port {PORT}")
|
||||||
|
httpd.serve_forever()
|
||||||
|
finally:
|
||||||
|
if httpd:
|
||||||
|
httpd.server_close()
|
||||||
|
print("Server closed.")
|
||||||
|
|
||||||
|
if __name__ == "__main__":
|
||||||
|
start_web_server()
|
||||||
62
start.py
Normal file
62
start.py
Normal file
@@ -0,0 +1,62 @@
|
|||||||
|
import json
|
||||||
|
import os
|
||||||
|
import logging
|
||||||
|
import asyncio
|
||||||
|
import aiohttp
|
||||||
|
from src.logger_setup import setup_logging
|
||||||
|
from src.crawler_core import crawl_source
|
||||||
|
from src.stats_manager import CrawlStats
|
||||||
|
|
||||||
|
def display_stats(all_stats):
|
||||||
|
"""Displays aggregated statistics from all crawl runs."""
|
||||||
|
total_visited = sum(s.total_visited for s in all_stats)
|
||||||
|
total_saved = sum(s.total_saved for s in all_stats)
|
||||||
|
total_errors = sum(s.errors for s in all_stats)
|
||||||
|
total_duration = sum(s.duration for s in all_stats)
|
||||||
|
total_data_mb = sum(s.total_data_volume for s in all_stats) / (1024 * 1024)
|
||||||
|
|
||||||
|
logging.info("--- Gesamte Crawling-Statistik ---")
|
||||||
|
logging.info(f"Gesamtdauer: {total_duration:.2f} Sekunden")
|
||||||
|
logging.info(f"Besuchte Seiten insgesamt: {total_visited}")
|
||||||
|
logging.info(f"Gespeicherte Seiten insgesamt: {total_saved}")
|
||||||
|
logging.info(f"Fehler insgesamt: {total_errors}")
|
||||||
|
logging.info(f"Gesamtdatenvolumen: {total_data_mb:.2f} MB")
|
||||||
|
logging.info("------------------------------------")
|
||||||
|
|
||||||
|
async def main():
|
||||||
|
"""
|
||||||
|
Main function to initialize and run the web crawler for all sources.
|
||||||
|
"""
|
||||||
|
# Load configuration
|
||||||
|
with open('config.json', 'r') as f:
|
||||||
|
config = json.load(f)
|
||||||
|
|
||||||
|
# Load URL list
|
||||||
|
with open('url_list.json', 'r') as f:
|
||||||
|
url_list = json.load(f)
|
||||||
|
|
||||||
|
# Setup logging
|
||||||
|
setup_logging(config)
|
||||||
|
logger = logging.getLogger(__name__)
|
||||||
|
|
||||||
|
all_stats = []
|
||||||
|
|
||||||
|
try:
|
||||||
|
async with aiohttp.ClientSession() as session:
|
||||||
|
for source in url_list:
|
||||||
|
logger.info(f"Processing source: {source['start_urls'][0]}")
|
||||||
|
stats = await crawl_source(session, source, config)
|
||||||
|
all_stats.append(stats)
|
||||||
|
logger.info(f"Finished source: {source['start_urls'][0]}")
|
||||||
|
|
||||||
|
except Exception as e:
|
||||||
|
logger.critical(f"A critical error occurred in the main loop: {e}", exc_info=True)
|
||||||
|
finally:
|
||||||
|
logger.info("All crawling tasks finished.")
|
||||||
|
display_stats(all_stats)
|
||||||
|
|
||||||
|
|
||||||
|
if __name__ == "__main__":
|
||||||
|
# Change the current working directory to the script's directory
|
||||||
|
os.chdir(os.path.dirname(os.path.abspath(__file__)))
|
||||||
|
asyncio.run(main())
|
||||||
308
url_list.json
Normal file
308
url_list.json
Normal file
@@ -0,0 +1,308 @@
|
|||||||
|
[
|
||||||
|
{
|
||||||
|
"start_urls": [
|
||||||
|
"https://sqlite.org/docs.html"
|
||||||
|
],
|
||||||
|
"crawl_mode": "single_page"
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"start_urls": [
|
||||||
|
"https://www.gesetze-im-internet.de/ao_1977/BJNR006130976.html"
|
||||||
|
],
|
||||||
|
"crawl_mode": "single_page"
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"start_urls": [
|
||||||
|
"https://www.gesetze-im-internet.de/gmbhg/BJNR004770892.html"
|
||||||
|
],
|
||||||
|
"crawl_mode": "single_page"
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"start_urls": [
|
||||||
|
"https://www.gesetze-im-internet.de/pangv_2022/BJNR492110021.html"
|
||||||
|
],
|
||||||
|
"crawl_mode": "single_page"
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"start_urls": [
|
||||||
|
"https://www.gesetze-im-internet.de/ustg_1980/BJNR119530979.html"
|
||||||
|
],
|
||||||
|
"crawl_mode": "single_page"
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"start_urls": [
|
||||||
|
"https://www.gesetze-im-internet.de/uwg_2004/BJNR141400004.html"
|
||||||
|
],
|
||||||
|
"crawl_mode": "single_page"
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"start_urls": [
|
||||||
|
"https://www.gesetze-im-internet.de/hgb/BJNR002190897.html"
|
||||||
|
],
|
||||||
|
"crawl_mode": "single_page"
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"start_urls": [
|
||||||
|
"https://www.gesetze-im-internet.de/bgb/BJNR001950896.html"
|
||||||
|
],
|
||||||
|
"crawl_mode": "single_page"
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"start_urls": [
|
||||||
|
"https://www.gesetze-im-internet.de/fahrschausbo_2012/BJNR131800012.html"
|
||||||
|
],
|
||||||
|
"crawl_mode": "single_page"
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"start_urls": [
|
||||||
|
"https://www.gesetze-im-internet.de/fahrlpr_fv/BJNR004200018.html"
|
||||||
|
],
|
||||||
|
"crawl_mode": "single_page"
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"start_urls": [
|
||||||
|
"https://www.gesetze-im-internet.de/fahrlg_2018/BJNR216210017.html"
|
||||||
|
],
|
||||||
|
"crawl_mode": "single_page"
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"start_urls": [
|
||||||
|
"https://www.gesetze-im-internet.de/fahrlausbv/BJNR001500018.html"
|
||||||
|
],
|
||||||
|
"crawl_mode": "single_page"
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"start_urls": [
|
||||||
|
"https://www.gesetze-im-internet.de/pbefg/BJNR002410961.html"
|
||||||
|
],
|
||||||
|
"crawl_mode": "single_page"
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"start_urls": [
|
||||||
|
"https://www.gesetze-im-internet.de/g_kg_1998/BJNR148510998.html"
|
||||||
|
],
|
||||||
|
"crawl_mode": "single_page"
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"start_urls": [
|
||||||
|
"https://www.gesetze-im-internet.de/fev_2010/BJNR198000010.html"
|
||||||
|
],
|
||||||
|
"crawl_mode": "single_page"
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"start_urls": [
|
||||||
|
"https://www.gesetze-im-internet.de/stvzo_2012/BJNR067910012.html"
|
||||||
|
],
|
||||||
|
"crawl_mode": "single_page"
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"start_urls": [
|
||||||
|
"https://www.gesetze-im-internet.de/stvg/BJNR004370909.html"
|
||||||
|
],
|
||||||
|
"crawl_mode": "single_page"
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"start_urls": [
|
||||||
|
"https://www.gesetze-im-internet.de/stvoausnv_8/BJNR113000998.html"
|
||||||
|
],
|
||||||
|
"crawl_mode": "single_page"
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"start_urls": [
|
||||||
|
"https://www.gesetze-im-internet.de/stvoausnv_5/BJNR062300994.html"
|
||||||
|
],
|
||||||
|
"crawl_mode": "single_page"
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"start_urls": [
|
||||||
|
"https://www.gesetze-im-internet.de/stvoausnv_9/BJNR317100998.html"
|
||||||
|
],
|
||||||
|
"crawl_mode": "single_page"
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"start_urls": [
|
||||||
|
"https://www.gesetze-im-internet.de/stvoausnv_12/BJNR086600005.html"
|
||||||
|
],
|
||||||
|
"crawl_mode": "single_page"
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"start_urls": [
|
||||||
|
"https://www.gesetze-im-internet.de/stvoausnv_4/BJNR011240992.html"
|
||||||
|
],
|
||||||
|
"crawl_mode": "single_page"
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"start_urls": [
|
||||||
|
"https://www.gesetze-im-internet.de/stvo_2013/BJNR036710013.html"
|
||||||
|
],
|
||||||
|
"crawl_mode": "single_page"
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"start_urls": [
|
||||||
|
"https://eur-lex.europa.eu/legal-content/DE/TXT/HTML/?uri=CELEX:32016R0679"
|
||||||
|
],
|
||||||
|
"crawl_mode": "single_page"
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"start_urls": [
|
||||||
|
"https://www.gnu.org/software/bash/manual/bash.html"
|
||||||
|
],
|
||||||
|
"crawl_mode": "single_page"
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"start_urls": [
|
||||||
|
"https://www.json.org/json-en.html"
|
||||||
|
],
|
||||||
|
"crawl_mode": "single_page"
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"start_urls": [
|
||||||
|
"https://www.bussgeldkatalog.org/"
|
||||||
|
],
|
||||||
|
"crawl_mode": "domain_wide"
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"start_urls": [
|
||||||
|
"https://docs.budibase.com/"
|
||||||
|
],
|
||||||
|
"crawl_mode": "domain_wide"
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"start_urls": [
|
||||||
|
"https://shadowhelix.de/"
|
||||||
|
],
|
||||||
|
"crawl_mode": "domain_wide"
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"start_urls": [
|
||||||
|
"https://markdown.de/"
|
||||||
|
],
|
||||||
|
"crawl_mode": "domain_wide"
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"start_urls": [
|
||||||
|
"https://mariadb.com/docs/"
|
||||||
|
],
|
||||||
|
"crawl_mode": "domain_wide"
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"start_urls": [
|
||||||
|
"https://html.spec.whatwg.org/"
|
||||||
|
],
|
||||||
|
"crawl_mode": "domain_wide"
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"start_urls": [
|
||||||
|
"https://restfulapi.net/"
|
||||||
|
],
|
||||||
|
"crawl_mode": "domain_wide"
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"start_urls": [
|
||||||
|
"https://react.dev/"
|
||||||
|
],
|
||||||
|
"crawl_mode": "domain_wide"
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"start_urls": [
|
||||||
|
"https://expressjs.com/"
|
||||||
|
],
|
||||||
|
"crawl_mode": "domain_wide"
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"start_urls": [
|
||||||
|
"https://docs.docker.com/"
|
||||||
|
],
|
||||||
|
"crawl_mode": "domain_wide"
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"start_urls": [
|
||||||
|
"https://docs.openwebui.com/"
|
||||||
|
],
|
||||||
|
"crawl_mode": "domain_wide"
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"start_urls": [
|
||||||
|
"https://www.php.net/manual/en/"
|
||||||
|
],
|
||||||
|
"crawl_mode": "domain_wide"
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"start_urls": [
|
||||||
|
"https://www.postgresql.org/docs/"
|
||||||
|
],
|
||||||
|
"crawl_mode": "domain_wide"
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"start_urls": [
|
||||||
|
"https://nginx.org/en/docs/"
|
||||||
|
],
|
||||||
|
"crawl_mode": "domain_wide"
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"start_urls": [
|
||||||
|
"https://docs.nginx.com/nginx/admin-guide/"
|
||||||
|
],
|
||||||
|
"crawl_mode": "domain_wide"
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"start_urls": [
|
||||||
|
"https://wordpress.org/documentation/"
|
||||||
|
],
|
||||||
|
"crawl_mode": "domain_wide"
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"start_urls": [
|
||||||
|
"https://github.com/fail2ban/"
|
||||||
|
],
|
||||||
|
"crawl_mode": "domain_wide"
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"start_urls": [
|
||||||
|
"httpshttps://grafana.com/docs/"
|
||||||
|
],
|
||||||
|
"crawl_mode": "domain_wide"
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"start_urls": [
|
||||||
|
"https://nodejs.org/docs/latest/"
|
||||||
|
],
|
||||||
|
"crawl_mode": "domain_wide"
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"start_urls": [
|
||||||
|
"https://www.virtualmin.com/docs/"
|
||||||
|
],
|
||||||
|
"crawl_mode": "domain_wide"
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"start_urls": [
|
||||||
|
"https://developer.mozilla.org/en-US/docs/Web/"
|
||||||
|
],
|
||||||
|
"crawl_mode": "domain_wide"
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"start_urls": [
|
||||||
|
"https://owasp.org/www-project-top-ten/"
|
||||||
|
],
|
||||||
|
"crawl_mode": "domain_wide"
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"start_urls": [
|
||||||
|
"https://cheatsheetseries.owasp.org/"
|
||||||
|
],
|
||||||
|
"crawl_mode": "domain_wide"
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"start_urls": [
|
||||||
|
"https://nodered.org/docs/"
|
||||||
|
],
|
||||||
|
"crawl_mode": "domain_wide"
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"start_urls": [
|
||||||
|
"https://www.home-assistant.io/docs/"
|
||||||
|
],
|
||||||
|
"crawl_mode": "domain_wide"
|
||||||
|
}
|
||||||
|
]
|
||||||
Reference in New Issue
Block a user