neu
This commit is contained in:
35
.roomodes
Normal file
35
.roomodes
Normal file
@@ -0,0 +1,35 @@
|
|||||||
|
customModes:
|
||||||
|
- slug: project-research
|
||||||
|
name: 🔍 Project Research
|
||||||
|
roleDefinition: |
|
||||||
|
You are a detailed-oriented research assistant specializing in examining and understanding codebases. Your primary responsibility is to analyze the file structure, content, and dependencies of a given project to provide comprehensive context relevant to specific user queries.
|
||||||
|
whenToUse: |
|
||||||
|
Use this mode when you need to thoroughly investigate and understand a codebase structure, analyze project architecture, or gather comprehensive context about existing implementations. Ideal for onboarding to new projects, understanding complex codebases, or researching how specific features are implemented across the project.
|
||||||
|
description: Investigate and analyze codebase structure
|
||||||
|
groups:
|
||||||
|
- read
|
||||||
|
source: project
|
||||||
|
customInstructions: |
|
||||||
|
Your role is to deeply investigate and summarize the structure and implementation details of the project codebase. To achieve this effectively, you must:
|
||||||
|
|
||||||
|
1. Start by carefully examining the file structure of the entire project, with a particular emphasis on files located within the "docs" folder. These files typically contain crucial context, architectural explanations, and usage guidelines.
|
||||||
|
|
||||||
|
2. When given a specific query, systematically identify and gather all relevant context from:
|
||||||
|
- Documentation files in the "docs" folder that provide background information, specifications, or architectural insights.
|
||||||
|
- Relevant type definitions and interfaces, explicitly citing their exact location (file path and line number) within the source code.
|
||||||
|
- Implementations directly related to the query, clearly noting their file locations and providing concise yet comprehensive summaries of how they function.
|
||||||
|
- Important dependencies, libraries, or modules involved in the implementation, including their usage context and significance to the query.
|
||||||
|
|
||||||
|
3. Deliver a structured, detailed report that clearly outlines:
|
||||||
|
- An overview of relevant documentation insights.
|
||||||
|
- Specific type definitions and their exact locations.
|
||||||
|
- Relevant implementations, including file paths, functions or methods involved, and a brief explanation of their roles.
|
||||||
|
- Critical dependencies and their roles in relation to the query.
|
||||||
|
|
||||||
|
4. Always cite precise file paths, function names, and line numbers to enhance clarity and ease of navigation.
|
||||||
|
|
||||||
|
5. Organize your findings in logical sections, making it straightforward for the user to understand the project's structure and implementation status relevant to their request.
|
||||||
|
|
||||||
|
6. Ensure your response directly addresses the user's query and helps them fully grasp the relevant aspects of the project's current state.
|
||||||
|
|
||||||
|
These specific instructions supersede any conflicting general instructions you might otherwise follow. Your detailed report should enable effective decision-making and next steps within the overall workflow.
|
||||||
@@ -50,9 +50,9 @@ def create_app(config: Optional[Dict[str, Any]] = None) -> FastAPI:
|
|||||||
pass
|
pass
|
||||||
|
|
||||||
try:
|
try:
|
||||||
from agent_api.router import agent_router
|
import agent_api
|
||||||
|
|
||||||
app.include_router(agent_router, prefix="/api")
|
app.include_router(agent_api.agent_router, prefix="/api")
|
||||||
except Exception:
|
except Exception:
|
||||||
# Agenten-API ist optional und wird bei fehlender Implementierung ignoriert
|
# Agenten-API ist optional und wird bei fehlender Implementierung ignoriert
|
||||||
pass
|
pass
|
||||||
|
|||||||
@@ -0,0 +1,19 @@
|
|||||||
|
# Agenten-API Modulkonfiguration (Standardwerte)
|
||||||
|
# Wird vom optionalen Modul geladen; Anpassungen erfolgen projektspezifisch.
|
||||||
|
|
||||||
|
metadata:
|
||||||
|
name: "agent_api"
|
||||||
|
description: "Konfiguration für die optionale Agenten-API-Erweiterung."
|
||||||
|
|
||||||
|
execution:
|
||||||
|
mode: "async" # Optionen: "async" oder "sync"
|
||||||
|
response_timeout_seconds: 30 # Maximale Wartezeit für synchrone Antworten (Sekunden)
|
||||||
|
queue_ttl_seconds: 300 # Lebensdauer eines Tasks in Sekunden
|
||||||
|
heartbeat_interval_seconds: 10 # Interval für Heartbeats der Hintergrund-Worker (Sekunden)
|
||||||
|
|
||||||
|
llm:
|
||||||
|
provider: "local_stub" # Setze z. B. "openai" wenn externe APIs verwendet werden
|
||||||
|
model: "local-agent"
|
||||||
|
api_base_url: null # Externe Basis-URL, z. B. https://api.openai.com/v1
|
||||||
|
api_key: null # Wird durch die Umgebungsvariable AGENT_API_LLM_KEY überschrieben
|
||||||
|
request_timeout_seconds: 15 # Request-Timeout für den LLM-Client in Sekunden
|
||||||
@@ -116,7 +116,33 @@ def init_logging(settings: Settings, app_config: Optional[Dict[str, Any]] = None
|
|||||||
logging.getLogger(__name__).exception(
|
logging.getLogger(__name__).exception(
|
||||||
"Fehler bei Initialisierung des internen SQLite-Loggings; Handler wird nicht aktiviert."
|
"Fehler bei Initialisierung des internen SQLite-Loggings; Handler wird nicht aktiviert."
|
||||||
)
|
)
|
||||||
|
|
||||||
|
# Optionale Integration: externes DB-Logging (01_Modulerweiterungen)
|
||||||
|
try:
|
||||||
|
external_cfg = ((app_config or {}).get("logging_external") or {})
|
||||||
|
except Exception:
|
||||||
|
external_cfg = {}
|
||||||
|
|
||||||
|
if isinstance(external_cfg, dict) and external_cfg.get("enabled"):
|
||||||
|
try:
|
||||||
|
import importlib
|
||||||
|
ext_mod = importlib.import_module("logging_external")
|
||||||
|
except Exception:
|
||||||
|
logging.getLogger(__name__).warning(
|
||||||
|
"logging_external Modul nicht importierbar; externer DB-Handler wird nicht aktiviert."
|
||||||
|
)
|
||||||
|
else:
|
||||||
|
try:
|
||||||
|
# ext_mod.init akzeptiert entweder connection_url ODER config-Dict
|
||||||
|
# Wir übergeben das gesamte Config-Dict und lassen das Modul die URL bauen.
|
||||||
|
ext_mod.init(config=external_cfg)
|
||||||
|
ext_handler = ext_mod.get_handler(level)
|
||||||
|
root.addHandler(ext_handler)
|
||||||
|
logging.getLogger(__name__).info("Externer DB-Log-Handler aktiviert.")
|
||||||
|
except Exception:
|
||||||
|
logging.getLogger(__name__).exception(
|
||||||
|
"Fehler bei Initialisierung des externen DB-Loggings; Handler wird nicht aktiviert."
|
||||||
|
)
|
||||||
|
|
||||||
def _reset_logger_handlers(logger: logging.Logger) -> None:
|
def _reset_logger_handlers(logger: logging.Logger) -> None:
|
||||||
"""Entfernt alle existierenden Handler vom Logger."""
|
"""Entfernt alle existierenden Handler vom Logger."""
|
||||||
|
|||||||
@@ -0,0 +1,23 @@
|
|||||||
|
# Konfiguration für externes Logging-Modul (MySQL/PostgreSQL über SQLAlchemy)
|
||||||
|
# Diese Datei folgt dem Schema der Konfigurationen unter
|
||||||
|
# 00_Globale_Richtlinien/Entworfener_Code/app/config/
|
||||||
|
# und kann später in die Haupt-App übernommen werden.
|
||||||
|
#
|
||||||
|
# Hinweise:
|
||||||
|
# - Secrets (user/password) sollten via Umgebungsvariablen überschrieben werden:
|
||||||
|
# LOG_EXT_USER, LOG_EXT_PASSWORD
|
||||||
|
# - Das Modul unterstützt Fallback ins interne SQLite-Logging, wenn aktiviert.
|
||||||
|
|
||||||
|
logging_external:
|
||||||
|
enabled: false # Modul aktivieren/deaktivieren
|
||||||
|
type: "postgresql" # "postgresql" | "mysql"
|
||||||
|
host: "localhost" # DB-Host
|
||||||
|
port: 5432 # 5432 für PostgreSQL, 3306 für MySQL
|
||||||
|
user: null # via ENV: LOG_EXT_USER (überschreibt diesen Wert)
|
||||||
|
password: null # via ENV: LOG_EXT_PASSWORD (überschreibt diesen Wert)
|
||||||
|
database: "logs" # Datenbankname
|
||||||
|
sslmode: "prefer" # nur für PostgreSQL relevant: disable|allow|prefer|require|verify-ca|verify-full
|
||||||
|
pool_size: 5 # Größe des Connection-Pools
|
||||||
|
connect_timeout: 10 # Verbindungs-Timeout in Sekunden
|
||||||
|
write_buffer_size: 100 # (reserviert für spätere Batch-Nutzung)
|
||||||
|
fallback_to_internal_on_error: true # Fallback in internes SQLite-Logging bei Fehlern
|
||||||
@@ -0,0 +1,12 @@
|
|||||||
|
# Konfiguration für internes Logging-Modul (SQLite)
|
||||||
|
# Diese Datei folgt dem selben Schema wie die Konfigurationen unter 00_Globale_Richtlinien/Entworfener_Code/app/config
|
||||||
|
# Sie kann später in die Haupt-App übernommen werden; Werte sind Default-Beispiele.
|
||||||
|
|
||||||
|
logging_internal:
|
||||||
|
enabled: true # Modul aktivieren/deaktivieren
|
||||||
|
db_path: "data/internal_logs.sqlite" # relativ zu app/
|
||||||
|
clean_database: false # Datenbank beim Start bereinigen (vorsichtig einsetzen)
|
||||||
|
retention_days: 30 # Aufbewahrungsdauer in Tagen
|
||||||
|
max_entries: 100000 # Max. Anzahl an Logeinträgen
|
||||||
|
vacuum_on_start: true # VACUUM nach Start/Cleanup
|
||||||
|
batch_write: 100 # zukünftige Batch-Größe (derzeit optional)
|
||||||
416
01_Modulerweiterungen/Entworfener_Code/app/src/agent_api.py
Normal file
416
01_Modulerweiterungen/Entworfener_Code/app/src/agent_api.py
Normal file
@@ -0,0 +1,416 @@
|
|||||||
|
"""
|
||||||
|
Agenten-API FastAPI Router für Aufgabenverarbeitung.
|
||||||
|
"""
|
||||||
|
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import logging
|
||||||
|
import os
|
||||||
|
import threading
|
||||||
|
import time
|
||||||
|
import uuid
|
||||||
|
from dataclasses import dataclass
|
||||||
|
from datetime import datetime, timedelta, timezone
|
||||||
|
from pathlib import Path
|
||||||
|
from typing import Any, Dict, List, Optional, TypedDict
|
||||||
|
|
||||||
|
import yaml
|
||||||
|
from fastapi import APIRouter, HTTPException, status
|
||||||
|
from fastapi.responses import JSONResponse
|
||||||
|
from pydantic import BaseModel, Field
|
||||||
|
|
||||||
|
logger = logging.getLogger(__name__)
|
||||||
|
|
||||||
|
|
||||||
|
@dataclass(frozen=True)
|
||||||
|
class ExecutionSettings:
|
||||||
|
"""Konfiguration der Ausführungspipeline."""
|
||||||
|
|
||||||
|
mode: str = "async"
|
||||||
|
queue_ttl_seconds: int = 300
|
||||||
|
heartbeat_interval_seconds: int = 10
|
||||||
|
response_timeout_seconds: int = 30
|
||||||
|
|
||||||
|
|
||||||
|
@dataclass(frozen=True)
|
||||||
|
class LLMSettings:
|
||||||
|
"""Konfiguration des LLM-Clients."""
|
||||||
|
|
||||||
|
provider: str = "local_stub"
|
||||||
|
model: str = "local-agent"
|
||||||
|
api_base_url: Optional[str] = None
|
||||||
|
api_key: Optional[str] = None
|
||||||
|
request_timeout_seconds: int = 15
|
||||||
|
|
||||||
|
|
||||||
|
@dataclass(frozen=True)
|
||||||
|
class AgentAPIConfig:
|
||||||
|
"""Gebündelte Agenten-API Konfiguration."""
|
||||||
|
|
||||||
|
execution: ExecutionSettings
|
||||||
|
llm: LLMSettings
|
||||||
|
|
||||||
|
|
||||||
|
class TaskRequest(BaseModel):
|
||||||
|
"""Eingangsmodell für eine Agenten-Aufgabe."""
|
||||||
|
|
||||||
|
user_input: str = Field(..., description="Auslöser der Task in natürlicher Sprache.")
|
||||||
|
context: Optional[Dict[str, Any]] = Field(default=None, description="Optionale Kontextdaten des Aufrufs.")
|
||||||
|
metadata: Optional[Dict[str, Any]] = Field(default=None, description="Zusätzliche Metadaten zur Korrelation.")
|
||||||
|
sync: Optional[bool] = Field(default=None, description="Erzwingt synchrone Verarbeitung, falls gesetzt.")
|
||||||
|
|
||||||
|
|
||||||
|
class TaskData(TypedDict):
|
||||||
|
"""Interne Darstellung eines Tasks im Speicher."""
|
||||||
|
|
||||||
|
task_id: str
|
||||||
|
status: str
|
||||||
|
result: Optional[Dict[str, Any]]
|
||||||
|
error: Optional[str]
|
||||||
|
created_at: datetime
|
||||||
|
updated_at: datetime
|
||||||
|
expires_at: datetime
|
||||||
|
last_heartbeat: datetime
|
||||||
|
retry_after: int
|
||||||
|
request: TaskRequest
|
||||||
|
|
||||||
|
|
||||||
|
def _utcnow() -> datetime:
|
||||||
|
"""Gibt die aktuelle UTC-Zeit zurück."""
|
||||||
|
return datetime.now(timezone.utc)
|
||||||
|
|
||||||
|
|
||||||
|
def _to_iso(dt: datetime) -> str:
|
||||||
|
"""Formatiert Datumswerte als ISO8601-String mit Z-Suffix."""
|
||||||
|
return dt.astimezone(timezone.utc).isoformat().replace("+00:00", "Z")
|
||||||
|
|
||||||
|
|
||||||
|
def _positive_int(value: Any, default: int) -> int:
|
||||||
|
"""Stellt sicher, dass ein Integer positiv ist, andernfalls gilt der Default."""
|
||||||
|
try:
|
||||||
|
candidate = int(value)
|
||||||
|
if candidate > 0:
|
||||||
|
return candidate
|
||||||
|
except (TypeError, ValueError):
|
||||||
|
pass
|
||||||
|
return default
|
||||||
|
|
||||||
|
|
||||||
|
def _candidate_config_paths(module_path: Path) -> List[Path]:
|
||||||
|
"""Ermittelt potenzielle Konfigurationspfade gemäß Projektstandard."""
|
||||||
|
candidates: List[Path] = []
|
||||||
|
local_app_path = module_path.parent.parent
|
||||||
|
candidates.append(local_app_path / "config" / "agent_api.yaml")
|
||||||
|
for parent in module_path.parents:
|
||||||
|
candidate = parent / "00_Globale_Richtlinien" / "Entworfener_Code" / "app" / "config" / "agent_api.yaml"
|
||||||
|
if candidate not in candidates:
|
||||||
|
candidates.append(candidate)
|
||||||
|
return candidates
|
||||||
|
|
||||||
|
|
||||||
|
def _load_agent_config() -> AgentAPIConfig:
|
||||||
|
"""Lädt die Agenten-Konfiguration und führt sie mit Defaults zusammen."""
|
||||||
|
module_path = Path(__file__).resolve()
|
||||||
|
config_data: Dict[str, Any] = {}
|
||||||
|
for path in _candidate_config_paths(module_path):
|
||||||
|
if not path.exists():
|
||||||
|
continue
|
||||||
|
try:
|
||||||
|
with path.open("r", encoding="utf-8") as handle:
|
||||||
|
loaded = yaml.safe_load(handle) or {}
|
||||||
|
config_data = loaded
|
||||||
|
logger.debug("Agenten-API-Konfiguration aus %s geladen.", path)
|
||||||
|
break
|
||||||
|
except Exception as exc:
|
||||||
|
logger.warning("Fehler beim Laden der Agenten-API-Konfiguration aus %s: %s", path, exc)
|
||||||
|
execution_data = config_data.get("execution", {})
|
||||||
|
llm_data = config_data.get("llm", {})
|
||||||
|
mode = str(execution_data.get("mode", "async")).lower()
|
||||||
|
if mode not in {"async", "sync"}:
|
||||||
|
mode = "async"
|
||||||
|
execution = ExecutionSettings(
|
||||||
|
mode=mode,
|
||||||
|
queue_ttl_seconds=_positive_int(execution_data.get("queue_ttl_seconds"), default=300),
|
||||||
|
heartbeat_interval_seconds=_positive_int(execution_data.get("heartbeat_interval_seconds"), default=10),
|
||||||
|
response_timeout_seconds=_positive_int(execution_data.get("response_timeout_seconds"), default=30),
|
||||||
|
)
|
||||||
|
env_api_key = os.getenv("AGENT_API_LLM_KEY")
|
||||||
|
api_key = env_api_key if env_api_key else llm_data.get("api_key")
|
||||||
|
llm = LLMSettings(
|
||||||
|
provider=str(llm_data.get("provider", "local_stub")),
|
||||||
|
model=str(llm_data.get("model", "local-agent")),
|
||||||
|
api_base_url=llm_data.get("api_base_url"),
|
||||||
|
api_key=str(api_key) if api_key else None,
|
||||||
|
request_timeout_seconds=_positive_int(llm_data.get("request_timeout_seconds"), default=15),
|
||||||
|
)
|
||||||
|
return AgentAPIConfig(execution=execution, llm=llm)
|
||||||
|
|
||||||
|
|
||||||
|
CONFIG = _load_agent_config()
|
||||||
|
|
||||||
|
|
||||||
|
class TaskRegistry:
|
||||||
|
"""Thread-sichere Verwaltung der In-Memory-Tasks."""
|
||||||
|
|
||||||
|
def __init__(self) -> None:
|
||||||
|
self._tasks: Dict[str, TaskData] = {}
|
||||||
|
self._lock = threading.Lock()
|
||||||
|
|
||||||
|
def create(self, request: TaskRequest, ttl_seconds: int, retry_after: int) -> TaskData:
|
||||||
|
"""Erzeugt eine neue Task-Struktur und speichert sie."""
|
||||||
|
now = _utcnow()
|
||||||
|
task: TaskData = {
|
||||||
|
"task_id": str(uuid.uuid4()),
|
||||||
|
"status": "processing",
|
||||||
|
"result": None,
|
||||||
|
"error": None,
|
||||||
|
"created_at": now,
|
||||||
|
"updated_at": now,
|
||||||
|
"expires_at": now + timedelta(seconds=ttl_seconds),
|
||||||
|
"last_heartbeat": now,
|
||||||
|
"retry_after": max(retry_after, 1),
|
||||||
|
"request": request,
|
||||||
|
}
|
||||||
|
with self._lock:
|
||||||
|
self._tasks[task["task_id"]] = task
|
||||||
|
return task.copy()
|
||||||
|
|
||||||
|
def get(self, task_id: str) -> Optional[TaskData]:
|
||||||
|
"""Liefert eine Kopie der Task und aktualisiert Expirationen."""
|
||||||
|
with self._lock:
|
||||||
|
task = self._tasks.get(task_id)
|
||||||
|
if task is None:
|
||||||
|
return None
|
||||||
|
self._expire_task_locked(task)
|
||||||
|
return task.copy()
|
||||||
|
|
||||||
|
def get_request(self, task_id: str) -> Optional[TaskRequest]:
|
||||||
|
"""Liefert das ursprüngliche Request-Objekt."""
|
||||||
|
with self._lock:
|
||||||
|
task = self._tasks.get(task_id)
|
||||||
|
if task is None:
|
||||||
|
return None
|
||||||
|
return task["request"]
|
||||||
|
|
||||||
|
def heartbeat(self, task_id: str) -> None:
|
||||||
|
"""Aktualisiert den Heartbeat einer laufenden Task."""
|
||||||
|
with self._lock:
|
||||||
|
task = self._tasks.get(task_id)
|
||||||
|
if task is None or task["status"] != "processing":
|
||||||
|
return
|
||||||
|
now = _utcnow()
|
||||||
|
task["last_heartbeat"] = now
|
||||||
|
task["updated_at"] = now
|
||||||
|
|
||||||
|
def mark_succeeded(self, task_id: str, result: Dict[str, Any], ttl_seconds: int) -> Optional[TaskData]:
|
||||||
|
"""Markiert eine Task als abgeschlossen."""
|
||||||
|
with self._lock:
|
||||||
|
task = self._tasks.get(task_id)
|
||||||
|
if task is None:
|
||||||
|
return None
|
||||||
|
if task["status"] in {"cancelled", "expired"}:
|
||||||
|
return task.copy()
|
||||||
|
now = _utcnow()
|
||||||
|
task["status"] = "succeeded"
|
||||||
|
task["result"] = result
|
||||||
|
task["error"] = None
|
||||||
|
task["updated_at"] = now
|
||||||
|
task["last_heartbeat"] = now
|
||||||
|
task["expires_at"] = now + timedelta(seconds=ttl_seconds)
|
||||||
|
return task.copy()
|
||||||
|
|
||||||
|
def mark_failed(self, task_id: str, error: str, ttl_seconds: int) -> Optional[TaskData]:
|
||||||
|
"""Markiert eine Task als fehlgeschlagen."""
|
||||||
|
with self._lock:
|
||||||
|
task = self._tasks.get(task_id)
|
||||||
|
if task is None:
|
||||||
|
return None
|
||||||
|
now = _utcnow()
|
||||||
|
task["status"] = "failed"
|
||||||
|
task["result"] = {"error": error}
|
||||||
|
task["error"] = error
|
||||||
|
task["updated_at"] = now
|
||||||
|
task["last_heartbeat"] = now
|
||||||
|
task["expires_at"] = now + timedelta(seconds=ttl_seconds)
|
||||||
|
return task.copy()
|
||||||
|
|
||||||
|
def cancel(self, task_id: str) -> Optional[TaskData]:
|
||||||
|
"""Bricht eine Task ab und markiert sie als cancelled."""
|
||||||
|
with self._lock:
|
||||||
|
task = self._tasks.get(task_id)
|
||||||
|
if task is None:
|
||||||
|
return None
|
||||||
|
now = _utcnow()
|
||||||
|
if task["status"] in {"succeeded", "failed", "expired", "cancelled"}:
|
||||||
|
self._expire_task_locked(task)
|
||||||
|
return task.copy()
|
||||||
|
task["status"] = "cancelled"
|
||||||
|
task["error"] = "cancelled by client"
|
||||||
|
task["result"] = None
|
||||||
|
task["updated_at"] = now
|
||||||
|
task["last_heartbeat"] = now
|
||||||
|
task["expires_at"] = now
|
||||||
|
return task.copy()
|
||||||
|
|
||||||
|
def _expire_task_locked(self, task: TaskData) -> None:
|
||||||
|
"""Setzt den Status auf expired, falls das Ablaufdatum erreicht ist."""
|
||||||
|
now = _utcnow()
|
||||||
|
if now < task["expires_at"] or task["status"] == "expired":
|
||||||
|
return
|
||||||
|
task["status"] = "expired"
|
||||||
|
task["error"] = "expired"
|
||||||
|
task["updated_at"] = now
|
||||||
|
|
||||||
|
|
||||||
|
REGISTRY = TaskRegistry()
|
||||||
|
|
||||||
|
|
||||||
|
def _local_stub_result(request: TaskRequest) -> Dict[str, Any]:
|
||||||
|
"""Lokale Fallback-Implementierung ohne externe LLM-Abhängigkeit."""
|
||||||
|
return {
|
||||||
|
"output": f"{request.user_input} [local_stub]",
|
||||||
|
"provider": "local_stub",
|
||||||
|
"metadata": {
|
||||||
|
"mode": CONFIG.execution.mode,
|
||||||
|
"context_present": bool(request.context),
|
||||||
|
"metadata_present": bool(request.metadata),
|
||||||
|
},
|
||||||
|
}
|
||||||
|
|
||||||
|
|
||||||
|
def _invoke_llm(request: TaskRequest) -> Dict[str, Any]:
|
||||||
|
"""Ruft das konfigurierte LLM auf oder fällt auf die Stub-Implementierung zurück."""
|
||||||
|
api_key = CONFIG.llm.api_key
|
||||||
|
base_url = CONFIG.llm.api_base_url
|
||||||
|
if not api_key or not base_url:
|
||||||
|
return _local_stub_result(request)
|
||||||
|
try:
|
||||||
|
import httpx # type: ignore
|
||||||
|
except ModuleNotFoundError:
|
||||||
|
logger.warning("httpx nicht verfügbar, fallback auf lokalen Stub.")
|
||||||
|
return _local_stub_result(request)
|
||||||
|
headers: Dict[str, str] = {}
|
||||||
|
if api_key:
|
||||||
|
headers["Authorization"] = f"Bearer {api_key}"
|
||||||
|
payload = {
|
||||||
|
"model": CONFIG.llm.model,
|
||||||
|
"input": request.user_input,
|
||||||
|
"context": request.context or {},
|
||||||
|
"metadata": request.metadata or {},
|
||||||
|
}
|
||||||
|
timeout = CONFIG.llm.request_timeout_seconds
|
||||||
|
try:
|
||||||
|
with httpx.Client(timeout=timeout) as client:
|
||||||
|
response = client.post(str(base_url), json=payload, headers=headers)
|
||||||
|
response.raise_for_status()
|
||||||
|
data = response.json()
|
||||||
|
except Exception as exc:
|
||||||
|
logger.warning("LLM-Request fehlgeschlagen (%s), fallback auf Stub.", exc)
|
||||||
|
return _local_stub_result(request)
|
||||||
|
return {
|
||||||
|
"output": data.get("output") or data.get("result") or data,
|
||||||
|
"provider": CONFIG.llm.provider,
|
||||||
|
"metadata": {
|
||||||
|
"source": "llm",
|
||||||
|
"model": CONFIG.llm.model,
|
||||||
|
},
|
||||||
|
}
|
||||||
|
|
||||||
|
|
||||||
|
def _execute_request(request: TaskRequest) -> Dict[str, Any]:
|
||||||
|
"""Führt Vorverarbeitung aus und delegiert an das LLM."""
|
||||||
|
# Simulierter Pre-Processing-Schritt sowie Heartbeat-Update
|
||||||
|
time.sleep(0.01)
|
||||||
|
return _invoke_llm(request)
|
||||||
|
|
||||||
|
|
||||||
|
def _process_task(task_id: str) -> None:
|
||||||
|
"""Hintergrundverarbeitung einer Task."""
|
||||||
|
request = REGISTRY.get_request(task_id)
|
||||||
|
if request is None:
|
||||||
|
return
|
||||||
|
try:
|
||||||
|
REGISTRY.heartbeat(task_id)
|
||||||
|
result = _execute_request(request)
|
||||||
|
REGISTRY.heartbeat(task_id)
|
||||||
|
REGISTRY.mark_succeeded(task_id, result, CONFIG.execution.queue_ttl_seconds)
|
||||||
|
except Exception as exc:
|
||||||
|
logger.exception("Fehler bei der Bearbeitung von Task %s", task_id)
|
||||||
|
REGISTRY.mark_failed(task_id, str(exc), CONFIG.execution.queue_ttl_seconds)
|
||||||
|
|
||||||
|
|
||||||
|
def _serialize_task(task: TaskData, include_result: bool = True, include_detail: bool = True) -> Dict[str, Any]:
|
||||||
|
"""Konvertiert den internen Task-Status in eine API-Antwort."""
|
||||||
|
payload: Dict[str, Any] = {
|
||||||
|
"task_id": task["task_id"],
|
||||||
|
"status": task["status"],
|
||||||
|
"expires_at": _to_iso(task["expires_at"]),
|
||||||
|
}
|
||||||
|
if task["status"] == "processing":
|
||||||
|
payload["retry_after"] = task["retry_after"]
|
||||||
|
if include_result and task["result"] is not None and task["status"] in {"succeeded", "failed"}:
|
||||||
|
payload["result"] = task["result"]
|
||||||
|
if include_detail and task.get("error"):
|
||||||
|
payload["detail"] = task["error"]
|
||||||
|
if include_detail and task["status"] == "cancelled" and "detail" not in payload:
|
||||||
|
payload["detail"] = "cancelled"
|
||||||
|
return payload
|
||||||
|
|
||||||
|
|
||||||
|
agent_router = APIRouter(prefix="/agent/v1", tags=["agent-api"])
|
||||||
|
|
||||||
|
|
||||||
|
@agent_router.post("/tasks")
|
||||||
|
def submit_task(request: TaskRequest) -> JSONResponse:
|
||||||
|
"""Erstellt eine neue Task und verarbeitet sie je nach Modus synchron oder asynchron."""
|
||||||
|
effective_mode = CONFIG.execution.mode
|
||||||
|
if request.sync is not None:
|
||||||
|
effective_mode = "sync" if request.sync else "async"
|
||||||
|
ttl = CONFIG.execution.queue_ttl_seconds
|
||||||
|
retry_after = CONFIG.execution.heartbeat_interval_seconds
|
||||||
|
task = REGISTRY.create(request, ttl_seconds=ttl, retry_after=retry_after)
|
||||||
|
task_id = task["task_id"]
|
||||||
|
worker = threading.Thread(
|
||||||
|
target=_process_task,
|
||||||
|
args=(task_id,),
|
||||||
|
name=f"agent-task-{task_id}",
|
||||||
|
daemon=True,
|
||||||
|
)
|
||||||
|
worker.start()
|
||||||
|
if effective_mode == "sync":
|
||||||
|
worker.join(timeout=CONFIG.execution.response_timeout_seconds)
|
||||||
|
final_task = REGISTRY.get(task_id)
|
||||||
|
if final_task is None:
|
||||||
|
raise HTTPException(status_code=status.HTTP_500_INTERNAL_SERVER_ERROR, detail="task_not_found")
|
||||||
|
if final_task["status"] == "processing":
|
||||||
|
content = {
|
||||||
|
"task_id": task_id,
|
||||||
|
"status": "processing",
|
||||||
|
"detail": "still_running",
|
||||||
|
"expires_at": _to_iso(final_task["expires_at"]),
|
||||||
|
"retry_after": final_task["retry_after"],
|
||||||
|
}
|
||||||
|
return JSONResponse(status_code=status.HTTP_408_REQUEST_TIMEOUT, content=content)
|
||||||
|
return JSONResponse(status_code=status.HTTP_200_OK, content=_serialize_task(final_task))
|
||||||
|
return JSONResponse(status_code=status.HTTP_202_ACCEPTED, content=_serialize_task(task, include_result=False, include_detail=False))
|
||||||
|
|
||||||
|
|
||||||
|
@agent_router.get("/tasks/{task_id}")
|
||||||
|
def get_task_status(task_id: str) -> Dict[str, Any]:
|
||||||
|
"""Liefert den Status einer bestehenden Task."""
|
||||||
|
task = REGISTRY.get(task_id)
|
||||||
|
if task is None:
|
||||||
|
raise HTTPException(status_code=status.HTTP_404_NOT_FOUND, detail="task_not_found")
|
||||||
|
return _serialize_task(task)
|
||||||
|
|
||||||
|
|
||||||
|
@agent_router.post("/tasks/{task_id}/cancel")
|
||||||
|
def cancel_task(task_id: str) -> Dict[str, Any]:
|
||||||
|
"""Bricht eine laufende Task ab."""
|
||||||
|
task = REGISTRY.cancel(task_id)
|
||||||
|
if task is None:
|
||||||
|
raise HTTPException(status_code=status.HTTP_404_NOT_FOUND, detail="task_not_found")
|
||||||
|
return _serialize_task(task)
|
||||||
|
|
||||||
|
|
||||||
|
__all__ = ["agent_router"]
|
||||||
@@ -1,6 +0,0 @@
|
|||||||
"""Initialisierungspaket für das Agenten-API-Modul."""
|
|
||||||
from __future__ import annotations
|
|
||||||
|
|
||||||
from .router import agent_router
|
|
||||||
|
|
||||||
__all__ = ["agent_router"]
|
|
||||||
@@ -1,130 +0,0 @@
|
|||||||
"""Konfigurationslader für das Agenten-API-Modul."""
|
|
||||||
from __future__ import annotations
|
|
||||||
|
|
||||||
import copy
|
|
||||||
import os
|
|
||||||
import threading
|
|
||||||
from pathlib import Path
|
|
||||||
from typing import Any, Dict, Mapping, cast
|
|
||||||
|
|
||||||
import yaml
|
|
||||||
|
|
||||||
_ENV_LLM_KEY = "AGENT_API_LLM_KEY"
|
|
||||||
|
|
||||||
_DEFAULT_CONFIG: Dict[str, Any] = {
|
|
||||||
"agent_api": {
|
|
||||||
"metadata": {
|
|
||||||
"version": "0.1.0",
|
|
||||||
"description": "Agent Gateway für Worker + OpenAI-kompatible LLMs",
|
|
||||||
},
|
|
||||||
"http": {
|
|
||||||
"base_path": "/api/agent/v1",
|
|
||||||
"timeout_seconds": 60,
|
|
||||||
"rate_limit_per_minute": 120,
|
|
||||||
"enable_cors": True,
|
|
||||||
},
|
|
||||||
"auth": {
|
|
||||||
"api_key_header": "x-agent-api-key",
|
|
||||||
"allowed_keys": [],
|
|
||||||
"allow_unauthenticated": True,
|
|
||||||
},
|
|
||||||
"worker": {
|
|
||||||
"adapter": "inline",
|
|
||||||
"endpoint": None,
|
|
||||||
"timeout_seconds": 30,
|
|
||||||
},
|
|
||||||
"llm": {
|
|
||||||
"provider": "openai",
|
|
||||||
"base_url": "https://api.openai.com/v1",
|
|
||||||
"model": "gpt-4o-mini",
|
|
||||||
"temperature": 0.2,
|
|
||||||
"max_tokens": 1_200,
|
|
||||||
"api_key": None,
|
|
||||||
"request_timeout_seconds": 45,
|
|
||||||
"retry": {
|
|
||||||
"max_attempts": 3,
|
|
||||||
"backoff_seconds": 2,
|
|
||||||
},
|
|
||||||
},
|
|
||||||
"execution": {
|
|
||||||
"mode": "async",
|
|
||||||
"response_timeout_seconds": 30,
|
|
||||||
"queue_ttl_seconds": 300,
|
|
||||||
"heartbeat_interval_seconds": 10,
|
|
||||||
"allow_long_polling": True,
|
|
||||||
},
|
|
||||||
"logging": {
|
|
||||||
"enabled": True,
|
|
||||||
"log_payloads": False,
|
|
||||||
"redact_fields": ["user_input", "llm_response"],
|
|
||||||
},
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
_config_cache: Dict[str, Any] | None = None
|
|
||||||
_CACHE_LOCK = threading.Lock()
|
|
||||||
|
|
||||||
|
|
||||||
def _config_path() -> Path:
|
|
||||||
base_dir = Path(__file__).resolve().parents[2]
|
|
||||||
return base_dir / "config" / "agent_api.yaml"
|
|
||||||
|
|
||||||
|
|
||||||
def _load_file(path: Path) -> Dict[str, Any]:
|
|
||||||
if not path.exists():
|
|
||||||
return {}
|
|
||||||
with path.open("r", encoding="utf-8") as handle:
|
|
||||||
raw_data: Any = yaml.safe_load(handle) or {}
|
|
||||||
if not isinstance(raw_data, dict):
|
|
||||||
raise ValueError(f"Ungültiges Konfigurationsformat in {path}")
|
|
||||||
return cast(Dict[str, Any], raw_data)
|
|
||||||
|
|
||||||
|
|
||||||
def _deep_merge(base: Dict[str, Any], override: Mapping[str, Any]) -> Dict[str, Any]:
|
|
||||||
merged: Dict[str, Any] = copy.deepcopy(base)
|
|
||||||
for key, value in override.items():
|
|
||||||
if (
|
|
||||||
key in merged
|
|
||||||
and isinstance(merged[key], dict)
|
|
||||||
and isinstance(value, Mapping)
|
|
||||||
):
|
|
||||||
merged[key] = _deep_merge(merged[key], value) # type: ignore[arg-type]
|
|
||||||
else:
|
|
||||||
merged[key] = copy.deepcopy(value)
|
|
||||||
return merged
|
|
||||||
|
|
||||||
|
|
||||||
def _apply_env_overrides(config: Dict[str, Any]) -> None:
|
|
||||||
api_key = os.getenv(_ENV_LLM_KEY)
|
|
||||||
if api_key:
|
|
||||||
config.setdefault("agent_api", {}).setdefault("llm", {})["api_key"] = api_key
|
|
||||||
|
|
||||||
|
|
||||||
def _load_config_uncached() -> Dict[str, Any]:
|
|
||||||
file_data = _load_file(_config_path())
|
|
||||||
merged = _deep_merge(_DEFAULT_CONFIG, file_data)
|
|
||||||
_apply_env_overrides(merged)
|
|
||||||
return merged
|
|
||||||
|
|
||||||
|
|
||||||
def get_config(force_reload: bool = False) -> Dict[str, Any]:
|
|
||||||
"""
|
|
||||||
Liefert die Agenten-API-Konfiguration als Dictionary.
|
|
||||||
|
|
||||||
Args:
|
|
||||||
force_reload: Wenn True, wird die Datei erneut eingelesen.
|
|
||||||
|
|
||||||
Returns:
|
|
||||||
Dict[str, Any]: Zusammengeführte Konfiguration aus Defaults,
|
|
||||||
YAML-Datei und ENV-Overrides.
|
|
||||||
"""
|
|
||||||
global _config_cache
|
|
||||||
if force_reload:
|
|
||||||
with _CACHE_LOCK:
|
|
||||||
_config_cache = _load_config_uncached()
|
|
||||||
return copy.deepcopy(_config_cache)
|
|
||||||
if _config_cache is None:
|
|
||||||
with _CACHE_LOCK:
|
|
||||||
if _config_cache is None:
|
|
||||||
_config_cache = _load_config_uncached()
|
|
||||||
return copy.deepcopy(_config_cache)
|
|
||||||
@@ -1,113 +0,0 @@
|
|||||||
"""HTTP-basierter LLM-Client für das Agenten-API-Modul."""
|
|
||||||
from __future__ import annotations
|
|
||||||
|
|
||||||
import json
|
|
||||||
import time
|
|
||||||
from typing import Any, Dict, Mapping, Optional
|
|
||||||
|
|
||||||
import httpx
|
|
||||||
|
|
||||||
from .config import get_config
|
|
||||||
|
|
||||||
|
|
||||||
class LLMClient:
|
|
||||||
"""
|
|
||||||
Kapselt Aufrufe an ein OpenAI-kompatibles REST-Interface.
|
|
||||||
|
|
||||||
Die Implementierung unterstützt einfache Retries, Timeout-Handling sowie
|
|
||||||
einen deterministischen Stub, falls kein API-Key konfiguriert wurde.
|
|
||||||
"""
|
|
||||||
|
|
||||||
def __init__(self, config: Optional[Mapping[str, Any]] = None) -> None:
|
|
||||||
cfg = dict(config or get_config())
|
|
||||||
agent_cfg = cfg.get("agent_api", {})
|
|
||||||
llm_cfg: Dict[str, Any] = dict(agent_cfg.get("llm", {}))
|
|
||||||
|
|
||||||
self.base_url: str = str(llm_cfg.get("base_url", "https://api.openai.com/v1")).rstrip("/")
|
|
||||||
self.model: str = str(llm_cfg.get("model", "gpt-4o-mini"))
|
|
||||||
self.temperature: float = float(llm_cfg.get("temperature", 0.2))
|
|
||||||
self.max_tokens: int = int(llm_cfg.get("max_tokens", 1_200))
|
|
||||||
self.api_key: Optional[str] = llm_cfg.get("api_key")
|
|
||||||
self.request_timeout_seconds: float = float(llm_cfg.get("request_timeout_seconds", 45))
|
|
||||||
retry_cfg: Dict[str, Any] = dict(llm_cfg.get("retry", {}))
|
|
||||||
self.max_attempts: int = max(1, int(retry_cfg.get("max_attempts", 3)))
|
|
||||||
self.backoff_seconds: float = float(retry_cfg.get("backoff_seconds", 2))
|
|
||||||
|
|
||||||
def generate(self, prompt: Dict[str, Any]) -> Dict[str, Any]:
|
|
||||||
"""
|
|
||||||
Sendet den Prompt an den konfigurierten LLM-Endpunkt oder liefert eine Stub-Antwort.
|
|
||||||
|
|
||||||
Args:
|
|
||||||
prompt: Strukturierter Prompt, erzeugt durch den WorkerAdapter.
|
|
||||||
|
|
||||||
Returns:
|
|
||||||
Dict[str, Any]: Antwort-Payload, der den OpenAI-ähnlichen Response widerspiegelt.
|
|
||||||
"""
|
|
||||||
if not self.api_key:
|
|
||||||
return self._local_stub(prompt)
|
|
||||||
|
|
||||||
url = f"{self.base_url}/chat/completions"
|
|
||||||
payload = {
|
|
||||||
"model": self.model,
|
|
||||||
"messages": self._build_messages(prompt),
|
|
||||||
"temperature": self.temperature,
|
|
||||||
"max_tokens": self.max_tokens,
|
|
||||||
}
|
|
||||||
headers = {
|
|
||||||
"Authorization": f"Bearer {self.api_key}",
|
|
||||||
"Content-Type": "application/json",
|
|
||||||
}
|
|
||||||
|
|
||||||
last_error: Optional[BaseException] = None
|
|
||||||
for attempt in range(1, self.max_attempts + 1):
|
|
||||||
try:
|
|
||||||
with httpx.Client(timeout=self.request_timeout_seconds) as client:
|
|
||||||
response = client.post(url, headers=headers, json=payload)
|
|
||||||
response.raise_for_status()
|
|
||||||
return response.json()
|
|
||||||
except (httpx.HTTPError, httpx.TimeoutException) as exc:
|
|
||||||
last_error = exc
|
|
||||||
if attempt >= self.max_attempts:
|
|
||||||
break
|
|
||||||
time.sleep(self.backoff_seconds)
|
|
||||||
|
|
||||||
if last_error:
|
|
||||||
raise RuntimeError(f"LLM request failed after {self.max_attempts} attempts") from last_error
|
|
||||||
raise RuntimeError("LLM request failed unexpectedly without additional context")
|
|
||||||
|
|
||||||
@staticmethod
|
|
||||||
def _build_messages(prompt: Mapping[str, Any]) -> Any:
|
|
||||||
user_input = str(prompt.get("user_input", ""))
|
|
||||||
context = prompt.get("context") or {}
|
|
||||||
metadata = prompt.get("metadata") or {}
|
|
||||||
parts = [
|
|
||||||
{"role": "system", "content": json.dumps({"context": context, "metadata": metadata})},
|
|
||||||
{"role": "user", "content": user_input},
|
|
||||||
]
|
|
||||||
return parts
|
|
||||||
|
|
||||||
@staticmethod
|
|
||||||
def _local_stub(prompt: Mapping[str, Any]) -> Dict[str, Any]:
|
|
||||||
user_input = prompt.get("user_input", "")
|
|
||||||
return {
|
|
||||||
"id": "local_stub_completion",
|
|
||||||
"object": "chat.completion",
|
|
||||||
"created": int(time.time()),
|
|
||||||
"model": "local-stub",
|
|
||||||
"choices": [
|
|
||||||
{
|
|
||||||
"index": 0,
|
|
||||||
"message": {
|
|
||||||
"role": "assistant",
|
|
||||||
"content": f"[local_stub] Echo: {user_input}",
|
|
||||||
},
|
|
||||||
"finish_reason": "stop",
|
|
||||||
}
|
|
||||||
],
|
|
||||||
"usage": {
|
|
||||||
"prompt_tokens": len(json.dumps(prompt)),
|
|
||||||
"completion_tokens": len(str(user_input)),
|
|
||||||
"total_tokens": len(json.dumps(prompt)) + len(str(user_input)),
|
|
||||||
},
|
|
||||||
"meta": {"note": "local_stub"},
|
|
||||||
}
|
|
||||||
@@ -1,49 +0,0 @@
|
|||||||
"""Pydantic-Modelle für das Agenten-API-Modul."""
|
|
||||||
from __future__ import annotations
|
|
||||||
|
|
||||||
from datetime import datetime
|
|
||||||
from typing import Any, Dict, Optional
|
|
||||||
|
|
||||||
from pydantic import BaseModel, Field
|
|
||||||
|
|
||||||
|
|
||||||
class TaskCreate(BaseModel):
|
|
||||||
"""
|
|
||||||
Repräsentiert den Payload zur Erstellung eines neuen Agenten-Tasks.
|
|
||||||
"""
|
|
||||||
|
|
||||||
user_input: str = Field(..., description="Freitext-Anfrage oder Prompt des Nutzers.")
|
|
||||||
context: Optional[Dict[str, Any]] = Field(
|
|
||||||
None, description="Optionaler Kontext mit zusätzlichen strukturierten Daten."
|
|
||||||
)
|
|
||||||
metadata: Optional[Dict[str, Any]] = Field(
|
|
||||||
None, description="Beliebige Metadaten zur Nachverfolgung von Tasks."
|
|
||||||
)
|
|
||||||
sync: Optional[bool] = Field(
|
|
||||||
None,
|
|
||||||
description=(
|
|
||||||
"Optionaler Override, um den Task synchron zu verarbeiten, "
|
|
||||||
"selbst wenn die Ausführung standardmäßig asynchron erfolgt."
|
|
||||||
),
|
|
||||||
)
|
|
||||||
|
|
||||||
|
|
||||||
class TaskStatusResponse(BaseModel):
|
|
||||||
"""
|
|
||||||
Antwortmodell für den Status eines bestehenden Tasks.
|
|
||||||
"""
|
|
||||||
|
|
||||||
task_id: str = Field(..., description="Eindeutige Kennung des Tasks.")
|
|
||||||
status: str = Field(..., description="Aktueller Status des Tasks.")
|
|
||||||
result: Optional[Dict[str, Any]] = Field(
|
|
||||||
None,
|
|
||||||
description="Ergebnisdaten des Tasks, sobald verfügbar.",
|
|
||||||
)
|
|
||||||
expires_at: Optional[datetime] = Field(
|
|
||||||
None,
|
|
||||||
description="Zeitpunkt, an dem der Task automatisch ausläuft.",
|
|
||||||
)
|
|
||||||
retry_after: Optional[int] = Field(
|
|
||||||
None,
|
|
||||||
description="Empfohlene Wartezeit in Sekunden für den nächsten Statusabruf.",
|
|
||||||
)
|
|
||||||
@@ -1,88 +0,0 @@
|
|||||||
"""FastAPI-Router für das Agenten-API-Modul."""
|
|
||||||
from __future__ import annotations
|
|
||||||
|
|
||||||
from typing import Dict, Optional
|
|
||||||
|
|
||||||
from fastapi import APIRouter, HTTPException, Response, status
|
|
||||||
from fastapi.responses import JSONResponse
|
|
||||||
|
|
||||||
from .models import TaskCreate
|
|
||||||
from .service import AgentService
|
|
||||||
|
|
||||||
agent_router = APIRouter(prefix="/agent/v1", tags=["agent_api"])
|
|
||||||
|
|
||||||
_service = AgentService()
|
|
||||||
|
|
||||||
|
|
||||||
def _build_response(
|
|
||||||
payload: Dict[str, Optional[str]],
|
|
||||||
status_code: int,
|
|
||||||
retry_after: Optional[int] = None,
|
|
||||||
) -> JSONResponse:
|
|
||||||
headers = {}
|
|
||||||
if retry_after is not None:
|
|
||||||
headers["Retry-After"] = str(retry_after)
|
|
||||||
return JSONResponse(content=payload, status_code=status_code, headers=headers)
|
|
||||||
|
|
||||||
|
|
||||||
@agent_router.post(
|
|
||||||
"/tasks",
|
|
||||||
response_model_exclude_none=True,
|
|
||||||
summary="Neuen Task einreichen",
|
|
||||||
)
|
|
||||||
async def submit_task(task: TaskCreate) -> JSONResponse:
|
|
||||||
"""
|
|
||||||
Legt einen neuen Task an und startet die Verarbeitung.
|
|
||||||
"""
|
|
||||||
result = _service.submit_task(task)
|
|
||||||
response_payload = result.response.model_dump()
|
|
||||||
if result.detail:
|
|
||||||
response_payload["detail"] = result.detail
|
|
||||||
return _build_response(response_payload, result.status_code, result.response.retry_after)
|
|
||||||
|
|
||||||
|
|
||||||
@agent_router.get(
|
|
||||||
"/tasks/{task_id}",
|
|
||||||
response_model_exclude_none=True,
|
|
||||||
summary="Status eines Tasks abrufen",
|
|
||||||
)
|
|
||||||
async def get_status(task_id: str) -> JSONResponse:
|
|
||||||
"""
|
|
||||||
Liefert den Status eines Tasks.
|
|
||||||
"""
|
|
||||||
status_map = {
|
|
||||||
"processing": status.HTTP_202_ACCEPTED,
|
|
||||||
"succeeded": status.HTTP_200_OK,
|
|
||||||
"failed": status.HTTP_500_INTERNAL_SERVER_ERROR,
|
|
||||||
"cancelled": status.HTTP_409_CONFLICT,
|
|
||||||
"expired": status.HTTP_410_GONE,
|
|
||||||
}
|
|
||||||
try:
|
|
||||||
status_response = _service.get_status(task_id)
|
|
||||||
except KeyError as exc:
|
|
||||||
raise HTTPException(status_code=status.HTTP_404_NOT_FOUND, detail=str(exc)) from exc
|
|
||||||
|
|
||||||
status_code = status_map.get(status_response.status, status.HTTP_200_OK)
|
|
||||||
payload = status_response.model_dump()
|
|
||||||
if status_response.status in {"failed", "cancelled", "expired"}:
|
|
||||||
payload.setdefault("detail", status_response.status)
|
|
||||||
return _build_response(payload, status_code, status_response.retry_after)
|
|
||||||
|
|
||||||
|
|
||||||
@agent_router.post(
|
|
||||||
"/tasks/{task_id}/cancel",
|
|
||||||
response_model_exclude_none=True,
|
|
||||||
summary="Laufenden Task abbrechen",
|
|
||||||
)
|
|
||||||
async def cancel_task(task_id: str) -> JSONResponse:
|
|
||||||
"""
|
|
||||||
Markiert einen laufenden Task als abgebrochen.
|
|
||||||
"""
|
|
||||||
try:
|
|
||||||
status_response = _service.cancel_task(task_id)
|
|
||||||
except KeyError as exc:
|
|
||||||
raise HTTPException(status_code=status.HTTP_404_NOT_FOUND, detail=str(exc)) from exc
|
|
||||||
|
|
||||||
payload = status_response.model_dump()
|
|
||||||
payload.setdefault("detail", "task_cancelled")
|
|
||||||
return _build_response(payload, status.HTTP_200_OK, status_response.retry_after)
|
|
||||||
@@ -1,310 +0,0 @@
|
|||||||
"""Service-Schicht für das Agenten-API-Modul."""
|
|
||||||
from __future__ import annotations
|
|
||||||
|
|
||||||
import threading
|
|
||||||
import uuid
|
|
||||||
from dataclasses import dataclass, field
|
|
||||||
from datetime import datetime, timedelta, timezone
|
|
||||||
from typing import Any, Dict, Mapping, Optional
|
|
||||||
|
|
||||||
from .config import get_config
|
|
||||||
from .llm_client import LLMClient
|
|
||||||
from .models import TaskCreate, TaskStatusResponse
|
|
||||||
from .worker_adapter import WorkerAdapter
|
|
||||||
|
|
||||||
_PROCESSING_STATUS = "processing"
|
|
||||||
_SUCCEEDED_STATUS = "succeeded"
|
|
||||||
_FAILED_STATUS = "failed"
|
|
||||||
_CANCELLED_STATUS = "cancelled"
|
|
||||||
_EXPIRED_STATUS = "expired"
|
|
||||||
|
|
||||||
|
|
||||||
@dataclass
|
|
||||||
class _TaskEntry:
|
|
||||||
payload: TaskCreate
|
|
||||||
status: str = _PROCESSING_STATUS
|
|
||||||
result: Optional[Dict[str, Any]] = None
|
|
||||||
created_at: datetime = field(
|
|
||||||
default_factory=lambda: datetime.now(timezone.utc)
|
|
||||||
)
|
|
||||||
expires_at: datetime = field(
|
|
||||||
default_factory=lambda: datetime.now(timezone.utc) + timedelta(minutes=5)
|
|
||||||
)
|
|
||||||
last_heartbeat: datetime = field(
|
|
||||||
default_factory=lambda: datetime.now(timezone.utc)
|
|
||||||
)
|
|
||||||
detail: Optional[str] = None
|
|
||||||
retry_after: Optional[int] = None
|
|
||||||
error: Optional[str] = None
|
|
||||||
|
|
||||||
def to_response(self, task_id: str) -> TaskStatusResponse:
|
|
||||||
return TaskStatusResponse(
|
|
||||||
task_id=task_id,
|
|
||||||
status=self.status,
|
|
||||||
result=self.result,
|
|
||||||
expires_at=self.expires_at if self.expires_at.tzinfo else self.expires_at.replace(tzinfo=timezone.utc),
|
|
||||||
retry_after=self.retry_after,
|
|
||||||
)
|
|
||||||
|
|
||||||
|
|
||||||
@dataclass
|
|
||||||
class SubmitResult:
|
|
||||||
response: TaskStatusResponse
|
|
||||||
status_code: int
|
|
||||||
detail: Optional[str] = None
|
|
||||||
|
|
||||||
|
|
||||||
class AgentService:
|
|
||||||
"""
|
|
||||||
Koordiniert die Lebenszyklen von Tasks, Worker-Adapter und LLM-Client.
|
|
||||||
"""
|
|
||||||
|
|
||||||
def __init__(
|
|
||||||
self,
|
|
||||||
*,
|
|
||||||
worker_adapter: Optional[WorkerAdapter] = None,
|
|
||||||
llm_client: Optional[LLMClient] = None,
|
|
||||||
) -> None:
|
|
||||||
self._lock = threading.RLock()
|
|
||||||
self._tasks: Dict[str, _TaskEntry] = {}
|
|
||||||
self._worker = worker_adapter or WorkerAdapter()
|
|
||||||
self._llm_client = llm_client or LLMClient()
|
|
||||||
|
|
||||||
# ------------------------------------------------------------------ #
|
|
||||||
# Öffentliche API
|
|
||||||
# ------------------------------------------------------------------ #
|
|
||||||
|
|
||||||
def submit_task(self, payload: TaskCreate) -> SubmitResult:
|
|
||||||
"""
|
|
||||||
Legt einen neuen Task an und triggert die Verarbeitung entsprechend
|
|
||||||
der konfigurierten Ausführungsstrategie.
|
|
||||||
"""
|
|
||||||
config = self._current_config()
|
|
||||||
execution_cfg = self._execution_config(config)
|
|
||||||
queue_ttl = int(execution_cfg.get("queue_ttl_seconds", 300))
|
|
||||||
retry_after = int(execution_cfg.get("heartbeat_interval_seconds", 10))
|
|
||||||
timeout_seconds = float(execution_cfg.get("response_timeout_seconds", 30))
|
|
||||||
default_mode = str(execution_cfg.get("mode", "async")).lower()
|
|
||||||
|
|
||||||
mode = default_mode
|
|
||||||
if payload.sync is True:
|
|
||||||
mode = "sync"
|
|
||||||
elif payload.sync is False:
|
|
||||||
mode = "async"
|
|
||||||
|
|
||||||
now = datetime.now(timezone.utc)
|
|
||||||
task_id = str(uuid.uuid4())
|
|
||||||
entry = _TaskEntry(
|
|
||||||
payload=payload,
|
|
||||||
status=_PROCESSING_STATUS,
|
|
||||||
created_at=now,
|
|
||||||
expires_at=now + timedelta(seconds=queue_ttl),
|
|
||||||
last_heartbeat=now,
|
|
||||||
retry_after=retry_after,
|
|
||||||
)
|
|
||||||
|
|
||||||
with self._lock:
|
|
||||||
self._tasks[task_id] = entry
|
|
||||||
|
|
||||||
thread = threading.Thread(
|
|
||||||
target=self.process_task,
|
|
||||||
args=(task_id,),
|
|
||||||
name=f"agent-task-{task_id}",
|
|
||||||
daemon=True,
|
|
||||||
)
|
|
||||||
thread.start()
|
|
||||||
|
|
||||||
if mode == "async":
|
|
||||||
return SubmitResult(
|
|
||||||
response=entry.to_response(task_id),
|
|
||||||
status_code=202,
|
|
||||||
detail=None,
|
|
||||||
)
|
|
||||||
|
|
||||||
# synchroner Modus
|
|
||||||
thread.join(timeout_seconds)
|
|
||||||
if thread.is_alive():
|
|
||||||
return SubmitResult(
|
|
||||||
response=self._task_response(task_id),
|
|
||||||
status_code=408,
|
|
||||||
detail="still_running",
|
|
||||||
)
|
|
||||||
|
|
||||||
entry = self._task_entry(task_id)
|
|
||||||
if entry.status == _SUCCEEDED_STATUS:
|
|
||||||
return SubmitResult(
|
|
||||||
response=entry.to_response(task_id),
|
|
||||||
status_code=200,
|
|
||||||
detail=None,
|
|
||||||
)
|
|
||||||
if entry.status == _FAILED_STATUS:
|
|
||||||
return SubmitResult(
|
|
||||||
response=entry.to_response(task_id),
|
|
||||||
status_code=500,
|
|
||||||
detail=entry.detail or "task_failed",
|
|
||||||
)
|
|
||||||
if entry.status == _CANCELLED_STATUS:
|
|
||||||
return SubmitResult(
|
|
||||||
response=entry.to_response(task_id),
|
|
||||||
status_code=409,
|
|
||||||
detail=entry.detail or "task_cancelled",
|
|
||||||
)
|
|
||||||
if entry.status == _EXPIRED_STATUS:
|
|
||||||
return SubmitResult(
|
|
||||||
response=entry.to_response(task_id),
|
|
||||||
status_code=410,
|
|
||||||
detail=entry.detail or "task_expired",
|
|
||||||
)
|
|
||||||
return SubmitResult(
|
|
||||||
response=entry.to_response(task_id),
|
|
||||||
status_code=202,
|
|
||||||
detail=None,
|
|
||||||
)
|
|
||||||
|
|
||||||
def process_task(self, task_id: str) -> None:
|
|
||||||
"""
|
|
||||||
Führt den Worker- und LLM-Workflow für einen gegebenen Task aus.
|
|
||||||
"""
|
|
||||||
try:
|
|
||||||
entry = self._task_entry(task_id)
|
|
||||||
except KeyError:
|
|
||||||
return
|
|
||||||
|
|
||||||
if entry.status in {_CANCELLED_STATUS, _EXPIRED_STATUS}:
|
|
||||||
return
|
|
||||||
|
|
||||||
self.update_heartbeat(task_id)
|
|
||||||
try:
|
|
||||||
prepared = self._worker.pre_process(entry.payload)
|
|
||||||
prompt = self._worker.run_task(prepared)
|
|
||||||
llm_response = self._llm_client.generate(prompt)
|
|
||||||
|
|
||||||
result_payload = {
|
|
||||||
"prepared_prompt": prompt,
|
|
||||||
"llm_response": llm_response,
|
|
||||||
}
|
|
||||||
now = datetime.now(timezone.utc)
|
|
||||||
with self._lock:
|
|
||||||
stored = self._tasks.get(task_id)
|
|
||||||
if stored is None or stored.status in {_CANCELLED_STATUS, _EXPIRED_STATUS}:
|
|
||||||
return
|
|
||||||
stored.status = _SUCCEEDED_STATUS
|
|
||||||
stored.result = result_payload
|
|
||||||
stored.last_heartbeat = now
|
|
||||||
stored.detail = None
|
|
||||||
stored.retry_after = None
|
|
||||||
except Exception as exc: # pragma: no cover - Fehlerpfad
|
|
||||||
now = datetime.now(timezone.utc)
|
|
||||||
with self._lock:
|
|
||||||
stored = self._tasks.get(task_id)
|
|
||||||
if stored is None:
|
|
||||||
return
|
|
||||||
if stored.status not in {_CANCELLED_STATUS, _EXPIRED_STATUS}:
|
|
||||||
stored.status = _FAILED_STATUS
|
|
||||||
stored.result = {"error": str(exc)}
|
|
||||||
stored.detail = exc.__class__.__name__
|
|
||||||
stored.error = repr(exc)
|
|
||||||
stored.last_heartbeat = now
|
|
||||||
stored.retry_after = None
|
|
||||||
finally:
|
|
||||||
self._finalize_task(task_id)
|
|
||||||
|
|
||||||
def get_status(self, task_id: str) -> TaskStatusResponse:
|
|
||||||
"""
|
|
||||||
Liefert den aktuellen Status eines Tasks.
|
|
||||||
"""
|
|
||||||
entry = self._task_entry(task_id)
|
|
||||||
self._evaluate_expiration(task_id, entry)
|
|
||||||
return self._task_response(task_id)
|
|
||||||
|
|
||||||
def cancel_task(self, task_id: str) -> TaskStatusResponse:
|
|
||||||
"""
|
|
||||||
Markiert einen laufenden Task als abgebrochen.
|
|
||||||
"""
|
|
||||||
entry = self._task_entry(task_id)
|
|
||||||
with self._lock:
|
|
||||||
if entry.status in {_SUCCEEDED_STATUS, _FAILED_STATUS, _EXPIRED_STATUS}:
|
|
||||||
return entry.to_response(task_id)
|
|
||||||
entry.status = _CANCELLED_STATUS
|
|
||||||
entry.result = entry.result or {"detail": "Task cancelled by client"}
|
|
||||||
entry.detail = "task_cancelled"
|
|
||||||
entry.last_heartbeat = datetime.now(timezone.utc)
|
|
||||||
entry.retry_after = None
|
|
||||||
return self._task_response(task_id)
|
|
||||||
|
|
||||||
def update_heartbeat(self, task_id: str) -> None:
|
|
||||||
"""
|
|
||||||
Aktualisiert den Heartbeat eines Tasks und verlängert dessen TTL.
|
|
||||||
"""
|
|
||||||
entry = self._task_entry(task_id)
|
|
||||||
config = self._current_config()
|
|
||||||
execution_cfg = self._execution_config(config)
|
|
||||||
queue_ttl = int(execution_cfg.get("queue_ttl_seconds", 300))
|
|
||||||
now = datetime.now(timezone.utc)
|
|
||||||
with self._lock:
|
|
||||||
stored = self._tasks.get(task_id)
|
|
||||||
if stored is None:
|
|
||||||
return
|
|
||||||
stored.last_heartbeat = now
|
|
||||||
stored.expires_at = now + timedelta(seconds=queue_ttl)
|
|
||||||
|
|
||||||
# ------------------------------------------------------------------ #
|
|
||||||
# Interne Hilfsfunktionen
|
|
||||||
# ------------------------------------------------------------------ #
|
|
||||||
|
|
||||||
def _task_entry(self, task_id: str) -> _TaskEntry:
|
|
||||||
with self._lock:
|
|
||||||
entry = self._tasks.get(task_id)
|
|
||||||
if entry is None:
|
|
||||||
raise KeyError(f"Task '{task_id}' not found")
|
|
||||||
return entry
|
|
||||||
|
|
||||||
def _task_response(self, task_id: str) -> TaskStatusResponse:
|
|
||||||
entry = self._task_entry(task_id)
|
|
||||||
return entry.to_response(task_id)
|
|
||||||
|
|
||||||
def _finalize_task(self, task_id: str) -> None:
|
|
||||||
entry = self._task_entry(task_id)
|
|
||||||
self._evaluate_expiration(task_id, entry)
|
|
||||||
|
|
||||||
def _evaluate_expiration(self, task_id: str, entry: _TaskEntry) -> None:
|
|
||||||
now = datetime.now(timezone.utc)
|
|
||||||
if entry.status in {_SUCCEEDED_STATUS, _FAILED_STATUS, _CANCELLED_STATUS}:
|
|
||||||
return
|
|
||||||
if now > entry.expires_at:
|
|
||||||
with self._lock:
|
|
||||||
stored = self._tasks.get(task_id)
|
|
||||||
if stored is None:
|
|
||||||
return
|
|
||||||
stored.status = _EXPIRED_STATUS
|
|
||||||
stored.result = stored.result or {"detail": "Task expired"}
|
|
||||||
stored.detail = "task_expired"
|
|
||||||
stored.last_heartbeat = now
|
|
||||||
stored.retry_after = None
|
|
||||||
|
|
||||||
@staticmethod
|
|
||||||
def _current_config() -> Dict[str, Any]:
|
|
||||||
return get_config()
|
|
||||||
|
|
||||||
@staticmethod
|
|
||||||
def _execution_config(config: Mapping[str, Any]) -> Mapping[str, Any]:
|
|
||||||
agent_cfg = config.get("agent_api", {})
|
|
||||||
return agent_cfg.get("execution", {})
|
|
||||||
|
|
||||||
# ------------------------------------------------------------------ #
|
|
||||||
# Debug-/Test-Hilfen
|
|
||||||
# ------------------------------------------------------------------ #
|
|
||||||
|
|
||||||
def clear_tasks(self) -> None:
|
|
||||||
"""
|
|
||||||
Löscht alle gemerkten Tasks. Hilfreich für Tests.
|
|
||||||
"""
|
|
||||||
with self._lock:
|
|
||||||
self._tasks.clear()
|
|
||||||
|
|
||||||
def list_tasks(self) -> Dict[str, _TaskEntry]:
|
|
||||||
"""
|
|
||||||
Gibt einen Snapshot der aktuellen Tasks zurück.
|
|
||||||
"""
|
|
||||||
with self._lock:
|
|
||||||
return dict(self._tasks)
|
|
||||||
@@ -1,50 +0,0 @@
|
|||||||
"""Worker-Adapter für das Agenten-API-Modul."""
|
|
||||||
from __future__ import annotations
|
|
||||||
|
|
||||||
import datetime as _dt
|
|
||||||
from typing import Any, Dict
|
|
||||||
|
|
||||||
from .models import TaskCreate
|
|
||||||
|
|
||||||
|
|
||||||
class WorkerAdapter:
|
|
||||||
"""
|
|
||||||
Einfacher Inline-Worker-Adapter.
|
|
||||||
|
|
||||||
Die Implementierung dient als Stub für spätere Erweiterungen. Sie nimmt den
|
|
||||||
eingehenden Payload entgegen, reichert diesen mit Metadaten an und liefert
|
|
||||||
einen Prompt für den LLM-Client zurück.
|
|
||||||
"""
|
|
||||||
|
|
||||||
@staticmethod
|
|
||||||
def pre_process(payload: TaskCreate) -> Dict[str, Any]:
|
|
||||||
"""
|
|
||||||
Bereitet den Task-Payload für den LLM-Aufruf vor.
|
|
||||||
|
|
||||||
Args:
|
|
||||||
payload: Valider TaskCreate-Payload.
|
|
||||||
|
|
||||||
Returns:
|
|
||||||
Dict[str, Any]: Strukturierter Prompt inklusive Metadaten.
|
|
||||||
"""
|
|
||||||
timestamp = _dt.datetime.now(_dt.timezone.utc)
|
|
||||||
prepared_prompt: Dict[str, Any] = {
|
|
||||||
"user_input": payload.user_input,
|
|
||||||
"context": payload.context or {},
|
|
||||||
"metadata": payload.metadata or {},
|
|
||||||
"created_at": timestamp.isoformat(),
|
|
||||||
}
|
|
||||||
return prepared_prompt
|
|
||||||
|
|
||||||
@staticmethod
|
|
||||||
def run_task(prepared: Dict[str, Any]) -> Dict[str, Any]:
|
|
||||||
"""
|
|
||||||
Führt optionale Worker-Logik aus und liefert den Prompt für den LLM-Client.
|
|
||||||
|
|
||||||
Args:
|
|
||||||
prepared: Vom WorkerAdapter.pre_process erzeugter Prompt.
|
|
||||||
|
|
||||||
Returns:
|
|
||||||
Dict[str, Any]: Prompt für den LLM-Client.
|
|
||||||
"""
|
|
||||||
return prepared
|
|
||||||
1
work/entworfener_code_working_copy/.venv/bin/python
Symbolic link
1
work/entworfener_code_working_copy/.venv/bin/python
Symbolic link
@@ -0,0 +1 @@
|
|||||||
|
python3
|
||||||
1
work/entworfener_code_working_copy/.venv/bin/python3
Symbolic link
1
work/entworfener_code_working_copy/.venv/bin/python3
Symbolic link
@@ -0,0 +1 @@
|
|||||||
|
/usr/bin/python3
|
||||||
1
work/entworfener_code_working_copy/.venv/bin/python3.10
Symbolic link
1
work/entworfener_code_working_copy/.venv/bin/python3.10
Symbolic link
@@ -0,0 +1 @@
|
|||||||
|
python3
|
||||||
1
work/entworfener_code_working_copy/.venv/lib64
Symbolic link
1
work/entworfener_code_working_copy/.venv/lib64
Symbolic link
@@ -0,0 +1 @@
|
|||||||
|
lib
|
||||||
3
work/entworfener_code_working_copy/.venv/pyvenv.cfg
Normal file
3
work/entworfener_code_working_copy/.venv/pyvenv.cfg
Normal file
@@ -0,0 +1,3 @@
|
|||||||
|
home = /usr/bin
|
||||||
|
include-system-site-packages = false
|
||||||
|
version = 3.10.12
|
||||||
1
work/entworfener_code_working_copy/1
Normal file
1
work/entworfener_code_working_copy/1
Normal file
@@ -0,0 +1 @@
|
|||||||
|
/usr/bin/python3
|
||||||
33
work/entworfener_code_working_copy/Dockerfile
Normal file
33
work/entworfener_code_working_copy/Dockerfile
Normal file
@@ -0,0 +1,33 @@
|
|||||||
|
# syntax=docker/dockerfile:1
|
||||||
|
ARG PYTHON_VERSION=3.11
|
||||||
|
|
||||||
|
FROM python:${PYTHON_VERSION}-slim AS base
|
||||||
|
|
||||||
|
ENV PYTHONDONTWRITEBYTECODE=1 \
|
||||||
|
PYTHONUNBUFFERED=1
|
||||||
|
|
||||||
|
# Optional: Systemabhängigkeiten (Minimalsatz)
|
||||||
|
RUN apt-get update && apt-get install -y --no-install-recommends \
|
||||||
|
ca-certificates curl build-essential \
|
||||||
|
&& rm -rf /var/lib/apt/lists/*
|
||||||
|
|
||||||
|
WORKDIR /app
|
||||||
|
|
||||||
|
# Dependencies zuerst kopieren für bessere Layer-Caching-Effizienz
|
||||||
|
COPY requirements.txt /app/requirements.txt
|
||||||
|
RUN pip install --no-cache-dir -r /app/requirements.txt
|
||||||
|
|
||||||
|
# Anwendungscode
|
||||||
|
COPY . /app
|
||||||
|
|
||||||
|
# Non-root User
|
||||||
|
RUN useradd -m appuser && chown -R appuser:appuser /app
|
||||||
|
USER appuser
|
||||||
|
|
||||||
|
# Standard-Config-Pfad für die App-Factory (neue Struktur unter /app/app/config)
|
||||||
|
ENV APP_CONFIG_PATH=/app/app/config/config.yaml
|
||||||
|
|
||||||
|
EXPOSE 8000
|
||||||
|
|
||||||
|
# Start über Uvicorn-Factory aus neuer Struktur: app/start.py stellt app_factory() bereit
|
||||||
|
CMD ["uvicorn", "app.start:app_factory", "--factory", "--host", "0.0.0.0", "--port", "8000"]
|
||||||
Binary file not shown.
Binary file not shown.
67
work/entworfener_code_working_copy/app/code/app/main.py
Normal file
67
work/entworfener_code_working_copy/app/code/app/main.py
Normal file
@@ -0,0 +1,67 @@
|
|||||||
|
from typing import Any, Dict, List, Optional
|
||||||
|
|
||||||
|
from fastapi import FastAPI
|
||||||
|
from fastapi.middleware.cors import CORSMiddleware
|
||||||
|
|
||||||
|
|
||||||
|
def create_app(config: Optional[Dict[str, Any]] = None) -> FastAPI:
|
||||||
|
"""
|
||||||
|
Erzeugt und konfiguriert die FastAPI-Anwendung.
|
||||||
|
|
||||||
|
Args:
|
||||||
|
config: Konfigurations-Dictionary (z. B. aus app/config/config.yaml geladen).
|
||||||
|
|
||||||
|
Returns:
|
||||||
|
FastAPI: Konfigurierte FastAPI-App.
|
||||||
|
"""
|
||||||
|
cfg: Dict[str, Any] = config or {}
|
||||||
|
meta: Dict[str, Any] = cfg.get("meta", {})
|
||||||
|
app_cfg: Dict[str, Any] = cfg.get("app", {})
|
||||||
|
cors_cfg: Dict[str, Any] = cfg.get("cors", {})
|
||||||
|
|
||||||
|
app = FastAPI(
|
||||||
|
title=app_cfg.get("name", "entworfener_code_service"),
|
||||||
|
version=str(meta.get("version", "0.1.0")),
|
||||||
|
docs_url="/docs",
|
||||||
|
redoc_url="/redoc",
|
||||||
|
)
|
||||||
|
|
||||||
|
# CORS
|
||||||
|
allow_origins: List[str] = cors_cfg.get("allow_origins", ["*"])
|
||||||
|
allow_credentials: bool = bool(cors_cfg.get("allow_credentials", True))
|
||||||
|
allow_methods: List[str] = cors_cfg.get("allow_methods", ["*"])
|
||||||
|
allow_headers: List[str] = cors_cfg.get("allow_headers", ["*"])
|
||||||
|
|
||||||
|
app.add_middleware(
|
||||||
|
CORSMiddleware,
|
||||||
|
allow_origins=allow_origins,
|
||||||
|
allow_credentials=allow_credentials,
|
||||||
|
allow_methods=allow_methods,
|
||||||
|
allow_headers=allow_headers,
|
||||||
|
)
|
||||||
|
|
||||||
|
# Router registrieren (unter app/code/api)
|
||||||
|
try:
|
||||||
|
from api.router import api_router
|
||||||
|
|
||||||
|
app.include_router(api_router, prefix="/api")
|
||||||
|
except Exception:
|
||||||
|
# Router existiert evtl. noch nicht beim ersten Scaffold
|
||||||
|
pass
|
||||||
|
|
||||||
|
try:
|
||||||
|
import agent_api
|
||||||
|
|
||||||
|
app.include_router(agent_api.agent_router, prefix="/api")
|
||||||
|
except Exception:
|
||||||
|
# Agenten-API ist optional und wird bei fehlender Implementierung ignoriert
|
||||||
|
pass
|
||||||
|
|
||||||
|
@app.get("/health", tags=["health"])
|
||||||
|
async def health() -> Dict[str, str]:
|
||||||
|
"""
|
||||||
|
Einfache Health-Check-Route.
|
||||||
|
"""
|
||||||
|
return {"status": "ok"}
|
||||||
|
|
||||||
|
return app
|
||||||
46
work/entworfener_code_working_copy/app/config/agent_api.yaml
Normal file
46
work/entworfener_code_working_copy/app/config/agent_api.yaml
Normal file
@@ -0,0 +1,46 @@
|
|||||||
|
agent_api:
|
||||||
|
metadata:
|
||||||
|
version: "0.1.0"
|
||||||
|
description: "Agent Gateway für Worker + OpenAI-kompatible LLMs"
|
||||||
|
|
||||||
|
http:
|
||||||
|
base_path: "/api/agent/v1"
|
||||||
|
timeout_seconds: 60
|
||||||
|
rate_limit_per_minute: 120
|
||||||
|
enable_cors: true
|
||||||
|
|
||||||
|
auth:
|
||||||
|
api_key_header: "x-agent-api-key"
|
||||||
|
allowed_keys: []
|
||||||
|
allow_unauthenticated: true
|
||||||
|
|
||||||
|
worker:
|
||||||
|
adapter: "inline"
|
||||||
|
endpoint: null
|
||||||
|
timeout_seconds: 30
|
||||||
|
|
||||||
|
llm:
|
||||||
|
provider: "openai"
|
||||||
|
base_url: "https://api.openai.com/v1"
|
||||||
|
model: "gpt-4o-mini"
|
||||||
|
temperature: 0.2
|
||||||
|
max_tokens: 1200
|
||||||
|
api_key: null
|
||||||
|
request_timeout_seconds: 45
|
||||||
|
retry:
|
||||||
|
max_attempts: 3
|
||||||
|
backoff_seconds: 2
|
||||||
|
|
||||||
|
execution:
|
||||||
|
mode: "async"
|
||||||
|
response_timeout_seconds: 30
|
||||||
|
queue_ttl_seconds: 300
|
||||||
|
heartbeat_interval_seconds: 10
|
||||||
|
allow_long_polling: true
|
||||||
|
|
||||||
|
logging:
|
||||||
|
enabled: true
|
||||||
|
log_payloads: false
|
||||||
|
redact_fields:
|
||||||
|
- "user_input"
|
||||||
|
- "llm_response"
|
||||||
60
work/entworfener_code_working_copy/app/config/config.yaml
Normal file
60
work/entworfener_code_working_copy/app/config/config.yaml
Normal file
@@ -0,0 +1,60 @@
|
|||||||
|
# Application settings
|
||||||
|
app:
|
||||||
|
name: "entworfener_code_service"
|
||||||
|
host: "0.0.0.0"
|
||||||
|
port: 8000
|
||||||
|
reload: true
|
||||||
|
|
||||||
|
# Logging settings
|
||||||
|
logging:
|
||||||
|
level: "INFO"
|
||||||
|
config_file: "config/logging.yaml"
|
||||||
|
|
||||||
|
# CORS settings
|
||||||
|
cors:
|
||||||
|
allow_origins:
|
||||||
|
- "*"
|
||||||
|
allow_credentials: true
|
||||||
|
allow_methods:
|
||||||
|
- "*"
|
||||||
|
allow_headers:
|
||||||
|
- "*"
|
||||||
|
|
||||||
|
# Paths (relativ zu app/)
|
||||||
|
paths:
|
||||||
|
log_dir: "logs"
|
||||||
|
data_dir: "data"
|
||||||
|
|
||||||
|
# Server timeouts
|
||||||
|
server:
|
||||||
|
timeout_keep_alive: 5
|
||||||
|
backlog: 2048
|
||||||
|
|
||||||
|
# Metadata
|
||||||
|
meta:
|
||||||
|
version: "0.1.0"
|
||||||
|
environment: "dev"
|
||||||
|
|
||||||
|
# Erweiterungen: interne/externe Logging-Module (01_Modulerweiterungen)
|
||||||
|
logging_internal:
|
||||||
|
enabled: true
|
||||||
|
db_path: "data/internal_logs.sqlite"
|
||||||
|
clean_database: false
|
||||||
|
retention_days: 30
|
||||||
|
max_entries: 100000
|
||||||
|
vacuum_on_start: true
|
||||||
|
batch_write: 100
|
||||||
|
|
||||||
|
logging_external:
|
||||||
|
enabled: false
|
||||||
|
type: "postgresql" # mysql | postgresql
|
||||||
|
host: "localhost"
|
||||||
|
port: 5432
|
||||||
|
user: "logger"
|
||||||
|
password: null # Secrets per Env-Var/Keystore, siehe [Security.md](01_Modulerweiterungen/Planung/Security.md:1)
|
||||||
|
database: "logs"
|
||||||
|
sslmode: "prefer"
|
||||||
|
pool_size: 5
|
||||||
|
connect_timeout: 10
|
||||||
|
write_buffer_size: 100
|
||||||
|
fallback_to_internal_on_error: true
|
||||||
57
work/entworfener_code_working_copy/app/config/logging.yaml
Normal file
57
work/entworfener_code_working_copy/app/config/logging.yaml
Normal file
@@ -0,0 +1,57 @@
|
|||||||
|
version: 1
|
||||||
|
disable_existing_loggers: false
|
||||||
|
|
||||||
|
formatters:
|
||||||
|
standard:
|
||||||
|
format: "%(asctime)s | %(levelname)s | %(name)s | %(filename)s:%(lineno)d | %(message)s"
|
||||||
|
datefmt: "%Y-%m-%d %H:%M:%S"
|
||||||
|
access:
|
||||||
|
format: '%(asctime)s | %(levelname)s | %(client_addr)s - "%(request_line)s" %(status_code)s'
|
||||||
|
datefmt: "%Y-%m-%d %H:%M:%S"
|
||||||
|
|
||||||
|
handlers:
|
||||||
|
console:
|
||||||
|
class: logging.StreamHandler
|
||||||
|
level: INFO
|
||||||
|
formatter: standard
|
||||||
|
stream: ext://sys.stdout
|
||||||
|
|
||||||
|
app_file:
|
||||||
|
class: logging.handlers.TimedRotatingFileHandler
|
||||||
|
level: INFO
|
||||||
|
formatter: standard
|
||||||
|
filename: app/logs/app.log
|
||||||
|
when: D
|
||||||
|
interval: 1
|
||||||
|
backupCount: 7
|
||||||
|
encoding: utf-8
|
||||||
|
|
||||||
|
access_file:
|
||||||
|
class: logging.handlers.TimedRotatingFileHandler
|
||||||
|
level: INFO
|
||||||
|
formatter: access
|
||||||
|
filename: app/logs/access.log
|
||||||
|
when: D
|
||||||
|
interval: 1
|
||||||
|
backupCount: 7
|
||||||
|
encoding: utf-8
|
||||||
|
|
||||||
|
loggers:
|
||||||
|
uvicorn:
|
||||||
|
level: INFO
|
||||||
|
handlers: [console, app_file]
|
||||||
|
propagate: false
|
||||||
|
|
||||||
|
uvicorn.error:
|
||||||
|
level: INFO
|
||||||
|
handlers: [console, app_file]
|
||||||
|
propagate: false
|
||||||
|
|
||||||
|
uvicorn.access:
|
||||||
|
level: INFO
|
||||||
|
handlers: [console, access_file]
|
||||||
|
propagate: false
|
||||||
|
|
||||||
|
root:
|
||||||
|
level: INFO
|
||||||
|
handlers: [console, app_file]
|
||||||
@@ -0,0 +1,23 @@
|
|||||||
|
# Konfiguration für externes Logging-Modul (MySQL/PostgreSQL über SQLAlchemy)
|
||||||
|
# Diese Datei folgt dem Schema der Konfigurationen unter
|
||||||
|
# 00_Globale_Richtlinien/Entworfener_Code/app/config/
|
||||||
|
# und kann später in die Haupt-App übernommen werden.
|
||||||
|
#
|
||||||
|
# Hinweise:
|
||||||
|
# - Secrets (user/password) sollten via Umgebungsvariablen überschrieben werden:
|
||||||
|
# LOG_EXT_USER, LOG_EXT_PASSWORD
|
||||||
|
# - Das Modul unterstützt Fallback ins interne SQLite-Logging, wenn aktiviert.
|
||||||
|
|
||||||
|
logging_external:
|
||||||
|
enabled: false # Modul aktivieren/deaktivieren
|
||||||
|
type: "postgresql" # "postgresql" | "mysql"
|
||||||
|
host: "localhost" # DB-Host
|
||||||
|
port: 5432 # 5432 für PostgreSQL, 3306 für MySQL
|
||||||
|
user: null # via ENV: LOG_EXT_USER (überschreibt diesen Wert)
|
||||||
|
password: null # via ENV: LOG_EXT_PASSWORD (überschreibt diesen Wert)
|
||||||
|
database: "logs" # Datenbankname
|
||||||
|
sslmode: "prefer" # nur für PostgreSQL relevant: disable|allow|prefer|require|verify-ca|verify-full
|
||||||
|
pool_size: 5 # Größe des Connection-Pools
|
||||||
|
connect_timeout: 10 # Verbindungs-Timeout in Sekunden
|
||||||
|
write_buffer_size: 100 # (reserviert für spätere Batch-Nutzung)
|
||||||
|
fallback_to_internal_on_error: true # Fallback in internes SQLite-Logging bei Fehlern
|
||||||
@@ -0,0 +1,12 @@
|
|||||||
|
# Konfiguration für internes Logging-Modul (SQLite)
|
||||||
|
# Diese Datei folgt dem selben Schema wie die Konfigurationen unter 00_Globale_Richtlinien/Entworfener_Code/app/config
|
||||||
|
# Sie kann später in die Haupt-App übernommen werden; Werte sind Default-Beispiele.
|
||||||
|
|
||||||
|
logging_internal:
|
||||||
|
enabled: true # Modul aktivieren/deaktivieren
|
||||||
|
db_path: "data/internal_logs.sqlite" # relativ zu app/
|
||||||
|
clean_database: false # Datenbank beim Start bereinigen (vorsichtig einsetzen)
|
||||||
|
retention_days: 30 # Aufbewahrungsdauer in Tagen
|
||||||
|
max_entries: 100000 # Max. Anzahl an Logeinträgen
|
||||||
|
vacuum_on_start: true # VACUUM nach Start/Cleanup
|
||||||
|
batch_write: 100 # zukünftige Batch-Größe (derzeit optional)
|
||||||
BIN
work/entworfener_code_working_copy/app/data/internal_logs.sqlite
Normal file
BIN
work/entworfener_code_working_copy/app/data/internal_logs.sqlite
Normal file
Binary file not shown.
8
work/entworfener_code_working_copy/app/logs/app.log
Normal file
8
work/entworfener_code_working_copy/app/logs/app.log
Normal file
@@ -0,0 +1,8 @@
|
|||||||
|
2025-11-14 00:04:28 | WARNING | src.logging_setup | logging_setup.py:92 | logging_internal Modul nicht importierbar; interner DB-Handler wird nicht aktiviert.
|
||||||
|
2025-11-14 00:04:28 | INFO | start | start.py:149 | Starte Uvicorn mit Reload (Factory-Modus, app/code).
|
||||||
|
2025-11-14 00:04:31 | WARNING | src.logging_setup | logging_setup.py:92 | logging_internal Modul nicht importierbar; interner DB-Handler wird nicht aktiviert.
|
||||||
|
2025-11-14 00:04:31 | INFO | watchfiles.main | main.py:308 | 6 changes detected
|
||||||
|
2025-11-14 00:16:16 | INFO | src.logging_setup | logging_setup.py:114 | Interner SQLite-Log-Handler aktiv: /home/hei/Downloads/porjekt/Unbenannter Ordner 2/work/entworfener_code_working_copy/app/data/internal_logs.sqlite
|
||||||
|
2025-11-14 00:16:16 | INFO | start | start.py:149 | Starte Uvicorn mit Reload (Factory-Modus, app/code).
|
||||||
|
2025-11-14 00:16:19 | INFO | src.logging_setup | logging_setup.py:114 | Interner SQLite-Log-Handler aktiv: /home/hei/Downloads/porjekt/Unbenannter Ordner 2/work/entworfener_code_working_copy/app/data/internal_logs.sqlite
|
||||||
|
2025-11-14 00:16:19 | INFO | watchfiles.main | main.py:308 | 6 changes detected
|
||||||
39
work/entworfener_code_working_copy/app/runtime_config.yaml
Normal file
39
work/entworfener_code_working_copy/app/runtime_config.yaml
Normal file
@@ -0,0 +1,39 @@
|
|||||||
|
# Laufzeitkonfiguration (Basis)
|
||||||
|
# Diese Datei liegt neben start.py und wird zur Initialisierung von Logging & Modulen verwendet.
|
||||||
|
|
||||||
|
global:
|
||||||
|
level:
|
||||||
|
beschreibung: Logging-Level
|
||||||
|
wert: INFO
|
||||||
|
max_log_size:
|
||||||
|
beschreibung: Maximale Gesamtgröße aller Log-Dateien (in MB)
|
||||||
|
wert: 10
|
||||||
|
retention_days:
|
||||||
|
beschreibung: Anzahl der Tage, die Log-Dateien aufbewahrt werden
|
||||||
|
wert: 7
|
||||||
|
log_dir:
|
||||||
|
beschreibung: Relativer Pfad zum Log-Verzeichnis (relativ zu app/)
|
||||||
|
wert: logs
|
||||||
|
|
||||||
|
modules:
|
||||||
|
webserver:
|
||||||
|
enabled:
|
||||||
|
beschreibung: Separaten Webserver aktivieren (liefert README aus)
|
||||||
|
wert: true
|
||||||
|
host:
|
||||||
|
beschreibung: Bind-Adresse des Webservers
|
||||||
|
wert: "127.0.0.1"
|
||||||
|
port:
|
||||||
|
beschreibung: Port des Webservers
|
||||||
|
wert: 8300
|
||||||
|
|
||||||
|
task_queue:
|
||||||
|
mode:
|
||||||
|
beschreibung: Abarbeitungsmodus der Aufgaben (sequential | parallel)
|
||||||
|
wert: sequential
|
||||||
|
worker_count:
|
||||||
|
beschreibung: Anzahl paralleler Worker, wenn mode=parallel
|
||||||
|
wert: 2
|
||||||
|
task_timeout_seconds:
|
||||||
|
beschreibung: Timeout für einzelne Aufgaben in Sekunden
|
||||||
|
wert: 300
|
||||||
10
work/entworfener_code_working_copy/app/src/__init__.py
Normal file
10
work/entworfener_code_working_copy/app/src/__init__.py
Normal file
@@ -0,0 +1,10 @@
|
|||||||
|
"""
|
||||||
|
Paket für Laufzeit-Module (Config/Logging/etc.) der Referenz-App.
|
||||||
|
"""
|
||||||
|
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
from .config_loader import Settings, load_runtime_config
|
||||||
|
from .logging_setup import init_logging
|
||||||
|
|
||||||
|
__all__ = ["Settings", "load_runtime_config", "init_logging"]
|
||||||
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
416
work/entworfener_code_working_copy/app/src/agent_api.py
Normal file
416
work/entworfener_code_working_copy/app/src/agent_api.py
Normal file
@@ -0,0 +1,416 @@
|
|||||||
|
"""
|
||||||
|
Agenten-API FastAPI Router für Aufgabenverarbeitung.
|
||||||
|
"""
|
||||||
|
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import logging
|
||||||
|
import os
|
||||||
|
import threading
|
||||||
|
import time
|
||||||
|
import uuid
|
||||||
|
from dataclasses import dataclass
|
||||||
|
from datetime import datetime, timedelta, timezone
|
||||||
|
from pathlib import Path
|
||||||
|
from typing import Any, Dict, List, Optional, TypedDict
|
||||||
|
|
||||||
|
import yaml
|
||||||
|
from fastapi import APIRouter, HTTPException, status
|
||||||
|
from fastapi.responses import JSONResponse
|
||||||
|
from pydantic import BaseModel, Field
|
||||||
|
|
||||||
|
logger = logging.getLogger(__name__)
|
||||||
|
|
||||||
|
|
||||||
|
@dataclass(frozen=True)
|
||||||
|
class ExecutionSettings:
|
||||||
|
"""Konfiguration der Ausführungspipeline."""
|
||||||
|
|
||||||
|
mode: str = "async"
|
||||||
|
queue_ttl_seconds: int = 300
|
||||||
|
heartbeat_interval_seconds: int = 10
|
||||||
|
response_timeout_seconds: int = 30
|
||||||
|
|
||||||
|
|
||||||
|
@dataclass(frozen=True)
|
||||||
|
class LLMSettings:
|
||||||
|
"""Konfiguration des LLM-Clients."""
|
||||||
|
|
||||||
|
provider: str = "local_stub"
|
||||||
|
model: str = "local-agent"
|
||||||
|
api_base_url: Optional[str] = None
|
||||||
|
api_key: Optional[str] = None
|
||||||
|
request_timeout_seconds: int = 15
|
||||||
|
|
||||||
|
|
||||||
|
@dataclass(frozen=True)
|
||||||
|
class AgentAPIConfig:
|
||||||
|
"""Gebündelte Agenten-API Konfiguration."""
|
||||||
|
|
||||||
|
execution: ExecutionSettings
|
||||||
|
llm: LLMSettings
|
||||||
|
|
||||||
|
|
||||||
|
class TaskRequest(BaseModel):
|
||||||
|
"""Eingangsmodell für eine Agenten-Aufgabe."""
|
||||||
|
|
||||||
|
user_input: str = Field(..., description="Auslöser der Task in natürlicher Sprache.")
|
||||||
|
context: Optional[Dict[str, Any]] = Field(default=None, description="Optionale Kontextdaten des Aufrufs.")
|
||||||
|
metadata: Optional[Dict[str, Any]] = Field(default=None, description="Zusätzliche Metadaten zur Korrelation.")
|
||||||
|
sync: Optional[bool] = Field(default=None, description="Erzwingt synchrone Verarbeitung, falls gesetzt.")
|
||||||
|
|
||||||
|
|
||||||
|
class TaskData(TypedDict):
|
||||||
|
"""Interne Darstellung eines Tasks im Speicher."""
|
||||||
|
|
||||||
|
task_id: str
|
||||||
|
status: str
|
||||||
|
result: Optional[Dict[str, Any]]
|
||||||
|
error: Optional[str]
|
||||||
|
created_at: datetime
|
||||||
|
updated_at: datetime
|
||||||
|
expires_at: datetime
|
||||||
|
last_heartbeat: datetime
|
||||||
|
retry_after: int
|
||||||
|
request: TaskRequest
|
||||||
|
|
||||||
|
|
||||||
|
def _utcnow() -> datetime:
|
||||||
|
"""Gibt die aktuelle UTC-Zeit zurück."""
|
||||||
|
return datetime.now(timezone.utc)
|
||||||
|
|
||||||
|
|
||||||
|
def _to_iso(dt: datetime) -> str:
|
||||||
|
"""Formatiert Datumswerte als ISO8601-String mit Z-Suffix."""
|
||||||
|
return dt.astimezone(timezone.utc).isoformat().replace("+00:00", "Z")
|
||||||
|
|
||||||
|
|
||||||
|
def _positive_int(value: Any, default: int) -> int:
|
||||||
|
"""Stellt sicher, dass ein Integer positiv ist, andernfalls gilt der Default."""
|
||||||
|
try:
|
||||||
|
candidate = int(value)
|
||||||
|
if candidate > 0:
|
||||||
|
return candidate
|
||||||
|
except (TypeError, ValueError):
|
||||||
|
pass
|
||||||
|
return default
|
||||||
|
|
||||||
|
|
||||||
|
def _candidate_config_paths(module_path: Path) -> List[Path]:
|
||||||
|
"""Ermittelt potenzielle Konfigurationspfade gemäß Projektstandard."""
|
||||||
|
candidates: List[Path] = []
|
||||||
|
local_app_path = module_path.parent.parent
|
||||||
|
candidates.append(local_app_path / "config" / "agent_api.yaml")
|
||||||
|
for parent in module_path.parents:
|
||||||
|
candidate = parent / "00_Globale_Richtlinien" / "Entworfener_Code" / "app" / "config" / "agent_api.yaml"
|
||||||
|
if candidate not in candidates:
|
||||||
|
candidates.append(candidate)
|
||||||
|
return candidates
|
||||||
|
|
||||||
|
|
||||||
|
def _load_agent_config() -> AgentAPIConfig:
|
||||||
|
"""Lädt die Agenten-Konfiguration und führt sie mit Defaults zusammen."""
|
||||||
|
module_path = Path(__file__).resolve()
|
||||||
|
config_data: Dict[str, Any] = {}
|
||||||
|
for path in _candidate_config_paths(module_path):
|
||||||
|
if not path.exists():
|
||||||
|
continue
|
||||||
|
try:
|
||||||
|
with path.open("r", encoding="utf-8") as handle:
|
||||||
|
loaded = yaml.safe_load(handle) or {}
|
||||||
|
config_data = loaded
|
||||||
|
logger.debug("Agenten-API-Konfiguration aus %s geladen.", path)
|
||||||
|
break
|
||||||
|
except Exception as exc:
|
||||||
|
logger.warning("Fehler beim Laden der Agenten-API-Konfiguration aus %s: %s", path, exc)
|
||||||
|
execution_data = config_data.get("execution", {})
|
||||||
|
llm_data = config_data.get("llm", {})
|
||||||
|
mode = str(execution_data.get("mode", "async")).lower()
|
||||||
|
if mode not in {"async", "sync"}:
|
||||||
|
mode = "async"
|
||||||
|
execution = ExecutionSettings(
|
||||||
|
mode=mode,
|
||||||
|
queue_ttl_seconds=_positive_int(execution_data.get("queue_ttl_seconds"), default=300),
|
||||||
|
heartbeat_interval_seconds=_positive_int(execution_data.get("heartbeat_interval_seconds"), default=10),
|
||||||
|
response_timeout_seconds=_positive_int(execution_data.get("response_timeout_seconds"), default=30),
|
||||||
|
)
|
||||||
|
env_api_key = os.getenv("AGENT_API_LLM_KEY")
|
||||||
|
api_key = env_api_key if env_api_key else llm_data.get("api_key")
|
||||||
|
llm = LLMSettings(
|
||||||
|
provider=str(llm_data.get("provider", "local_stub")),
|
||||||
|
model=str(llm_data.get("model", "local-agent")),
|
||||||
|
api_base_url=llm_data.get("api_base_url"),
|
||||||
|
api_key=str(api_key) if api_key else None,
|
||||||
|
request_timeout_seconds=_positive_int(llm_data.get("request_timeout_seconds"), default=15),
|
||||||
|
)
|
||||||
|
return AgentAPIConfig(execution=execution, llm=llm)
|
||||||
|
|
||||||
|
|
||||||
|
CONFIG = _load_agent_config()
|
||||||
|
|
||||||
|
|
||||||
|
class TaskRegistry:
|
||||||
|
"""Thread-sichere Verwaltung der In-Memory-Tasks."""
|
||||||
|
|
||||||
|
def __init__(self) -> None:
|
||||||
|
self._tasks: Dict[str, TaskData] = {}
|
||||||
|
self._lock = threading.Lock()
|
||||||
|
|
||||||
|
def create(self, request: TaskRequest, ttl_seconds: int, retry_after: int) -> TaskData:
|
||||||
|
"""Erzeugt eine neue Task-Struktur und speichert sie."""
|
||||||
|
now = _utcnow()
|
||||||
|
task: TaskData = {
|
||||||
|
"task_id": str(uuid.uuid4()),
|
||||||
|
"status": "processing",
|
||||||
|
"result": None,
|
||||||
|
"error": None,
|
||||||
|
"created_at": now,
|
||||||
|
"updated_at": now,
|
||||||
|
"expires_at": now + timedelta(seconds=ttl_seconds),
|
||||||
|
"last_heartbeat": now,
|
||||||
|
"retry_after": max(retry_after, 1),
|
||||||
|
"request": request,
|
||||||
|
}
|
||||||
|
with self._lock:
|
||||||
|
self._tasks[task["task_id"]] = task
|
||||||
|
return task.copy()
|
||||||
|
|
||||||
|
def get(self, task_id: str) -> Optional[TaskData]:
|
||||||
|
"""Liefert eine Kopie der Task und aktualisiert Expirationen."""
|
||||||
|
with self._lock:
|
||||||
|
task = self._tasks.get(task_id)
|
||||||
|
if task is None:
|
||||||
|
return None
|
||||||
|
self._expire_task_locked(task)
|
||||||
|
return task.copy()
|
||||||
|
|
||||||
|
def get_request(self, task_id: str) -> Optional[TaskRequest]:
|
||||||
|
"""Liefert das ursprüngliche Request-Objekt."""
|
||||||
|
with self._lock:
|
||||||
|
task = self._tasks.get(task_id)
|
||||||
|
if task is None:
|
||||||
|
return None
|
||||||
|
return task["request"]
|
||||||
|
|
||||||
|
def heartbeat(self, task_id: str) -> None:
|
||||||
|
"""Aktualisiert den Heartbeat einer laufenden Task."""
|
||||||
|
with self._lock:
|
||||||
|
task = self._tasks.get(task_id)
|
||||||
|
if task is None or task["status"] != "processing":
|
||||||
|
return
|
||||||
|
now = _utcnow()
|
||||||
|
task["last_heartbeat"] = now
|
||||||
|
task["updated_at"] = now
|
||||||
|
|
||||||
|
def mark_succeeded(self, task_id: str, result: Dict[str, Any], ttl_seconds: int) -> Optional[TaskData]:
|
||||||
|
"""Markiert eine Task als abgeschlossen."""
|
||||||
|
with self._lock:
|
||||||
|
task = self._tasks.get(task_id)
|
||||||
|
if task is None:
|
||||||
|
return None
|
||||||
|
if task["status"] in {"cancelled", "expired"}:
|
||||||
|
return task.copy()
|
||||||
|
now = _utcnow()
|
||||||
|
task["status"] = "succeeded"
|
||||||
|
task["result"] = result
|
||||||
|
task["error"] = None
|
||||||
|
task["updated_at"] = now
|
||||||
|
task["last_heartbeat"] = now
|
||||||
|
task["expires_at"] = now + timedelta(seconds=ttl_seconds)
|
||||||
|
return task.copy()
|
||||||
|
|
||||||
|
def mark_failed(self, task_id: str, error: str, ttl_seconds: int) -> Optional[TaskData]:
|
||||||
|
"""Markiert eine Task als fehlgeschlagen."""
|
||||||
|
with self._lock:
|
||||||
|
task = self._tasks.get(task_id)
|
||||||
|
if task is None:
|
||||||
|
return None
|
||||||
|
now = _utcnow()
|
||||||
|
task["status"] = "failed"
|
||||||
|
task["result"] = {"error": error}
|
||||||
|
task["error"] = error
|
||||||
|
task["updated_at"] = now
|
||||||
|
task["last_heartbeat"] = now
|
||||||
|
task["expires_at"] = now + timedelta(seconds=ttl_seconds)
|
||||||
|
return task.copy()
|
||||||
|
|
||||||
|
def cancel(self, task_id: str) -> Optional[TaskData]:
|
||||||
|
"""Bricht eine Task ab und markiert sie als cancelled."""
|
||||||
|
with self._lock:
|
||||||
|
task = self._tasks.get(task_id)
|
||||||
|
if task is None:
|
||||||
|
return None
|
||||||
|
now = _utcnow()
|
||||||
|
if task["status"] in {"succeeded", "failed", "expired", "cancelled"}:
|
||||||
|
self._expire_task_locked(task)
|
||||||
|
return task.copy()
|
||||||
|
task["status"] = "cancelled"
|
||||||
|
task["error"] = "cancelled by client"
|
||||||
|
task["result"] = None
|
||||||
|
task["updated_at"] = now
|
||||||
|
task["last_heartbeat"] = now
|
||||||
|
task["expires_at"] = now
|
||||||
|
return task.copy()
|
||||||
|
|
||||||
|
def _expire_task_locked(self, task: TaskData) -> None:
|
||||||
|
"""Setzt den Status auf expired, falls das Ablaufdatum erreicht ist."""
|
||||||
|
now = _utcnow()
|
||||||
|
if now < task["expires_at"] or task["status"] == "expired":
|
||||||
|
return
|
||||||
|
task["status"] = "expired"
|
||||||
|
task["error"] = "expired"
|
||||||
|
task["updated_at"] = now
|
||||||
|
|
||||||
|
|
||||||
|
REGISTRY = TaskRegistry()
|
||||||
|
|
||||||
|
|
||||||
|
def _local_stub_result(request: TaskRequest) -> Dict[str, Any]:
|
||||||
|
"""Lokale Fallback-Implementierung ohne externe LLM-Abhängigkeit."""
|
||||||
|
return {
|
||||||
|
"output": f"{request.user_input} [local_stub]",
|
||||||
|
"provider": "local_stub",
|
||||||
|
"metadata": {
|
||||||
|
"mode": CONFIG.execution.mode,
|
||||||
|
"context_present": bool(request.context),
|
||||||
|
"metadata_present": bool(request.metadata),
|
||||||
|
},
|
||||||
|
}
|
||||||
|
|
||||||
|
|
||||||
|
def _invoke_llm(request: TaskRequest) -> Dict[str, Any]:
|
||||||
|
"""Ruft das konfigurierte LLM auf oder fällt auf die Stub-Implementierung zurück."""
|
||||||
|
api_key = CONFIG.llm.api_key
|
||||||
|
base_url = CONFIG.llm.api_base_url
|
||||||
|
if not api_key or not base_url:
|
||||||
|
return _local_stub_result(request)
|
||||||
|
try:
|
||||||
|
import httpx # type: ignore
|
||||||
|
except ModuleNotFoundError:
|
||||||
|
logger.warning("httpx nicht verfügbar, fallback auf lokalen Stub.")
|
||||||
|
return _local_stub_result(request)
|
||||||
|
headers: Dict[str, str] = {}
|
||||||
|
if api_key:
|
||||||
|
headers["Authorization"] = f"Bearer {api_key}"
|
||||||
|
payload = {
|
||||||
|
"model": CONFIG.llm.model,
|
||||||
|
"input": request.user_input,
|
||||||
|
"context": request.context or {},
|
||||||
|
"metadata": request.metadata or {},
|
||||||
|
}
|
||||||
|
timeout = CONFIG.llm.request_timeout_seconds
|
||||||
|
try:
|
||||||
|
with httpx.Client(timeout=timeout) as client:
|
||||||
|
response = client.post(str(base_url), json=payload, headers=headers)
|
||||||
|
response.raise_for_status()
|
||||||
|
data = response.json()
|
||||||
|
except Exception as exc:
|
||||||
|
logger.warning("LLM-Request fehlgeschlagen (%s), fallback auf Stub.", exc)
|
||||||
|
return _local_stub_result(request)
|
||||||
|
return {
|
||||||
|
"output": data.get("output") or data.get("result") or data,
|
||||||
|
"provider": CONFIG.llm.provider,
|
||||||
|
"metadata": {
|
||||||
|
"source": "llm",
|
||||||
|
"model": CONFIG.llm.model,
|
||||||
|
},
|
||||||
|
}
|
||||||
|
|
||||||
|
|
||||||
|
def _execute_request(request: TaskRequest) -> Dict[str, Any]:
|
||||||
|
"""Führt Vorverarbeitung aus und delegiert an das LLM."""
|
||||||
|
# Simulierter Pre-Processing-Schritt sowie Heartbeat-Update
|
||||||
|
time.sleep(0.01)
|
||||||
|
return _invoke_llm(request)
|
||||||
|
|
||||||
|
|
||||||
|
def _process_task(task_id: str) -> None:
|
||||||
|
"""Hintergrundverarbeitung einer Task."""
|
||||||
|
request = REGISTRY.get_request(task_id)
|
||||||
|
if request is None:
|
||||||
|
return
|
||||||
|
try:
|
||||||
|
REGISTRY.heartbeat(task_id)
|
||||||
|
result = _execute_request(request)
|
||||||
|
REGISTRY.heartbeat(task_id)
|
||||||
|
REGISTRY.mark_succeeded(task_id, result, CONFIG.execution.queue_ttl_seconds)
|
||||||
|
except Exception as exc:
|
||||||
|
logger.exception("Fehler bei der Bearbeitung von Task %s", task_id)
|
||||||
|
REGISTRY.mark_failed(task_id, str(exc), CONFIG.execution.queue_ttl_seconds)
|
||||||
|
|
||||||
|
|
||||||
|
def _serialize_task(task: TaskData, include_result: bool = True, include_detail: bool = True) -> Dict[str, Any]:
|
||||||
|
"""Konvertiert den internen Task-Status in eine API-Antwort."""
|
||||||
|
payload: Dict[str, Any] = {
|
||||||
|
"task_id": task["task_id"],
|
||||||
|
"status": task["status"],
|
||||||
|
"expires_at": _to_iso(task["expires_at"]),
|
||||||
|
}
|
||||||
|
if task["status"] == "processing":
|
||||||
|
payload["retry_after"] = task["retry_after"]
|
||||||
|
if include_result and task["result"] is not None and task["status"] in {"succeeded", "failed"}:
|
||||||
|
payload["result"] = task["result"]
|
||||||
|
if include_detail and task.get("error"):
|
||||||
|
payload["detail"] = task["error"]
|
||||||
|
if include_detail and task["status"] == "cancelled" and "detail" not in payload:
|
||||||
|
payload["detail"] = "cancelled"
|
||||||
|
return payload
|
||||||
|
|
||||||
|
|
||||||
|
agent_router = APIRouter(prefix="/agent/v1", tags=["agent-api"])
|
||||||
|
|
||||||
|
|
||||||
|
@agent_router.post("/tasks")
|
||||||
|
def submit_task(request: TaskRequest) -> JSONResponse:
|
||||||
|
"""Erstellt eine neue Task und verarbeitet sie je nach Modus synchron oder asynchron."""
|
||||||
|
effective_mode = CONFIG.execution.mode
|
||||||
|
if request.sync is not None:
|
||||||
|
effective_mode = "sync" if request.sync else "async"
|
||||||
|
ttl = CONFIG.execution.queue_ttl_seconds
|
||||||
|
retry_after = CONFIG.execution.heartbeat_interval_seconds
|
||||||
|
task = REGISTRY.create(request, ttl_seconds=ttl, retry_after=retry_after)
|
||||||
|
task_id = task["task_id"]
|
||||||
|
worker = threading.Thread(
|
||||||
|
target=_process_task,
|
||||||
|
args=(task_id,),
|
||||||
|
name=f"agent-task-{task_id}",
|
||||||
|
daemon=True,
|
||||||
|
)
|
||||||
|
worker.start()
|
||||||
|
if effective_mode == "sync":
|
||||||
|
worker.join(timeout=CONFIG.execution.response_timeout_seconds)
|
||||||
|
final_task = REGISTRY.get(task_id)
|
||||||
|
if final_task is None:
|
||||||
|
raise HTTPException(status_code=status.HTTP_500_INTERNAL_SERVER_ERROR, detail="task_not_found")
|
||||||
|
if final_task["status"] == "processing":
|
||||||
|
content = {
|
||||||
|
"task_id": task_id,
|
||||||
|
"status": "processing",
|
||||||
|
"detail": "still_running",
|
||||||
|
"expires_at": _to_iso(final_task["expires_at"]),
|
||||||
|
"retry_after": final_task["retry_after"],
|
||||||
|
}
|
||||||
|
return JSONResponse(status_code=status.HTTP_408_REQUEST_TIMEOUT, content=content)
|
||||||
|
return JSONResponse(status_code=status.HTTP_200_OK, content=_serialize_task(final_task))
|
||||||
|
return JSONResponse(status_code=status.HTTP_202_ACCEPTED, content=_serialize_task(task, include_result=False, include_detail=False))
|
||||||
|
|
||||||
|
|
||||||
|
@agent_router.get("/tasks/{task_id}")
|
||||||
|
def get_task_status(task_id: str) -> Dict[str, Any]:
|
||||||
|
"""Liefert den Status einer bestehenden Task."""
|
||||||
|
task = REGISTRY.get(task_id)
|
||||||
|
if task is None:
|
||||||
|
raise HTTPException(status_code=status.HTTP_404_NOT_FOUND, detail="task_not_found")
|
||||||
|
return _serialize_task(task)
|
||||||
|
|
||||||
|
|
||||||
|
@agent_router.post("/tasks/{task_id}/cancel")
|
||||||
|
def cancel_task(task_id: str) -> Dict[str, Any]:
|
||||||
|
"""Bricht eine laufende Task ab."""
|
||||||
|
task = REGISTRY.cancel(task_id)
|
||||||
|
if task is None:
|
||||||
|
raise HTTPException(status_code=status.HTTP_404_NOT_FOUND, detail="task_not_found")
|
||||||
|
return _serialize_task(task)
|
||||||
|
|
||||||
|
|
||||||
|
__all__ = ["agent_router"]
|
||||||
239
work/entworfener_code_working_copy/app/src/config_loader.py
Normal file
239
work/entworfener_code_working_copy/app/src/config_loader.py
Normal file
@@ -0,0 +1,239 @@
|
|||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
from dataclasses import dataclass
|
||||||
|
from pathlib import Path
|
||||||
|
from typing import Any, Dict, Optional, Union, cast
|
||||||
|
|
||||||
|
import yaml
|
||||||
|
|
||||||
|
|
||||||
|
@dataclass
|
||||||
|
class Settings:
|
||||||
|
"""
|
||||||
|
Universeller Settings-Wrapper für die zur Laufzeit geladene YAML-Konfiguration.
|
||||||
|
|
||||||
|
Unterstützt zwei Schemata, um kompatibel mit den Anforderungen zu bleiben:
|
||||||
|
|
||||||
|
1) "Einfaches Variablen-Schema" (vom Nutzer gefordert):
|
||||||
|
global:
|
||||||
|
variablenname:
|
||||||
|
beschreibung: <text>
|
||||||
|
wert: <any>
|
||||||
|
modulname:
|
||||||
|
variablenname:
|
||||||
|
beschreibung: <text>
|
||||||
|
wert: <any>
|
||||||
|
|
||||||
|
2) "Verschachteltes Schema" (optionale, spätere Erweiterung):
|
||||||
|
global:
|
||||||
|
logging:
|
||||||
|
max_log_size: 10
|
||||||
|
retention_days: 14
|
||||||
|
modules:
|
||||||
|
webserver:
|
||||||
|
port: 8300
|
||||||
|
|
||||||
|
Abfrage-Regeln:
|
||||||
|
- Settings.value("global", "max_log_size")
|
||||||
|
- Prüft zuerst einfaches Schema: global.max_log_size.wert
|
||||||
|
- Dann verschachtelt: global.max_log_size
|
||||||
|
- Alternativ mit Dot-Pfad: value("global", "logging.max_log_size")
|
||||||
|
- Für Module:
|
||||||
|
- module_value("webserver", "port") oder module_value("webserver", "logging.max_log_size")
|
||||||
|
|
||||||
|
Hinweise:
|
||||||
|
- Diese Klasse validiert bewusst minimal, um ohne zusätzliche Dependencies auszukommen.
|
||||||
|
Detaillierte Validierung (z. B. via pydantic) kann später ergänzt werden.
|
||||||
|
"""
|
||||||
|
|
||||||
|
raw: Dict[str, Any]
|
||||||
|
base_dir: Path
|
||||||
|
|
||||||
|
# ---------- Public API ----------
|
||||||
|
|
||||||
|
def section(self, name: str) -> Dict[str, Any]:
|
||||||
|
return _ensure_dict(self.raw.get(name, {}))
|
||||||
|
|
||||||
|
def value(self, section: str, key: str, default: Any = None) -> Any:
|
||||||
|
"""
|
||||||
|
Liest einen Wert aus einem Abschnitt.
|
||||||
|
|
||||||
|
- Unterstützt Dot-Pfade (z. B. "logging.max_log_size").
|
||||||
|
- Unterstützt das "wert"-Wrapper-Format (variablenname: {beschreibung, wert}).
|
||||||
|
"""
|
||||||
|
node = self.section(section)
|
||||||
|
if not node:
|
||||||
|
return default
|
||||||
|
|
||||||
|
# 1) Einfacher Variablenname ohne Dot-Pfad
|
||||||
|
if "." not in key:
|
||||||
|
if key in node:
|
||||||
|
return _unwrap_variable(node.get(key), default)
|
||||||
|
# ggf. verschachteltes Schema: global[key]
|
||||||
|
return _unwrap_variable(node.get(key), default)
|
||||||
|
|
||||||
|
# 2) Dot-Pfad (verschachtelt)
|
||||||
|
current: Any = node
|
||||||
|
for part in key.split("."):
|
||||||
|
if isinstance(current, dict) and part in current:
|
||||||
|
current = cast(Dict[str, Any], current)[part]
|
||||||
|
else:
|
||||||
|
return default
|
||||||
|
return _unwrap_variable(current, default)
|
||||||
|
|
||||||
|
def module_value(self, module: str, key: str, default: Any = None) -> Any:
|
||||||
|
"""
|
||||||
|
Liest einen Wert im Modul-Abschnitt. Es werden folgende Orte geprüft:
|
||||||
|
- raw[module]
|
||||||
|
- raw.get("modules", {}).get(module)
|
||||||
|
"""
|
||||||
|
# Direktes Modul (einfaches Schema)
|
||||||
|
direct_section = _ensure_dict(self.raw.get(module, {}))
|
||||||
|
if direct_section:
|
||||||
|
out = _value_from_node(direct_section, key, default)
|
||||||
|
if out is not _MISSING:
|
||||||
|
return out
|
||||||
|
|
||||||
|
# Verschachteltes "modules"-Schema
|
||||||
|
modules_section = _ensure_dict(self.raw.get("modules", {}))
|
||||||
|
module_section = _ensure_dict(modules_section.get(module, {}))
|
||||||
|
if module_section:
|
||||||
|
out = _value_from_node(module_section, key, default)
|
||||||
|
if out is not _MISSING:
|
||||||
|
return out
|
||||||
|
|
||||||
|
return default
|
||||||
|
|
||||||
|
def get_logging_params(self) -> Dict[str, Any]:
|
||||||
|
"""
|
||||||
|
Liefert gebräuchliche Logging-Parameter mit sinnvollen Defaults.
|
||||||
|
Folgende Schlüssel werden versucht (in dieser Reihenfolge):
|
||||||
|
|
||||||
|
- level: global.level, global.logging.level
|
||||||
|
- max_log_size: global.max_log_size (MB), global.logging.max_log_size
|
||||||
|
- retention_days: global.retention_days, global.logging.retention_days
|
||||||
|
- log_dir: global.log_dir, global.paths.log_dir
|
||||||
|
"""
|
||||||
|
level = (
|
||||||
|
self.value("global", "level")
|
||||||
|
or self.value("global", "logging.level")
|
||||||
|
or "INFO"
|
||||||
|
)
|
||||||
|
|
||||||
|
max_log_size_mb = (
|
||||||
|
self.value("global", "max_log_size")
|
||||||
|
or self.value("global", "logging.max_log_size")
|
||||||
|
or 10
|
||||||
|
)
|
||||||
|
|
||||||
|
retention_days = (
|
||||||
|
self.value("global", "retention_days")
|
||||||
|
or self.value("global", "logging.retention_days")
|
||||||
|
or 7
|
||||||
|
)
|
||||||
|
|
||||||
|
log_dir = (
|
||||||
|
self.value("global", "log_dir")
|
||||||
|
or self.value("global", "paths.log_dir")
|
||||||
|
or "logs"
|
||||||
|
)
|
||||||
|
|
||||||
|
return {
|
||||||
|
"level": str(level),
|
||||||
|
"max_log_size_mb": int(max_log_size_mb),
|
||||||
|
"retention_days": int(retention_days),
|
||||||
|
"log_dir": str(log_dir),
|
||||||
|
}
|
||||||
|
|
||||||
|
def resolve_path(self, p: Union[str, Path]) -> Path:
|
||||||
|
"""
|
||||||
|
Löst relative Pfade relativ zur base_dir (app/) auf.
|
||||||
|
"""
|
||||||
|
pp = Path(p)
|
||||||
|
if pp.is_absolute():
|
||||||
|
return pp
|
||||||
|
return (self.base_dir / pp).resolve()
|
||||||
|
|
||||||
|
# ---------- Constructors ----------
|
||||||
|
|
||||||
|
@classmethod
|
||||||
|
def from_file(cls, path: Path, base_dir: Optional[Path] = None) -> "Settings":
|
||||||
|
data = _load_yaml_safe(path)
|
||||||
|
return cls(raw=data, base_dir=base_dir or path.parent)
|
||||||
|
|
||||||
|
@classmethod
|
||||||
|
def empty(cls, base_dir: Path) -> "Settings":
|
||||||
|
return cls(raw={}, base_dir=base_dir)
|
||||||
|
|
||||||
|
|
||||||
|
# ---------- Module-level helpers ----------
|
||||||
|
|
||||||
|
def load_runtime_config(config_path: Optional[Path] = None, base_dir: Optional[Path] = None) -> Settings:
|
||||||
|
"""
|
||||||
|
Lädt die Laufzeitkonfiguration. Standard:
|
||||||
|
- base_dir = app/ (Ordner der start.py)
|
||||||
|
- config_path = base_dir / "runtime_config.yaml"
|
||||||
|
|
||||||
|
Gibt Settings mit leerer Rohstruktur zurück, falls Datei fehlt.
|
||||||
|
"""
|
||||||
|
# Versuche app/ als Basis zu bestimmen: .../app/src/config_loader.py -> parents[1] = app/
|
||||||
|
default_base = Path(__file__).resolve().parents[1]
|
||||||
|
bdir = base_dir or default_base
|
||||||
|
|
||||||
|
cfg_path = config_path or (bdir / "runtime_config.yaml")
|
||||||
|
if cfg_path.exists():
|
||||||
|
return Settings.from_file(cfg_path, base_dir=bdir)
|
||||||
|
return Settings.empty(base_dir=bdir)
|
||||||
|
|
||||||
|
|
||||||
|
def _ensure_dict(v: Any) -> Dict[str, Any]:
|
||||||
|
return cast(Dict[str, Any], v) if isinstance(v, dict) else {}
|
||||||
|
|
||||||
|
|
||||||
|
def _unwrap_variable(v: Any, default: Any = None) -> Any:
|
||||||
|
"""
|
||||||
|
Falls v ein Mapping nach dem Muster {beschreibung, wert} ist, gib 'wert' zurück.
|
||||||
|
Andernfalls gib v (als Any) selbst oder default zurück.
|
||||||
|
"""
|
||||||
|
if isinstance(v, dict):
|
||||||
|
mv: Dict[str, Any] = cast(Dict[str, Any], v)
|
||||||
|
if "wert" in mv:
|
||||||
|
return mv.get("wert", default)
|
||||||
|
# Pylance: explizit als Any zurückgeben, um Unknown-Typen zu vermeiden
|
||||||
|
return cast(Any, v) if v is not None else default
|
||||||
|
|
||||||
|
|
||||||
|
_MISSING = object()
|
||||||
|
|
||||||
|
|
||||||
|
def _value_from_node(node: Dict[str, Any], key: str, default: Any) -> Any:
|
||||||
|
"""
|
||||||
|
Holt 'key' aus node. Unterstützt Dot-Pfade und das {beschreibung, wert}-Muster.
|
||||||
|
Gibt _MISSING zurück, wenn der Schlüssel nicht existiert (damit Aufrufer fallbacken kann).
|
||||||
|
"""
|
||||||
|
# Direkter Schlüssel ohne Dot
|
||||||
|
if "." not in key:
|
||||||
|
if key in node:
|
||||||
|
return _unwrap_variable(node.get(key), default)
|
||||||
|
return _MISSING
|
||||||
|
|
||||||
|
# Dot-Pfad
|
||||||
|
current: Any = node
|
||||||
|
for part in key.split("."):
|
||||||
|
if isinstance(current, dict) and part in current:
|
||||||
|
current = cast(Dict[str, Any], current)[part]
|
||||||
|
else:
|
||||||
|
return _MISSING
|
||||||
|
return _unwrap_variable(current, default)
|
||||||
|
|
||||||
|
|
||||||
|
def _load_yaml_safe(path: Path) -> Dict[str, Any]:
|
||||||
|
with path.open("r", encoding="utf-8") as f:
|
||||||
|
data = yaml.safe_load(f)
|
||||||
|
return data or {}
|
||||||
|
|
||||||
|
|
||||||
|
__all__ = [
|
||||||
|
"Settings",
|
||||||
|
"load_runtime_config",
|
||||||
|
]
|
||||||
372
work/entworfener_code_working_copy/app/src/logging_external.py
Normal file
372
work/entworfener_code_working_copy/app/src/logging_external.py
Normal file
@@ -0,0 +1,372 @@
|
|||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
"""
|
||||||
|
Externes Logging-Modul (MySQL / PostgreSQL) auf Basis von SQLAlchemy
|
||||||
|
====================================================================
|
||||||
|
|
||||||
|
Zweck
|
||||||
|
- Persistentes Logging in eine externe Datenbank (MySQL oder PostgreSQL).
|
||||||
|
- Optionale Fallback-Strategie auf internes SQLite-Logging bei Fehlern.
|
||||||
|
|
||||||
|
Features
|
||||||
|
- Connection-Pooling via SQLAlchemy Engine.
|
||||||
|
- Schema-Erzeugung (Tabelle logs) via SQLAlchemy Core.
|
||||||
|
- Batch-Schreiben von Logeinträgen.
|
||||||
|
- Health-Check (SELECT 1).
|
||||||
|
- logging.Handler zum direkten Schreiben aus Python-Logging.
|
||||||
|
- Optionaler Fallback (Schreiben in internes SQLite-Logging), wenn aktiviert.
|
||||||
|
|
||||||
|
Konfiguration (Beispiel; siehe 01_Modulerweiterungen/Planung/Architektur.md)
|
||||||
|
logging_external:
|
||||||
|
enabled: false
|
||||||
|
type: "postgresql" # mysql | postgresql
|
||||||
|
host: "localhost"
|
||||||
|
port: 5432
|
||||||
|
user: "logger"
|
||||||
|
password: null
|
||||||
|
database: "logs"
|
||||||
|
sslmode: "prefer"
|
||||||
|
pool_size: 5
|
||||||
|
connect_timeout: 10
|
||||||
|
write_buffer_size: 100
|
||||||
|
fallback_to_internal_on_error: true
|
||||||
|
"""
|
||||||
|
|
||||||
|
import json
|
||||||
|
import logging
|
||||||
|
import time
|
||||||
|
import os
|
||||||
|
import threading
|
||||||
|
from dataclasses import dataclass
|
||||||
|
from datetime import datetime, timezone
|
||||||
|
from typing import Any, Dict, Iterable, List, Optional, Tuple, Union
|
||||||
|
|
||||||
|
try:
|
||||||
|
from sqlalchemy import (
|
||||||
|
create_engine,
|
||||||
|
Table,
|
||||||
|
Column,
|
||||||
|
Integer,
|
||||||
|
String,
|
||||||
|
Text,
|
||||||
|
MetaData,
|
||||||
|
DateTime,
|
||||||
|
text as sa_text,
|
||||||
|
insert as sa_insert,
|
||||||
|
)
|
||||||
|
from sqlalchemy.engine import Engine, Connection
|
||||||
|
except Exception as e: # pragma: no cover
|
||||||
|
raise RuntimeError(
|
||||||
|
"SQLAlchemy ist erforderlich für logging_external. Bitte Abhängigkeiten installieren."
|
||||||
|
) from e
|
||||||
|
|
||||||
|
|
||||||
|
def _env_override(key: str, default: Optional[str] = None) -> Optional[str]:
|
||||||
|
"""Liest optionale Secrets aus Umgebungsvariablen, falls gesetzt."""
|
||||||
|
v = os.environ.get(key)
|
||||||
|
return v if v is not None else default
|
||||||
|
|
||||||
|
|
||||||
|
def _safe_quote(v: str) -> str:
|
||||||
|
# Nur einfache, naive Maskierung für URL-Bestandteile
|
||||||
|
return v.replace("@", "%40").replace(":", "%3A").replace("/", "%2F")
|
||||||
|
|
||||||
|
|
||||||
|
def _now_utc_iso() -> str:
|
||||||
|
return (
|
||||||
|
datetime.now(timezone.utc)
|
||||||
|
.astimezone(timezone.utc)
|
||||||
|
.replace(tzinfo=timezone.utc)
|
||||||
|
.isoformat(timespec="milliseconds")
|
||||||
|
.replace("+00:00", "Z")
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
@dataclass
|
||||||
|
class ExternalConfig:
|
||||||
|
db_type: str = "postgresql" # "mysql" | "postgresql"
|
||||||
|
host: str = "localhost"
|
||||||
|
port: int = 5432
|
||||||
|
user: Optional[str] = None
|
||||||
|
password: Optional[str] = None
|
||||||
|
database: str = "logs"
|
||||||
|
sslmode: Optional[str] = None # nur relevant für PostgreSQL
|
||||||
|
pool_size: int = 5
|
||||||
|
connect_timeout: int = 10
|
||||||
|
fallback_to_internal_on_error: bool = True
|
||||||
|
|
||||||
|
@classmethod
|
||||||
|
def from_dict(cls, cfg: Dict[str, Any]) -> "ExternalConfig":
|
||||||
|
# Secrets via ENV überschreiben, wenn vorhanden
|
||||||
|
user = cfg.get("user")
|
||||||
|
password = cfg.get("password")
|
||||||
|
user = _env_override("LOG_EXT_USER", user)
|
||||||
|
password = _env_override("LOG_EXT_PASSWORD", password)
|
||||||
|
|
||||||
|
return cls(
|
||||||
|
db_type=str(cfg.get("type", "postgresql")),
|
||||||
|
host=str(cfg.get("host", "localhost")),
|
||||||
|
port=int(cfg.get("port", 5432)),
|
||||||
|
user=(None if user is None else str(user)),
|
||||||
|
password=(None if password is None else str(password)),
|
||||||
|
database=str(cfg.get("database", "logs")),
|
||||||
|
sslmode=(cfg.get("sslmode") if cfg.get("sslmode") is not None else None),
|
||||||
|
pool_size=int(cfg.get("pool_size", 5)),
|
||||||
|
connect_timeout=int(cfg.get("connect_timeout", 10)),
|
||||||
|
fallback_to_internal_on_error=bool(cfg.get("fallback_to_internal_on_error", True)),
|
||||||
|
)
|
||||||
|
|
||||||
|
def to_url(self) -> str:
|
||||||
|
"""
|
||||||
|
Baut den SQLAlchemy-URL-String auf Basis der Konfiguration.
|
||||||
|
- PostgreSQL: postgresql+psycopg2://user:password@host:port/database?sslmode=...
|
||||||
|
- MySQL: mysql+pymysql://user:password@host:port/database
|
||||||
|
"""
|
||||||
|
user_part = ""
|
||||||
|
if self.user:
|
||||||
|
if self.password:
|
||||||
|
user_part = f"{_safe_quote(self.user)}:{_safe_quote(self.password)}@"
|
||||||
|
else:
|
||||||
|
user_part = f"{_safe_quote(self.user)}@"
|
||||||
|
|
||||||
|
if self.db_type.lower() in ("postgres", "postgresql", "pg"):
|
||||||
|
base = f"postgresql+psycopg2://{user_part}{self.host}:{self.port}/{self.database}"
|
||||||
|
if self.sslmode:
|
||||||
|
base = f"{base}?sslmode={self.sslmode}"
|
||||||
|
return base
|
||||||
|
|
||||||
|
if self.db_type.lower() in ("mysql", "mariadb"):
|
||||||
|
return f"mysql+pymysql://{user_part}{self.host}:{self.port}/{self.database}"
|
||||||
|
|
||||||
|
raise ValueError(f"Unsupported db_type: {self.db_type}")
|
||||||
|
|
||||||
|
|
||||||
|
class ExternalDBLogger:
|
||||||
|
"""
|
||||||
|
Verwaltet Engine und Schema für externes Logging.
|
||||||
|
"""
|
||||||
|
|
||||||
|
def __init__(self, engine: Engine) -> None:
|
||||||
|
self.engine = engine
|
||||||
|
self.meta = MetaData()
|
||||||
|
# Hinweis: ts als String/DateTime. Für maximale Kompatibilität nutzen wir String(30) ISO-Zeitstempel.
|
||||||
|
self.logs = Table(
|
||||||
|
"logs",
|
||||||
|
self.meta,
|
||||||
|
Column("id", Integer, primary_key=True, autoincrement=True),
|
||||||
|
Column("ts", String(30), nullable=False),
|
||||||
|
Column("level", String(16), nullable=True),
|
||||||
|
Column("logger", String(128), nullable=True),
|
||||||
|
Column("message", Text, nullable=True),
|
||||||
|
Column("meta", Text, nullable=True), # JSON-String
|
||||||
|
)
|
||||||
|
|
||||||
|
# ---------- API ----------
|
||||||
|
|
||||||
|
def ensure_schema(self) -> None:
|
||||||
|
self.meta.create_all(self.engine, checkfirst=True)
|
||||||
|
|
||||||
|
def health_check(self, timeout_s: int = 5) -> bool:
|
||||||
|
try:
|
||||||
|
with self.engine.connect() as conn:
|
||||||
|
conn.execute(sa_text("SELECT 1"))
|
||||||
|
return True
|
||||||
|
except Exception:
|
||||||
|
return False
|
||||||
|
|
||||||
|
def write_batch(self, entries: Iterable[Dict[str, Any]]) -> None:
|
||||||
|
rows: List[Dict[str, Any]] = []
|
||||||
|
for e in entries:
|
||||||
|
ts = e.get("ts") or _now_utc_iso()
|
||||||
|
level = e.get("level")
|
||||||
|
loggername = e.get("logger")
|
||||||
|
message = e.get("message")
|
||||||
|
meta = e.get("meta")
|
||||||
|
try:
|
||||||
|
meta_json = json.dumps(meta, ensure_ascii=False, separators=(",", ":")) if meta is not None else None
|
||||||
|
except Exception:
|
||||||
|
meta_json = json.dumps({"_repr": repr(meta)}, ensure_ascii=False)
|
||||||
|
rows.append(
|
||||||
|
{
|
||||||
|
"ts": str(ts),
|
||||||
|
"level": (None if level is None else str(level)),
|
||||||
|
"logger": (None if loggername is None else str(loggername)),
|
||||||
|
"message": (None if message is None else str(message)),
|
||||||
|
"meta": meta_json,
|
||||||
|
}
|
||||||
|
)
|
||||||
|
if not rows:
|
||||||
|
return
|
||||||
|
|
||||||
|
with self.engine.begin() as conn:
|
||||||
|
conn.execute(sa_insert(self.logs), rows)
|
||||||
|
|
||||||
|
|
||||||
|
class ExternalDBHandler(logging.Handler):
|
||||||
|
"""
|
||||||
|
Logging-Handler, der LogRecords in die externe DB schreibt.
|
||||||
|
Mit einfachem Retry/Backoff und optionalem Fallback auf internes SQLite.
|
||||||
|
"""
|
||||||
|
|
||||||
|
def __init__(self, writer: ExternalDBLogger, *, fallback_to_internal: bool = True) -> None:
|
||||||
|
super().__init__()
|
||||||
|
self._writer = writer
|
||||||
|
self._fallback_to_internal = fallback_to_internal
|
||||||
|
|
||||||
|
def emit(self, record: logging.LogRecord) -> None:
|
||||||
|
payload = {
|
||||||
|
"ts": _now_utc_iso(),
|
||||||
|
"level": record.levelname,
|
||||||
|
"logger": record.name,
|
||||||
|
"message": self.format(record) if self.formatter else record.getMessage(),
|
||||||
|
"meta": _extract_meta(record),
|
||||||
|
}
|
||||||
|
# Kleiner Retry mit Backoff: 3 Versuche 50ms/150ms
|
||||||
|
delay_ms = [0.05, 0.15, 0.5]
|
||||||
|
last_err: Optional[BaseException] = None
|
||||||
|
for d in delay_ms:
|
||||||
|
try:
|
||||||
|
self._writer.write_batch([payload])
|
||||||
|
return
|
||||||
|
except Exception as e:
|
||||||
|
last_err = e
|
||||||
|
time.sleep(d)
|
||||||
|
# Fallback
|
||||||
|
if self._fallback_to_internal:
|
||||||
|
try:
|
||||||
|
import importlib
|
||||||
|
mod = importlib.import_module("logging_internal")
|
||||||
|
mod.instance().write(payload) # nutzt bestehende interne Instanz
|
||||||
|
logging.getLogger(__name__).warning(
|
||||||
|
"External logging failed; wrote to internal SQLite fallback."
|
||||||
|
)
|
||||||
|
return
|
||||||
|
except Exception:
|
||||||
|
pass
|
||||||
|
# Letzter Ausweg: Fehler im Handler nicht nach außen eskalieren
|
||||||
|
if last_err:
|
||||||
|
self.handleError(record)
|
||||||
|
|
||||||
|
|
||||||
|
def _extract_meta(record: logging.LogRecord) -> Dict[str, Any]:
|
||||||
|
meta: Dict[str, Any] = {
|
||||||
|
"pathname": record.pathname,
|
||||||
|
"lineno": record.lineno,
|
||||||
|
"funcName": record.funcName,
|
||||||
|
"process": record.process,
|
||||||
|
"threadName": record.threadName,
|
||||||
|
}
|
||||||
|
standard = {
|
||||||
|
"name",
|
||||||
|
"msg",
|
||||||
|
"args",
|
||||||
|
"levelname",
|
||||||
|
"levelno",
|
||||||
|
"pathname",
|
||||||
|
"filename",
|
||||||
|
"module",
|
||||||
|
"exc_info",
|
||||||
|
"exc_text",
|
||||||
|
"stack_info",
|
||||||
|
"lineno",
|
||||||
|
"funcName",
|
||||||
|
"created",
|
||||||
|
"msecs",
|
||||||
|
"relativeCreated",
|
||||||
|
"thread",
|
||||||
|
"threadName",
|
||||||
|
"processName",
|
||||||
|
"process",
|
||||||
|
"message",
|
||||||
|
}
|
||||||
|
for k, v in record.__dict__.items():
|
||||||
|
if k not in standard:
|
||||||
|
try:
|
||||||
|
json.dumps(v)
|
||||||
|
meta[k] = v
|
||||||
|
except Exception:
|
||||||
|
meta[k] = repr(v)
|
||||||
|
if record.exc_info:
|
||||||
|
try:
|
||||||
|
meta["exc_info"] = logging.Formatter().formatException(record.exc_info)
|
||||||
|
except Exception:
|
||||||
|
pass
|
||||||
|
return meta
|
||||||
|
|
||||||
|
|
||||||
|
# ---------------------------------------------------------
|
||||||
|
# Modul-Singleton und Helper
|
||||||
|
# ---------------------------------------------------------
|
||||||
|
|
||||||
|
_EXTERNAL_INSTANCE: Optional[ExternalDBLogger] = None
|
||||||
|
_INSTANCE_LOCK = threading.Lock()
|
||||||
|
|
||||||
|
|
||||||
|
def init(
|
||||||
|
*,
|
||||||
|
connection_url: Optional[str] = None,
|
||||||
|
config: Optional[Dict[str, Any]] = None,
|
||||||
|
) -> ExternalDBLogger:
|
||||||
|
"""
|
||||||
|
Initialisiert den globalen ExternalDBLogger. Entweder via connection_url oder via config.
|
||||||
|
Mehrfache Aufrufe liefern dieselbe Instanz zurück.
|
||||||
|
"""
|
||||||
|
global _EXTERNAL_INSTANCE
|
||||||
|
with _INSTANCE_LOCK:
|
||||||
|
if _EXTERNAL_INSTANCE is not None:
|
||||||
|
return _EXTERNAL_INSTANCE
|
||||||
|
|
||||||
|
if connection_url is None:
|
||||||
|
if not isinstance(config, dict):
|
||||||
|
raise ValueError("Entweder connection_url angeben oder config (Dict) bereitstellen.")
|
||||||
|
cfg = ExternalConfig.from_dict(config)
|
||||||
|
connection_url = cfg.to_url()
|
||||||
|
pool_size = cfg.pool_size
|
||||||
|
connect_timeout = cfg.connect_timeout
|
||||||
|
else:
|
||||||
|
# Fallback-Parameter wenn URL direkt übergeben wird
|
||||||
|
pool_size = int((config or {}).get("pool_size", 5))
|
||||||
|
connect_timeout = int((config or {}).get("connect_timeout", 10))
|
||||||
|
cfg = ExternalConfig.from_dict(config or {})
|
||||||
|
|
||||||
|
engine = create_engine(
|
||||||
|
connection_url,
|
||||||
|
pool_pre_ping=True,
|
||||||
|
pool_size=pool_size,
|
||||||
|
connect_args={}, # sslmode ist bei PG bereits in der URL verarbeitet
|
||||||
|
echo=False,
|
||||||
|
)
|
||||||
|
logger = ExternalDBLogger(engine)
|
||||||
|
logger.ensure_schema()
|
||||||
|
_EXTERNAL_INSTANCE = logger
|
||||||
|
# Speichere Fallback-Flag am Logger-Objekt (attributiv), damit Handler darauf zugreifen kann, wenn benötigt
|
||||||
|
setattr(_EXTERNAL_INSTANCE, "_fallback_to_internal_on_error", cfg.fallback_to_internal_on_error)
|
||||||
|
return _EXTERNAL_INSTANCE
|
||||||
|
|
||||||
|
|
||||||
|
def instance() -> ExternalDBLogger:
|
||||||
|
if _EXTERNAL_INSTANCE is None:
|
||||||
|
raise RuntimeError("ExternalDBLogger ist nicht initialisiert. Bitte init(...) zuerst aufrufen.")
|
||||||
|
return _EXTERNAL_INSTANCE
|
||||||
|
|
||||||
|
|
||||||
|
def get_handler(level: Union[int, str] = logging.INFO) -> logging.Handler:
|
||||||
|
if _EXTERNAL_INSTANCE is None:
|
||||||
|
raise RuntimeError("ExternalDBLogger ist nicht initialisiert. Bitte init(...) zuerst aufrufen.")
|
||||||
|
h = ExternalDBHandler(
|
||||||
|
_EXTERNAL_INSTANCE,
|
||||||
|
fallback_to_internal=bool(getattr(_EXTERNAL_INSTANCE, "_fallback_to_internal_on_error", True)),
|
||||||
|
)
|
||||||
|
if isinstance(level, str):
|
||||||
|
level = getattr(logging, level.upper(), logging.INFO)
|
||||||
|
h.setLevel(int(level))
|
||||||
|
return h
|
||||||
|
|
||||||
|
|
||||||
|
__all__ = [
|
||||||
|
"ExternalConfig",
|
||||||
|
"ExternalDBLogger",
|
||||||
|
"init",
|
||||||
|
"instance",
|
||||||
|
"get_handler",
|
||||||
|
]
|
||||||
476
work/entworfener_code_working_copy/app/src/logging_internal.py
Normal file
476
work/entworfener_code_working_copy/app/src/logging_internal.py
Normal file
@@ -0,0 +1,476 @@
|
|||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
"""
|
||||||
|
SQLite-basiertes internes Logging-Modul
|
||||||
|
=======================================
|
||||||
|
|
||||||
|
Zweck
|
||||||
|
- Persistentes, leichtgewichtiges Logging in eine interne SQLite-Datenbank.
|
||||||
|
- Geeignet für interne Daten wie Hash-Werte, Status-Events, Metadaten.
|
||||||
|
|
||||||
|
Fähigkeiten
|
||||||
|
- Schema-Management (Tabellen und Indizes falls nicht vorhanden).
|
||||||
|
- Optionales Säubern der Datenbank beim Start (clean_database).
|
||||||
|
- Aufbewahrung/Retention nach Tagen (retention_days).
|
||||||
|
- Begrenzung der Gesamtanzahl (max_entries).
|
||||||
|
- Bereitstellung eines logging.Handler, der LogRecords direkt in SQLite schreibt.
|
||||||
|
- Abfrage-API mit Filtern und Paging.
|
||||||
|
|
||||||
|
Konfiguration (Beispiel, siehe Planung/Architektur.md)
|
||||||
|
logging_internal:
|
||||||
|
enabled: true
|
||||||
|
db_path: "data/internal_logs.sqlite"
|
||||||
|
clean_database: false
|
||||||
|
retention_days: 30
|
||||||
|
max_entries: 100000
|
||||||
|
vacuum_on_start: true
|
||||||
|
batch_write: 100
|
||||||
|
"""
|
||||||
|
|
||||||
|
import json
|
||||||
|
import logging
|
||||||
|
import sqlite3
|
||||||
|
import threading
|
||||||
|
from dataclasses import dataclass
|
||||||
|
from datetime import datetime, timedelta, timezone
|
||||||
|
from pathlib import Path
|
||||||
|
from typing import Any, Dict, Iterable, List, Optional, Tuple, Union
|
||||||
|
|
||||||
|
# ---------------------------------------------------------
|
||||||
|
# Hilfsfunktionen
|
||||||
|
# ---------------------------------------------------------
|
||||||
|
|
||||||
|
|
||||||
|
def _utc_now_iso() -> str:
|
||||||
|
"""Aktuelle UTC-Zeit als ISO 8601 mit Millisekunden und 'Z'-Suffix."""
|
||||||
|
return datetime.now(timezone.utc).astimezone(timezone.utc).replace(tzinfo=timezone.utc).isoformat(timespec="milliseconds").replace("+00:00", "Z")
|
||||||
|
|
||||||
|
|
||||||
|
def _ensure_dir_for(path: Path) -> None:
|
||||||
|
"""Erzeugt das Zielverzeichnis für eine Datei, falls erforderlich."""
|
||||||
|
path.parent.mkdir(parents=True, exist_ok=True)
|
||||||
|
|
||||||
|
|
||||||
|
def _to_iso(ts: Union[str, datetime, None]) -> Optional[str]:
|
||||||
|
if ts is None:
|
||||||
|
return None
|
||||||
|
if isinstance(ts, str):
|
||||||
|
return ts
|
||||||
|
if isinstance(ts, datetime):
|
||||||
|
if ts.tzinfo is None:
|
||||||
|
ts = ts.replace(tzinfo=timezone.utc)
|
||||||
|
ts = ts.astimezone(timezone.utc)
|
||||||
|
return ts.isoformat(timespec="milliseconds").replace("+00:00", "Z")
|
||||||
|
return None
|
||||||
|
|
||||||
|
|
||||||
|
# ---------------------------------------------------------
|
||||||
|
# Kern: SQLiteLogger
|
||||||
|
# ---------------------------------------------------------
|
||||||
|
|
||||||
|
|
||||||
|
@dataclass
|
||||||
|
class RetentionPolicy:
|
||||||
|
retention_days: int = 30
|
||||||
|
max_entries: int = 100_000
|
||||||
|
|
||||||
|
|
||||||
|
class SQLiteLogger:
|
||||||
|
"""
|
||||||
|
Verwaltet die SQLite-Datenbank für interne Logs.
|
||||||
|
|
||||||
|
Thread-sicher durch Lock, eine Connection pro Prozess (check_same_thread=False).
|
||||||
|
"""
|
||||||
|
|
||||||
|
def __init__(
|
||||||
|
self,
|
||||||
|
db_path: Union[str, Path],
|
||||||
|
vacuum_on_start: bool = True,
|
||||||
|
clean_database: bool = False,
|
||||||
|
retention: Optional[RetentionPolicy] = None,
|
||||||
|
) -> None:
|
||||||
|
self.db_path = Path(db_path)
|
||||||
|
self.vacuum_on_start = bool(vacuum_on_start)
|
||||||
|
self.clean_database = bool(clean_database)
|
||||||
|
self.retention = retention or RetentionPolicy()
|
||||||
|
self._lock = threading.RLock()
|
||||||
|
self._conn: Optional[sqlite3.Connection] = None
|
||||||
|
|
||||||
|
self._initialize_db()
|
||||||
|
|
||||||
|
# ---------- Public API ----------
|
||||||
|
|
||||||
|
def write(self, entry: Dict[str, Any]) -> None:
|
||||||
|
"""
|
||||||
|
Schreibt einen einzelnen Log-Eintrag.
|
||||||
|
|
||||||
|
Erwartete Keys:
|
||||||
|
- ts: ISO 8601, sonst wird automatisch gesetzt
|
||||||
|
- level: TEXT
|
||||||
|
- logger: TEXT
|
||||||
|
- message: TEXT
|
||||||
|
- meta: dict | JSON-serialisierbar | None
|
||||||
|
"""
|
||||||
|
data = self._normalize_entry(entry)
|
||||||
|
with self._lock, self._connection() as con:
|
||||||
|
con.execute(
|
||||||
|
"""
|
||||||
|
INSERT INTO logs (ts, level, logger, message, meta)
|
||||||
|
VALUES (?, ?, ?, ?, ?)
|
||||||
|
""",
|
||||||
|
(data["ts"], data.get("level"), data.get("logger"), data.get("message"), data.get("meta")),
|
||||||
|
)
|
||||||
|
con.commit()
|
||||||
|
|
||||||
|
def write_many(self, entries: Iterable[Dict[str, Any]]) -> None:
|
||||||
|
"""Batch-Insert für mehrere Einträge."""
|
||||||
|
rows: List[Tuple[str, Optional[str], Optional[str], Optional[str], Optional[str]]] = []
|
||||||
|
for e in entries:
|
||||||
|
d = self._normalize_entry(e)
|
||||||
|
rows.append((d["ts"], d.get("level"), d.get("logger"), d.get("message"), d.get("meta")))
|
||||||
|
if not rows:
|
||||||
|
return
|
||||||
|
with self._lock, self._connection() as con:
|
||||||
|
con.executemany(
|
||||||
|
"INSERT INTO logs (ts, level, logger, message, meta) VALUES (?, ?, ?, ?, ?)",
|
||||||
|
rows,
|
||||||
|
)
|
||||||
|
con.commit()
|
||||||
|
|
||||||
|
def query(
|
||||||
|
self,
|
||||||
|
*,
|
||||||
|
logger: Optional[str] = None,
|
||||||
|
level: Optional[str] = None,
|
||||||
|
from_ts: Optional[Union[str, datetime]] = None,
|
||||||
|
to_ts: Optional[Union[str, datetime]] = None,
|
||||||
|
text: Optional[str] = None,
|
||||||
|
limit: int = 100,
|
||||||
|
offset: int = 0,
|
||||||
|
order_desc: bool = True,
|
||||||
|
) -> List[Dict[str, Any]]:
|
||||||
|
"""
|
||||||
|
Liefert Log-Einträge gefiltert und paginiert zurück.
|
||||||
|
- logger: exakter Match
|
||||||
|
- level: exakter Match
|
||||||
|
- from_ts / to_ts: Grenzen (inklusive), ISO 8601 oder datetime
|
||||||
|
- text: Fulltext-ähnliche Suche per LIKE auf message
|
||||||
|
"""
|
||||||
|
clauses: List[str] = []
|
||||||
|
params: List[Any] = []
|
||||||
|
|
||||||
|
if logger:
|
||||||
|
clauses.append("logger = ?")
|
||||||
|
params.append(logger)
|
||||||
|
if level:
|
||||||
|
clauses.append("level = ?")
|
||||||
|
params.append(level)
|
||||||
|
if from_ts is not None:
|
||||||
|
v = _to_iso(from_ts) or _utc_now_iso()
|
||||||
|
clauses.append("ts >= ?")
|
||||||
|
params.append(v)
|
||||||
|
if to_ts is not None:
|
||||||
|
v = _to_iso(to_ts) or _utc_now_iso()
|
||||||
|
clauses.append("ts <= ?")
|
||||||
|
params.append(v)
|
||||||
|
if text:
|
||||||
|
clauses.append("message LIKE ?")
|
||||||
|
params.append(f"%{text}%")
|
||||||
|
|
||||||
|
where = f"WHERE {' AND '.join(clauses)}" if clauses else ""
|
||||||
|
order = "DESC" if order_desc else "ASC"
|
||||||
|
sql = f"""
|
||||||
|
SELECT id, ts, level, logger, message, meta
|
||||||
|
FROM logs
|
||||||
|
{where}
|
||||||
|
ORDER BY ts {order}, id {order}
|
||||||
|
LIMIT ? OFFSET ?
|
||||||
|
"""
|
||||||
|
params.extend([int(max(limit, 0)), int(max(offset, 0))])
|
||||||
|
|
||||||
|
with self._lock, self._connection() as con:
|
||||||
|
cur = con.execute(sql, params)
|
||||||
|
rows = cur.fetchall() or []
|
||||||
|
|
||||||
|
out: List[Dict[str, Any]] = []
|
||||||
|
for r in rows:
|
||||||
|
meta_val: Optional[str] = r[5]
|
||||||
|
meta_obj: Optional[Any] = None
|
||||||
|
if meta_val:
|
||||||
|
try:
|
||||||
|
meta_obj = json.loads(meta_val)
|
||||||
|
except Exception:
|
||||||
|
meta_obj = meta_val # fallback: Rohwert
|
||||||
|
out.append(
|
||||||
|
{
|
||||||
|
"id": r[0],
|
||||||
|
"ts": r[1],
|
||||||
|
"level": r[2],
|
||||||
|
"logger": r[3],
|
||||||
|
"message": r[4],
|
||||||
|
"meta": meta_obj,
|
||||||
|
}
|
||||||
|
)
|
||||||
|
return out
|
||||||
|
|
||||||
|
def cleanup(self) -> None:
|
||||||
|
"""
|
||||||
|
Führt Aufräumregeln aus:
|
||||||
|
- Lösche Einträge älter als retention_days (wenn > 0).
|
||||||
|
- Reduziere auf max_entries (wenn > 0), lösche älteste zuerst.
|
||||||
|
"""
|
||||||
|
with self._lock, self._connection() as con:
|
||||||
|
# 1) Zeitbasierte Aufbewahrung
|
||||||
|
if self.retention.retention_days and self.retention.retention_days > 0:
|
||||||
|
cutoff = datetime.now(timezone.utc) - timedelta(days=int(self.retention.retention_days))
|
||||||
|
cutoff_iso = _to_iso(cutoff) or _utc_now_iso()
|
||||||
|
con.execute("DELETE FROM logs WHERE ts < ?", (cutoff_iso,))
|
||||||
|
con.commit()
|
||||||
|
|
||||||
|
# 2) Anzahl begrenzen
|
||||||
|
if self.retention.max_entries and self.retention.max_entries > 0:
|
||||||
|
cur = con.execute("SELECT COUNT(*) FROM logs")
|
||||||
|
total = int(cur.fetchone()[0])
|
||||||
|
overflow = total - int(self.retention.max_entries)
|
||||||
|
if overflow > 0:
|
||||||
|
# Lösche die ältesten N Einträge
|
||||||
|
con.execute(
|
||||||
|
"""
|
||||||
|
DELETE FROM logs
|
||||||
|
WHERE id IN (
|
||||||
|
SELECT id FROM logs
|
||||||
|
ORDER BY ts ASC, id ASC
|
||||||
|
LIMIT ?
|
||||||
|
)
|
||||||
|
""",
|
||||||
|
(overflow,),
|
||||||
|
)
|
||||||
|
con.commit()
|
||||||
|
|
||||||
|
def get_handler(self, level: Union[int, str] = logging.INFO) -> logging.Handler:
|
||||||
|
"""
|
||||||
|
Erzeugt einen logging.Handler, der LogRecords in die SQLite-DB schreibt.
|
||||||
|
Der zurückgegebene Handler ist threadsicher und kann dem Root-Logger zugewiesen werden.
|
||||||
|
"""
|
||||||
|
h = SQLiteLogHandler(self)
|
||||||
|
if isinstance(level, str):
|
||||||
|
level = getattr(logging, level.upper(), logging.INFO)
|
||||||
|
h.setLevel(int(level))
|
||||||
|
return h
|
||||||
|
|
||||||
|
# ---------- Internals ----------
|
||||||
|
|
||||||
|
def _initialize_db(self) -> None:
|
||||||
|
if self.clean_database and self.db_path.exists():
|
||||||
|
try:
|
||||||
|
self.db_path.unlink()
|
||||||
|
except FileNotFoundError:
|
||||||
|
pass
|
||||||
|
|
||||||
|
_ensure_dir_for(self.db_path)
|
||||||
|
|
||||||
|
with self._lock, self._connection() as con:
|
||||||
|
# Pragmas für Stabilität/Performance
|
||||||
|
con.execute("PRAGMA journal_mode=WAL;")
|
||||||
|
con.execute("PRAGMA synchronous=NORMAL;")
|
||||||
|
con.execute("PRAGMA foreign_keys=ON;")
|
||||||
|
con.execute("PRAGMA temp_store=MEMORY;")
|
||||||
|
|
||||||
|
# Schema
|
||||||
|
con.execute(
|
||||||
|
"""
|
||||||
|
CREATE TABLE IF NOT EXISTS logs (
|
||||||
|
id INTEGER PRIMARY KEY AUTOINCREMENT,
|
||||||
|
ts TEXT NOT NULL,
|
||||||
|
level TEXT,
|
||||||
|
logger TEXT,
|
||||||
|
message TEXT,
|
||||||
|
meta TEXT
|
||||||
|
)
|
||||||
|
"""
|
||||||
|
)
|
||||||
|
con.execute("CREATE INDEX IF NOT EXISTS idx_logs_ts ON logs (ts);")
|
||||||
|
con.execute("CREATE INDEX IF NOT EXISTS idx_logs_logger ON logs (logger);")
|
||||||
|
con.commit()
|
||||||
|
|
||||||
|
if self.vacuum_on_start:
|
||||||
|
# VACUUM darf nicht innerhalb einer aktiven Transaktion laufen
|
||||||
|
prev_iso = con.isolation_level
|
||||||
|
try:
|
||||||
|
con.isolation_level = None
|
||||||
|
con.execute("VACUUM;")
|
||||||
|
finally:
|
||||||
|
con.isolation_level = prev_iso
|
||||||
|
|
||||||
|
# Nach Schema-Erstellung sofort Cleanup-Regeln anwenden
|
||||||
|
self.cleanup()
|
||||||
|
|
||||||
|
def _normalize_entry(self, entry: Dict[str, Any]) -> Dict[str, Any]:
|
||||||
|
ts = _to_iso(entry.get("ts")) or _utc_now_iso()
|
||||||
|
meta_val = entry.get("meta")
|
||||||
|
if meta_val is None:
|
||||||
|
meta_json = None
|
||||||
|
else:
|
||||||
|
try:
|
||||||
|
meta_json = json.dumps(meta_val, ensure_ascii=False, separators=(",", ":"))
|
||||||
|
except Exception:
|
||||||
|
meta_json = json.dumps({"_repr": repr(meta_val)}, ensure_ascii=False)
|
||||||
|
return {
|
||||||
|
"ts": ts,
|
||||||
|
"level": str(entry.get("level")) if entry.get("level") is not None else None,
|
||||||
|
"logger": str(entry.get("logger")) if entry.get("logger") is not None else None,
|
||||||
|
"message": str(entry.get("message")) if entry.get("message") is not None else None,
|
||||||
|
"meta": meta_json,
|
||||||
|
}
|
||||||
|
|
||||||
|
def _connection(self) -> sqlite3.Connection:
|
||||||
|
if self._conn is None:
|
||||||
|
# check_same_thread=False: erlaubt Nutzung über mehrere Threads, wir schützen per Lock
|
||||||
|
self._conn = sqlite3.connect(str(self.db_path), check_same_thread=False)
|
||||||
|
return self._conn
|
||||||
|
|
||||||
|
|
||||||
|
# ---------------------------------------------------------
|
||||||
|
# logging.Handler-Integration
|
||||||
|
# ---------------------------------------------------------
|
||||||
|
|
||||||
|
|
||||||
|
class SQLiteLogHandler(logging.Handler):
|
||||||
|
"""
|
||||||
|
Logging-Handler, der LogRecords in die SQLite-Datenbank schreibt.
|
||||||
|
"""
|
||||||
|
|
||||||
|
def __init__(self, writer: SQLiteLogger) -> None:
|
||||||
|
super().__init__()
|
||||||
|
self._writer = writer
|
||||||
|
|
||||||
|
def emit(self, record: logging.LogRecord) -> None:
|
||||||
|
try:
|
||||||
|
# Standardfelder
|
||||||
|
payload: Dict[str, Any] = {
|
||||||
|
"ts": _utc_now_iso(),
|
||||||
|
"level": record.levelname,
|
||||||
|
"logger": record.name,
|
||||||
|
"message": self.format(record) if self.formatter else record.getMessage(),
|
||||||
|
}
|
||||||
|
|
||||||
|
# Meta: ausgewählte Felder + Extras
|
||||||
|
meta: Dict[str, Any] = {
|
||||||
|
"pathname": record.pathname,
|
||||||
|
"lineno": record.lineno,
|
||||||
|
"funcName": record.funcName,
|
||||||
|
"process": record.process,
|
||||||
|
"threadName": record.threadName,
|
||||||
|
}
|
||||||
|
|
||||||
|
# Extras erkennen: all jene keys, die nicht Standard sind
|
||||||
|
standard = {
|
||||||
|
"name",
|
||||||
|
"msg",
|
||||||
|
"args",
|
||||||
|
"levelname",
|
||||||
|
"levelno",
|
||||||
|
"pathname",
|
||||||
|
"filename",
|
||||||
|
"module",
|
||||||
|
"exc_info",
|
||||||
|
"exc_text",
|
||||||
|
"stack_info",
|
||||||
|
"lineno",
|
||||||
|
"funcName",
|
||||||
|
"created",
|
||||||
|
"msecs",
|
||||||
|
"relativeCreated",
|
||||||
|
"thread",
|
||||||
|
"threadName",
|
||||||
|
"processName",
|
||||||
|
"process",
|
||||||
|
"message",
|
||||||
|
}
|
||||||
|
for k, v in record.__dict__.items():
|
||||||
|
if k not in standard:
|
||||||
|
# Versuch: JSON-serialisierbar machen
|
||||||
|
try:
|
||||||
|
json.dumps(v)
|
||||||
|
meta[k] = v
|
||||||
|
except Exception:
|
||||||
|
meta[k] = repr(v)
|
||||||
|
|
||||||
|
if record.exc_info:
|
||||||
|
# Ausnahmeinformationen hinzufügen (als Text)
|
||||||
|
meta["exc_info"] = logging.Formatter().formatException(record.exc_info)
|
||||||
|
|
||||||
|
payload["meta"] = meta
|
||||||
|
|
||||||
|
self._writer.write(payload)
|
||||||
|
except Exception:
|
||||||
|
# Handler darf niemals den Prozess crashen
|
||||||
|
self.handleError(record)
|
||||||
|
|
||||||
|
|
||||||
|
# ---------------------------------------------------------
|
||||||
|
# Modulweite Singletons/Helper-Funktionen
|
||||||
|
# ---------------------------------------------------------
|
||||||
|
|
||||||
|
_logger_instance: Optional[SQLiteLogger] = None
|
||||||
|
_INSTANCE_LOCK = threading.Lock()
|
||||||
|
|
||||||
|
|
||||||
|
def init(
|
||||||
|
*,
|
||||||
|
db_path: Union[str, Path],
|
||||||
|
vacuum_on_start: bool = True,
|
||||||
|
clean_database: bool = False,
|
||||||
|
retention_days: int = 30,
|
||||||
|
max_entries: int = 100_000,
|
||||||
|
) -> SQLiteLogger:
|
||||||
|
"""
|
||||||
|
Initialisiert den globalen SQLiteLogger (Singleton pro Prozess).
|
||||||
|
Wird erneut aufgerufen, wird die bestehende Instanz zurückgegeben.
|
||||||
|
"""
|
||||||
|
global _logger_instance
|
||||||
|
with _INSTANCE_LOCK:
|
||||||
|
if _logger_instance is not None:
|
||||||
|
return _logger_instance
|
||||||
|
|
||||||
|
p = Path(db_path)
|
||||||
|
# relative Pfade sind relativ zum Arbeitsverzeichnis des Prozesses;
|
||||||
|
# in Integrationen sollte nach Bedarf auf app/-Pfad aufgelöst werden.
|
||||||
|
retention = RetentionPolicy(retention_days=int(retention_days), max_entries=int(max_entries))
|
||||||
|
_logger_instance = SQLiteLogger(
|
||||||
|
db_path=p,
|
||||||
|
vacuum_on_start=vacuum_on_start,
|
||||||
|
clean_database=clean_database,
|
||||||
|
retention=retention,
|
||||||
|
)
|
||||||
|
return _logger_instance
|
||||||
|
|
||||||
|
|
||||||
|
def instance() -> SQLiteLogger:
|
||||||
|
"""Gibt die initialisierte Instanz zurück oder wirft einen Fehler."""
|
||||||
|
if _logger_instance is None:
|
||||||
|
raise RuntimeError("SQLiteLogger ist nicht initialisiert. Bitte init(...) zuerst aufrufen.")
|
||||||
|
return _logger_instance
|
||||||
|
|
||||||
|
|
||||||
|
def get_engineered_handler(level: Union[int, str] = logging.INFO) -> logging.Handler:
|
||||||
|
"""
|
||||||
|
Liefert einen konfigurierten Handler auf Basis der globalen Instanz.
|
||||||
|
Beispiel:
|
||||||
|
from logging import getLogger
|
||||||
|
from logging_internal import init, get_engineered_handler
|
||||||
|
|
||||||
|
init(db_path='data/internal_logs.sqlite', retention_days=30, max_entries=100_000)
|
||||||
|
root = logging.getLogger()
|
||||||
|
root.addHandler(get_engineered_handler(logging.INFO))
|
||||||
|
"""
|
||||||
|
return instance().get_handler(level)
|
||||||
|
|
||||||
|
|
||||||
|
__all__ = [
|
||||||
|
"SQLiteLogger",
|
||||||
|
"RetentionPolicy",
|
||||||
|
"init",
|
||||||
|
"instance",
|
||||||
|
"get_engineered_handler",
|
||||||
|
]
|
||||||
263
work/entworfener_code_working_copy/app/src/logging_setup.py
Normal file
263
work/entworfener_code_working_copy/app/src/logging_setup.py
Normal file
@@ -0,0 +1,263 @@
|
|||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import logging
|
||||||
|
import logging.handlers
|
||||||
|
import os
|
||||||
|
from datetime import datetime, timedelta
|
||||||
|
from pathlib import Path
|
||||||
|
from typing import Iterable, List, Tuple, Dict, Optional, Any
|
||||||
|
|
||||||
|
from .config_loader import Settings
|
||||||
|
|
||||||
|
|
||||||
|
DEFAULT_FORMAT = "%(asctime)s | %(levelname)s | %(name)s | %(filename)s:%(lineno)d | %(message)s"
|
||||||
|
DEFAULT_DATEFMT = "%Y-%m-%d %H:%M:%S"
|
||||||
|
|
||||||
|
|
||||||
|
def init_logging(settings: Settings, app_config: Optional[Dict[str, Any]] = None) -> None:
|
||||||
|
"""
|
||||||
|
Initialisiert das Logging gemäß den Laufzeit-Settings.
|
||||||
|
|
||||||
|
Erfüllt die Anforderungen:
|
||||||
|
- Unter logs/ je Tag eine neue Datei (TimedRotatingFileHandler -> midnight).
|
||||||
|
- Konfigurierbare Aufbewahrungsdauer (retention_days).
|
||||||
|
- Konfigurierbare maximale Log-Größe (interpretiert als maximale Gesamtgröße des Log-Verzeichnisses);
|
||||||
|
älteste Dateien werden bei Überschreitung entfernt.
|
||||||
|
|
||||||
|
Hinweise:
|
||||||
|
- Diese Funktion setzt das Root-Logging neu (entfernt bestehende Handler), um konsistentes Verhalten
|
||||||
|
gegenüber evtl. vorhandener dictConfig zu gewährleisten.
|
||||||
|
- Zusätzlich zur Datei wird immer ein Console-Handler eingerichtet.
|
||||||
|
"""
|
||||||
|
params = settings.get_logging_params()
|
||||||
|
level_name: str = params.get("level", "INFO")
|
||||||
|
level = getattr(logging, level_name.upper(), logging.INFO)
|
||||||
|
|
||||||
|
# Verzeichnis vorbereiten
|
||||||
|
raw_log_dir: str = params.get("log_dir", "logs")
|
||||||
|
log_dir: Path = settings.resolve_path(raw_log_dir)
|
||||||
|
log_dir.mkdir(parents=True, exist_ok=True)
|
||||||
|
|
||||||
|
# Root-Logger zurücksetzen und konfigurieren
|
||||||
|
root = logging.getLogger()
|
||||||
|
_reset_logger_handlers(root)
|
||||||
|
root.setLevel(level)
|
||||||
|
|
||||||
|
# Formatter
|
||||||
|
formatter = logging.Formatter(fmt=DEFAULT_FORMAT, datefmt=DEFAULT_DATEFMT)
|
||||||
|
|
||||||
|
# Console-Handler
|
||||||
|
console_handler = logging.StreamHandler()
|
||||||
|
console_handler.setLevel(level)
|
||||||
|
console_handler.setFormatter(formatter)
|
||||||
|
root.addHandler(console_handler)
|
||||||
|
|
||||||
|
# Datei-Handler: tägliche Rotation um Mitternacht
|
||||||
|
logfile = log_dir / "app.log"
|
||||||
|
file_handler = logging.handlers.TimedRotatingFileHandler(
|
||||||
|
filename=str(logfile),
|
||||||
|
when="midnight",
|
||||||
|
interval=1,
|
||||||
|
backupCount=int(params.get("retention_days", 7)),
|
||||||
|
encoding="utf-8",
|
||||||
|
utc=False,
|
||||||
|
)
|
||||||
|
file_handler.setLevel(level)
|
||||||
|
file_handler.setFormatter(formatter)
|
||||||
|
root.addHandler(file_handler)
|
||||||
|
|
||||||
|
# Cleanup-Regeln anwenden (maximale Gesamtgröße und zusätzliche tagbasierte Aufbewahrung)
|
||||||
|
try:
|
||||||
|
_cleanup_logs(
|
||||||
|
log_dir=log_dir,
|
||||||
|
max_total_mb=int(params.get("max_log_size_mb", params.get("max_log_size", 10))),
|
||||||
|
retention_days=int(params.get("retention_days", 7)),
|
||||||
|
file_prefix="app",
|
||||||
|
extensions=(".log",),
|
||||||
|
)
|
||||||
|
except Exception: # Schutz vor Start-Abbruch durch Aufräumfehler
|
||||||
|
logging.getLogger(__name__).exception("Fehler beim Log-Cleanup ignoriert.")
|
||||||
|
|
||||||
|
# Optionale Integration: internes SQLite-Logging (01_Modulerweiterungen)
|
||||||
|
try:
|
||||||
|
internal_cfg = ((app_config or {}).get("logging_internal") or {})
|
||||||
|
except Exception:
|
||||||
|
internal_cfg = {}
|
||||||
|
|
||||||
|
if isinstance(internal_cfg, dict) and internal_cfg.get("enabled"):
|
||||||
|
try:
|
||||||
|
import importlib
|
||||||
|
mod = importlib.import_module("logging_internal")
|
||||||
|
except Exception:
|
||||||
|
logging.getLogger(__name__).warning(
|
||||||
|
"logging_internal Modul nicht importierbar; interner DB-Handler wird nicht aktiviert."
|
||||||
|
)
|
||||||
|
else:
|
||||||
|
try:
|
||||||
|
db_path = internal_cfg.get("db_path", "data/internal_logs.sqlite")
|
||||||
|
clean_db = bool(internal_cfg.get("clean_database", False))
|
||||||
|
retention_days_cfg = int(internal_cfg.get("retention_days", 30))
|
||||||
|
max_entries_cfg = int(internal_cfg.get("max_entries", 100000))
|
||||||
|
vacuum_on_start = bool(internal_cfg.get("vacuum_on_start", True))
|
||||||
|
|
||||||
|
resolved_db_path = settings.resolve_path(str(db_path))
|
||||||
|
|
||||||
|
mod.init(
|
||||||
|
db_path=str(resolved_db_path),
|
||||||
|
vacuum_on_start=vacuum_on_start,
|
||||||
|
clean_database=clean_db,
|
||||||
|
retention_days=retention_days_cfg,
|
||||||
|
max_entries=max_entries_cfg,
|
||||||
|
)
|
||||||
|
handler = mod.get_engineered_handler(level)
|
||||||
|
root.addHandler(handler)
|
||||||
|
logging.getLogger(__name__).info("Interner SQLite-Log-Handler aktiv: %s", str(resolved_db_path))
|
||||||
|
except Exception:
|
||||||
|
logging.getLogger(__name__).exception(
|
||||||
|
"Fehler bei Initialisierung des internen SQLite-Loggings; Handler wird nicht aktiviert."
|
||||||
|
)
|
||||||
|
|
||||||
|
# Optionale Integration: externes DB-Logging (01_Modulerweiterungen)
|
||||||
|
try:
|
||||||
|
external_cfg = ((app_config or {}).get("logging_external") or {})
|
||||||
|
except Exception:
|
||||||
|
external_cfg = {}
|
||||||
|
|
||||||
|
if isinstance(external_cfg, dict) and external_cfg.get("enabled"):
|
||||||
|
try:
|
||||||
|
import importlib
|
||||||
|
ext_mod = importlib.import_module("logging_external")
|
||||||
|
except Exception:
|
||||||
|
logging.getLogger(__name__).warning(
|
||||||
|
"logging_external Modul nicht importierbar; externer DB-Handler wird nicht aktiviert."
|
||||||
|
)
|
||||||
|
else:
|
||||||
|
try:
|
||||||
|
# ext_mod.init akzeptiert entweder connection_url ODER config-Dict
|
||||||
|
# Wir übergeben das gesamte Config-Dict und lassen das Modul die URL bauen.
|
||||||
|
ext_mod.init(config=external_cfg)
|
||||||
|
ext_handler = ext_mod.get_handler(level)
|
||||||
|
root.addHandler(ext_handler)
|
||||||
|
logging.getLogger(__name__).info("Externer DB-Log-Handler aktiviert.")
|
||||||
|
except Exception:
|
||||||
|
logging.getLogger(__name__).exception(
|
||||||
|
"Fehler bei Initialisierung des externen DB-Loggings; Handler wird nicht aktiviert."
|
||||||
|
)
|
||||||
|
|
||||||
|
def _reset_logger_handlers(logger: logging.Logger) -> None:
|
||||||
|
"""Entfernt alle existierenden Handler vom Logger."""
|
||||||
|
for h in list(logger.handlers):
|
||||||
|
try:
|
||||||
|
logger.removeHandler(h)
|
||||||
|
try:
|
||||||
|
h.close()
|
||||||
|
except Exception:
|
||||||
|
pass
|
||||||
|
except Exception:
|
||||||
|
pass
|
||||||
|
|
||||||
|
|
||||||
|
def _cleanup_logs(
|
||||||
|
log_dir: Path,
|
||||||
|
max_total_mb: int,
|
||||||
|
retention_days: int,
|
||||||
|
file_prefix: str = "app",
|
||||||
|
extensions: Tuple[str, ...] = (".log",),
|
||||||
|
) -> None:
|
||||||
|
"""
|
||||||
|
Bereinigt das Log-Verzeichnis nach zwei Regeln:
|
||||||
|
|
||||||
|
1) Zeitbasierte Aufbewahrung (retention_days):
|
||||||
|
- Löscht Dateien, deren Änderungszeitpunkt älter als retention_days ist.
|
||||||
|
|
||||||
|
2) Größenlimit (max_total_mb):
|
||||||
|
- Sicherstellt, dass die Gesamtgröße aller Log-Dateien (passend zu extensions/prefix)
|
||||||
|
das Limit nicht überschreitet. Bei Überschreitung werden die ältesten Dateien
|
||||||
|
entfernt, bis das Limit eingehalten wird.
|
||||||
|
|
||||||
|
Diese Regeln ergänzen den TimedRotatingFileHandler (backupCount), der ohnehin
|
||||||
|
nur eine feste Anzahl von Backups behält. Mit diesem zusätzlichen Cleanup
|
||||||
|
wird explizit die Aufbewahrungsdauer (in Tagen) und die Gesamtgröße kontrolliert.
|
||||||
|
"""
|
||||||
|
if max_total_mb <= 0 and retention_days <= 0:
|
||||||
|
return
|
||||||
|
|
||||||
|
candidates: List[Path] = _collect_log_files(log_dir, file_prefix, extensions)
|
||||||
|
|
||||||
|
# 1) Zeitbasierte Aufbewahrung (zusätzlich zu backupCount)
|
||||||
|
if retention_days > 0:
|
||||||
|
cutoff = datetime.now() - timedelta(days=retention_days)
|
||||||
|
for f in candidates:
|
||||||
|
try:
|
||||||
|
mtime = datetime.fromtimestamp(f.stat().st_mtime)
|
||||||
|
if mtime < cutoff:
|
||||||
|
_safe_unlink(f)
|
||||||
|
except FileNotFoundError:
|
||||||
|
# Wurde zwischenzeitlich rotiert/gelöscht
|
||||||
|
continue
|
||||||
|
|
||||||
|
# Refresh nach Löschungen
|
||||||
|
candidates = _collect_log_files(log_dir, file_prefix, extensions)
|
||||||
|
|
||||||
|
# 2) Größenlimit
|
||||||
|
if max_total_mb > 0:
|
||||||
|
limit_bytes = max_total_mb * 1024 * 1024
|
||||||
|
# Sortiere nach mtime aufsteigend (älteste zuerst)
|
||||||
|
sorted_by_age = sorted(
|
||||||
|
candidates,
|
||||||
|
key=lambda p: (p.stat().st_mtime if p.exists() else float("inf")),
|
||||||
|
)
|
||||||
|
total_size = _total_size(sorted_by_age)
|
||||||
|
|
||||||
|
for f in sorted_by_age:
|
||||||
|
if total_size <= limit_bytes:
|
||||||
|
break
|
||||||
|
try:
|
||||||
|
size_before = f.stat().st_size if f.exists() else 0
|
||||||
|
_safe_unlink(f)
|
||||||
|
total_size -= size_before
|
||||||
|
except FileNotFoundError:
|
||||||
|
# Bereits gelöscht
|
||||||
|
continue
|
||||||
|
|
||||||
|
|
||||||
|
def _collect_log_files(log_dir: Path, prefix: str, exts: Tuple[str, ...]) -> List[Path]:
|
||||||
|
if not log_dir.exists():
|
||||||
|
return []
|
||||||
|
result: List[Path] = []
|
||||||
|
try:
|
||||||
|
for item in log_dir.iterdir():
|
||||||
|
if not item.is_file():
|
||||||
|
continue
|
||||||
|
if not item.suffix.lower() in exts:
|
||||||
|
continue
|
||||||
|
# Akzeptiere Standard-Dateinamen und Rotationssuffixe (z. B. app.log, app.log.2025-11-13, ...)
|
||||||
|
name = item.name
|
||||||
|
if name.startswith(prefix):
|
||||||
|
result.append(item)
|
||||||
|
except Exception:
|
||||||
|
# Defensive: bei Problemen einfach keine Kandidaten liefern
|
||||||
|
return []
|
||||||
|
return result
|
||||||
|
|
||||||
|
|
||||||
|
def _total_size(files: Iterable[Path]) -> int:
|
||||||
|
s = 0
|
||||||
|
for f in files:
|
||||||
|
try:
|
||||||
|
s += f.stat().st_size
|
||||||
|
except FileNotFoundError:
|
||||||
|
continue
|
||||||
|
return s
|
||||||
|
|
||||||
|
|
||||||
|
def _safe_unlink(p: Path) -> None:
|
||||||
|
try:
|
||||||
|
p.unlink(missing_ok=True) # py3.8+: attribute missing_ok in 3.8? Actually 3.8+: No, use exists() check
|
||||||
|
except TypeError:
|
||||||
|
# Fallback für Python-Versionen ohne missing_ok
|
||||||
|
if p.exists():
|
||||||
|
os.remove(p)
|
||||||
|
|
||||||
|
|
||||||
|
__all__ = ["init_logging"]
|
||||||
170
work/entworfener_code_working_copy/app/start.py
Normal file
170
work/entworfener_code_working_copy/app/start.py
Normal file
@@ -0,0 +1,170 @@
|
|||||||
|
from typing import Any, Dict, Optional, List
|
||||||
|
import argparse
|
||||||
|
import logging
|
||||||
|
import logging.config
|
||||||
|
import os
|
||||||
|
from pathlib import Path
|
||||||
|
import sys
|
||||||
|
|
||||||
|
import uvicorn
|
||||||
|
import yaml
|
||||||
|
from fastapi import FastAPI
|
||||||
|
|
||||||
|
# Basisverzeichnis: .../Entworfener_Code/app
|
||||||
|
BASE_DIR: Path = Path(__file__).resolve().parent
|
||||||
|
CODE_DIR: Path = BASE_DIR / "code" # .../app/code
|
||||||
|
CONFIG_DIR: Path = BASE_DIR / "config" # .../app/config
|
||||||
|
SRC_DIR: Path = BASE_DIR / "src" # .../app/src
|
||||||
|
|
||||||
|
# Sicherstellen, dass .../app/code und .../app/src importierbar sind
|
||||||
|
if str(CODE_DIR) not in sys.path:
|
||||||
|
sys.path.insert(0, str(CODE_DIR))
|
||||||
|
if str(SRC_DIR) not in sys.path:
|
||||||
|
sys.path.insert(0, str(SRC_DIR))
|
||||||
|
|
||||||
|
# Jetzt Import aus neuer Struktur (app/code/app/main.py)
|
||||||
|
from app.main import create_app # noqa: E402
|
||||||
|
from src.config_loader import load_runtime_config # noqa: E402
|
||||||
|
from src.logging_setup import init_logging # noqa: E402
|
||||||
|
|
||||||
|
APP_CONFIG_ENV = "APP_CONFIG_PATH"
|
||||||
|
|
||||||
|
|
||||||
|
def load_yaml(path: Path) -> Dict[str, Any]:
|
||||||
|
"""
|
||||||
|
Lädt eine YAML-Datei und gibt den Inhalt als Dictionary zurück.
|
||||||
|
"""
|
||||||
|
with path.open("r", encoding="utf-8") as f:
|
||||||
|
return yaml.safe_load(f) or {}
|
||||||
|
|
||||||
|
|
||||||
|
def setup_logging(cfg: Dict[str, Any]) -> None:
|
||||||
|
"""
|
||||||
|
Initialisiert das Logging anhand der in app/config/logging.yaml definierten Konfiguration.
|
||||||
|
Stellt sicher, dass das Log-Verzeichnis existiert.
|
||||||
|
"""
|
||||||
|
logging_cfg = cfg.get("logging", {}) or {}
|
||||||
|
paths_cfg = cfg.get("paths", {}) or {}
|
||||||
|
|
||||||
|
log_dir = Path(paths_cfg.get("log_dir", "logs"))
|
||||||
|
if not log_dir.is_absolute():
|
||||||
|
log_dir = BASE_DIR / log_dir
|
||||||
|
log_dir.mkdir(parents=True, exist_ok=True)
|
||||||
|
|
||||||
|
logging_config_file = logging_cfg.get("config_file", "config/logging.yaml")
|
||||||
|
logging_path = Path(logging_config_file)
|
||||||
|
if not logging_path.is_absolute():
|
||||||
|
logging_path = BASE_DIR / logging_path
|
||||||
|
|
||||||
|
if logging_path.exists():
|
||||||
|
with logging_path.open("r", encoding="utf-8") as f:
|
||||||
|
config_dict = yaml.safe_load(f) or {}
|
||||||
|
logging.config.dictConfig(config_dict)
|
||||||
|
else:
|
||||||
|
logging.basicConfig(
|
||||||
|
level=getattr(logging, (logging_cfg.get("level") or "INFO").upper(), logging.INFO),
|
||||||
|
format="%(asctime)s | %(levelname)s | %(name)s | %(filename)s:%(lineno)d | %(message)s",
|
||||||
|
)
|
||||||
|
|
||||||
|
logging.getLogger(__name__).debug("Logging initialisiert. Log-Verzeichnis: %s", str(log_dir))
|
||||||
|
|
||||||
|
|
||||||
|
def load_config_from_env() -> Dict[str, Any]:
|
||||||
|
"""
|
||||||
|
Lädt die Anwendungskonfiguration aus der durch APP_CONFIG_PATH angegebenen Datei.
|
||||||
|
Fällt zurück auf app/config/config.yaml.
|
||||||
|
"""
|
||||||
|
env_path = os.environ.get(APP_CONFIG_ENV)
|
||||||
|
if env_path:
|
||||||
|
cfg_path = Path(env_path)
|
||||||
|
else:
|
||||||
|
cfg_path = CONFIG_DIR / "config.yaml"
|
||||||
|
|
||||||
|
if not cfg_path.exists():
|
||||||
|
print(f"Konfigurationsdatei nicht gefunden: {cfg_path}", file=sys.stderr)
|
||||||
|
return {}
|
||||||
|
|
||||||
|
return load_yaml(cfg_path)
|
||||||
|
|
||||||
|
|
||||||
|
def app_factory() -> FastAPI:
|
||||||
|
"""
|
||||||
|
Uvicorn-Factory für Reload-Betrieb (importierbare App-Fabrik).
|
||||||
|
Liest Konfiguration, initialisiert Logging und erzeugt die FastAPI-App.
|
||||||
|
"""
|
||||||
|
cfg = load_config_from_env()
|
||||||
|
# Laufzeit-Settings laden und Logging initialisieren (überschreibt ggf. YAML-Logging)
|
||||||
|
settings = load_runtime_config(base_dir=BASE_DIR)
|
||||||
|
init_logging(settings, cfg)
|
||||||
|
app = create_app(cfg)
|
||||||
|
return app
|
||||||
|
|
||||||
|
|
||||||
|
def parse_args(argv: Optional[List[str]] = None) -> argparse.Namespace:
|
||||||
|
"""
|
||||||
|
CLI-Argumente parsen.
|
||||||
|
"""
|
||||||
|
parser = argparse.ArgumentParser(description="Startet den FastAPI-Server (app/code-Struktur).")
|
||||||
|
parser.add_argument(
|
||||||
|
"--config",
|
||||||
|
type=str,
|
||||||
|
default=str(CONFIG_DIR / "config.yaml"),
|
||||||
|
help="Pfad zur Konfigurationsdatei (YAML).",
|
||||||
|
)
|
||||||
|
return parser.parse_args(argv)
|
||||||
|
|
||||||
|
|
||||||
|
def main(argv: Optional[List[str]] = None) -> None:
|
||||||
|
"""
|
||||||
|
Einstiegspunkt: lädt Konfiguration, initialisiert Logging, startet Uvicorn.
|
||||||
|
"""
|
||||||
|
args = parse_args(argv)
|
||||||
|
|
||||||
|
# Resolve config path
|
||||||
|
config_path = Path(args.config)
|
||||||
|
if not config_path.is_absolute():
|
||||||
|
config_path = (BASE_DIR / args.config).resolve()
|
||||||
|
|
||||||
|
if not config_path.exists():
|
||||||
|
print(f"Konfigurationsdatei nicht gefunden: {config_path}", file=sys.stderr)
|
||||||
|
sys.exit(2)
|
||||||
|
|
||||||
|
# Set env var für Factory
|
||||||
|
os.environ[APP_CONFIG_ENV] = str(config_path)
|
||||||
|
|
||||||
|
cfg = load_yaml(config_path)
|
||||||
|
# Laufzeit-Settings laden und Logging initialisieren (überschreibt ggf. YAML-Logging)
|
||||||
|
settings = load_runtime_config(base_dir=BASE_DIR)
|
||||||
|
init_logging(settings, cfg)
|
||||||
|
|
||||||
|
app_cfg = cfg.get("app", {}) or {}
|
||||||
|
host = str(app_cfg.get("host", "0.0.0.0"))
|
||||||
|
port = int(app_cfg.get("port", 8000))
|
||||||
|
reload_enabled = bool(app_cfg.get("reload", False))
|
||||||
|
|
||||||
|
logger = logging.getLogger("start")
|
||||||
|
|
||||||
|
if reload_enabled:
|
||||||
|
# Für Reload muss eine importierbare App-Factory übergeben werden
|
||||||
|
logger.info("Starte Uvicorn mit Reload (Factory-Modus, app/code).")
|
||||||
|
module_name = Path(__file__).stem # "start"
|
||||||
|
uvicorn.run(
|
||||||
|
f"{module_name}:app_factory",
|
||||||
|
factory=True,
|
||||||
|
host=host,
|
||||||
|
port=port,
|
||||||
|
reload=True,
|
||||||
|
)
|
||||||
|
else:
|
||||||
|
logger.info("Starte Uvicorn ohne Reload (app/code).")
|
||||||
|
app = create_app(cfg)
|
||||||
|
uvicorn.run(
|
||||||
|
app,
|
||||||
|
host=host,
|
||||||
|
port=port,
|
||||||
|
reload=False,
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
if __name__ == "__main__":
|
||||||
|
main()
|
||||||
27368
work/entworfener_code_working_copy/get-pip.py
Normal file
27368
work/entworfener_code_working_copy/get-pip.py
Normal file
File diff suppressed because it is too large
Load Diff
15
work/entworfener_code_working_copy/requirements.txt
Normal file
15
work/entworfener_code_working_copy/requirements.txt
Normal file
@@ -0,0 +1,15 @@
|
|||||||
|
# Core web framework and server
|
||||||
|
fastapi>=0.115.0
|
||||||
|
uvicorn[standard]>=0.30.0
|
||||||
|
|
||||||
|
# Configuration
|
||||||
|
PyYAML>=6.0.2
|
||||||
|
|
||||||
|
# Testing
|
||||||
|
pytest>=8.3.1
|
||||||
|
httpx>=0.27.2
|
||||||
|
|
||||||
|
# Optional developer tooling (can be moved to requirements-dev.txt)
|
||||||
|
mypy>=1.11.2
|
||||||
|
ruff>=0.6.4
|
||||||
|
black>=24.10.0
|
||||||
17
work/entworfener_code_working_copy/server.log
Normal file
17
work/entworfener_code_working_copy/server.log
Normal file
@@ -0,0 +1,17 @@
|
|||||||
|
2025-11-14 00:16:16 | INFO | src.logging_setup | logging_setup.py:114 | Interner SQLite-Log-Handler aktiv: /home/hei/Downloads/porjekt/Unbenannter Ordner 2/work/entworfener_code_working_copy/app/data/internal_logs.sqlite
|
||||||
|
2025-11-14 00:16:16 | INFO | start | start.py:149 | Starte Uvicorn mit Reload (Factory-Modus, app/code).
|
||||||
|
INFO: Will watch for changes in these directories: ['/home/hei/Downloads/porjekt/Unbenannter Ordner 2/work/entworfener_code_working_copy']
|
||||||
|
INFO: Uvicorn running on http://0.0.0.0:8000 (Press CTRL+C to quit)
|
||||||
|
INFO: Started reloader process [78315] using WatchFiles
|
||||||
|
2025-11-14 00:16:19 | INFO | src.logging_setup | logging_setup.py:114 | Interner SQLite-Log-Handler aktiv: /home/hei/Downloads/porjekt/Unbenannter Ordner 2/work/entworfener_code_working_copy/app/data/internal_logs.sqlite
|
||||||
|
2025-11-14 00:16:19 | INFO | watchfiles.main | main.py:308 | 6 changes detected
|
||||||
|
INFO: Started server process [78350]
|
||||||
|
INFO: Waiting for application startup.
|
||||||
|
INFO: Application startup complete.
|
||||||
|
INFO: 127.0.0.1:53060 - "GET /health HTTP/1.1" 200 OK
|
||||||
|
INFO: 127.0.0.1:53066 - "POST /api/agent/v1/tasks HTTP/1.1" 200 OK
|
||||||
|
INFO: Shutting down
|
||||||
|
INFO: Waiting for application shutdown.
|
||||||
|
INFO: Application shutdown complete.
|
||||||
|
INFO: Finished server process [78350]
|
||||||
|
INFO: Stopping reloader process [78315]
|
||||||
1
work/entworfener_code_working_copy/server.pid
Normal file
1
work/entworfener_code_working_copy/server.pid
Normal file
@@ -0,0 +1 @@
|
|||||||
|
PID=78315
|
||||||
Reference in New Issue
Block a user