madgerm/KI-Cluster-Roadmap
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

neu

Browse Source
This commit is contained in:
madgerm
2025-11-15 08:42:00 +01:00
parent 47a22ac2a6
commit afe3db934c
48 changed files with 30234 additions and 749 deletions
Download Patch File Download Diff File Expand all files Collapse all files

35
.roomodes Normal file
View File

@@ -0,0 +1,35 @@
customModes:
- slug: project-research
name: 🔍 Project Research
roleDefinition: |
You are a detailed-oriented research assistant specializing in examining and understanding codebases. Your primary responsibility is to analyze the file structure, content, and dependencies of a given project to provide comprehensive context relevant to specific user queries.
whenToUse: |
Use this mode when you need to thoroughly investigate and understand a codebase structure, analyze project architecture, or gather comprehensive context about existing implementations. Ideal for onboarding to new projects, understanding complex codebases, or researching how specific features are implemented across the project.
description: Investigate and analyze codebase structure
groups:
- read
source: project
customInstructions: |
Your role is to deeply investigate and summarize the structure and implementation details of the project codebase. To achieve this effectively, you must:
1. Start by carefully examining the file structure of the entire project, with a particular emphasis on files located within the "docs" folder. These files typically contain crucial context, architectural explanations, and usage guidelines.
2. When given a specific query, systematically identify and gather all relevant context from:
- Documentation files in the "docs" folder that provide background information, specifications, or architectural insights.
- Relevant type definitions and interfaces, explicitly citing their exact location (file path and line number) within the source code.
- Implementations directly related to the query, clearly noting their file locations and providing concise yet comprehensive summaries of how they function.
- Important dependencies, libraries, or modules involved in the implementation, including their usage context and significance to the query.
3. Deliver a structured, detailed report that clearly outlines:
- An overview of relevant documentation insights.
- Specific type definitions and their exact locations.
- Relevant implementations, including file paths, functions or methods involved, and a brief explanation of their roles.
- Critical dependencies and their roles in relation to the query.
4. Always cite precise file paths, function names, and line numbers to enhance clarity and ease of navigation.
5. Organize your findings in logical sections, making it straightforward for the user to understand the project's structure and implementation status relevant to their request.
6. Ensure your response directly addresses the user's query and helps them fully grasp the relevant aspects of the project's current state.
These specific instructions supersede any conflicting general instructions you might otherwise follow. Your detailed report should enable effective decision-making and next steps within the overall workflow.

4
00_Globale_Richtlinien/Entworfener_Code/app/code/app/main.py
View File

@@ -50,9 +50,9 @@ def create_app(config: Optional[Dict[str, Any]] = None) -> FastAPI:
pass
try:
from agent_api.router import agent_router
import agent_api
app.include_router(agent_router, prefix="/api")
app.include_router(agent_api.agent_router, prefix="/api")
except Exception:
# Agenten-API ist optional und wird bei fehlender Implementierung ignoriert
pass

19
00_Globale_Richtlinien/Entworfener_Code/app/config/agent_api.yaml Normal file
View File

@@ -0,0 +1,19 @@
# Agenten-API Modulkonfiguration (Standardwerte)
# Wird vom optionalen Modul geladen; Anpassungen erfolgen projektspezifisch.
metadata:
name: "agent_api"
description: "Konfiguration für die optionale Agenten-API-Erweiterung."
execution:
mode: "async" # Optionen: "async" oder "sync"
response_timeout_seconds: 30 # Maximale Wartezeit für synchrone Antworten (Sekunden)
queue_ttl_seconds: 300 # Lebensdauer eines Tasks in Sekunden
heartbeat_interval_seconds: 10 # Interval für Heartbeats der Hintergrund-Worker (Sekunden)
llm:
provider: "local_stub" # Setze z. B. "openai" wenn externe APIs verwendet werden
model: "local-agent"
api_base_url: null # Externe Basis-URL, z. B. https://api.openai.com/v1
api_key: null # Wird durch die Umgebungsvariable AGENT_API_LLM_KEY überschrieben
request_timeout_seconds: 15 # Request-Timeout für den LLM-Client in Sekunden

26
00_Globale_Richtlinien/Entworfener_Code/app/src/logging_setup.py
View File

@@ -117,6 +117,32 @@ def init_logging(settings: Settings, app_config: Optional[Dict[str, Any]] = None
"Fehler bei Initialisierung des internen SQLite-Loggings; Handler wird nicht aktiviert."
)
# Optionale Integration: externes DB-Logging (01_Modulerweiterungen)
try:
external_cfg = ((app_config or {}).get("logging_external") or {})
except Exception:
external_cfg = {}
if isinstance(external_cfg, dict) and external_cfg.get("enabled"):
try:
import importlib
ext_mod = importlib.import_module("logging_external")
except Exception:
logging.getLogger(__name__).warning(
"logging_external Modul nicht importierbar; externer DB-Handler wird nicht aktiviert."
)
else:
try:
# ext_mod.init akzeptiert entweder connection_url ODER config-Dict
# Wir übergeben das gesamte Config-Dict und lassen das Modul die URL bauen.
ext_mod.init(config=external_cfg)
ext_handler = ext_mod.get_handler(level)
root.addHandler(ext_handler)
logging.getLogger(__name__).info("Externer DB-Log-Handler aktiviert.")
except Exception:
logging.getLogger(__name__).exception(
"Fehler bei Initialisierung des externen DB-Loggings; Handler wird nicht aktiviert."
)
def _reset_logger_handlers(logger: logging.Logger) -> None:
"""Entfernt alle existierenden Handler vom Logger."""

23
01_Modulerweiterungen/Entworfener_Code/app/config/logging_external.yaml Normal file
View File

@@ -0,0 +1,23 @@
# Konfiguration für externes Logging-Modul (MySQL/PostgreSQL über SQLAlchemy)
# Diese Datei folgt dem Schema der Konfigurationen unter
# 00_Globale_Richtlinien/Entworfener_Code/app/config/
# und kann später in die Haupt-App übernommen werden.
#
# Hinweise:
# - Secrets (user/password) sollten via Umgebungsvariablen überschrieben werden:
# LOG_EXT_USER, LOG_EXT_PASSWORD
# - Das Modul unterstützt Fallback ins interne SQLite-Logging, wenn aktiviert.
logging_external:
enabled: false # Modul aktivieren/deaktivieren
type: "postgresql" # "postgresql" | "mysql"
host: "localhost" # DB-Host
port: 5432 # 5432 für PostgreSQL, 3306 für MySQL
user: null # via ENV: LOG_EXT_USER (überschreibt diesen Wert)
password: null # via ENV: LOG_EXT_PASSWORD (überschreibt diesen Wert)
database: "logs" # Datenbankname
sslmode: "prefer" # nur für PostgreSQL relevant: disable|allow|prefer|require|verify-ca|verify-full
pool_size: 5 # Größe des Connection-Pools
connect_timeout: 10 # Verbindungs-Timeout in Sekunden
write_buffer_size: 100 # (reserviert für spätere Batch-Nutzung)
fallback_to_internal_on_error: true # Fallback in internes SQLite-Logging bei Fehlern

12
01_Modulerweiterungen/Entworfener_Code/app/config/logging_internal.yaml Normal file
View File

@@ -0,0 +1,12 @@
# Konfiguration für internes Logging-Modul (SQLite)
# Diese Datei folgt dem selben Schema wie die Konfigurationen unter 00_Globale_Richtlinien/Entworfener_Code/app/config
# Sie kann später in die Haupt-App übernommen werden; Werte sind Default-Beispiele.
logging_internal:
enabled: true # Modul aktivieren/deaktivieren
db_path: "data/internal_logs.sqlite" # relativ zu app/
clean_database: false # Datenbank beim Start bereinigen (vorsichtig einsetzen)
retention_days: 30 # Aufbewahrungsdauer in Tagen
max_entries: 100000 # Max. Anzahl an Logeinträgen
vacuum_on_start: true # VACUUM nach Start/Cleanup
batch_write: 100 # zukünftige Batch-Größe (derzeit optional)

416
01_Modulerweiterungen/Entworfener_Code/app/src/agent_api.py Normal file
View File

@@ -0,0 +1,416 @@
"""
Agenten-API FastAPI Router für Aufgabenverarbeitung.
"""
from __future__ import annotations
import logging
import os
import threading
import time
import uuid
from dataclasses import dataclass
from datetime import datetime, timedelta, timezone
from pathlib import Path
from typing import Any, Dict, List, Optional, TypedDict
import yaml
from fastapi import APIRouter, HTTPException, status
from fastapi.responses import JSONResponse
from pydantic import BaseModel, Field
logger = logging.getLogger(__name__)
@dataclass(frozen=True)
class ExecutionSettings:
"""Konfiguration der Ausführungspipeline."""
mode: str = "async"
queue_ttl_seconds: int = 300
heartbeat_interval_seconds: int = 10
response_timeout_seconds: int = 30
@dataclass(frozen=True)
class LLMSettings:
"""Konfiguration des LLM-Clients."""
provider: str = "local_stub"
model: str = "local-agent"
api_base_url: Optional[str] = None
api_key: Optional[str] = None
request_timeout_seconds: int = 15
@dataclass(frozen=True)
class AgentAPIConfig:
"""Gebündelte Agenten-API Konfiguration."""
execution: ExecutionSettings
llm: LLMSettings
class TaskRequest(BaseModel):
"""Eingangsmodell für eine Agenten-Aufgabe."""
user_input: str = Field(..., description="Auslöser der Task in natürlicher Sprache.")
context: Optional[Dict[str, Any]] = Field(default=None, description="Optionale Kontextdaten des Aufrufs.")
metadata: Optional[Dict[str, Any]] = Field(default=None, description="Zusätzliche Metadaten zur Korrelation.")
sync: Optional[bool] = Field(default=None, description="Erzwingt synchrone Verarbeitung, falls gesetzt.")
class TaskData(TypedDict):
"""Interne Darstellung eines Tasks im Speicher."""
task_id: str
status: str
result: Optional[Dict[str, Any]]
error: Optional[str]
created_at: datetime
updated_at: datetime
expires_at: datetime
last_heartbeat: datetime
retry_after: int
request: TaskRequest
def _utcnow() -> datetime:
"""Gibt die aktuelle UTC-Zeit zurück."""
return datetime.now(timezone.utc)
def _to_iso(dt: datetime) -> str:
"""Formatiert Datumswerte als ISO8601-String mit Z-Suffix."""
return dt.astimezone(timezone.utc).isoformat().replace("+00:00", "Z")
def _positive_int(value: Any, default: int) -> int:
"""Stellt sicher, dass ein Integer positiv ist, andernfalls gilt der Default."""
try:
candidate = int(value)
if candidate > 0:
return candidate
except (TypeError, ValueError):
pass
return default
def _candidate_config_paths(module_path: Path) -> List[Path]:
"""Ermittelt potenzielle Konfigurationspfade gemäß Projektstandard."""
candidates: List[Path] = []
local_app_path = module_path.parent.parent
candidates.append(local_app_path / "config" / "agent_api.yaml")
for parent in module_path.parents:
candidate = parent / "00_Globale_Richtlinien" / "Entworfener_Code" / "app" / "config" / "agent_api.yaml"
if candidate not in candidates:
candidates.append(candidate)
return candidates
def _load_agent_config() -> AgentAPIConfig:
"""Lädt die Agenten-Konfiguration und führt sie mit Defaults zusammen."""
module_path = Path(__file__).resolve()
config_data: Dict[str, Any] = {}
for path in _candidate_config_paths(module_path):
if not path.exists():
continue
try:
with path.open("r", encoding="utf-8") as handle:
loaded = yaml.safe_load(handle) or {}
config_data = loaded
logger.debug("Agenten-API-Konfiguration aus %s geladen.", path)
break
except Exception as exc:
logger.warning("Fehler beim Laden der Agenten-API-Konfiguration aus %s: %s", path, exc)
execution_data = config_data.get("execution", {})
llm_data = config_data.get("llm", {})
mode = str(execution_data.get("mode", "async")).lower()
if mode not in {"async", "sync"}:
mode = "async"
execution = ExecutionSettings(
mode=mode,
queue_ttl_seconds=_positive_int(execution_data.get("queue_ttl_seconds"), default=300),
heartbeat_interval_seconds=_positive_int(execution_data.get("heartbeat_interval_seconds"), default=10),
response_timeout_seconds=_positive_int(execution_data.get("response_timeout_seconds"), default=30),
)
env_api_key = os.getenv("AGENT_API_LLM_KEY")
api_key = env_api_key if env_api_key else llm_data.get("api_key")
llm = LLMSettings(
provider=str(llm_data.get("provider", "local_stub")),
model=str(llm_data.get("model", "local-agent")),
api_base_url=llm_data.get("api_base_url"),
api_key=str(api_key) if api_key else None,
request_timeout_seconds=_positive_int(llm_data.get("request_timeout_seconds"), default=15),
)
return AgentAPIConfig(execution=execution, llm=llm)
CONFIG = _load_agent_config()
class TaskRegistry:
"""Thread-sichere Verwaltung der In-Memory-Tasks."""
def __init__(self) -> None:
self._tasks: Dict[str, TaskData] = {}
self._lock = threading.Lock()
def create(self, request: TaskRequest, ttl_seconds: int, retry_after: int) -> TaskData:
"""Erzeugt eine neue Task-Struktur und speichert sie."""
now = _utcnow()
task: TaskData = {
"task_id": str(uuid.uuid4()),
"status": "processing",
"result": None,
"error": None,
"created_at": now,
"updated_at": now,
"expires_at": now + timedelta(seconds=ttl_seconds),
"last_heartbeat": now,
"retry_after": max(retry_after, 1),
"request": request,
}
with self._lock:
self._tasks[task["task_id"]] = task
return task.copy()
def get(self, task_id: str) -> Optional[TaskData]:
"""Liefert eine Kopie der Task und aktualisiert Expirationen."""
with self._lock:
task = self._tasks.get(task_id)
if task is None:
return None
self._expire_task_locked(task)
return task.copy()
def get_request(self, task_id: str) -> Optional[TaskRequest]:
"""Liefert das ursprüngliche Request-Objekt."""
with self._lock:
task = self._tasks.get(task_id)
if task is None:
return None
return task["request"]
def heartbeat(self, task_id: str) -> None:
"""Aktualisiert den Heartbeat einer laufenden Task."""
with self._lock:
task = self._tasks.get(task_id)
if task is None or task["status"] != "processing":
return
now = _utcnow()
task["last_heartbeat"] = now
task["updated_at"] = now
def mark_succeeded(self, task_id: str, result: Dict[str, Any], ttl_seconds: int) -> Optional[TaskData]:
"""Markiert eine Task als abgeschlossen."""
with self._lock:
task = self._tasks.get(task_id)
if task is None:
return None
if task["status"] in {"cancelled", "expired"}:
return task.copy()
now = _utcnow()
task["status"] = "succeeded"
task["result"] = result
task["error"] = None
task["updated_at"] = now
task["last_heartbeat"] = now
task["expires_at"] = now + timedelta(seconds=ttl_seconds)
return task.copy()
def mark_failed(self, task_id: str, error: str, ttl_seconds: int) -> Optional[TaskData]:
"""Markiert eine Task als fehlgeschlagen."""
with self._lock:
task = self._tasks.get(task_id)
if task is None:
return None
now = _utcnow()
task["status"] = "failed"
task["result"] = {"error": error}
task["error"] = error
task["updated_at"] = now
task["last_heartbeat"] = now
task["expires_at"] = now + timedelta(seconds=ttl_seconds)
return task.copy()
def cancel(self, task_id: str) -> Optional[TaskData]:
"""Bricht eine Task ab und markiert sie als cancelled."""
with self._lock:
task = self._tasks.get(task_id)
if task is None:
return None
now = _utcnow()
if task["status"] in {"succeeded", "failed", "expired", "cancelled"}:
self._expire_task_locked(task)
return task.copy()
task["status"] = "cancelled"
task["error"] = "cancelled by client"
task["result"] = None
task["updated_at"] = now
task["last_heartbeat"] = now
task["expires_at"] = now
return task.copy()
def _expire_task_locked(self, task: TaskData) -> None:
"""Setzt den Status auf expired, falls das Ablaufdatum erreicht ist."""
now = _utcnow()
if now < task["expires_at"] or task["status"] == "expired":
return
task["status"] = "expired"
task["error"] = "expired"
task["updated_at"] = now
REGISTRY = TaskRegistry()
def _local_stub_result(request: TaskRequest) -> Dict[str, Any]:
"""Lokale Fallback-Implementierung ohne externe LLM-Abhängigkeit."""
return {
"output": f"{request.user_input} [local_stub]",
"provider": "local_stub",
"metadata": {
"mode": CONFIG.execution.mode,
"context_present": bool(request.context),
"metadata_present": bool(request.metadata),
},
}
def _invoke_llm(request: TaskRequest) -> Dict[str, Any]:
"""Ruft das konfigurierte LLM auf oder fällt auf die Stub-Implementierung zurück."""
api_key = CONFIG.llm.api_key
base_url = CONFIG.llm.api_base_url
if not api_key or not base_url:
return _local_stub_result(request)
try:
import httpx # type: ignore
except ModuleNotFoundError:
logger.warning("httpx nicht verfügbar, fallback auf lokalen Stub.")
return _local_stub_result(request)
headers: Dict[str, str] = {}
if api_key:
headers["Authorization"] = f"Bearer {api_key}"
payload = {
"model": CONFIG.llm.model,
"input": request.user_input,
"context": request.context or {},
"metadata": request.metadata or {},
}
timeout = CONFIG.llm.request_timeout_seconds
try:
with httpx.Client(timeout=timeout) as client:
response = client.post(str(base_url), json=payload, headers=headers)
response.raise_for_status()
data = response.json()
except Exception as exc:
logger.warning("LLM-Request fehlgeschlagen (%s), fallback auf Stub.", exc)
return _local_stub_result(request)
return {
"output": data.get("output") or data.get("result") or data,
"provider": CONFIG.llm.provider,
"metadata": {
"source": "llm",
"model": CONFIG.llm.model,
},
}
def _execute_request(request: TaskRequest) -> Dict[str, Any]:
"""Führt Vorverarbeitung aus und delegiert an das LLM."""
# Simulierter Pre-Processing-Schritt sowie Heartbeat-Update
time.sleep(0.01)
return _invoke_llm(request)
def _process_task(task_id: str) -> None:
"""Hintergrundverarbeitung einer Task."""
request = REGISTRY.get_request(task_id)
if request is None:
return
try:
REGISTRY.heartbeat(task_id)
result = _execute_request(request)
REGISTRY.heartbeat(task_id)
REGISTRY.mark_succeeded(task_id, result, CONFIG.execution.queue_ttl_seconds)
except Exception as exc:
logger.exception("Fehler bei der Bearbeitung von Task %s", task_id)
REGISTRY.mark_failed(task_id, str(exc), CONFIG.execution.queue_ttl_seconds)
def _serialize_task(task: TaskData, include_result: bool = True, include_detail: bool = True) -> Dict[str, Any]:
"""Konvertiert den internen Task-Status in eine API-Antwort."""
payload: Dict[str, Any] = {
"task_id": task["task_id"],
"status": task["status"],
"expires_at": _to_iso(task["expires_at"]),
}
if task["status"] == "processing":
payload["retry_after"] = task["retry_after"]
if include_result and task["result"] is not None and task["status"] in {"succeeded", "failed"}:
payload["result"] = task["result"]
if include_detail and task.get("error"):
payload["detail"] = task["error"]
if include_detail and task["status"] == "cancelled" and "detail" not in payload:
payload["detail"] = "cancelled"
return payload
agent_router = APIRouter(prefix="/agent/v1", tags=["agent-api"])
@agent_router.post("/tasks")
def submit_task(request: TaskRequest) -> JSONResponse:
"""Erstellt eine neue Task und verarbeitet sie je nach Modus synchron oder asynchron."""
effective_mode = CONFIG.execution.mode
if request.sync is not None:
effective_mode = "sync" if request.sync else "async"
ttl = CONFIG.execution.queue_ttl_seconds
retry_after = CONFIG.execution.heartbeat_interval_seconds
task = REGISTRY.create(request, ttl_seconds=ttl, retry_after=retry_after)
task_id = task["task_id"]
worker = threading.Thread(
target=_process_task,
args=(task_id,),
name=f"agent-task-{task_id}",
daemon=True,
)
worker.start()
if effective_mode == "sync":
worker.join(timeout=CONFIG.execution.response_timeout_seconds)
final_task = REGISTRY.get(task_id)
if final_task is None:
raise HTTPException(status_code=status.HTTP_500_INTERNAL_SERVER_ERROR, detail="task_not_found")
if final_task["status"] == "processing":
content = {
"task_id": task_id,
"status": "processing",
"detail": "still_running",
"expires_at": _to_iso(final_task["expires_at"]),
"retry_after": final_task["retry_after"],
}
return JSONResponse(status_code=status.HTTP_408_REQUEST_TIMEOUT, content=content)
return JSONResponse(status_code=status.HTTP_200_OK, content=_serialize_task(final_task))
return JSONResponse(status_code=status.HTTP_202_ACCEPTED, content=_serialize_task(task, include_result=False, include_detail=False))
@agent_router.get("/tasks/{task_id}")
def get_task_status(task_id: str) -> Dict[str, Any]:
"""Liefert den Status einer bestehenden Task."""
task = REGISTRY.get(task_id)
if task is None:
raise HTTPException(status_code=status.HTTP_404_NOT_FOUND, detail="task_not_found")
return _serialize_task(task)
@agent_router.post("/tasks/{task_id}/cancel")
def cancel_task(task_id: str) -> Dict[str, Any]:
"""Bricht eine laufende Task ab."""
task = REGISTRY.cancel(task_id)
if task is None:
raise HTTPException(status_code=status.HTTP_404_NOT_FOUND, detail="task_not_found")
return _serialize_task(task)
__all__ = ["agent_router"]

6
01_Modulerweiterungen/Entworfener_Code/app/src/agent_api/__init__.py
View File

@@ -1,6 +0,0 @@
"""Initialisierungspaket für das Agenten-API-Modul."""
from __future__ import annotations
from .router import agent_router
__all__ = ["agent_router"]

130
01_Modulerweiterungen/Entworfener_Code/app/src/agent_api/config.py
View File

@@ -1,130 +0,0 @@
"""Konfigurationslader für das Agenten-API-Modul."""
from __future__ import annotations
import copy
import os
import threading
from pathlib import Path
from typing import Any, Dict, Mapping, cast
import yaml
_ENV_LLM_KEY = "AGENT_API_LLM_KEY"
_DEFAULT_CONFIG: Dict[str, Any] = {
"agent_api": {
"metadata": {
"version": "0.1.0",
"description": "Agent Gateway für Worker + OpenAI-kompatible LLMs",
},
"http": {
"base_path": "/api/agent/v1",
"timeout_seconds": 60,
"rate_limit_per_minute": 120,
"enable_cors": True,
},
"auth": {
"api_key_header": "x-agent-api-key",
"allowed_keys": [],
"allow_unauthenticated": True,
},
"worker": {
"adapter": "inline",
"endpoint": None,
"timeout_seconds": 30,
},
"llm": {
"provider": "openai",
"base_url": "https://api.openai.com/v1",
"model": "gpt-4o-mini",
"temperature": 0.2,
"max_tokens": 1_200,
"api_key": None,
"request_timeout_seconds": 45,
"retry": {
"max_attempts": 3,
"backoff_seconds": 2,
},
},
"execution": {
"mode": "async",
"response_timeout_seconds": 30,
"queue_ttl_seconds": 300,
"heartbeat_interval_seconds": 10,
"allow_long_polling": True,
},
"logging": {
"enabled": True,
"log_payloads": False,
"redact_fields": ["user_input", "llm_response"],
},
}
}
_config_cache: Dict[str, Any] | None = None
_CACHE_LOCK = threading.Lock()
def _config_path() -> Path:
base_dir = Path(__file__).resolve().parents[2]
return base_dir / "config" / "agent_api.yaml"
def _load_file(path: Path) -> Dict[str, Any]:
if not path.exists():
return {}
with path.open("r", encoding="utf-8") as handle:
raw_data: Any = yaml.safe_load(handle) or {}
if not isinstance(raw_data, dict):
raise ValueError(f"Ungültiges Konfigurationsformat in {path}")
return cast(Dict[str, Any], raw_data)
def _deep_merge(base: Dict[str, Any], override: Mapping[str, Any]) -> Dict[str, Any]:
merged: Dict[str, Any] = copy.deepcopy(base)
for key, value in override.items():
if (
key in merged
and isinstance(merged[key], dict)
and isinstance(value, Mapping)
):
merged[key] = _deep_merge(merged[key], value) # type: ignore[arg-type]
else:
merged[key] = copy.deepcopy(value)
return merged
def _apply_env_overrides(config: Dict[str, Any]) -> None:
api_key = os.getenv(_ENV_LLM_KEY)
if api_key:
config.setdefault("agent_api", {}).setdefault("llm", {})["api_key"] = api_key
def _load_config_uncached() -> Dict[str, Any]:
file_data = _load_file(_config_path())
merged = _deep_merge(_DEFAULT_CONFIG, file_data)
_apply_env_overrides(merged)
return merged
def get_config(force_reload: bool = False) -> Dict[str, Any]:
"""
Liefert die Agenten-API-Konfiguration als Dictionary.
Args:
force_reload: Wenn True, wird die Datei erneut eingelesen.
Returns:
Dict[str, Any]: Zusammengeführte Konfiguration aus Defaults,
YAML-Datei und ENV-Overrides.
"""
global _config_cache
if force_reload:
with _CACHE_LOCK:
_config_cache = _load_config_uncached()
return copy.deepcopy(_config_cache)
if _config_cache is None:
with _CACHE_LOCK:
if _config_cache is None:
_config_cache = _load_config_uncached()
return copy.deepcopy(_config_cache)

113
01_Modulerweiterungen/Entworfener_Code/app/src/agent_api/llm_client.py
View File

@@ -1,113 +0,0 @@
"""HTTP-basierter LLM-Client für das Agenten-API-Modul."""
from __future__ import annotations
import json
import time
from typing import Any, Dict, Mapping, Optional
import httpx
from .config import get_config
class LLMClient:
"""
Kapselt Aufrufe an ein OpenAI-kompatibles REST-Interface.
Die Implementierung unterstützt einfache Retries, Timeout-Handling sowie
einen deterministischen Stub, falls kein API-Key konfiguriert wurde.
"""
def __init__(self, config: Optional[Mapping[str, Any]] = None) -> None:
cfg = dict(config or get_config())
agent_cfg = cfg.get("agent_api", {})
llm_cfg: Dict[str, Any] = dict(agent_cfg.get("llm", {}))
self.base_url: str = str(llm_cfg.get("base_url", "https://api.openai.com/v1")).rstrip("/")
self.model: str = str(llm_cfg.get("model", "gpt-4o-mini"))
self.temperature: float = float(llm_cfg.get("temperature", 0.2))
self.max_tokens: int = int(llm_cfg.get("max_tokens", 1_200))
self.api_key: Optional[str] = llm_cfg.get("api_key")
self.request_timeout_seconds: float = float(llm_cfg.get("request_timeout_seconds", 45))
retry_cfg: Dict[str, Any] = dict(llm_cfg.get("retry", {}))
self.max_attempts: int = max(1, int(retry_cfg.get("max_attempts", 3)))
self.backoff_seconds: float = float(retry_cfg.get("backoff_seconds", 2))
def generate(self, prompt: Dict[str, Any]) -> Dict[str, Any]:
"""
Sendet den Prompt an den konfigurierten LLM-Endpunkt oder liefert eine Stub-Antwort.
Args:
prompt: Strukturierter Prompt, erzeugt durch den WorkerAdapter.
Returns:
Dict[str, Any]: Antwort-Payload, der den OpenAI-ähnlichen Response widerspiegelt.
"""
if not self.api_key:
return self._local_stub(prompt)
url = f"{self.base_url}/chat/completions"
payload = {
"model": self.model,
"messages": self._build_messages(prompt),
"temperature": self.temperature,
"max_tokens": self.max_tokens,
}
headers = {
"Authorization": f"Bearer {self.api_key}",
"Content-Type": "application/json",
}
last_error: Optional[BaseException] = None
for attempt in range(1, self.max_attempts + 1):
try:
with httpx.Client(timeout=self.request_timeout_seconds) as client:
response = client.post(url, headers=headers, json=payload)
response.raise_for_status()
return response.json()
except (httpx.HTTPError, httpx.TimeoutException) as exc:
last_error = exc
if attempt >= self.max_attempts:
break
time.sleep(self.backoff_seconds)
if last_error:
raise RuntimeError(f"LLM request failed after {self.max_attempts} attempts") from last_error
raise RuntimeError("LLM request failed unexpectedly without additional context")
@staticmethod
def _build_messages(prompt: Mapping[str, Any]) -> Any:
user_input = str(prompt.get("user_input", ""))
context = prompt.get("context") or {}
metadata = prompt.get("metadata") or {}
parts = [
{"role": "system", "content": json.dumps({"context": context, "metadata": metadata})},
{"role": "user", "content": user_input},
]
return parts
@staticmethod
def _local_stub(prompt: Mapping[str, Any]) -> Dict[str, Any]:
user_input = prompt.get("user_input", "")
return {
"id": "local_stub_completion",
"object": "chat.completion",
"created": int(time.time()),
"model": "local-stub",
"choices": [
{
"index": 0,
"message": {
"role": "assistant",
"content": f"[local_stub] Echo: {user_input}",
},
"finish_reason": "stop",
}
],
"usage": {
"prompt_tokens": len(json.dumps(prompt)),
"completion_tokens": len(str(user_input)),
"total_tokens": len(json.dumps(prompt)) + len(str(user_input)),
},
"meta": {"note": "local_stub"},
}

49
01_Modulerweiterungen/Entworfener_Code/app/src/agent_api/models.py
View File

@@ -1,49 +0,0 @@
"""Pydantic-Modelle für das Agenten-API-Modul."""
from __future__ import annotations
from datetime import datetime
from typing import Any, Dict, Optional
from pydantic import BaseModel, Field
class TaskCreate(BaseModel):
"""
Repräsentiert den Payload zur Erstellung eines neuen Agenten-Tasks.
"""
user_input: str = Field(..., description="Freitext-Anfrage oder Prompt des Nutzers.")
context: Optional[Dict[str, Any]] = Field(
None, description="Optionaler Kontext mit zusätzlichen strukturierten Daten."
)
metadata: Optional[Dict[str, Any]] = Field(
None, description="Beliebige Metadaten zur Nachverfolgung von Tasks."
)
sync: Optional[bool] = Field(
None,
description=(
"Optionaler Override, um den Task synchron zu verarbeiten, "
"selbst wenn die Ausführung standardmäßig asynchron erfolgt."
),
)
class TaskStatusResponse(BaseModel):
"""
Antwortmodell für den Status eines bestehenden Tasks.
"""
task_id: str = Field(..., description="Eindeutige Kennung des Tasks.")
status: str = Field(..., description="Aktueller Status des Tasks.")
result: Optional[Dict[str, Any]] = Field(
None,
description="Ergebnisdaten des Tasks, sobald verfügbar.",
)
expires_at: Optional[datetime] = Field(
None,
description="Zeitpunkt, an dem der Task automatisch ausläuft.",
)
retry_after: Optional[int] = Field(
None,
description="Empfohlene Wartezeit in Sekunden für den nächsten Statusabruf.",
)

88
01_Modulerweiterungen/Entworfener_Code/app/src/agent_api/router.py
View File

@@ -1,88 +0,0 @@
"""FastAPI-Router für das Agenten-API-Modul."""
from __future__ import annotations
from typing import Dict, Optional
from fastapi import APIRouter, HTTPException, Response, status
from fastapi.responses import JSONResponse
from .models import TaskCreate
from .service import AgentService
agent_router = APIRouter(prefix="/agent/v1", tags=["agent_api"])
_service = AgentService()
def _build_response(
payload: Dict[str, Optional[str]],
status_code: int,
retry_after: Optional[int] = None,
) -> JSONResponse:
headers = {}
if retry_after is not None:
headers["Retry-After"] = str(retry_after)
return JSONResponse(content=payload, status_code=status_code, headers=headers)
@agent_router.post(
"/tasks",
response_model_exclude_none=True,
summary="Neuen Task einreichen",
)
async def submit_task(task: TaskCreate) -> JSONResponse:
"""
Legt einen neuen Task an und startet die Verarbeitung.
"""
result = _service.submit_task(task)
response_payload = result.response.model_dump()
if result.detail:
response_payload["detail"] = result.detail
return _build_response(response_payload, result.status_code, result.response.retry_after)
@agent_router.get(
"/tasks/{task_id}",
response_model_exclude_none=True,
summary="Status eines Tasks abrufen",
)
async def get_status(task_id: str) -> JSONResponse:
"""
Liefert den Status eines Tasks.
"""
status_map = {
"processing": status.HTTP_202_ACCEPTED,
"succeeded": status.HTTP_200_OK,
"failed": status.HTTP_500_INTERNAL_SERVER_ERROR,
"cancelled": status.HTTP_409_CONFLICT,
"expired": status.HTTP_410_GONE,
}
try:
status_response = _service.get_status(task_id)
except KeyError as exc:
raise HTTPException(status_code=status.HTTP_404_NOT_FOUND, detail=str(exc)) from exc
status_code = status_map.get(status_response.status, status.HTTP_200_OK)
payload = status_response.model_dump()
if status_response.status in {"failed", "cancelled", "expired"}:
payload.setdefault("detail", status_response.status)
return _build_response(payload, status_code, status_response.retry_after)
@agent_router.post(
"/tasks/{task_id}/cancel",
response_model_exclude_none=True,
summary="Laufenden Task abbrechen",
)
async def cancel_task(task_id: str) -> JSONResponse:
"""
Markiert einen laufenden Task als abgebrochen.
"""
try:
status_response = _service.cancel_task(task_id)
except KeyError as exc:
raise HTTPException(status_code=status.HTTP_404_NOT_FOUND, detail=str(exc)) from exc
payload = status_response.model_dump()
payload.setdefault("detail", "task_cancelled")
return _build_response(payload, status.HTTP_200_OK, status_response.retry_after)

310
01_Modulerweiterungen/Entworfener_Code/app/src/agent_api/service.py
View File

@@ -1,310 +0,0 @@
"""Service-Schicht für das Agenten-API-Modul."""
from __future__ import annotations
import threading
import uuid
from dataclasses import dataclass, field
from datetime import datetime, timedelta, timezone
from typing import Any, Dict, Mapping, Optional
from .config import get_config
from .llm_client import LLMClient
from .models import TaskCreate, TaskStatusResponse
from .worker_adapter import WorkerAdapter
_PROCESSING_STATUS = "processing"
_SUCCEEDED_STATUS = "succeeded"
_FAILED_STATUS = "failed"
_CANCELLED_STATUS = "cancelled"
_EXPIRED_STATUS = "expired"
@dataclass
class _TaskEntry:
payload: TaskCreate
status: str = _PROCESSING_STATUS
result: Optional[Dict[str, Any]] = None
created_at: datetime = field(
default_factory=lambda: datetime.now(timezone.utc)
)
expires_at: datetime = field(
default_factory=lambda: datetime.now(timezone.utc) + timedelta(minutes=5)
)
last_heartbeat: datetime = field(
default_factory=lambda: datetime.now(timezone.utc)
)
detail: Optional[str] = None
retry_after: Optional[int] = None
error: Optional[str] = None
def to_response(self, task_id: str) -> TaskStatusResponse:
return TaskStatusResponse(
task_id=task_id,
status=self.status,
result=self.result,
expires_at=self.expires_at if self.expires_at.tzinfo else self.expires_at.replace(tzinfo=timezone.utc),
retry_after=self.retry_after,
)
@dataclass
class SubmitResult:
response: TaskStatusResponse
status_code: int
detail: Optional[str] = None
class AgentService:
"""
Koordiniert die Lebenszyklen von Tasks, Worker-Adapter und LLM-Client.
"""
def __init__(
self,
*,
worker_adapter: Optional[WorkerAdapter] = None,
llm_client: Optional[LLMClient] = None,
) -> None:
self._lock = threading.RLock()
self._tasks: Dict[str, _TaskEntry] = {}
self._worker = worker_adapter or WorkerAdapter()
self._llm_client = llm_client or LLMClient()
# ------------------------------------------------------------------ #
# Öffentliche API
# ------------------------------------------------------------------ #
def submit_task(self, payload: TaskCreate) -> SubmitResult:
"""
Legt einen neuen Task an und triggert die Verarbeitung entsprechend
der konfigurierten Ausführungsstrategie.
"""
config = self._current_config()
execution_cfg = self._execution_config(config)
queue_ttl = int(execution_cfg.get("queue_ttl_seconds", 300))
retry_after = int(execution_cfg.get("heartbeat_interval_seconds", 10))
timeout_seconds = float(execution_cfg.get("response_timeout_seconds", 30))
default_mode = str(execution_cfg.get("mode", "async")).lower()
mode = default_mode
if payload.sync is True:
mode = "sync"
elif payload.sync is False:
mode = "async"
now = datetime.now(timezone.utc)
task_id = str(uuid.uuid4())
entry = _TaskEntry(
payload=payload,
status=_PROCESSING_STATUS,
created_at=now,
expires_at=now + timedelta(seconds=queue_ttl),
last_heartbeat=now,
retry_after=retry_after,
)
with self._lock:
self._tasks[task_id] = entry
thread = threading.Thread(
target=self.process_task,
args=(task_id,),
name=f"agent-task-{task_id}",
daemon=True,
)
thread.start()
if mode == "async":
return SubmitResult(
response=entry.to_response(task_id),
status_code=202,
detail=None,
)
# synchroner Modus
thread.join(timeout_seconds)
if thread.is_alive():
return SubmitResult(
response=self._task_response(task_id),
status_code=408,
detail="still_running",
)
entry = self._task_entry(task_id)
if entry.status == _SUCCEEDED_STATUS:
return SubmitResult(
response=entry.to_response(task_id),
status_code=200,
detail=None,
)
if entry.status == _FAILED_STATUS:
return SubmitResult(
response=entry.to_response(task_id),
status_code=500,
detail=entry.detail or "task_failed",
)
if entry.status == _CANCELLED_STATUS:
return SubmitResult(
response=entry.to_response(task_id),
status_code=409,
detail=entry.detail or "task_cancelled",
)
if entry.status == _EXPIRED_STATUS:
return SubmitResult(
response=entry.to_response(task_id),
status_code=410,
detail=entry.detail or "task_expired",
)
return SubmitResult(
response=entry.to_response(task_id),
status_code=202,
detail=None,
)
def process_task(self, task_id: str) -> None:
"""
Führt den Worker- und LLM-Workflow für einen gegebenen Task aus.
"""
try:
entry = self._task_entry(task_id)
except KeyError:
return
if entry.status in {_CANCELLED_STATUS, _EXPIRED_STATUS}:
return
self.update_heartbeat(task_id)
try:
prepared = self._worker.pre_process(entry.payload)
prompt = self._worker.run_task(prepared)
llm_response = self._llm_client.generate(prompt)
result_payload = {
"prepared_prompt": prompt,
"llm_response": llm_response,
}
now = datetime.now(timezone.utc)
with self._lock:
stored = self._tasks.get(task_id)
if stored is None or stored.status in {_CANCELLED_STATUS, _EXPIRED_STATUS}:
return
stored.status = _SUCCEEDED_STATUS
stored.result = result_payload
stored.last_heartbeat = now
stored.detail = None
stored.retry_after = None
except Exception as exc: # pragma: no cover - Fehlerpfad
now = datetime.now(timezone.utc)
with self._lock:
stored = self._tasks.get(task_id)
if stored is None:
return
if stored.status not in {_CANCELLED_STATUS, _EXPIRED_STATUS}:
stored.status = _FAILED_STATUS
stored.result = {"error": str(exc)}
stored.detail = exc.__class__.__name__
stored.error = repr(exc)
stored.last_heartbeat = now
stored.retry_after = None
finally:
self._finalize_task(task_id)
def get_status(self, task_id: str) -> TaskStatusResponse:
"""
Liefert den aktuellen Status eines Tasks.
"""
entry = self._task_entry(task_id)
self._evaluate_expiration(task_id, entry)
return self._task_response(task_id)
def cancel_task(self, task_id: str) -> TaskStatusResponse:
"""
Markiert einen laufenden Task als abgebrochen.
"""
entry = self._task_entry(task_id)
with self._lock:
if entry.status in {_SUCCEEDED_STATUS, _FAILED_STATUS, _EXPIRED_STATUS}:
return entry.to_response(task_id)
entry.status = _CANCELLED_STATUS
entry.result = entry.result or {"detail": "Task cancelled by client"}
entry.detail = "task_cancelled"
entry.last_heartbeat = datetime.now(timezone.utc)
entry.retry_after = None
return self._task_response(task_id)
def update_heartbeat(self, task_id: str) -> None:
"""
Aktualisiert den Heartbeat eines Tasks und verlängert dessen TTL.
"""
entry = self._task_entry(task_id)
config = self._current_config()
execution_cfg = self._execution_config(config)
queue_ttl = int(execution_cfg.get("queue_ttl_seconds", 300))
now = datetime.now(timezone.utc)
with self._lock:
stored = self._tasks.get(task_id)
if stored is None:
return
stored.last_heartbeat = now
stored.expires_at = now + timedelta(seconds=queue_ttl)
# ------------------------------------------------------------------ #
# Interne Hilfsfunktionen
# ------------------------------------------------------------------ #
def _task_entry(self, task_id: str) -> _TaskEntry:
with self._lock:
entry = self._tasks.get(task_id)
if entry is None:
raise KeyError(f"Task '{task_id}' not found")
return entry
def _task_response(self, task_id: str) -> TaskStatusResponse:
entry = self._task_entry(task_id)
return entry.to_response(task_id)
def _finalize_task(self, task_id: str) -> None:
entry = self._task_entry(task_id)
self._evaluate_expiration(task_id, entry)
def _evaluate_expiration(self, task_id: str, entry: _TaskEntry) -> None:
now = datetime.now(timezone.utc)
if entry.status in {_SUCCEEDED_STATUS, _FAILED_STATUS, _CANCELLED_STATUS}:
return
if now > entry.expires_at:
with self._lock:
stored = self._tasks.get(task_id)
if stored is None:
return
stored.status = _EXPIRED_STATUS
stored.result = stored.result or {"detail": "Task expired"}
stored.detail = "task_expired"
stored.last_heartbeat = now
stored.retry_after = None
@staticmethod
def _current_config() -> Dict[str, Any]:
return get_config()
@staticmethod
def _execution_config(config: Mapping[str, Any]) -> Mapping[str, Any]:
agent_cfg = config.get("agent_api", {})
return agent_cfg.get("execution", {})
# ------------------------------------------------------------------ #
# Debug-/Test-Hilfen
# ------------------------------------------------------------------ #
def clear_tasks(self) -> None:
"""
Löscht alle gemerkten Tasks. Hilfreich für Tests.
"""
with self._lock:
self._tasks.clear()
def list_tasks(self) -> Dict[str, _TaskEntry]:
"""
Gibt einen Snapshot der aktuellen Tasks zurück.
"""
with self._lock:
return dict(self._tasks)

50
01_Modulerweiterungen/Entworfener_Code/app/src/agent_api/worker_adapter.py
View File

@@ -1,50 +0,0 @@
"""Worker-Adapter für das Agenten-API-Modul."""
from __future__ import annotations
import datetime as _dt
from typing import Any, Dict
from .models import TaskCreate
class WorkerAdapter:
"""
Einfacher Inline-Worker-Adapter.
Die Implementierung dient als Stub für spätere Erweiterungen. Sie nimmt den
eingehenden Payload entgegen, reichert diesen mit Metadaten an und liefert
einen Prompt für den LLM-Client zurück.
"""
@staticmethod
def pre_process(payload: TaskCreate) -> Dict[str, Any]:
"""
Bereitet den Task-Payload für den LLM-Aufruf vor.
Args:
payload: Valider TaskCreate-Payload.
Returns:
Dict[str, Any]: Strukturierter Prompt inklusive Metadaten.
"""
timestamp = _dt.datetime.now(_dt.timezone.utc)
prepared_prompt: Dict[str, Any] = {
"user_input": payload.user_input,
"context": payload.context or {},
"metadata": payload.metadata or {},
"created_at": timestamp.isoformat(),
}
return prepared_prompt
@staticmethod
def run_task(prepared: Dict[str, Any]) -> Dict[str, Any]:
"""
Führt optionale Worker-Logik aus und liefert den Prompt für den LLM-Client.
Args:
prepared: Vom WorkerAdapter.pre_process erzeugter Prompt.
Returns:
Dict[str, Any]: Prompt für den LLM-Client.
"""
return prepared

1
work/entworfener_code_working_copy/.venv/bin/python Symbolic link
View File

@@ -0,0 +1 @@
python3

1
work/entworfener_code_working_copy/.venv/bin/python3 Symbolic link
View File

@@ -0,0 +1 @@
/usr/bin/python3

1
work/entworfener_code_working_copy/.venv/bin/python3.10 Symbolic link
View File

@@ -0,0 +1 @@
python3

1
work/entworfener_code_working_copy/.venv/lib64 Symbolic link
View File

@@ -0,0 +1 @@
lib

3
work/entworfener_code_working_copy/.venv/pyvenv.cfg Normal file
View File

@@ -0,0 +1,3 @@
home = /usr/bin
include-system-site-packages = false
version = 3.10.12

1
work/entworfener_code_working_copy/1 Normal file
View File

@@ -0,0 +1 @@
/usr/bin/python3

33
work/entworfener_code_working_copy/Dockerfile Normal file
View File

@@ -0,0 +1,33 @@
# syntax=docker/dockerfile:1
ARG PYTHON_VERSION=3.11
FROM python:${PYTHON_VERSION}-slim AS base
ENV PYTHONDONTWRITEBYTECODE=1 \
PYTHONUNBUFFERED=1
# Optional: Systemabhängigkeiten (Minimalsatz)
RUN apt-get update && apt-get install -y --no-install-recommends \
ca-certificates curl build-essential \
&& rm -rf /var/lib/apt/lists/*
WORKDIR /app
# Dependencies zuerst kopieren für bessere Layer-Caching-Effizienz
COPY requirements.txt /app/requirements.txt
RUN pip install --no-cache-dir -r /app/requirements.txt
# Anwendungscode
COPY . /app
# Non-root User
RUN useradd -m appuser && chown -R appuser:appuser /app
USER appuser
# Standard-Config-Pfad für die App-Factory (neue Struktur unter /app/app/config)
ENV APP_CONFIG_PATH=/app/app/config/config.yaml
EXPOSE 8000
# Start über Uvicorn-Factory aus neuer Struktur: app/start.py stellt app_factory() bereit
CMD ["uvicorn", "app.start:app_factory", "--factory", "--host", "0.0.0.0", "--port", "8000"]

BIN
work/entworfener_code_working_copy/app/__pycache__/start.cpython-310.pyc Normal file
View File

Binary file not shown.

BIN
work/entworfener_code_working_copy/app/code/app/__pycache__/main.cpython-310.pyc Normal file
View File

Binary file not shown.

67
work/entworfener_code_working_copy/app/code/app/main.py Normal file
View File

@@ -0,0 +1,67 @@
from typing import Any, Dict, List, Optional
from fastapi import FastAPI
from fastapi.middleware.cors import CORSMiddleware
def create_app(config: Optional[Dict[str, Any]] = None) -> FastAPI:
"""
Erzeugt und konfiguriert die FastAPI-Anwendung.
Args:
config: Konfigurations-Dictionary (z. B. aus app/config/config.yaml geladen).
Returns:
FastAPI: Konfigurierte FastAPI-App.
"""
cfg: Dict[str, Any] = config or {}
meta: Dict[str, Any] = cfg.get("meta", {})
app_cfg: Dict[str, Any] = cfg.get("app", {})
cors_cfg: Dict[str, Any] = cfg.get("cors", {})
app = FastAPI(
title=app_cfg.get("name", "entworfener_code_service"),
version=str(meta.get("version", "0.1.0")),
docs_url="/docs",
redoc_url="/redoc",
)
# CORS
allow_origins: List[str] = cors_cfg.get("allow_origins", ["*"])
allow_credentials: bool = bool(cors_cfg.get("allow_credentials", True))
allow_methods: List[str] = cors_cfg.get("allow_methods", ["*"])
allow_headers: List[str] = cors_cfg.get("allow_headers", ["*"])
app.add_middleware(
CORSMiddleware,
allow_origins=allow_origins,
allow_credentials=allow_credentials,
allow_methods=allow_methods,
allow_headers=allow_headers,
)
# Router registrieren (unter app/code/api)
try:
from api.router import api_router
app.include_router(api_router, prefix="/api")
except Exception:
# Router existiert evtl. noch nicht beim ersten Scaffold
pass
try:
import agent_api
app.include_router(agent_api.agent_router, prefix="/api")
except Exception:
# Agenten-API ist optional und wird bei fehlender Implementierung ignoriert
pass
@app.get("/health", tags=["health"])
async def health() -> Dict[str, str]:
"""
Einfache Health-Check-Route.
"""
return {"status": "ok"}
return app

46
work/entworfener_code_working_copy/app/config/agent_api.yaml Normal file
View File

@@ -0,0 +1,46 @@
agent_api:
metadata:
version: "0.1.0"
description: "Agent Gateway für Worker + OpenAI-kompatible LLMs"
http:
base_path: "/api/agent/v1"
timeout_seconds: 60
rate_limit_per_minute: 120
enable_cors: true
auth:
api_key_header: "x-agent-api-key"
allowed_keys: []
allow_unauthenticated: true
worker:
adapter: "inline"
endpoint: null
timeout_seconds: 30
llm:
provider: "openai"
base_url: "https://api.openai.com/v1"
model: "gpt-4o-mini"
temperature: 0.2
max_tokens: 1200
api_key: null
request_timeout_seconds: 45
retry:
max_attempts: 3
backoff_seconds: 2
execution:
mode: "async"
response_timeout_seconds: 30
queue_ttl_seconds: 300
heartbeat_interval_seconds: 10
allow_long_polling: true
logging:
enabled: true
log_payloads: false
redact_fields:
- "user_input"
- "llm_response"

60
work/entworfener_code_working_copy/app/config/config.yaml Normal file
View File

@@ -0,0 +1,60 @@
# Application settings
app:
name: "entworfener_code_service"
host: "0.0.0.0"
port: 8000
reload: true
# Logging settings
logging:
level: "INFO"
config_file: "config/logging.yaml"
# CORS settings
cors:
allow_origins:
- "*"
allow_credentials: true
allow_methods:
- "*"
allow_headers:
- "*"
# Paths (relativ zu app/)
paths:
log_dir: "logs"
data_dir: "data"
# Server timeouts
server:
timeout_keep_alive: 5
backlog: 2048
# Metadata
meta:
version: "0.1.0"
environment: "dev"
# Erweiterungen: interne/externe Logging-Module (01_Modulerweiterungen)
logging_internal:
enabled: true
db_path: "data/internal_logs.sqlite"
clean_database: false
retention_days: 30
max_entries: 100000
vacuum_on_start: true
batch_write: 100
logging_external:
enabled: false
type: "postgresql" # mysql | postgresql
host: "localhost"
port: 5432
user: "logger"
password: null # Secrets per Env-Var/Keystore, siehe [Security.md](01_Modulerweiterungen/Planung/Security.md:1)
database: "logs"
sslmode: "prefer"
pool_size: 5
connect_timeout: 10
write_buffer_size: 100
fallback_to_internal_on_error: true

57
work/entworfener_code_working_copy/app/config/logging.yaml Normal file
View File

@@ -0,0 +1,57 @@
version: 1
disable_existing_loggers: false
formatters:
standard:
format: "%(asctime)s | %(levelname)s | %(name)s | %(filename)s:%(lineno)d | %(message)s"
datefmt: "%Y-%m-%d %H:%M:%S"
access:
format: '%(asctime)s | %(levelname)s | %(client_addr)s - "%(request_line)s" %(status_code)s'
datefmt: "%Y-%m-%d %H:%M:%S"
handlers:
console:
class: logging.StreamHandler
level: INFO
formatter: standard
stream: ext://sys.stdout
app_file:
class: logging.handlers.TimedRotatingFileHandler
level: INFO
formatter: standard
filename: app/logs/app.log
when: D
interval: 1
backupCount: 7
encoding: utf-8
access_file:
class: logging.handlers.TimedRotatingFileHandler
level: INFO
formatter: access
filename: app/logs/access.log
when: D
interval: 1
backupCount: 7
encoding: utf-8
loggers:
uvicorn:
level: INFO
handlers: [console, app_file]
propagate: false
uvicorn.error:
level: INFO
handlers: [console, app_file]
propagate: false
uvicorn.access:
level: INFO
handlers: [console, access_file]
propagate: false
root:
level: INFO
handlers: [console, app_file]

23
work/entworfener_code_working_copy/app/config/logging_external.yaml Normal file
View File

@@ -0,0 +1,23 @@
# Konfiguration für externes Logging-Modul (MySQL/PostgreSQL über SQLAlchemy)
# Diese Datei folgt dem Schema der Konfigurationen unter
# 00_Globale_Richtlinien/Entworfener_Code/app/config/
# und kann später in die Haupt-App übernommen werden.
#
# Hinweise:
# - Secrets (user/password) sollten via Umgebungsvariablen überschrieben werden:
# LOG_EXT_USER, LOG_EXT_PASSWORD
# - Das Modul unterstützt Fallback ins interne SQLite-Logging, wenn aktiviert.
logging_external:
enabled: false # Modul aktivieren/deaktivieren
type: "postgresql" # "postgresql" | "mysql"
host: "localhost" # DB-Host
port: 5432 # 5432 für PostgreSQL, 3306 für MySQL
user: null # via ENV: LOG_EXT_USER (überschreibt diesen Wert)
password: null # via ENV: LOG_EXT_PASSWORD (überschreibt diesen Wert)
database: "logs" # Datenbankname
sslmode: "prefer" # nur für PostgreSQL relevant: disable|allow|prefer|require|verify-ca|verify-full
pool_size: 5 # Größe des Connection-Pools
connect_timeout: 10 # Verbindungs-Timeout in Sekunden
write_buffer_size: 100 # (reserviert für spätere Batch-Nutzung)
fallback_to_internal_on_error: true # Fallback in internes SQLite-Logging bei Fehlern

12
work/entworfener_code_working_copy/app/config/logging_internal.yaml Normal file
View File

@@ -0,0 +1,12 @@
# Konfiguration für internes Logging-Modul (SQLite)
# Diese Datei folgt dem selben Schema wie die Konfigurationen unter 00_Globale_Richtlinien/Entworfener_Code/app/config
# Sie kann später in die Haupt-App übernommen werden; Werte sind Default-Beispiele.
logging_internal:
enabled: true # Modul aktivieren/deaktivieren
db_path: "data/internal_logs.sqlite" # relativ zu app/
clean_database: false # Datenbank beim Start bereinigen (vorsichtig einsetzen)
retention_days: 30 # Aufbewahrungsdauer in Tagen
max_entries: 100000 # Max. Anzahl an Logeinträgen
vacuum_on_start: true # VACUUM nach Start/Cleanup
batch_write: 100 # zukünftige Batch-Größe (derzeit optional)

BIN
work/entworfener_code_working_copy/app/data/internal_logs.sqlite Normal file
View File

Binary file not shown.

8
work/entworfener_code_working_copy/app/logs/app.log Normal file
View File

@@ -0,0 +1,8 @@
2025-11-14 00:04:28 | WARNING | src.logging_setup | logging_setup.py:92 | logging_internal Modul nicht importierbar; interner DB-Handler wird nicht aktiviert.
2025-11-14 00:04:28 | INFO | start | start.py:149 | Starte Uvicorn mit Reload (Factory-Modus, app/code).
2025-11-14 00:04:31 | WARNING | src.logging_setup | logging_setup.py:92 | logging_internal Modul nicht importierbar; interner DB-Handler wird nicht aktiviert.
2025-11-14 00:04:31 | INFO | watchfiles.main | main.py:308 | 6 changes detected
2025-11-14 00:16:16 | INFO | src.logging_setup | logging_setup.py:114 | Interner SQLite-Log-Handler aktiv: /home/hei/Downloads/porjekt/Unbenannter Ordner 2/work/entworfener_code_working_copy/app/data/internal_logs.sqlite
2025-11-14 00:16:16 | INFO | start | start.py:149 | Starte Uvicorn mit Reload (Factory-Modus, app/code).
2025-11-14 00:16:19 | INFO | src.logging_setup | logging_setup.py:114 | Interner SQLite-Log-Handler aktiv: /home/hei/Downloads/porjekt/Unbenannter Ordner 2/work/entworfener_code_working_copy/app/data/internal_logs.sqlite
2025-11-14 00:16:19 | INFO | watchfiles.main | main.py:308 | 6 changes detected

39
work/entworfener_code_working_copy/app/runtime_config.yaml Normal file
View File

@@ -0,0 +1,39 @@
# Laufzeitkonfiguration (Basis)
# Diese Datei liegt neben start.py und wird zur Initialisierung von Logging & Modulen verwendet.
global:
level:
beschreibung: Logging-Level
wert: INFO
max_log_size:
beschreibung: Maximale Gesamtgröße aller Log-Dateien (in MB)
wert: 10
retention_days:
beschreibung: Anzahl der Tage, die Log-Dateien aufbewahrt werden
wert: 7
log_dir:
beschreibung: Relativer Pfad zum Log-Verzeichnis (relativ zu app/)
wert: logs
modules:
webserver:
enabled:
beschreibung: Separaten Webserver aktivieren (liefert README aus)
wert: true
host:
beschreibung: Bind-Adresse des Webservers
wert: "127.0.0.1"
port:
beschreibung: Port des Webservers
wert: 8300
task_queue:
mode:
beschreibung: Abarbeitungsmodus der Aufgaben (sequential | parallel)
wert: sequential
worker_count:
beschreibung: Anzahl paralleler Worker, wenn mode=parallel
wert: 2
task_timeout_seconds:
beschreibung: Timeout für einzelne Aufgaben in Sekunden
wert: 300

10
work/entworfener_code_working_copy/app/src/__init__.py Normal file
View File

@@ -0,0 +1,10 @@
"""
Paket für Laufzeit-Module (Config/Logging/etc.) der Referenz-App.
"""
from __future__ import annotations
from .config_loader import Settings, load_runtime_config
from .logging_setup import init_logging
__all__ = ["Settings", "load_runtime_config", "init_logging"]

BIN
work/entworfener_code_working_copy/app/src/__pycache__/__init__.cpython-310.pyc Normal file
View File

Binary file not shown.

BIN
work/entworfener_code_working_copy/app/src/__pycache__/agent_api.cpython-310.pyc Normal file
View File

Binary file not shown.

BIN
work/entworfener_code_working_copy/app/src/__pycache__/config_loader.cpython-310.pyc Normal file
View File

Binary file not shown.

BIN
work/entworfener_code_working_copy/app/src/__pycache__/logging_internal.cpython-310.pyc Normal file
View File

Binary file not shown.

BIN
work/entworfener_code_working_copy/app/src/__pycache__/logging_setup.cpython-310.pyc Normal file
View File

Binary file not shown.

416
work/entworfener_code_working_copy/app/src/agent_api.py Normal file
View File

@@ -0,0 +1,416 @@
"""
Agenten-API FastAPI Router für Aufgabenverarbeitung.
"""
from __future__ import annotations
import logging
import os
import threading
import time
import uuid
from dataclasses import dataclass
from datetime import datetime, timedelta, timezone
from pathlib import Path
from typing import Any, Dict, List, Optional, TypedDict
import yaml
from fastapi import APIRouter, HTTPException, status
from fastapi.responses import JSONResponse
from pydantic import BaseModel, Field
logger = logging.getLogger(__name__)
@dataclass(frozen=True)
class ExecutionSettings:
"""Konfiguration der Ausführungspipeline."""
mode: str = "async"
queue_ttl_seconds: int = 300
heartbeat_interval_seconds: int = 10
response_timeout_seconds: int = 30
@dataclass(frozen=True)
class LLMSettings:
"""Konfiguration des LLM-Clients."""
provider: str = "local_stub"
model: str = "local-agent"
api_base_url: Optional[str] = None
api_key: Optional[str] = None
request_timeout_seconds: int = 15
@dataclass(frozen=True)
class AgentAPIConfig:
"""Gebündelte Agenten-API Konfiguration."""
execution: ExecutionSettings
llm: LLMSettings
class TaskRequest(BaseModel):
"""Eingangsmodell für eine Agenten-Aufgabe."""
user_input: str = Field(..., description="Auslöser der Task in natürlicher Sprache.")
context: Optional[Dict[str, Any]] = Field(default=None, description="Optionale Kontextdaten des Aufrufs.")
metadata: Optional[Dict[str, Any]] = Field(default=None, description="Zusätzliche Metadaten zur Korrelation.")
sync: Optional[bool] = Field(default=None, description="Erzwingt synchrone Verarbeitung, falls gesetzt.")
class TaskData(TypedDict):
"""Interne Darstellung eines Tasks im Speicher."""
task_id: str
status: str
result: Optional[Dict[str, Any]]
error: Optional[str]
created_at: datetime
updated_at: datetime
expires_at: datetime
last_heartbeat: datetime
retry_after: int
request: TaskRequest
def _utcnow() -> datetime:
"""Gibt die aktuelle UTC-Zeit zurück."""
return datetime.now(timezone.utc)
def _to_iso(dt: datetime) -> str:
"""Formatiert Datumswerte als ISO8601-String mit Z-Suffix."""
return dt.astimezone(timezone.utc).isoformat().replace("+00:00", "Z")
def _positive_int(value: Any, default: int) -> int:
"""Stellt sicher, dass ein Integer positiv ist, andernfalls gilt der Default."""
try:
candidate = int(value)
if candidate > 0:
return candidate
except (TypeError, ValueError):
pass
return default
def _candidate_config_paths(module_path: Path) -> List[Path]:
"""Ermittelt potenzielle Konfigurationspfade gemäß Projektstandard."""
candidates: List[Path] = []
local_app_path = module_path.parent.parent
candidates.append(local_app_path / "config" / "agent_api.yaml")
for parent in module_path.parents:
candidate = parent / "00_Globale_Richtlinien" / "Entworfener_Code" / "app" / "config" / "agent_api.yaml"
if candidate not in candidates:
candidates.append(candidate)
return candidates
def _load_agent_config() -> AgentAPIConfig:
"""Lädt die Agenten-Konfiguration und führt sie mit Defaults zusammen."""
module_path = Path(__file__).resolve()
config_data: Dict[str, Any] = {}
for path in _candidate_config_paths(module_path):
if not path.exists():
continue
try:
with path.open("r", encoding="utf-8") as handle:
loaded = yaml.safe_load(handle) or {}
config_data = loaded
logger.debug("Agenten-API-Konfiguration aus %s geladen.", path)
break
except Exception as exc:
logger.warning("Fehler beim Laden der Agenten-API-Konfiguration aus %s: %s", path, exc)
execution_data = config_data.get("execution", {})
llm_data = config_data.get("llm", {})
mode = str(execution_data.get("mode", "async")).lower()
if mode not in {"async", "sync"}:
mode = "async"
execution = ExecutionSettings(
mode=mode,
queue_ttl_seconds=_positive_int(execution_data.get("queue_ttl_seconds"), default=300),
heartbeat_interval_seconds=_positive_int(execution_data.get("heartbeat_interval_seconds"), default=10),
response_timeout_seconds=_positive_int(execution_data.get("response_timeout_seconds"), default=30),
)
env_api_key = os.getenv("AGENT_API_LLM_KEY")
api_key = env_api_key if env_api_key else llm_data.get("api_key")
llm = LLMSettings(
provider=str(llm_data.get("provider", "local_stub")),
model=str(llm_data.get("model", "local-agent")),
api_base_url=llm_data.get("api_base_url"),
api_key=str(api_key) if api_key else None,
request_timeout_seconds=_positive_int(llm_data.get("request_timeout_seconds"), default=15),
)
return AgentAPIConfig(execution=execution, llm=llm)
CONFIG = _load_agent_config()
class TaskRegistry:
"""Thread-sichere Verwaltung der In-Memory-Tasks."""
def __init__(self) -> None:
self._tasks: Dict[str, TaskData] = {}
self._lock = threading.Lock()
def create(self, request: TaskRequest, ttl_seconds: int, retry_after: int) -> TaskData:
"""Erzeugt eine neue Task-Struktur und speichert sie."""
now = _utcnow()
task: TaskData = {
"task_id": str(uuid.uuid4()),
"status": "processing",
"result": None,
"error": None,
"created_at": now,
"updated_at": now,
"expires_at": now + timedelta(seconds=ttl_seconds),
"last_heartbeat": now,
"retry_after": max(retry_after, 1),
"request": request,
}
with self._lock:
self._tasks[task["task_id"]] = task
return task.copy()
def get(self, task_id: str) -> Optional[TaskData]:
"""Liefert eine Kopie der Task und aktualisiert Expirationen."""
with self._lock:
task = self._tasks.get(task_id)
if task is None:
return None
self._expire_task_locked(task)
return task.copy()
def get_request(self, task_id: str) -> Optional[TaskRequest]:
"""Liefert das ursprüngliche Request-Objekt."""
with self._lock:
task = self._tasks.get(task_id)
if task is None:
return None
return task["request"]
def heartbeat(self, task_id: str) -> None:
"""Aktualisiert den Heartbeat einer laufenden Task."""
with self._lock:
task = self._tasks.get(task_id)
if task is None or task["status"] != "processing":
return
now = _utcnow()
task["last_heartbeat"] = now
task["updated_at"] = now
def mark_succeeded(self, task_id: str, result: Dict[str, Any], ttl_seconds: int) -> Optional[TaskData]:
"""Markiert eine Task als abgeschlossen."""
with self._lock:
task = self._tasks.get(task_id)
if task is None:
return None
if task["status"] in {"cancelled", "expired"}:
return task.copy()
now = _utcnow()
task["status"] = "succeeded"
task["result"] = result
task["error"] = None
task["updated_at"] = now
task["last_heartbeat"] = now
task["expires_at"] = now + timedelta(seconds=ttl_seconds)
return task.copy()
def mark_failed(self, task_id: str, error: str, ttl_seconds: int) -> Optional[TaskData]:
"""Markiert eine Task als fehlgeschlagen."""
with self._lock:
task = self._tasks.get(task_id)
if task is None:
return None
now = _utcnow()
task["status"] = "failed"
task["result"] = {"error": error}
task["error"] = error
task["updated_at"] = now
task["last_heartbeat"] = now
task["expires_at"] = now + timedelta(seconds=ttl_seconds)
return task.copy()
def cancel(self, task_id: str) -> Optional[TaskData]:
"""Bricht eine Task ab und markiert sie als cancelled."""
with self._lock:
task = self._tasks.get(task_id)
if task is None:
return None
now = _utcnow()
if task["status"] in {"succeeded", "failed", "expired", "cancelled"}:
self._expire_task_locked(task)
return task.copy()
task["status"] = "cancelled"
task["error"] = "cancelled by client"
task["result"] = None
task["updated_at"] = now
task["last_heartbeat"] = now
task["expires_at"] = now
return task.copy()
def _expire_task_locked(self, task: TaskData) -> None:
"""Setzt den Status auf expired, falls das Ablaufdatum erreicht ist."""
now = _utcnow()
if now < task["expires_at"] or task["status"] == "expired":
return
task["status"] = "expired"
task["error"] = "expired"
task["updated_at"] = now
REGISTRY = TaskRegistry()
def _local_stub_result(request: TaskRequest) -> Dict[str, Any]:
"""Lokale Fallback-Implementierung ohne externe LLM-Abhängigkeit."""
return {
"output": f"{request.user_input} [local_stub]",
"provider": "local_stub",
"metadata": {
"mode": CONFIG.execution.mode,
"context_present": bool(request.context),
"metadata_present": bool(request.metadata),
},
}
def _invoke_llm(request: TaskRequest) -> Dict[str, Any]:
"""Ruft das konfigurierte LLM auf oder fällt auf die Stub-Implementierung zurück."""
api_key = CONFIG.llm.api_key
base_url = CONFIG.llm.api_base_url
if not api_key or not base_url:
return _local_stub_result(request)
try:
import httpx # type: ignore
except ModuleNotFoundError:
logger.warning("httpx nicht verfügbar, fallback auf lokalen Stub.")
return _local_stub_result(request)
headers: Dict[str, str] = {}
if api_key:
headers["Authorization"] = f"Bearer {api_key}"
payload = {
"model": CONFIG.llm.model,
"input": request.user_input,
"context": request.context or {},
"metadata": request.metadata or {},
}
timeout = CONFIG.llm.request_timeout_seconds
try:
with httpx.Client(timeout=timeout) as client:
response = client.post(str(base_url), json=payload, headers=headers)
response.raise_for_status()
data = response.json()
except Exception as exc:
logger.warning("LLM-Request fehlgeschlagen (%s), fallback auf Stub.", exc)
return _local_stub_result(request)
return {
"output": data.get("output") or data.get("result") or data,
"provider": CONFIG.llm.provider,
"metadata": {
"source": "llm",
"model": CONFIG.llm.model,
},
}
def _execute_request(request: TaskRequest) -> Dict[str, Any]:
"""Führt Vorverarbeitung aus und delegiert an das LLM."""
# Simulierter Pre-Processing-Schritt sowie Heartbeat-Update
time.sleep(0.01)
return _invoke_llm(request)
def _process_task(task_id: str) -> None:
"""Hintergrundverarbeitung einer Task."""
request = REGISTRY.get_request(task_id)
if request is None:
return
try:
REGISTRY.heartbeat(task_id)
result = _execute_request(request)
REGISTRY.heartbeat(task_id)
REGISTRY.mark_succeeded(task_id, result, CONFIG.execution.queue_ttl_seconds)
except Exception as exc:
logger.exception("Fehler bei der Bearbeitung von Task %s", task_id)
REGISTRY.mark_failed(task_id, str(exc), CONFIG.execution.queue_ttl_seconds)
def _serialize_task(task: TaskData, include_result: bool = True, include_detail: bool = True) -> Dict[str, Any]:
"""Konvertiert den internen Task-Status in eine API-Antwort."""
payload: Dict[str, Any] = {
"task_id": task["task_id"],
"status": task["status"],
"expires_at": _to_iso(task["expires_at"]),
}
if task["status"] == "processing":
payload["retry_after"] = task["retry_after"]
if include_result and task["result"] is not None and task["status"] in {"succeeded", "failed"}:
payload["result"] = task["result"]
if include_detail and task.get("error"):
payload["detail"] = task["error"]
if include_detail and task["status"] == "cancelled" and "detail" not in payload:
payload["detail"] = "cancelled"
return payload
agent_router = APIRouter(prefix="/agent/v1", tags=["agent-api"])
@agent_router.post("/tasks")
def submit_task(request: TaskRequest) -> JSONResponse:
"""Erstellt eine neue Task und verarbeitet sie je nach Modus synchron oder asynchron."""
effective_mode = CONFIG.execution.mode
if request.sync is not None:
effective_mode = "sync" if request.sync else "async"
ttl = CONFIG.execution.queue_ttl_seconds
retry_after = CONFIG.execution.heartbeat_interval_seconds
task = REGISTRY.create(request, ttl_seconds=ttl, retry_after=retry_after)
task_id = task["task_id"]
worker = threading.Thread(
target=_process_task,
args=(task_id,),
name=f"agent-task-{task_id}",
daemon=True,
)
worker.start()
if effective_mode == "sync":
worker.join(timeout=CONFIG.execution.response_timeout_seconds)
final_task = REGISTRY.get(task_id)
if final_task is None:
raise HTTPException(status_code=status.HTTP_500_INTERNAL_SERVER_ERROR, detail="task_not_found")
if final_task["status"] == "processing":
content = {
"task_id": task_id,
"status": "processing",
"detail": "still_running",
"expires_at": _to_iso(final_task["expires_at"]),
"retry_after": final_task["retry_after"],
}
return JSONResponse(status_code=status.HTTP_408_REQUEST_TIMEOUT, content=content)
return JSONResponse(status_code=status.HTTP_200_OK, content=_serialize_task(final_task))
return JSONResponse(status_code=status.HTTP_202_ACCEPTED, content=_serialize_task(task, include_result=False, include_detail=False))
@agent_router.get("/tasks/{task_id}")
def get_task_status(task_id: str) -> Dict[str, Any]:
"""Liefert den Status einer bestehenden Task."""
task = REGISTRY.get(task_id)
if task is None:
raise HTTPException(status_code=status.HTTP_404_NOT_FOUND, detail="task_not_found")
return _serialize_task(task)
@agent_router.post("/tasks/{task_id}/cancel")
def cancel_task(task_id: str) -> Dict[str, Any]:
"""Bricht eine laufende Task ab."""
task = REGISTRY.cancel(task_id)
if task is None:
raise HTTPException(status_code=status.HTTP_404_NOT_FOUND, detail="task_not_found")
return _serialize_task(task)
__all__ = ["agent_router"]

239
work/entworfener_code_working_copy/app/src/config_loader.py Normal file
View File

@@ -0,0 +1,239 @@
from __future__ import annotations
from dataclasses import dataclass
from pathlib import Path
from typing import Any, Dict, Optional, Union, cast
import yaml
@dataclass
class Settings:
"""
Universeller Settings-Wrapper für die zur Laufzeit geladene YAML-Konfiguration.
Unterstützt zwei Schemata, um kompatibel mit den Anforderungen zu bleiben:
1) "Einfaches Variablen-Schema" (vom Nutzer gefordert):
global:
variablenname:
beschreibung: <text>
wert: <any>
modulname:
variablenname:
beschreibung: <text>
wert: <any>
2) "Verschachteltes Schema" (optionale, spätere Erweiterung):
global:
logging:
max_log_size: 10
retention_days: 14
modules:
webserver:
port: 8300
Abfrage-Regeln:
- Settings.value("global", "max_log_size")
- Prüft zuerst einfaches Schema: global.max_log_size.wert
- Dann verschachtelt: global.max_log_size
- Alternativ mit Dot-Pfad: value("global", "logging.max_log_size")
- Für Module:
- module_value("webserver", "port") oder module_value("webserver", "logging.max_log_size")
Hinweise:
- Diese Klasse validiert bewusst minimal, um ohne zusätzliche Dependencies auszukommen.
Detaillierte Validierung (z. B. via pydantic) kann später ergänzt werden.
"""
raw: Dict[str, Any]
base_dir: Path
# ---------- Public API ----------
def section(self, name: str) -> Dict[str, Any]:
return _ensure_dict(self.raw.get(name, {}))
def value(self, section: str, key: str, default: Any = None) -> Any:
"""
Liest einen Wert aus einem Abschnitt.
- Unterstützt Dot-Pfade (z. B. "logging.max_log_size").
- Unterstützt das "wert"-Wrapper-Format (variablenname: {beschreibung, wert}).
"""
node = self.section(section)
if not node:
return default
# 1) Einfacher Variablenname ohne Dot-Pfad
if "." not in key:
if key in node:
return _unwrap_variable(node.get(key), default)
# ggf. verschachteltes Schema: global[key]
return _unwrap_variable(node.get(key), default)
# 2) Dot-Pfad (verschachtelt)
current: Any = node
for part in key.split("."):
if isinstance(current, dict) and part in current:
current = cast(Dict[str, Any], current)[part]
else:
return default
return _unwrap_variable(current, default)
def module_value(self, module: str, key: str, default: Any = None) -> Any:
"""
Liest einen Wert im Modul-Abschnitt. Es werden folgende Orte geprüft:
- raw[module]
- raw.get("modules", {}).get(module)
"""
# Direktes Modul (einfaches Schema)
direct_section = _ensure_dict(self.raw.get(module, {}))
if direct_section:
out = _value_from_node(direct_section, key, default)
if out is not _MISSING:
return out
# Verschachteltes "modules"-Schema
modules_section = _ensure_dict(self.raw.get("modules", {}))
module_section = _ensure_dict(modules_section.get(module, {}))
if module_section:
out = _value_from_node(module_section, key, default)
if out is not _MISSING:
return out
return default
def get_logging_params(self) -> Dict[str, Any]:
"""
Liefert gebräuchliche Logging-Parameter mit sinnvollen Defaults.
Folgende Schlüssel werden versucht (in dieser Reihenfolge):
- level: global.level, global.logging.level
- max_log_size: global.max_log_size (MB), global.logging.max_log_size
- retention_days: global.retention_days, global.logging.retention_days
- log_dir: global.log_dir, global.paths.log_dir
"""
level = (
self.value("global", "level")
or self.value("global", "logging.level")
or "INFO"
)
max_log_size_mb = (
self.value("global", "max_log_size")
or self.value("global", "logging.max_log_size")
or 10
)
retention_days = (
self.value("global", "retention_days")
or self.value("global", "logging.retention_days")
or 7
)
log_dir = (
self.value("global", "log_dir")
or self.value("global", "paths.log_dir")
or "logs"
)
return {
"level": str(level),
"max_log_size_mb": int(max_log_size_mb),
"retention_days": int(retention_days),
"log_dir": str(log_dir),
}
def resolve_path(self, p: Union[str, Path]) -> Path:
"""
Löst relative Pfade relativ zur base_dir (app/) auf.
"""
pp = Path(p)
if pp.is_absolute():
return pp
return (self.base_dir / pp).resolve()
# ---------- Constructors ----------
@classmethod
def from_file(cls, path: Path, base_dir: Optional[Path] = None) -> "Settings":
data = _load_yaml_safe(path)
return cls(raw=data, base_dir=base_dir or path.parent)
@classmethod
def empty(cls, base_dir: Path) -> "Settings":
return cls(raw={}, base_dir=base_dir)
# ---------- Module-level helpers ----------
def load_runtime_config(config_path: Optional[Path] = None, base_dir: Optional[Path] = None) -> Settings:
"""
Lädt die Laufzeitkonfiguration. Standard:
- base_dir = app/ (Ordner der start.py)
- config_path = base_dir / "runtime_config.yaml"
Gibt Settings mit leerer Rohstruktur zurück, falls Datei fehlt.
"""
# Versuche app/ als Basis zu bestimmen: .../app/src/config_loader.py -> parents[1] = app/
default_base = Path(__file__).resolve().parents[1]
bdir = base_dir or default_base
cfg_path = config_path or (bdir / "runtime_config.yaml")
if cfg_path.exists():
return Settings.from_file(cfg_path, base_dir=bdir)
return Settings.empty(base_dir=bdir)
def _ensure_dict(v: Any) -> Dict[str, Any]:
return cast(Dict[str, Any], v) if isinstance(v, dict) else {}
def _unwrap_variable(v: Any, default: Any = None) -> Any:
"""
Falls v ein Mapping nach dem Muster {beschreibung, wert} ist, gib 'wert' zurück.
Andernfalls gib v (als Any) selbst oder default zurück.
"""
if isinstance(v, dict):
mv: Dict[str, Any] = cast(Dict[str, Any], v)
if "wert" in mv:
return mv.get("wert", default)
# Pylance: explizit als Any zurückgeben, um Unknown-Typen zu vermeiden
return cast(Any, v) if v is not None else default
_MISSING = object()
def _value_from_node(node: Dict[str, Any], key: str, default: Any) -> Any:
"""
Holt 'key' aus node. Unterstützt Dot-Pfade und das {beschreibung, wert}-Muster.
Gibt _MISSING zurück, wenn der Schlüssel nicht existiert (damit Aufrufer fallbacken kann).
"""
# Direkter Schlüssel ohne Dot
if "." not in key:
if key in node:
return _unwrap_variable(node.get(key), default)
return _MISSING
# Dot-Pfad
current: Any = node
for part in key.split("."):
if isinstance(current, dict) and part in current:
current = cast(Dict[str, Any], current)[part]
else:
return _MISSING
return _unwrap_variable(current, default)
def _load_yaml_safe(path: Path) -> Dict[str, Any]:
with path.open("r", encoding="utf-8") as f:
data = yaml.safe_load(f)
return data or {}
__all__ = [
"Settings",
"load_runtime_config",
]

372
work/entworfener_code_working_copy/app/src/logging_external.py Normal file
View File

@@ -0,0 +1,372 @@
from __future__ import annotations
"""
Externes Logging-Modul (MySQL / PostgreSQL) auf Basis von SQLAlchemy
====================================================================
Zweck
- Persistentes Logging in eine externe Datenbank (MySQL oder PostgreSQL).
- Optionale Fallback-Strategie auf internes SQLite-Logging bei Fehlern.
Features
- Connection-Pooling via SQLAlchemy Engine.
- Schema-Erzeugung (Tabelle logs) via SQLAlchemy Core.
- Batch-Schreiben von Logeinträgen.
- Health-Check (SELECT 1).
- logging.Handler zum direkten Schreiben aus Python-Logging.
- Optionaler Fallback (Schreiben in internes SQLite-Logging), wenn aktiviert.
Konfiguration (Beispiel; siehe 01_Modulerweiterungen/Planung/Architektur.md)
logging_external:
enabled: false
type: "postgresql" # mysql | postgresql
host: "localhost"
port: 5432
user: "logger"
password: null
database: "logs"
sslmode: "prefer"
pool_size: 5
connect_timeout: 10
write_buffer_size: 100
fallback_to_internal_on_error: true
"""
import json
import logging
import time
import os
import threading
from dataclasses import dataclass
from datetime import datetime, timezone
from typing import Any, Dict, Iterable, List, Optional, Tuple, Union
try:
from sqlalchemy import (
create_engine,
Table,
Column,
Integer,
String,
Text,
MetaData,
DateTime,
text as sa_text,
insert as sa_insert,
)
from sqlalchemy.engine import Engine, Connection
except Exception as e: # pragma: no cover
raise RuntimeError(
"SQLAlchemy ist erforderlich für logging_external. Bitte Abhängigkeiten installieren."
) from e
def _env_override(key: str, default: Optional[str] = None) -> Optional[str]:
"""Liest optionale Secrets aus Umgebungsvariablen, falls gesetzt."""
v = os.environ.get(key)
return v if v is not None else default
def _safe_quote(v: str) -> str:
# Nur einfache, naive Maskierung für URL-Bestandteile
return v.replace("@", "%40").replace(":", "%3A").replace("/", "%2F")
def _now_utc_iso() -> str:
return (
datetime.now(timezone.utc)
.astimezone(timezone.utc)
.replace(tzinfo=timezone.utc)
.isoformat(timespec="milliseconds")
.replace("+00:00", "Z")
)
@dataclass
class ExternalConfig:
db_type: str = "postgresql" # "mysql" | "postgresql"
host: str = "localhost"
port: int = 5432
user: Optional[str] = None
password: Optional[str] = None
database: str = "logs"
sslmode: Optional[str] = None # nur relevant für PostgreSQL
pool_size: int = 5
connect_timeout: int = 10
fallback_to_internal_on_error: bool = True
@classmethod
def from_dict(cls, cfg: Dict[str, Any]) -> "ExternalConfig":
# Secrets via ENV überschreiben, wenn vorhanden
user = cfg.get("user")
password = cfg.get("password")
user = _env_override("LOG_EXT_USER", user)
password = _env_override("LOG_EXT_PASSWORD", password)
return cls(
db_type=str(cfg.get("type", "postgresql")),
host=str(cfg.get("host", "localhost")),
port=int(cfg.get("port", 5432)),
user=(None if user is None else str(user)),
password=(None if password is None else str(password)),
database=str(cfg.get("database", "logs")),
sslmode=(cfg.get("sslmode") if cfg.get("sslmode") is not None else None),
pool_size=int(cfg.get("pool_size", 5)),
connect_timeout=int(cfg.get("connect_timeout", 10)),
fallback_to_internal_on_error=bool(cfg.get("fallback_to_internal_on_error", True)),
)
def to_url(self) -> str:
"""
Baut den SQLAlchemy-URL-String auf Basis der Konfiguration.
- PostgreSQL: postgresql+psycopg2://user:password@host:port/database?sslmode=...
- MySQL: mysql+pymysql://user:password@host:port/database
"""
user_part = ""
if self.user:
if self.password:
user_part = f"{_safe_quote(self.user)}:{_safe_quote(self.password)}@"
else:
user_part = f"{_safe_quote(self.user)}@"
if self.db_type.lower() in ("postgres", "postgresql", "pg"):
base = f"postgresql+psycopg2://{user_part}{self.host}:{self.port}/{self.database}"
if self.sslmode:
base = f"{base}?sslmode={self.sslmode}"
return base
if self.db_type.lower() in ("mysql", "mariadb"):
return f"mysql+pymysql://{user_part}{self.host}:{self.port}/{self.database}"
raise ValueError(f"Unsupported db_type: {self.db_type}")
class ExternalDBLogger:
"""
Verwaltet Engine und Schema für externes Logging.
"""
def __init__(self, engine: Engine) -> None:
self.engine = engine
self.meta = MetaData()
# Hinweis: ts als String/DateTime. Für maximale Kompatibilität nutzen wir String(30) ISO-Zeitstempel.
self.logs = Table(
"logs",
self.meta,
Column("id", Integer, primary_key=True, autoincrement=True),
Column("ts", String(30), nullable=False),
Column("level", String(16), nullable=True),
Column("logger", String(128), nullable=True),
Column("message", Text, nullable=True),
Column("meta", Text, nullable=True), # JSON-String
)
# ---------- API ----------
def ensure_schema(self) -> None:
self.meta.create_all(self.engine, checkfirst=True)
def health_check(self, timeout_s: int = 5) -> bool:
try:
with self.engine.connect() as conn:
conn.execute(sa_text("SELECT 1"))
return True
except Exception:
return False
def write_batch(self, entries: Iterable[Dict[str, Any]]) -> None:
rows: List[Dict[str, Any]] = []
for e in entries:
ts = e.get("ts") or _now_utc_iso()
level = e.get("level")
loggername = e.get("logger")
message = e.get("message")
meta = e.get("meta")
try:
meta_json = json.dumps(meta, ensure_ascii=False, separators=(",", ":")) if meta is not None else None
except Exception:
meta_json = json.dumps({"_repr": repr(meta)}, ensure_ascii=False)
rows.append(
{
"ts": str(ts),
"level": (None if level is None else str(level)),
"logger": (None if loggername is None else str(loggername)),
"message": (None if message is None else str(message)),
"meta": meta_json,
}
)
if not rows:
return
with self.engine.begin() as conn:
conn.execute(sa_insert(self.logs), rows)
class ExternalDBHandler(logging.Handler):
"""
Logging-Handler, der LogRecords in die externe DB schreibt.
Mit einfachem Retry/Backoff und optionalem Fallback auf internes SQLite.
"""
def __init__(self, writer: ExternalDBLogger, *, fallback_to_internal: bool = True) -> None:
super().__init__()
self._writer = writer
self._fallback_to_internal = fallback_to_internal
def emit(self, record: logging.LogRecord) -> None:
payload = {
"ts": _now_utc_iso(),
"level": record.levelname,
"logger": record.name,
"message": self.format(record) if self.formatter else record.getMessage(),
"meta": _extract_meta(record),
}
# Kleiner Retry mit Backoff: 3 Versuche 50ms/150ms
delay_ms = [0.05, 0.15, 0.5]
last_err: Optional[BaseException] = None
for d in delay_ms:
try:
self._writer.write_batch([payload])
return
except Exception as e:
last_err = e
time.sleep(d)
# Fallback
if self._fallback_to_internal:
try:
import importlib
mod = importlib.import_module("logging_internal")
mod.instance().write(payload) # nutzt bestehende interne Instanz
logging.getLogger(__name__).warning(
"External logging failed; wrote to internal SQLite fallback."
)
return
except Exception:
pass
# Letzter Ausweg: Fehler im Handler nicht nach außen eskalieren
if last_err:
self.handleError(record)
def _extract_meta(record: logging.LogRecord) -> Dict[str, Any]:
meta: Dict[str, Any] = {
"pathname": record.pathname,
"lineno": record.lineno,
"funcName": record.funcName,
"process": record.process,
"threadName": record.threadName,
}
standard = {
"name",
"msg",
"args",
"levelname",
"levelno",
"pathname",
"filename",
"module",
"exc_info",
"exc_text",
"stack_info",
"lineno",
"funcName",
"created",
"msecs",
"relativeCreated",
"thread",
"threadName",
"processName",
"process",
"message",
}
for k, v in record.__dict__.items():
if k not in standard:
try:
json.dumps(v)
meta[k] = v
except Exception:
meta[k] = repr(v)
if record.exc_info:
try:
meta["exc_info"] = logging.Formatter().formatException(record.exc_info)
except Exception:
pass
return meta
# ---------------------------------------------------------
# Modul-Singleton und Helper
# ---------------------------------------------------------
_EXTERNAL_INSTANCE: Optional[ExternalDBLogger] = None
_INSTANCE_LOCK = threading.Lock()
def init(
*,
connection_url: Optional[str] = None,
config: Optional[Dict[str, Any]] = None,
) -> ExternalDBLogger:
"""
Initialisiert den globalen ExternalDBLogger. Entweder via connection_url oder via config.
Mehrfache Aufrufe liefern dieselbe Instanz zurück.
"""
global _EXTERNAL_INSTANCE
with _INSTANCE_LOCK:
if _EXTERNAL_INSTANCE is not None:
return _EXTERNAL_INSTANCE
if connection_url is None:
if not isinstance(config, dict):
raise ValueError("Entweder connection_url angeben oder config (Dict) bereitstellen.")
cfg = ExternalConfig.from_dict(config)
connection_url = cfg.to_url()
pool_size = cfg.pool_size
connect_timeout = cfg.connect_timeout
else:
# Fallback-Parameter wenn URL direkt übergeben wird
pool_size = int((config or {}).get("pool_size", 5))
connect_timeout = int((config or {}).get("connect_timeout", 10))
cfg = ExternalConfig.from_dict(config or {})
engine = create_engine(
connection_url,
pool_pre_ping=True,
pool_size=pool_size,
connect_args={}, # sslmode ist bei PG bereits in der URL verarbeitet
echo=False,
)
logger = ExternalDBLogger(engine)
logger.ensure_schema()
_EXTERNAL_INSTANCE = logger
# Speichere Fallback-Flag am Logger-Objekt (attributiv), damit Handler darauf zugreifen kann, wenn benötigt
setattr(_EXTERNAL_INSTANCE, "_fallback_to_internal_on_error", cfg.fallback_to_internal_on_error)
return _EXTERNAL_INSTANCE
def instance() -> ExternalDBLogger:
if _EXTERNAL_INSTANCE is None:
raise RuntimeError("ExternalDBLogger ist nicht initialisiert. Bitte init(...) zuerst aufrufen.")
return _EXTERNAL_INSTANCE
def get_handler(level: Union[int, str] = logging.INFO) -> logging.Handler:
if _EXTERNAL_INSTANCE is None:
raise RuntimeError("ExternalDBLogger ist nicht initialisiert. Bitte init(...) zuerst aufrufen.")
h = ExternalDBHandler(
_EXTERNAL_INSTANCE,
fallback_to_internal=bool(getattr(_EXTERNAL_INSTANCE, "_fallback_to_internal_on_error", True)),
)
if isinstance(level, str):
level = getattr(logging, level.upper(), logging.INFO)
h.setLevel(int(level))
return h
__all__ = [
"ExternalConfig",
"ExternalDBLogger",
"init",
"instance",
"get_handler",
]

476
work/entworfener_code_working_copy/app/src/logging_internal.py Normal file
View File

@@ -0,0 +1,476 @@
from __future__ import annotations
"""
SQLite-basiertes internes Logging-Modul
=======================================
Zweck
- Persistentes, leichtgewichtiges Logging in eine interne SQLite-Datenbank.
- Geeignet für interne Daten wie Hash-Werte, Status-Events, Metadaten.
Fähigkeiten
- Schema-Management (Tabellen und Indizes falls nicht vorhanden).
- Optionales Säubern der Datenbank beim Start (clean_database).
- Aufbewahrung/Retention nach Tagen (retention_days).
- Begrenzung der Gesamtanzahl (max_entries).
- Bereitstellung eines logging.Handler, der LogRecords direkt in SQLite schreibt.
- Abfrage-API mit Filtern und Paging.
Konfiguration (Beispiel, siehe Planung/Architektur.md)
logging_internal:
enabled: true
db_path: "data/internal_logs.sqlite"
clean_database: false
retention_days: 30
max_entries: 100000
vacuum_on_start: true
batch_write: 100
"""
import json
import logging
import sqlite3
import threading
from dataclasses import dataclass
from datetime import datetime, timedelta, timezone
from pathlib import Path
from typing import Any, Dict, Iterable, List, Optional, Tuple, Union
# ---------------------------------------------------------
# Hilfsfunktionen
# ---------------------------------------------------------
def _utc_now_iso() -> str:
"""Aktuelle UTC-Zeit als ISO 8601 mit Millisekunden und 'Z'-Suffix."""
return datetime.now(timezone.utc).astimezone(timezone.utc).replace(tzinfo=timezone.utc).isoformat(timespec="milliseconds").replace("+00:00", "Z")
def _ensure_dir_for(path: Path) -> None:
"""Erzeugt das Zielverzeichnis für eine Datei, falls erforderlich."""
path.parent.mkdir(parents=True, exist_ok=True)
def _to_iso(ts: Union[str, datetime, None]) -> Optional[str]:
if ts is None:
return None
if isinstance(ts, str):
return ts
if isinstance(ts, datetime):
if ts.tzinfo is None:
ts = ts.replace(tzinfo=timezone.utc)
ts = ts.astimezone(timezone.utc)
return ts.isoformat(timespec="milliseconds").replace("+00:00", "Z")
return None
# ---------------------------------------------------------
# Kern: SQLiteLogger
# ---------------------------------------------------------
@dataclass
class RetentionPolicy:
retention_days: int = 30
max_entries: int = 100_000
class SQLiteLogger:
"""
Verwaltet die SQLite-Datenbank für interne Logs.
Thread-sicher durch Lock, eine Connection pro Prozess (check_same_thread=False).
"""
def __init__(
self,
db_path: Union[str, Path],
vacuum_on_start: bool = True,
clean_database: bool = False,
retention: Optional[RetentionPolicy] = None,
) -> None:
self.db_path = Path(db_path)
self.vacuum_on_start = bool(vacuum_on_start)
self.clean_database = bool(clean_database)
self.retention = retention or RetentionPolicy()
self._lock = threading.RLock()
self._conn: Optional[sqlite3.Connection] = None
self._initialize_db()
# ---------- Public API ----------
def write(self, entry: Dict[str, Any]) -> None:
"""
Schreibt einen einzelnen Log-Eintrag.
Erwartete Keys:
- ts: ISO 8601, sonst wird automatisch gesetzt
- level: TEXT
- logger: TEXT
- message: TEXT
- meta: dict | JSON-serialisierbar | None
"""
data = self._normalize_entry(entry)
with self._lock, self._connection() as con:
con.execute(
"""
INSERT INTO logs (ts, level, logger, message, meta)
VALUES (?, ?, ?, ?, ?)
""",
(data["ts"], data.get("level"), data.get("logger"), data.get("message"), data.get("meta")),
)
con.commit()
def write_many(self, entries: Iterable[Dict[str, Any]]) -> None:
"""Batch-Insert für mehrere Einträge."""
rows: List[Tuple[str, Optional[str], Optional[str], Optional[str], Optional[str]]] = []
for e in entries:
d = self._normalize_entry(e)
rows.append((d["ts"], d.get("level"), d.get("logger"), d.get("message"), d.get("meta")))
if not rows:
return
with self._lock, self._connection() as con:
con.executemany(
"INSERT INTO logs (ts, level, logger, message, meta) VALUES (?, ?, ?, ?, ?)",
rows,
)
con.commit()
def query(
self,
*,
logger: Optional[str] = None,
level: Optional[str] = None,
from_ts: Optional[Union[str, datetime]] = None,
to_ts: Optional[Union[str, datetime]] = None,
text: Optional[str] = None,
limit: int = 100,
offset: int = 0,
order_desc: bool = True,
) -> List[Dict[str, Any]]:
"""
Liefert Log-Einträge gefiltert und paginiert zurück.
- logger: exakter Match
- level: exakter Match
- from_ts / to_ts: Grenzen (inklusive), ISO 8601 oder datetime
- text: Fulltext-ähnliche Suche per LIKE auf message
"""
clauses: List[str] = []
params: List[Any] = []
if logger:
clauses.append("logger = ?")
params.append(logger)
if level:
clauses.append("level = ?")
params.append(level)
if from_ts is not None:
v = _to_iso(from_ts) or _utc_now_iso()
clauses.append("ts >= ?")
params.append(v)
if to_ts is not None:
v = _to_iso(to_ts) or _utc_now_iso()
clauses.append("ts <= ?")
params.append(v)
if text:
clauses.append("message LIKE ?")
params.append(f"%{text}%")
where = f"WHERE {' AND '.join(clauses)}" if clauses else ""
order = "DESC" if order_desc else "ASC"
sql = f"""
SELECT id, ts, level, logger, message, meta
FROM logs
{where}
ORDER BY ts {order}, id {order}
LIMIT ? OFFSET ?
"""
params.extend([int(max(limit, 0)), int(max(offset, 0))])
with self._lock, self._connection() as con:
cur = con.execute(sql, params)
rows = cur.fetchall() or []
out: List[Dict[str, Any]] = []
for r in rows:
meta_val: Optional[str] = r[5]
meta_obj: Optional[Any] = None
if meta_val:
try:
meta_obj = json.loads(meta_val)
except Exception:
meta_obj = meta_val # fallback: Rohwert
out.append(
{
"id": r[0],
"ts": r[1],
"level": r[2],
"logger": r[3],
"message": r[4],
"meta": meta_obj,
}
)
return out
def cleanup(self) -> None:
"""
Führt Aufräumregeln aus:
- Lösche Einträge älter als retention_days (wenn > 0).
- Reduziere auf max_entries (wenn > 0), lösche älteste zuerst.
"""
with self._lock, self._connection() as con:
# 1) Zeitbasierte Aufbewahrung
if self.retention.retention_days and self.retention.retention_days > 0:
cutoff = datetime.now(timezone.utc) - timedelta(days=int(self.retention.retention_days))
cutoff_iso = _to_iso(cutoff) or _utc_now_iso()
con.execute("DELETE FROM logs WHERE ts < ?", (cutoff_iso,))
con.commit()
# 2) Anzahl begrenzen
if self.retention.max_entries and self.retention.max_entries > 0:
cur = con.execute("SELECT COUNT(*) FROM logs")
total = int(cur.fetchone()[0])
overflow = total - int(self.retention.max_entries)
if overflow > 0:
# Lösche die ältesten N Einträge
con.execute(
"""
DELETE FROM logs
WHERE id IN (
SELECT id FROM logs
ORDER BY ts ASC, id ASC
LIMIT ?
)
""",
(overflow,),
)
con.commit()
def get_handler(self, level: Union[int, str] = logging.INFO) -> logging.Handler:
"""
Erzeugt einen logging.Handler, der LogRecords in die SQLite-DB schreibt.
Der zurückgegebene Handler ist threadsicher und kann dem Root-Logger zugewiesen werden.
"""
h = SQLiteLogHandler(self)
if isinstance(level, str):
level = getattr(logging, level.upper(), logging.INFO)
h.setLevel(int(level))
return h
# ---------- Internals ----------
def _initialize_db(self) -> None:
if self.clean_database and self.db_path.exists():
try:
self.db_path.unlink()
except FileNotFoundError:
pass
_ensure_dir_for(self.db_path)
with self._lock, self._connection() as con:
# Pragmas für Stabilität/Performance
con.execute("PRAGMA journal_mode=WAL;")
con.execute("PRAGMA synchronous=NORMAL;")
con.execute("PRAGMA foreign_keys=ON;")
con.execute("PRAGMA temp_store=MEMORY;")
# Schema
con.execute(
"""
CREATE TABLE IF NOT EXISTS logs (
id INTEGER PRIMARY KEY AUTOINCREMENT,
ts TEXT NOT NULL,
level TEXT,
logger TEXT,
message TEXT,
meta TEXT
)
"""
)
con.execute("CREATE INDEX IF NOT EXISTS idx_logs_ts ON logs (ts);")
con.execute("CREATE INDEX IF NOT EXISTS idx_logs_logger ON logs (logger);")
con.commit()
if self.vacuum_on_start:
# VACUUM darf nicht innerhalb einer aktiven Transaktion laufen
prev_iso = con.isolation_level
try:
con.isolation_level = None
con.execute("VACUUM;")
finally:
con.isolation_level = prev_iso
# Nach Schema-Erstellung sofort Cleanup-Regeln anwenden
self.cleanup()
def _normalize_entry(self, entry: Dict[str, Any]) -> Dict[str, Any]:
ts = _to_iso(entry.get("ts")) or _utc_now_iso()
meta_val = entry.get("meta")
if meta_val is None:
meta_json = None
else:
try:
meta_json = json.dumps(meta_val, ensure_ascii=False, separators=(",", ":"))
except Exception:
meta_json = json.dumps({"_repr": repr(meta_val)}, ensure_ascii=False)
return {
"ts": ts,
"level": str(entry.get("level")) if entry.get("level") is not None else None,
"logger": str(entry.get("logger")) if entry.get("logger") is not None else None,
"message": str(entry.get("message")) if entry.get("message") is not None else None,
"meta": meta_json,
}
def _connection(self) -> sqlite3.Connection:
if self._conn is None:
# check_same_thread=False: erlaubt Nutzung über mehrere Threads, wir schützen per Lock
self._conn = sqlite3.connect(str(self.db_path), check_same_thread=False)
return self._conn
# ---------------------------------------------------------
# logging.Handler-Integration
# ---------------------------------------------------------
class SQLiteLogHandler(logging.Handler):
"""
Logging-Handler, der LogRecords in die SQLite-Datenbank schreibt.
"""
def __init__(self, writer: SQLiteLogger) -> None:
super().__init__()
self._writer = writer
def emit(self, record: logging.LogRecord) -> None:
try:
# Standardfelder
payload: Dict[str, Any] = {
"ts": _utc_now_iso(),
"level": record.levelname,
"logger": record.name,
"message": self.format(record) if self.formatter else record.getMessage(),
}
# Meta: ausgewählte Felder + Extras
meta: Dict[str, Any] = {
"pathname": record.pathname,
"lineno": record.lineno,
"funcName": record.funcName,
"process": record.process,
"threadName": record.threadName,
}
# Extras erkennen: all jene keys, die nicht Standard sind
standard = {
"name",
"msg",
"args",
"levelname",
"levelno",
"pathname",
"filename",
"module",
"exc_info",
"exc_text",
"stack_info",
"lineno",
"funcName",
"created",
"msecs",
"relativeCreated",
"thread",
"threadName",
"processName",
"process",
"message",
}
for k, v in record.__dict__.items():
if k not in standard:
# Versuch: JSON-serialisierbar machen
try:
json.dumps(v)
meta[k] = v
except Exception:
meta[k] = repr(v)
if record.exc_info:
# Ausnahmeinformationen hinzufügen (als Text)
meta["exc_info"] = logging.Formatter().formatException(record.exc_info)
payload["meta"] = meta
self._writer.write(payload)
except Exception:
# Handler darf niemals den Prozess crashen
self.handleError(record)
# ---------------------------------------------------------
# Modulweite Singletons/Helper-Funktionen
# ---------------------------------------------------------
_logger_instance: Optional[SQLiteLogger] = None
_INSTANCE_LOCK = threading.Lock()
def init(
*,
db_path: Union[str, Path],
vacuum_on_start: bool = True,
clean_database: bool = False,
retention_days: int = 30,
max_entries: int = 100_000,
) -> SQLiteLogger:
"""
Initialisiert den globalen SQLiteLogger (Singleton pro Prozess).
Wird erneut aufgerufen, wird die bestehende Instanz zurückgegeben.
"""
global _logger_instance
with _INSTANCE_LOCK:
if _logger_instance is not None:
return _logger_instance
p = Path(db_path)
# relative Pfade sind relativ zum Arbeitsverzeichnis des Prozesses;
# in Integrationen sollte nach Bedarf auf app/-Pfad aufgelöst werden.
retention = RetentionPolicy(retention_days=int(retention_days), max_entries=int(max_entries))
_logger_instance = SQLiteLogger(
db_path=p,
vacuum_on_start=vacuum_on_start,
clean_database=clean_database,
retention=retention,
)
return _logger_instance
def instance() -> SQLiteLogger:
"""Gibt die initialisierte Instanz zurück oder wirft einen Fehler."""
if _logger_instance is None:
raise RuntimeError("SQLiteLogger ist nicht initialisiert. Bitte init(...) zuerst aufrufen.")
return _logger_instance
def get_engineered_handler(level: Union[int, str] = logging.INFO) -> logging.Handler:
"""
Liefert einen konfigurierten Handler auf Basis der globalen Instanz.
Beispiel:
from logging import getLogger
from logging_internal import init, get_engineered_handler
init(db_path='data/internal_logs.sqlite', retention_days=30, max_entries=100_000)
root = logging.getLogger()
root.addHandler(get_engineered_handler(logging.INFO))
"""
return instance().get_handler(level)
__all__ = [
"SQLiteLogger",
"RetentionPolicy",
"init",
"instance",
"get_engineered_handler",
]

263
work/entworfener_code_working_copy/app/src/logging_setup.py Normal file
View File

@@ -0,0 +1,263 @@
from __future__ import annotations
import logging
import logging.handlers
import os
from datetime import datetime, timedelta
from pathlib import Path
from typing import Iterable, List, Tuple, Dict, Optional, Any
from .config_loader import Settings
DEFAULT_FORMAT = "%(asctime)s | %(levelname)s | %(name)s | %(filename)s:%(lineno)d | %(message)s"
DEFAULT_DATEFMT = "%Y-%m-%d %H:%M:%S"
def init_logging(settings: Settings, app_config: Optional[Dict[str, Any]] = None) -> None:
"""
Initialisiert das Logging gemäß den Laufzeit-Settings.
Erfüllt die Anforderungen:
- Unter logs/ je Tag eine neue Datei (TimedRotatingFileHandler -> midnight).
- Konfigurierbare Aufbewahrungsdauer (retention_days).
- Konfigurierbare maximale Log-Größe (interpretiert als maximale Gesamtgröße des Log-Verzeichnisses);
älteste Dateien werden bei Überschreitung entfernt.
Hinweise:
- Diese Funktion setzt das Root-Logging neu (entfernt bestehende Handler), um konsistentes Verhalten
gegenüber evtl. vorhandener dictConfig zu gewährleisten.
- Zusätzlich zur Datei wird immer ein Console-Handler eingerichtet.
"""
params = settings.get_logging_params()
level_name: str = params.get("level", "INFO")
level = getattr(logging, level_name.upper(), logging.INFO)
# Verzeichnis vorbereiten
raw_log_dir: str = params.get("log_dir", "logs")
log_dir: Path = settings.resolve_path(raw_log_dir)
log_dir.mkdir(parents=True, exist_ok=True)
# Root-Logger zurücksetzen und konfigurieren
root = logging.getLogger()
_reset_logger_handlers(root)
root.setLevel(level)
# Formatter
formatter = logging.Formatter(fmt=DEFAULT_FORMAT, datefmt=DEFAULT_DATEFMT)
# Console-Handler
console_handler = logging.StreamHandler()
console_handler.setLevel(level)
console_handler.setFormatter(formatter)
root.addHandler(console_handler)
# Datei-Handler: tägliche Rotation um Mitternacht
logfile = log_dir / "app.log"
file_handler = logging.handlers.TimedRotatingFileHandler(
filename=str(logfile),
when="midnight",
interval=1,
backupCount=int(params.get("retention_days", 7)),
encoding="utf-8",
utc=False,
)
file_handler.setLevel(level)
file_handler.setFormatter(formatter)
root.addHandler(file_handler)
# Cleanup-Regeln anwenden (maximale Gesamtgröße und zusätzliche tagbasierte Aufbewahrung)
try:
_cleanup_logs(
log_dir=log_dir,
max_total_mb=int(params.get("max_log_size_mb", params.get("max_log_size", 10))),
retention_days=int(params.get("retention_days", 7)),
file_prefix="app",
extensions=(".log",),
)
except Exception: # Schutz vor Start-Abbruch durch Aufräumfehler
logging.getLogger(__name__).exception("Fehler beim Log-Cleanup ignoriert.")
# Optionale Integration: internes SQLite-Logging (01_Modulerweiterungen)
try:
internal_cfg = ((app_config or {}).get("logging_internal") or {})
except Exception:
internal_cfg = {}
if isinstance(internal_cfg, dict) and internal_cfg.get("enabled"):
try:
import importlib
mod = importlib.import_module("logging_internal")
except Exception:
logging.getLogger(__name__).warning(
"logging_internal Modul nicht importierbar; interner DB-Handler wird nicht aktiviert."
)
else:
try:
db_path = internal_cfg.get("db_path", "data/internal_logs.sqlite")
clean_db = bool(internal_cfg.get("clean_database", False))
retention_days_cfg = int(internal_cfg.get("retention_days", 30))
max_entries_cfg = int(internal_cfg.get("max_entries", 100000))
vacuum_on_start = bool(internal_cfg.get("vacuum_on_start", True))
resolved_db_path = settings.resolve_path(str(db_path))
mod.init(
db_path=str(resolved_db_path),
vacuum_on_start=vacuum_on_start,
clean_database=clean_db,
retention_days=retention_days_cfg,
max_entries=max_entries_cfg,
)
handler = mod.get_engineered_handler(level)
root.addHandler(handler)
logging.getLogger(__name__).info("Interner SQLite-Log-Handler aktiv: %s", str(resolved_db_path))
except Exception:
logging.getLogger(__name__).exception(
"Fehler bei Initialisierung des internen SQLite-Loggings; Handler wird nicht aktiviert."
)
# Optionale Integration: externes DB-Logging (01_Modulerweiterungen)
try:
external_cfg = ((app_config or {}).get("logging_external") or {})
except Exception:
external_cfg = {}
if isinstance(external_cfg, dict) and external_cfg.get("enabled"):
try:
import importlib
ext_mod = importlib.import_module("logging_external")
except Exception:
logging.getLogger(__name__).warning(
"logging_external Modul nicht importierbar; externer DB-Handler wird nicht aktiviert."
)
else:
try:
# ext_mod.init akzeptiert entweder connection_url ODER config-Dict
# Wir übergeben das gesamte Config-Dict und lassen das Modul die URL bauen.
ext_mod.init(config=external_cfg)
ext_handler = ext_mod.get_handler(level)
root.addHandler(ext_handler)
logging.getLogger(__name__).info("Externer DB-Log-Handler aktiviert.")
except Exception:
logging.getLogger(__name__).exception(
"Fehler bei Initialisierung des externen DB-Loggings; Handler wird nicht aktiviert."
)
def _reset_logger_handlers(logger: logging.Logger) -> None:
"""Entfernt alle existierenden Handler vom Logger."""
for h in list(logger.handlers):
try:
logger.removeHandler(h)
try:
h.close()
except Exception:
pass
except Exception:
pass
def _cleanup_logs(
log_dir: Path,
max_total_mb: int,
retention_days: int,
file_prefix: str = "app",
extensions: Tuple[str, ...] = (".log",),
) -> None:
"""
Bereinigt das Log-Verzeichnis nach zwei Regeln:
1) Zeitbasierte Aufbewahrung (retention_days):
- Löscht Dateien, deren Änderungszeitpunkt älter als retention_days ist.
2) Größenlimit (max_total_mb):
- Sicherstellt, dass die Gesamtgröße aller Log-Dateien (passend zu extensions/prefix)
das Limit nicht überschreitet. Bei Überschreitung werden die ältesten Dateien
entfernt, bis das Limit eingehalten wird.
Diese Regeln ergänzen den TimedRotatingFileHandler (backupCount), der ohnehin
nur eine feste Anzahl von Backups behält. Mit diesem zusätzlichen Cleanup
wird explizit die Aufbewahrungsdauer (in Tagen) und die Gesamtgröße kontrolliert.
"""
if max_total_mb <= 0 and retention_days <= 0:
return
candidates: List[Path] = _collect_log_files(log_dir, file_prefix, extensions)
# 1) Zeitbasierte Aufbewahrung (zusätzlich zu backupCount)
if retention_days > 0:
cutoff = datetime.now() - timedelta(days=retention_days)
for f in candidates:
try:
mtime = datetime.fromtimestamp(f.stat().st_mtime)
if mtime < cutoff:
_safe_unlink(f)
except FileNotFoundError:
# Wurde zwischenzeitlich rotiert/gelöscht
continue
# Refresh nach Löschungen
candidates = _collect_log_files(log_dir, file_prefix, extensions)
# 2) Größenlimit
if max_total_mb > 0:
limit_bytes = max_total_mb * 1024 * 1024
# Sortiere nach mtime aufsteigend (älteste zuerst)
sorted_by_age = sorted(
candidates,
key=lambda p: (p.stat().st_mtime if p.exists() else float("inf")),
)
total_size = _total_size(sorted_by_age)
for f in sorted_by_age:
if total_size <= limit_bytes:
break
try:
size_before = f.stat().st_size if f.exists() else 0
_safe_unlink(f)
total_size -= size_before
except FileNotFoundError:
# Bereits gelöscht
continue
def _collect_log_files(log_dir: Path, prefix: str, exts: Tuple[str, ...]) -> List[Path]:
if not log_dir.exists():
return []
result: List[Path] = []
try:
for item in log_dir.iterdir():
if not item.is_file():
continue
if not item.suffix.lower() in exts:
continue
# Akzeptiere Standard-Dateinamen und Rotationssuffixe (z. B. app.log, app.log.2025-11-13, ...)
name = item.name
if name.startswith(prefix):
result.append(item)
except Exception:
# Defensive: bei Problemen einfach keine Kandidaten liefern
return []
return result
def _total_size(files: Iterable[Path]) -> int:
s = 0
for f in files:
try:
s += f.stat().st_size
except FileNotFoundError:
continue
return s
def _safe_unlink(p: Path) -> None:
try:
p.unlink(missing_ok=True) # py3.8+: attribute missing_ok in 3.8? Actually 3.8+: No, use exists() check
except TypeError:
# Fallback für Python-Versionen ohne missing_ok
if p.exists():
os.remove(p)
__all__ = ["init_logging"]

170
work/entworfener_code_working_copy/app/start.py Normal file
View File

@@ -0,0 +1,170 @@
from typing import Any, Dict, Optional, List
import argparse
import logging
import logging.config
import os
from pathlib import Path
import sys
import uvicorn
import yaml
from fastapi import FastAPI
# Basisverzeichnis: .../Entworfener_Code/app
BASE_DIR: Path = Path(__file__).resolve().parent
CODE_DIR: Path = BASE_DIR / "code" # .../app/code
CONFIG_DIR: Path = BASE_DIR / "config" # .../app/config
SRC_DIR: Path = BASE_DIR / "src" # .../app/src
# Sicherstellen, dass .../app/code und .../app/src importierbar sind
if str(CODE_DIR) not in sys.path:
sys.path.insert(0, str(CODE_DIR))
if str(SRC_DIR) not in sys.path:
sys.path.insert(0, str(SRC_DIR))
# Jetzt Import aus neuer Struktur (app/code/app/main.py)
from app.main import create_app # noqa: E402
from src.config_loader import load_runtime_config # noqa: E402
from src.logging_setup import init_logging # noqa: E402
APP_CONFIG_ENV = "APP_CONFIG_PATH"
def load_yaml(path: Path) -> Dict[str, Any]:
"""
Lädt eine YAML-Datei und gibt den Inhalt als Dictionary zurück.
"""
with path.open("r", encoding="utf-8") as f:
return yaml.safe_load(f) or {}
def setup_logging(cfg: Dict[str, Any]) -> None:
"""
Initialisiert das Logging anhand der in app/config/logging.yaml definierten Konfiguration.
Stellt sicher, dass das Log-Verzeichnis existiert.
"""
logging_cfg = cfg.get("logging", {}) or {}
paths_cfg = cfg.get("paths", {}) or {}
log_dir = Path(paths_cfg.get("log_dir", "logs"))
if not log_dir.is_absolute():
log_dir = BASE_DIR / log_dir
log_dir.mkdir(parents=True, exist_ok=True)
logging_config_file = logging_cfg.get("config_file", "config/logging.yaml")
logging_path = Path(logging_config_file)
if not logging_path.is_absolute():
logging_path = BASE_DIR / logging_path
if logging_path.exists():
with logging_path.open("r", encoding="utf-8") as f:
config_dict = yaml.safe_load(f) or {}
logging.config.dictConfig(config_dict)
else:
logging.basicConfig(
level=getattr(logging, (logging_cfg.get("level") or "INFO").upper(), logging.INFO),
format="%(asctime)s | %(levelname)s | %(name)s | %(filename)s:%(lineno)d | %(message)s",
)
logging.getLogger(__name__).debug("Logging initialisiert. Log-Verzeichnis: %s", str(log_dir))
def load_config_from_env() -> Dict[str, Any]:
"""
Lädt die Anwendungskonfiguration aus der durch APP_CONFIG_PATH angegebenen Datei.
Fällt zurück auf app/config/config.yaml.
"""
env_path = os.environ.get(APP_CONFIG_ENV)
if env_path:
cfg_path = Path(env_path)
else:
cfg_path = CONFIG_DIR / "config.yaml"
if not cfg_path.exists():
print(f"Konfigurationsdatei nicht gefunden: {cfg_path}", file=sys.stderr)
return {}
return load_yaml(cfg_path)
def app_factory() -> FastAPI:
"""
Uvicorn-Factory für Reload-Betrieb (importierbare App-Fabrik).
Liest Konfiguration, initialisiert Logging und erzeugt die FastAPI-App.
"""
cfg = load_config_from_env()
# Laufzeit-Settings laden und Logging initialisieren (überschreibt ggf. YAML-Logging)
settings = load_runtime_config(base_dir=BASE_DIR)
init_logging(settings, cfg)
app = create_app(cfg)
return app
def parse_args(argv: Optional[List[str]] = None) -> argparse.Namespace:
"""
CLI-Argumente parsen.
"""
parser = argparse.ArgumentParser(description="Startet den FastAPI-Server (app/code-Struktur).")
parser.add_argument(
"--config",
type=str,
default=str(CONFIG_DIR / "config.yaml"),
help="Pfad zur Konfigurationsdatei (YAML).",
)
return parser.parse_args(argv)
def main(argv: Optional[List[str]] = None) -> None:
"""
Einstiegspunkt: lädt Konfiguration, initialisiert Logging, startet Uvicorn.
"""
args = parse_args(argv)
# Resolve config path
config_path = Path(args.config)
if not config_path.is_absolute():
config_path = (BASE_DIR / args.config).resolve()
if not config_path.exists():
print(f"Konfigurationsdatei nicht gefunden: {config_path}", file=sys.stderr)
sys.exit(2)
# Set env var für Factory
os.environ[APP_CONFIG_ENV] = str(config_path)
cfg = load_yaml(config_path)
# Laufzeit-Settings laden und Logging initialisieren (überschreibt ggf. YAML-Logging)
settings = load_runtime_config(base_dir=BASE_DIR)
init_logging(settings, cfg)
app_cfg = cfg.get("app", {}) or {}
host = str(app_cfg.get("host", "0.0.0.0"))
port = int(app_cfg.get("port", 8000))
reload_enabled = bool(app_cfg.get("reload", False))
logger = logging.getLogger("start")
if reload_enabled:
# Für Reload muss eine importierbare App-Factory übergeben werden
logger.info("Starte Uvicorn mit Reload (Factory-Modus, app/code).")
module_name = Path(__file__).stem # "start"
uvicorn.run(
f"{module_name}:app_factory",
factory=True,
host=host,
port=port,
reload=True,
)
else:
logger.info("Starte Uvicorn ohne Reload (app/code).")
app = create_app(cfg)
uvicorn.run(
app,
host=host,
port=port,
reload=False,
)
if __name__ == "__main__":
main()

27368
work/entworfener_code_working_copy/get-pip.py Normal file
View File

File diff suppressed because it is too large Load Diff

15
work/entworfener_code_working_copy/requirements.txt Normal file
View File

@@ -0,0 +1,15 @@
# Core web framework and server
fastapi>=0.115.0
uvicorn[standard]>=0.30.0
# Configuration
PyYAML>=6.0.2
# Testing
pytest>=8.3.1
httpx>=0.27.2
# Optional developer tooling (can be moved to requirements-dev.txt)
mypy>=1.11.2
ruff>=0.6.4
black>=24.10.0

17
work/entworfener_code_working_copy/server.log Normal file
View File

@@ -0,0 +1,17 @@
2025-11-14 00:16:16 | INFO | src.logging_setup | logging_setup.py:114 | Interner SQLite-Log-Handler aktiv: /home/hei/Downloads/porjekt/Unbenannter Ordner 2/work/entworfener_code_working_copy/app/data/internal_logs.sqlite
2025-11-14 00:16:16 | INFO | start | start.py:149 | Starte Uvicorn mit Reload (Factory-Modus, app/code).
INFO: Will watch for changes in these directories: ['/home/hei/Downloads/porjekt/Unbenannter Ordner 2/work/entworfener_code_working_copy']
INFO: Uvicorn running on http://0.0.0.0:8000 (Press CTRL+C to quit)
INFO: Started reloader process [78315] using WatchFiles
2025-11-14 00:16:19 | INFO | src.logging_setup | logging_setup.py:114 | Interner SQLite-Log-Handler aktiv: /home/hei/Downloads/porjekt/Unbenannter Ordner 2/work/entworfener_code_working_copy/app/data/internal_logs.sqlite
2025-11-14 00:16:19 | INFO | watchfiles.main | main.py:308 | 6 changes detected
INFO: Started server process [78350]
INFO: Waiting for application startup.
INFO: Application startup complete.
INFO: 127.0.0.1:53060 - "GET /health HTTP/1.1" 200 OK
INFO: 127.0.0.1:53066 - "POST /api/agent/v1/tasks HTTP/1.1" 200 OK
INFO: Shutting down
INFO: Waiting for application shutdown.
INFO: Application shutdown complete.
INFO: Finished server process [78350]
INFO: Stopping reloader process [78315]

1
work/entworfener_code_working_copy/server.pid Normal file
View File

@@ -0,0 +1 @@
PID=78315