Changelog

Added

• Virtual Datasets — Chatbot-driven dataset creation with auto-introspection of source schemas. Federated joins across DuckDB, PostgreSQL, and MySQL. Full lifecycle: SAVE / CREATE / DELETE via chat. convert_type support in create_virtual_table().
• isShared Sharing System — isShared flag on all shareable resources: datasets, DB connections, AI connections, recipes, preparation rules, notebooks. datasetGroup stores owning group; correct query scoping. Frontend toggle UI with group badge and attribution. Admin group dropdown on User details and DB connection edit pages. Removed group-00 sharing restriction.
• Recipe Execution Pipeline — Recipe execution wired to Python backend (replaced Prisma-only stub). LLM-controlled stage progression through all 10 stages. Data preparation stage executor with _apply_operation_in_memory. Cancelled execution kill support. Virtual dataset support. Preparation rules preview endpoint. Column schema fetch endpoint.
• Database Design Tool — Schema analysis for PostgreSQL & MySQL. RAG integration: full-text search via pg_trgm + AI-powered answer generation with citations. Semi-structured data analysis: code system detection (SIC, HS), keyword extraction, category classification. AI-powered normalisation suggestions. PDF export with translated labels. Email PDF functionality. Graceful 503 on ECONNREFUSED. Zero-tables warning with GRANT instructions.
• Statistics & Visualisation — Statistics router: descriptive stats and correlation matrix (pearson/spearman/kendall) with shared-memory caching. Visualisation stage in recipe execution: bar, pie, line, scatter via Plotly. Download chart as PNG.
• Jupyter Notebook Import — Multi-engine storage: DuckDB, PostgreSQL, MySQL. Notebook share toggle with PATCH proxy. Persistent error/success messages.
• Platform Configuration — platformMaxUploadSizeMB global upload cap with frontend file-size warnings and oversized-file blocking. set_platform_upload_max.py install script. 'Clear Default' buttons on DB and AI connection settings.
• Example User (Read-Only) Mode — Read-only access across admin, settings, visualisation, recipes. isExampleUser guards on all mutation endpoints. Auto-logout on auth pages. Banner hidden on auth pages.
• Mobile Responsive — Phases 1–3 — Hamburger drawer navigation, responsive sidebars, touch-friendly targets, keyboard padding.
• Internationalization — Japanese (ja) added: 965 frontend + 195 backend keys (now 9 languages: de, en, es, fr, ja, nl, ru, th, zh). Translation strategy upgrade: TranslationKey objects, useTranslate() hook. Memoized t() with useCallback. Chat language context: AI responds in user's language.
• UI/UX Polish — sentuny.com link on login page. 'Existing' sidebar on define dataset page. 'View Schema' button with SchemaViewDialog modal. Dynamic filter operators based on column type. Red "*" on all mandatory fields. About page 'Get Started' opens in new tab.
• Backend Infrastructure — DuckDB Quack protocol: full dataset storage (replaced 10-row snapshot). quack-proxy as git submodule with structured logger. Shared memory manager with UltraDict + TTL expiration. Backend logging with timestamps. Cascade user delete + orphan cleanup. Powerlifting sample data (386K lifters, 8.5K meets). MySQL extract_mysql_schema() and connection test.
• Workflow & AGENTS.md Rules — Bug Fix Workflow, HARD STOP gates, report-don't-extend, no silent simplification, proposal/implementation separation, pre-commit verification (pnpm build required), production commands ('status', 'deploy simple frontend fix', 'deploy simple backend fix'), stored commands ('list commands', 'new session', 'generate context transfer').

Changed

• Navigation — "Datasets" → "My Data", "Upload Dataset" → "Upload Data", "Define Dataset" → "Connect Database", "Prepare" → "Transform", "Chat" → expandable "AI Chat" section (General, Database design, Dataset cleaning, Analysis rules), "About" moved to last position, /tools page deleted.
• Architecture — Backend modular routers: recipes (13 files), datasets (crud/upload/define/schema/describe/preparation), chat (connection/crud/messages/title/language/manager/providers). Recipe architecture: datasetId at recipe level. Dataset storage: sourceType, storageType, sourceConnectionId, DuckDB fields. API proxies for chat + dataset endpoints. model column → defaultModel in UserAIConnection.
• Translation system — extract-translations.py uses parentheses matching. 657 valid keys. Detailed import reporting. 27 extra keys translated across 8 languages.
• UI refinements — Edit button prominent (outline + pencil icon). Save/Cancel in edit mode. Dataset list shows type and origin. Shared resources with "*" prefix and attribution.

Fixed

• Sharing — isShared ambiguous column in recipe JOINs. datasetGroup set correctly on creation. Shared connections viewable by all. Share toggle URL + state refresh. Models route group check for admins. Admin groups dropdown updates on selection.
• Recipe Execution — Shared dataset support. Correct owner name. All stages use same dataset ID. Defined dataset execution. pd.to_datetime warning handled.
• Default Connections — Chat respects default AI. datasets/define uses useDB. datasets/upload, ai/dataclean, ai/recipe respect defaults. Shared connections settable as default. String/int type mismatch. connectionId fallback lookup.
• Datasets — 401 on /api/datasets (proxies). Schema from DuckDB + external DBs. Full DuckDB storage (was 10 rows). Description generation. Missing table error handling.
• UI — Persistent messages at top of recipe page. Cancel resets form. Shared toggle disabled outside edit mode. Admin groups field name. Language change immediate. Login toast language.
• Backend — NaN/JSON sanitization (_sanitize_for_json). MySQL schema scoping. DuckDB Quack: ATTACH, DDL detection, table_exists. connection_id=None for notebooks. statsmodels 0.14.6. psycopg2-binary in requirements. PDF bytearray handling. DB design router registration. Graceful 503. Password decryption. pg_description joins. MS SQL Server removed from dropdown. Powerlifting '+' stripping.
• Build — Multiple pnpm build fixes. TypeScript errors. Missing imports. tw-animate-css. next-env.d.ts in .gitignore. ESM cron compatibility.
• Example User — isExampleUser guards on PATCH/password/profile. Auto-logout on auth pages. Banner hidden on auth pages. Translation memoization prevents render loops. 'More' button enabled for read-only detail.
• Chat — Shared connection support. connection_id type mismatch. Frontend connection_id=0 bug. History persistence with useEffect. Execution auth header. Missing typing imports.

Security

• Admin users excluded from automated cleanup.
• Cascade user delete across owned tables prevents orphans.
• isExampleUser guards on all mutation endpoints.
• API proxies preserve Authorization header.
• JWT token authentication on all backend API routes.
• Upload limits enforced server-side (count + size).

Known Issues

• Recipe wizard back button on Step 1 (low priority).
• Title generation endpoint (/chat/{id}/generate-title) returns 400.
• Dataset detail page may show "Rows: 0" for some datasets.
• Login toast shows in page language, not user's saved language (cosmetic).
• UltraDict on Windows not installed (in-memory fallback works).

Migration Notes

# Frontend
cd frontend && pnpm install && npx prisma migrate deploy && npx prisma generate && pnpm build

# Backend
cd backend && source venv/bin/activate && pip install -r requirements.txt && deactivate

# quack-proxy submodule
git submodule update --init

Schema changes: Recipe + RecipeExecution + DataPreparationRule + RecipePreparationRule tables. isShared flag on UserDataset, UserDatabaseConnection, UserAIConnection, Recipe, DataPreparationRule, Notebook. Group fields. DuckDB fields on UserDataset. model → defaultModel. Foreign keys + indexes.

9 languages imported with full key sets. Japanese newly added.