Feat/ai update - #98
Closed
Luongduytoan2006 wants to merge 5 commits into
Closed
Conversation
- llm.py: khi USE_NINE_ROUTER=true mà 9router lỗi hết retry, rớt thẳng
về local qwen3 (OLLAMA_FALLBACK_MODEL) thay vì raise 500. Áp cho cả
complete() async lẫn complete_sync().
- vbpl.py: VBPL editor tách số Điều ra nhiều <span> ("Điều 1 3") làm vỡ
số hiệu; gộp lại chữ số ngay sau "Điều" trong _strip_html.
Co-Authored-By: Claude <noreply@anthropic.com>
…orkflows, document parsing, and database publishing services
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
There was a problem hiding this comment.
🟡 Not ready to approve
It introduces a critical JWT verification weakness (hard-coded fallback secrets) and a runtime-breaking removal (pdf_download_url) that is still referenced by doc_preview.
Once you've addressed the issues Copilot identified, you can request another Copilot review.
This review doesn't count toward merge requirements. Sign up for the private preview to control whether Copilot approvals count.
Pull request overview
This PR updates the FairInSight AI/chat experience and admin legal-documents management, while introducing a new Python-based AI/rag_agent backend stack (FastAPI + SQLAlchemy + LangGraph) and removing the legacy AI/Backend and AI/backend_reasoning code.
Changes:
- Frontend: add “reasoning steps” loading ticker in chat UI and add admin delete confirmation flow for legal documents.
- Frontend: add
DELETE /documents/{id}API client helper. - Backend (AI): add new
AI/rag_agentservice with schema/models, DTOs, API routes, ingest + retrieval utilities, dockerization; remove old backend folders.
File summaries
| File | Description |
|---|---|
| frontend/src/pages/users/ai-chat/components/ChatMessages.tsx | Add reasoning ticker UI during loading + adjust post-action buttons logic |
| frontend/src/pages/admin/legal-documents/index.tsx | Add admin delete modal flow + optimistic UI delete |
| frontend/src/pages/admin/legal-documents/components/document-list-table.tsx | Pass delete/admin props through table |
| frontend/src/pages/admin/legal-documents/components/delete-confirm-modal.tsx | New confirm-delete modal component |
| frontend/src/core/services/law-ai.service.ts | Add deleteLaw() API call |
| frontend/src/_mocks/chat-data-mock.ts | Add Message.isLoading flag |
| AI/rag_agent/src/workflows/states/lawyer_state.py | New LangGraph state for lawyer agent |
| AI/rag_agent/src/workflows/states/ingest_state.py | Update ingest state fields/comments |
| AI/rag_agent/src/workflows/states/chat_state.py | New chat graph state definition |
| AI/rag_agent/src/workflows/states/init.py | Package init |
| AI/rag_agent/src/workflows/nodes/chat_nodes.py | Adjust abusive regex boundaries |
| AI/rag_agent/src/workflows/nodes/init.py | Package init |
| AI/rag_agent/src/workflows/ingest_graph.py | Update ingest graph topology (remove markup node) |
| AI/rag_agent/src/workflows/init.py | Package init |
| AI/rag_agent/src/services/vbpl.py | VBPL parsing/fields tweaks; (needs pdf URL helper restored) |
| AI/rag_agent/src/services/reranker.py | New cross-encoder reranker service |
| AI/rag_agent/src/services/reasoning.py | Increase evidence snippet length for composer |
| AI/rag_agent/src/services/llm.py | Improve JSON parsing robustness (unicode spaces + raw_decode fallback) |
| AI/rag_agent/src/services/lawyer/rerank.py | New evidence rerank helper for lawyer flow |
| AI/rag_agent/src/services/lawyer/hypothesis.py | New hypothesis builder + fact grading helpers |
| AI/rag_agent/src/services/lawyer/condition.py | New structured conditions reasoning step |
| AI/rag_agent/src/services/lawyer/init.py | Package init |
| AI/rag_agent/src/services/gpu.py | New VRAM swap helpers (OCR vs LLM) |
| AI/rag_agent/src/services/extraction/types.py | New extraction shared types |
| AI/rag_agent/src/services/extraction/pdf.py | New PDF profiling/extraction via PyMuPDF |
| AI/rag_agent/src/services/extraction/init.py | Export extraction APIs/types |
| AI/rag_agent/src/services/cloudinary.py | New Cloudinary raw-PDF management service |
| AI/rag_agent/src/services/init.py | Package init |
| AI/rag_agent/src/schema/models/unit.py | New SQLAlchemy Unit model |
| AI/rag_agent/src/schema/models/source_file.py | New SourceFile model |
| AI/rag_agent/src/schema/models/review_item.py | New ReviewItem model |
| AI/rag_agent/src/schema/models/reference.py | New Reference model |
| AI/rag_agent/src/schema/models/legal_tag.py | New LegalTag + DocumentTag models |
| AI/rag_agent/src/schema/models/document.py | New Document model |
| AI/rag_agent/src/schema/models/chunk.py | New Chunk model (pgvector) |
| AI/rag_agent/src/schema/models/chat.py | New ChatSession/ChatMessage models |
| AI/rag_agent/src/schema/models/base.py | New Base + timestamp mixin |
| AI/rag_agent/src/schema/models/amendment.py | New Amendment model |
| AI/rag_agent/src/schema/models/init.py | Aggregate model imports for metadata |
| AI/rag_agent/src/schema/enums/unit.py | New UnitType/UnitStatus enums |
| AI/rag_agent/src/schema/enums/tag.py | New tag + review enums |
| AI/rag_agent/src/schema/enums/relation.py | New relation/amendment enums |
| AI/rag_agent/src/schema/enums/init.py | Export enums |
| AI/rag_agent/src/schema/dto/lookup.py | New lookup DTOs |
| AI/rag_agent/src/schema/dto/lawyer.py | New lawyer request DTO |
| AI/rag_agent/src/schema/dto/document.py | New document list/detail DTOs |
| AI/rag_agent/src/schema/dto/chat.py | New chat request DTO |
| AI/rag_agent/src/schema/dto/init.py | Package init |
| AI/rag_agent/src/schema/init.py | Package init |
| AI/rag_agent/src/retrieval/hybrid.py | Adjust rerank candidate pool sizing |
| AI/rag_agent/src/retrieval/init.py | Package init |
| AI/rag_agent/src/repositories/chat_repo.py | New chat persistence repo |
| AI/rag_agent/src/repositories/asset_repo.py | New document PDF asset repo |
| AI/rag_agent/src/repositories/init.py | Package init |
| AI/rag_agent/src/ingest/tagging.py | New deterministic+LLM domain tagging |
| AI/rag_agent/src/ingest/relation_judge.py | Update hierarchy gate rules |
| AI/rag_agent/src/ingest/pipeline.py | Thin adapter for ingest graph |
| AI/rag_agent/src/ingest/codes.py | Normalize “Số:” label in official codes |
| AI/rag_agent/src/ingest/init.py | Package init |
| AI/rag_agent/src/data/sync_session.py | New sync DB session factory |
| AI/rag_agent/src/data/migrations/versions/d1f2a3b4c5d6_add_clause_point_to_relations.py | New Alembic migration |
| AI/rag_agent/src/data/migrations/versions/ce11ac153b9f_add_target_article_to_relations.py | New Alembic migration |
| AI/rag_agent/src/data/migrations/versions/a1b2c3d4e5f6_add_user_id_to_chat_sessions.py | New Alembic migration |
| AI/rag_agent/src/data/migrations/script.py.mako | New Alembic template |
| AI/rag_agent/src/data/migrations/env.py | New Alembic env |
| AI/rag_agent/src/data/init.py | Package init |
| AI/rag_agent/src/config/init.py | Package init |
| AI/rag_agent/src/api/v1/lookup.py | New lookup routes |
| AI/rag_agent/src/api/v1/lawyer.py | New lawyer agent route |
| AI/rag_agent/src/api/v1/ingest.py | New background ingest route |
| AI/rag_agent/src/api/v1/documents.py | New documents list/detail/update routes |
| AI/rag_agent/src/api/v1/chat.py | New chat route wiring to lawyer graph |
| AI/rag_agent/src/api/v1/init.py | Package init |
| AI/rag_agent/src/api/core/database.py | New async DB engine/session |
| AI/rag_agent/src/api/core/auth.py | JWT decode changes (security concerns noted) |
| AI/rag_agent/src/api/core/app.py | App init: add UTF-8 stdio + register routes |
| AI/rag_agent/src/api/core/init.py | Package init |
| AI/rag_agent/src/api/init.py | Package init |
| AI/rag_agent/src/init.py | Package init |
| AI/rag_agent/pyproject.toml | New Python project + deps (uv) |
| AI/rag_agent/main.py | New uvicorn entrypoint |
| AI/rag_agent/Dockerfile | New container build (preload models) |
| AI/rag_agent/docker-compose.yml | New production compose |
| AI/rag_agent/docker-compose.dev.yml | New dev compose (host overrides) |
| AI/rag_agent/alembic.ini | New Alembic config |
| AI/rag_agent/.python-version | Pin Python 3.12 |
| AI/rag_agent/.env.example | New env template |
| AI/rag_agent/.dockerignore | New dockerignore |
| AI/rag_agent/.claude/CLAUDE.MD | New AI assistant guidelines doc |
| AI/Backend/tests/test_llm_router.py | Remove legacy backend test |
| AI/Backend/tests/test_embedding.py | Remove legacy backend test |
| AI/Backend/tests/test_chunking.py | Remove legacy backend test |
| AI/Backend/test_local.py | Remove legacy script |
| AI/Backend/test_jina.py | Remove legacy script |
| AI/Backend/scripts/sync_runtime_requirements.py | Remove legacy tooling |
| AI/Backend/scripts/seed.py | Remove legacy seeding |
| AI/Backend/scripts/migrate_user_profile.py | Remove legacy migration script |
| AI/Backend/scratch/test_pipeline_run.py | Remove legacy scratch script |
| AI/Backend/scratch/scan_imports.py | Remove legacy scratch script |
| AI/Backend/requirements.runtime.txt | Remove legacy runtime requirements |
| AI/Backend/README.md | Remove legacy backend README |
| AI/Backend/pyproject.toml | Remove legacy backend project config |
| AI/Backend/Makefile | Remove legacy make targets |
| AI/Backend/feature_engineering/pipeline.py | Remove legacy feature pipeline |
| AI/Backend/feature_engineering/ingest.py | Remove legacy feature ingest |
| AI/Backend/feature_engineering/enricher.py | Remove legacy enricher |
| AI/Backend/feature_engineering/embedder.py | Remove legacy embedder |
| AI/Backend/feature_engineering/cleaner.py | Remove legacy cleaner |
| AI/Backend/feature_engineering/init.py | Remove legacy init |
| AI/Backend/Dockerfile | Remove legacy dockerfile |
| AI/Backend/docker-compose.yml | Remove legacy compose |
| AI/Backend/app/utils/ws_manager.py | Remove legacy ws manager |
| AI/Backend/app/utils/validators.py | Remove legacy validators |
| AI/Backend/app/utils/text_utils.py | Remove legacy text utils |
| AI/Backend/app/utils/settings.py | Remove legacy settings |
| AI/Backend/app/utils/logger.py | Remove legacy logger |
| AI/Backend/app/utils/init.py | Remove legacy init |
| AI/Backend/app/services/query_service.py | Remove legacy query service |
| AI/Backend/app/services/llm_service.py | Remove legacy llm service |
| AI/Backend/app/services/import_service.py | Remove legacy import service |
| AI/Backend/app/services/health_service.py | Remove legacy health service |
| AI/Backend/app/services/embedding_service.py | Remove legacy embedding service |
| AI/Backend/app/services/cloudinary_service.py | Remove legacy cloudinary service |
| AI/Backend/app/services/chat_service.py | Remove legacy chat service |
| AI/Backend/app/services/auth_service.py | Remove legacy auth service |
| AI/Backend/app/nodes/store.py | Remove legacy node |
| AI/Backend/app/nodes/search.py | Remove legacy node |
| AI/Backend/app/nodes/scan.py | Remove legacy node |
| AI/Backend/app/nodes/rewrite.py | Remove legacy node |
| AI/Backend/app/nodes/rerank.py | Remove legacy node |
| AI/Backend/app/nodes/generate.py | Remove legacy node |
| AI/Backend/app/nodes/embed.py | Remove legacy node |
| AI/Backend/app/nodes/init.py | Remove legacy init |
| AI/Backend/app/main.py | Remove legacy FastAPI entrypoint |
| AI/Backend/app/llm/prompts.py | Remove legacy prompts |
| AI/Backend/app/llm/init.py | Remove legacy init |
| AI/Backend/app/graphs/state.py | Remove legacy graph state |
| AI/Backend/app/graphs/query_graph.py | Remove legacy query graph |
| AI/Backend/app/graphs/import_graph.py | Remove legacy import graph |
| AI/Backend/app/graphs/init.py | Remove legacy init |
| AI/Backend/app/db/session.py | Remove legacy DB session |
| AI/Backend/app/db/models.py | Remove legacy DB models |
| AI/Backend/app/db/connection.py | Remove legacy DB connection |
| AI/Backend/app/db/init.py | Remove legacy init |
| AI/Backend/app/api/routers/query.py | Remove legacy query route |
| AI/Backend/app/api/routers/llm.py | Remove legacy llm route |
| AI/Backend/app/api/routers/health.py | Remove legacy health route |
| AI/Backend/app/api/routers/cloudinary.py | Remove legacy cloudinary route |
| AI/Backend/app/api/routers/auth.py | Remove legacy auth route |
| AI/Backend/app/init.py | Remove legacy init |
| AI/Backend/.env.example | Remove legacy env example |
| AI/Backend/.dockerignore | Remove legacy dockerignore |
| AI/backend_reasoning/tests/conftest.py | Remove backend_reasoning test fixtures |
| AI/backend_reasoning/src/ingest/normalizer.py | Remove legacy normalizer implementation |
| AI/backend_reasoning/src/api/v1/chat.py | Remove legacy chat route |
| AI/backend_reasoning/_ingest_local_test.py | Remove local ingest runner |
| .gitignore | Ignore new AI/rag_graph path |
Review details
Suppressed comments (2)
AI/rag_agent/src/api/core/auth.py:40
- Hard-coding fallback JWT secrets ("vjppro", "123456") makes token verification bypassable (an attacker can mint a valid token with a known secret). Token verification should only accept secrets that are explicitly configured via environment/settings.
AI/rag_agent/src/services/vbpl.py:82 services/doc_preview.pystill callsvbpl_module.pdf_download_url(...), but this helper was removed fromvbpl.py, which will raiseAttributeErrorwhencloudinary_urlis missing. Re-introducepdf_download_url()(URL builder only) to keep the fallback flow working.
- Files reviewed: 122/214 changed files
- Comments generated: 0
- Review effort level: Lite
We're testing this review assessment. Please use 👍 or 👎 to tell us if it's correct.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.