feat: implement ML-assisted content moderation system - #449
Open
iyanumajekodunmi756 wants to merge 3 commits into
Open
feat: implement ML-assisted content moderation system#449iyanumajekodunmi756 wants to merge 3 commits into
iyanumajekodunmi756 wants to merge 3 commits into
Conversation
Implements a comprehensive ML-based content moderation system that automatically pre-screens submissions for policy violations, routes high-risk content to a human review queue, and uses moderator decisions to improve the model over time. Key features: - ML pre-screening service with pattern-based policy violation detection across 14 policy categories (hate speech, spam, NSFW, violence, etc.) - Multi-factor risk scoring: text analysis, metadata, user history, content similarity, and per-policy violation scores - Auto-approval/rejection for high-confidence decisions with configurable thresholds - Priority-based human review queue with moderator assignment and load balancing - Complete appeal flow: rejected content authors can submit appeals with evidence, reviewed by admins - Model feedback loop: moderator decisions are tracked as correct/incorrect predictions for accuracy monitoring and retraining - Async background scoring worker for processing batches of submissions - RESTful API at /api/moderation with full validation (submit, score, queue, decide, appeal) - Enhanced admin moderation UI with ML risk score visualization, severity indicators, policy breakdown, and dark-themed appeal review - Comprehensive test suite (23 tests covering scoring, queuing, appeals, integration lifecycle, and performance) Files modified: - backend/src/models/Moderation.ts (new: data model/types) - backend/src/services/moderation/ (new: scoring, queue, appeal services) - backend/src/workers/moderationJob.ts (new: async scoring worker) - backend/src/controllers/moderationController.ts (new: REST controller) - backend/src/routes/moderation.ts (new: API routes with validation) - backend/src/index.ts (register moderation routes and worker) - backend/tests/moderation.test.js (new: test suite) - backend/tests/setup.js (fix: mock roles utility for UserRole export) - backend/jest.moderation.config.js (new: standalone test config) - frontend/src/app/admin/content/moderation/page.tsx (enhanced UI) Closes AetherEdu#402 🤖 Generated with Codebuff Co-Authored-By: Codebuff <noreply@codebuff.com>
- Removed the `isClient` guard in TourProvider that skipped rendering the context during SSR, causing `useTour()` to throw in all child components (OnboardingModal, TourGuide, ProfilePageClient) during Next.js static page generation. The provider now always renders so components calling useTour() receive a valid context regardless of rendering phase. - Fixed bundle-budget workflow to use `npm ci -w frontend` from the monorepo root instead of `npm ci` in the frontend directory, which failed because workspace dependencies require root-level resolution. Fixes the failing Build Frontend, CI Status Check, and Bundle Budget CI checks on PR AetherEdu#449. 🤖 Generated with Codebuff Co-Authored-By: Codebuff <noreply@codebuff.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Implements a comprehensive ML-based content moderation system that automatically pre-screens submissions for policy violations, routes high-risk content to a human review queue, and uses moderator decisions to improve the model over time.
Key features:
Files modified:
Closes #402