Skip to content

feat: implement ML-assisted content moderation system - #449

Open
iyanumajekodunmi756 wants to merge 3 commits into
AetherEdu:mainfrom
iyanumajekodunmi756:feat/ml-assisted-content-moderation
Open

feat: implement ML-assisted content moderation system#449
iyanumajekodunmi756 wants to merge 3 commits into
AetherEdu:mainfrom
iyanumajekodunmi756:feat/ml-assisted-content-moderation

Conversation

@iyanumajekodunmi756

Copy link
Copy Markdown
Contributor

Implements a comprehensive ML-based content moderation system that automatically pre-screens submissions for policy violations, routes high-risk content to a human review queue, and uses moderator decisions to improve the model over time.

Key features:

  • ML pre-screening service with pattern-based policy violation detection across 14 policy categories (hate speech, spam, NSFW, violence, etc.)
  • Multi-factor risk scoring: text analysis, metadata, user history, content similarity, and per-policy violation scores
  • Auto-approval/rejection for high-confidence decisions with configurable thresholds
  • Priority-based human review queue with moderator assignment and load balancing
  • Complete appeal flow: rejected content authors can submit appeals with evidence, reviewed by admins
  • Model feedback loop: moderator decisions are tracked as correct/incorrect predictions for accuracy monitoring and retraining
  • Async background scoring worker for processing batches of submissions
  • RESTful API at /api/moderation with full validation (submit, score, queue, decide, appeal)
  • Enhanced admin moderation UI with ML risk score visualization, severity indicators, policy breakdown, and dark-themed appeal review
  • Comprehensive test suite (23 tests covering scoring, queuing, appeals, integration lifecycle, and performance)

Files modified:

  • backend/src/models/Moderation.ts (new: data model/types)
  • backend/src/services/moderation/ (new: scoring, queue, appeal services)
  • backend/src/workers/moderationJob.ts (new: async scoring worker)
  • backend/src/controllers/moderationController.ts (new: REST controller)
  • backend/src/routes/moderation.ts (new: API routes with validation)
  • backend/src/index.ts (register moderation routes and worker)
  • backend/tests/moderation.test.js (new: test suite)
  • backend/tests/setup.js (fix: mock roles utility for UserRole export)
  • backend/jest.moderation.config.js (new: standalone test config)
  • frontend/src/app/admin/content/moderation/page.tsx (enhanced UI)

Closes #402

iyanumajekodunmi756 and others added 3 commits August 22, 2026 16:28
Implements a comprehensive ML-based content moderation system that
automatically pre-screens submissions for policy violations, routes
high-risk content to a human review queue, and uses moderator decisions
to improve the model over time.

Key features:
- ML pre-screening service with pattern-based policy violation detection
  across 14 policy categories (hate speech, spam, NSFW, violence, etc.)
- Multi-factor risk scoring: text analysis, metadata, user history,
  content similarity, and per-policy violation scores
- Auto-approval/rejection for high-confidence decisions with configurable
  thresholds
- Priority-based human review queue with moderator assignment and load
  balancing
- Complete appeal flow: rejected content authors can submit appeals with
  evidence, reviewed by admins
- Model feedback loop: moderator decisions are tracked as correct/incorrect
  predictions for accuracy monitoring and retraining
- Async background scoring worker for processing batches of submissions
- RESTful API at /api/moderation with full validation (submit, score,
  queue, decide, appeal)
- Enhanced admin moderation UI with ML risk score visualization,
  severity indicators, policy breakdown, and dark-themed appeal review
- Comprehensive test suite (23 tests covering scoring, queuing, appeals,
  integration lifecycle, and performance)

Files modified:
- backend/src/models/Moderation.ts (new: data model/types)
- backend/src/services/moderation/ (new: scoring, queue, appeal services)
- backend/src/workers/moderationJob.ts (new: async scoring worker)
- backend/src/controllers/moderationController.ts (new: REST controller)
- backend/src/routes/moderation.ts (new: API routes with validation)
- backend/src/index.ts (register moderation routes and worker)
- backend/tests/moderation.test.js (new: test suite)
- backend/tests/setup.js (fix: mock roles utility for UserRole export)
- backend/jest.moderation.config.js (new: standalone test config)
- frontend/src/app/admin/content/moderation/page.tsx (enhanced UI)

Closes AetherEdu#402

🤖 Generated with Codebuff
Co-Authored-By: Codebuff <noreply@codebuff.com>
- Removed the `isClient` guard in TourProvider that skipped rendering
  the context during SSR, causing `useTour()` to throw in all child
  components (OnboardingModal, TourGuide, ProfilePageClient) during
  Next.js static page generation. The provider now always renders so
  components calling useTour() receive a valid context regardless of
  rendering phase.

- Fixed bundle-budget workflow to use `npm ci -w frontend` from the
  monorepo root instead of `npm ci` in the frontend directory, which
  failed because workspace dependencies require root-level resolution.

Fixes the failing Build Frontend, CI Status Check, and Bundle Budget
CI checks on PR AetherEdu#449.

🤖 Generated with Codebuff
Co-Authored-By: Codebuff <noreply@codebuff.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Backend] ML-assisted content moderation queue

1 participant