Arabic Text Normalizer

This project provides a Python-based tool for normalizing Arabic text. It offers two normalization models: a standard model for rule-based normalization and an LLM-based model for more advanced, context-aware normalization using Google's Gemini API.

Features

• Dual Normalization Models: Choose between a standard, rule-based normalizer and a powerful LLM-based normalizer.
• Customizable Configuration: Easily configure normalization rules through a JSON file.
• Secure API Key Handling: Uses a .env file to securely manage your Gemini API key.
• Batch Processing: Process text files in batches with progress tracking.
• Easy to Use: Comes with a simple command-line interface and a run.bat script for easy execution.

Installation

1. Clone the repository:

   git clone <repository-url>
   cd normalizer

2. Create a Python virtual environment:

   python -m venv venv

3. Activate the virtual environment:

   • On Windows:

     venv\Scripts\activate

   • On macOS/Linux:

     source venv/bin/activate

4. Install the required dependencies:

   pip install -r requirements.txt

Configuration

1. Create a .env file:

   • Create a file named .env in the root directory of the project.
   • Add your Gemini API key to the .env file as follows:

     GEMINI_API_KEY="your_api_key_here"
     

2. Customize the normalization rules (optional):

   • The config.json file contains the rules for the standard normalizer. You can modify this file to add, remove, or change the normalization rules.

Usage

This project can be run from the command line using the main.py script or the run.bat script on Windows.

Using the run.bat script (Windows)

The run.bat script provides a convenient way to run the normalizer. It activates the virtual environment and provides examples of how to run the main.py script.

1. Open the run.bat file in a text editor.
2. Uncomment one of the example commands and replace the placeholder file paths with your actual input and output file paths.
3. Save the run.bat file and run it from the command line.

Using the main.py script

You can also run the main.py script directly from the command line.

python main.py --input_file <input_path> --output_file <output_path> --model <model_type> --config <config_path>

Arguments:

• --input_file: The path to the input text file.
• --output_file: The path to save the normalized text.
• --model: The normalization model to use. Can be standard or llm.
• --config: The path to the JSON config file for the normalizer (default is config.json).

Examples:

• Standard Normalization:

  python main.py --input_file "data/sample_input.txt" --output_file "data/sample_output_standard.txt" --model "standard"

• LLM-based Normalization:

  python main.py --input_file "data/sample_input.txt" --output_file "data/sample_output_llm.txt" --model "llm"

Models

Standard Normalizer

The standard normalizer is a rule-based normalizer that applies a set of predefined rules to the text. These rules are defined in the config.json file and are executed by the arabic_normalizer.py script. This model is fast and efficient for common normalization tasks.

LLM-based Normalizer

The llm normalizer uses a Large Language Model (LLM) from Google's Gemini API to perform more advanced, context-aware normalization. This model is more powerful than the standard normalizer and can handle more complex normalization tasks, but it requires an API key and an internet connection.

Contributing

Contributions are welcome! If you have any suggestions, bug reports, or feature requests, please open an issue or submit a pull request.