diff --git a/.idea/claudeCodeEditorTabs.xml b/.idea/claudeCodeEditorTabs.xml new file mode 100644 index 0000000..5b81d50 --- /dev/null +++ b/.idea/claudeCodeEditorTabs.xml @@ -0,0 +1,21 @@ + + + + + + + + \ No newline at end of file diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index 627759d..e64f34e 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -95,10 +95,12 @@ pytest -v --tb=long | File | Coverage area | |------|---------------| -| `test_batch_creation.py` | JSONL batch generation — prompt construction, schema building, `forbid_additional_props`, non-ASCII text, edge cases | -| `test_batch_parsing.py` | Result parsing (`_safe_parse_model_text`), OpenAI client creation, `get_batch_results` with mocked API (success, auth failure, rate limit, timeout, malformed output, fenced JSON) | +| `test_task.py` | The Task engine — prompt/list rendering (all list formats), dynamic output-model/schema building per field type, `validate_task`, `Task.to_dict`/`from_dict` round-trip, `forbid_additional_props` | +| `test_presets.py` | The three built-in presets are valid, editable `Task`s | +| `test_batch_creation.py` | JSONL batch generation from a `Task` — prompt rendering reaching the request body, system message handling, strict schema attached, non-ASCII text, edge cases | +| `test_batch_parsing.py` | Result parsing (`_safe_parse_model_text`), OpenAI client creation, `send_batch` validation + sidecar writing, `get_batch_results` with mocked API (success, auth failure, rate limit, timeout, malformed output, fenced JSON, sidecar rejoin) | | `test_data_conversion.py` | `make_str_enum`, `to_long_df` (including multi-label explode), `join_datasets` | -| `test_data_import.py` | Delimiter sniffing, CSV/TSV/Excel reading, non-ASCII content, empty files, realistic fixture | +| `test_data_import.py` | Delimiter sniffing, CSV/TSV/Excel reading, non-ASCII content, empty files, realistic fixture, full-table `load_table` | | `test_settings.py` | User config load/save/roundtrip, `get_setting` precedence | | `test_reliability.py` | Cohen's kappa — perfect agreement, known value, edge cases (empty, mismatched lengths, single label, unicode labels) | @@ -126,21 +128,25 @@ CodebookAI/ ├── requirements-dev.txt Test / development dependencies ├── pytest.ini Pytest configuration │ +├── core/ The Task engine (pure, fully testable) +│ ├── task.py Task/TaskList/OutputField, prompt rendering, +│ │ dynamic Pydantic model + strict schema builder +│ └── presets.py Built-in Task presets (single/multi/keyword) +│ ├── batch_processing/ OpenAI Batch API workflow -│ ├── batch_creation.py JSONL generation (pure, fully testable) -│ ├── batch_method.py Batch submission / retrieval / result parsing +│ ├── batch_creation.py JSONL generation from a Task (pure, fully testable) +│ ├── batch_method.py Batch submission / retrieval / sidecar rejoin │ └── batch_error_handling.py Error reporting UI │ -├── live_processing/ Real-time classification -│ ├── single_label_live.py Single-label pipeline -│ ├── multi_label_live.py Multi-label pipeline +├── live_processing/ Real-time (synchronous) execution + analysis tools +│ ├── task_live.py Runs any Task row-by-row via the live API │ ├── reliability_calculator.py Cohen's kappa (pure, fully testable) -│ ├── keyword_extraction_live.py │ ├── correlogram.py │ └── sampler.py │ ├── file_handling/ File I/O (mostly pure, fully testable) -│ ├── data_import.py CSV / Excel reader + import-wizard GUI +│ ├── data_import.py CSV / Excel reader; single-column wizard (for +│ │ Lists) + full-table loader (for the Task Builder) │ └── data_conversion.py make_str_enum, to_long_df, join_datasets │ ├── settings/ Configuration @@ -150,6 +156,7 @@ CodebookAI/ │ └── models_registry.py OpenAI model list cache │ ├── ui/ Tkinter GUI components +│ └── task_builder.py The Task Builder -- the app's central window │ ├── tests/ Automated test suite │ ├── conftest.py Shared fixtures; in-memory keyring setup @@ -157,6 +164,8 @@ CodebookAI/ │ │ ├── sample_labels.csv │ │ ├── sample_quotes.csv │ │ └── realistic_dataset.csv +│ ├── test_task.py +│ ├── test_presets.py │ ├── test_batch_creation.py │ ├── test_batch_parsing.py │ ├── test_data_conversion.py diff --git a/README.md b/README.md index c165eb4..48159da 100644 --- a/README.md +++ b/README.md @@ -1,7 +1,7 @@ ![CodebookAI Logo](./assets/BannerNarrow.png) -CodebookAI is a tool designed to assist qualitative researchers in processing large datasets through OpenAI's GPT models (e.g., 4o, 5, o3, etc.). It enables batch processing of text snippets against a set of labels, significantly reducing the cost and time associated with manual coding, as well as a variety of other tools aimed at qualitative data preparation and analysis. +CodebookAI is a tool designed to assist qualitative researchers in processing large datasets through OpenAI's GPT models (e.g., 4o, 5, o3, etc.). Its **Task Builder** lets you write your own prompt against an imported table, define named label Lists that constrain the model's response to on-list values only (no hallucinated or reworded labels to clean up), and choose exactly which columns -- including existing IDs -- carry through to your results. Run a task live for quick results or submit it as a batch for large, cost-effective jobs. The app also includes a variety of other tools aimed at qualitative data preparation and analysis. ## Getting Started diff --git a/batch_processing/batch_creation.py b/batch_processing/batch_creation.py index ccacef5..64de0f1 100644 --- a/batch_processing/batch_creation.py +++ b/batch_processing/batch_creation.py @@ -1,194 +1,62 @@ """ -JSON schema handling and batch file generation for OpenAI text classification. +JSONL batch file generation for OpenAI Structured Outputs requests. -This module provides utilities for creating JSON schemas that enforce structured -responses from OpenAI models, and for generating JSONL batch files for processing -multiple text classification requests efficiently. +This used to be three near-identical functions (generate_single_label_batch, +generate_multi_label_batch, generate_keyword_extraction_batch), each with its +own hardcoded prompt and Pydantic model. They're now one function driven by a +core.task.Task -- the prompt, the constant/per-row split, and the output +schema all come from the task instead of being baked in here. """ from __future__ import annotations -import io, json -from enum import Enum -from io import BytesIO -from typing import Optional, List -from pydantic import BaseModel, ConfigDict, Field - -from file_handling.data_conversion import make_str_enum -from file_handling.data_import import import_data -from settings.config import model - - -def forbid_additional_props(schema: dict) -> dict: - """Recursively set additionalProperties:false on every object schema.""" - if not isinstance(schema, dict): - return schema - - # If this schema node is an object, set additionalProperties: false - if schema.get("type") == "object": - schema.setdefault("properties", {}) - schema["additionalProperties"] = False - - # Recurse into each property - for prop_schema in schema.get("properties", {}).values(): - forbid_additional_props(prop_schema) - - # Recurse into patternProperties (rare but safe) - for prop_schema in schema.get("patternProperties", {}).values(): - forbid_additional_props(prop_schema) - - # If it's an array, recurse into "items" - if schema.get("type") == "array" and "items" in schema: - forbid_additional_props(schema["items"]) - # oneOf/anyOf/allOf branches - for key in ("oneOf", "anyOf", "allOf"): - if key in schema and isinstance(schema[key], list): - for s in schema[key]: - forbid_additional_props(s) +import io +import json +from io import BytesIO - return schema +from core.task import Task, build_request_params, build_strict_schema, render_prompt -def generate_single_label_batch(labels, quotes) -> BytesIO | None: +def generate_batch(task: Task, rows: list[dict]) -> BytesIO: """ Create a JSONL batch as bytes, in memory (no disk writes). - Returns a BytesIO whose .name is set to 'batchinput.jsonl'. - """ - labels_enum = make_str_enum("Label", labels) - - class LabeledQuote(BaseModel): - label: labels_enum # STRICT: must be one of labels - model_config = ConfigDict(use_enum_values=True, extra='forbid') - - SCHEMA = LabeledQuote.model_json_schema() - STRICT_SCHEMA = forbid_additional_props(SCHEMA) - - buf = io.BytesIO() - # Give the buffer a filename so the API knows its type - buf.name = "batchinput.jsonl" - - lines = [] - for i, q in enumerate(quotes, 1): - lines.append({ - "custom_id": f"quote-{i:05d}", - "method": "POST", - "url": "/v1/responses", - "body": { - "model": model, - "input": [ - {"role": "user", - "content": f"Label this quote with exactly one label from the allowed set.\n" - f"Allowed: {', '.join(labels)}\nQuote: {q}"} - ], - "text": { - "format": { - "type": "json_schema", - "name": "LabeledQuote", - "schema": STRICT_SCHEMA, - "strict": True - } - }, - "metadata": {"quote": q} - } - }) - - for line in lines: - buf.write((json.dumps(line, ensure_ascii=False) + "\n").encode("utf-8")) - - buf.seek(0) - return buf + One row -> one request, custom_id = "row-00001", "row-00002", ... + (1-based, matching the row order of `rows`). - -def generate_multi_label_batch(labels, quotes) -> BytesIO | None: - """ - Create a JSONL batch as bytes, in memory (no disk writes). - Returns a BytesIO whose .name is set to 'batchinput.jsonl'. + Row data itself is NOT sent as request metadata -- OpenAI's metadata has + a ~512 char/value limit that arbitrary carried columns could exceed. + Carried columns are rejoined locally by row position after download + (see batch_processing.batch_method). """ - labels_enum = make_str_enum("Label", labels) - - class LabeledQuoteMulti(BaseModel): - label: List[labels_enum] = Field(..., min_length=1) - model_config = ConfigDict(use_enum_values=True, extra='forbid') - - SCHEMA = LabeledQuoteMulti.model_json_schema() - STRICT_SCHEMA = forbid_additional_props(SCHEMA) + schema = build_strict_schema(task) + request_params = build_request_params(task) buf = io.BytesIO() - # Give the buffer a filename so the API knows its type buf.name = "batchinput.jsonl" - lines = [] - for i, q in enumerate(quotes, 1): - lines.append({ - "custom_id": f"quote-{i:05d}", - "method": "POST", - "url": "/v1/responses", - "body": { - "model": model, - "input": [ - {"role": "user", - "content": "Label this quote with labels from the allowed set only. \n" - f"Allowed: {', '.join(labels)}\nQuote: {q}"} - ], - "text": { - "format": { - "type": "json_schema", - "name": "LabeledQuoteMulti", - "schema": STRICT_SCHEMA, - "strict": True - } - }, - "metadata": {"quote": q} - } - }) - - for line in lines: - buf.write((json.dumps(line, ensure_ascii=False) + "\n").encode("utf-8")) - - buf.seek(0) - return buf - - -def generate_keyword_extraction_batch(texts) -> BytesIO | None: - """ - Create a JSONL batch as bytes, in memory (no disk writes). - Returns a BytesIO whose .name is set to 'batchinput.jsonl'. - """ - class KeywordExtraction(BaseModel): - keywords: list[str] = Field(..., min_length=1) - model_config = ConfigDict(extra='forbid') + for i, row in enumerate(rows, 1): + messages = [] + if task.system: + messages.append({"role": "system", "content": render_prompt(task, row, text=task.system)}) + messages.append({"role": "user", "content": render_prompt(task, row)}) - SCHEMA = KeywordExtraction.model_json_schema() - STRICT_SCHEMA = forbid_additional_props(SCHEMA) - - buf = io.BytesIO() - # Give the buffer a filename so the API knows its type - buf.name = "batchinput.jsonl" - - lines = [] - for i, txt in enumerate(texts, 1): - lines.append({ - "custom_id": f"text-{i:05d}", + line = { + "custom_id": f"row-{i:05d}", "method": "POST", "url": "/v1/responses", "body": { - "model": model, - "input": [{"role": "system", "content": "You are an expert at structured data extraction."}, - {"role": "user", "content": f"Extract the keywords from this text: {txt}"} - ], + "input": messages, "text": { "format": { "type": "json_schema", - "name": "KeywordExtraction", - "schema": STRICT_SCHEMA, - "strict": True + "name": "TaskOutput", + "schema": schema, + "strict": True, } }, - "metadata": {"quote": txt} - } - }) - - for line in lines: + **request_params, + }, + } buf.write((json.dumps(line, ensure_ascii=False) + "\n").encode("utf-8")) buf.seek(0) diff --git a/batch_processing/batch_method.py b/batch_processing/batch_method.py index 2380c09..87bdbe2 100644 --- a/batch_processing/batch_method.py +++ b/batch_processing/batch_method.py @@ -2,23 +2,29 @@ OpenAI batch processing functionality for text classification. This module handles the creation, monitoring, and result retrieval of OpenAI -batch processing jobs. It provides functions to submit large collections of -text classification requests efficiently using OpenAI's batch API. +batch processing jobs, driven by a core.task.Task instead of six hardcoded +per-tool flows. File import now happens once, before send_batch is called +(the Task Builder owns that), and carried columns are joined back in from a +local sidecar file rather than round-tripped through OpenAI request metadata. """ import json -from batch_processing.batch_error_handling import handle_batch_fail -from file_handling.data_conversion import to_long_df, save_as_csv, join_datasets -from file_handling.data_import import import_data -from settings import config, secrets_store -from settings.user_config import get_setting -from openai import OpenAI -from batch_processing.batch_creation import generate_single_label_batch, generate_multi_label_batch, \ - generate_keyword_extraction_batch +import re +import shutil from datetime import datetime -from zoneinfo import ZoneInfo +from pathlib import Path from typing import Any -import re +from zoneinfo import ZoneInfo + +import pandas as pd +from openai import OpenAI + +from batch_processing.batch_creation import generate_batch +from batch_processing.batch_error_handling import handle_batch_fail +from core.task import Task, build_request_params, validate_task +from file_handling.data_conversion import save_as_csv, to_long_df +from settings import secrets_store +from settings.user_config import get_setting, get_user_config_dir def get_client() -> OpenAI: @@ -37,92 +43,113 @@ def get_client() -> OpenAI: return OpenAI(api_key=api_key) -def send_batch(root: Any, type: str) -> Any: - """ - Create and submit a new batch processing job to OpenAI. +# --------------------------------------------------------------------------- +# Local sidecar: carries task.carry_columns through the batch round-trip. +# OpenAI's request metadata caps values at ~512 chars, which arbitrary carried +# columns (or even a single long quote) could exceed -- so carried data is +# kept locally, keyed by batch ID, and rejoined by row position on download. +# --------------------------------------------------------------------------- + +def _sidecar_dir() -> Path: + d = get_user_config_dir() / "sidecars" + d.mkdir(parents=True, exist_ok=True) + return d + + +def _sidecar_path(batch_id: str) -> Path: + return _sidecar_dir() / f"{batch_id}.csv" + + +def _write_sidecar(batch_id: str, df: "pd.DataFrame", carry_columns: list[str]) -> None: + if not carry_columns: + return + df[carry_columns].to_csv(_sidecar_path(batch_id), index=False) + + +def _read_sidecar(batch_id: str) -> "pd.DataFrame | None": + path = _sidecar_path(batch_id) + if not path.exists(): + return None + return pd.read_csv(path, dtype=str, keep_default_na=False) + + +def _row_index_from_custom_id(custom_id: str | None) -> int | None: + """'row-00001' -> 0 (matches sidecar's 0-based row order).""" + if not custom_id: + return None + m = re.match(r"row-(\d+)$", custom_id) + return int(m.group(1)) - 1 if m else None - This function prompts the user to select CSV files containing labels and - text, generates a properly formatted batch request, and submits it to - OpenAI's batch processing API. + +def _task_type(task: Task) -> str: + """Derive a display label for the batches table from the task's output shape. + + ponytail: precedence heuristic, not a formal classifier -- refine only if a + task genuinely mixes output field types and the label needs to reflect that. + """ + field_types = {f.type for f in task.output_fields} + if "multi_choice" in field_types: + return "multi-label" + if "choice" in field_types: + return "classification" + if "text_list" in field_types: + return "keyword extraction" + return "free-text" + + +def _batch_metadata(task: Task, row_count: int, dataset: str | None) -> dict[str, str]: + """Build the OpenAI batch metadata dict, including only the inference + settings that actually apply to the chosen model (reuses build_request_params + so the standard-vs-reasoning rule lives in exactly one place).""" + params = build_request_params(task) + md = { + "model": task.model, + "rows": str(row_count), + "type": _task_type(task), + "dataset": (dataset or "")[:64], + } + if "temperature" in params: + md["temperature"] = str(params["temperature"]) + if "reasoning" in params: + md["reasoning"] = params["reasoning"]["effort"] + if "max_output_tokens" in params: + md["max_output_tokens"] = str(params["max_output_tokens"]) + return md + + +def send_batch(task: Task, df: "pd.DataFrame", dataset: str | None = None) -> Any: + """ + Submit a Task run against an already-imported DataFrame as a new batch job. Args: - root: Tkinter root window for file dialog ownership + task: the built Task (prompt, lists, output fields, carried columns) + df: the full imported table + dataset: display name of the source dataset (e.g. the imported filename), + recorded in batch metadata for the batches table Returns: OpenAI batch object containing job details and status Raises: - Exception: If file selection is cancelled, files are invalid, or API call fails + TaskValidationError: if the task references unknown columns/lists + Exception: if the API call fails """ client = get_client() - datasets = () - # Generate the JSONL batch file in memory - if type == "single_label": - # Get labels data - from_import = import_data(root, "Select the labels data") - if from_import is None: - return # user hit Cancel - labels, labels_nickname = from_import - - # Get text data - from_import = import_data(root, "Select the text data") - if from_import is None: - return # user hit Cancel - text, quotes_nickname = from_import - - batch_bytes = generate_single_label_batch(labels, text) - datasets = (labels_nickname, quotes_nickname) - joined_datasets = join_datasets(datasets) - - elif type == "multi_label": - # Get labels data - from_import = import_data(root, "Select the labels data") - if from_import is None: - return # user hit Cancel - labels, labels_nickname = from_import - - # Get text data - from_import = import_data(root, "Select the text data") - if from_import is None: - return # user hit Cancel - text, quotes_nickname = from_import - - datasets = (labels_nickname, quotes_nickname) - joined_datasets = join_datasets(datasets) - batch_bytes = generate_multi_label_batch(labels, text) - - elif type == "keyword_extraction": - # Get text data - from_import = import_data(root, "Select the text data") - if from_import is None: - return # user hit Cancel - text, text_nickname = from_import - - joined_datasets = join_datasets(text_nickname) - batch_bytes = generate_keyword_extraction_batch(text) - else: - raise ValueError(f"Unknown batch type: {type}") - - # Upload the batch file to OpenAI - batch_input_file = client.files.create( - file=batch_bytes, - purpose="batch", - ) + validate_task(task, list(df.columns)) + + rows = df.to_dict(orient="records") + batch_bytes = generate_batch(task, rows) - batch_input_file_id = batch_input_file.id + batch_input_file = client.files.create(file=batch_bytes, purpose="batch") - # Create the batch processing job batch = client.batches.create( - input_file_id=batch_input_file_id, + input_file_id=batch_input_file.id, endpoint="/v1/responses", completion_window="24h", - metadata={ - "model": get_setting("model", "gpt-4o"), - "type": type, - "dataset(s)": joined_datasets - } + metadata=_batch_metadata(task, len(rows), dataset), ) + _write_sidecar(batch.id, df, task.carry_columns) return batch @@ -140,7 +167,6 @@ def get_batch_status(batch_id: str) -> Any: return client.batches.retrieve(batch_id) - # Extract classification results from the API responses def _safe_parse_model_text(s: str): t = s.strip() # strip ```json fences if present @@ -165,19 +191,15 @@ def get_batch_results(batch_id: str) -> None: """ Download and save the results of a completed batch processing job. - This function retrieves the output file from a completed batch job, - parses the classification results, and prompts the user to save them - as a CSV file. + Retrieves the output file, parses each response, rejoins carried columns + from the local sidecar (by row position), and prompts the user to save + the combined table as a CSV. Args: batch_id: Unique identifier for the completed batch job Raises: Exception: If batch is not complete, results are malformed, or save fails - - Note: - Results are automatically saved as a DataFrame with columns for - quote, label, and confidence from the classification response. """ client = get_client() status = get_batch_status(batch_id) @@ -195,6 +217,8 @@ def get_batch_results(batch_id: str) -> None: if line.strip() ] + sidecar = _read_sidecar(batch_id) # row order matches the original request order + responses, bad_rows = [], [] for res in results: @@ -205,18 +229,20 @@ def get_batch_results(batch_id: str) -> None: if parsed is None: bad_rows.append({ "custom_id": res.get("custom_id"), - "quote": body.get("metadata", {}).get("quote", ""), "raw_text": text_output }) continue - metadata = body.get('metadata', {}) - combined = {"custom_id": res.get("custom_id"), **metadata, **parsed} + combined = {"custom_id": res.get("custom_id")} + if sidecar is not None: + idx = _row_index_from_custom_id(res.get("custom_id")) + if idx is not None and 0 <= idx < len(sidecar): + combined.update(sidecar.iloc[idx].to_dict()) + combined.update(parsed) if note: combined["repair_note"] = note responses.append(combined) - # Now continue with your DF creation df = to_long_df(responses) save_as_csv(df) @@ -235,16 +261,20 @@ def cancel_batch(batch_id: str) -> Any: return client.batches.cancel(batch_id) -def list_batches(): - """ - Retrieve and categorize recent batch processing jobs. +# Statuses that mean the batch is still active (used by the UI to decide +# whether to offer Cancel vs Download in the batches table). +ONGOING_STATUSES = {"validating", "in_progress", "cancelling", "finalizing"} - This function fetches the most recent batch jobs and separates them into - ongoing (active) and completed batches based on their status. + +def list_batches() -> list[dict[str, str]]: + """ + Retrieve recent batch processing jobs with their display metadata. Returns: - Tuple of (ongoing_batches, done_batches) where each is a list of tuples - containing (batch_id, status, created_timestamp) + List of dicts, most-recent-first, one per batch, with keys: + id, status, type, dataset, model, rows, progress, created, + temperature, reasoning, max_output_tokens. + Metadata fields default to "" when a batch predates their introduction. Note: The number of batches returned is limited by config.max_batches. @@ -252,31 +282,68 @@ def list_batches(): """ limit = get_setting("max_batches", 4) client = get_client() + tz = ZoneInfo(get_setting("time_zone", "UTC")) + + result = [] + for batch in client.batches.list(limit=limit): + created_time = datetime.fromtimestamp(batch.created_at, tz) + md = batch.metadata or {} + + rc = batch.request_counts + progress = f"{rc.completed}/{rc.total}" if rc else "" + if rc and rc.failed: + progress += f" (+{rc.failed} failed)" + + result.append({ + "id": batch.id, + "status": batch.status, + "type": md.get("type", ""), + "dataset": md.get("dataset", ""), + "model": md.get("model", ""), + "rows": md.get("rows", ""), + "progress": progress, + "created": created_time.strftime("%Y-%m-%d %H:%M"), + "temperature": md.get("temperature", ""), + "reasoning": md.get("reasoning", ""), + "max_output_tokens": md.get("max_output_tokens", ""), + }) + return result + + +def rerun_batch(batch_id: str, count: int = 1) -> list[str]: + """ + Resubmit a batch with the exact same settings, `count` times. - batches = client.batches.list(limit=limit) - ongoing_batches = [] - done_batches = [] - - # Categorize batches by status - ongoing_statuses = {"validating", "in_progress", "cancelling", "finalizing"} - - for batch in batches: - # Convert timestamp to configured timezone - created_time = datetime.fromtimestamp(batch.created_at, ZoneInfo(get_setting("time_zone", "UTC"))) - - md = (batch.metadata or {}) - tuple_of_batch_data = ( - batch.id, - batch.status, - created_time, - md.get("model", ""), - md.get("type", ""), - md.get("dataset(s)", "") - ) + Reuses the original batch's retained input file and metadata directly -- + no Task or DataFrame needed -- so the new batches run the identical + requests. The local sidecar (carried columns) is copied to each new batch + id so results still rejoin correctly on download. + + Args: + batch_id: the batch to rerun + count: how many times to resubmit (default 1) - if batch.status in ongoing_statuses: - ongoing_batches.append(tuple_of_batch_data) - else: - done_batches.append(tuple_of_batch_data) + Returns: + List of new batch ids, in submission order. - return ongoing_batches, done_batches + Raises: + Exception: if the API call fails (e.g. the input file was deleted -- + ponytail: OpenAI retains input files for a limited time; a rerun + of a very old batch may simply error, which is surfaced as-is). + """ + client = get_client() + orig = client.batches.retrieve(batch_id) + + new_ids = [] + for _ in range(count): + new_batch = client.batches.create( + input_file_id=orig.input_file_id, + endpoint=orig.endpoint, + completion_window=orig.completion_window or "24h", + metadata=dict(orig.metadata or {}), + ) + src = _sidecar_path(batch_id) + if src.exists(): + shutil.copyfile(src, _sidecar_path(new_batch.id)) + new_ids.append(new_batch.id) + return new_ids diff --git a/core/__init__.py b/core/__init__.py new file mode 100644 index 0000000..e69de29 diff --git a/core/presets.py b/core/presets.py new file mode 100644 index 0000000..2cebb73 --- /dev/null +++ b/core/presets.py @@ -0,0 +1,59 @@ +""" +Built-in Task presets reproducing today's three prescriptive tools +(single-label, multi-label, keyword extraction) as starting points for the +Task Builder. Presets are fully editable -- opening one just pre-fills the +builder instead of starting from a blank task. + +Every preset expects a "text" column in the imported data. If the user's file +doesn't have a column literally named "text", they rename the {text} +placeholder in the prompt (or their column) -- the builder does no implicit +column mapping. +""" + +from __future__ import annotations + +from typing import Callable + +from core.task import OutputField, Task, TaskList + +PresetFactory = tuple[str, Callable[[], Task]] + + +def single_label_preset() -> Task: + return Task( + prompt=( + "Classify this text using exactly one label from the allowed set.\n" + "Allowed: {labels}\n" + "Text: {text}" + ), + lists={"labels": TaskList(values=[], format="comma")}, + output_fields=[OutputField(name="label", type="choice", list_ref="labels")], + ) + + +def multi_label_preset() -> Task: + return Task( + prompt=( + "Label this text with one or more labels from the allowed set only.\n" + "Allowed: {labels}\n" + "Text: {text}" + ), + lists={"labels": TaskList(values=[], format="comma")}, + output_fields=[OutputField(name="labels", type="multi_choice", list_ref="labels")], + ) + + +def keyword_extraction_preset() -> Task: + return Task( + prompt="Extract the keywords from this text.\nText: {text}", + system="You are an expert at structured data extraction.", + output_fields=[OutputField(name="keywords", type="text_list")], + ) + + +# key -> (display name, factory). Order matters for menu display. +PRESETS: dict[str, PresetFactory] = { + "single_label": ("Single-Label Classification", single_label_preset), + "multi_label": ("Multi-Label Classification", multi_label_preset), + "keyword_extraction": ("Keyword Extraction", keyword_extraction_preset), +} diff --git a/core/task.py b/core/task.py new file mode 100644 index 0000000..25e708e --- /dev/null +++ b/core/task.py @@ -0,0 +1,282 @@ +""" +Task engine: the single definition of "prompt + constrained output + column roles" +that replaces the six hardcoded classification/extraction pipelines +(single/multi/keyword x batch/live). + +A Task is: +- prompt / system: user-authored text with {column} and {list_name} placeholders. + Braces = substituted per row (columns) or per task (lists); anything else is + sent to the model verbatim. +- lists: named constant value sets, each with its own rendering format. A list + can be dropped into the prompt AND bound to an output field, so the allowed + values shown to the model and the values it's constrained to are always the + same list, edited in one place. +- output_fields: the shape of the structured response. This is what used to be + six separate hardcoded Pydantic models (LabeledQuote, LabeledQuoteMulti, + KeywordExtraction, and their *_live.py twins) -- now one dynamic model built + from a list of (name, type) pairs. +- carry_columns: input columns that ride into the output CSV untouched. This is + independent of the prompt -- a column can be carried without ever being used + as a {placeholder} (e.g. a pre-existing quote_id). +""" + +from __future__ import annotations + +import re +from dataclasses import dataclass, field +from typing import Any, Literal + +from pydantic import BaseModel, ConfigDict, Field, create_model + +from file_handling.data_conversion import make_str_enum + +PLACEHOLDER_RE = re.compile(r"\{([^{}]+)\}") + +ListFormat = Literal["comma", "hyphen", "space", "newline"] +FieldType = Literal["choice", "multi_choice", "text_list", "free_text", "number", "yes_no"] + +_CHOICE_TYPES = {"choice", "multi_choice"} + +DEFAULT_MODEL = "gpt-4o-mini" + +# ponytail: name heuristic, not an API capability lookup. Extend if OpenAI adds +# reasoning families -- upgrade to a models_registry capability field if this +# ever needs to be authoritative rather than a best guess. +_REASONING_PREFIXES = ("o1", "o3", "o4", "gpt-5") + + +def is_reasoning_model(model: str) -> bool: + """Whether `model` takes `reasoning.effort` instead of `temperature` on the + Responses API.""" + return model.startswith(_REASONING_PREFIXES) + + +class TaskValidationError(ValueError): + """Raised when a Task is internally inconsistent (bad placeholder, dangling + list reference, etc.) -- always before anything is sent to the API.""" + + +@dataclass +class TaskList: + """A named, constant value set (e.g. labels) with a prompt rendering format.""" + values: list[str] + format: ListFormat = "comma" + + def render(self) -> str: + if self.format == "comma": + return ", ".join(self.values) + if self.format == "hyphen": + return "\n".join(f"- {v}" for v in self.values) + if self.format == "space": + return " ".join(self.values) + if self.format == "newline": + return "\n".join(self.values) + raise TaskValidationError(f"Unknown list format: {self.format!r}") + + def to_dict(self) -> dict: + return {"values": list(self.values), "format": self.format} + + @classmethod + def from_dict(cls, data: dict) -> "TaskList": + return cls(values=list(data["values"]), format=data.get("format", "comma")) + + +@dataclass +class OutputField: + """One field of the structured output -- one column in the result CSV.""" + name: str + type: FieldType + list_ref: str | None = None # required for choice / multi_choice + + def to_dict(self) -> dict: + return {"name": self.name, "type": self.type, "list_ref": self.list_ref} + + @classmethod + def from_dict(cls, data: dict) -> "OutputField": + return cls(name=data["name"], type=data["type"], list_ref=data.get("list_ref")) + + +@dataclass +class Task: + prompt: str + system: str | None = None + lists: dict[str, TaskList] = field(default_factory=dict) + output_fields: list[OutputField] = field(default_factory=list) + carry_columns: list[str] = field(default_factory=list) + model: str = DEFAULT_MODEL + temperature: float | None = None # standard models only + reasoning_effort: str | None = None # "low" | "medium" | "high" -- reasoning models only + max_output_tokens: int | None = None + + def to_dict(self) -> dict: + """Serialize for saving as a reusable task file.""" + return { + "prompt": self.prompt, + "system": self.system, + "lists": {name: lst.to_dict() for name, lst in self.lists.items()}, + "output_fields": [f.to_dict() for f in self.output_fields], + "carry_columns": list(self.carry_columns), + "model": self.model, + "temperature": self.temperature, + "reasoning_effort": self.reasoning_effort, + "max_output_tokens": self.max_output_tokens, + } + + @classmethod + def from_dict(cls, data: dict) -> "Task": + return cls( + prompt=data["prompt"], + system=data.get("system"), + lists={name: TaskList.from_dict(v) for name, v in data.get("lists", {}).items()}, + output_fields=[OutputField.from_dict(f) for f in data.get("output_fields", [])], + carry_columns=list(data.get("carry_columns", [])), + model=data.get("model", DEFAULT_MODEL), + temperature=data.get("temperature"), + reasoning_effort=data.get("reasoning_effort"), + max_output_tokens=data.get("max_output_tokens"), + ) + + +def build_request_params(task: Task) -> dict: + """Model + inference params for the Responses API, applying per-model rules + (the API rejects `temperature` on reasoning models and `reasoning` on + standard ones).""" + params: dict = {"model": task.model} + if is_reasoning_model(task.model): + if task.reasoning_effort: + params["reasoning"] = {"effort": task.reasoning_effort} + elif task.temperature is not None: + params["temperature"] = task.temperature + if task.max_output_tokens is not None: + params["max_output_tokens"] = task.max_output_tokens + return params + + +def placeholders_in(text: str | None) -> set[str]: + """All {name} placeholders referenced in a piece of text.""" + return set(PLACEHOLDER_RE.findall(text or "")) + + +def validate_task(task: Task, columns: list[str]) -> None: + """ + Check a task against the set of available input columns. Raises + TaskValidationError with every problem found (not just the first) so the + UI can show a complete error list before submission. + """ + errors: list[str] = [] + known = set(columns) | set(task.lists) + + for ph in placeholders_in(task.prompt) | placeholders_in(task.system): + if ph not in known: + errors.append(f"Placeholder {{{ph}}} does not match any input column or list.") + + if not task.output_fields: + errors.append("Task must define at least one output field.") + + names = [f.name for f in task.output_fields] + if len(names) != len(set(names)): + errors.append("Output field names must be unique.") + + for f in task.output_fields: + if f.type in _CHOICE_TYPES: + if not f.list_ref: + errors.append(f"Output field '{f.name}' ({f.type}) must reference a list.") + elif f.list_ref not in task.lists: + errors.append(f"Output field '{f.name}' references unknown list '{f.list_ref}'.") + + for col in task.carry_columns: + if col not in columns: + errors.append(f"Carried column '{col}' is not in the input data.") + + if errors: + raise TaskValidationError("\n".join(errors)) + + +def render_prompt(task: Task, row: dict[str, Any], *, text: str | None = None) -> str: + """ + Substitute {column} (from row) and {list_name} (from task.lists, constant) + placeholders in `text` (defaults to task.prompt). Used both to build the + actual request and to power the UI's per-row prompt preview, so preview + and reality can never diverge. + """ + src = task.prompt if text is None else text + + def _sub(m: re.Match) -> str: + key = m.group(1) + if key in task.lists: + return task.lists[key].render() + if key in row: + return str(row[key]) + raise TaskValidationError(f"Unresolved placeholder: {{{key}}}") + + return PLACEHOLDER_RE.sub(_sub, src) + + +def build_output_model(task: Task) -> type[BaseModel]: + """ + Dynamically build one Pydantic model from task.output_fields -- this + replaces the six hardcoded LabeledQuote/LabeledQuoteMulti/KeywordExtraction + models. Choice/multi_choice fields become strict string enums bound to the + referenced list, so the model literally cannot emit an off-list value. + """ + if not task.output_fields: + raise TaskValidationError("Task must define at least one output field.") + + annotations: dict[str, tuple] = {} + for f in task.output_fields: + if f.type == "choice": + if f.list_ref not in task.lists: + raise TaskValidationError(f"Output field '{f.name}' references unknown list '{f.list_ref}'.") + enum = make_str_enum(f"{f.name}_Choice", task.lists[f.list_ref].values) + annotations[f.name] = (enum, ...) + elif f.type == "multi_choice": + if f.list_ref not in task.lists: + raise TaskValidationError(f"Output field '{f.name}' references unknown list '{f.list_ref}'.") + enum = make_str_enum(f"{f.name}_Choice", task.lists[f.list_ref].values) + annotations[f.name] = (list[enum], Field(..., min_length=1)) + elif f.type == "text_list": + annotations[f.name] = (list[str], Field(..., min_length=1)) + elif f.type == "free_text": + annotations[f.name] = (str, ...) + elif f.type == "number": + annotations[f.name] = (float, ...) + elif f.type == "yes_no": + annotations[f.name] = (bool, ...) + else: + raise TaskValidationError(f"Unknown output field type: {f.type!r}") + + return create_model( + "TaskOutput", + __config__=ConfigDict(use_enum_values=True, extra="forbid"), + **annotations, + ) + + +def forbid_additional_props(schema: dict) -> dict: + """Recursively set additionalProperties:false on every object schema. + Required for OpenAI Structured Outputs 'strict' mode.""" + if not isinstance(schema, dict): + return schema + + if schema.get("type") == "object": + schema.setdefault("properties", {}) + schema["additionalProperties"] = False + for prop_schema in schema.get("properties", {}).values(): + forbid_additional_props(prop_schema) + for prop_schema in schema.get("patternProperties", {}).values(): + forbid_additional_props(prop_schema) + + if schema.get("type") == "array" and "items" in schema: + forbid_additional_props(schema["items"]) + + for key in ("oneOf", "anyOf", "allOf"): + if key in schema and isinstance(schema[key], list): + for s in schema[key]: + forbid_additional_props(s) + + return schema + + +def build_strict_schema(task: Task) -> dict: + """The full json_schema payload for the OpenAI 'text.format' request field.""" + return forbid_additional_props(build_output_model(task).model_json_schema()) diff --git a/file_handling/data_import.py b/file_handling/data_import.py index 3651303..f763705 100644 --- a/file_handling/data_import.py +++ b/file_handling/data_import.py @@ -123,6 +123,32 @@ def _read_with(file, enc=None): raise RuntimeError(f"Failed to read delimited text: {e}") +def load_table(path: str, has_headers: bool = True) -> tuple[list[str], list[list[str]]]: + """ + Load an entire table (not a single column) for the Task Builder: every + column becomes available as a placeholder/carry candidate, rather than + forcing a separate import per column like the old single-column wizard. + + Returns (column_names, data_rows), reusing the same header-normalization + logic as the single-column import wizard (blank/missing headers become + "Column N"). + """ + rows = _load_tabular(path) + if not rows: + return [], [] + if has_headers: + raw_header, data = rows[0], rows[1:] + else: + max_cols = max((len(r) for r in rows), default=1) + raw_header, data = [f"Column {i + 1}" for i in range(max_cols)], rows + + header = [ + str(h) if (h is not None and str(h).strip() != "") else f"Column {i + 1}" + for i, h in enumerate(raw_header) + ] + return header, data + + def _load_tabular(path: str, max_rows: int | None = None) -> list[list[str]]: if _is_excel(path): return _read_excel(path, max_rows=max_rows) diff --git a/live_processing/keyword_extraction_live.py b/live_processing/keyword_extraction_live.py deleted file mode 100644 index 109681f..0000000 --- a/live_processing/keyword_extraction_live.py +++ /dev/null @@ -1,76 +0,0 @@ -""" -Live text classification processing using OpenAI API. - -This module provides real-time text classification functionality that processes -text snippets immediately using OpenAI's API, as opposed to batch processing. -This is useful for smaller datasets or when immediate results are needed. -""" - -from typing import Optional -import tkinter as tk -from tkinter import messagebox -from pydantic import BaseModel, ValidationError, Field, ConfigDict -from file_handling.data_import import import_data -from file_handling.data_conversion import save_as_csv, to_long_df -from settings import config -from batch_processing.batch_method import get_client - -# Progress UI lives in a separate module -from ui.progress_ui import ProgressController - - -def keyword_extraction_pipeline(parent: Optional[tk.Misc] = None): - """ - Prompt for quotes CSVs, extract keywords from each quote, - show progress, then save results to CSV. - """ - try: - client = get_client() - except Exception as e: - messagebox.showerror("API Key Required", str(e)) - return - - # Get quotes data - from_import = import_data(parent, "Select the quotes data") - if from_import is None: - return # user hit Cancel - quotes, quotes_nickname = from_import - - class KeywordExtraction(BaseModel): - id: int | None = None - quote: str - keywords: list[str] = Field(..., min_length=1) - model_config = ConfigDict() - - total = len(quotes) - - progress = ProgressController.open(parent=parent, total_count=total, title="Processing quotes…") - - results: list[KeywordExtraction] = [] - try: - for idx, q in enumerate(quotes, start=1): - try: - resp = client.responses.parse( - model=config.model, - input=[{"role": "system", "content": "You are an expert at structured data extraction."}, - {"role": "user", "content": f"Extract the keywords from this quote: {q}"}], - text_format=KeywordExtraction, - ) - decision = resp.output_parsed - row = KeywordExtraction( - id=idx, - quote=q, - **decision.model_dump(exclude={'id', 'quote'}) # <- prevents duplicate kwargs - ) - results.append(row) - except ValidationError as ve: - print(f"[VALIDATION ERROR] {str(q)[:60]}... -> {ve}") - except Exception as e: - print(f"[API ERROR] {str(q)[:60]}... -> {e}") - finally: - progress.update(idx, message=f"Processed {idx} of {total} quotes") - finally: - progress.close() - - df = to_long_df(results) - save_as_csv(df) \ No newline at end of file diff --git a/live_processing/multi_label_live.py b/live_processing/multi_label_live.py deleted file mode 100644 index e40b8f0..0000000 --- a/live_processing/multi_label_live.py +++ /dev/null @@ -1,88 +0,0 @@ -""" -Live text classification processing using OpenAI API. - -This module provides real-time text classification functionality that processes -text snippets immediately using OpenAI's API, as opposed to batch processing. -This is useful for smaller datasets or when immediate results are needed. -""" - -from typing import Optional, List -import tkinter as tk -from tkinter import messagebox -from pydantic import BaseModel, ValidationError, Field, ConfigDict -from file_handling.data_import import import_data -from file_handling.data_conversion import make_str_enum, save_as_csv, to_long_df -from settings import config -from batch_processing.batch_method import get_client - -# Progress UI lives in a separate module -from ui.progress_ui import ProgressController - - -def multi_label_pipeline(parent: Optional[tk.Misc] = None): - """ - Prompt for labels/quotes CSVs, classify each quote with 1+ labels, - show progress, then save results to CSV. - """ - try: - client = get_client() - except Exception as e: - messagebox.showerror("API Key Required", str(e)) - return - - # Get labels data - from_import = import_data(parent, "Select the labels data") - if from_import is None: - return # user hit Cancel - label_values, labels_nickname = from_import - labels = make_str_enum("Label", label_values) - - # Get quotes data - from_import = import_data(parent, "Select the quotes data") - if from_import is None: - return # user hit Cancel - quotes, quotes_nickname = from_import - - class LabeledQuoteMulti(BaseModel): - id: int | None = None - quote: str - label: List[labels] = Field(..., min_length=1) - model_config = ConfigDict(use_enum_values=True, extra='forbid') - - total = len(quotes) - - progress = ProgressController.open(parent=parent, total_count=total, title="Processing quotes…") - - results: list[LabeledQuoteMulti] = [] - try: - for idx, q in enumerate(quotes, start=1): - try: - resp = client.responses.parse( - model=config.model, - input=[{ - "role": "user", - "content": ( - "Label this quote with labels from the allowed set only. " - f"Allowed: {', '.join(labels)}\nQuote: {q}" - ), - }], - text_format=LabeledQuoteMulti, - ) - decision = resp.output_parsed - row = LabeledQuoteMulti( - id=idx, - quote=q, - **decision.model_dump(exclude={'id', 'quote'}) # <- prevents duplicate kwargs - ) - results.append(row) - except ValidationError as ve: - print(f"[VALIDATION ERROR] {str(q)[:60]}... -> {ve}") - except Exception as e: - print(f"[API ERROR] {str(q)[:60]}... -> {e}") - finally: - progress.update(idx, message=f"Processed {idx} of {total} quotes") - finally: - progress.close() - - df = to_long_df(results) - save_as_csv(df) diff --git a/live_processing/single_label_live.py b/live_processing/single_label_live.py deleted file mode 100644 index 896232c..0000000 --- a/live_processing/single_label_live.py +++ /dev/null @@ -1,89 +0,0 @@ -""" -Live text classification processing using OpenAI API. - -This module provides real-time text classification functionality that processes -text snippets immediately using OpenAI's API, as opposed to batch processing. -This is useful for smaller datasets or when immediate results are needed. -""" - -from typing import Optional -import tkinter as tk -from tkinter import messagebox -from pydantic import BaseModel, ValidationError, Field, ConfigDict -from file_handling.data_import import import_data -from file_handling.data_conversion import make_str_enum, save_as_csv, to_long_df -from settings import config -from batch_processing.batch_method import get_client - -# Progress UI lives in a separate module -from ui.progress_ui import ProgressController - - -def single_label_pipeline(parent: Optional[tk.Misc] = None): - """ - Prompt for labels/quotes CSVs, classify each quote with exactly one label, - show progress, then save results to CSV. - """ - try: - client = get_client() - except Exception as e: - messagebox.showerror("API Key Required", str(e)) - return - - # Get labels data - from_import = import_data(parent, "Select the labels data") - if from_import is None: - return # user hit Cancel - label_values, labels_nickname = from_import - labels = make_str_enum("Label", label_values) - - # Get quotes data - from_import = import_data(parent, "Select the quotes data") - if from_import is None: - return # user hit Cancel - quotes, quotes_nickname = from_import - - class LabeledQuote(BaseModel): - id: int | None = None - quote: str - label: labels # STRICT: must be one of labels - model_config = ConfigDict(use_enum_values=True, extra='forbid') - - - total = len(quotes) - - progress = ProgressController.open(parent=parent, total_count=total, title="Processing quotes…") - - results: list[LabeledQuote] = [] - try: - for idx, q in enumerate(quotes, start=1): - try: - resp = client.responses.parse( - model=config.model, - input=[{ - "role": "user", - "content": ( - f"Label this quote with exactly one label from the allowed set.\n" - f"Quote: {q}" - ), - }], - text_format=LabeledQuote, - ) - decision = resp.output_parsed - row = LabeledQuote( - id=idx, - quote=q, - **decision.model_dump(exclude={'id', 'quote'}) # <- prevents duplicate kwargs - ) - results.append(row) - except ValidationError as ve: - print(f"[VALIDATION ERROR] {str(q)[:60]}... -> {ve}") - except Exception as e: - print(f"[API ERROR] {str(q)[:60]}... -> {e}") - finally: - progress.update(idx, message=f"Processed {idx} of {total} quotes") - finally: - progress.close() - - df = to_long_df(results) - save_as_csv(df) \ No newline at end of file diff --git a/live_processing/task_live.py b/live_processing/task_live.py new file mode 100644 index 0000000..3644adc --- /dev/null +++ b/live_processing/task_live.py @@ -0,0 +1,69 @@ +""" +Live (synchronous, one-request-per-row) execution of a core.task.Task. + +Replaces single_label_live.py / multi_label_live.py / keyword_extraction_live.py +-- one loop now handles any task shape, including combinations none of the old +tools could do alone (e.g. a label AND keywords AND a free-text reason in the +same run). +""" + +from typing import Any, Optional + +import tkinter as tk +from tkinter import messagebox + +import pandas as pd +from pydantic import ValidationError + +from core.task import Task, build_output_model, build_request_params, render_prompt, validate_task +from file_handling.data_conversion import save_as_csv, to_long_df +from batch_processing.batch_method import get_client + +from ui.progress_ui import ProgressController + + +def run_task_live(parent: Optional[tk.Misc], task: Task, df: "pd.DataFrame") -> None: + """ + Run `task` against every row of `df` synchronously, showing progress, then + prompt to save the results (carried columns + output fields) as CSV. + """ + try: + client = get_client() + except Exception as e: + messagebox.showerror("API Key Required", str(e)) + return + + validate_task(task, list(df.columns)) + output_model = build_output_model(task) + request_params = build_request_params(task) + + rows = df.to_dict(orient="records") + total = len(rows) + progress = ProgressController.open(parent=parent, total_count=total, title="Processing rows…") + + results: list[dict[str, Any]] = [] + try: + for idx, row in enumerate(rows, start=1): + try: + messages = [] + if task.system: + messages.append({"role": "system", "content": render_prompt(task, row, text=task.system)}) + messages.append({"role": "user", "content": render_prompt(task, row)}) + + resp = client.responses.parse(input=messages, text_format=output_model, **request_params) + parsed = resp.output_parsed.model_dump() + + combined = {col: row.get(col) for col in task.carry_columns} + combined.update(parsed) + results.append(combined) + except ValidationError as ve: + print(f"[VALIDATION ERROR] row {idx} -> {ve}") + except Exception as e: + print(f"[API ERROR] row {idx} -> {e}") + finally: + progress.update(idx, message=f"Processed {idx} of {total} rows") + finally: + progress.close() + + result_df = to_long_df(results) + save_as_csv(result_df) diff --git a/settings/config.py b/settings/config.py index 621a840..157a20d 100644 --- a/settings/config.py +++ b/settings/config.py @@ -1,5 +1,4 @@ # Non-secret runtime settings -model = 'gpt-4o-mini' max_batches = 4 time_zone = 'US/Eastern' diff --git a/tests/test_batch_creation.py b/tests/test_batch_creation.py index 5fd0b11..c561b83 100644 --- a/tests/test_batch_creation.py +++ b/tests/test_batch_creation.py @@ -1,262 +1,127 @@ """ Tests for batch_processing/batch_creation.py -Covers: -- forbid_additional_props schema transformation -- generate_single_label_batch: JSONL output, prompt content, custom_id format, schema -- generate_multi_label_batch: JSONL output and array schema -- generate_keyword_extraction_batch: JSONL output -- Edge cases: empty input, non-ASCII text, punctuation-heavy text +The old single/multi/keyword generate_*_batch functions are gone -- one +generate_batch(task, rows) now covers every shape. Covers: +- JSONL structure: custom_id format, endpoint, model +- Prompt rendering (per-row columns + constant lists) reaches the request body +- System message only appears when the task defines one +- Strict schema (additionalProperties: false) is attached +- Edge cases: empty rows, non-ASCII, multiple output fields """ import io import json -import pytest +from batch_processing.batch_creation import generate_batch +from core.task import OutputField, Task, TaskList -from batch_processing.batch_creation import ( - forbid_additional_props, - generate_single_label_batch, - generate_multi_label_batch, - generate_keyword_extraction_batch, -) - - -# --------------------------------------------------------------------------- -# forbid_additional_props -# --------------------------------------------------------------------------- - -class TestForbidAdditionalProps: - def test_sets_flag_on_object(self): - schema = {"type": "object", "properties": {"x": {"type": "string"}}} - result = forbid_additional_props(schema) - assert result["additionalProperties"] is False - - def test_recursive_nested_object(self): - schema = { - "type": "object", - "properties": { - "inner": { - "type": "object", - "properties": {"y": {"type": "integer"}}, - } - }, - } - result = forbid_additional_props(schema) - assert result["additionalProperties"] is False - assert result["properties"]["inner"]["additionalProperties"] is False - - def test_array_items_object(self): - schema = { - "type": "array", - "items": { - "type": "object", - "properties": {"z": {"type": "string"}}, - }, - } - result = forbid_additional_props(schema) - assert result["items"]["additionalProperties"] is False - - def test_non_object_schema_unchanged(self): - schema = {"type": "string"} - result = forbid_additional_props(schema) - assert "additionalProperties" not in result - - def test_creates_empty_properties_on_object(self): - schema = {"type": "object"} - result = forbid_additional_props(schema) - assert result["additionalProperties"] is False - assert "properties" in result - - def test_does_not_modify_string_inside_array(self): - schema = {"type": "array", "items": {"type": "string"}} - result = forbid_additional_props(schema) - assert "additionalProperties" not in result["items"] - - def test_handles_non_dict_gracefully(self): - # Passing a non-dict should be returned unchanged - assert forbid_additional_props("string") == "string" - assert forbid_additional_props(42) == 42 - - -# --------------------------------------------------------------------------- -# Helpers -# --------------------------------------------------------------------------- def _parse_jsonl(buf: io.BytesIO) -> list[dict]: - """Read all lines from a BytesIO and parse them as JSON.""" buf.seek(0) return [json.loads(line) for line in buf.read().decode("utf-8").splitlines() if line.strip()] -# --------------------------------------------------------------------------- -# generate_single_label_batch -# --------------------------------------------------------------------------- +def _single_label_task(labels=("positive", "negative", "neutral")) -> Task: + return Task( + prompt="Label this quote with exactly one label from the allowed set.\nAllowed: {labels}\nQuote: {quote}", + lists={"labels": TaskList(list(labels), "comma")}, + output_fields=[OutputField(name="label", type="choice", list_ref="labels")], + ) -class TestGenerateSingleLabelBatch: - def test_returns_bytesio(self, sample_labels, sample_quotes): - result = generate_single_label_batch(sample_labels, sample_quotes) - assert isinstance(result, io.BytesIO) - def test_has_jsonl_name(self, sample_labels, sample_quotes): - result = generate_single_label_batch(sample_labels, sample_quotes) +class TestGenerateBatch: + def test_returns_bytesio_named_jsonl(self): + result = generate_batch(_single_label_task(), [{"quote": "hi"}]) + assert isinstance(result, io.BytesIO) assert result.name == "batchinput.jsonl" - def test_line_count_matches_quotes(self, sample_labels, sample_quotes): - result = generate_single_label_batch(sample_labels, sample_quotes) - lines = _parse_jsonl(result) - assert len(lines) == len(sample_quotes) - - def test_custom_id_format(self, sample_labels, sample_quotes): - result = generate_single_label_batch(sample_labels, sample_quotes) - lines = _parse_jsonl(result) - for i, line in enumerate(lines, start=1): - assert line["custom_id"] == f"quote-{i:05d}", f"Wrong custom_id at index {i}" - - def test_prompt_contains_all_labels(self, sample_labels, sample_quotes): - result = generate_single_label_batch(sample_labels, sample_quotes) - lines = _parse_jsonl(result) - for line in lines: - content = line["body"]["input"][0]["content"] - for label in sample_labels: - assert label in content, f"Label '{label}' missing from prompt" - - def test_prompt_contains_quote_text(self, sample_labels, sample_quotes): - result = generate_single_label_batch(sample_labels, sample_quotes) - lines = _parse_jsonl(result) - for line, quote in zip(lines, sample_quotes): - assert quote in line["body"]["input"][0]["content"] - - def test_schema_strict_flag(self, sample_labels, sample_quotes): - result = generate_single_label_batch(sample_labels, sample_quotes) - lines = _parse_jsonl(result) - for line in lines: - fmt = line["body"]["text"]["format"] - assert fmt["strict"] is True - - def test_schema_additional_props_false(self, sample_labels, sample_quotes): - result = generate_single_label_batch(sample_labels, sample_quotes) - lines = _parse_jsonl(result) - for line in lines: - schema = line["body"]["text"]["format"]["schema"] - assert schema.get("additionalProperties") is False - - def test_metadata_includes_quote(self, sample_labels, sample_quotes): - result = generate_single_label_batch(sample_labels, sample_quotes) - lines = _parse_jsonl(result) - for line, quote in zip(lines, sample_quotes): - assert line["body"]["metadata"]["quote"] == quote - - def test_empty_quotes_returns_empty_jsonl(self, sample_labels): - result = generate_single_label_batch(sample_labels, []) - lines = _parse_jsonl(result) - assert lines == [] - - def test_nonascii_text_round_trips(self, sample_labels): - quotes = ["Ünïcödé tëxt wïth spëcïäl châräctërs!", "日本語テキスト", "Héllo wörld"] - result = generate_single_label_batch(sample_labels, quotes) - lines = _parse_jsonl(result) + def test_line_count_matches_rows(self): + rows = [{"quote": "a"}, {"quote": "b"}, {"quote": "c"}] + lines = _parse_jsonl(generate_batch(_single_label_task(), rows)) assert len(lines) == 3 - assert "日本語テキスト" in lines[1]["body"]["input"][0]["content"] - - def test_punctuation_heavy_text(self, sample_labels): - quotes = ['He said, "It\'s great!" — really? (Yes!) #amazing @user $100'] - result = generate_single_label_batch(sample_labels, quotes) - lines = _parse_jsonl(result) - assert len(lines) == 1 - - def test_long_excerpt(self, sample_labels): - long_quote = "This is a very long excerpt. " * 100 - result = generate_single_label_batch(sample_labels, [long_quote]) - lines = _parse_jsonl(result) - assert len(lines) == 1 - - def test_short_single_word_excerpt(self, sample_labels): - result = generate_single_label_batch(sample_labels, ["great"]) - lines = _parse_jsonl(result) - assert len(lines) == 1 - - def test_url_endpoint_is_responses(self, sample_labels, sample_quotes): - result = generate_single_label_batch(sample_labels, sample_quotes) - lines = _parse_jsonl(result) - for line in lines: - assert line["url"] == "/v1/responses" - assert line["method"] == "POST" - - -# --------------------------------------------------------------------------- -# generate_multi_label_batch -# --------------------------------------------------------------------------- - -class TestGenerateMultiLabelBatch: - def test_line_count_matches_quotes(self, sample_labels, sample_quotes): - result = generate_multi_label_batch(sample_labels, sample_quotes) - lines = _parse_jsonl(result) - assert len(lines) == len(sample_quotes) - - def test_schema_label_field_is_array(self, sample_labels, sample_quotes): - result = generate_multi_label_batch(sample_labels, sample_quotes) - lines = _parse_jsonl(result) - for line in lines: - schema = line["body"]["text"]["format"]["schema"] - label_prop = schema["properties"]["label"] - assert label_prop["type"] == "array" - def test_prompt_contains_all_labels(self, sample_labels, sample_quotes): - result = generate_multi_label_batch(sample_labels, sample_quotes) - lines = _parse_jsonl(result) - for line in lines: - content = line["body"]["input"][0]["content"] - for label in sample_labels: - assert label in content - - def test_empty_quotes(self, sample_labels): - result = generate_multi_label_batch(sample_labels, []) - assert _parse_jsonl(result) == [] - - def test_nonascii_labels(self, sample_quotes): - labels = ["positivo", "négative", "中立"] - result = generate_multi_label_batch(labels, sample_quotes) - lines = _parse_jsonl(result) - assert len(lines) == len(sample_quotes) - for line in lines: - content = line["body"]["input"][0]["content"] - assert "中立" in content - - -# --------------------------------------------------------------------------- -# generate_keyword_extraction_batch -# --------------------------------------------------------------------------- - -class TestGenerateKeywordExtractionBatch: - def test_line_count_matches_texts(self, sample_quotes): - result = generate_keyword_extraction_batch(sample_quotes) - lines = _parse_jsonl(result) - assert len(lines) == len(sample_quotes) - - def test_custom_id_prefix_is_text(self, sample_quotes): - result = generate_keyword_extraction_batch(sample_quotes) - lines = _parse_jsonl(result) + def test_custom_id_format(self): + rows = [{"quote": f"q{i}"} for i in range(3)] + lines = _parse_jsonl(generate_batch(_single_label_task(), rows)) for i, line in enumerate(lines, start=1): - assert line["custom_id"] == f"text-{i:05d}" - - def test_schema_has_keywords_array(self, sample_quotes): - result = generate_keyword_extraction_batch(sample_quotes) - lines = _parse_jsonl(result) - for line in lines: - schema = line["body"]["text"]["format"]["schema"] - assert "keywords" in schema.get("properties", {}) - assert schema["properties"]["keywords"]["type"] == "array" - - def test_prompt_contains_text(self, sample_quotes): - result = generate_keyword_extraction_batch(sample_quotes) - lines = _parse_jsonl(result) - for line, text in zip(lines, sample_quotes): - user_msg = next( - m for m in line["body"]["input"] if m["role"] == "user" - ) - assert text in user_msg["content"] - - def test_empty_texts(self): - result = generate_keyword_extraction_batch([]) - assert _parse_jsonl(result) == [] + assert line["custom_id"] == f"row-{i:05d}" + + def test_prompt_contains_rendered_labels_and_quote(self): + rows = [{"quote": "great product"}] + lines = _parse_jsonl(generate_batch(_single_label_task(), rows)) + content = lines[0]["body"]["input"][-1]["content"] + assert "positive" in content and "negative" in content and "neutral" in content + assert "great product" in content + + def test_no_system_message_when_task_has_none(self): + lines = _parse_jsonl(generate_batch(_single_label_task(), [{"quote": "x"}])) + assert len(lines[0]["body"]["input"]) == 1 + assert lines[0]["body"]["input"][0]["role"] == "user" + + def test_system_message_present_when_task_defines_one(self): + task = Task( + prompt="Extract keywords.\nText: {text}", + system="You are an expert at structured data extraction.", + output_fields=[OutputField(name="keywords", type="text_list")], + ) + lines = _parse_jsonl(generate_batch(task, [{"text": "hello world"}])) + roles = [m["role"] for m in lines[0]["body"]["input"]] + assert roles == ["system", "user"] + assert lines[0]["body"]["input"][0]["content"] == "You are an expert at structured data extraction." + + def test_schema_strict_and_forbids_additional_props(self): + lines = _parse_jsonl(generate_batch(_single_label_task(), [{"quote": "x"}])) + fmt = lines[0]["body"]["text"]["format"] + assert fmt["strict"] is True + assert fmt["schema"]["additionalProperties"] is False + + def test_no_metadata_key_in_request_body(self): + """Row data is no longer round-tripped through OpenAI request metadata + (512-char limit risk) -- carried columns are joined locally instead.""" + lines = _parse_jsonl(generate_batch(_single_label_task(), [{"quote": "x"}])) + assert "metadata" not in lines[0]["body"] + + def test_empty_rows_returns_empty_jsonl(self): + assert _parse_jsonl(generate_batch(_single_label_task(), [])) == [] + + def test_nonascii_text_round_trips(self): + rows = [{"quote": "日本語テキスト"}] + lines = _parse_jsonl(generate_batch(_single_label_task(), rows)) + assert "日本語テキスト" in lines[0]["body"]["input"][0]["content"] + + def test_url_and_method(self): + lines = _parse_jsonl(generate_batch(_single_label_task(), [{"quote": "x"}])) + assert lines[0]["url"] == "/v1/responses" + assert lines[0]["method"] == "POST" + + def test_multiple_output_fields_in_schema(self): + task = Task( + prompt="{quote}", + lists={"labels": TaskList(["a", "b"])}, + output_fields=[ + OutputField(name="label", type="choice", list_ref="labels"), + OutputField(name="reason", type="free_text"), + ], + ) + lines = _parse_jsonl(generate_batch(task, [{"quote": "x"}])) + props = lines[0]["body"]["text"]["format"]["schema"]["properties"] + assert set(props) == {"label", "reason"} + + def test_extra_columns_not_referenced_in_prompt_are_ignored(self): + """A carry-only column (e.g. quote_id) exists in the row dict but + isn't a {placeholder} -- it must not appear in the prompt text.""" + rows = [{"quote": "hello", "quote_id": "R001"}] + lines = _parse_jsonl(generate_batch(_single_label_task(), rows)) + content = lines[0]["body"]["input"][0]["content"] + assert "R001" not in content + + def test_multiple_per_row_placeholders(self): + task = Task( + prompt="Q: {survey_q}\nA: {quote}", + output_fields=[OutputField(name="summary", type="free_text")], + ) + rows = [{"survey_q": "How was it?", "quote": "Great"}] + lines = _parse_jsonl(generate_batch(task, rows)) + content = lines[0]["body"]["input"][0]["content"] + assert content == "Q: How was it?\nA: Great" diff --git a/tests/test_batch_parsing.py b/tests/test_batch_parsing.py index 92d78fa..5ff5a7a 100644 --- a/tests/test_batch_parsing.py +++ b/tests/test_batch_parsing.py @@ -5,22 +5,49 @@ - _safe_parse_model_text: valid JSON, fenced JSON, truncated JSON, non-JSON, completely unparsable, empty string, whitespace - get_client: missing API key raises, valid key returns client -- get_batch_results: successful response, malformed rows, missing output file -- Authentication failures, rate limit errors, timeout/connection errors are - all handled via the same Exception path that get_batch_results delegates to - the caller (get_client raises, or the mock client raises). +- send_batch: validates the task before calling the API, writes a sidecar + for carried columns, skips the sidecar when nothing is carried +- get_batch_results: successful response, malformed rows, missing output + file, and rejoining carried columns from the local sidecar +- Authentication failures, rate limit errors, timeout/connection errors """ import json from unittest.mock import MagicMock +import pandas as pd import pytest from batch_processing.batch_method import ( + _read_sidecar, + _row_index_from_custom_id, _safe_parse_model_text, + _sidecar_path, + _task_type, get_batch_results, get_client, + rerun_batch, + send_batch, ) +from core.task import OutputField, Task, TaskList, TaskValidationError + + +@pytest.fixture(autouse=True) +def isolated_config_dir(tmp_path, monkeypatch): + """Redirect the sidecar directory to a temp dir (mirrors test_settings.py).""" + config_dir = tmp_path / "CodebookAI" + config_dir.mkdir() + monkeypatch.setattr("batch_processing.batch_method.get_user_config_dir", lambda: config_dir) + yield config_dir + + +def _single_label_task(carry_columns=()) -> Task: + return Task( + prompt="Allowed: {labels}\nQuote: {quote}", + lists={"labels": TaskList(["positive", "negative", "neutral"])}, + output_fields=[OutputField(name="label", type="choice", list_ref="labels")], + carry_columns=list(carry_columns), + ) # --------------------------------------------------------------------------- @@ -129,19 +156,205 @@ def test_returns_openai_client_when_key_present(self, mocker): assert client is mock_openai_cls.return_value +# --------------------------------------------------------------------------- +# _row_index_from_custom_id +# --------------------------------------------------------------------------- + +class TestRowIndexFromCustomId: + def test_first_row(self): + assert _row_index_from_custom_id("row-00001") == 0 + + def test_later_row(self): + assert _row_index_from_custom_id("row-00042") == 41 + + def test_none_returns_none(self): + assert _row_index_from_custom_id(None) is None + + def test_malformed_returns_none(self): + assert _row_index_from_custom_id("not-a-row-id") is None + + +# --------------------------------------------------------------------------- +# send_batch +# --------------------------------------------------------------------------- + +class TestSendBatch: + def _mock_client(self, mocker): + mock_client = MagicMock() + mocker.patch("batch_processing.batch_method.get_client", return_value=mock_client) + mock_client.files.create.return_value.id = "file-abc" + mock_client.batches.create.return_value.id = "batch-abc" + return mock_client + + def test_invalid_task_raises_before_calling_api(self, mocker): + mock_client = self._mock_client(mocker) + task = Task(prompt="{ghost}", output_fields=[]) # unknown placeholder, no output fields + df = pd.DataFrame({"quote": ["hi"]}) + with pytest.raises(TaskValidationError): + send_batch(task, df) + mock_client.files.create.assert_not_called() + + def test_valid_task_uploads_and_creates_batch(self, mocker): + mock_client = self._mock_client(mocker) + df = pd.DataFrame({"quote": ["a", "b"]}) + batch = send_batch(_single_label_task(), df) + mock_client.files.create.assert_called_once() + mock_client.batches.create.assert_called_once() + assert batch.id == "batch-abc" + + def test_carried_columns_written_to_sidecar(self, mocker, isolated_config_dir): + self._mock_client(mocker) + df = pd.DataFrame({"quote": ["a", "b"], "quote_id": ["R001", "R002"]}) + send_batch(_single_label_task(carry_columns=["quote_id"]), df) + + sidecar = _read_sidecar("batch-abc") + assert sidecar is not None + assert list(sidecar["quote_id"]) == ["R001", "R002"] + + def test_no_carried_columns_skips_sidecar(self, mocker, isolated_config_dir): + self._mock_client(mocker) + df = pd.DataFrame({"quote": ["a"]}) + send_batch(_single_label_task(carry_columns=[]), df) + assert not _sidecar_path("batch-abc").exists() + + def test_metadata_includes_type_and_dataset(self, mocker): + mock_client = self._mock_client(mocker) + df = pd.DataFrame({"quote": ["a", "b"]}) + send_batch(_single_label_task(), df, dataset="reviews") + + md = mock_client.batches.create.call_args.kwargs["metadata"] + assert md["type"] == "classification" + assert md["dataset"] == "reviews" + assert md["rows"] == "2" + + def test_metadata_dataset_blank_when_not_given(self, mocker): + mock_client = self._mock_client(mocker) + df = pd.DataFrame({"quote": ["a"]}) + send_batch(_single_label_task(), df) + + md = mock_client.batches.create.call_args.kwargs["metadata"] + assert md["dataset"] == "" + + def test_metadata_standard_model_has_temperature_not_reasoning(self, mocker): + mock_client = self._mock_client(mocker) + df = pd.DataFrame({"quote": ["a"]}) + task = _single_label_task() + task.model = "gpt-4o-mini" + task.temperature = 0.5 + task.reasoning_effort = "high" # ignored: not a reasoning model + send_batch(task, df) + + md = mock_client.batches.create.call_args.kwargs["metadata"] + assert md["temperature"] == "0.5" + assert "reasoning" not in md + + def test_metadata_reasoning_model_has_reasoning_not_temperature(self, mocker): + mock_client = self._mock_client(mocker) + df = pd.DataFrame({"quote": ["a"]}) + task = _single_label_task() + task.model = "o3" + task.temperature = 0.5 # ignored: reasoning models don't take temperature + task.reasoning_effort = "high" + send_batch(task, df) + + md = mock_client.batches.create.call_args.kwargs["metadata"] + assert md["reasoning"] == "high" + assert "temperature" not in md + + +# --------------------------------------------------------------------------- +# _task_type +# --------------------------------------------------------------------------- + +class TestTaskType: + def _task_with(self, field_type: str) -> Task: + list_ref = "labels" if field_type in ("choice", "multi_choice") else None + return Task( + prompt="x", + lists={"labels": TaskList(["a", "b"])} if list_ref else {}, + output_fields=[OutputField(name="out", type=field_type, list_ref=list_ref)], + ) + + def test_multi_choice_is_multi_label(self): + assert _task_type(self._task_with("multi_choice")) == "multi-label" + + def test_choice_is_classification(self): + assert _task_type(self._task_with("choice")) == "classification" + + def test_text_list_is_keyword_extraction(self): + assert _task_type(self._task_with("text_list")) == "keyword extraction" + + def test_free_text_is_free_text(self): + assert _task_type(self._task_with("free_text")) == "free-text" + + +# --------------------------------------------------------------------------- +# rerun_batch +# --------------------------------------------------------------------------- + +class TestRerunBatch: + def _mock_client_for_rerun(self, mocker): + mock_client = MagicMock() + mocker.patch("batch_processing.batch_method.get_client", return_value=mock_client) + orig = MagicMock() + orig.input_file_id = "file-orig" + orig.endpoint = "/v1/responses" + orig.completion_window = "24h" + orig.metadata = {"model": "gpt-4o-mini", "type": "classification"} + mock_client.batches.retrieve.return_value = orig + mock_client.batches.create.side_effect = [ + MagicMock(id="batch-rerun-1"), MagicMock(id="batch-rerun-2"), + ] + return mock_client + + def test_reruns_reuse_input_file_and_metadata(self, mocker): + mock_client = self._mock_client_for_rerun(mocker) + new_ids = rerun_batch("batch-orig", count=2) + + assert new_ids == ["batch-rerun-1", "batch-rerun-2"] + assert mock_client.batches.create.call_count == 2 + for call in mock_client.batches.create.call_args_list: + assert call.kwargs["input_file_id"] == "file-orig" + assert call.kwargs["metadata"] == {"model": "gpt-4o-mini", "type": "classification"} + + def test_default_count_is_one(self, mocker): + mock_client = self._mock_client_for_rerun(mocker) + new_ids = rerun_batch("batch-orig") + assert new_ids == ["batch-rerun-1"] + + def test_sidecar_copied_to_each_new_batch(self, mocker, isolated_config_dir): + self._mock_client_for_rerun(mocker) + df = pd.DataFrame({"quote_id": ["R001", "R002"]}) + df.to_csv(_sidecar_path("batch-orig"), index=False) + + rerun_batch("batch-orig", count=2) + + for new_id in ("batch-rerun-1", "batch-rerun-2"): + sidecar = _read_sidecar(new_id) + assert sidecar is not None + assert list(sidecar["quote_id"]) == ["R001", "R002"] + + def test_no_sidecar_is_fine(self, mocker, isolated_config_dir): + self._mock_client_for_rerun(mocker) + new_ids = rerun_batch("batch-orig", count=1) + assert new_ids == ["batch-rerun-1"] + assert _read_sidecar("batch-rerun-1") is None + + # --------------------------------------------------------------------------- # get_batch_results – mocked OpenAI client # --------------------------------------------------------------------------- -def _make_response_line(custom_id: str, text_output: str, quote: str = "") -> str: - """Build a single JSONL line matching the OpenAI batch output format.""" +def _make_response_line(custom_id: str, text_output: str) -> str: + """Build a single JSONL line matching the OpenAI batch output format. + Request metadata is no longer echoed (dropped in favor of the local + sidecar), so the response body carries no 'metadata' key.""" return json.dumps( { "custom_id": custom_id, "response": { "body": { "output": [{"content": [{"text": text_output}]}], - "metadata": {"quote": quote}, } }, } @@ -161,9 +374,7 @@ def _setup_mock_client(self, mocker, output_file_id, file_content_bytes): return mock_client, mock_status def test_successful_single_row(self, mocker): - line = _make_response_line( - "quote-00001", '{"label": "positive"}', quote="Great product!" - ) + line = _make_response_line("row-00001", '{"label": "positive"}') mock_client, _ = self._setup_mock_client( mocker, "file-123", (line + "\n").encode("utf-8") ) @@ -179,9 +390,9 @@ def test_successful_single_row(self, mocker): def test_multiple_rows_all_parsed(self, mocker): lines = "\n".join( [ - _make_response_line("quote-00001", '{"label": "positive"}', "Good."), - _make_response_line("quote-00002", '{"label": "negative"}', "Bad."), - _make_response_line("quote-00003", '{"label": "neutral"}', "OK."), + _make_response_line("row-00001", '{"label": "positive"}'), + _make_response_line("row-00002", '{"label": "negative"}'), + _make_response_line("row-00003", '{"label": "neutral"}'), ] ) self._setup_mock_client(mocker, "file-234", (lines + "\n").encode("utf-8")) @@ -195,9 +406,7 @@ def test_multiple_rows_all_parsed(self, mocker): def test_malformed_row_excluded_from_results(self, mocker): """A row whose text cannot be parsed should be skipped (bad_rows), not crash.""" - bad_line = _make_response_line( - "quote-00001", "COMPLETELY INVALID OUTPUT", "Some quote." - ) + bad_line = _make_response_line("row-00001", "COMPLETELY INVALID OUTPUT") self._setup_mock_client( mocker, "file-345", (bad_line + "\n").encode("utf-8") ) @@ -212,9 +421,9 @@ def test_mixed_good_and_bad_rows(self, mocker): """Only parsable rows end up in the saved DataFrame.""" lines = "\n".join( [ - _make_response_line("quote-00001", '{"label": "positive"}', "Good."), - _make_response_line("quote-00002", "NOT JSON AT ALL", "Bad."), - _make_response_line("quote-00003", '{"label": "neutral"}', "OK."), + _make_response_line("row-00001", '{"label": "positive"}'), + _make_response_line("row-00002", "NOT JSON AT ALL"), + _make_response_line("row-00003", '{"label": "neutral"}'), ] ) self._setup_mock_client(mocker, "file-456", (lines + "\n").encode("utf-8")) @@ -228,9 +437,7 @@ def test_mixed_good_and_bad_rows(self, mocker): def test_repaired_truncated_row_gets_repair_note(self, mocker): """A repaired row should have a repair_note column in the output.""" - line = _make_response_line( - "quote-00001", '{"label":"positive', "Truncated." - ) + line = _make_response_line("row-00001", '{"label":"positive') self._setup_mock_client( mocker, "file-567", (line + "\n").encode("utf-8") ) @@ -296,10 +503,7 @@ def test_connection_error_propagates(self, mocker): def test_unexpected_response_schema_row_excluded(self, mocker): """A row with an unexpected but valid JSON schema still goes through _safe_parse.""" - # Valid JSON but not the expected schema – save_as_csv still called - line = _make_response_line( - "quote-00001", '{"unexpected_field": "value"}', "Quote." - ) + line = _make_response_line("row-00001", '{"unexpected_field": "value"}') self._setup_mock_client( mocker, "file-schema", (line + "\n").encode("utf-8") ) @@ -309,16 +513,11 @@ def test_unexpected_response_schema_row_excluded(self, mocker): mock_save.assert_called_once() df = mock_save.call_args[0][0] - # unexpected_field was parsed and included assert "unexpected_field" in df.columns def test_fenced_json_in_response(self, mocker): """A response with ```json fences is correctly parsed.""" - line = _make_response_line( - "quote-00001", - "```json\n{\"label\": \"positive\"}\n```", - "Good.", - ) + line = _make_response_line("row-00001", "```json\n{\"label\": \"positive\"}\n```") self._setup_mock_client( mocker, "file-fenced", (line + "\n").encode("utf-8") ) @@ -338,3 +537,37 @@ def test_empty_file_content_saves_empty_df(self, mocker): df = mock_save.call_args[0][0] assert len(df) == 0 + + def test_carried_columns_rejoined_from_sidecar(self, mocker, isolated_config_dir, tmp_path): + """Carried columns written by send_batch are rejoined by row position + when results come back -- this is the quote_id pass-through case.""" + from batch_processing.batch_method import _sidecar_dir + sidecar_df = pd.DataFrame({"quote_id": ["R001", "R002"]}) + sidecar_df.to_csv(_sidecar_dir() / "batch-sidecar.csv", index=False) + + lines = "\n".join( + [ + _make_response_line("row-00001", '{"label": "positive"}'), + _make_response_line("row-00002", '{"label": "negative"}'), + ] + ) + self._setup_mock_client(mocker, "file-sidecar", (lines + "\n").encode("utf-8")) + mock_save = mocker.patch("batch_processing.batch_method.save_as_csv") + + get_batch_results("batch-sidecar") + + df = mock_save.call_args[0][0] + assert list(df["quote_id"]) == ["R001", "R002"] + assert list(df["label"]) == ["positive", "negative"] + + def test_no_sidecar_file_still_produces_results(self, mocker): + """If no sidecar was ever written (no carried columns), results + still parse fine -- carried columns are simply absent.""" + line = _make_response_line("row-00001", '{"label": "positive"}') + self._setup_mock_client(mocker, "file-nosidecar", (line + "\n").encode("utf-8")) + mock_save = mocker.patch("batch_processing.batch_method.save_as_csv") + + get_batch_results("batch-nosidecar-xyz") + + df = mock_save.call_args[0][0] + assert df["label"].iloc[0] == "positive" diff --git a/tests/test_data_import.py b/tests/test_data_import.py index 97b64ed..667c490 100644 --- a/tests/test_data_import.py +++ b/tests/test_data_import.py @@ -24,6 +24,7 @@ _load_tabular, _read_text_table, _sniff_delimiter, + load_table, ) @@ -175,3 +176,38 @@ def test_label_fixture_single_column(self): assert "positive" in values assert "negative" in values assert "neutral" in values + + +# --------------------------------------------------------------------------- +# load_table -- full-table import for the Task Builder (all columns, not one) +# --------------------------------------------------------------------------- + +class TestLoadTable: + def test_returns_header_and_data_rows(self, tmp_csv): + path = tmp_csv("text,id,emotion\nhello,R1,joy\nbye,R2,sadness\n") + header, data = load_table(str(path)) + assert header == ["text", "id", "emotion"] + assert data == [["hello", "R1", "joy"], ["bye", "R2", "sadness"]] + + def test_no_headers_generates_column_names(self, tmp_csv): + path = tmp_csv("hello,R1\nbye,R2\n") + header, data = load_table(str(path), has_headers=False) + assert header == ["Column 1", "Column 2"] + assert data == [["hello", "R1"], ["bye", "R2"]] + + def test_blank_header_cell_gets_placeholder_name(self, tmp_csv): + path = tmp_csv("text,,emotion\nhello,R1,joy\n") + header, _ = load_table(str(path)) + assert header == ["text", "Column 2", "emotion"] + + def test_empty_file_returns_empty(self, tmp_csv): + path = tmp_csv("") + header, data = load_table(str(path)) + assert header == [] + assert data == [] + + def test_header_only_file_returns_no_data_rows(self, tmp_csv): + path = tmp_csv("text,id\n") + header, data = load_table(str(path)) + assert header == ["text", "id"] + assert data == [] diff --git a/tests/test_presets.py b/tests/test_presets.py new file mode 100644 index 0000000..073efb5 --- /dev/null +++ b/tests/test_presets.py @@ -0,0 +1,56 @@ +""" +Tests for core/presets.py -- the three built-in Task presets that replace the +old hardcoded single/multi/keyword tools as editable starting points. +""" + +import pytest + +from core.presets import PRESETS +from core.task import Task, TaskValidationError, build_output_model, validate_task + + +class TestPresets: + @pytest.mark.parametrize("key", list(PRESETS)) + def test_preset_is_a_task(self, key): + _, factory = PRESETS[key] + assert isinstance(factory(), Task) + + @pytest.mark.parametrize("key", list(PRESETS)) + def test_preset_valid_once_lists_populated(self, key): + """Presets ship with empty label lists (the user fills these in) -- + once populated with any values, the task must validate against a + 'text' column and build a real output model.""" + _, factory = PRESETS[key] + task = factory() + for lst in task.lists.values(): + lst.values = ["a", "b"] + validate_task(task, columns=["text"]) + build_output_model(task) # must not raise + + @pytest.mark.parametrize("key", list(PRESETS)) + def test_preset_prompt_references_text_column(self, key): + _, factory = PRESETS[key] + assert "{text}" in factory().prompt + + def test_single_label_preset_is_choice(self): + task = PRESETS["single_label"][1]() + assert task.output_fields[0].type == "choice" + assert task.output_fields[0].list_ref == "labels" + + def test_multi_label_preset_is_multi_choice(self): + task = PRESETS["multi_label"][1]() + assert task.output_fields[0].type == "multi_choice" + + def test_keyword_preset_has_no_list_dependency(self): + task = PRESETS["keyword_extraction"][1]() + assert task.lists == {} + assert task.output_fields[0].type == "text_list" + + def test_empty_labels_list_accepts_no_value(self): + """An unpopulated preset label list builds a model (structurally + valid) but with zero allowed enum values -- so it can never actually + be satisfied, which is exactly the nudge to fill labels in first.""" + task = PRESETS["single_label"][1]() + model = build_output_model(task) + with pytest.raises(Exception): + model(label="anything") diff --git a/tests/test_settings.py b/tests/test_settings.py index 3666dc1..6a86ab1 100644 --- a/tests/test_settings.py +++ b/tests/test_settings.py @@ -109,13 +109,13 @@ def test_empty_dict_saved(self): class TestGetSetting: def test_returns_default_config_attribute(self): - # 'model' is defined in settings/config.py as 'gpt-4o-mini' - result = get_setting("model") - assert result == "gpt-4o-mini" + # 'time_zone' is defined in settings/config.py as 'US/Eastern' + result = get_setting("time_zone") + assert result == "US/Eastern" def test_user_setting_overrides_config(self): - save_user_settings({"model": "gpt-5-override"}) - assert get_setting("model") == "gpt-5-override" + save_user_settings({"time_zone": "UTC"}) + assert get_setting("time_zone") == "UTC" def test_returns_provided_default_when_key_missing(self): result = get_setting("nonexistent_key_xyz", default="my_default") diff --git a/tests/test_task.py b/tests/test_task.py new file mode 100644 index 0000000..d88421f --- /dev/null +++ b/tests/test_task.py @@ -0,0 +1,445 @@ +""" +Tests for core/task.py -- the shared engine that replaced the six hardcoded +single/multi/keyword x batch/live pipelines. + +Covers: +- TaskList rendering in all four formats +- build_output_model / build_strict_schema for every output field type +- render_prompt substitution of columns and lists +- validate_task catching unknown placeholders and dangling list refs +- Task.to_dict / from_dict round-trip (task file save/load) +""" + +import pytest + +from core.task import ( + DEFAULT_MODEL, + OutputField, + Task, + TaskList, + TaskValidationError, + build_output_model, + build_request_params, + build_strict_schema, + forbid_additional_props, + is_reasoning_model, + render_prompt, + validate_task, +) + + +# --------------------------------------------------------------------------- +# forbid_additional_props (moved here from batch_processing/batch_creation.py) +# --------------------------------------------------------------------------- + +class TestForbidAdditionalProps: + def test_sets_flag_on_object(self): + schema = {"type": "object", "properties": {"x": {"type": "string"}}} + result = forbid_additional_props(schema) + assert result["additionalProperties"] is False + + def test_recursive_nested_object(self): + schema = { + "type": "object", + "properties": { + "inner": { + "type": "object", + "properties": {"y": {"type": "integer"}}, + } + }, + } + result = forbid_additional_props(schema) + assert result["additionalProperties"] is False + assert result["properties"]["inner"]["additionalProperties"] is False + + def test_array_items_object(self): + schema = { + "type": "array", + "items": { + "type": "object", + "properties": {"z": {"type": "string"}}, + }, + } + result = forbid_additional_props(schema) + assert result["items"]["additionalProperties"] is False + + def test_non_object_schema_unchanged(self): + schema = {"type": "string"} + result = forbid_additional_props(schema) + assert "additionalProperties" not in result + + def test_creates_empty_properties_on_object(self): + schema = {"type": "object"} + result = forbid_additional_props(schema) + assert result["additionalProperties"] is False + assert "properties" in result + + def test_does_not_modify_string_inside_array(self): + schema = {"type": "array", "items": {"type": "string"}} + result = forbid_additional_props(schema) + assert "additionalProperties" not in result["items"] + + def test_handles_non_dict_gracefully(self): + assert forbid_additional_props("string") == "string" + assert forbid_additional_props(42) == 42 + + +# --------------------------------------------------------------------------- +# TaskList rendering +# --------------------------------------------------------------------------- + +class TestTaskListRender: + def test_comma(self): + assert TaskList(["a", "b", "c"], "comma").render() == "a, b, c" + + def test_hyphen(self): + assert TaskList(["a", "b"], "hyphen").render() == "- a\n- b" + + def test_space(self): + assert TaskList(["a", "b", "c"], "space").render() == "a b c" + + def test_newline(self): + assert TaskList(["a", "b"], "newline").render() == "a\nb" + + def test_default_format_is_comma(self): + assert TaskList(["a", "b"]).render() == "a, b" + + +# --------------------------------------------------------------------------- +# render_prompt +# --------------------------------------------------------------------------- + +class TestRenderPrompt: + def test_substitutes_column(self): + task = Task(prompt="Quote: {quote}") + assert render_prompt(task, {"quote": "hello"}) == "Quote: hello" + + def test_substitutes_list_constant_across_rows(self): + task = Task( + prompt="Allowed: {labels}\nQuote: {quote}", + lists={"labels": TaskList(["pos", "neg"], "comma")}, + ) + out1 = render_prompt(task, {"quote": "a"}) + out2 = render_prompt(task, {"quote": "b"}) + assert "Allowed: pos, neg" in out1 + assert "Allowed: pos, neg" in out2 + assert "Quote: a" in out1 + assert "Quote: b" in out2 + + def test_multiple_per_row_columns(self): + task = Task(prompt="Q: {survey_q}\nText: {quote}") + rendered = render_prompt(task, {"survey_q": "How was it?", "quote": "Great"}) + assert rendered == "Q: How was it?\nText: Great" + + def test_literal_text_outside_braces_is_constant(self): + task = Task(prompt="Classify strictly. Text: {quote}") + assert render_prompt(task, {"quote": "x"}).startswith("Classify strictly. Text:") + + def test_renders_system_text_explicitly(self): + task = Task(prompt="{quote}", system="You are grading {quote}.") + rendered = render_prompt(task, {"quote": "hi"}, text=task.system) + assert rendered == "You are grading hi." + + def test_unresolved_placeholder_raises(self): + task = Task(prompt="{nonexistent}") + with pytest.raises(TaskValidationError): + render_prompt(task, {"quote": "x"}) + + def test_list_format_applied_in_prompt(self): + task = Task( + prompt="Labels:\n{labels}", + lists={"labels": TaskList(["pos", "neg"], "hyphen")}, + ) + assert render_prompt(task, {}) == "Labels:\n- pos\n- neg" + + +# --------------------------------------------------------------------------- +# validate_task +# --------------------------------------------------------------------------- + +class TestValidateTask: + def _valid_task(self) -> Task: + return Task( + prompt="Allowed: {labels}\nQuote: {quote}", + lists={"labels": TaskList(["pos", "neg"])}, + output_fields=[OutputField(name="label", type="choice", list_ref="labels")], + carry_columns=["quote_id"], + ) + + def test_valid_task_passes(self): + validate_task(self._valid_task(), columns=["quote", "quote_id"]) + + def test_unknown_placeholder_column_raises(self): + task = Task( + prompt="{does_not_exist}", + output_fields=[OutputField(name="kw", type="text_list")], + ) + with pytest.raises(TaskValidationError, match="does_not_exist"): + validate_task(task, columns=["quote"]) + + def test_no_output_fields_raises(self): + task = Task(prompt="{quote}") + with pytest.raises(TaskValidationError, match="output field"): + validate_task(task, columns=["quote"]) + + def test_duplicate_output_field_names_raises(self): + task = Task( + prompt="{quote}", + output_fields=[ + OutputField(name="label", type="free_text"), + OutputField(name="label", type="number"), + ], + ) + with pytest.raises(TaskValidationError, match="unique"): + validate_task(task, columns=["quote"]) + + def test_choice_field_missing_list_ref_raises(self): + task = Task( + prompt="{quote}", + output_fields=[OutputField(name="label", type="choice", list_ref=None)], + ) + with pytest.raises(TaskValidationError, match="reference a list"): + validate_task(task, columns=["quote"]) + + def test_choice_field_dangling_list_ref_raises(self): + task = Task( + prompt="{quote}", + output_fields=[OutputField(name="label", type="choice", list_ref="ghost")], + ) + with pytest.raises(TaskValidationError, match="ghost"): + validate_task(task, columns=["quote"]) + + def test_carried_column_not_in_input_raises(self): + task = Task( + prompt="{quote}", + output_fields=[OutputField(name="kw", type="text_list")], + carry_columns=["missing_id"], + ) + with pytest.raises(TaskValidationError, match="missing_id"): + validate_task(task, columns=["quote"]) + + def test_carry_only_column_not_in_prompt_is_fine(self): + # quote_id is carried to output but never referenced in the prompt -- + # this must be allowed (the whole point of separating the two roles). + task = Task( + prompt="Quote: {quote}", + output_fields=[OutputField(name="kw", type="text_list")], + carry_columns=["quote_id"], + ) + validate_task(task, columns=["quote", "quote_id"]) + + def test_reports_multiple_errors_at_once(self): + task = Task(prompt="{ghost}", output_fields=[]) + with pytest.raises(TaskValidationError) as exc_info: + validate_task(task, columns=["quote"]) + msg = str(exc_info.value) + assert "ghost" in msg + assert "output field" in msg + + +# --------------------------------------------------------------------------- +# build_output_model / build_strict_schema -- one per field type +# --------------------------------------------------------------------------- + +class TestBuildOutputModel: + def test_choice_field_is_enum_constrained(self): + task = Task( + prompt="{quote}", + lists={"labels": TaskList(["pos", "neg", "neutral"])}, + output_fields=[OutputField(name="label", type="choice", list_ref="labels")], + ) + model = build_output_model(task) + inst = model(label="pos") + assert inst.label == "pos" + with pytest.raises(Exception): + model(label="not_a_label") + + def test_multi_choice_field_is_list_of_enum(self): + task = Task( + prompt="{quote}", + lists={"labels": TaskList(["a", "b", "c"])}, + output_fields=[OutputField(name="labels_out", type="multi_choice", list_ref="labels")], + ) + model = build_output_model(task) + inst = model(labels_out=["a", "b"]) + assert inst.labels_out == ["a", "b"] + with pytest.raises(Exception): + model(labels_out=[]) # min_length=1 + with pytest.raises(Exception): + model(labels_out=["not_valid"]) + + def test_text_list_field(self): + task = Task(prompt="{quote}", output_fields=[OutputField(name="keywords", type="text_list")]) + model = build_output_model(task) + inst = model(keywords=["a", "b"]) + assert inst.keywords == ["a", "b"] + with pytest.raises(Exception): + model(keywords=[]) + + def test_free_text_field(self): + task = Task(prompt="{quote}", output_fields=[OutputField(name="summary", type="free_text")]) + model = build_output_model(task) + assert model(summary="a reason").summary == "a reason" + + def test_number_field(self): + task = Task(prompt="{quote}", output_fields=[OutputField(name="score", type="number")]) + model = build_output_model(task) + assert model(score=4.5).score == 4.5 + + def test_yes_no_field(self): + task = Task(prompt="{quote}", output_fields=[OutputField(name="relevant", type="yes_no")]) + model = build_output_model(task) + assert model(relevant=True).relevant is True + + def test_multiple_output_fields_combined(self): + task = Task( + prompt="{quote}", + lists={"labels": TaskList(["pos", "neg"])}, + output_fields=[ + OutputField(name="label", type="choice", list_ref="labels"), + OutputField(name="keywords", type="text_list"), + OutputField(name="reason", type="free_text"), + ], + ) + model = build_output_model(task) + inst = model(label="pos", keywords=["k1"], reason="because") + assert (inst.label, inst.keywords, inst.reason) == ("pos", ["k1"], "because") + + def test_no_output_fields_raises(self): + task = Task(prompt="{quote}") + with pytest.raises(TaskValidationError): + build_output_model(task) + + def test_extra_fields_forbidden(self): + task = Task(prompt="{quote}", output_fields=[OutputField(name="summary", type="free_text")]) + model = build_output_model(task) + with pytest.raises(Exception): + model(summary="x", unexpected="y") + + +class TestBuildStrictSchema: + def test_schema_forbids_additional_props(self): + task = Task(prompt="{quote}", output_fields=[OutputField(name="summary", type="free_text")]) + schema = build_strict_schema(task) + assert schema["additionalProperties"] is False + + def test_schema_has_field_names(self): + task = Task( + prompt="{quote}", + lists={"labels": TaskList(["pos", "neg"])}, + output_fields=[ + OutputField(name="label", type="choice", list_ref="labels"), + OutputField(name="score", type="number"), + ], + ) + schema = build_strict_schema(task) + assert set(schema["properties"]) == {"label", "score"} + + +# --------------------------------------------------------------------------- +# Task.to_dict / from_dict round-trip (task file save/load) +# --------------------------------------------------------------------------- + +class TestTaskSerialization: + def test_round_trip(self): + task = Task( + prompt="Allowed: {labels}\nQuote: {quote}", + system="Be terse.", + lists={"labels": TaskList(["pos", "neg"], "hyphen")}, + output_fields=[ + OutputField(name="label", type="choice", list_ref="labels"), + OutputField(name="keywords", type="text_list"), + ], + carry_columns=["quote_id"], + model="o3", + temperature=0.7, + reasoning_effort="high", + max_output_tokens=500, + ) + restored = Task.from_dict(task.to_dict()) + assert restored.prompt == task.prompt + assert restored.system == task.system + assert restored.lists["labels"].values == ["pos", "neg"] + assert restored.lists["labels"].format == "hyphen" + assert [f.name for f in restored.output_fields] == ["label", "keywords"] + assert restored.carry_columns == ["quote_id"] + assert restored.model == "o3" + assert restored.temperature == 0.7 + assert restored.reasoning_effort == "high" + assert restored.max_output_tokens == 500 + + def test_old_task_json_without_model_defaults(self): + """Task files saved before model/params existed must still load.""" + restored = Task.from_dict({ + "prompt": "{quote}", + "output_fields": [{"name": "label", "type": "free_text"}], + }) + assert restored.model == DEFAULT_MODEL + assert restored.temperature is None + assert restored.reasoning_effort is None + assert restored.max_output_tokens is None + + def test_round_trip_via_json(self): + import json + + task = Task( + prompt="{quote}", + lists={"labels": TaskList(["a", "b"])}, + output_fields=[OutputField(name="label", type="choice", list_ref="labels")], + ) + blob = json.dumps(task.to_dict()) + restored = Task.from_dict(json.loads(blob)) + assert restored.lists["labels"].values == ["a", "b"] + + def test_minimal_task_defaults(self): + task = Task.from_dict({"prompt": "{quote}"}) + assert task.system is None + assert task.lists == {} + assert task.output_fields == [] + assert task.carry_columns == [] + assert task.model == DEFAULT_MODEL + + +# --------------------------------------------------------------------------- +# is_reasoning_model / build_request_params +# --------------------------------------------------------------------------- + +class TestIsReasoningModel: + @pytest.mark.parametrize("model", ["o1", "o1-mini", "o3", "o3-mini", "o4-mini", "gpt-5", "gpt-5-mini"]) + def test_reasoning_models(self, model): + assert is_reasoning_model(model) is True + + @pytest.mark.parametrize("model", ["gpt-4o", "gpt-4o-mini", "gpt-4.1"]) + def test_standard_models(self, model): + assert is_reasoning_model(model) is False + + +class TestBuildRequestParams: + def _task(self, **kwargs) -> Task: + return Task(prompt="{quote}", output_fields=[OutputField(name="x", type="free_text")], **kwargs) + + def test_standard_model_sends_temperature_not_reasoning(self): + params = build_request_params(self._task(model="gpt-4o-mini", temperature=0.3, reasoning_effort="high")) + assert params["model"] == "gpt-4o-mini" + assert params["temperature"] == 0.3 + assert "reasoning" not in params + + def test_reasoning_model_sends_reasoning_not_temperature(self): + params = build_request_params(self._task(model="o3", temperature=0.3, reasoning_effort="high")) + assert params["model"] == "o3" + assert params["reasoning"] == {"effort": "high"} + assert "temperature" not in params + + def test_reasoning_model_without_effort_omits_reasoning_key(self): + params = build_request_params(self._task(model="o3")) + assert "reasoning" not in params + + def test_max_output_tokens_passes_through_for_both(self): + std = build_request_params(self._task(model="gpt-4o-mini", max_output_tokens=200)) + reasoning = build_request_params(self._task(model="o3", max_output_tokens=200)) + assert std["max_output_tokens"] == 200 + assert reasoning["max_output_tokens"] == 200 + + def test_unset_optional_params_omitted(self): + params = build_request_params(self._task(model="gpt-4o-mini")) + assert params == {"model": "gpt-4o-mini"} diff --git a/ui/batch_operations.py b/ui/batch_operations.py index 714292a..9534f3e 100644 --- a/ui/batch_operations.py +++ b/ui/batch_operations.py @@ -13,35 +13,11 @@ # Handle imports based on how the script is run try: from batch_processing import batch_method - from batch_processing.batch_method import list_batches - from ui.ui_utils import populate_treeview except ImportError: import sys import os sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.abspath(__file__)))) from batch_processing import batch_method - from batch_processing.batch_method import list_batches - from ui.ui_utils import populate_treeview - - -def call_batch_async(parent: tk.Tk, type) -> None: - """ - Start a new batch processing job on a background thread. - - This function launches the batch creation process in a separate thread - to prevent the UI from freezing during file selection and API calls. - - Args: - parent: Parent Tkinter window for error dialog ownership - """ - def _worker(): - try: - result = batch_method.send_batch(parent, type) - refresh_batches_async(parent) - parent.after(0, lambda: print("batch_method finished:", result)) - except Exception as error: - parent.after(0, lambda: messagebox.showerror("Batch Error", str(error))) - threading.Thread(target=_worker, daemon=True).start() def call_batch_download_async(parent: tk.Tk, batch_id: str) -> None: @@ -59,26 +35,34 @@ def _worker(): def refresh_batches_async(parent: tk.Tk) -> None: """ - Refresh the batch job lists on a background thread. + Refresh the batches table. - This function fetches the latest batch job status from OpenAI and - updates both the ongoing and completed batch tables. + Thin shim to the panel's own refresh (ui/batches_view.py owns the fetch + thread and rendering now); kept so existing call sites (e.g. after a + batch submission) don't need to know about the panel directly. Args: - parent: Parent Tkinter window containing the batch tables + parent: The main window, which holds `.batches_panel` """ - def _worker(): - try: - ongoing_batches, done_batches = list_batches() + parent.batches_panel.refresh() + - def _update_ui(): - cols = ("id", "status", "created_at", "model", "type", "dataset(s)") - populate_treeview(parent.tree_ongoing, cols, ongoing_batches) - populate_treeview(parent.tree_done, cols, done_batches) +def rerun_batch_async(parent: tk.Tk, batch_id: str, count: int) -> None: + """ + Resubmit a batch with identical settings `count` times, on a background + thread, then refresh the batches table. - parent.after(0, _update_ui) + Args: + parent: The main window, which holds `.batches_panel` + batch_id: The batch to rerun + count: How many times to resubmit + """ + def _worker(): + try: + batch_method.rerun_batch(batch_id, count) + parent.after(0, parent.batches_panel.refresh) except Exception as error: - parent.after(0, partial(messagebox.showerror, "Refresh Error", str(error))) + parent.after(0, partial(messagebox.showerror, "Rerun Error", str(error))) threading.Thread(target=_worker, daemon=True).start() diff --git a/ui/batches_view.py b/ui/batches_view.py new file mode 100644 index 0000000..99bb666 --- /dev/null +++ b/ui/batches_view.py @@ -0,0 +1,434 @@ +""" +Batches table: a single sortable/filterable list of all batch jobs (ongoing and +done merged, distinguished by their "status" column) with a column picker, a +detail pane for the selected batch, and a status-aware right-click menu +(Cancel / Download / Rerun). + +Replaces the old two-tab (Ongoing/Done) Treeview pair in main_window.py, which +had no sort, no filter, and always showed every column. +""" + +import threading +import tkinter as tk +from datetime import datetime +from tkinter import ttk + +try: + from ui_helpers import popup_menu + from batch_operations import cancel_batch_async, call_batch_download_async, rerun_batch_async +except ImportError: # fallback when running as a package (ui.*) + from ui.ui_helpers import popup_menu + from ui.batch_operations import cancel_batch_async, call_batch_download_async, rerun_batch_async + +from batch_processing.batch_method import ONGOING_STATUSES, list_batches +from settings.user_config import get_setting, set_setting + +COLUMNS = ( + "id", "status", "type", "dataset", "model", "rows", + "progress", "created", "temperature", "reasoning", "max_output_tokens", +) +HEADERS = { + "id": "ID", "status": "Status", "type": "Type", "dataset": "Dataset", + "model": "Model", "rows": "Rows", "progress": "Progress", "created": "Created", + "temperature": "Temp", "reasoning": "Reasoning", "max_output_tokens": "Max Tokens", +} +# Shown by default; the rest (id, temperature, reasoning, max_output_tokens) are +# still visible in the detail pane and can be toggled on via "Columns". +DEFAULT_VISIBLE = ["status", "type", "dataset", "model", "rows", "progress", "created"] + +COLUMNS_SETTING_KEY = "batches_columns" + +# Columns with a small, closed set of values -- rendered as a dropdown of +# whatever values are actually present in the current data (not a hardcoded +# OpenAI status enum, which could drift out of sync). +CHOICE_COLUMNS = ("status", "type", "model", "reasoning") +# Columns that hold a plain number -- rendered as a min/max entry pair. +NUMBER_COLUMNS = ("temperature", "rows", "max_output_tokens") +# Columns that hold a "YYYY-MM-DD HH:MM" timestamp -- rendered as a from/to +# entry pair, compared by date (the ISO-ish format sorts/compares lexically). +DATE_COLUMNS = ("created",) + +BLANK_LABEL = "(blank)" + + +def _sort_key(value: str): + """Numeric-aware sort key: numbers sort as numbers, blanks sort last, + everything else sorts as a case-insensitive string.""" + if value == "": + return (1, "") + try: + return (0, float(value)) + except (TypeError, ValueError): + return (0, value.casefold()) + + +def _in_range(raw: str, min_s: str, max_s: str) -> bool: + """Numeric range check for one field. Blank/non-numeric field values fail + any *active* bound (e.g. filtering temperature excludes reasoning-model + batches that have none). Malformed min/max entries are simply ignored.""" + min_s, max_s = min_s.strip(), max_s.strip() + if not min_s and not max_s: + return True + try: + val = float(raw) + except (TypeError, ValueError): + return False + if min_s: + try: + if val < float(min_s): + return False + except ValueError: + pass + if max_s: + try: + if val > float(max_s): + return False + except ValueError: + pass + return True + + +def _date_in_range(raw: str, from_s: str, to_s: str) -> bool: + """Date range check against a 'YYYY-MM-DD HH:MM' field value, using only + its date portion. Malformed from/to entries are ignored (that bound is + dropped rather than the row).""" + from_s, to_s = from_s.strip(), to_s.strip() + if not from_s and not to_s: + return True + date_part = raw[:10] + + def _valid(bound: str) -> bool: + try: + datetime.strptime(bound, "%Y-%m-%d") + return True + except ValueError: + return False + + if from_s and _valid(from_s) and date_part < from_s: + return False + if to_s and _valid(to_s) and date_part > to_s: + return False + return True + + +def _center_over(win: tk.Toplevel, parent: tk.Misc) -> None: + """Center a just-built Toplevel over the main window instead of wherever + the window manager defaults to (top-left on Windows).""" + win.update_idletasks() + root = parent.winfo_toplevel() + x = root.winfo_rootx() + (root.winfo_width() - win.winfo_width()) // 2 + y = root.winfo_rooty() + (root.winfo_height() - win.winfo_height()) // 2 + win.geometry(f"+{max(x, 0)}+{max(y, 0)}") + + +def _ask_rerun_count(parent: tk.Misc) -> "int | None": + """Small modal dialog: how many times to rerun a batch. Default 1.""" + dlg = tk.Toplevel(parent) + dlg.title("Rerun Batch") + dlg.transient(parent.winfo_toplevel()) + dlg.grab_set() + dlg.resizable(False, False) + + ttk.Label(dlg, text="Number of reruns:").grid(row=0, column=0, padx=10, pady=10, sticky="w") + count_var = tk.StringVar(value="1") + spin = ttk.Spinbox(dlg, from_=1, to=50, textvariable=count_var, width=6) + spin.grid(row=0, column=1, padx=(0, 10), pady=10) + spin.focus_set() + + result: dict = {"count": None} + + def _ok(_event=None): + try: + n = int(count_var.get()) + except ValueError: + n = 1 + result["count"] = max(1, min(50, n)) + dlg.destroy() + + def _cancel(_event=None): + dlg.destroy() + + btns = ttk.Frame(dlg) + btns.grid(row=1, column=0, columnspan=2, pady=(0, 10)) + ttk.Button(btns, text="OK", command=_ok).grid(row=0, column=0, padx=4) + ttk.Button(btns, text="Cancel", command=_cancel).grid(row=0, column=1, padx=4) + + dlg.bind("", _ok) + dlg.bind("", _cancel) + + _center_over(dlg, parent) + parent.wait_window(dlg) + return result["count"] + + +class BatchesPanel(ttk.Frame): + """Owns the batches table's data, widgets, and actions.""" + + def __init__(self, parent: tk.Misc): + super().__init__(parent) + self.columnconfigure(0, weight=1) + self.rowconfigure(2, weight=1) + + self._rows: list[dict] = [] # full cache from the last refresh + self._by_id: dict[str, dict] = {} # batch id -> row dict, rebuilt each render + self._sort_col: str | None = None + self._sort_reverse = False + self._visible = set(get_setting(COLUMNS_SETTING_KEY, DEFAULT_VISIBLE)) + + self._choice_vars: dict[str, tk.StringVar] = {} + self._choice_combos: dict[str, ttk.Combobox] = {} + self._range_vars: dict[str, tuple[tk.StringVar, tk.StringVar]] = {} + self._date_vars: dict[str, tuple[tk.StringVar, tk.StringVar]] = {} + + self._build_toolbar() + self._build_body() + + # ----- construction ----------------------------------------------- + + def _build_toolbar(self): + row1 = ttk.Frame(self) + row1.grid(row=0, column=0, sticky="ew", pady=(0, 4)) + row2 = ttk.Frame(self) + row2.grid(row=1, column=0, sticky="ew", pady=(0, 6)) + + # Row 1: free-text search, then a dropdown per small-closed-set column. + ttk.Label(row1, text="Search:").pack(side="left") + self._search_var = tk.StringVar() + self._search_var.trace_add("write", lambda *_: self._render()) + ttk.Entry(row1, textvariable=self._search_var, width=16).pack(side="left", padx=(4, 0)) + + for col in CHOICE_COLUMNS: + self._add_choice_control(row1, col) + + columns_btn = ttk.Menubutton(row1, text="Columns ▾") + columns_btn.pack(side="right") + self._columns_menu = tk.Menu(columns_btn, tearoff=False) + columns_btn["menu"] = self._columns_menu + self._column_vars: dict[str, tk.BooleanVar] = {} + for col in COLUMNS: + var = tk.BooleanVar(value=col in self._visible) + self._column_vars[col] = var + self._columns_menu.add_checkbutton( + label=HEADERS[col], variable=var, + command=lambda c=col: self._on_column_toggle(c), + ) + + # Row 2: a from/to entry pair per date column, min/max per number + # column, and a way to reset everything at once. + for col in DATE_COLUMNS: + self._add_range_control(row2, col, f"{HEADERS[col]} (YYYY-MM-DD):", self._date_vars) + for col in NUMBER_COLUMNS: + self._add_range_control(row2, col, f"{HEADERS[col]}:", self._range_vars) + + ttk.Button(row2, text="Clear Filters", command=self._clear_filters).pack(side="right") + + def _add_choice_control(self, parent, col: str): + ttk.Label(parent, text=f"{HEADERS[col]}:").pack(side="left", padx=(10, 2)) + var = tk.StringVar(value="All") + combo = ttk.Combobox(parent, textvariable=var, state="readonly", width=13, values=["All"]) + combo.pack(side="left") + # A trace (not just <>) so any programmatic change + # -- e.g. _refresh_choice_options resetting a stale selection -- + # re-renders too, not just direct user picks. + var.trace_add("write", lambda *_: self._render()) + self._choice_vars[col] = var + self._choice_combos[col] = combo + + def _add_range_control(self, parent, col: str, label: str, store: dict): + ttk.Label(parent, text=label).pack(side="left", padx=(10, 2)) + low, high = tk.StringVar(), tk.StringVar() + ttk.Entry(parent, textvariable=low, width=9).pack(side="left") + ttk.Label(parent, text="–").pack(side="left", padx=2) + ttk.Entry(parent, textvariable=high, width=9).pack(side="left") + low.trace_add("write", lambda *_: self._render()) + high.trace_add("write", lambda *_: self._render()) + store[col] = (low, high) + + def _build_body(self): + body = ttk.PanedWindow(self, orient="horizontal") + body.grid(row=2, column=0, sticky="nsew") + + tree_frame = ttk.Frame(body) + tree_frame.columnconfigure(0, weight=1) + tree_frame.rowconfigure(0, weight=1) + + self.tree = ttk.Treeview(tree_frame, columns=COLUMNS, show="headings", height=12) + for col in COLUMNS: + self.tree.heading(col, text=HEADERS[col], command=lambda c=col: self._sort_by(c)) + self.tree.column(col, anchor="w", stretch=True, width=90, minwidth=40) + self.tree["displaycolumns"] = self._displaycolumns() + + v_scroll = ttk.Scrollbar(tree_frame, orient="vertical", command=self.tree.yview) + self.tree.configure(yscrollcommand=v_scroll.set) + self.tree.grid(row=0, column=0, sticky="nsew") + v_scroll.grid(row=0, column=1, sticky="ns") + + self.tree.bind("<>", lambda e: self._update_detail()) + self.tree.bind("", self._on_right_click) + self.tree.bind("", self._on_right_click) + # Redistribute column widths to fill the available space whenever the + # tree is resized (window resize, pane drag) -- ttk's own stretch + # doesn't recompute this on its own after displaycolumns changes. + self.tree.bind("", self._autosize_columns) + + body.add(tree_frame, weight=3) + + self._detail_frame = ttk.Frame(body, padding=(12, 0)) + body.add(self._detail_frame, weight=1) + self._detail_labels: dict[str, ttk.Label] = {} + for i, col in enumerate(COLUMNS): + ttk.Label(self._detail_frame, text=f"{HEADERS[col]}:", font=("Segoe UI", 9, "bold")).grid( + row=i, column=0, sticky="ne", pady=2) + value_lbl = ttk.Label(self._detail_frame, text="", wraplength=220, justify="left") + value_lbl.grid(row=i, column=1, sticky="nw", padx=(6, 0), pady=2) + self._detail_labels[col] = value_lbl + + def _displaycolumns(self): + return tuple(c for c in COLUMNS if c in self._visible) + + def _autosize_columns(self, event=None): + """Spread the visible columns evenly across the tree's current width.""" + cols = self.tree["displaycolumns"] or COLUMNS + total = self.tree.winfo_width() + if total <= 1: # not yet realized (e.g. during initial construction) + return + width = max(total // len(cols), 40) + for c in cols: + self.tree.column(c, width=width, stretch=True) + + # ----- data loading -------------------------------------------------- + + def refresh(self): + """Fetch batches from the API on a background thread, then repopulate.""" + def _worker(): + try: + rows = list_batches() + self.after(0, lambda: self._set_rows(rows)) + except Exception as error: + from tkinter import messagebox + self.after(0, lambda: messagebox.showerror("Refresh Error", str(error))) + threading.Thread(target=_worker, daemon=True).start() + + def _set_rows(self, rows: list[dict]): + self._rows = rows + self._refresh_choice_options() + self._render() + + def _distinct_options(self, col: str) -> list[str]: + values = {row.get(col, "") for row in self._rows} + options = sorted(v for v in values if v != "") + if "" in values: + options.append(BLANK_LABEL) + return ["All"] + options + + def _refresh_choice_options(self): + """Repopulate each dropdown from the data actually on hand, so it + always reflects real statuses/models/etc. rather than a guessed list. + Resets a selection that no longer has any matching rows.""" + for col, combo in self._choice_combos.items(): + options = self._distinct_options(col) + combo["values"] = options + if self._choice_vars[col].get() not in options: + self._choice_vars[col].set("All") + + # ----- filtering: search text, choice dropdowns, numeric/date ranges --- + + def _passes_filters(self, row: dict) -> bool: + search = self._search_var.get().strip().lower() + if search and search not in " ".join(str(v) for v in row.values()).lower(): + return False + + for col, var in self._choice_vars.items(): + selected = var.get() + if selected and selected != "All": + target = "" if selected == BLANK_LABEL else selected + if row.get(col, "") != target: + return False + + for col, (min_var, max_var) in self._range_vars.items(): + if not _in_range(str(row.get(col, "")), min_var.get(), max_var.get()): + return False + + for col, (from_var, to_var) in self._date_vars.items(): + if not _date_in_range(str(row.get(col, "")), from_var.get(), to_var.get()): + return False + + return True + + def _clear_filters(self): + self._search_var.set("") + for var in self._choice_vars.values(): + var.set("All") + for low, high in list(self._range_vars.values()) + list(self._date_vars.values()): + low.set("") + high.set("") + + # ----- rendering: filter + sort, then repopulate the tree ------------ + + def _render(self): + rows = [r for r in self._rows if self._passes_filters(r)] + + if self._sort_col: + rows = sorted(rows, key=lambda r: _sort_key(str(r.get(self._sort_col, ""))), + reverse=self._sort_reverse) + + selected = self.tree.selection() + for iid in self.tree.get_children(): + self.tree.delete(iid) + + self._by_id = {} + for row in rows: + self.tree.insert("", "end", iid=row["id"], values=[row.get(c, "") for c in COLUMNS]) + self._by_id[row["id"]] = row + + if selected and selected[0] in self._by_id: + self.tree.selection_set(selected[0]) + self._update_detail() + + def _sort_by(self, col: str): + if self._sort_col == col: + self._sort_reverse = not self._sort_reverse + else: + self._sort_col, self._sort_reverse = col, False + self._render() + + def _on_column_toggle(self, col: str): + if self._column_vars[col].get(): + self._visible.add(col) + else: + self._visible.discard(col) + self.tree["displaycolumns"] = self._displaycolumns() + self._autosize_columns() + set_setting(COLUMNS_SETTING_KEY, [c for c in COLUMNS if c in self._visible]) + + # ----- detail pane ----------------------------------------------------- + + def _update_detail(self): + sel = self.tree.selection() + row = self._by_id.get(sel[0]) if sel else None + for col, lbl in self._detail_labels.items(): + lbl.config(text=(row.get(col, "") or "—") if row else "") + + # ----- right-click menu: Rerun always, Cancel/Download by status ------- + + def _on_right_click(self, event): + iid = self.tree.identify_row(event.y) + row = self._by_id.get(iid) + if row is None: + return + + root = self.winfo_toplevel() + menu = tk.Menu(self, tearoff=False) + menu.add_command(label="Rerun…", command=lambda: self._on_rerun(root, row["id"])) + if row["status"] in ONGOING_STATUSES: + menu.add_command(label="Cancel", command=lambda: cancel_batch_async(root, row["id"])) + elif row["status"] == "completed": + menu.add_command(label="Download", command=lambda: call_batch_download_async(root, row["id"])) + + popup_menu(event, self.tree, menu) + + def _on_rerun(self, root, batch_id: str): + count = _ask_rerun_count(self) + if not count: + return + rerun_batch_async(root, batch_id, count) diff --git a/ui/main_window.py b/ui/main_window.py index 9f898d4..17ce61a 100644 --- a/ui/main_window.py +++ b/ui/main_window.py @@ -3,18 +3,15 @@ This module provides the primary user interface for the CodebookAI application, including batch job management, live processing controls, and settings access. -The interface displays ongoing and completed batch jobs in tabbed tables. +The interface displays all batch jobs in one sortable/filterable table with a +detail pane (see ui/batches_view.py). -Updated per request: -- Removed add_btn, tools_btn, and settings_btn (refresh button retained). -- Added a top menu bar with the following structure: +Top menu bar structure: File > Settings, Exit Data Prep > Sample - LLM Tools > Live Methods > Single Label Text Classification, Multi-Label Text Classification, Text Extraction - > Batch Methods > Single Label Text Classification - Data Analysis > Reliability Statistics - Help > Github Repo -- Added a "Batches" title above the table area at the bottom of the page. + LLM Tools > New Task (blank Task Builder), Presets > {built-in presets} + Data Analysis > Reliability Statistics, Correlogram + Help > Help Docs, Report a bug """ import sys, os @@ -24,6 +21,7 @@ import webbrowser from asset_path import asset_path +from core.presets import PRESETS from live_processing.correlogram import open_correlogram_wizard from live_processing.reliability_calculator import open_reliability_wizard from live_processing.sampler import sample_data @@ -33,38 +31,16 @@ from settings_window import SettingsWindow from tooltip import ToolTip from ui_utils import center_window - from ui_helpers import make_tab_with_tree, popup_menu, popup_menu_below_widget - from batch_operations import ( - call_batch_async, - refresh_batches_async, - call_batch_download_async, - cancel_batch_async, - ) + from batches_view import BatchesPanel + from batch_operations import refresh_batches_async + from task_builder import open_task_builder except ImportError: # fallback when running as a package (ui.*) from ui.settings_window import SettingsWindow from ui.tooltip import ToolTip from ui.ui_utils import center_window - from ui.ui_helpers import make_tab_with_tree, popup_menu, popup_menu_below_widget - from ui.batch_operations import ( - call_batch_async, - refresh_batches_async, - call_batch_download_async, - cancel_batch_async, - ) - -# Ensure live modules can be imported when run directly -try: - import live_processing.multi_label_live - import live_processing.single_label_live - import live_processing.keyword_extraction_live -except ImportError: - import sys - import os - - sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.abspath(__file__)))) - import live_processing.multi_label_live - import live_processing.single_label_live - import live_processing.keyword_extraction_live + from ui.batches_view import BatchesPanel + from ui.batch_operations import refresh_batches_async + from ui.task_builder import open_task_builder APP_TITLE = "CodebookAI" APP_SUBTITLE = "A qualitative research tool based on OpenAI's Playground API." @@ -85,8 +61,8 @@ def build_ui(root: tk.Tk) -> None: This function creates the complete UI layout including: - Header with app title & subtitle - Top menubar for navigation and actions - - Tabbed interface for ongoing and completed batch jobs - - Refresh control and context menus for batch operations + - A single sortable/filterable batches table with a detail pane + - Refresh control and a status-aware right-click menu for batch operations Args: root: The main Tkinter window to build the UI in @@ -102,8 +78,8 @@ def build_ui(root: tk.Tk) -> None: # ===== Top-level grid: header, spacer, table area ===== root.columnconfigure(0, weight=1) root.rowconfigure(0, weight=0) # header - root.rowconfigure(1, weight=1) # spacer/filler (kept for compatibility) - root.rowconfigure(2, weight=0) # table area + root.rowconfigure(1, weight=0) # spacer/filler (kept for compatibility) + root.rowconfigure(2, weight=1) # table area now grows with the window # ===== Header (title & subtitle only; no buttons) ===== # ===== Header (banner image) ===== @@ -135,46 +111,19 @@ def build_ui(root: tk.Tk) -> None: data_prep_menu.add_command(label="Sample", command=lambda: sample_data(root)) menubar.add_cascade(label="Data Prep", menu=data_prep_menu) - # LLM Tools > Live Methods / Batch Methods + # LLM Tools > New Task / Presets + # The old six hardcoded tools (single/multi/keyword x live/batch) are gone -- + # one Task Builder window covers all of it. Presets just pre-fill the builder. llm_tools_menu = tk.Menu(menubar, tearoff=False) - - # Live Methods submenu - live_methods_menu = tk.Menu(llm_tools_menu, tearoff=False) - - def _single_label_live_call(): - live_processing.single_label_live.single_label_pipeline(root) - - def _multi_label_live_call(): - live_processing.multi_label_live.multi_label_pipeline(root) - - def _keyword_extraction_live_call(): - live_processing.keyword_extraction_live.keyword_extraction_pipeline(root) - - live_methods_menu.add_command( - label="Single Label Text Classification", command=_single_label_live_call - ) - live_methods_menu.add_command( - label="Multi-Label Text Classification", command=_multi_label_live_call - ) - live_methods_menu.add_command(label="Keyword Extraction", command=_keyword_extraction_live_call) - - # Batch Methods submenu - batch_methods_menu = tk.Menu(llm_tools_menu, tearoff=False) - batch_methods_menu.add_command( - label="Single Label Text Classification", - command=lambda: call_batch_async(root, type="single_label"), - ) - batch_methods_menu.add_command( - label="Multi-Label Text Classification", - command=lambda: call_batch_async(root, type="multi_label"), - ) - batch_methods_menu.add_command( - label="Keyword Extraction", - command=lambda: call_batch_async(root, type="keyword_extraction"), - ) - - llm_tools_menu.add_cascade(label="Live Methods", menu=live_methods_menu) - llm_tools_menu.add_cascade(label="Batch Methods", menu=batch_methods_menu) + llm_tools_menu.add_command(label="New Task", command=lambda: open_task_builder(root)) + + presets_menu = tk.Menu(llm_tools_menu, tearoff=False) + for key, (display_name, factory) in PRESETS.items(): + presets_menu.add_command( + label=display_name, + command=lambda factory=factory: open_task_builder(root, factory()), + ) + llm_tools_menu.add_cascade(label="Presets", menu=presets_menu) menubar.add_cascade(label="LLM Tools", menu=llm_tools_menu) # Data Analysis @@ -193,7 +142,7 @@ def _keyword_extraction_live_call(): # ===== Table area ===== table_area = ttk.Frame(root, padding=(16, 12)) - table_area.grid(row=2, column=0, sticky="ew") + table_area.grid(row=2, column=0, sticky="nsew") table_area.columnconfigure(0, weight=1) # Controls row with section title (left) and refresh button (right) @@ -209,49 +158,12 @@ def _keyword_extraction_live_call(): refresh_btn.grid(row=0, column=1, sticky="e") ToolTip(refresh_btn, "Refresh - Update the batch job lists with current status") - # Notebook with two tabs - notebook = ttk.Notebook(table_area) - notebook.grid(row=1, column=0, sticky="ew") - - ongoing_tab, tree_ongoing = make_tab_with_tree(notebook) - done_tab, tree_done = make_tab_with_tree(notebook) - - notebook.add(ongoing_tab, text="Ongoing") - notebook.add(done_tab, text="Done") - - # --- Context menus for rows --- - # Ongoing tab: "Cancel" - ongoing_menu = tk.Menu(root, tearoff=False) - - def _on_cancel(): - sel = tree_ongoing.selection() - if sel: - values = tree_ongoing.item(sel[0], "values") - cancel_batch_async(root, values[0]) # values[0] is the batch ID - - ongoing_menu.add_command(label="Cancel", command=_on_cancel) - - # Done tab: "Download" - done_menu = tk.Menu(root, tearoff=False) - - def _on_download(): - sel = tree_done.selection() - if sel: - values = tree_done.item(sel[0], "values") - call_batch_download_async(root, values[0]) # values[0] is the batch ID - - done_menu.add_command(label="Download", command=_on_download) - - # Bind right-clicks - tree_ongoing.bind("", lambda e: popup_menu(e, tree_ongoing, ongoing_menu)) - tree_done.bind("", lambda e: popup_menu(e, tree_done, done_menu)) - # Optional: Control-click as an extra trigger - tree_ongoing.bind("", lambda e: popup_menu(e, tree_ongoing, ongoing_menu)) - tree_done.bind("", lambda e: popup_menu(e, tree_done, done_menu)) - - # Save references on root - root.tree_ongoing = tree_ongoing - root.tree_done = tree_done + # Single merged, sortable/filterable table with a detail pane and a + # status-aware right-click menu (Cancel / Download / Rerun). + batches_panel = BatchesPanel(table_area) + batches_panel.grid(row=1, column=0, sticky="nsew") + table_area.rowconfigure(1, weight=1) + root.batches_panel = batches_panel # Initial load refresh_batches_async(root) diff --git a/ui/settings_window.py b/ui/settings_window.py index 7f0f169..9519a21 100644 --- a/ui/settings_window.py +++ b/ui/settings_window.py @@ -3,10 +3,12 @@ This module provides a modal settings dialog that allows users to configure: - OpenAI API key (stored securely in system keyring) -- AI model selection (with on-demand refresh from API) - Maximum number of batch jobs to display - Timezone for timestamp display +Model selection now lives per-task in the Task Builder (see ui/task_builder.py), +not here. + The settings are split between sensitive data (API keys) stored in the OS keyring and non-sensitive configuration values stored in the config.py file. """ @@ -18,57 +20,10 @@ from tkinter import ttk, messagebox from zoneinfo import available_timezones from settings.user_config import get_setting, set_setting -from settings.models_registry import get_models, refresh_models, refresh_client +from settings.models_registry import refresh_client from settings.secrets_store import save_api_key, load_api_key, clear_api_key -# -------------------- Simple tooltip helper -------------------- -class Tooltip: - """Lightweight tooltip for Tk/ttk widgets.""" - def __init__(self, widget: tk.Widget, text: str, delay_ms: int = 500): - self.widget = widget - self.text = text - self.delay_ms = delay_ms - self._id = None - self._tip = None - widget.bind("", self._schedule) - widget.bind("", self._hide) - widget.bind("", self._hide) - - def _schedule(self, _): - self._unschedule() - self._id = self.widget.after(self.delay_ms, self._show) - - def _unschedule(self): - if self._id is not None: - self.widget.after_cancel(self._id) - self._id = None - - def _show(self): - if self._tip or not self.text: - return - # position near the widget - x, y, cx, cy = self.widget.bbox("insert") if hasattr(self.widget, "bbox") else (0, 0, 0, 0) - x = x + self.widget.winfo_rootx() + 10 - y = y + self.widget.winfo_rooty() + cy + 10 - self._tip = tw = tk.Toplevel(self.widget) - tw.wm_overrideredirect(True) - tw.wm_geometry(f"+{x}+{y}") - lbl = tk.Label( - tw, text=self.text, justify="left", - background="#ffffe0", relief="solid", borderwidth=1, - font=("TkDefaultFont", 9), padx=6, pady=4 - ) - lbl.pack() - - def _hide(self, _=None): - self._unschedule() - if self._tip is not None: - self._tip.destroy() - self._tip = None -# --------------------------------------------------------------- - - class SettingsWindow(tk.Toplevel): """ Modal settings configuration dialog. @@ -82,7 +37,6 @@ def __init__(self, parent: tk.Tk | tk.Toplevel): # --- State variables --- self.var_api_key = tk.StringVar(value=load_api_key() or "") - self.var_model = tk.StringVar(value=get_setting("model", "gpt-4o")) self.var_max_batches = tk.StringVar(value=str(get_setting("max_batches", 20))) self.var_timezone = tk.StringVar(value=get_setting("time_zone", "UTC")) self.var_show_key = tk.BooleanVar(value=False) @@ -123,30 +77,8 @@ def __init__(self, parent: tk.Tk | tk.Toplevel): ) chk_show.grid(row=0, column=2, sticky="w", **pad) - # ---- Model ---- - ttk.Label(frm, text="Model:").grid(row=1, column=0, sticky="w", **pad) - - # Model combobox in column 1 - self.cmb_model = ttk.Combobox( - frm, - textvariable=self.var_model, - width=45, - values=get_models(), # cached list - state="readonly", - ) - self.cmb_model.grid(row=1, column=1, sticky="w", **pad) - - # Refresh button in column 2, with tooltip - self.btn_refresh_models = ttk.Button(frm, text="↻", width=3, command=self._on_refresh_models) - self.btn_refresh_models.grid(row=1, column=2, sticky="w", **pad) - Tooltip( - self.btn_refresh_models, - "Refresh the list of models from OpenAI's API.\n" - "Use this if new models were added to your account." - ) - # ---- Time Zone ---- - ttk.Label(frm, text="Time Zone:").grid(row=2, column=0, sticky="w", **pad) + ttk.Label(frm, text="Time Zone:").grid(row=1, column=0, sticky="w", **pad) self.cmb_timezone = ttk.Combobox( frm, textvariable=self.var_timezone, @@ -154,10 +86,10 @@ def __init__(self, parent: tk.Tk | tk.Toplevel): values=sorted(available_timezones()), state="readonly", ) - self.cmb_timezone.grid(row=2, column=1, columnspan=2, sticky="w", **pad) + self.cmb_timezone.grid(row=1, column=1, columnspan=2, sticky="w", **pad) # ---- Max Batches ---- - ttk.Label(frm, text="Max Batches:").grid(row=3, column=0, sticky="w", **pad) + ttk.Label(frm, text="Max Batches:").grid(row=2, column=0, sticky="w", **pad) self.ent_max = ttk.Spinbox( frm, from_=1, @@ -165,11 +97,11 @@ def __init__(self, parent: tk.Tk | tk.Toplevel): textvariable=self.var_max_batches, width=10, ) - self.ent_max.grid(row=3, column=1, sticky="w", **pad) + self.ent_max.grid(row=2, column=1, sticky="w", **pad) # ---- Buttons ---- btns = ttk.Frame(frm) - btns.grid(row=4, column=0, columnspan=3, sticky="e", pady=(12, 0)) + btns.grid(row=3, column=0, columnspan=3, sticky="e", pady=(12, 0)) ttk.Button(btns, text="Reset to File", command=self._reset_from_file).grid(row=0, column=0, padx=6) ttk.Button(btns, text="Clear Key", command=self._clear_key).grid(row=0, column=1, padx=6) ttk.Button(btns, text="Cancel", command=self.destroy).grid(row=0, column=2, padx=6) @@ -202,7 +134,6 @@ def _reset_from_file(self): """Reload settings from defaults and user config, reset UI fields. API key reloads from keyring.""" try: self.var_api_key.set(load_api_key() or "") - self.var_model.set(get_setting("model", "gpt-4o")) self.var_max_batches.set(str(get_setting("max_batches", 20))) self.var_timezone.set(get_setting("time_zone", "UTC")) except Exception as e: @@ -235,7 +166,6 @@ def _save(self): return api_key = self.var_api_key.get().strip() - model = self.var_model.get().strip() max_batches = int(self.var_max_batches.get().strip()) time_zone = self.var_timezone.get().strip() @@ -247,33 +177,9 @@ def _save(self): refresh_client() # 2) Save non-secrets to user config (JSON file in user directory) - set_setting("model", model) - set_setting("max_batches", max_batches) + set_setting("max_batches", max_batches) set_setting("time_zone", time_zone) self.destroy() except Exception as e: messagebox.showerror("Save Error", f"Could not save settings:\n{e}") - - # ---- UI callbacks ---- - def _on_refresh_models(self): - """ - Refresh the model list via API, update the combobox values, - and keep selection if still present; else select first item. - """ - try: - current = self.var_model.get().strip() - models = refresh_models() or [] - if not models: - messagebox.showwarning("Models", "No models returned. Check your API key and network.") - return - self.cmb_model["values"] = models - # Keep prior selection if still available; otherwise pick first - if current in models: - # ensure combobox reflects the value (in case of case differences) - self.var_model.set(current) - else: - self.var_model.set(models[0]) - messagebox.showinfo("Models", "Model list refreshed from OpenAI.") - except Exception as e: - messagebox.showerror("Models", f"Failed to refresh models:\n{e}") diff --git a/ui/task_builder.py b/ui/task_builder.py new file mode 100644 index 0000000..c59c923 --- /dev/null +++ b/ui/task_builder.py @@ -0,0 +1,770 @@ +""" +Task Builder: the flexible UI that replaces the six hardcoded LLM tools +(single/multi/keyword x batch/live). + +Import one table, write a prompt with {column} / {list_name} placeholders, +define named Lists (typed or imported), define output fields (Choice / +Multi-choice bind to a List, so the same values shown in the prompt are the +ones enforced on the output), pick which columns carry through to the result +untouched, then Run Live or Submit Batch. +""" + +from __future__ import annotations + +import json +import re +import threading +import tkinter as tk +from pathlib import Path +from tkinter import filedialog, messagebox, simpledialog, ttk + +import pandas as pd + +from batch_processing import batch_method +from core.task import ( + DEFAULT_MODEL, + FieldType, + ListFormat, + OutputField, + Task, + TaskList, + TaskValidationError, + is_reasoning_model, + render_prompt, + validate_task, +) +from file_handling.data_import import import_data, load_table +from live_processing.task_live import run_task_live +from settings.models_registry import get_models +from ui.drag_drop import enable_file_drop +from ui.ui_utils import center_window, populate_treeview + +FIELD_TYPES: list[FieldType] = ["choice", "multi_choice", "text_list", "free_text", "number", "yes_no"] +LIST_FORMATS: list[ListFormat] = ["comma", "hyphen", "space", "newline"] +REASONING_EFFORTS = ["", "low", "medium", "high"] + + +def _parse_optional_float(s: str) -> float | None: + s = s.strip() + if not s: + return None + try: + return float(s) + except ValueError: + return None + + +def _parse_optional_int(s: str) -> int | None: + s = s.strip() + if not s: + return None + try: + return int(s) + except ValueError: + return None + +COLUMN_PILL_BG = "#dbe9ff" +LIST_PILL_BG = "#d9f2df" +UNKNOWN_PLACEHOLDER_BG = "#ffd6d6" + + +def open_task_builder(parent: tk.Misc, preset_task: Task | None = None) -> None: + TaskBuilder(parent, preset_task) + + +# --------------------------------------------------------------------------- +# Small reusable widgets +# --------------------------------------------------------------------------- + +class _ScrollableFrame(ttk.Frame): + """A vertically scrollable frame -- the builder has more sections than + fit in a fixed window; content goes in `.body`.""" + + def __init__(self, master): + super().__init__(master) + canvas = tk.Canvas(self, highlightthickness=0) + vsb = ttk.Scrollbar(self, orient="vertical", command=canvas.yview) + self.body = ttk.Frame(canvas) + self.body.bind("", lambda e: canvas.configure(scrollregion=canvas.bbox("all"))) + canvas.create_window((0, 0), window=self.body, anchor="nw") + canvas.configure(yscrollcommand=vsb.set) + canvas.grid(row=0, column=0, sticky="nsew") + vsb.grid(row=0, column=1, sticky="ns") + self.rowconfigure(0, weight=1) + self.columnconfigure(0, weight=1) + + def _on_enter(_): + canvas.bind_all("", lambda e: canvas.yview_scroll(int(-1 * (e.delta / 120)), "units")) + + def _on_leave(_): + canvas.unbind_all("") + + canvas.bind("", _on_enter) + canvas.bind("", _on_leave) + + +class _PillRow(ttk.Frame): + """A horizontally-scrollable row of clickable/draggable placeholder pills. + Click inserts at the active text widget's cursor; dragging onto a text + widget inserts at the drop point.""" + + def __init__(self, master, on_pick): + super().__init__(master) + self.on_pick = on_pick + self.canvas = tk.Canvas(self, height=34, highlightthickness=0) + self.inner = ttk.Frame(self.canvas) + self.window_id = self.canvas.create_window(0, 0, window=self.inner, anchor="nw") + hsb = ttk.Scrollbar(self, orient="horizontal", command=self.canvas.xview) + self.canvas.configure(xscrollcommand=hsb.set) + self.canvas.grid(row=0, column=0, sticky="ew") + hsb.grid(row=1, column=0, sticky="ew") + self.columnconfigure(0, weight=1) + self.inner.bind("", self._on_inner_configure) + + def _on_inner_configure(self, _event=None): + self.canvas.configure(scrollregion=self.canvas.bbox(self.window_id)) + + def set_pills(self, items: list[tuple[str, str]]): + """items: list of (name, kind) where kind in {'column', 'list'}.""" + for w in self.inner.winfo_children(): + w.destroy() + if not items: + ttk.Label(self.inner, text="(none yet)", foreground="#888").grid(row=0, column=0, padx=6, pady=6) + for i, (name, kind) in enumerate(items): + bg = COLUMN_PILL_BG if kind == "column" else LIST_PILL_BG + pill = tk.Label(self.inner, text=name, bg=bg, padx=8, pady=3, relief="raised", borderwidth=1, cursor="hand2") + pill.grid(row=0, column=i, padx=4, pady=4) + pill.bind("", lambda e, n=name: self._on_release(e, n)) + self.inner.update_idletasks() + self._on_inner_configure() + + def _on_release(self, event, name: str): + target = event.widget.winfo_containing(event.x_root, event.y_root) + self.on_pick(name, target) + + +# --------------------------------------------------------------------------- +# Main window +# --------------------------------------------------------------------------- + +class TaskBuilder(tk.Toplevel): + def __init__(self, parent: tk.Misc, preset_task: Task | None = None): + super().__init__(parent) + self.title("Task Builder") + self.geometry("1000x760") + self.transient(parent) + + # ----- state ----- + self.df: "pd.DataFrame | None" = None + self.columns: list[str] = [] + self.current_file: str = "" + self.lists: dict[str, TaskList] = {} + self.carry_vars: dict[str, tk.BooleanVar] = {} + self._field_rows: list[dict] = [] + self._active_text_widget: tk.Text | None = None + self.preview_mode = False + self.preview_row_idx = 0 + + outer = _ScrollableFrame(self) + outer.grid(row=0, column=0, sticky="nsew") + self.rowconfigure(0, weight=1) + self.columnconfigure(0, weight=1) + root = outer.body + root.columnconfigure(0, weight=1) + + self._build_input_section(root) + self._build_prompt_section(root) + self._build_lists_section(root) + self._build_output_fields_section(root) + self._build_carry_section(root) + self._build_model_section(root) + self._build_run_section(root) + + if preset_task is not None: + self._apply_task(preset_task) + + center_window(self, 1000, 760) + + # ----------------------------------------------------------------- + # Input section: import a table, show column pills, optional preview + # ----------------------------------------------------------------- + def _build_input_section(self, root): + frm = ttk.LabelFrame(root, text="Input", padding=10) + frm.grid(row=0, column=0, sticky="ew", padx=10, pady=(10, 6)) + frm.columnconfigure(1, weight=1) + + ttk.Label(frm, text="File:").grid(row=0, column=0, sticky="w") + self.file_var = tk.StringVar() + file_entry = ttk.Entry(frm, textvariable=self.file_var) + file_entry.grid(row=0, column=1, sticky="ew", padx=6) + ttk.Button(frm, text="Browse…", command=self._on_browse).grid(row=0, column=2) + if enable_file_drop is not None: + enable_file_drop(file_entry, self._load_file, [".csv", ".tsv", ".txt", ".xlsx", ".xls"]) + + opts = ttk.Frame(frm) + opts.grid(row=1, column=0, columnspan=3, sticky="w", pady=(6, 0)) + self.has_headers_var = tk.BooleanVar(value=True) + ttk.Checkbutton(opts, text="File has headers", variable=self.has_headers_var, + command=self._reload_current_file).grid(row=0, column=0, padx=(0, 16)) + self.show_preview_var = tk.BooleanVar(value=False) + ttk.Checkbutton(opts, text="Show preview (top 5 rows)", variable=self.show_preview_var, + command=self._refresh_preview_table).grid(row=0, column=1) + + ttk.Label(frm, text="Columns (click to insert, or drag into the prompt):").grid( + row=2, column=0, columnspan=3, sticky="w", pady=(8, 2)) + self.column_pills = _PillRow(frm, self._insert_placeholder) + self.column_pills.grid(row=3, column=0, columnspan=3, sticky="ew") + + self.preview_tree = ttk.Treeview(frm, show="headings", height=5) + self.preview_tree.grid(row=4, column=0, columnspan=3, sticky="ew", pady=(8, 0)) + self.preview_tree.grid_remove() + + def _on_browse(self): + path = filedialog.askopenfilename( + parent=self, + title="Import Data", + filetypes=[ + ("All supported", "*.csv *.tsv *.txt *.xlsx *.xls"), + ("CSV files", "*.csv"), + ("Excel files", "*.xlsx *.xls"), + ("All files", "*.*"), + ], + ) + if path: + self.file_var.set(path) + self._load_file(path) + + def _reload_current_file(self): + if self.file_var.get().strip(): + self._load_file(self.file_var.get().strip()) + + def _load_file(self, path: str): + try: + header, data = load_table(path, self.has_headers_var.get()) + except Exception as e: + messagebox.showerror("Import Error", str(e), parent=self) + return + if not header: + messagebox.showerror("Import Error", "The file appears to have no data.", parent=self) + return + + self.file_var.set(path) + self.current_file = path + self.df = pd.DataFrame(data, columns=header) + self.columns = header + self.carry_vars = {c: tk.BooleanVar(value=True) for c in header} + + self._refresh_column_pills() + self._refresh_carry_ui() + self._refresh_preview_table() + self._highlight_placeholders(self.prompt_edit) + if self.system_var.get(): + self._highlight_placeholders(self.system_edit) + + def _refresh_preview_table(self): + if not self.show_preview_var.get() or self.df is None: + self.preview_tree.grid_remove() + return + rows = [tuple(r) for r in self.df.head(5).itertuples(index=False, name=None)] + populate_treeview(self.preview_tree, tuple(self.columns), rows) + self.preview_tree.grid() + + # ----------------------------------------------------------------- + # Prompt section: editor + system box + edit/preview toggle + # ----------------------------------------------------------------- + def _build_prompt_section(self, root): + frm = ttk.LabelFrame(root, text="Prompt", padding=10) + frm.grid(row=1, column=0, sticky="ew", padx=10, pady=6) + frm.columnconfigure(0, weight=1) + + toolbar = ttk.Frame(frm) + toolbar.grid(row=0, column=0, sticky="ew") + toolbar.columnconfigure(0, weight=1) + ttk.Label(toolbar, text="Use {column} / {list} placeholders. Everything else is sent as-is.").grid( + row=0, column=0, sticky="w") + self.preview_toggle_btn = ttk.Button(toolbar, text="Preview", command=self._toggle_preview) + self.preview_toggle_btn.grid(row=0, column=1, sticky="e") + + self.prompt_container = ttk.Frame(frm) + self.prompt_container.grid(row=1, column=0, sticky="nsew", pady=(4, 4)) + self.prompt_container.columnconfigure(0, weight=1) + + self.prompt_edit = tk.Text(self.prompt_container, height=8, wrap="word") + self.prompt_edit.grid(row=0, column=0, sticky="nsew") + self._configure_placeholder_tags(self.prompt_edit) + self.prompt_edit.bind("", lambda e: self._highlight_placeholders(self.prompt_edit)) + self.prompt_edit.bind("", lambda e: setattr(self, "_active_text_widget", self.prompt_edit)) + + self.prompt_preview = tk.Text(self.prompt_container, height=8, wrap="word", state="disabled", bg="#f5f5f5") + self.prompt_preview.grid(row=0, column=0, sticky="nsew") + self.prompt_preview.grid_remove() + + self.preview_stepper = ttk.Frame(frm) + self.preview_stepper.grid(row=2, column=0, sticky="w") + ttk.Button(self.preview_stepper, text="◀", width=3, command=lambda: self._step_preview(-1)).grid(row=0, column=0) + self.row_stepper_label = ttk.Label(self.preview_stepper, text="Row 0 of 0") + self.row_stepper_label.grid(row=0, column=1, padx=8) + ttk.Button(self.preview_stepper, text="▶", width=3, command=lambda: self._step_preview(1)).grid(row=0, column=2) + self.preview_stepper.grid_remove() + + # System instructions -- hidden by default (progressive disclosure) + self.system_var = tk.BooleanVar(value=False) + ttk.Checkbutton(frm, text="+ System instructions (advanced)", variable=self.system_var, + command=self._toggle_system_box).grid(row=3, column=0, sticky="w", pady=(4, 0)) + + self.system_frame = ttk.Frame(frm) + self.system_frame.columnconfigure(0, weight=1) + self.system_edit = tk.Text(self.system_frame, height=3, wrap="word") + self.system_edit.grid(row=0, column=0, sticky="ew") + self._configure_placeholder_tags(self.system_edit) + self.system_edit.bind("", lambda e: self._highlight_placeholders(self.system_edit)) + self.system_edit.bind("", lambda e: setattr(self, "_active_text_widget", self.system_edit)) + self.system_frame.grid(row=4, column=0, sticky="ew") + self.system_frame.grid_remove() + + def _configure_placeholder_tags(self, widget: tk.Text): + widget.tag_configure("column_ph", background=COLUMN_PILL_BG) + widget.tag_configure("list_ph", background=LIST_PILL_BG) + widget.tag_configure("unknown_ph", background=UNKNOWN_PLACEHOLDER_BG) + + def _highlight_placeholders(self, widget: tk.Text): + for tag in ("column_ph", "list_ph", "unknown_ph"): + widget.tag_remove(tag, "1.0", "end") + text = widget.get("1.0", "end-1c") + for m in re.finditer(r"\{([^{}]+)\}", text): + name = m.group(1) + start, end = f"1.0+{m.start()}c", f"1.0+{m.end()}c" + if name in self.lists: + widget.tag_add("list_ph", start, end) + elif name in self.columns: + widget.tag_add("column_ph", start, end) + else: + widget.tag_add("unknown_ph", start, end) + + def _toggle_system_box(self): + if self.system_var.get(): + self.system_frame.grid() + else: + self.system_frame.grid_remove() + + def _insert_placeholder(self, name: str, drop_target: tk.Widget | None = None): + if self.preview_mode: + return + widget = drop_target if drop_target in (self.prompt_edit, self.system_edit) else self._active_text_widget + if widget is None: + widget = self.prompt_edit + if widget is self.system_edit and not self.system_var.get(): + self.system_var.set(True) + self._toggle_system_box() + if drop_target is widget: + x = widget.winfo_pointerx() - widget.winfo_rootx() + y = widget.winfo_pointery() - widget.winfo_rooty() + widget.mark_set("insert", widget.index(f"@{x},{y}")) + widget.insert("insert", f"{{{name}}}") + widget.focus_set() + self._highlight_placeholders(widget) + + def _toggle_preview(self): + self.preview_mode = not self.preview_mode + if self.preview_mode: + self.prompt_edit.grid_remove() + self.prompt_preview.grid() + self.preview_stepper.grid() + self.preview_toggle_btn.config(text="Edit") + self.preview_row_idx = 0 + self._render_preview_row() + else: + self.prompt_preview.grid_remove() + self.preview_stepper.grid_remove() + self.prompt_edit.grid() + self.preview_toggle_btn.config(text="Preview") + + def _step_preview(self, delta: int): + if self.df is None or len(self.df) == 0: + return + self.preview_row_idx = max(0, min(self.preview_row_idx + delta, len(self.df) - 1)) + self._render_preview_row() + + def _render_preview_row(self): + self.prompt_preview.config(state="normal") + self.prompt_preview.delete("1.0", "end") + if self.df is None or len(self.df) == 0: + self.prompt_preview.insert("1.0", "(Import data to preview the rendered prompt.)") + self.row_stepper_label.config(text="Row 0 of 0") + else: + preview_task = Task(prompt=self.prompt_edit.get("1.0", "end-1c"), lists=dict(self.lists)) + row = self.df.iloc[self.preview_row_idx].to_dict() + parts = [] + if self.system_var.get(): + try: + parts.append("[SYSTEM]\n" + render_prompt(preview_task, row, text=self.system_edit.get("1.0", "end-1c"))) + except TaskValidationError as e: + parts.append(f"[SYSTEM] (cannot render: {e})") + try: + parts.append("[USER]\n" + render_prompt(preview_task, row)) + except TaskValidationError as e: + parts.append(f"[USER] (cannot render: {e})") + self.prompt_preview.insert("1.0", "\n\n".join(parts)) + self.row_stepper_label.config(text=f"Row {self.preview_row_idx + 1} of {len(self.df)}") + self.prompt_preview.config(state="disabled") + + # ----------------------------------------------------------------- + # Lists section: named constant value sets, used in prompt + output + # ----------------------------------------------------------------- + def _build_lists_section(self, root): + frm = ttk.LabelFrame(root, text="Lists", padding=10) + frm.grid(row=2, column=0, sticky="ew", padx=10, pady=6) + frm.columnconfigure(0, weight=1) + + self.lists_rows_frame = ttk.Frame(frm) + self.lists_rows_frame.grid(row=0, column=0, sticky="ew") + ttk.Button(frm, text="+ Add List", command=self._on_add_list).grid(row=1, column=0, sticky="w", pady=(6, 0)) + self._list_row_widgets: dict[str, dict] = {} + self._refresh_lists_ui() + + def _on_add_list(self): + name = simpledialog.askstring("New List", "List name:", parent=self) + if not name: + return + name = name.strip() + if not name or name in self.lists: + messagebox.showerror("Invalid Name", "Enter a unique, non-empty list name.", parent=self) + return + self.lists[name] = TaskList(values=[], format="comma") + self._refresh_lists_ui() + + def _refresh_lists_ui(self): + for w in self.lists_rows_frame.winfo_children(): + w.destroy() + self._list_row_widgets.clear() + + for r, name in enumerate(self.lists): + lst = self.lists[name] + ttk.Label(self.lists_rows_frame, text=name, width=14).grid(row=r, column=0, sticky="w", padx=(0, 6), pady=3) + + values_var = tk.StringVar(value=", ".join(lst.values)) + entry = ttk.Entry(self.lists_rows_frame, textvariable=values_var, width=36) + entry.grid(row=r, column=1, sticky="ew", padx=4) + + ttk.Button(self.lists_rows_frame, text="Import…", width=9, + command=lambda n=name: self._on_import_list(n)).grid(row=r, column=2, padx=4) + + format_var = tk.StringVar(value=lst.format) + fmt_cmb = ttk.Combobox(self.lists_rows_frame, textvariable=format_var, values=LIST_FORMATS, + state="readonly", width=8) + fmt_cmb.grid(row=r, column=3, padx=4) + + preview_var = tk.StringVar() + ttk.Label(self.lists_rows_frame, textvariable=preview_var, foreground="#555", width=28).grid( + row=r, column=4, sticky="w", padx=4) + + ttk.Button(self.lists_rows_frame, text="Remove", width=8, + command=lambda n=name: self._on_remove_list(n)).grid(row=r, column=5, padx=4) + + def _sync(*_args, n=name, vv=values_var, fv=format_var, pv=preview_var): + values = [v.strip() for v in vv.get().split(",") if v.strip()] + new_list = TaskList(values=values, format=fv.get()) + self.lists[n] = new_list + pv.set(new_list.render() if values else "(empty)") + self._highlight_placeholders(self.prompt_edit) + if self.system_var.get(): + self._highlight_placeholders(self.system_edit) + + values_var.trace_add("write", _sync) + format_var.trace_add("write", _sync) + _sync() + + self._list_row_widgets[name] = {"values_var": values_var, "format_var": format_var} + + self.lists_rows_frame.columnconfigure(1, weight=1) + self._refresh_column_pills() + self._refresh_output_field_list_choices() + + def _on_import_list(self, name: str): + result = import_data(self, f"Import values for list '{name}'") + if result is None: + return + values, _nickname = result + self._list_row_widgets[name]["values_var"].set(", ".join(values)) + + def _on_remove_list(self, name: str): + self.lists.pop(name, None) + self._refresh_lists_ui() + + # ----------------------------------------------------------------- + # Column + list pill palette (shared between Input and Lists sections) + # ----------------------------------------------------------------- + def _refresh_column_pills(self): + items = [(c, "column") for c in self.columns] + [(n, "list") for n in self.lists] + self.column_pills.set_pills(items) + + # ----------------------------------------------------------------- + # Output fields section + # ----------------------------------------------------------------- + def _build_output_fields_section(self, root): + frm = ttk.LabelFrame(root, text="Output Fields", padding=10) + frm.grid(row=3, column=0, sticky="ew", padx=10, pady=6) + frm.columnconfigure(0, weight=1) + + self.fields_rows_frame = ttk.Frame(frm) + self.fields_rows_frame.grid(row=0, column=0, sticky="ew") + ttk.Button(frm, text="+ Add Field", command=self._on_add_field).grid(row=1, column=0, sticky="w", pady=(6, 0)) + self._refresh_output_fields_ui() + + def _on_add_field(self): + self._field_rows.append({ + "name_var": tk.StringVar(value=f"field{len(self._field_rows) + 1}"), + "type_var": tk.StringVar(value="free_text"), + "list_var": tk.StringVar(value=""), + }) + self._refresh_output_fields_ui() + + def _refresh_output_fields_ui(self): + if not hasattr(self, "fields_rows_frame"): + return # Lists section builds before Output Fields section exists yet + for w in self.fields_rows_frame.winfo_children(): + w.destroy() + + for r, row in enumerate(self._field_rows): + ttk.Entry(self.fields_rows_frame, textvariable=row["name_var"], width=16).grid( + row=r, column=0, sticky="w", padx=(0, 6), pady=3) + + type_cmb = ttk.Combobox(self.fields_rows_frame, textvariable=row["type_var"], values=FIELD_TYPES, + state="readonly", width=13) + type_cmb.grid(row=r, column=1, padx=4) + + list_cmb = ttk.Combobox(self.fields_rows_frame, textvariable=row["list_var"], + values=list(self.lists), state="readonly", width=14) + list_cmb.grid(row=r, column=2, padx=4) + + def _sync_list_visibility(_e=None, t=row["type_var"], w=list_cmb): + w.configure(state="readonly" if t.get() in ("choice", "multi_choice") else "disabled") + + type_cmb.bind("<>", _sync_list_visibility) + _sync_list_visibility() + + ttk.Button(self.fields_rows_frame, text="Remove", width=8, + command=lambda i=r: self._on_remove_field(i)).grid(row=r, column=3, padx=4) + + self.fields_rows_frame.columnconfigure(0, weight=1) + + def _refresh_output_field_list_choices(self): + # Rebuild so each row's list combobox reflects current list names. + self._refresh_output_fields_ui() + + def _on_remove_field(self, index: int): + del self._field_rows[index] + self._refresh_output_fields_ui() + + def _collect_output_fields(self) -> list[OutputField]: + fields = [] + for row in self._field_rows: + name = row["name_var"].get().strip() + if not name: + continue + ftype = row["type_var"].get() + list_ref = row["list_var"].get().strip() or None + if ftype not in ("choice", "multi_choice"): + list_ref = None + fields.append(OutputField(name=name, type=ftype, list_ref=list_ref)) + return fields + + # ----------------------------------------------------------------- + # Carry-to-output section + # ----------------------------------------------------------------- + def _build_carry_section(self, root): + frm = ttk.LabelFrame(root, text="Carry to Output", padding=10) + frm.grid(row=4, column=0, sticky="ew", padx=10, pady=6) + ttk.Label(frm, text="Input columns to include untouched in the output CSV " + "(e.g. an existing ID column) -- independent of whether " + "they're used in the prompt.").grid(row=0, column=0, sticky="w") + self.carry_checks_frame = ttk.Frame(frm) + self.carry_checks_frame.grid(row=1, column=0, sticky="ew", pady=(6, 0)) + + def _refresh_carry_ui(self): + for w in self.carry_checks_frame.winfo_children(): + w.destroy() + for i, (col, var) in enumerate(self.carry_vars.items()): + row, colpos = divmod(i, 5) + ttk.Checkbutton(self.carry_checks_frame, text=col, variable=var).grid( + row=row, column=colpos, sticky="w", padx=6, pady=2) + + # ----------------------------------------------------------------- + # Model: which model runs this task, and its inference params. Standard + # models take temperature; reasoning models (o1/o3/o4/gpt-5) take a + # reasoning effort instead -- the API rejects whichever doesn't apply, so + # the UI shows only the one that does. + # ----------------------------------------------------------------- + def _build_model_section(self, root): + frm = ttk.LabelFrame(root, text="Model", padding=10) + frm.grid(row=5, column=0, sticky="ew", padx=10, pady=6) + + ttk.Label(frm, text="Model:").grid(row=0, column=0, sticky="w") + self.model_var = tk.StringVar(value=DEFAULT_MODEL) + self.cmb_model = ttk.Combobox( + frm, textvariable=self.model_var, values=get_models(), state="readonly", width=28) + self.cmb_model.grid(row=0, column=1, sticky="w", padx=6) + self.cmb_model.bind("<>", lambda e: self._refresh_model_params_visibility()) + + ttk.Label(frm, text="Max output tokens:").grid(row=0, column=2, sticky="w", padx=(16, 0)) + self.max_tokens_var = tk.StringVar() + ttk.Entry(frm, textvariable=self.max_tokens_var, width=10).grid(row=0, column=3, sticky="w", padx=6) + + self.temp_label = ttk.Label(frm, text="Temperature:") + self.temperature_var = tk.StringVar() + self.temp_entry = ttk.Entry(frm, textvariable=self.temperature_var, width=10) + + self.reasoning_label = ttk.Label(frm, text="Reasoning effort:") + self.reasoning_var = tk.StringVar() + self.reasoning_combo = ttk.Combobox( + frm, textvariable=self.reasoning_var, values=REASONING_EFFORTS, state="readonly", width=10) + + self._refresh_model_params_visibility() + + def _refresh_model_params_visibility(self): + if is_reasoning_model(self.model_var.get()): + self.temp_label.grid_remove() + self.temp_entry.grid_remove() + self.reasoning_label.grid(row=1, column=0, sticky="w", pady=(6, 0)) + self.reasoning_combo.grid(row=1, column=1, sticky="w", padx=6, pady=(6, 0)) + else: + self.reasoning_label.grid_remove() + self.reasoning_combo.grid_remove() + self.temp_label.grid(row=1, column=0, sticky="w", pady=(6, 0)) + self.temp_entry.grid(row=1, column=1, sticky="w", padx=6, pady=(6, 0)) + + # ----------------------------------------------------------------- + # Run / Save / Load + # ----------------------------------------------------------------- + def _build_run_section(self, root): + frm = ttk.Frame(root, padding=10) + frm.grid(row=6, column=0, sticky="ew", padx=10, pady=(6, 10)) + ttk.Button(frm, text="Load Task…", command=self._on_load_task).grid(row=0, column=0, padx=4) + ttk.Button(frm, text="Save Task…", command=self._on_save_task).grid(row=0, column=1, padx=4) + ttk.Button(frm, text="Run Live", command=self._on_run_live).grid(row=0, column=2, padx=(24, 4)) + ttk.Button(frm, text="Submit Batch", style="Accent.TButton", command=self._on_submit_batch).grid( + row=0, column=3, padx=4) + + def _collect_task(self) -> Task: + prompt = self.prompt_edit.get("1.0", "end-1c") + system = self.system_edit.get("1.0", "end-1c").strip() if self.system_var.get() else None + carry_columns = [c for c, v in self.carry_vars.items() if v.get()] + return Task( + prompt=prompt, + system=system or None, + lists=dict(self.lists), + output_fields=self._collect_output_fields(), + carry_columns=carry_columns, + model=self.model_var.get().strip() or DEFAULT_MODEL, + temperature=_parse_optional_float(self.temperature_var.get()), + reasoning_effort=self.reasoning_var.get().strip() or None, + max_output_tokens=_parse_optional_int(self.max_tokens_var.get()), + ) + + def _validate_and_collect(self) -> Task | None: + if self.df is None: + messagebox.showerror("Task Error", "Import data first.", parent=self) + return None + task = self._collect_task() + try: + validate_task(task, self.columns) + except TaskValidationError as e: + messagebox.showerror("Task Error", str(e), parent=self) + return None + return task + + def _on_run_live(self): + task = self._validate_and_collect() + if task is None: + return + run_task_live(self, task, self.df) + + def _on_submit_batch(self): + task = self._validate_and_collect() + if task is None: + return + df = self.df + dataset = Path(self.current_file).stem if self.current_file else None + + def _worker(): + try: + batch_method.send_batch(task, df, dataset=dataset) + self.after(0, self._on_batch_submitted) + except Exception as e: + self.after(0, lambda: messagebox.showerror("Batch Error", str(e), parent=self)) + + threading.Thread(target=_worker, daemon=True).start() + + def _on_batch_submitted(self): + messagebox.showinfo("Batch Submitted", "Batch job submitted. Check the main window's Batches list.", parent=self) + try: + from ui.batch_operations import refresh_batches_async + refresh_batches_async(self.master) + except Exception: + pass + + def _on_save_task(self): + task = self._collect_task() + path = filedialog.asksaveasfilename( + parent=self, title="Save Task", defaultextension=".json", + filetypes=[("Task JSON", "*.json"), ("All files", "*.*")], + ) + if not path: + return + try: + Path(path).write_text(json.dumps(task.to_dict(), indent=2, ensure_ascii=False), encoding="utf-8") + messagebox.showinfo("Task Saved", f"Saved to {path}", parent=self) + except OSError as e: + messagebox.showerror("Save Error", str(e), parent=self) + + def _on_load_task(self): + path = filedialog.askopenfilename( + parent=self, title="Load Task", + filetypes=[("Task JSON", "*.json"), ("All files", "*.*")], + ) + if not path: + return + try: + data = json.loads(Path(path).read_text(encoding="utf-8")) + task = Task.from_dict(data) + except (OSError, json.JSONDecodeError, KeyError) as e: + messagebox.showerror("Load Error", str(e), parent=self) + return + self._apply_task(task) + + def _apply_task(self, task: Task): + self.prompt_edit.delete("1.0", "end") + self.prompt_edit.insert("1.0", task.prompt) + + self.system_var.set(bool(task.system)) + self._toggle_system_box() + self.system_edit.delete("1.0", "end") + if task.system: + self.system_edit.insert("1.0", task.system) + + self.lists = dict(task.lists) + self._refresh_lists_ui() + + self._field_rows = [ + { + "name_var": tk.StringVar(value=f.name), + "type_var": tk.StringVar(value=f.type), + "list_var": tk.StringVar(value=f.list_ref or ""), + } + for f in task.output_fields + ] + self._refresh_output_fields_ui() + + if self.df is not None: + for c, v in self.carry_vars.items(): + v.set(c in task.carry_columns) + + self.model_var.set(task.model) + self.temperature_var.set("" if task.temperature is None else str(task.temperature)) + self.reasoning_var.set(task.reasoning_effort or "") + self.max_tokens_var.set("" if task.max_output_tokens is None else str(task.max_output_tokens)) + self._refresh_model_params_visibility() + + self._highlight_placeholders(self.prompt_edit) + if self.system_var.get(): + self._highlight_placeholders(self.system_edit) diff --git a/ui/ui_helpers.py b/ui/ui_helpers.py index d4cdacf..9d73b13 100644 --- a/ui/ui_helpers.py +++ b/ui/ui_helpers.py @@ -8,36 +8,6 @@ import tkinter as tk from tkinter import ttk -# Constants for UI components -TABLE_HEIGHT_ROWS = 12 - - -def make_tab_with_tree(parent_frame: ttk.Frame) -> tuple[ttk.Frame, ttk.Treeview]: - """ - Create a Treeview widget with horizontal scrollbar inside a tab frame. - - Args: - parent_frame: The parent frame to contain the tree and scrollbar - - Returns: - Tuple of (tab_inner_frame, treeview_widget) - """ - tab_inner = ttk.Frame(parent_frame) - tab_inner.columnconfigure(0, weight=1) - - columns = ("id", "status", "created_at", "model", "type", "dataset(s)") - tree = ttk.Treeview(tab_inner, columns=columns, show="headings", height=TABLE_HEIGHT_ROWS) - for c in columns: - tree.heading(c, text=c) - tree.column(c, width=tab_inner.winfo_width()//len(columns), anchor="w") - - h_scroll = ttk.Scrollbar(tab_inner, orient="horizontal", command=tree.xview) - tree.configure(xscrollcommand=h_scroll.set) - - tree.grid(row=0, column=0, sticky="ew") - h_scroll.grid(row=1, column=0, sticky="ew") - return tab_inner, tree - def popup_menu(event: tk.Event, tree: ttk.Treeview, menu: tk.Menu) -> None: """ diff --git a/ui/ui_utils.py b/ui/ui_utils.py index a8571f9..b90924e 100644 --- a/ui/ui_utils.py +++ b/ui/ui_utils.py @@ -30,6 +30,9 @@ def populate_treeview(tree: ttk.Treeview, columns: tuple[str, ...], rows: list[t """ Configure and populate a Treeview widget with tabular data. + Used by the Task Builder's data preview table (ui/task_builder.py). The + batches table has its own rendering now (see ui/batches_view.py). + Args: tree: The Treeview widget to populate columns: Tuple of column header names diff --git a/wiki/LLMTools/BatchMethods/BatchMethods.md b/wiki/LLMTools/BatchMethods/BatchMethods.md index 62e2399..2078cfd 100644 --- a/wiki/LLMTools/BatchMethods/BatchMethods.md +++ b/wiki/LLMTools/BatchMethods/BatchMethods.md @@ -1,9 +1,7 @@ -# Batch Methods +# Batch Methods (moved) -## Tools +The separate Single-Label / Multi-Label / Keyword Extraction batch tools have been replaced by the **Task Builder**, where you build one task and choose "Submit Batch" to run it via OpenAI's Batch API. -- [Single Label Classification](./SingleLabelClassification.md) - Tool for classifying text data into one of several predefined categories using a language model. -- [Multi-Label Classification](./MultiLabelClassification.md) - Tool for classifying text data into one or more of several predefined categories using a language model. -- [Keyword Extraction](./KeywordExtraction.md) - Tool for extracting key phrases or terms from text data using a language model. +See [LLM Tools](../LLMTools.md) for the current workflow. The old presets (LLM Tools > Presets) still give you the same starting points, but now as an editable prompt instead of a fixed tool. ---- \ No newline at end of file +--- diff --git a/wiki/LLMTools/BatchMethods/KeywordExtraction.md b/wiki/LLMTools/BatchMethods/KeywordExtraction.md index a134f91..3887bda 100644 --- a/wiki/LLMTools/BatchMethods/KeywordExtraction.md +++ b/wiki/LLMTools/BatchMethods/KeywordExtraction.md @@ -1,21 +1,7 @@ -# Keyword Extraction Tool (Batch) +# Keyword Extraction Tool (Batch) -- moved -## Accessing Keyword Extraction Tool (Batch) -**Navigate**: LLM Tools > Batch Methods > Keyword Extraction +This fixed tool has been replaced by the **Task Builder**: LLM Tools > Presets > Keyword Extraction opens the same starting point as an editable prompt, then choose "Submit Batch" to run it. -## Keyword Extraction Purpose -The Keyword Extraction tool allows you to extract key phrases or terms from text data using a language model. This is useful for tasks such as summarizing content, identifying important topics, or enhancing search functionality by highlighting relevant keywords. -This is the batch version and thus will take up to 24 hours to process using the OpenAI API. -You can check the progress by refresh the Batches table on the main window. +See [LLM Tools](../LLMTools.md) for the current workflow, including how output fields work. -## Using the Keyword Extraction Tool -1. The import wizard will appear once to request the text data. -2. Select which column contains the text data you want to extract keywords from. -3. The Dataset field is used to name your batch elements for easy identification later on the Batches table on the main window. - -Accepted file types: csv, tsv, Excel, and Parquet - -## Batch Pricing -For batch pricing by model see OpenAI's [pricing page](https://platform.openai.com/docs/pricing). - ---- \ No newline at end of file +--- diff --git a/wiki/LLMTools/BatchMethods/MultiLabelClassification.md b/wiki/LLMTools/BatchMethods/MultiLabelClassification.md index d5060ef..3eb6976 100644 --- a/wiki/LLMTools/BatchMethods/MultiLabelClassification.md +++ b/wiki/LLMTools/BatchMethods/MultiLabelClassification.md @@ -1,21 +1,7 @@ -# Multi-Label Classification Tool (Batch) +# Multi-Label Classification Tool (Batch) -- moved -## Accessing Multi-Label Classification Tool (Batch) -**Navigate**: LLM Tools > Batch Methods > Multi-Label Classification +This fixed tool has been replaced by the **Task Builder**: LLM Tools > Presets > Multi-Label Classification opens the same starting point as an editable prompt, then choose "Submit Batch" to run it. -## Multi-Label Classification Purpose -The Multi-Label Classification tool allows you to classify text data into one or more of several predefined categories using a language model. This is useful for tasks such as thematic analysis, topic categorization, or any scenario where you need to assign a one or more labels to each piece of text based on its content. -This is the batch version and thus will take up to 24 hours to process using the OpenAI API. -You can check the progress by refresh the Batches table on the main window. +See [LLM Tools](../LLMTools.md) for the current workflow, including how Lists (your labels) and output fields work. -## Using the Multi-Label Classification Tool -1. The import wizard will appear twice: first to request the labels and second to request the text data. -2. For each import, select which column you want to use for the labels and text data, respectively. -3. The Dataset field is used to name your batch elements for easy identification later on the Batches table on the main window. - -Accepted file types: csv, tsv, Excel, and Parquet - -## Batch Pricing -For batch pricing by model see OpenAI's [pricing page](https://platform.openai.com/docs/pricing). - ---- \ No newline at end of file +--- diff --git a/wiki/LLMTools/BatchMethods/SingleLabelClassification.md b/wiki/LLMTools/BatchMethods/SingleLabelClassification.md index f26fa0b..5bf8d84 100644 --- a/wiki/LLMTools/BatchMethods/SingleLabelClassification.md +++ b/wiki/LLMTools/BatchMethods/SingleLabelClassification.md @@ -1,21 +1,7 @@ -# Single Label Classification Tool (Batch) +# Single Label Classification Tool (Batch) -- moved -## Accessing Single Label Classification Tool (Batch) -**Navigate**: LLM Tools > Batch Methods > Single Label Classification +This fixed tool has been replaced by the **Task Builder**: LLM Tools > Presets > Single-Label Classification opens the same starting point as an editable prompt, then choose "Submit Batch" to run it. -## Single Label Classification Purpose -The Single Label Classification tool allows you to classify text data into one of several predefined categories using a language model. This is useful for tasks such as sentiment analysis, topic categorization, or any scenario where you need to assign a single label to each piece of text based on its content. -This is the batch version and thus will take up to 24 hours to process using the OpenAI API. -You can check the progress by refresh the Batches table on the main window. +See [LLM Tools](../LLMTools.md) for the current workflow, including how Lists (your labels) and output fields work. -## Using the Single Label Classification Tool -1. The import wizard will appear twice: first to request the labels and second to request the text data. -2. For each import, select which column you want to use for the labels and text data, respectively. -3. The Dataset field is used to name your batch elements for easy identification later on the Batches table on the main window. - -Accepted file types: csv, tsv, Excel, and Parquet - -## Batch Pricing -For batch pricing by model see OpenAI's [pricing page](https://platform.openai.com/docs/pricing). - ---- \ No newline at end of file +--- diff --git a/wiki/LLMTools/LLMTools.md b/wiki/LLMTools/LLMTools.md index ad95ae8..83887f9 100644 --- a/wiki/LLMTools/LLMTools.md +++ b/wiki/LLMTools/LLMTools.md @@ -1,29 +1,38 @@ # LLM Tools -## Tools +## Accessing the Task Builder +**Navigate**: LLM Tools > New Task (blank) or LLM Tools > Presets > *a starting point* -- [Live Methods](./LiveMethods/LiveMethods.md) - Real-time API calls for quick responses. -- [Batch Methods](./BatchMethods/BatchMethods.md) - Cost-effective processing for large datasets. +CodebookAI no longer ships six separate fixed tools. Instead there is one **Task Builder** window where you define exactly what you want the model to do, then run it either live or as a batch. The old Single-Label, Multi-Label, and Keyword Extraction tools are now **presets** -- one-click starting points that pre-fill the builder with an editable prompt and output shape, rather than separate menu items. -## Overview +## What a Task is -When working with OpenAI's API there are two primary modes of operation: live and batch. Each mode has its own advantages and disadvantages, and the choice between them depends on the specific use case and requirements. +A task has four parts, all editable in the builder: -## Live Mode -In live mode, requests are sent to the API in real-time, and responses are received quickly. However, it is more expensive than batch mode. -Useful for: -- Small datasets -- Quick responses -- Testing and experimentation before batching +- **Input** -- import one table (CSV/TSV/TXT/Excel). Every column becomes available as a `{placeholder}`. +- **Prompt** -- your own text. Anywhere you write `{column_name}`, that column's value is substituted per row; anywhere you write `{list_name}`, a constant value set is substituted (the same every row). An optional, hidden-by-default "system instructions" box is available for advanced use. Toggle **Preview** to see the exact rendered prompt for any row before you run anything. +- **Lists** -- named, reusable value sets (e.g. your labels), either typed directly (`positive, negative, neutral`) or imported from a file column. A list can be dropped into the prompt *and* bound to an output field at the same time, so the values shown to the model and the values it's constrained to are always identical -- edit the list once, both places update. +- **Output fields** -- the shape of the response, and the columns of your result CSV. Each field has a type: + - **Choice** -- exactly one value from a List (this is what enforces "never a hallucinated label" -- a strict schema, not free text) + - **Multi-choice** -- one or more values from a List + - **Text list** -- free-form list of strings (e.g. extracted keywords) + - **Free text** -- a single free-form string (e.g. a one-line justification) + - **Number** / **Yes-No** -- numeric or boolean output -## Batch Mode -In batch mode, requests are grouped together and sent to the API in a single call. This is more cost-effective for large datasets but takes up to 24 hours to process. -Useful for: -- Large datasets -- Cost efficiency -- Non-urgent processing + You can define more than one output field per run -- e.g. a label *and* a free-text reason in the same request. +- **Carry to output** -- input columns that ride into the result CSV untouched, independent of whether they're used in the prompt. This is how an existing ID column follows your results back out even though it's never shown to the model. + +## Live vs. Batch + +Once a task is built, run it either way from the same window: + +### Live +Requests are sent one at a time and you see results immediately. More expensive per request. Useful for small datasets, quick checks, and testing a task before committing to a full batch run. + +### Batch +All requests are submitted together via OpenAI's Batch API and processed within 24 hours. Substantially cheaper for large datasets. Check progress in the Batches table on the main window; download results once the job is marked Done. ## Pricing For pricing by model see OpenAI's [pricing page](https://platform.openai.com/docs/pricing). ---- \ No newline at end of file +--- diff --git a/wiki/LLMTools/LiveMethods/KeywordExtraction.md b/wiki/LLMTools/LiveMethods/KeywordExtraction.md index 94abd5c..d26b779 100644 --- a/wiki/LLMTools/LiveMethods/KeywordExtraction.md +++ b/wiki/LLMTools/LiveMethods/KeywordExtraction.md @@ -1,20 +1,7 @@ -# Keyword Extraction Tool (Live) +# Keyword Extraction Tool (Live) -- moved -## Accessing Keyword Extraction Tool (Live) -**Navigate**: LLM Tools > Live Methods > Keyword Extraction +This fixed tool has been replaced by the **Task Builder**: LLM Tools > Presets > Keyword Extraction opens the same starting point as an editable prompt, then choose "Run Live" to run it. -## Keyword Extraction Purpose -The Keyword Extraction tool allows you to extract key phrases or terms from text data using a language model. This is useful for tasks such as summarizing content, identifying important topics, or enhancing search functionality by highlighting relevant keywords. -This is the live version and thus will process data in real-time using the OpenAI API. +See [LLM Tools](../LLMTools.md) for the current workflow, including how output fields work. -## Using the Keyword Extraction Tool -1. The import wizard will appear once to request the text data. -2. Select which column contains the text data you want to extract keywords from. -3. The Dataset field is not used for this tool, so you can ignore it. - -Accepted file types: csv, tsv, Excel, and Parquet - -## Live Pricing -For live pricing by model see OpenAI's [pricing page](https://platform.openai.com/docs/pricing). - ---- \ No newline at end of file +--- diff --git a/wiki/LLMTools/LiveMethods/LiveMethods.md b/wiki/LLMTools/LiveMethods/LiveMethods.md index d4399cb..a7ffd35 100644 --- a/wiki/LLMTools/LiveMethods/LiveMethods.md +++ b/wiki/LLMTools/LiveMethods/LiveMethods.md @@ -1,9 +1,7 @@ -# Live Methods +# Live Methods (moved) -## Tools +The separate Single-Label / Multi-Label / Keyword Extraction live tools have been replaced by the **Task Builder**, where you build one task and choose "Run Live" to run it synchronously with immediate results. -- [Single Label Classification](./SingleLabelClassification.md) - Tool for classifying text data into one of several predefined categories using a language model. -- [Multi-Label Classification](./MultiLabelClassification.md) - Tool for classifying text data into one or more of several predefined categories using a language model. -- [Keyword Extraction](./KeywordExtraction.md) - Tool for extracting key phrases or terms from text data using a language model. +See [LLM Tools](../LLMTools.md) for the current workflow. The old presets (LLM Tools > Presets) still give you the same starting points, but now as an editable prompt instead of a fixed tool. ---- \ No newline at end of file +--- diff --git a/wiki/LLMTools/LiveMethods/MultiLabelClassification.md b/wiki/LLMTools/LiveMethods/MultiLabelClassification.md index 3850461..b735819 100644 --- a/wiki/LLMTools/LiveMethods/MultiLabelClassification.md +++ b/wiki/LLMTools/LiveMethods/MultiLabelClassification.md @@ -1,20 +1,7 @@ -# Multi-Label Classification Tool (Live) +# Multi-Label Classification Tool (Live) -- moved -## Accessing Multi-Label Classification Tool (Live) -**Navigate**: LLM Tools > Live Methods > Multi-Label Classification +This fixed tool has been replaced by the **Task Builder**: LLM Tools > Presets > Multi-Label Classification opens the same starting point as an editable prompt, then choose "Run Live" to run it. -## Multi-Label Classification Purpose -The Multi-Label Classification tool allows you to classify text data into one or more of several predefined categories using a language model. This is useful for tasks such as thematic analysis, topic categorization, or any scenario where you need to assign a one or more labels to each piece of text based on its content. -This is the live version and thus will process data in real-time using the OpenAI API. +See [LLM Tools](../LLMTools.md) for the current workflow, including how Lists (your labels) and output fields work. -## Using the Multi-Label Classification Tool -1. The import wizard will appear twice: first to request the labels and second to request the text data. -2. For each import, select which column you want to use for the labels and text data, respectively. -3. The Dataset field is not used for this tool, so you can ignore it. - -Accepted file types: csv, tsv, Excel, and Parquet - -## Live Pricing -For live pricing by model see OpenAI's [pricing page](https://platform.openai.com/docs/pricing). - ---- \ No newline at end of file +--- diff --git a/wiki/LLMTools/LiveMethods/SingleLabelClassification.md b/wiki/LLMTools/LiveMethods/SingleLabelClassification.md index 701d1ab..0da2e3c 100644 --- a/wiki/LLMTools/LiveMethods/SingleLabelClassification.md +++ b/wiki/LLMTools/LiveMethods/SingleLabelClassification.md @@ -1,20 +1,7 @@ -# Single Label Classification Tool (Live) +# Single Label Classification Tool (Live) -- moved -## Accessing Single Label Classification Tool (Live) -**Navigate**: LLM Tools > Live Methods > Single Label Classification +This fixed tool has been replaced by the **Task Builder**: LLM Tools > Presets > Single-Label Classification opens the same starting point as an editable prompt, then choose "Run Live" to run it. -## Single Label Classification Purpose -The Single Label Classification tool allows you to classify text data into one of several predefined categories using a language model. This is useful for tasks such as sentiment analysis, topic categorization, or any scenario where you need to assign a single label to each piece of text based on its content. -This is the live version and thus will process data in real-time using the OpenAI API. +See [LLM Tools](../LLMTools.md) for the current workflow, including how Lists (your labels) and output fields work. -## Using the Single Label Classification Tool -1. The import wizard will appear twice: first to request the labels and second to request the text data. -2. For each import, select which column you want to use for the labels and text data, respectively. -3. The Dataset field is not used for this tool, so you can ignore it. - -Accepted file types: csv, tsv, Excel, and Parquet - -## Live Pricing -For live pricing by model see OpenAI's [pricing page](https://platform.openai.com/docs/pricing). - ---- \ No newline at end of file +---