Redesign

.github/ISSUE_TEMPLATE/datasource_request.yml

@@ -0,0 +1,100 @@
+name: Data source request
+description: Request a new data source for 4CAT.
+title: "[Data source] "
+labels: ["module-request", "data source"]
+body:
+  - type: markdown
+    attributes:
+      value: |
+        Before you start:
+
+        - Check the [list of available data sources](https://github.com/digitalmethodsinitiative/4cat/wiki) and the open
+          [data source requests](https://github.com/digitalmethodsinitiative/4cat/issues?q=is%3Aissue+label%3A%22data+source%22)
+          first.
+        - Request new [Zeeschuimer data sources on its own GitHub]
+        (https://github.com/digitalmethodsinitiative/zeeschuimer) instead of here.
+
+  - type: input
+    id: datasource
+    attributes:
+      label: Data source name
+      description: The name of the datasource (platform, upload, etc) you want to collect data from.
+      placeholder: e.g. 'BlueSky'
+    validations:
+      required: true
+
+  - type: input
+    id: datasource_url
+    attributes:
+      label: URL of data source
+      placeholder: https://...
+    validations:
+      required: false
+
+  - type: dropdown
+    id: access
+    attributes:
+      label: How is the data reachable?
+      description: Your best guess is fine — say so in the details below if you are unsure.
+      options:
+        - It has a documented public API
+        - It has an API, but it needs approval, an account, or payment
+        - No API — the data is only visible in the browser
+        - It offers a bulk export or data archive
+        - I don't know
+    validations:
+      required: true
+
+  - type: textarea
+    id: access_details
+    attributes:
+      label: API documentation and access details
+      description: >
+        Link to the API documentation if there is any. Does it need credentials, an approved research application, or a
+        paid plan? Are there rate limits or caps you know of? If there is no API, describe how you currently get at the
+        data.
+    validations:
+      required: false
+
+  - type: textarea
+    id: items
+    attributes:
+      label: What should one item in the dataset be?
+      description: >
+        4CAT datasets are tables of items. Say what a single item should be for this platform (a post, a comment, a
+        video, a profile, …) and which fields matter to you — timestamp, author, body text, engagement counts, media
+        URLs, and so on.
+    validations:
+      required: true
+
+  - type: textarea
+    id: use_case
+    attributes:
+      label: What would you use this for?
+      description: >
+        A sentence or two on the research you have in mind. This helps us judge which fields and query options actually
+        need to exist.
+    validations:
+      required: true
+
+  - type: textarea
+    id: sample
+    attributes:
+      label: Example data
+      description: >
+        If you can share a small sample of the raw data (an API response, an exported file), paste or attach it here.
+        This is the single most useful thing you can add. Remove anything personal or sensitive first.
+    validations:
+      required: false
+
+  - type: checkboxes
+    id: acknowledgements
+    attributes:
+      label: Before submitting
+      options:
+        - label: >
+            I have considered whether collecting this data is compatible with the platform's terms of service and with
+            the privacy of the people in it, and I am not asking for a way to circumvent access restrictions.
+          required: true
+        - label: I searched the existing issues and this data source has not been requested yet.
+          required: true

.github/ISSUE_TEMPLATE/processor_request.yml

@@ -0,0 +1,92 @@
+name: Processor request
+description: Ask for a new analysis step that can be run on a 4CAT dataset.
+title: "[Processor] "
+labels: ["module-request", "processor"]
+body:
+  - type: markdown
+    attributes:
+      value: |
+        Before you start:
+
+        - Check the [list of available processors](https://github.com/digitalmethodsinitiative/4cat/wiki/Available-processors)
+          and the open [processor requests](https://github.com/digitalmethodsinitiative/4cat/issues?q=is%3Aissue+label%3Aprocessor)
+          first — it may already exist.
+        - Processors take one dataset and produce another. If what you want is a different way of *collecting* data,
+          open a data source request instead.
+        - If you want to write it yourself, see
+          [How to make a processor](https://github.com/digitalmethodsinitiative/4cat/wiki/How-to-make-a-processor). It
+          does not have to live in 4CAT itself — processors can also be distributed as an extension.
+        - The clearer the input and output below, the more likely someone can pick this up and build it.
+
+  - type: textarea
+    id: what
+    attributes:
+      label: What should it do?
+      description: >
+        One or two sentences, starting with a verb — the way a processor describes itself in the interface.
+      placeholder: e.g. Calculate the lexical diversity of each post and add it as a column.
+    validations:
+      required: true
+
+  - type: textarea
+    id: input
+    attributes:
+      label: What should it run on?
+      description: >
+        Which datasets should this be available for — all csv datasets, any dataset with text, only datasets from a 
+        specific data source, only the output of another processor? Name that processor if so.
+    validations:
+      required: true
+
+  - type: textarea
+    id: output
+    attributes:
+      label: What should it produce?
+      description: >
+        The output format (CSV, NDJSON, an image, a network file, …) and what is in it. For a table, list the columns
+        and what one row represents.
+    validations:
+      required: true
+
+  - type: textarea
+    id: options
+    attributes:
+      label: Options
+      description: >
+        Anything the user should be able to configure before running it — thresholds, columns to work on, a language, a
+        model to use. Include sensible defaults if you have them in mind.
+    validations:
+      required: false
+
+  - type: textarea
+    id: reference
+    attributes:
+      label: Reference implementation
+      description: >
+        A paper, method, library, script, or an equivalent feature in another tool. If there is code that already does
+        this, linking to it is the most useful thing you can add.
+    validations:
+      required: false
+
+  - type: dropdown
+    id: requirements
+    attributes:
+      label: Does it need anything beyond plain Python?
+      options:
+        - "No — it can be computed from the dataset itself"
+        - "A Python library that 4CAT does not ship yet"
+        - "A machine learning model (running locally)"
+        - "A GPU"
+        - "An external API or paid service"
+        - "I don't know"
+    validations:
+      required: true
+
+  - type: textarea
+    id: use_case
+    attributes:
+      label: What would you use this for?
+      description: >
+        A sentence or two on the research you have in mind. This helps us get the output shape right the first time.
+    validations:
+      required: true

AGENTS.md

@@ -160,6 +160,7 @@ Invocation (the plain `docker compose` command uses `docker-compose.yml`):
 - Define reusable Jinja2 components when patterns emerge, but avoid over-engineering for future reuse.
 - Views are organized by concern in `webtool/views/`. API endpoints are in `api_tool.py` and `api_standalone.py`.
 - Static assets go in `webtool/static/`; templates in `webtool/templates/`.
+- The project uses htmx `4.0.0-beta*`.
 
 ## Testing Expectations
 - Run tests with `pytest` from the repo root. Config is in `pytest.ini`.

backend/lib/processor.py

@@ -1,6 +1,7 @@
 """
 Basic post-processor worker - should be inherited by workers to post-process results
 """
+from dataclasses import dataclass, field
 import traceback
 import inspect as py_inspect
 import zipfile
@@ -30,6 +31,55 @@
 # Shared instance for the legacy default `compatibility`
 _DEFAULT_COMPATIBILITY = Compatibility(top_dataset_only=True)
 
+@dataclass
+class ProcessorDescription:
+    """
+    A processor's user-facing description: the information shown about it in
+    the web interface. A processor can declare one of these directly (as its
+    `description` attribute) or declare the individual attributes; either way
+    the result is available via `Processor.get_description()`.
+    """
+    title: str
+    description: str
+    category: str = ""  # for backwards compatability
+    tags: typing.List[str] = field(default_factory=list)
+    references: typing.List[str] = field(default_factory=list)
+    info: typing.List[str] = field(default_factory=list)
+    warnings: typing.List[str] = field(default_factory=list)
+    icon: str = ""
+
+    def __post_init__(self):
+        if self.category:
+            self.category = self.category[0].upper() + self.category[1:]
+        self.tags = [tag.strip() for tag in self.tags]
+
+        # `category` is kept as the first entry of `tags` (as its lower-case
+        # tag form), so 4CAT can move to tags (which allow several per processor
+        # and can be filtered on) while `category` keeps working. Derive
+        # whichever is missing; when both are given, make sure the category
+        # leads the tag list.
+        category_tag = self.category.lower()
+        if self.category and not self.tags:
+            self.tags = [category_tag]
+        elif self.tags and not self.category:
+            self.category = self.tags[0][0].upper() + self.tags[0][1:]
+        elif self.category and self.tags:
+            self.tags = [category_tag] + [tag for tag in self.tags if tag != category_tag]
+
+
+class _DescriptionField:
+    """
+    Exposes one ProcessorDescription field as an attribute on the processor,
+    e.g. `Processor.title`. A plain `property` only runs on instance access;
+    this descriptor also runs on class access (`owner` is the class in both
+    cases), so `Processor.title` and `self.title` both return the value from
+    the processor's ProcessorDescription.
+    """
+    def __init__(self, name):
+        self.name = name
+
+    def __get__(self, obj, owner):
+        return getattr(owner._processor_description, self.name)
 
 class BasicProcessor(FourcatModule, BasicWorker, metaclass=abc.ABCMeta):
     """
@@ -81,11 +131,30 @@ def is_compatible_with(cls, module=None, config=None):
     #: The file that is being processed
     source_file = None
 
-    #: Processor description, which will be displayed in the web interface
-    description = "No description available"
-
-    #: Category identifier, used to group processors in the web interface
-    category = "Other"
+    #: The processor's user-facing description (title, category, description
+    #: text and references) as a single object. A processor may set this
+    #: directly, or set the individual attributes below; both are normalised
+    #: into `_processor_description` when the class is defined.
+    _processor_description = ProcessorDescription(
+        title="",
+        description="No description available",
+        references=[],
+        info=[],
+        warnings=[],
+        icon=""
+        )
+
+    #: Title, category, description text and references, read from the
+    #: processor's ProcessorDescription. Defined as descriptors so that both
+    #: `Processor.title` and `self.title` resolve to the stored value.
+    title = _DescriptionField("title")
+    category = _DescriptionField("category")
+    description = _DescriptionField("description")
+    references = _DescriptionField("references")
+    info = _DescriptionField("info")
+    warnings = _DescriptionField("warnings")
+    tags = _DescriptionField("tags")
+    icon = _DescriptionField("icon")
 
     #: Extension of the file created by the processor
     extension = "csv"
@@ -991,6 +1060,18 @@ def _validate_map_item_post_run(self):
             except Exception:
                 pass
 
+    @classmethod
+    def get_repo_link(cls, config):
+        """
+        Get a link to the processor's source code repository
+
+        :param ConfigManager config:  Configuration reader
+        :return str:  URL to the processor's source code repository
+        """
+        repo_url = config.get("4cat.github_url")
+        path = cls.filepath.replace("\\", "/").lstrip("/")
+        return f"{repo_url.rstrip('/')}/blob/master/{path}"
+
     @classmethod
     def is_compatible_with(cls, module=None, config=None):
         """
@@ -1026,10 +1107,10 @@ def is_filter(cls):
         Filters do not produce their own dataset but replace the source_dataset dataset
         instead.
 
-        :todo: Make this a bit more robust than sniffing the processor category
+        :todo: Make this a bit more robust than sniffing the processor tags
         :return bool:
         """
-        return (hasattr(cls, "category") and cls.category and "filter" in cls.category.lower()) or (hasattr(cls, "filter") and cls.filter)
+        return (hasattr(cls, "tags") and cls.tags and "filtering" in [tag.lower() for tag in cls.tags]) or (hasattr(cls, "filter") and cls.filter)
 
     @classmethod
     def get_options(cls, parent_dataset=None, config=None) -> dict:
@@ -1134,6 +1215,58 @@ def exclude_followup_processors(cls, processor_type=None):
             return True
         return False
 
+    def __init_subclass__(cls, **kwargs):
+        """
+        Normalise a processor's description when its class is defined.
+
+        A processor may declare its description either as a ProcessorDescription
+        object (assigned to `description`) or as the individual attributes
+        (title, category, description, references, info, warnings, icon, tags).
+        Either way it is folded into a single `_processor_description` object
+        here, and the raw attributes are removed so the descriptors on
+        BasicProcessor provide access to them.
+        """
+        super().__init_subclass__(**kwargs)
+
+        # the description inherited from the nearest ancestor (cls has none yet)
+        inherited = getattr(cls, "_processor_description", None)
+
+        # read the raw class-body value, bypassing the descriptor
+        declared = cls.__dict__.get("description")
+        if isinstance(declared, ProcessorDescription):
+            description = declared
+        else:
+            # build from the flat attributes, falling back to inherited values
+            # so a legacy subclass keeps anything an ancestor set
+            description = ProcessorDescription(
+                title=cls.__dict__.get("title", inherited.title),
+                category=cls.__dict__.get("category", inherited.category),
+                description=cls.__dict__.get("description", inherited.description),
+                references=list(cls.__dict__.get("references", inherited.references)),
+                info=list(cls.__dict__.get("info", inherited.info)),
+                warnings=list(cls.__dict__.get("warnings", inherited.warnings)),
+                # tags default to this class's category (see __post_init__), so
+                # don't inherit them — re-derive from the resolved category
+                tags=list(cls.__dict__.get("tags", [])),
+                icon=cls.__dict__.get("icon", inherited.icon),   
+            )
+
+        # remove raw attributes so the inherited descriptors govern access
+        for name in ("title", "category", "description", "references", "info", "warnings", "tags", "icon"):
+            if name in cls.__dict__:
+                delattr(cls, name)
+
+        cls._processor_description = description
+
+    @classmethod
+    def get_description(cls):
+        """
+        Get the processor's user-facing description
+
+        :return ProcessorDescription:  Description of this processor
+        """
+        return cls._processor_description
+
     @abc.abstractmethod
     def process(self):
         """
@@ -1162,4 +1295,4 @@ def is_preset():
 
         :return:  False
         """
-        return False
+        return False