Chapter10 - #35
Conversation
Morgan's additions to chapter 10
Includes segmentation as a running example.
| overseg_label = label(dilation(filled, disk(5))) | ||
| overseg_overlay = label2rgb(overseg_label, image=image, bg_label=0) | ||
|
|
||
| # 3. Missing cells: correct borders, but border cells and ~1/3 of interior cells removed |
There was a problem hiding this comment.
Can we add false positives too?
There was a problem hiding this comment.
And ig possible a split nucleus? These are examples that I briefly reference in the what metric to pick section
| For example, AP involves sorting predictions by confidence. | ||
| However, just like previously with MSE, we can measure them. | ||
| The practical workflow is therefore: choose a differentiable loss to train the model, then evaluate a richer set of metrics on held-out data to judge whether the trained model actually behaves the way you need it to. | ||
|
|
There was a problem hiding this comment.
Idea: do a part 2 of the segmentation examples figure above and calculate BCE, MSE, DICE, Hausdorff, and AP for each image after applying the error introducing function to a larger portion of the dataset
There was a problem hiding this comment.
Could also link to this https://metrics-reloaded.dkfz.de/metric-demo/segmentation
There was a problem hiding this comment.
The idea of showing the figure again is great, but if you'd prefer to leave it at the Metrics Reloaded link below, that is also okay with us.
|
|
||
| A single metric is rarely sufficient to fully evaluate a models performance as illustrated by @sec-cell-tracking-metrics. If multiple metrics are needed to fully evaluate a model, when do you run these metrics and on what data? As discussed in @sec-9.3.5.4, you will minimally compute a loss both on your training and validation sets during the course of training. Some metrics can be easily computed on the validation dataset and used to select a checkpoint for further testing. These metrics should be computationally inexpensive to compute in order to avoid slowing down the training process. Typically these would be metrics that are computed on a pixel-pixel level. In contrast, consider metrics that are computed on an object level such as those discussed in @sec-cell-tracking-metrics. Each cell tracking metric requires a preliminary stage in which each ground truth cell is matched to a predicted cell prior to evaluation. While there are many different ways to perform the matching step, it is ultimately a slow and expensive process. Metrics that rely on a matching stage are best reserved for running on the final test data split. @tbl-metric-stages elaborates on the stages of model evaluation and illustrates how metrics might be selected for a segmentation model. | ||
|
|
||
| ::: {#tbl-metric-stages .list-table} |
There was a problem hiding this comment.
@adjavon let me know what you think of this table. Not sure if it's helpful or if I should just elaborate more in the actual paragraph.
There was a problem hiding this comment.
We like the table! It could be helpful to give it a caption clarifying the columns (e.g., in what sense speed is being used).
|
@opp1231 @ScientistRachel We're ready for you to take a look! 🎉 Thank you for your patience! |
| In @sec-9, you learned about how to train a machine learning model. | ||
| This required selecting a loss function: a differentiable function that represents how far the model's output is from your desired ground truth. | ||
| The model's parameters were optimized to minimize the loss on a training dataset, and you evaluated the same loss on a held-out validation dataset to get an idea of how well your model would perform on new data. | ||
| As training relies on gradient descent, the loss must be differentiable: the algorithm needs the slope of the loss with respect to every parameter to know which direction to step. |
There was a problem hiding this comment.
We suggest moving the sentence on line 45 before the sentence on line 44.
|
|
||
| This loss penalizes the model based on how confident it is in the wrong answer. For ground truth label $y_i \in \{0,1\}$ and predicted probability $\hat{y}_i \in [0,1]$: | ||
| $$\text{BCE} = -\frac{1}{N} \sum_{i=1}^{N} \bigl[ y_i \log(\hat{y}_i) + (1-y_i)\log(1-\hat{y}_i) \bigr]$$ | ||
| BCE requires the output to be interpretable as a probability — it asks "is this pixel foreground or background?" and treats all wrongly-classified pixels the same way, whether they are at a cell boundary, inside a hole, or belong to an entirely missing cell. |
There was a problem hiding this comment.
We suggest moving this line in between the two sentences on line 178 to help with readability for those readers that are less familiar.
|
|
||
| If we are interested in counting the number of nuclei, a model which under ([@fig-segmentations]a) or over segments a nucleus ([@fig-segmentations]b) and therefore changes the size of the nucleus will not impact the analysis. | ||
| However a model that misses nuclei ([@fig-segmentations]c), merges multiple nuclei into one ([@fig-segmentations]b), or splits a single nucleus into multiple ([@fig-segmentations]c) would introduce errors into the final counts. | ||
| In this case, a metric built around per-instance detection, such as precision and recall (or F1-score) after matching predicted masks to ground truth, captures the problems that matter to the analysis. It will also reveal the false positives and false negatives would need correction. |
There was a problem hiding this comment.
This is the first introduction to precision and recall, which are not discussed much in other chapters. We feel it would be helpful to add a couple of sentences of explanation here to define those terms (which will also be added to the glossary).
| In this case a more appropriate metric would be pixel-overlap metrics like Intersection over Union (IoU) or the Dice coefficient. | ||
| A metric that captures this will look very different from BCE or MSE. | ||
| Different downstream uses call for different metrics. | ||
| Let's look at a few that are commonly used in bioimage analysis. |
There was a problem hiding this comment.
We suggest either (a) rewording this to make it clear that they need to open the boxes below or (b) having the three boxes with collapse="false."
| :::{.callout-note appearance="simple" icon=false collapse="true"} | ||
| # IoU and F1 are overlap metrics. | ||
|
|
||
| Both metrics quantify the overlap between a predicted region $B$ and the ground truth $A$, and range from 0 (no overlap) to 1 (perfect overlap): |
There was a problem hiding this comment.
| Both metrics quantify the overlap between a predicted region $B$ and the ground truth $A$, and range from 0 (no overlap) to 1 (perfect overlap): | |
| Both metrics quantify the overlap between a predicted region $B$ and the ground truth $A$, and range from 0 (no overlap) to 1 (perfect overlap). Overlap between is measured using the intersection ($$\cap$$) while the sum of the two areas is measured by the union ($$\cup$$). |
We suggest adding a defining sentence here to make the symbols clear; we made a suggestion but please feel free to edit.
| legend_elements = [ | ||
| Patch(facecolor=navy, alpha=0.5, label="A"), | ||
| Patch(facecolor=orange, alpha=0.5, label="B"), | ||
| Patch(facecolor=green, alpha=0.85, hatch="////", label="A ∩ B"), |
There was a problem hiding this comment.
Does it make sense to also include the union as an outer line or something like that?
| xytext=(A_pts[w, 0], A_pts[w, 1]), | ||
| arrowprops=dict(arrowstyle="<->", color=green, lw=2.0), zorder=3) | ||
| mid = (A_pts[w] + B_pts[nn[w]]) / 2 | ||
| ax.text(mid[0] + 0.1, mid[1], "$d_H$", color=green, fontsize=13, va="center") |
There was a problem hiding this comment.
This shows up obscured when we render it. It might be helpful to shift it down in y.
| When selecting a metric or metrics to use for evaluating a model, there are two key axes to consider. The first is the downstream analysis task that the data will be used for. The second is the time/cost needed to correct errors prior to that analysis task. In many cases the downstream analysis task will influence which types of errors can be tolerated. | ||
|
|
||
|
|
||
| This is why there is rarely one "correct" metric for a task like segmentation; the same set of predicted masks can look excellent by one measure and mediocre by another, and the right choice depends on tracing through how errors in the output propagate into errors in the analysis you actually care about. In some cases no single number can capture this tradeoff, and getting a picture of quality you can act on requires reporting several complementary metrics side by side, or splitting evaluation into stages that reflect distinct sources of error. This tension is especially visible in cell tracking, where a solution can be scored separately on how well it detects cells in each frame and on how well it links them across frames, and both pieces have to be weighed together to judge whether the result is fit for the application at hand. |
There was a problem hiding this comment.
We really like this paragraph!
| Choosing the right way to measure quality is therefore just as important as the training itself. | ||
|
|
||
| In this chapter we will look at how metrics and losses differ, how to pick measures that actually reflect your scientific goal, and how to use held-out test data to get an honest estimate of how your model will behave on new images. | ||
| We will focus on a *segmentation* as a concrete example, but the principles apply broadly. |
There was a problem hiding this comment.
| We will focus on a *segmentation* as a concrete example, but the principles apply broadly. | |
| We will focus on *segmentation* and *cell tracking* as concrete examples, but the principles apply broadly. |
|
|
||
| <!-- Tracking error types --> | ||
| ```{python} | ||
| #| label: fig-track-errors |
There was a problem hiding this comment.
We are trying to avoid using red-green in figures. Please check if this figure is accessible (e.g., using a tool like https://www.color-blindness.com/coblis-color-blindness-simulator/).
| Consider the three failure modes in @fig-sparse-edges. The two unified metrics produce very different rankings of solutions. The TRA score ranks no divisions (0.957) similarly to sparse edges (0.947) and far above missing nodes (0.732). In contrast, while all solutions score relatively low on CHOTA, the missing node solution ranks highest (0.560) followed by no divisions (0.333) and sparse edges (0.297). Because TRA and CHOTA weight node, edge and lineage errors differently, two established "single number" metrics can disagree about which solution is preferable. Neither score reveals whether the underlying problem is due to missed detections, broken links, or failure to find divisions. To identify the specific failure modes, we need to examine metrics for that we need to look at the node, edge and division metrics reported alongside them. | ||
|
|
||
| The per-component metrics reveal the distinct failure mode in each solution. The missing node solution (b) has the lowest node F1 (0.858) of the three because a quarter of detections are dropped; since each dropped node also severs the edges that pass through it, this single error mode depresses edge F1 (0.798) and division F1 (0.399) as well. The sparse edge (c) and no-division (d) solutions both keep every detection (node F1 = 1.000) and lose the same fraction of edges (edge F1 = 0.800), yet they are far from equivalent: sparse edges still recovers roughly half of the true divisions (division F1 = 0.505) because the deleted edges are scattered at random, while the no-division solution deterministically removes every division edge, so no division is ever proposed and division F1 is undefined. | ||
|
|
There was a problem hiding this comment.
@fig-sparse-edges might make more sense here, after the two paragraphs that describe it.
|
|
||
| Biological data varies from experiment to experiment, however, and whether the model will be robust to these variations remains somewhat of an open question. | ||
| In the ideal case, we would like the models to be able to output a level of certainty along with its predicted segmentations or tracks. | ||
| Unfortunately, uncertainty estimation remains an unsolved problem, methods such as test-time augmentation [@(reference)] can give you an estimate if you have a good model of your data's sources of noise. |
There was a problem hiding this comment.
Flagging the [@(reference)] to be filled in.
| """ | ||
| smoothed = gaussian(image, sigma=sigma) | ||
| bw = closing(smoothed > threshold_otsu(smoothed), footprint_rectangle((3, 3))) | ||
| bw = remove_small_objects(bw, min_size=min_size) |
There was a problem hiding this comment.
We receive a warning about min_size being a deprecated parameter. Would you be okay with updating this to max_size?
ScientistRachel
left a comment
ScientistRachel
left a comment
There was a problem hiding this comment.
Hi @msschwartz21 and @adjavon, Thank you for the chapter draft! It looks really good. We left our suggested edits as GitHub comments. Please take a look and submit a revised draft. After you revise, we will do things like cross-reference the glossary, copy-edit, etc., before merging with main.
Some of your terms are already in the glossary; if you are okay with the existing definitions, nothing needs to change.
- training data
- loss function
- quality control
- semantic segmentation
- F1-score
- false positive
- false negative
- point detections (link to centroid coordinate)
- bounding box (link to bounding region)
These terms are not yet in the glossary. Please feel free to provide definitions, otherwise we will pull them from your chapter text:
- test dataset
- validation dataset
- binary cross entropy
- mean squared error
- distance transform
- confidence score
- precision
- recall
- harmonic mean
- intersection
- union
- Hausdorff distance
- Average Precision (AP)
4 participants
Diane and I are going to use this PR as a place to review the chunks we each have written and to leave comments. Please don't merge it until I confirm that it's ready.