scripts.analyze_agreement module

analyze_agreement.py is a script that analyzes the agreement between two annotations of the same file. The script measures:

• Object counts: are they the same?

• Object assignment: given the least-squares mapping of objects onto each other, to what extent do they differ?

For an overview of command-line options, call:

analyze_agreement.py -h

Alignment algorithm

The script uses a greedy alignment procedure.

First, it computes for each (truth, prediction) symbol pair their recall, precision, and f-score over pixels that fall within the mask (bounding box overlap may be misleading, mainly for parallel beams).

Each predicted symbol is then aligned to the ground truth symbol with the highest f-score. If the symbol classes of a (truth, prediction) pair do not match, their score gets set to 0. (This can be turned off using the --no_strict_class_names option.)

Next, the alignment is cleaned up: if multiple predictions are aligned to a single ground truth, the one with the highest f-score is chosen and the other predicted symbols are considered unaligned.

Computing the output f-score

Finally, we sum all the f-scores of (truth, prediction) symbol pairs in the alignment.

Ground truth symbols that are not aligned to any predicted object also contribute a zero to the overall f-score.

scripts.analyze_agreement.align_nodes(truth: List[Node], prediction: List[Node], fscore=None) → List[Tuple[int, int]]

Aligns prediction Nodes to truth.

Parameters:

• truth – A list of the ground truth Nodes.

• prediction – A list of the predicted Nodes.

Returns:

A list of (t, p) pairs of Node indices into the truth and prediction lists. There will be one pair for each predicted symbol.

scripts.analyze_agreement.bounding_box_intersection(origin: Tuple[int, int, int, int], intersect: Tuple[int, int, int, int]) → Tuple[int, int, int, int] | None

Returns the coordinates of the origin bounding box that are intersected by the intersect bounding box.

>>> bounding_box = 10, 100, 30, 110
>>> other_bbox = 20, 100, 40, 105
>>> bounding_box_intersection(bounding_box, other_bbox)
(10, 0, 20, 5)
>>> bounding_box_intersection(other_bbox, bounding_box)
(0, 0, 10, 5)
>>> containing_bbox = 4, 55, 44, 115
>>> bounding_box_intersection(bounding_box, containing_bbox)
(0, 0, 20, 10)
>>> contained_bbox = 12, 102, 22, 108
>>> bounding_box_intersection(bounding_box, contained_bbox)
(2, 2, 12, 8)
>>> non_overlapping_bbox = 0, 0, 3, 3
>>> bounding_box_intersection(bounding_box, non_overlapping_bbox) is None
True

scripts.analyze_agreement.build_argument_parser()

scripts.analyze_agreement.compute_recall_precision_fscore(truth: List[Node], prediction: List[Node]) → Tuple[ndarray, ndarray, ndarray]

Computes Node pixel-level metrics.

Parameters:

• truth – A list of the ground truth Nodes.

• prediction – A list of the predicted Nodes.

Returns:

Three matrices with shape (len(truth), len(prediction): recall, precision, and f-score for each truth/prediction Node pair. Truth Nodes are rows, prediction columns.

scripts.analyze_agreement.compute_recall_precision_fscore_given_an_alignment(alignment: List[Tuple[int, int]], individual_recalls, individual_precisions, n_not_aligned: int = 0, strict_classnames: bool = True, truths: List[Node] = None, predictions: List[Node] = None) → Tuple[float, float, float]

scripts.analyze_agreement.main(args)

scripts.analyze_agreement.pixel_metrics(truth: Node, prediction: Node) → Tuple[float, float, float]

Computes the recall, precision and f-score for the prediction Node given the truth Node.