Metrics API

PolyGraph implements metrics that provide either a single estimate or an interval to quantify uncertainty. We provide a minimal common interface for metrics as the protocol GenerationMetric. The only requirement to satisfy this interface is to implement a compute method that accepts a collection of graphs. In practice, these graphs may either be nx.Graph or rdkit.Chem.Mol objects, as determined by the GraphType generic parameter.

Metrics that implement this interface may be evaluated jointly using the MetricCollection class.

from polygraph.metrics import MetricCollection, StandardPGD
from polygraph.metrics.rbf_mmd import RBFOrbitMMD2
from polygraph.datasets import PlanarGraphDataset, SBMGraphDataset

reference_graphs = PlanarGraphDataset("val").to_nx()
generated_graphs = SBMGraphDataset("val").to_nx()

metrics = MetricCollection(
    metrics={
        "rbf_orbit": RBFOrbitMMD2(reference_graphs=reference_graphs),
        "pgd": StandardPGD(reference_graphs=reference_graphs),
    }
)
print(metrics.compute(generated_graphs))

We implement the following metrics:

  • MMD - Classical Maximum Mean Discrepancy
  • PolyGraphDiscrepancy - Lower bounds on probability metrics via classification
  • VUN - Validity, Uniqueness, Novelty
  • Fréchet Distance - Optimal transport distance between fitted Gaussians

Metrics that perform uncertainty quantification may return a MetricInterval object.

Metric Interface

polygraph.metrics.base.GenerationMetric

Bases: Protocol, Generic[GraphType]

Interface for all graph generation metrics.

compute(generated_graphs)

Compute the metric on the generated graphs.

Parameters:
  • generated_graphs (Collection[GraphType]) –

    Collection of generated graphs to evaluate.

Metric Collections

polygraph.metrics.base.MetricCollection

Bases: GenerationMetric[GraphType], Generic[GraphType]

Collection of metrics that are evaluated jointly.

compute(generated_graphs)

Compute the metrics on the generated graphs.

Parameters:
  • generated_graphs (Collection[GraphType]) –

    Collection of generated graphs.

Uncertainty Quantification

polygraph.metrics.base.MetricInterval

Class for representing uncertainty quantifications of a metric.

Attributes:
  • mean –

    Mean of the metric samples.

  • std –

    Standard deviation of the metric samples.

  • low –

    Lower bound of the metric interval, if coverage is provided.

  • high –

    Upper bound of the metric interval, if coverage is provided.

  • coverage –

    Coverage of the metric interval. If None, no bounds are computed, only the mean and standard deviation are set.

from_samples(samples, coverage=None) classmethod

Create a MetricInterval from a collection of samples.

Parameters:
  • samples (ndarray) –

    1D array of samples.

  • coverage (Optional[float], default: None ) –

    Coverage of the interval. If None, no bounds are computed, only the mean and standard deviation are set.