swh.storage.proxies.counter module#

class swh.storage.proxies.counter.CountingProxyStorage(counters, storage: Dict[str, Any])[source]#

Bases: object

Counting Storage Proxy.

This is in charge of adding objects directly to swh-counters, without going through Kafka/swh-journal. This is meant as a simple way to setup counters for experiments; production should use swh-journal to reduce load/latency of the storage server.

Additionally, unlike the journal-based counting, it does not count persons or the number of origins per netloc.

Sample configuration use case for filtering storage:

swh.storage.proxies.counter module#

This Page