swh.lister.julia.lister module

class swh.lister.julia.lister.JuliaListerState(last_seen_commit: str | None = None)

Bases: object

Store lister state for incremental mode operations

last_seen_commit: str | None = None

Hash of the latest Git commit when lister was executed

class swh.lister.julia.lister.JuliaLister(scheduler: SchedulerInterface, url: str = 'https://github.com/JuliaRegistries/General.git', instance: str = 'julia', credentials: Dict[str, Dict[str, List[Dict[str, str]]]] | None = None, max_origins_per_page: int | None = None, max_pages: int | None = None, enable_origins: bool = True)

Bases: Lister[JuliaListerState, Dict[str, Any]]

List Julia packages origins

LISTER_NAME: str = 'julia'

VISIT_TYPE = 'git'

INSTANCE = 'julia'

REPO_URL = 'https://github.com/JuliaRegistries/General.git'

REPO_PATH = PosixPath('/tmp/tmp_33yk6_d/General')

get_registry_repository() → None

Get Julia General Registry Git repository up to date on disk

state_from_dict(d: Dict[str, Any]) → JuliaListerState

Convert the state stored in the scheduler backend (as a dict), to the concrete StateType for this lister.

state_to_dict(state: JuliaListerState) → Dict[str, Any]

Convert the StateType for this lister to its serialization as dict for storage in the scheduler.

Values must be JSON-compatible as that’s what the backend database expects.

get_origin_data(entry: WalkEntry) → Dict[str, Any]

Given an entry object parse its commit message and other attributes to detect if the commit is valid to describe a new package or a new package version.

Returns a dict with origin url as key and iso8601 commit date as value

get_pages() → Iterator[Dict[str, Any]]

Yield an iterator which returns ‘page’

To build a list of origins the Julia General registry Git repository is cloned to look at commits history to discover new package and new package versions.

Depending on last_seen_commit state it initiate a commit walker since the last time the lister has been executed.

There is only one page that list all origins urls.

get_origins_from_page(page: Dict[str, Any]) → Iterator[ListedOrigin]

Iterate on all pages and yield ListedOrigin instances

Each directory of the Git repository have a Package.toml file from where we get the Git repository url as an origin for each package.

finalize() → None

Custom hook to finalize the lister state before returning from the main loop.

This method must set updated if the lister has done some work.

If relevant, this method can use :meth`get_state_from_scheduler` to merge the current lister state with the one from the scheduler backend, reducing the risk of race conditions if we’re running concurrent listings.

This method is called in a finally block, which means it will also run when the lister fails.