import Frontier from 'squidwarc/lib/frontier/index.js'

public class | source

Frontier

In memory implementation of a frontier

Constructor Summary

Public Constructor
public	constructor() Create a new frontier object

Public Members
public	current: {url: string, mode: Symbol, cdepth: number, tracker: string} Information pertaining to the current URL being crawled
public	queue: {url: string, mode: Symbol, cdepth: number, tracker: string}}[] URLs to be crawled
public	trackers: Map<string, SeedTracker> Tracks the depth and crawl config per starting seed

Public Methods
public	exhausted(): boolean Is the frontier exhausted
public	init(starting: Seed[] \| Seed) Initialize the initial frontier
public	next(): string Get the next URL to crawl from the frontier, queue length - 1
public	process(links: Array<{href: string, pathname: string, host: string}>) Process discovered outlinks of a page based on the originating seeds configuration
public	size(): number Returns the number of URLs left in the queue

Create a new frontier object

Information pertaining to the current URL being crawled

URLs to be crawled

Tracks the depth and crawl config per starting seed

Is the frontier exhausted

Initialize the initial frontier

Name	Type	Attribute	Description
starting	Seed[] \| Seed

Get the next URL to crawl from the frontier, queue length - 1

string (nullable: true)

Process discovered outlinks of a page based on the originating seeds configuration

Name	Type	Attribute	Description
links	Array<{href: string, pathname: string, host: string}>		list of seeds to consider

Returns the number of URLs left in the queue