Anthelion is an open-source Apache Nutch plugin that uses online learning to focus crawling on web pages containing semantic annotations, extracting structured data from HTML as it crawls.
What is Anthelion?
Anthelion is a Nutch plugin that extends the Apache Nutch web crawler to prioritize pages rich in semantic data. It takes as input the HTML pages fetched by Nutch, classifies newly discovered outlinks with an online scoring filter, and outputs extracted RDF triples (stored in the Content field) plus additional index fields such as containsSem. The project is maintained under the Yahoo GitHub organization, is released under the Apache License 2.0, and ships with the complete Nutch 1.6 distribution including the plugin source in /src/plugin/parse-anth.
Key Features
- Online learning classifier — Uses page context and feedback from previously parsed pages to score outlinks; the scores determine which URLs are selected in the Generate stage of the crawl cycle.
- Semantic data extraction — The WdcParser parses HTML and extracts Microdata, Microformats, and RDFa annotations using the Apache Any23 1.2 library.
- Scoring integration — Implements Nutch's ScoringFilter interface via AnthelionScoringFilter, so relevant outlinks receive higher scores and are fetched first.
- Index enrichment — TripleExtractor implements Nutch's IndexingFilter interface to store new fields (e.g., containsSem) in the Solr schema, enabling queries for semantic content.
- Configurable classifier — The online classifier is tuned through properties such as anth.scoring.classifier.PropsFilePath, which points to a baseline.properties file containing all classifier settings.
- Plugin deployment flexibility — Can be added to any Nutch distribution by copying the parse-anth folder, enabling plugin.includes, and adding the alias and mimeType entries in parse-plugins.xml.
Who is it for?
- Semantic web researchers — Build focused crawls that increase the fraction of crawled pages containing structured data, as demonstrated by the precision tables in the README (e.g., 0.78 precision with 1000 seeds and depth 5, versus 0.06 for standard scoring).
- Knowledge graph engineers — Use the extracted triples from Microdata, Microformats, and RDFa to populate triple stores or enrich internal knowledge bases.
- Search and crawling engineers — Integrate Anthelion into an existing Nutch pipeline to filter index documents based on semantic markup, using the added containsSem field for querying.
Use cases
- E-commerce data collection: Crawl product pages and extract product price, availability, and review annotations from HTML microdata to build a product feed.
- Linked Data dataset creation: Collect RDFa statements from a targeted set of domains and export them as triples for use in a semantic web dataset.
- Search index enhancement: Add the containsSem field to Solr and boost or filter search results that contain semantic annotations.
How does Anthelion work?
The crawl cycle starts with Nutch seeding URLs; the AnthelionScoringFilter classifies each outlink as relevant or not and assigns a score. The WdcParser then extracts triples from fetched pages using Any23, and the TripleExtractor writes index fields. The Generate stage picks the next batch of URLs based on the assigned scores, and the classifier receives feedback from the parsed pages to improve future predictions.
Pricing
Anthelion is free and open-source software released under the Apache License 2.0. No paid tiers are offered.
FAQ
Is Anthelion free?
Yes, Anthelion is completely free and open-source. It is released under the Apache License 2.0, and the source code is publicly available on GitHub under the yahoo/anthelion repository.
What semantic formats does Anthelion extract?
The WdcParser extracts three types of semantic annotation from HTML: Microdata, Microformats, and RDFa. Extraction is done through the Apache Any23 library version 1.2.
Which Nutch version does Anthelion use?
The repository contains the full Apache Nutch 1.6 distribution, including the plugin located at /src/plugin/parse-anth. The plugin can also be added to other Nutch versions by following the configuration steps in the README.
Does Anthelion require Apache Solr?
No, Solr is optional. The TripleExtractor adds index fields for querying, and the README shows how to add the containsSem field to schema.xml when using Solr. If you only need focused crawling without indexing, you can use the scoring and parsing extensions on their own.
How does the scoring filter make decisions?
The AnthelionScoringFilter wraps an online classifier that predicts whether a newly discovered page is data-rich. It uses the context of the page and feedback from the extraction of metadata from previously seen pages, which the README cites from the research paper "Focused Crawling for Structured Data."
