|
StormCrawler
StormCrawler is an open-source collection of resources for building low-latency, scalable web crawlers on Apache Storm. It is provided under Apache License and is written mostly in Java (programming language). StormCrawler is modular and consists of a core module, which provides the basic building blocks of a web crawler such as fetching, parsing, URL filtering. Apart from the core components, the project also provides external resources, like for instance spout and bolts for Elasticsearch and Apache Solr or a ParserBolt which uses Apache Tika to parse various document formats. The project is used by various organisations, notably Common Crawl for generating a large and publicly available dataset of news. Linux.com published a Q&A in October 2016 with the author of StormCrawler. InfoQ ran one in December 2016. A comparative benchmark with Apache Nutch was published in January 2017 on dzone.com. Several research papers mentioned the use of StormCrawler, in particular: * Cra ... [...More Info...] [...Related Items...] OR: [Wikipedia] [Google] [Baidu] |
Web Crawler
Web crawler, sometimes called a spider or spiderbot and often shortened to crawler, is an Internet bot that systematically browses the World Wide Web and that is typically operated by search engines for the purpose of Web indexing (''web spidering''). Web search engines and some other websites use Web crawling or spidering software to update their web content or indices of other sites' web content. Web crawlers copy pages for processing by a search engine, which Index (search engine), indexes the downloaded pages so that users can search more efficiently. Crawlers consume resources on visited systems and often visit sites unprompted. Issues of schedule, load, and "politeness" come into play when large collections of pages are accessed. Mechanisms exist for public sites not wishing to be crawled to make this known to the crawling agent. For example, including a robots.txt file can request Software agent, bots to index only parts of a website, or nothing at all. The number of In ... [...More Info...] [...Related Items...] OR: [Wikipedia] [Google] [Baidu] |
Java (programming Language)
Java is a High-level programming language, high-level, General-purpose programming language, general-purpose, Memory safety, memory-safe, object-oriented programming, object-oriented programming language. It is intended to let programmers ''write once, run anywhere'' (Write once, run anywhere, WORA), meaning that compiler, compiled Java code can run on all platforms that support Java without the need to recompile. Java applications are typically compiled to Java bytecode, bytecode that can run on any Java virtual machine (JVM) regardless of the underlying computer architecture. The syntax (programming languages), syntax of Java is similar to C (programming language), C and C++, but has fewer low-level programming language, low-level facilities than either of them. The Java runtime provides dynamic capabilities (such as Reflective programming, reflection and runtime code modification) that are typically not available in traditional compiled languages. Java gained popularity sh ... [...More Info...] [...Related Items...] OR: [Wikipedia] [Google] [Baidu] |
Apache License
The Apache License is a permissive free software license written by the Apache Software Foundation (ASF). It allows users to use the software for any purpose, to distribute it, to modify it, and to distribute modified versions of the software under the terms of the license, without concern for royalties. The ASF and its projects release their software products under the Apache License. The license is also used by many non-ASF projects. History Beginning in 1995, the Apache Group (later the Apache Software Foundation) released successive versions of the Apache HTTP Server. Its initial license was essentially the same as the original 4-clause BSD license, with only the names of the organizations changed, and with an additional clause forbidding derivative works from bearing the Apache name. In July 1999, the Berkeley Software Distribution accepted the argument put to it by the Free Software Foundation and retired their ''advertising clause'' (clause 3) to form the new 3-clau ... [...More Info...] [...Related Items...] OR: [Wikipedia] [Google] [Baidu] |
Open-source Software
Open-source software (OSS) is Software, computer software that is released under a Open-source license, license in which the copyright holder grants users the rights to use, study, change, and Software distribution, distribute the software and its source code to anyone and for any purpose. Open-source software may be developed in a collaborative, public manner. Open-source software is a prominent example of open collaboration, meaning any capable user is able to online collaboration, participate online in development, making the number of possible contributors indefinite. The ability to examine the code facilitates public trust in the software. Open-source software development can bring in diverse perspectives beyond those of a single company. A 2024 estimate of the value of open-source software to firms is $8.8 trillion, as firms would need to spend 3.5 times the amount they currently do without the use of open source software. Open-source code can be used for studying and a ... [...More Info...] [...Related Items...] OR: [Wikipedia] [Google] [Baidu] |
Storm (event Processor)
Apache Storm is a distributed stream processing computation framework written predominantly in the Clojure programming language. Originally created by Nathan Marz and team at BackType, the project was open sourced after being acquired by Twitter. It uses custom created "spouts" and "bolts" to define information sources and manipulations to allow batch, distributed processing of streaming data. The initial release was on 17 September 2011. A Storm application is designed as a "topology" in the shape of a directed acyclic graph (DAG) with spouts and bolts acting as the graph vertices. Edges on the graph are named streams and direct data from one node to another. Together, the topology acts as a data transformation pipeline. At a superficial level the general topology structure is similar to a MapReduce job, with the main difference being that data is processed in real time as opposed to in individual batches. Additionally, Storm topologies run indefinitely until killed, while a ... [...More Info...] [...Related Items...] OR: [Wikipedia] [Google] [Baidu] |
Elasticsearch
Elasticsearch is a Search engine (computing), search engine based on Apache Lucene, a free and open-source search engine. It provides a distributed, Multitenancy, multitenant-capable full-text search engine with an HTTP web interface and schema-free JSON documents. Official clients are available in Java (programming language), Java, .NET Framework, .NET (C Sharp (programming language), C#), PHP (programming language), PHP, Python (programming language), Python, Ruby (programming language), Ruby and many other languages. According to the DB-Engines ranking, Elasticsearch is the most popular enterprise search engine. History Shay Banon created the precursor to Elasticsearch, called Compass, in 2004. While thinking about the third version of Compass he realized that it would be necessary to rewrite big parts of Compass to "create a scalable search solution". So he created "a solution built from the ground up to be distributed" and used a common interface, JSON over HTTP, suitable ... [...More Info...] [...Related Items...] OR: [Wikipedia] [Google] [Baidu] |
Apache Solr
Solr (pronounced "solar") is an open-source enterprise-search platform, written in Java. Its major features include full-text search, hit highlighting, faceted search, real-time indexing, dynamic clustering, database integration, NoSQL features and rich document (e.g., Word, PDF) handling. Providing distributed search and index replication, Solr is designed for scalability and fault tolerance. Solr is widely used for enterprise search and analytics use cases and has an active development community and regular releases. Solr runs as a standalone full-text search server. It uses the Lucene Java search library at its core for full-text indexing and search, and has REST-like HTTP/XML and JSON APIs that make it usable from most popular programming languages. Solr's external configuration allows it to be tailored to many types of applications without Java coding, and it has a plugin architecture to support more advanced customization. Apache Solr is developed in an open, collaborat ... [...More Info...] [...Related Items...] OR: [Wikipedia] [Google] [Baidu] |
Apache Tika
Apache Tika is a content detection and analysis framework, written in Java, stewarded at the Apache Software Foundation. It detects and extracts metadata and text from over a thousand different file types, and as well as providing a Java library, has server and command-line editions suitable for use from other programming languages. History The project originated as part of the Apache Nutch codebase, to provide content identification and extraction when crawling. In 2007, it was separated out, to make it more extensible and usable by content management systems, other Web crawlers, and information retrieval systems. The standalone Tika was founded by Jérôme Charron, Chris Mattmann and Jukka Zitting. In 2011 Chris Mattmann and Jukka Zitting released the Manning book "Tika in Action", and the project released version 1.0. Features Tika provides capabilities for identification of more than 1400 file types from the Internet Assigned Numbers Authority taxonomy of MIME types. Fo ... [...More Info...] [...Related Items...] OR: [Wikipedia] [Google] [Baidu] |
Common Crawl
Common Crawl is a nonprofit organization, nonprofit 501(c) organization#501.28c.29.283.29, 501(c)(3) organization that web crawler, crawls the web and freely provides its archives and datasets to the public. Common Crawl's Web archiving, web archive consists of petabyte, petabytes of data collected since 2008. It completes crawls approximately once a month. Common Crawl was founded by Gil Elbaz. Advisors to the non-profit include Peter Norvig and Joi Ito. The organization's crawlers respect nofollow and Robot exclusion standard, robots.txt policies. Open source code for processing Common Crawl's data set is publicly available. The Common Crawl dataset includes copyrighted work and is distributed from the US under fair use claims. Researchers in other countries have made use of techniques such as shuffling sentences or referencing the Common Crawl dataset to work around copyright law in other Jurisdiction, legal jurisdictions. Contents archived by Common Crawl are mirrored and ma ... [...More Info...] [...Related Items...] OR: [Wikipedia] [Google] [Baidu] |
Linux
Linux ( ) is a family of open source Unix-like operating systems based on the Linux kernel, an kernel (operating system), operating system kernel first released on September 17, 1991, by Linus Torvalds. Linux is typically package manager, packaged as a Linux distribution (distro), which includes the kernel and supporting system software and library (computing), libraries—most of which are provided by third parties—to create a complete operating system, designed as a clone of Unix and released under the copyleft GPL license. List of Linux distributions, Thousands of Linux distributions exist, many based directly or indirectly on other distributions; popular Linux distributions include Debian, Fedora Linux, Linux Mint, Arch Linux, and Ubuntu, while commercial distributions include Red Hat Enterprise Linux, SUSE Linux Enterprise, and ChromeOS. Linux distributions are frequently used in server platforms. Many Linux distributions use the word "Linux" in their name, but the Free ... [...More Info...] [...Related Items...] OR: [Wikipedia] [Google] [Baidu] |
