Common Crawl


Common Crawl is a nonprofit 501 organization that crawls the web and freely provides its archives and datasets to the public. Common Crawl's web archive consists of petabytes of data collected since 2011. It completes crawls generally every month.
Common Crawl was founded by Gil Elbaz. Advisors to the non-profit include Peter Norvig and Joi Ito. The organization's crawlers respect nofollow and robots.txt policies. Open source code for processing Common Crawl's data set is publicly available.

History

began hosting Common Crawl's archive through its Public Data Sets program in 2012.
The organization began releasing metadata files and the text output of the crawlers alongside.arc files in July of that year. Common Crawl's archives had only included.arc files previously.
In December 2012, blekko donated to Common Crawl search engine metadata blekko gathered from crawls it conducted from February to October 2012. The donated data helped Common Crawl "improve its crawl while avoiding spam, porn and the influence of excessive SEO."
In 2013, Common Crawl began using Apache Software Foundation's Nutch webcrawler instead of a custom crawler. Common Crawl switched from using.arc files to.warc files with its November 2013 crawl.
Common Crawl was used to train OpenAI's GPT-3 language model, announced in 2020.

History of Common Crawl data

The following data have been collected from the official
Crawl DateSize in TiBBillions of pagesComments
November 20182202.6
October 20182403.0
September 20182202.8
August 2018
July 20182553.25
June 20182353.05
May 20182152.75
April 20182303.1
March 20182503.2
February 20182703.4
January 20182703.4
December 20172402.9
November 20172603.2
October 20173003.65
September 20172503.01
August 20172803.28
July 20172402.89
June 20172603.16
May 20172502.96
April 20172502.94
March 20172503.07
February 20172503.08
January 20172503.14
December 2016-2.85
October 2016-3.25
September 2016-1.72
August 2016-1.61
July 2016-1.73
June 2016-1.23
May 2016-1.46
April 2016-1.33
February 2016-1.73
November 20151511.82
September 20151061.32
August 20151491.84
July 20151451.81
June 20151311.67
May 20151592.05
April 20151682.11
March 20151241.64
February 20151451.9
January 20151391.82
December 20141602.08
November 20141351.95
October 20142543.7
September 20142202.8
August 20142002.8
July 20142663.6
April 20141832.6
March 20142232.8First Nutch crawl
January 20141482.3Crawls performed monthly
November 20131022Data in Warc file format
July 2012--Data in Arc file format
January 2012--Public Data Set of Amazon Web Services
November 2011405First availability on Amazon

Norvig Web Data Science Award

In corroboration with SURFsara, Common Crawl sponsors the Norvig Web Data Science Award, a competition open to students and researchers in Benelux. The award is named for Peter Norvig who also chairs the judging committee for the award.