Search by

spatie/robots-txt dependents (26) Order by: downloads | none Show: all | require | require-dev

  • PHP

    spatie/crawler

    Crawl all internal links found on a website

    Latest version requires spatie/robots-txt ^2.0

    20 877 636 2 842

  • PHP

    roach-php/core

    A complete web scraping toolkit for PHP

    Latest version requires spatie/robots-txt ^2.0.3

    440 618 1 455

  • PHP

    vdb/php-spider

    A configurable and extensible PHP web spider

    Latest version requires spatie/robots-txt ^2.0

    189 400 1 355

  • PHP

    piedweb/extractor

    Library to extract some content from an html Page

    Latest version requires spatie/robots-txt ^2.0

    36 576 0

  • PHP

    piedweb/url-harvester

    Harvest statistics and meta data from an URL or his source code (seo oriented).

    Latest version requires spatie/robots-txt ^1.0.10|^2

    20 889 1

  • PHP

    bringyourownideas/laravel-sitemap

    A simple website crawler & sitemap generator without a headless browser for Laravel 5.8+

    Latest version requires spatie/robots-txt ^1.0||^2.0

    7 977 9

  • acanto/crawler

    Crawl all internal links found on a website

    Latest version requires spatie/robots-txt ^1.0.9|^2.0

    2 921 0

  • PHP

    imagina/cms-platform

    The AsgardCms application.

    Latest version requires spatie/robots-txt ^2.0

    285 2

  • PHP

    wengooooo/blackspider

    A complete web scraping toolkit for PHP

    Latest version requires spatie/robots-txt ^2.0

    70 0

  • PHP

    atoolo/crawler-teaser-indexer

    Automated crawler-based generation and indexing to Solr.

    Latest version requires spatie/robots-txt ^2.5.4

    69 0

  • PHP

    bukankalengkaleng/laravel-packages

    A collection of Laravel must-have packages

    Latest version requires spatie/robots-txt dev-master

    68 2

  • PHP

    imaginacms/platform

    The AsgardCms application.

    Latest version requires spatie/robots-txt ^2.0

    23 2

  • PHP

    vhs/roach-core

    A complete web scraping toolkit for PHP

    Latest version requires spatie/robots-txt ^2.0.3

    15 0

  • PHP

    alex-kassel/roach-php-core

    Community fork of roach-php/core with on-demand lazy DOM loading and native JSON response parsing

    Latest version requires spatie/robots-txt ^2.0.3

    12 0

  • dkm/typo3-robots-guard

    Centrally manages robots.txt, the robots head meta tag and XML sitemap inclusion for any TYPO3 site (single-site or multisite), with environment-aware safety logic that guards production against accidental crawl-blocking while keeping staging/dev fully disallowed.

    Latest version requires spatie/robots-txt ^2.5

    11 0