# Bot-Sperrlisten-Empfehlung - https://www.scraptrap.de - Stand: 15.10.2026 - 03:23:08
User-agent: 2ip bot/1.1 (+https://2ip.io/bot/)
Disallow: /
User-agent: AboutUsTrustSignals/1.0 (+https://aboutus.com)
Disallow: /
User-agent: Agent incident monitor/0.1
Disallow: /
User-agent: ai-freebie-radar/0.1 (+https://github.com/)
Disallow: /
User-agent: AI-NewsBot/3.0 article-extractor
Disallow: /
User-agent: AICUIScout/0.1 (+https://example.local; product research)
Disallow: /
User-agent: AIToolCompareBot/1.0 (+https://sortaitech.com/bot)
Disallow: /
User-agent: AkoniLeadHuntBot/0.1 (+https://www.recruiting-systems.com)
Disallow: /
User-agent: Apache-HttpClient/5.3.1 (Java/25.0.4)
Disallow: /
User-agent: AppWatchBot/0.1 (+https://appwatch.ai/bot)
Disallow: /
User-agent: AppWatchBot/0.1 (+https://appwatch.ai/bot; freshness check only)
Disallow: /
User-agent: AppWatchBot/0.1 (+https://appwatch.ai/bot; liveness check only, no content stored)
Disallow: /
User-agent: arca-allowlist-screen/0.1 (kashyap.puranik@databricks.com)
Disallow: /
User-agent: arschiv-ngx/1.0
Disallow: /
User-agent: AstraciaBot/1.0 (+https://astracia.com; affiliate program check)
Disallow: /
User-agent: AteveSearchSourceUrlDiscovery/0.1 (+mailto:crawler@example.com)
Disallow: /
User-agent: axios/1.13.5
Disallow: /
User-agent: bearblog-outreach/1.0 (personal writer-outreach research; contact via blog)
Disallow: /
User-agent: BergamotToolRadarBot/0.1 (+https://bergamotadvisors.com)
Disallow: /
User-agent: BergaTools-ForumAggregator/1.0 (+https://github.com/BergaDev/PersonalTools)
Disallow: /
User-agent: Bryf/1.0 (Article Processor)
Disallow: /
User-agent: BubblemediaCrawler/1.0 (+https://bubblemedia.app/crawler)
Disallow: /
User-agent: CalatheaNews/2.0 (+https://github.com/nbitslabs/calathea-content-curator)
Disallow: /
User-agent: ClarFlow/2.0 (+https://clarflow.cn)
Disallow: /
User-agent: CheckMarkNetwork/1.0 (+http://www.checkmarknetwork.com/spider.html)
Disallow: /
User-agent: CheKakStartupBot/1.0 (+https://startup.chekak.store/about)
Disallow: /
User-agent: CMS-Security-Auditor/1.0 (+authorized self-check; contact: local-admin)
Disallow: /
User-agent: CodaBot/1.0
Disallow: /
User-agent: COIBotParser/2.0
Disallow: /
User-agent: CollectorBridge-SaaSProducer/1
Disallow: /
User-agent: crawlberg/1.3.0
Disallow: /
User-agent: CS-Hijack/0.1.2 (passive homepage compromise detection; no vulnerability testing; +https://www.X-XXXXXX-XXXXXXXXXX.XXX/firmengruendung-in-bulgarien/contact-4eu/)
Disallow: /
User-agent: csint.dev/1.0 (public research; contact via site)
Disallow: /
User-agent: ct-sender-poc/1.0 (stage 3 fetch)
Disallow: /
User-agent: curl/7.29.0
Disallow: /
User-agent: curl/7.4.0
Disallow: /
User-agent: curl/8.3.0
Disallow: /
User-agent: curl/8.5.0
Disallow: /
User-agent: curl/8.7.1
Disallow: /
User-agent: curl-proxy/1.0
Disallow: /
User-agent: Dalvik/2.1.0 (Linux; U; Android 9.0; ZTE BA520 Build/MRA58K)
Disallow: /
User-agent: das-deutsche.net crawler/1.0 (+https://das-deutsche.net/impressum)
Disallow: /
User-agent: de-scanner/1.0 (+link discovery bot; kontakt: markusbra@gmail.com)
Disallow: /
User-agent: DeCrawler/1.0 (DACH Domain Crawler)
Disallow: /
User-agent: DENIC-Crawler/42 (https://www.denic.de/crawler)
Disallow: /
User-agent: Deno/2.1.4 (variant; SupabaseEdgeRuntime/1.74.3)
Disallow: /
User-agent: DidItSurviveBot/1.0 (+https://diditsurvive.com/about)
Disallow: /
User-agent: directree-ingest/1.0 (+https://www.directree.io)
Disallow: /
User-agent: directree-outreach/1.0 (+https://www.directree.io)
Disallow: /
User-agent: DisbanResearchBot/0.1 (+https://disban.io; info@disban.io)
Disallow: /
User-agent: dnsq-domainintel-web-probe/1.0
Disallow: /
User-agent: DomainStatsBot/1.0 (https://domainstats.com/pages/our-bot)
Disallow: /
User-agent: DomainTargetDetector/1.0
Disallow: /
User-agent: dp-discover/1.0 (+https://godigitalpigeon.com/bot)
Disallow: /
User-agent: Drakma/2.0.10 (SBCL 2.6.1; FreeBSD; 15.0-RELEASE-p10; http://weitz.de/drakma/)
Disallow: /
User-agent: Earthshine/1.0 (earthshine.fibonacci.ai; contact: hello@fibonacci.ai)
Disallow: /
User-agent: EMAIL-EU-Academic-Crawler/4.1
Disallow: /
User-agent: Embarcadero URI Client/1.0
Disallow: /
User-agent: emerging-apps-monitor/2.0 (research; +https://hn.algolia.com/)
Disallow: /
User-agent: EnrichHatBot/0.1 (+https://hatos.internal/bot; first-party enrichment)
Disallow: /
User-agent: EPST-maker-source-verifier/1.0
Disallow: /
User-agent: EuroWebAtlasBot/1.0
Disallow: /
User-agent: everyfeed-spider/2.0 (http://www.everyfeed.com)
Disallow: /
User-agent: FinData News Capture/1.0 (private research)
Disallow: /
User-agent: FixBeforeLaunchBot/0.1 (+https://fixbeforelaunch.com/bot; readiness scanner)
Disallow: /
User-agent: FlamingoBot (+https://hackernews.pink)
Disallow: /
User-agent: FoundryScope/1.0 (startup analysis bot; contact: amine.1.mestour@gmail.com)
Disallow: /
User-agent: FreshRSS/1.29.1 (Linux; https://freshrss.org)
Disallow: /
User-agent: GeneralCrawlBot/0.1
Disallow: /
User-agent: get_titles/1.0
Disallow: /
User-agent: github-wikidata-bot/0.4.0 (https://github.com/konstin/github-wikidata-bot)
Disallow: /
User-agent: GitHubCopilotRuntime-WebFetch
Disallow: /
User-agent: Go-http-client/1.1
Disallow: /
User-agent: go-resty/2.16.5 (https://github.com/go-resty/resty)
Disallow: /
User-agent: GoldiBot/0.1 (zlati@meatborsa.com)
Disallow: /
User-agent: Grammarly/1.0 (http://www.grammarly.com)
Disallow: /
User-agent: greedyhand/0.1
Disallow: /
User-agent: GreenWebChecker/1.0 (https://www.thegreenwebfoundation.org/green-web-checker-faq/)
Disallow: /
User-agent: GuzzleHttp/7
Disallow: /
User-agent: HackerNews/1569 CFNetwork/3860.700.1 Darwin/25.6.0
Disallow: /
User-agent: HackerSearchBot/0.1 (article-content enrichment)
Disallow: /
User-agent: HakuLabCollector/0.1 (+https://n8n.hakulab.cc)
Disallow: /
User-agent: Harmonic-HN-Android/3.1.2/fossRelease
Disallow: /
User-agent: Harmonic-HN-Android/3.1.2/playRelease
Disallow: /
User-agent: Hello from Palo Alto Networks, find out more about our scans in https://docs-cortex.paloaltonetworks.com/r/1/Cortex-Xpanse/Scanning-activity
Disallow: /
User-agent: hermes-newsroom/0.1 (research project; +https://github.com/hermes-newsroom)
Disallow: /
User-agent: Hermes-SallyPageFetch/1.0
Disallow: /
User-agent: hn Reader (https://e8f3ede5-ab43-42ec-9ebb-007f0f6e7523.netlify.app)
Disallow: /
User-agent: hn-youtube-fetcher/0.2 (personal script)
Disallow: /
User-agent: http.rb/5.3.1
Disallow: /
User-agent: httpx - Open-source project (github.com/projectdiscovery/httpx)
Disallow: /
User-agent: humanet-page-scraper/1.0
Disallow: /
User-agent: icon.horse/2.0
Disallow: /
User-agent: Iframely/1.3.1 (+https://iframely.com/docs/about)
Disallow: /
User-agent: iHerd-surfer/0.1 (+https://i-herd.com; content discovery agent)
Disallow: /
User-agent: Indexaro/1.0 (+https://indexaro.com)
Disallow: /
User-agent: IndieRisersBot/0.2 (+https://indie-risers.wangtiechuichui.chatgpt.site/bot)
Disallow: /
User-agent: InfomateBot/1.0 (+https://infomate.club)
Disallow: /
User-agent: InsuranceLeadMachine/1.0 (+lead discovery; contact: ops@insuranceleadmachine.com)
Disallow: /
User-agent: InterestingFindsBot/0.1 (+https://nikhilkulkarni1755.com/bot)
Disallow: /
User-agent: IonCrawl
Disallow: /
User-agent: IronFountain-Leads/1.0 (+https://www.ironfountain.com)
Disallow: /
User-agent: IronReader/0.1 (+https://example.com; RSS reader)
Disallow: /
User-agent: Java/21.0.11
Disallow: /
User-agent: JBBCareerDiscovery/4.0 (career-site discovery; contact: local-operator)
Disallow: /
User-agent: JGK-Firmendb/1.0 (+https://sevuma.de)
Disallow: /
User-agent: KelviqLeadResearch/1.0 (+https://kelviq.com)
Disallow: /
User-agent: ken/0.1 (+https://github.com/zcag/ken)
Disallow: /
User-agent: KnisNewsCollector/1.0 (+local)
Disallow: /
User-agent: Ktor http-Client
Disallow: /
User-agent: LanternBot/1.0 (+https://selfhost-search.rork.app/about; independent search index)
Disallow: /
User-agent: LaravelReviewScanner/1.0 (+https://webauditscanner.de)
Disallow: /
User-agent: LaunchToWinBot/1.0 (+https://launchtowin.com)
Disallow: /
User-agent: LaunchToWinBot/1.0 (+https://www.theill.com)
Disallow: /
User-agent: LindyReportBot/1.0 (+https://lindyreport.com/methodology)
Disallow: /
User-agent: LinkAce/2 (https://github.com/Kovah/LinkAce)
Disallow: /
User-agent: LinkAnalyser/2.0
Disallow: /
User-agent: LivelapBot/0.2 (http://site.livelap.com/crawler)
Disallow: /
User-agent: LohiSoftBot/1.0 (+message@lohisoft.com)
Disallow: /
User-agent: Lumina-News/0.1 (+local triage)
Disallow: /
User-agent: machines-outreach/1.0 (lead research; contact caledaptproductdelivery@zohomail.eu)
Disallow: /
User-agent: MakerJury-Discovery/1.0 (+https://makerjury.com/contact)
Disallow: /
User-agent: MalveonLeadBot/0.2 (+https://malveon.com; contact ladsonselvam1998@gmail.com)
Disallow: /
User-agent: MapTheNetBot/1.0 (+https://www.mapthenet.org/bot)
Disallow: /
User-agent: MarkdownWorker/1.0
Disallow: /
User-agent: markethunt2/0.1 (+research)
Disallow: /
User-agent: Mattermost-Bot/1.1
Disallow: /
User-agent: MBC-Signal-Crawler/1.0 (+https://mybusinessconsult.de)
Disallow: /
User-agent: metabase-cve-2026-72898-detect/1.0 (benign detection probes only)
Disallow: /
User-agent: MetaInspector/5.16.0 (+https://github.com/jaimeiniesta/metainspector)
Disallow: /
User-agent: MetaRabbitBot/2.0 (+https://www.metarabbit.dev/terms#automated-fetching)
Disallow: /
User-agent: MetaScopeBot/0.1 (+https://metascope.dev) fulltext-fetcher
Disallow: /
User-agent: Mozilla/5.0 (compatible; MJ12bot/v1.4.8; http://mj12bot.com/)
Disallow: /
User-agent: MockposeLeadBot/1.0 (+https://mockpose.com)
Disallow: /
User-agent: ModelContextProtocol/1.0 (Autonomous; +https://github.com/modelcontextprotocol/servers)
Disallow: /
User-agent: Mozilla/5.0
Disallow: /
User-agent: Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; KeenableBot/1.0; +https://keenable.ai/)
Disallow: /
User-agent: Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; Keenable-User/1.0; +https://keenable.ai/)
Disallow: /
User-agent: Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; LighthouseBot/1.2; +https://lighthouseapp.io/bot-info) Chrome/136.0.0.0 Safari/537.36
Disallow: /
User-agent: Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; SleepBot/1.0; +http://sleepbot.com/) Chrome/131.0.0.0 Safari/537.36
Disallow: /
User-agent: Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; YouBot/1.0; +https://docs.you.com/youbot; env:prod) Chrome/142.0.0.0 Safari/537.36
Disallow: /
User-agent: Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko) Chrome/123.0.6312.86 Safari/537.36 BitSightBot/1.0
Disallow: /
User-agent: Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; SEOJuice-SearchBot/1.0; +https://seojuice.io/bot
Disallow: /
User-agent: Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; ShapBot/0.1.0
Disallow: /
User-agent: Mozilla/5.0 compatible logDRILLER crawler
Disallow: /
User-agent: Mozilla/5.0 EmergingAppsMonitor/2.0
Disallow: /
User-agent: Mozilla/5.0 Firefox/33.0
Disallow: /
User-agent: Mozilla/5.0 SirenHunt/1.0
Disallow: /
User-agent: Mozilla/5.0 (compatible; A11yScanBot/1.0; +http://46.62.237.229/bot)
Disallow: /
User-agent: Mozilla/5.0 (compatible; AdalineStartupResearch/1.0; public-web-research)
Disallow: /
User-agent: Mozilla/5.0 (compatible; adscan/1.0)
Disallow: /
User-agent: Mozilla/5.0 (kompatibel; AffScan/1.0)
Disallow: /
User-agent: Mozilla/5.0 (compatible; AgentInfoVault/1.0)
Disallow: /
User-agent: Mozilla/5.0 (compatible; AhrefsBot/7.0; +http://ahrefs.com/robot/)
Disallow: /
User-agent: Mozilla/5.0 (compatible; AI-News-Aggregator/1.0)
Disallow: /
User-agent: Mozilla/5.0 (compatible; aiii.news/1.0)
Disallow: /
User-agent: Mozilla/5.0 (compatible; AkoniLeadHuntBot/1.0; +https://www.recruiting-systems.com) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/126.0 Safari/537.36
Disallow: /
User-agent: Mozilla/5.0 (compatible; AppAdvisorBot/1.0; +https://appadvisor.in/about)
Disallow: /
User-agent: Mozilla/5.0 (compatible; archive.org_bot +http://archive.org/details/archive.org_bot) Zeno/d3712ae warc/v0.8.99
Disallow: /
User-agent: Mozilla/5.0 (compatible; AwarioBot/1.0; +https://awario.com/bots.html)
Disallow: /
User-agent: Mozilla/5.0 (compatible; BacklinksExtendedBot)
Disallow: /
User-agent: Mozilla/5.0 (compatible; Barkrowler/0.9; +https://babbar.tech/crawler)
Disallow: /
User-agent: Mozilla/5.0 (compatible; BLEXBot/1.0; +http://webmeup-crawler.com/)
Disallow: /
User-agent: Mozilla/5.0 (compatible; bot/1.0)
Disallow: /
User-agent: Mozilla/5.0 (compatible; catchup-agent/1.0)
Disallow: /
User-agent: Mozilla/5.0 (compatible; CensysInspect/1.1; +https://about.censys.io/)
Disallow: /
User-agent: Mozilla/5.0 (compatible; CertSignalBot/1.0)
Disallow: /
User-agent: Mozilla/5.0 (compatible; ch-domain-research/1.0)
Disallow: /
User-agent: Mozilla/5.0 (compatible; CMS-Checker/1.0; +https://example.com)
Disallow: /
User-agent: Mozilla/5.0 (compatible; CodeSecBot/1.0)
Disallow: /
User-agent: Mozilla/5.0 (compatible; ContentStudio/1.0; +https://contentstudio.io)
Disallow: /
User-agent: Mozilla/5.0 (compatible; CopyvioDetector/0.4.2.dev0; +wikipedia.earwig[at]gmail.com)
Disallow: /
User-agent: Mozilla/5.0 (compatible; DataAIHub/1.0; +https://www.dataaihub.co)
Disallow: /
User-agent: Mozilla/5.0 (compatible; DataForSeoBot/1.0; +https://dataforseo.com/dataforseo-bot)
Disallow: /
User-agent: Mozilla/5.0 (compatible; Dataprovider.com)
Disallow: /
User-agent: Mozilla/5.0 (compatible; DefaultsExposed/1.0; +https://defaults.exposed/headers)
Disallow: /
User-agent: Mozilla/5.0 (compatible; Defuddle/1.0; +https://defuddle.md)
Disallow: /
User-agent: Mozilla/5.0 (compatible; directree-facts/1.0; +https://www.directree.io)
Disallow: /
User-agent: Mozilla/5.0 (compatible; directree-techdetect/1.0; +https://www.directree.io/tools/tech-stack-detector)
Disallow: /
User-agent: Mozilla/5.0 (compatible; DiscoveryBot/1.0)
Disallow: /
User-agent: Mozilla/5.0 (compatible; DomainAnalyzer/1.0)
Disallow: /
User-agent: Mozilla/5.0 (compatible; DomainScores/2.1; +https://www.domainscores.net/headers)
Disallow: /
User-agent: Mozilla/5.0 (compatible; DotBot/1.2; +https://opensiteexplorer.org/dotbot; help@moz.com)
Disallow: /
User-agent: Mozilla/5.0 (compatible; DyonysosRadar/2.0; +https://dyonysos.fr)
Disallow: /
User-agent: Mozilla/5.0 (compatible; EarlyTermsDiscovery/1.0)
Disallow: /
User-agent: Mozilla/5.0 (compatible; Embedly/0.2; +http://support.embed.ly/
Disallow: /
User-agent: Mozilla/5.0 (compatible; EnvScanner/2.0)
Disallow: /
User-agent: Mozilla/5.0 (compatible; EpistemicFilter/0.1; personal aggregator)
Disallow: /
User-agent: Host: ip-188-194-086-015.vkd90.pools.vodafone-ip.de
Disallow: /
User-agent: Mozilla/5.0 (compatible; eu-domains-enrich/0.1; +https://gitlab.com/parkmo)
Disallow: /
User-agent: Mozilla/5.0 (compatible; ev-crawler/1.0; +https://headline.com/legal/crawler)
Disallow: /
User-agent: Mozilla/5.0 (compatible; Example3/1.0; +https://www.example3.com/domain/[URL])
Disallow: /
User-agent: Mozilla/5.0 (compatible; ExaSearchBot/1.0; +https://crawler.exa.ai/)
Disallow: /
User-agent: Mozilla/5.0 (compatible; fastfollow-radar/0.1; +https://fastfollow.lol/how-it-works)
Disallow: /
User-agent: Mozilla/5.0 (compatible; Feedsearch-Crawler; +https://feedsearch.dev)
Disallow: /
User-agent: Mozilla/5.0 (compatible; Feedspot/1.0 (+https://www.feedspot.com/fs/fetcher; like FeedFetcher-Google)
Disallow: /
User-agent: Mozilla/5.0 (compatible; ForestEngine/1.0; +https://forestengine.net/)
Disallow: /
User-agent: Mozilla/5.0 (compatible; FounderDeckBot/1.0; +https://founderdeck.app)
Disallow: /
User-agent: Mozilla/5.0 (compatible; GCombinator/1.0; +https://news.gcombinator.com)
Disallow: /
User-agent: Mozilla/5.0 (compatible; GeniusDataBot/1.0; +https://geniusdata.com)
Disallow: /
User-agent: Mozilla/5.0 (compatible; getdomaindata/1.0; +https://getdomaindata.com/bot)
Disallow: /
User-agent: Mozilla/5.0 (compatible; global-new-sites-live-check/1.0; +https://example.invalid)
Disallow: /
User-agent: Mozilla/5.0 (compatible; Google-Apps-Script; beanserver; +https://script.google.com; id: UAEmdDd_b9Ocb4YPBbVaLDwvUS_5DqGlFqiw)
Disallow: /
User-agent: Mozilla/5.0 (compatible; GrowstaSales/1.0)
Disallow: /
User-agent: Mozilla/5.0 (compatible; HaloBot/1.0)
Disallow: /
User-agent: Mozilla/5.0 (compatible; HermesPublicResearch/1.0)
Disallow: /
User-agent: Mozilla/5.0 (compatible; HN-archiver/1.0)
Disallow: /
User-agent: Mozilla/5.0 (compatible; INETDEX-BOT/2.1; +https://inetdex.com/bot.php)
Disallow: /
User-agent: Mozilla/5.0 (compatible; intel-collector/0.1.0; +https://github.com/enterprise-playbook)
Disallow: /
User-agent: Mozilla/5.0 (compatible; intelx.io_bot +https://intelx.io)
Disallow: /
User-agent: Mozilla/5.0 (compatible; InternetMeasurement/1.0; +https://internet-measurement.com/)
Disallow: /
User-agent: Mozilla/5.0 (compatible; ListSignalBot/1.0; +https://listsignal.com/bot)
Disallow: /
User-agent: Mozilla/5.0 (compatible; Miniflux/2.3.1; +https://miniflux.app)
Disallow: /
User-agent: Mozilla/5.0 (compatible; MSIE 8.0; Windows NT 5.1) KStandBot/1.0
Disallow: /
User-agent: Mozilla/5.0 (compatible; MyltiNewsBot/1.0; +https://mylti.app) (variant; SupabaseEdgeRuntime/1.77.0; ref=hxbvilhzblvtbrmokmvp)
Disallow: /
User-agent: Mozilla/5.0 (compatible; NetcraftSurveyAgent/1.0; +info@netcraft.com)
Disallow: /
User-agent: Mozilla/5.0 (compatible; newsomo/1.0; +https://newsomo.com)
Disallow: /
User-agent: Mozilla/5.0 (compatible; NewsParser/0.1)
Disallow: /
User-agent: Mozilla/5.0 (compatible; NextPixelBot/1.0; +https://nextpixel.dev) Impressum-Check
Disallow: /
User-agent: Mozilla/5.0 (compatible; NoctraRecon/1.0)
Disallow: /
User-agent: Mozilla/5.0 (compatible; outreach-enrich/1.0; +local)
Disallow: /
User-agent: Mozilla/5.0 (compatible; PBN.BOT Monitoring/1.0)
Disallow: /
User-agent: Mozilla/5.0 (compatible; PitchframeBot/1.0; +https://pitchfra.me)
Disallow: /
User-agent: Mozilla/5.0 (compatible; PlagAwareBot/3.2; +https://www.plagaware.com/bot)
Disallow: /
User-agent: Mozilla/5.0 (compatible; PlytikBot/1.0; +https://plytik.com/bot)
Disallow: /
User-agent: Mozilla/5.0 (compatible; PostedIn/1.0)
Disallow: /
User-agent: Mozilla/5.0 (compatible; PostureBot/1.0; +passive-security-check)
Disallow: /
User-agent: Mozilla/5.0 (compatible; PublicWWWBot/1.0; +https://publicwww.com/bot.html)
Disallow: /
User-agent: Mozilla/5.0 (compatible; QlyzeBot/1.0; +https://app.qlyze.io/bot)
Disallow: /
User-agent: Mozilla/5.0 (compatible; recraft-image-dataset/0.1; +https://recraft.ai)
Disallow: /
User-agent: Mozilla/5.0 (compatible; Reflectionbot/1.0; +https://reflection.ai/bot)
Disallow: /
User-agent: Mozilla/5.0 (compatible; research-probe; joeljeon7@gmail.com)
Disallow: /
User-agent: Mozilla/5.0 (compatible; ResearchBot/1.0)
Disallow: /
User-agent: Mozilla/5.0 (compatible; rust_sniffer/0.1; +https://github.com/)
Disallow: /
User-agent: Mozilla/5.0 (compatible; scrape_central/1.0)
Disallow: /
User-agent: Mozilla/5.0 (compatible; SectorBot/0.1; +https://example.org/bot) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/124 Safari/537.36
Disallow: /
User-agent: Mozilla/5.0 (compatible; SecurityScanner/1.0)
Disallow: /
User-agent: Mozilla/5.0 (compatible; SecurityScanner/1.0; +https://tokenfaucet.fun)
Disallow: /
User-agent: Mozilla/5.0 (compatible; SeekportBot; +https://bot.seekport.com)
Disallow: /
User-agent: Mozilla/5.0 (compatible; SemrushBot/7~bl; +http://www.semrush.com/bot.html)
Disallow: /
User-agent: Mozilla/5.0 (compatible; seo-center/1.0; hn show hn enrichment)
Disallow: /
User-agent: Mozilla/5.0 (compatible; SeznamBot/4.0; +https://o-seznam.cz/napoveda/vyhledavani/en/seznambot-crawler/)
Disallow: /
User-agent: Mozilla/5.0 (compatible; SirenHunt/1.0; +https://mysiren.ai)
Disallow: /
User-agent: Mozilla/5.0 (compatible; SiteAuditBot/0.97; +http://www.semrush.com/bot.html)
Disallow: /
User-agent: Mozilla/5.0 (compatible; SquareTownBot/1.0; +https://square.pov.town)
Disallow: /
User-agent: Mozilla/5.0 (compatible; StackScopeBot/1.0; +https://stackscope.dev/bot)
Disallow: /
User-agent: Mozilla/5.0 (compatible; StockNewsDashboard/2.0)
Disallow: /
User-agent: Mozilla/5.0 (compatible; SummitPointLinkCheck/1.0; +https://summitpoint.co)
Disallow: /
User-agent: Mozilla/5.0 (compatible; SummitPointLinkCheck/1.0; +https://summitpoint.co) (variant; SupabaseEdgeRuntime/1.76.0; ref=wsnhmewcwgxpfpoelwfa)
Disallow: /
User-agent: Mozilla/5.0 (compatible; SurdotlyBot/1.0; +http://sur.ly/bot.html)
Disallow: /
User-agent: Mozilla/5.0 (compatible; TaskBountyContactBot/1.0)
Disallow: /
User-agent: Mozilla/5.0 (compatible; TavilyBot/1.0)
Disallow: /
User-agent: Mozilla/5.0 (compatible; TeardownBriefs/1.0; +https://whop.com/48-hour-teardown)
Disallow: /
User-agent: Mozilla/5.0 (compatible; TechBloggerBot/1.0; +personal content-research tool)
Disallow: /
User-agent: Mozilla/5.0 (compatible; The Lounge IRC Client; +https://github.com/lordbex/thelounge) facebookexternalhit/1.1 Twitterbot/1.0
Disallow: /
User-agent: Mozilla/5.0 (compatible; TYPO3-TLD-Census/0.1)
Disallow: /
User-agent: Mozilla/5.0 (compatible; urlsuma/2.0; +https://urlsuma.de/bot.html)
Disallow: /
User-agent: Mozilla/5.0 (compatible; VelenPublicWebCrawler/1.0; +https://velen.io)
Disallow: /
User-agent: Mozilla/5.0 (compatible; velvt-site-model-research/0.1; +https://getvelvt.com)
Disallow: /
User-agent: Mozilla/5.0 (compatible; ViucomWebsiteCheck/1.0)
Disallow: /
User-agent: Mozilla/5.0 (compatible; webatlabot/1.0; +https://webatla.com/bot; abuse@webatla.com)
Disallow: /
User-agent: Mozilla/5.0 (compatible; WebCrawler/1.0)
Disallow: /
User-agent: Mozilla/5.0 (compatible; webXtrendBot/1.1; +https://webxtrend.com/crawler)
Disallow: /
User-agent: Mozilla/5.0 (compatible; WordPressPostChecker/1.0)
Disallow: /
User-agent: Mozilla/5.0 (compatible; WP-Safe-Scanner/1.0)
Disallow: /
User-agent: Mozilla/5.0 (compatible; wpbot/1.4; +https://forms.gle/ajBaxygz9jSR8p8G9)
Disallow: /
User-agent: Mozilla/5.0 (compatible; XcoutBot/1.0; +https://intel.xcout.app/bot) (variant; SupabaseEdgeRuntime/1.76.0; ref=[diverse]
Disallow: /
User-agent: Mozilla/5.0 (compatible; YandexBot/3.0; +http://yandex.com/bots)
Disallow: /
User-agent: Mozilla/5.0 (compatible; yavchn-reader/0.1; +https://github.com/paulmooreparks/yavchn)
Disallow: /
User-agent: Mozilla/5.0 (compatible;PetalBot;+https://webmaster.petalsearch.com/site/petalbot)
Disallow: /
User-agent: Mozilla/5.0 (iPhone; CPU iPhone OS 6_0 like Mac OS X) AppleWebKit/536.26 (KHTML, like Gecko) Version/6.0 Mobile/10A5376e Safari/8536.25 (compatible; SiteAuditBot/0.97; +http://www.semrush.com/bot.html)
Disallow: /
User-agent: Mozilla/5.0 (lead-qualifier; passive fingerprint)
Disallow: /
User-agent: Mozilla/5.0 (Linux; Android 13; Pixel 4a (5G) Build/TQ2A.230505.002; wv) AppleWebKit/537.36 (KHTML, like Gecko) Version/4.0 Chrome/112.0.5615.136 Mobile Safari/537.36 GoogleApp/14.16.27.29.arm64 AppEngine-Google; (+http://code.google.com/appengine; appid: s~virustotalcloud)
Disallow: /
User-agent: Mozilla/5.0 (Linux; arm_64; Android 12; CPH2205) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/110.0.0.0 YaBrowser/23.3.3.86.00 SA/3 Mobile Safari/537.36
Disallow: /
User-agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120 Safari/537.36 (variant; SupabaseEdgeRuntime/1.76.0; ref=tekemubwihsjdzittoqf)
Disallow: /
User-agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120 Safari/537.36 (variant; SupabaseEdgeRuntime/1.77.0; ref=tekemubwihsjdzittoqf)
Disallow: /
User-agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/128.0 Safari/537.36 salesguy-fit/1
Disallow: /
User-agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_1) AppleWebKit/600.2.5 (KHTML, like Gecko) Version/8.0.2 Safari/600.2.5 (Gort)
Disallow: /
User-agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) News Explorer/2.2
Disallow: /
User-agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/15.5 Safari/605.1.15 AppEngine-Google; (+http://code.google.com/appengine; appid: s~virustotalcloud)
Disallow: /
User-agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 14_0) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/126 Safari/537.36 lightdrift-research
Disallow: /
User-agent: Mozilla/5.0 (newsdash; +local)
Disallow: /
User-agent: Mozilla/5.0 (Pentestas-Discovery/1.0; +https://pentestas.com/discovery)
Disallow: /
User-agent: Mozilla/5.0 (research crawler; HN narrative experiment)
Disallow: /
User-agent: Mozilla/5.0 (research; security.txt survey)
Disallow: /
User-agent: Mozilla/5.0 (ThreatIntelAgent/1.0)
Disallow: /
User-agent: Mozilla/5.0 (Windows NT 6.1; Win64; x64; +http://url-classification.io/wiki/index.php?title=URL_server_crawler) KStandBot/1.0
Disallow: /
User-agent: Mozilla/5.0 (Windows NT 6.1; WOW64) SkypeUriPreview Preview/0.5 skype-url-preview@microsoft.com
Disallow: /
User-agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64)
Disallow: /
User-agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/128.0 Safari/537.36 TransixProspector/1.0
Disallow: /
User-agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/135.0.0.0 Safari/537.36 StockerNewsBot/1.0
Disallow: /
User-agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/136.0.0.0 Safari/537.36 KR-InfoRadar/1.0
Disallow: /
User-agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/140.0.0.0 Safari/537.36 Edg/140.0.0.0; 360Spider
Disallow: /
User-agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 Chrome/131 Safari/537.36 CodexProductMonitor/1.0
Disallow: /
User-agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; trendictionbot0.5.0; trendiction search; http://www.trendiction.de/bot; please let us know of any problems; web at trendiction.com) Gecko/20100101 Firefox/125.0
Disallow: /
User-agent: Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/151.0.7922.34 Safari/537.36 FixBeforeLaunchBot/0.1 (+https://fixbeforelaunch.com/bot; readiness scanner)
Disallow: /
User-agent: Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko; GeedoShopProductFinder) Chrome/142.0.0.0 Safari/537.36
Disallow: /
User-agent: Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) HeadlessChrome/134.0.6998.35 Safari/537.36
Disallow: /
User-agent: Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) HeadlessChrome/151.0.7922.71 Safari/537.36 WP.com mShots
Disallow: /
User-agent: Mozilla/5.0 (X11; Linux x86_64) hn-sweep
Disallow: /
User-agent: Mozilla/5.0 (X11; Linux x86_64) OpenClaw-NewsPPT/1.0
Disallow: /
User-agent: Mozilla/5.0 (X11; Linux x86_64) research
Disallow: /
User-agent: Mozilla/5.0 (X11; Linux x86_64) research-read-only
Disallow: /
User-agent: multiply-sourcing/1.0
Disallow: /
User-agent: MuutedFeedFetcher/1.0 (+https://muuted.com/bot; support@muuted.com)
Disallow: /
User-agent: MyNews/0.1 (personal news aggregator; +https://github.com/mynews)
Disallow: /
User-agent: n0v8ba6-research/1.0 (+https://n0v8ba6.com)
Disallow: /
User-agent: n8n
Disallow: /
User-agent: nameserver-checker-http-validator/1.0
Disallow: /
User-agent: NetAPI/1.1 (+https://netapi.com/bot.html)
Disallow: /
User-agent: netscan-research/1.0 (+https://netscan.org/about; admin@netscan.org)
Disallow: /
User-agent: newspaper/0.2.8
Disallow: /
User-agent: newspaper/0.3.0
Disallow: /
User-agent: NextPickBot/1.0 (+local product research)
Disallow: /
User-agent: NextShopwareRadarHarvester/0.1 (+https://www.nextshops.de/radar)
Disallow: /
User-agent: NichetrailBot/0.2 (+https://indie-risers.wangtiechuichui.chatgpt.site/bot)
Disallow: /
User-agent: node
Disallow: /
User-agent: Novan-Research/0.1 (+novan)
Disallow: /
User-agent: okf-engine-capture/1.0
Disallow: /
User-agent: okhttp/5.4.0
Disallow: /
User-agent: omni-mvp/0.1 (personal research aggregator; +https://github.com/)
Disallow: /
User-agent: openmagpie/1.0 (+https://github.com/obris-dev/openmagpie)
Disallow: /
User-agent: OpportunityRadar/0.1 (private local research)
Disallow: /
User-agent: OWLer
Disallow: /
User-agent: Pandalytics/2.0 (https://domainsbot.com/pandalytics/)
Disallow: /
User-agent: Pandalytics/2.0 (+https://domainsbot.com/network-crawler-pandalytics-kdc68dhsbd/)
Disallow: /
User-agent: pilot/1.0 (+https://github.com/ryan-king/pilot)
Disallow: /
User-agent: PipericBot/1.0 (+https://piperic.com/bot)
Disallow: /
User-agent: pixelgrab-radar/1.0 (+https://radar.pixelgrab.lol/about; support@pixelgrab.lol)
Disallow: /
User-agent: PriceVersionBot/0.1 (+https://priceversion.com/bot; official-source price checker)
Disallow: /
User-agent: primal-kernel/0.1
Disallow: /
User-agent: primal-kernel/0.1, pyprimalnews-export/1.0
Disallow: /
User-agent: PromptingBot/1.0.0
Disallow: /
User-agent: Protopage/3.0 (http://www.protopage.com)
Disallow: /
User-agent: PTRagent/1
Disallow: /
User-agent: pulsed/0.4.0 readability-probe
Disallow: /
User-agent: PulseGate-Indexer/1.0 (+https://www.pulsegate.ai/bot)
Disallow: /
User-agent: PulseWatchBot/0.1 (+public reputation monitoring; contact configured by operator)
Disallow: /
User-agent: PUMPKN Research hello@pumpkn.ai
Disallow: /
User-agent: PWAsAppBot/1.0 (+https://pwas.app/)
Disallow: /
User-agent: python-httpx/0.28.1
Disallow: /
User-agent: python-requests/2.28.2
Disallow: /
User-agent: python-requests/2.31.0
Disallow: /
User-agent: python-requests/2.32.4
Disallow: /
User-agent: python-requests/2.32.5
Disallow: /
User-agent: Python/3.12 aiohttp/3.13.5
Disallow: /
User-agent: Readybot.io (https://readybot.io/webmasters)
Disallow: /
User-agent: rss-portal/0.1
Disallow: /
User-agent: RSS-Sweep/2.0 (Content Extractor)
Disallow: /
User-agent: rss/4.5.6
Disallow: /
User-agent: RSSPoller/1.0
Disallow: /
User-agent: Ruby
Disallow: /
User-agent: RuleZeroBot/1.0 (+https://rulezero.de/bot)
Disallow: /
User-agent: SallyProspecting/1.0 (public feedback qualification)
Disallow: /
User-agent: ScourRSSBot/1.0 (+https://scour.ing/bot)
Disallow: /
User-agent: ScrapTrap-Audit-Tool-Pro/1.0
Disallow: /
User-agent: ScrapTrap-Scanner-Pro/1.0
Disallow: /
User-agent: Scrapy/2.16.0 (+https://scrapy.org)
Disallow: /
User-agent: Scrapy/2.17.0 (+https://scrapy.org)
Disallow: /
User-agent: ScryBot/1.0 (+https://scry.io; ethical public-content crawler)
Disallow: /
User-agent: SearchEngine/1.0 (+https://search.tuna-lizard.ts.net/about)
Disallow: /
User-agent: SeobilityBot (SEO Tool; https://www.seobility.net/sites/bot.html)
Disallow: /
User-agent: SignalHat/0.1 (+https://hatos.internal/signalhat)
Disallow: /
User-agent: siteglass-intel/1.0 (+https://siteglass.io)
Disallow: /
User-agent: siteradar/0.1 contact: ops@yyai.studio
Disallow: /
User-agent: Sitetals-Compliance-Scanner/1.0 (+https://sitetals.com/scanner)
Disallow: /
User-agent: SituationAnalysis/1.0 (article fetch; situation-analysis)
Disallow: /
User-agent: Slackbot-LinkExpanding 1.0 (+https://api.slack.com/robots)
Disallow: /
User-agent: slop-bucket/0.1 (+https://slopbucket.app)
Disallow: /
User-agent: Snyfer-Availability/1.0
Disallow: /
User-agent: SoftNews/0.1 (+https://github.com/softnews)
Disallow: /
User-agent: SofyaBot/1.0 (+https://sofya.co/bot)
Disallow: /
User-agent: Sogou web spider/4.0(+http://www.sogou.com/docs/help/webmasters.htm#07)
Disallow: /
User-agent: SonarPrime RedirectMap/1.0
Disallow: /
User-agent: SpaceCowboys Android RSS Reader / 2.22.0(4050)
Disallow: /
User-agent: sqlmap
Disallow: /
User-agent: startup-scan/1.0 (personal market research; contact via site owner)
Disallow: /
User-agent: statamic-lead-research/1.0 (+https://stoffel.io; alex@stoffel.io)
Disallow: /
User-agent: stealth-lab-trends/1.0 (by /u/manougv)
Disallow: /
User-agent: SuperfluidityBot/1.0 (+https://superfluidity.ai/bot)
Disallow: /
User-agent: TALOS/1.0 (+intelligence platform; contact via source owner)
Disallow: /
User-agent: TechFeed/1.0
Disallow: /
User-agent: TechFeedBot/1.0 (Lambda; content-aggregator)
Disallow: /
User-agent: TheMapBot/0.1 (SaaS product enumeration research; +mailto:henryfried67@gmail.com)
Disallow: /
User-agent: ThreatIntelBot/1.0 (security research)
Disallow: /
User-agent: timing-research/1 (contact: gn@gabornyeki.com)
Disallow: /
User-agent: Timpibot/1.0 (+http://timpi.io/crawler)
Disallow: /
User-agent: Tiny Tiny RSS/25.02-169ff6de (https://tt-rss.org/)
Disallow: /
User-agent: touchstone/0.1 (+personal feed reader)
Disallow: /
User-agent: trafilatura/1.12.2 (+https://github.com/adbar/trafilatura)
Disallow: /
User-agent: trend-pulse/hn-enrich
Disallow: /
User-agent: Turnitin (https://bit.ly/2UvnfoQ)
Disallow: /
User-agent: twitter-reader-paper-resolver
Disallow: /
User-agent: ucp-crawler/0.1 (+https://github.com/huckleberry-inc/ucp-crawler)
Disallow: /
User-agent: uk-nhs-data/0.0 (internal fraud-research)
Disallow: /
User-agent: undici
Disallow: /
User-agent: vegapunk-collector/1.0 (+HN trend research)
Disallow: /
User-agent: VeristriaCensus/1.0 (+https://veristria.com)
Disallow: /
User-agent: vibelaunched/1.0 (+https://vibelaunched.com; market-scan bot)
Disallow: /
User-agent: visionheight.com/scan Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) Chrome/126.0.0.0 Safari/537.36
Disallow: /
User-agent: vroni-job-signals/1.0 (+manual review)
Disallow: /
User-agent: VSBot/4.0 (+https://vsbot.de/bot)
Disallow: /
User-agent: w3m/0.5.6
Disallow: /
User-agent: webshopok-kutatas/0.1 (+kapcsolat: test@datalistshop.eu; belso piackutatas, .de webshopok)
Disallow: /
User-agent: wedgemap-crawler/1.0 (launch trend research)
Disallow: /
User-agent: WordPress.com; https://metaaldetectievlaanderen.com
Disallow: /
User-agent: WordPress/7.1; https://leafext.de/hp
Disallow: /
User-agent: wp2shell
Disallow: /