URLertBot

Automated Security Scanner Policy

URLertBot is the automated security scanner for URLert.com. We analyze URLs to detect malicious content, including phishing, scams, and malware distribution.

Purpose & Mission

The primary purpose of URLertBot is Public Safety. We scan websites to identify threats that endanger internet users, organizations, and platforms.

  • Category: Phishing, Scam, & Malware Detection
  • Benefit: Our data protects the internet ecosystem by:
    • Protecting Users: Verifying links in emails or messages (e.g., detecting fake banking login pages) before a user clicks.
    • Securing Platforms: Helping public sharing services and platforms identify if users are hosting malicious data or bad links.
    • Assisting Owners: Alerting domain owners if their infrastructure has been compromised or injected with malicious code.

Identification

URLertBot identifies itself clearly in the HTTP User-Agent header.

User-Agent: Mozilla/5.0 (compatible; URLertBot/1.0; +https://urlert.com/bot)

Verification

To verify that traffic is genuinely from URLert, checking the User-Agent is not sufficient as it can be spoofed. We provide two ways to verify official URLertBot traffic:

1. Reverse DNS Verification (Recommended)

The most reliable way to verify URLertBot is using Reverse DNS (PTR record). All official requests originate from IP addresses that reverse resolve to bot.urlert.com.

Example verification steps:

  1. Run a reverse DNS lookup on the source IP (e.g., host 34.61.148.192).
  2. Verify that the domain name is bot.urlert.com.
  3. Run a forward DNS lookup on that domain name (e.g., host bot.urlert.com) and confirm it points back to the same source IP.

2. Public IP List

If you prefer IP-based whitelisting, we provide a strictly defined list of IP addresses that our official bot operates from.

Note: While we strive for stability, our IP addresses may change over time. We recommend using the Reverse DNS method or periodically polling the IP list for updates.

Behavior & Ethics

We strive to be a "Good Citizen" of the internet. Our scanning behavior is designed to be non-intrusive.

  • Security-First Scanning: URLertBot is a security scanner, not a search engine crawler. Because threat actors routinely add robots.txt disallow rules to conceal malicious content from detection, following those rules would directly undermine our mission to protect users. URLertBot therefore scans only the specific URLs submitted for analysis, regardless of robots.txt directives.
  • Traffic Volume: We do not index entire websites. We only fetch specific URLs that have been submitted for security analysis.
  • Abuse Prevention: Our bot is not used for credential stuffing, inventory scalping, or aggressive scraping. It is strictly a read-only analysis tool used to detect threats.

Controlling URLertBot

Because URLertBot is a security tool rather than a general-purpose crawler, robots.txt disallow directives are not honored. This is intentional — malicious actors routinely use robots.txt to hide harmful content from automated scanners. Honoring these rules would allow attackers to evade detection and put your users at risk.

If you would like to opt your domain out of URLertBot scanning entirely, please contact us directly at support@urlert.com. We review opt-out requests on a case-by-case basis.

Contact

If you believe URLertBot is malfunctioning, accessing sensitive data, or causing issues for your infrastructure, please contact our engineering team immediately.