CustomDomain Docs
Account and security

CustomDomainBot

What CustomDomainBot checks and how to opt out.

CustomDomainBot is how CustomDomain™ checks whether a domain that someone entered responds on the public internet. If you found CustomDomainBot/1.0 in your server logs or your DNS query logs, this page says what it did and what we keep.

Why it visits

When a person enters your domain in CustomDomain™ (to search for it, check it, price it, connect it or buy it), or signs up with an address at your company's domain, we look at the domain's public DNS records and make one ordinary request for its home page. We do it to help connect the domain, and to tell a live domain from a parked, redirecting or dead one. We have no interest in what your site says.

What it requests

  • Public DNS. The domain's addresses, nameservers, mail and certificate authorization records, and its SPF and DMARC records. We read no other TXT record. The questions go to public resolvers, never to servers your DNS named.
  • One web request. GET / over HTTPS, or over HTTP when HTTPS cannot connect, shake hands or answer in time. It follows up to five redirects and then stops. It never asks for another path, never submits a form, never sends cookies or credentials and never sends a referrer.
  • Only ordinary ports. It connects to ports 80 and 443 on public addresses, and refuses anything private, internal or reserved, including on a redirect.
  • A plain introduction. User-Agent: CustomDomainBot/1.0 (+https://docs.customdomain.ai/docs/security/domain-checks).
  • Rarely. At most once a day for a domain, however many people entered it. A domain that does not answer is tried again after a longer and longer wait, up to a day.

What we keep

The DNS records we read, whether the domain answered and with what status, where it redirects (without query strings), the type and size of the page, whether its certificate was valid, and the page title. We read at most the first 64 KiB of an HTML page and keep nothing from it but the title. We keep no cookies, no other headers and no page content, and the result is a fact about the domain, not about any person or account.

When you save an origin

There is one more request that carries the same name. A customer of CustomDomain™ who sends their visitors through our edge saves an origin, the server that answers for those domains, under Routing in the console. When they save it, and when they choose Check again, our server makes one request to that origin so the console can say whether it answers. If CustomDomainBot visited a server you run and the address was typed into a workspace, this is the likeliest reason.

  • One web request. GET / to the origin that workspace saved, with a 6 second limit. It reads the status line and drops the connection; it never reads the page. It follows no redirect (a 3xx is an answer), sends no cookies, credentials or referrer, and never asks for another path.
  • Only ordinary ports. Ports 80 and 443, on public addresses. An origin on any other port is not requested, and a private, internal or reserved address is refused when the connection is made.
  • Rarely. Only when someone saves that origin or asks for the check, and not more often than once every 15 seconds for the same origin.
  • What we keep. The origin that was checked, whether it answered and with what status (or why it did not), and when. No page content and no headers. The result belongs to that workspace and is shown only for the origin it was run against.
  • Advice only. The result never stops anyone saving, and nothing breaks if the check is refused: you can block CustomDomainBot at your server or firewall. If a workspace saved the address of a server that is not theirs, write to [email protected] with the address.

Ask us to stop

Write to [email protected] with the domain. We stop checking it and delete what we hold on it. You can also block CustomDomainBot at your server or firewall: it identifies itself in the User-Agent header, and nothing breaks if it is refused.

The Privacy Policy describes domain checks in section 3.3.

On this page