asked on July 16, 2024
I have configured Federated Search to crawl a client's webpage, and it's completing but returning zero documents. The client has reviewed their WAF logs, and are seeing blocks around the scheduled run time where the requests do not have User Agent strings. Does the crawler include user agent strings in its requests? The client says they automatically block all requests without them.
1
0