Here’s how to analyze a web page from spider’s point of view from Linux terminal:
Issue the following command at the terminal –
wget -r –spider <URL to the web page>
For example, to analyze ‘www.google.com’ from spider’s point of view, issue the following command at the terminal –
wget -r –spider www.google.com
