A user agent is a string sent to the webserver when a page is requested by a browser. User Agent is a request header that is included with other HTTP headers and sent by the browser to identify itself which operating system and type of browser it is. The user agent is sent with every HTTP request it makes to a web server. Each browser provides a built-in user agent string when making a request, but this string can be changed by a user. An example of a user agent string may look like this:

As you can see above, the user agent string is a series of character strings that describes the platform, operating system, and browser version. On the example above, the user agent identifies itself as macOS 11.2.1 (Big Sur) running Chrome 88.0 version and pretending to be a Safari browser. Due to a complex history of the browser market, and sharing of technology within the browser industry caused the user agent string to be lengthy and cryptic.

How to change user agent?

As not all web browsers display HTML documents the same way, web developers style the web pages to fit every type of browser, and CSS plays a significant role in it. However, a user may arbitrarily change the user agent with a switcher, and Google Chrome offers one as part of its developer tools. Start Chrome -> Settings -> More Tools -> Developer Tools, and click on the "Console" tab. At the bottom of the screen, click on the three dots next to the "console", and choose "Network Conditions" as shown below.

Google Chrome Developer Tools - User Agent Switcher

On firefox, you may have to use about:config page by typing "about:config" on the address bar. Other browsers provide a similar method to change the browser's user agent. Alternatively, you may use a browser extension to change the user agent before requesting a page.

If you're requesting a page from a Linux console (i.e. wget) or through a PHP script library such as Snoopy, you may arbitrarily change user agent via command-line option or through a variable assignment.

What are user agent types?

Web browser is one type of agent requesting web pages, and there are other tools working as agents and retrieve web pages. For example, search engine bots acting as agents to crawl web pages and console games such as Nintendo DS and Playstation act as agents to deliver games within the built-in browsers. Here are a few types of user agents:

• Web Browsers: Google Chrome, Safari, Firefox, Microsoft Edge, Opera, and more.
• Bots: Googlebot, Baidu Spider, BingBot, Yandex Bot, and more.
• Consoles: Playstation, Wii, Nintendo, and etc.
• Programming Tools: wget, Snoopy, and etc.
• Link checker: W3C-checklink

User Agents and Robots.txt

A robots.txt file on the web server lets the search engines (crawlers) know which user agents can access which pages or files. This is to allow search bots to avoid crawling "dev" or "qa" sites, and backend (admin) pages. Similarly, "noindex" and "nofollow" directives along with password-protected folders can keep bots out of web pages.