Understanding and Troubleshooting Web Application Security Scanning

The Web Application Security risk vector uses a headless web browser to gather information about web servers and applications to perform assessments related to the security controls implemented.

What Is a Headless Web Browser?

Headless browsers provide automated control of a web page in an environment similar to popular web browsers, but they are executed using a command-line interface or network communication. Bitsight uses this functionality to render and understand HTML the same way a browser would, including styling elements (page layout, color, font selection, and execution of JavaScript) which are usually not available when using other testing methods.

How Does the Data Collection Work?

In the scope of WAS, the headless browser is used for hostname-based scanning, gathering information about web servers and best practices as well as information about web applications being served from those web servers. This includes information such as:

• HTTP headers.
• Content Security Policies.
• Web application components and version detection (e.g. JavaScript libraries).
• Web page contents and their security context (e.g. HTTP resources loaded from an HTTPS context).
• Internet-accessible administration pages (e.g. admin portals for Wordpress, Drupal, and other Content Management Systems).

The headless browser loads the root web page of every domain on HTTP (port 80) and HTTPS (port 443) and follows any redirects just like a regular browser would. We wait for the page to finish loading or for a timeout to elapse, then gather the headless browser data. This includes low-level information on network interactions, security issues reported by the browser, files loaded, etc. We use this information to make all risk vector assessments, and those assessments generate the findings.

Troubleshooting Web Application Security Scanning

Bitsight’s headless browser follows the Robots Exclusion Protocol. This means that depending on the configuration of a web server, the headless browser may not be allowed to scan a web application and retrieve the contents needed for the assessments.

The following is a sample configuration of the application's robots.txt file with an Allow rule for the BitSightBot agent:

User-agent: BitSightBot
Allow: /

This may not be necessary for all applications and the specific configuration may vary. Check the Robots Exclusion Protocol for more information.