Chrome Extension: Reading the BODY of an HTTP response object

This article is for people who have a basic understanding of how Chrome Extensions work.

The Chrome Extension ecosystem provides APIs that allow us to partially read and modify request/response headers out of the box. We have many extensions that leverage these APIs like Requestly, Tamper Chrome.

However, when it comes to reading the body of an HTTP request, things get a bit tricky. In this article, we are going to explore a couple of methods to achieve this and while doing so, we’ll also look at the drawbacks associated with these methods.

Intercepting data using content scripts

Content scripts are conditionally injected scripts that run in the context of the current web page. A content script doesn’t have access to the JavaScript running inside a web page. Instead, it runs in isolation and has permissions to read the DOM and write to it using standard JavaScript APIs.

We are going to inject a custom script inside the DOM using our content script and use it to read the response body of HTTP requests. Now, let’s configure our content script.

Here’s what our manifest.json looks like:

{
    "content_scripts": [{
        "js": ["contentScript.js"],
        "run_at": "document_start"
    }]
}

The run_at field tells the extension to inject the content script in the web page even before the DOM has been constructed. Let’s look at the contents of the content script now:

function interceptData() {
  var xhrOverrideScript = document.createElement('script');
  xhrOverrideScript.type = 'text/javascript';
  xhrOverrideScript.innerHTML = `
  (function() {
    var XHR = XMLHttpRequest.prototype;
    var send = XHR.send;
    var open = XHR.open;
    XHR.open = function(method, url) {
        this.url = url; // the request url
        return open.apply(this, arguments);
    }
    XHR.send = function() {
        this.addEventListener('load', function() {
            if (this.url.includes('<url-you-want-to-intercept>')) {
                var dataDOMElement = document.createElement('div');
                dataDOMElement.id = '__interceptedData';
                dataDOMElement.innerText = this.response;
                dataDOMElement.style.height = 0;
                dataDOMElement.style.overflow = 'hidden';
                document.body.appendChild(dataDOMElement);
            }               
        });
        return send.apply(this, arguments);
    };
  })();
  `
  document.head.prepend(xhrOverrideScript);
}
function checkForDOM() {
  if (document.body && document.head) {
    interceptData();
  } else {
    requestIdleCallback(checkForDOM);
  }
}
requestIdleCallback(checkForDOM);

Firstly, what we are doing here is extending XMLHttpRequest’s send and open methods with our own code by overriding it’s prototype.

The ‘load’ event listener inside send is fired when we get a response for the request made. In this code sample, ‘this’ refers to a specific instance of XHR and this.response contains the HTTP response for the instance.

Next, we store the response inside the DOM using a ‘div’ element . Also, notice that in order to avoid disturbing the styling/content of the web page we use the ‘height’ and ‘overflow’ CSS properties to hide this element but at the same time be readable using native DOM APIs.

Now, we need to pass on this stored data to the extension. For this, we use our content script to read it. This wouldn’t have been possible with a different type of extension because only a content script can have access to a web page’s DOM.

Let’s add a function to the bottom of our content script that continuously checks whether the data has been appended to the hidden DOM element or not:

function scrapeData() {
    var responseContainingEle = document.getElementById('__interceptedData');
    if (responseContainingEle) {
        var response = JSON.parse(responseContainingEle.innerHTML);
    } else {
        requestIdleCallback(scrapeData);
    }
}
requestIdleCallback(scrapeData); 

The requestIdleCallback is there to ensure that the main thread of the content script doesn’t get stuck in an infinite loop.

That’s about it! Since the response data is now available in our content script, we can now send it to our background script which, in turn, can send it to any part of our extension. The reason why we may want to send the data to the background page is that content scripts have limited access to extension APIs compared to a background page.

Drawback

There’s no way to make this work with requests that are fired on load of the page or a server side rendered page because our content script is initiated only when the DOM construction starts in the browser.

Intercepting data using DevTools extension:

A DevTools extension provides us with an out of the box solution to read response data. Let’s configure it in the manifest:

{
    "devtools_page": "devtools.html"
}

As you can see, we only need to point the manifest to our devtools html page. Let’s see the contents of devtools.html now:

<script src="devtools.js"></script>

devtools.html only needs to point to the js file. Let’s see how devtools.js looks:

chrome.devtools.panels.create("MyPanel", null, 'panel.html');

devtools.js creates a custom panel in the Chrome DevTools that looks like this:

panel.html contains the HTML that will displayed in the DevTools extension. Since, we don’t need to have any HTML for our use case, we will just have a very simple file:

<html>
    <body>
        <script src="panel.js"></script>
    </body>
</html>

Let’s look at the final piece of our puzzle, which is panel.js:

chrome.devtools.network.onRequestFinished.addListener(request => {
  request.getContent((body) => {
    if (request.request && request.request.url) {
      if (request.request.url.includes('<url-to-intercept>')) {
        chrome.runtime.sendMessage({
            response: body
        });
      }
    }
  });
});

We use chrome.devtools.network.onRequestFinished.addListener which adds a listener for all requests being made in the web page. In the callback, we use the request.getContent API to read the response of each request and use chrome.runtime.sendMessage to send the response to other parts of the extension.

Drawback

The drawback for this method is that we have to keep the Chrome DevTools open all the time because DevTools extensions are only activated when DevTools is open.

Conclusion

Both the ways we discussed come with their own drawbacks. The first method is more complicated but works better when you want to intercept a request that is fired after the page is loaded. On the other hand, the second method is simpler and works for all cases but requires the DevTools to be open at all times.

I have read that there’s another way of achieving our goal which is by using the chrome.debugger API but I haven’t checked it out yet.

Feedbacks and critiques are welcome! Also, if you experience any difficulty in implementing or understanding the above solutions do let me know in the comments below.

Thank you for reading! 😃