Handle interactions, JSON and error page

Hi there, openbullet is an amazing tool that as I see can replicate all browser interactions
I would like some help to discover what wrong is going on, I am started this job 18 hour ago can’t sleep

My target URL makes available an API key that we can use by curl or whatever but they are seriously limited the number of requests, blocking and making it impossible to use. By otherside, the requests made by web browser works perfectly like no restrictions

An resume of problem: I am receiving a index response, with no results and in second block request I am receivind an 404 page error. I have some guesses (see below)

The code with url working is on paste links at side of code quotes

OPENBULLET LOLICODE WORKING CODE (removed)
Password: ruiopenbullet

Here is the openbullet loli code

REQUEST GET "http/www.url.br/busca?termo=value" 
  
  HEADER "Host: www.url.br" 
  HEADER "User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:78.0) Gecko/20100101 Firefox/78.0 Waterfox/78.12.0" 
  HEADER "Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8" 
  HEADER "Accept-Language: en-US,en;q=0.5" 
  HEADER "Accept-Encoding: gzip, deflate" 
  HEADER "Connection: keep-alive" 
  HEADER "Upgrade-Insecure-Requests: 1" 

Here is the LOG result of above code

Response code: 200 (OK)
Received headers:
Expires: 0
Cache-Control: no-cache, no-store, max-age=0, must-revalidate
X-XSS-Protection: 1; mode=block
Pragma: no-cache
X-FRAME-OPTIONS: SAMEORIGIN
Date: Fri, 18 Jun 2021 06:04:11 GMT
Connection: keep-alive
Vary: Accept-Encoding
X-Content-Type-Options: nosniff
Content-Type: text/html;charset=UTF-8
Content-Language: en-US
Content-Encoding: gzip
content-length: 15724
Received cookies:
JSESSIONID: MVbN7N69yBZXjWsibFwEeqvnq1N3wWFREVL8LM_n.idc-jboss2-ap8-p
REQUEST GET "http/www.url.br/resultado?termo=value&pagina=1&tamanhoPagina=10&t=8fcnCCGGtdlLPP3hcu8x" 
  
  HEADER "User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:78.0) Gecko/20100101 Firefox/78.0 Waterfox/78.12.0" 
  HEADER "Accept: */*" 
  HEADER "Accept-Language: en-US,en;q=0.5" 
  HEADER "Accept-Encoding: gzip, deflate" 
  HEADER "X-Requested-With: XMLHttpRequest" 
  HEADER "Connection: keep-alive" 
  HEADER "Referer: http/www.url.br/busca?termo=value" 

Here is the LOG result of above code

headers and cookies...
Response code: 404

The response is the requested page, and it’s all.
But acessing by browser by the request get url “http/www.url.br/busca?termo=value” the results is displayed correctly.

SUPOSES

On charles proxy, I can compose the URL request and also can get results, in summary tab will display
the URL fired by GET REQUEST “http/www.url.br/busca?termo=value” (type TEXT/HTML) but more interesting, shows a new resource named “url.br/resultado?=values&some&data” (type application/json), this JSON code is the needed response, that was the URL specified on my second REQUEST GET, as showed above on LOLI CODE.

CHARLES AND CHROME SAVED SESSION (removed)

On chrome I discovered also that the result text will not appear on openbullet index page preview because it’s JSON, but one time received the data they will appear on log. But for some reason when firing the lolicode by openbullet I am not able to receive the expected result (the content of “resultado?” JSON code)

The network tab will show

(status/method/domain/file/initiator)
200/GET/URL.BR/busca?termo=value/document
200/GET/URL.BR/resultado?=values&some&data/jquery-1.12.0.min.js:4(XHR)

What is the next step?

Considering that first request is using the correctly URL, the URL that when called by a address bar in a browser load by himself the complete data, same in charles and if the data is coming probably have a way to capture. I already tried to capture but not data is coming. My best try was make another request trying to call directly resultado?=values&some&data but the result is an 404 page.

I supose that is because the “resultado?=values&some&data” is called by jquery, and how to replicate into openbullet?

Using now OpenBullet 1.4.4
Thank you for read

http removed because I am a new user and have restrictions
PS: No captcha is needed.
Please do not paste working url in reply, if possible upload it on pastebin.pl with pass or remove/replace working url and let the rest code.

Update:

Not sleeped yet :frowning:
I was analyzing again the first request (a index search page) “busca?termo=value” in chrome network and I seem that they have a token, I just done updated my code to parse token but still not showing results. In fact, this was a problem but still not the reason not to load the results.

The token comes in response body of first request

function buscar() {

        var token = '&t=xJe4LVGixZQsStwUMMah';

        if (!$("#form-busca-indice").valid()) {
            return;
        }
        $("#resultados").html("");
        $(".loading-busca").show();
        $.get(springUrl + "busca/resultado", $("#form-busca-indice").serialize() + getFiltros() + token, exibirResultado);
        window.history.replaceState({termo: $("#termo").val()}, document.title, "?termo="+$("#termo").val()+getFiltros());
    }

    $('.no-collapsable').on('click', function (e) {
        e.stopPropagation();
    });

And analyzing again initiators from the "resultado?=values&some&data” (the result page)
I found this:

(function inject() {
        var open = XMLHttpRequest.prototype.open;

        XMLHttpRequest.prototype.open = function () {
          this.requestMethod = arguments[0];
          open.apply(this, arguments);
        };

        var send = XMLHttpRequest.prototype.send;

        XMLHttpRequest.prototype.send = function () {
          var onreadystatechange = this.onreadystatechange;

          this.onreadystatechange = function () {
            function GenerateQuickId() {
              var randomStrId = Math.random().toString(36).substring(2, 15) + Math.random().toString(36).substring(2, 15) + Math.random().toString(36).substring(2, 15);
              return randomStrId.substring(0, 22);
            }

            try {
              if (this.readyState === 4) {
                var id = 'detector';
                var mes = {
                  posdMessageId: 'PANELOS_MESSAGE',
                  posdHash: GenerateQuickId(),
                  type: 'VIDEO_XHR_CANDIDATE',
                  from: id,
                  to: id.substring(0, id.length - 2),
                  content: {
                    requestMethod: this.requestMethod,
                    url: this.responseURL,
                    type: this.getResponseHeader('content-type'),
                    content: this.response
                  }
                };
                window.postMessage(mes, '*');
              }
            } catch (e) {}

            if (onreadystatechange) {
              return onreadystatechange.apply(this, arguments);
            }
          };

          return send.apply(this, arguments);
        };
      })();

I can’t wait to be answered by ruri :smiley: