Problem marking as a capture when using IF and JUMP!

AINTHATEASY · August 23, 2022, 9:04pm

Hello people!

So I tried to make a statement where whenever the config finds a specific condition it has to jump and try something else then parse something and mark it as a capture. Unfortunately this did not work with me because it does mark it as a capture in the Log section but I can’t see it marked as a capture in the Variables section, also I’ve tried to run a job with that and still can’t see it in the capture section when I get a hit.

In the bottom you’ll find the LoliCode. I created the first request with a wrong URL in purpose just to deliver my point which is when the config finds that condition with a specific variable it has to jump and try another request with another variable in the URL but if the condition does not match, it should go for the first request only and make the parses then just stop and complete the other requests bellow. Sorry for the long explanation

Here is the output in the Log section:

Here is the output (all blank) in the Variables section:

I used the site @Ruri usually uses for tests. This is the whole LoliCode:

BLOCK:HttpRequest
LABEL:GET-V1
  url = "https://webscraper.io/test-sites/e-commerce/test"
  TYPE:STANDARD
  $""
  "application/x-www-form-urlencoded"
ENDBLOCK
IF STRINGKEY @data.RESPONSECODE EqualTo "404"
JUMP #ANOTHERONE
BLOCK:Parse
LABEL:PRICES
  input = @data.SOURCE
  leftDelim = "class=\"pull-right price\">"
  rightDelim = "</"
  attributeName = ""
  RECURSIVE
  MODE:LR
  => CAP @PRICES
ENDBLOCK

BLOCK:Parse
LABEL:DESCRIPTION
  input = @data.SOURCE
  leftDelim = "class=\"description\">"
  rightDelim = "</"
  attributeName = ""
  RECURSIVE
  MODE:LR
  => CAP @DESCRIPTION
ENDBLOCK

#ANOTHERONE
BLOCK:HttpRequest
LABEL:GET-V2
  url = "https://webscraper.io/test-sites/e-commerce/allinone"
  TYPE:STANDARD
  $""
  "application/x-www-form-urlencoded"
ENDBLOCK

BLOCK:Parse
LABEL:PRICES
  input = @data.SOURCE
  leftDelim = "class=\"pull-right price\">"
  rightDelim = "</"
  attributeName = ""
  RECURSIVE
  MODE:LR
  => CAP @PRICES
ENDBLOCK

BLOCK:Parse
LABEL:DESCRIPTION
  input = @data.SOURCE
  leftDelim = "class=\"description\">"
  rightDelim = "</"
  attributeName = ""
  RECURSIVE
  MODE:LR
  => CAP @DESCRIPTION
ENDBLOCK

END

DJHooligan · August 24, 2022, 10:39pm

put your IF block lower, this works;

BLOCK:HttpRequest
LABEL:GET-V1
  url = "https://webscraper.io/test-sites/e-commerce/test"
  securityProtocol = TLS12
  TYPE:STANDARD
  $""
  "application/x-www-form-urlencoded"
ENDBLOCK

BLOCK:Parse
LABEL:PRICES
  input = @data.SOURCE
  leftDelim = "class=\"pull-right price\">"
  rightDelim = "</"
  attributeName = ""
  RECURSIVE
  MODE:LR
  => CAP @PRICES
ENDBLOCK

BLOCK:Parse
LABEL:DESCRIPTION
  input = @data.SOURCE
  leftDelim = "class=\"description\">"
  rightDelim = "</"
  attributeName = ""
  RECURSIVE
  MODE:LR
  => CAP @DESCRIPTION
ENDBLOCK
IF STRINGKEY @data.RESPONSECODE EqualTo "404"
JUMP #ANOTHERONE

#ANOTHERONE
BLOCK:HttpRequest
LABEL:GET-V2
  url = "https://webscraper.io/test-sites/e-commerce/allinone"
  securityProtocol = TLS12
  TYPE:STANDARD
  $""
  "application/x-www-form-urlencoded"
ENDBLOCK

BLOCK:Parse
LABEL:PRICES
  input = @data.SOURCE
  leftDelim = "class=\"pull-right price\">"
  rightDelim = "</"
  attributeName = ""
  RECURSIVE
  MODE:LR
  => CAP @PRICES
ENDBLOCK

BLOCK:Parse
LABEL:DESCRIPTION
  input = @data.SOURCE
  leftDelim = "class=\"description\">"
  rightDelim = "</"
  attributeName = ""
  RECURSIVE
  MODE:LR
  => CAP @DESCRIPTION
ENDBLOCK

END```

AINTHATEASY · August 24, 2022, 10:57pm

I will check it out.

AINTHATEASY · August 24, 2022, 11:10pm

Actually I tried that one several times and here is what I get:

It doesn’t allow the config to proceed and read the IF STATEMENT so that’s exactly why I want to put the IF STATEMENT right under the request not under the parsing blocks in order to make it jump to the other request if the RESPONSECODE is EqualTo 404 but if the RESPONSECODE is EqualTo 200 then it should only do the first request with the parsing blocks under it and then complete the other requests below (not the second request that was meant for the RESPONSECODE is EqualTo 404)

DJHooligan · August 25, 2022, 6:10am

BLOCK:HttpRequest
  url = "https://checkerproxy.net/api/archive/"
  securityProtocol = TLS12
  customHeaders = {("accept", "text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3"), ("accept-encoding", "gzip, deflate, br"), ("accept-language", "en-US,en;q=0.9"), ("cookie", "_ga=GA1.2.1349792006.1653848537; _gid=GA1.2.463677074.1661400530; SL_G_WPT_TO=nl; SL_GWPT_Show_Hide_tmp=1; SL_wptGlobTipTmp=1; _gat=1"), ("dnt", "1"), ("sec-fetch-mode", "navigate"), ("sec-fetch-site", "none"), ("sec-fetch-user", "?1"), ("upgrade-insecure-requests", "1"), ("user-agent", "Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/77.0.3865.10 Safari/537.36 Edg/77.0.235.5")}
  TYPE:STANDARD
  $""
  "application/json"
ENDBLOCK

BLOCK:Parse
  input = @data.SOURCE
  leftDelim = "\"date\":\""
  rightDelim = "\",\""
  caseSensitive = False
  jToken = "$..date"
  RECURSIVE
  MODE:Json
  => CAP @PARSED
ENDBLOCK

IF STRINGKEY @data.RESPONSECODE EqualTo "404"
JUMP #ANOTHERONE

#ANOTHERONE
BLOCK:HttpRequest
  url = "https://checkerproxy.net/api/archive/2022-08-25"
  securityProtocol = TLS12
  customHeaders = {("accept", "text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3"), ("accept-encoding", "gzip, deflate, br"), ("accept-language", "en-US,en;q=0.9"), ("cookie", "_ga=GA1.2.1349792006.1653848537; _gid=GA1.2.463677074.1661400530; SL_G_WPT_TO=nl; SL_GWPT_Show_Hide_tmp=1; SL_wptGlobTipTmp=1; _gat=1"), ("dnt", "1"), ("sec-fetch-mode", "navigate"), ("sec-fetch-site", "none"), ("sec-fetch-user", "?1"), ("upgrade-insecure-requests", "1"), ("user-agent", "Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/77.0.3865.10 Safari/537.36 Edg/77.0.235.5")}
  TYPE:STANDARD
  $""
  "application/json"
ENDBLOCK

BLOCK:Parse
  input = @data.SOURCE
  leftDelim = "\"id\":"
  rightDelim = ",\""
  jToken = "$..id"
  RECURSIVE
  MODE:Json
  => CAP @PARSED
ENDBLOCK

END

the error in the printscreen means that you provided ill formed Json data in a POST request…
check out the example above, it parses json data fine!

if you are trying to enter login data in as json, check the format.
regards

AINTHATEASY · August 25, 2022, 6:17am

In fact it is a GET request not POST! Let me give you more details, this GET request works fine sometimes but other times it doesn’t, because I have to change the URL to get the response that I need to parse from, that’s why I have to make two requests with an IF STATEMENT which another URL with the same variable parsed earlier, when the first request goes wrong with that variable it has to try the second request with another URL but the same variable, you got me?

AINTHATEASY · August 25, 2022, 6:25am

For instance this is my variable: PARSEDPRICE

The 1st request is with this URL: https://checkerproxy.net/api/archive/latest/<PARSEDPRICE>
The 2nd request is with this URL: https://checkerproxy.net/api/archive/<PARSEDPRICE>

So whenever the first request goes 404 RESPONSECODE the config has to try the other request right under it, which has the same response as the first but only when the first request goes 404 RESPONSECODE I hope this makes it easier to understand.

DJHooligan · August 25, 2022, 6:46am

sorry the error means that you are trying to parse some data as json, but data.SOURCE isn’t json…

AINTHATEASY · August 25, 2022, 6:52am

Okay, I got what you mean, I was using parsing through Json so that’s why I get that error when the response is not in Json right?
Now I tried to parse usign LR and it didn’t give me that error but still not marking it as capture!

DJHooligan · August 25, 2022, 6:57am

is your config in this order?

Http Request
Parse
LoliCode
Http Request
Parse
LoliCode (END)

AINTHATEASY · August 25, 2022, 7:02am

Yes exactly, it just worked fine! I don’t know how but it marked them as a capture once I rec-created new parsing blocks. However what I still don’t understand is why that error pops up when I use Json parsing method why it doesn’t just ignore that? Or is that how Json works no matter what?

DJHooligan · August 25, 2022, 7:06am

that’s the way Ruri programmed his rurilib!

AINTHATEASY · August 25, 2022, 7:11am

Alright, I should wait for Ruri to arrive and ask him for that. Big respect man I really appreciate your help and I wish I can provide some tips like you do for the others posting in this amazing forum

AINTHATEASY · August 26, 2022, 7:30am

Hello Mr. @Ruri any information about this one?

Ruri · August 27, 2022, 9:30pm

Hi, I see you’re trying to parse HTML using the Parse block in JSON mode. I guess this only happens in some cases. You could put a keycheck before the parse to identify the 404 and either FAIL or BAN the proxy, or you could just mark the Parse block as SAFE so it will not crash regardless of what happens inside the block itself.

AINTHATEASY · August 28, 2022, 6:39am

Fully understood, thanks for the info.

AINTHATEASY · August 29, 2022, 12:17am

Forgive me but I have another question about the variable output name, for example I want to mark my variable as a capture and name it like this: FIRST ONE or FIRST-ONE it automatically gets renamed to this: FIRSTONE removing every symbol or space character in it! We used to do this without any problem in OB1, I’d like to know if there is any setting I should configure to allow me put any character in the output name.

Ruri · August 29, 2022, 6:16am

OB1 used a different system, OB2 compiles down to C# so if it’s not a valid variable name in C# then it’s not accepted. You can use FIRST_ONE as it’s a valid variable name.

AINTHATEASY · August 29, 2022, 7:06am

I see, so there is no way that we can use spaces or symbols other than the underscores and numbers in the variable name or there might be an update for that? I really hope an update could be possible in the future. Thank you.

Ruri · August 30, 2022, 4:46pm

Exactly. I could implement something like an alias which maps variable names to “pretty” ones before the capture string is constructed. Please open an issue on github mentioning this thread if you want me to implement it.