Iterating on lists

Let’s say you parsed a list with some URLs using the parse block in recursive mode, and then you want to iterate over that list to execute something for each element in the list, let’s say for example parsing some value from each page and then add it to a master list.

BLOCK:HttpRequest
  url = "https://webscraper.io/test-sites/e-commerce/allinone"
  TYPE:STANDARD
  $""
  "application/x-www-form-urlencoded"
ENDBLOCK

BLOCK:Parse
  input = @data.SOURCE
  prefix = "https://webscraper.io/"
  cssSelector = ".thumbnail .caption h4 a"
  attributeName = "href"
  RECURSIVE
  MODE:CSS
  => VAR @urls
ENDBLOCK

BLOCK:ConstantList
  value = []
  => CAP @prices
ENDBLOCK

FOREACH url IN urls
// Query the single URL
BLOCK:HttpRequest
  url = @url
  TYPE:STANDARD
  $""
  "application/x-www-form-urlencoded"
ENDBLOCK

BLOCK:Parse
  input = @data.SOURCE
  cssSelector = ".price"
  attributeName = "innerText"
  MODE:CSS
  => VAR @price
ENDBLOCK

BLOCK:AddToList
  list = @prices
  item = @price
ENDBLOCK
END

8 Likes

Here’s an example with a C# for loop, where we change the original list directly without creating a new one.

BLOCK:ConstantList
  value = ["one", "two", "three"]
  => VAR @list
ENDBLOCK

for (int i = 0; i < list.Count; i++)
{
  BLOCK:ToUppercase
    input = @list[i]
    => VAR @upper
  ENDBLOCK

  list[i] = upper;
}
2 Likes

Here’s another example which will search for the elements that are unique only to LIST1 (and not present in LIST2) and add them to a new list called UNIQUE

BLOCK:ConstantList
  value = []
  => VAR @UNIQUE
ENDBLOCK

BLOCK:ConstantList
  value = ["3", "5", "6"]
  => VAR @LIST1
ENDBLOCK

BLOCK:ConstantList
  value = ["1", "2", "3"]
  => VAR @LIST2
ENDBLOCK

FOREACH ITEM IN LIST1
IF LISTKEY @LIST2 DoesNotContain @ITEM
BLOCK:AddToList
  list = @UNIQUE
  item = @ITEM
ENDBLOCK
END
END
3 Likes