Divide read file line in chunks

[uxs0wgq9gk]

Hi i’m reading a 41523 lines file (read file line block) and i would like to divide it into predefined var/cap of maximum 1000 “chunks” i tried to create a code with jint and iron python, but i couldn’t do it. i want to create a kind of list of thing because i should make requests with index of each element and respective content and NO the TAKE command is not what i’m looking for as it doesn’t divide read file lines.

The request should look like this in the POST (I am reading a file that contains ip:port of public proxy) - (index of element)

Content-Disposition: form-data; name="ip_addr[]"


54.37.74.220:80-0

-----------------------------435902341004570564730231420768

Content-Disposition: form-data; name="ip_addr[]"


85.215.64.49:80-1

-----------------------------435902341004570564730231420768

Content-Disposition: form-data; name="ip_addr[]"


89.145.162.81:3128-2

-----------------------------435902341004570564730231420768

Content-Disposition: form-data; name="ip_addr[]"


114.129.2.82:8081-3

-----------------------------435902341004570564730231420768

Content-Disposition: form-data; name="ip_addr[]"


198.49.68.80:80-4

-----------------------------435902341004570564730231420768

Content-Disposition: form-data; name="ip_addr[]"


8.213.151.128:3128-5

-----------------------------435902341004570564730231420768

Content-Disposition: form-data; name="ip_addr[]"


47.178.24.220:80-6

-----------------------------435902341004570564730231420768

Content-Disposition: form-data; name="ip_addr[]"


77.83.246.25:80-7

-----------------------------435902341004570564730231420768

Content-Disposition: form-data; name="ip_addr[]"


43.134.68.153:3128-8

-----------------------------435902341004570564730231420768

Content-Disposition: form-data; name="ip_addr[]"


51.89.255.67:80-9

-----------------------------435902341004570564730231420768

Content-Disposition: form-data; name="ip_addr[]"


103.159.194.33:1111-10

-----------------------------435902341004570564730231420768

Content-Disposition: form-data; name="ip_addr[]"


45.92.177.60:8080-11

-----------------------------435902341004570564730231420768

Content-Disposition: form-data; name="ip_addr[]"


121.8.215.106:9797-12

do you know how to do this? there is a block that does this @Ruri how should I do for each “cap/var list created” make a request with its contents and index number?

[uxs0wgq9gk]

@Ruri in the web console I can see that it prints them but I can’t convert them as data.LogVariableAssignment(nameof(lines)); to use them as variables for requests so that I can then calculate the index of each variable for subsequent requests could you help me?

BLOCK:FileReadLines
  path = "UserData\\ProxyList.txt"
  => VAR @lines
ENDBLOCK
// Method to split a list into chunks
List<List<string>> SplitArray(List<string> array, int chunkSize)
{
    var chunks = new List<List<string>>();
    for (int i = 0; i < array.Count; i += chunkSize)
    {
        chunks.Add(array.GetRange(i, Math.Min(chunkSize, array.Count - i)));
    }
    return chunks;
}

// Main execution block
void Main()
{
    int chunkSize = 1000;

    // Split the lines into chunks
    var chunks = SplitArray(lines, chunkSize);

    // Create a list to store the names of the chunk variables
    List<string> chunkVariableNames = new List<string>();

    // Process each chunk
    for (int i = 0; i < chunks.Count; i++)
    {
        // Create a variable name for the chunk
        string chunkVariableName = $"chunk{i}";

        // Add the variable name to the list
        chunkVariableNames.Add(chunkVariableName);

        // Create a new variable for the chunk
        // Note: In C#, you can't dynamically create variables, so we'll use a dictionary instead
        // to store the chunk variables.
        // For demonstration purposes, we'll just print the chunk variable name and its contents.
        Console.WriteLine($"Creating variable {chunkVariableName} with {chunks[i].Count} elements:");
        Console.WriteLine(string.Join(", ", chunks[i]));

        // Define the chunk variable
        // This will be used in your program
        string chunkVariable = $"var {chunkVariableName} = [{string.Join(", ", chunks[i])}];";
        Console.WriteLine(chunkVariable);
    }

    // Create a "main variable" that contains the names of all the chunk variables
    string mainVariableName = "mainVariable";
    Console.WriteLine($"Creating main variable {mainVariableName} with the following chunk variable names:");
    Console.WriteLine(string.Join(", ", chunkVariableNames));

    // Define the main variable
    // This will be used in your program
    string mainVariable = $"var {mainVariableName} = [{string.Join(", ", chunkVariableNames)}];";
    Console.WriteLine(mainVariable);
}

// Call the main execution block
Main();

the console LOG

Creating variable chunk39 with 1000 elements:
45.153.165.170:999, 45.174.79.95:999, 45.189.117.199:999, 45.236.242.70:8080, 46.172.192.68:61378
var chunk39 = [45.153.165.170:999, 45.174.79.95:999, 45.189.117.199:999, 45.236.242.70:8080, 46.172.192.68:61378];
Creating variable chunk40 with 510 elements:
72.10.160.90:13495, 72.10.160.90:13927, 72.10.160.90:14297, 72.10.160.90:15867, 72.10.160.90:1631, 72.10.160.90:16537, 72.10.160.90:17223
var chunk40 = [72.10.160.90:13495, 72.10.160.90:13927, 72.10.160.90:14297, 72.10.160.90:15867, 72.10.160.90:1631, 72.10.160.90:16537, 72.10.160.90:17223];
Creating main variable mainVariable with the following chunk variable names:
chunk0, chunk1, chunk2, chunk3, chunk4, chunk5, chunk6, chunk7, chunk8, chunk9, chunk10, chunk11, chunk12, chunk13, chunk14, chunk15, chunk16, chunk17, chunk18, chunk19, chunk20, chunk21, chunk22, chunk23, chunk24, chunk25, chunk26, chunk27, chunk28, chunk29, chunk30, chunk31, chunk32, chunk33, chunk34, chunk35, chunk36, chunk37, chunk38, chunk39, chunk40
var mainVariable = [chunk0, chunk1, chunk2, chunk3, chunk4, chunk5, chunk6, chunk7, chunk8, chunk9, chunk10, chunk11, chunk12, chunk13, chunk14, chunk15, chunk16, chunk17, chunk18, chunk19, chunk20, chunk21, chunk22, chunk23, chunk24, chunk25, chunk26, chunk27, chunk28, chunk29, chunk30, chunk31, chunk32, chunk33, chunk34, chunk35, chunk36, chunk37, chunk38, chunk39, chunk40];

[Pure]

You are overcomplicating it.

Since version 0.1.15 you can define shared resources across all your bots. It’s in the config settings.

example:

Using that you can then use this simple command
TAKE <AMOUNT> FROM "WHAT YOU NAMED THE RESOURCE" => "NAME OF LIST"

After that you can do something like this to create a global List that holds your chunks and is indexed by the bot number so each bot has a unique chunk.

With this example each chunk is 1000 entries long. So for 40k lines you would use 40 bots.

[uxs0wgq9gk]

if I don’t know how many lines the file contains is there any way to make that 40000 as a value automatic by reading it from the file?

thanks this works, but how can I make the multipart request with all the contents of the variable list called Proxy set as ip:port-“Proxy variable element index” which means there must be maximum 1000 elements in the POST of the request?

I guess I have to integrate it in the loop for each but I don’t know what to do and how to do it