No sites work and proxy doesnt work #33

authsoftware · 2021-06-09T18:55:33Z

I have tried to use proxies but none worked and it keeps using local to open the sites.
Also I cannot get anything to actually checkout, footlocker keeps saying product not found on both product id and link, nike wont add to bag, shopify keeps giving an error of

[6/9/2021, 9:52:39 PM] - Error: Evaluation failed: TypeError: Cannot read property 'id' of undefined
    at <anonymous>:3:72

on champs it is stuck on closing modal.
Supreme this is the error I receive

[6/9/2021, 10:50:33 PM] - Navigating to URL
[6/9/2021, 10:50:36 PM] - Error: Evaluation failed: TypeError: Cannot read property 'querySelector' of null
    at <anonymous>:9:36
    at ExecutionContext._evaluateInternal (C:\Users\dev\Desktop\SneakerBot-master\node_modules\puppeteer\lib\cjs\puppeteer\common\ExecutionContext.js:217:19)
    at processTicksAndRejections (node:internal/process/task_queues:96:5)
    at async ExecutionContext.evaluate (C:\Users\dev\Desktop\SneakerBot-master\node_modules\puppeteer\lib\cjs\puppeteer\common\ExecutionContext.js:106:16)
    at async Object.exports.guestCheckout (C:\Users\dev\Desktop\SneakerBot-master\sites\supremenewyork.js:229:7)
    at async C:\Users\dev\Desktop\SneakerBot-master\helpers\cluster.js:85:34

The text was updated successfully, but these errors were encountered:

samc621 · 2021-06-12T23:47:14Z

@authsoftware thanks for opening this ticket. Regarding the proxy issues, please see #29. And regarding the nike issues, see #28. I am looking into the other issues.

samc621 · 2021-06-13T01:38:03Z

Can you also provide some details to reproduce the issue on Footlocker? Preferably a Task object I can test with? I cannot reproduce it from my end.

I recently fixed an unrelated issue with the footsites here.

samc621 · 2021-06-13T01:42:28Z

I am also not having any issues with shopify or supreme. Please provide a Task object for testing.

authsoftware · 2021-06-13T17:00:44Z

Yes I will send you the tasks I have so you could test them on Foots, Shopify, Supreme

authsoftware · 2021-06-13T17:03:51Z

Supreme

{
            "id": 24,
            "site_id": 5,
            "url": "https://www.supremenewyork.com/shop/shirts/b29vjuoq3",
            "style_index": null,
            "size": null,
            "shipping_speed_index": null,
            "billing_address_id": 1,
            "shipping_address_id": 1,
            "notification_email_address": "[email protected]",
            "created_at": "2021-06-09T19:29:52.576Z",
            "updated_at": "2021-06-09T19:29:52.576Z",
            "is_deleted": false,
            "auto_solve_captchas": true,
            "product_code": null,
            "site_name": "supremenewyork"
        },

Shopify

{
            "id": 19,
            "site_id": 3,
            "url": "https://kith.com/collections/kith-hoodies/products/kh2657-603",
            "style_index": null,
            "size": null,
            "shipping_speed_index": null,
            "billing_address_id": 1,
            "shipping_address_id": 1,
            "notification_email_address": "[email protected]",
            "created_at": "2021-06-09T18:49:04.645Z",
            "updated_at": "2021-06-09T18:49:04.645Z",
            "is_deleted": false,
            "auto_solve_captchas": true,
            "product_code": null,
            "site_name": "shopify"
        },

Footlocker

{
            "id": 16,
            "site_id": 2,
            "url": "https://footlocker.com",
            "style_index": null,
            "size": null,
            "shipping_speed_index": null,
            "billing_address_id": 1,
            "shipping_address_id": 1,
            "notification_email_address": "[email protected]",
            "created_at": "2021-06-09T18:33:10.523Z",
            "updated_at": "2021-06-09T18:33:10.523Z",
            "is_deleted": false,
            "auto_solve_captchas": true,
            "product_code": "W2288111",
            "site_name": "footsites"
        },

samc621 · 2021-06-15T12:54:43Z

@authsoftware using a style_index and size of null will not work for these products. For products without more than one style, you can use 0 for the style_index. I would apply the same for the shipping_speed_index. But you will still need to apply a size.

The reason I made these fields nullable is for site integrations or individual products where they are N/A e.g no shipping speed or style options on Supreme, or products without size options.

authsoftware · 2021-06-15T20:34:45Z

Okay thank you, I will test that and let you know

authsoftware · 2021-06-15T20:46:54Z

Okay so after fixing that and fixing the proxies issue, this is the error I have now

[6/15/2021, 11:45:23 PM] - Searching for product by product code
[6/15/2021, 11:45:23 PM] - Error: net::ERR_CONNECTION_RESET at https://www.champssports.com/search?query=EY2763
    at navigate (C:\Users\dev\Desktop\SneakerBot-master\node_modules\puppeteer\lib\cjs\puppeteer\common\FrameManager.js:115:23)
    at processTicksAndRejections (node:internal/process/task_queues:96:5)
    at async FrameManager.navigateFrame (C:\Users\dev\Desktop\SneakerBot-master\node_modules\puppeteer\lib\cjs\puppeteer\common\FrameManager.js:90:21)
    at async Frame.goto (C:\Users\dev\Desktop\SneakerBot-master\node_modules\puppeteer\lib\cjs\puppeteer\common\FrameManager.js:416:16)
    at async Page.goto (C:\Users\dev\Desktop\SneakerBot-master\node_modules\puppeteer\lib\cjs\puppeteer\common\Page.js:821:16)
    at async searchByProductCode (C:\Users\dev\Desktop\SneakerBot-master\sites\footsites.js:276:5)
    at async Object.exports.guestCheckout (C:\Users\dev\Desktop\SneakerBot-master\sites\footsites.js:323:7)
    at async C:\Users\dev\Desktop\SneakerBot-master\helpers\cluster.js:94:34

task object

{
        "id": 25,
        "site_id": 2,
        "url": "https://www.champssports.com/",
        "style_index": 0,
        "size": "12",
        "shipping_speed_index": 0,
        "billing_address_id": 1,
        "shipping_address_id": 1,
        "notification_email_address": "[email protected]",
        "created_at": "2021-06-15T20:45:11.970Z",
        "updated_at": "2021-06-15T20:45:11.970Z",
        "is_deleted": false,
        "auto_solve_captchas": true,
        "product_code": "EY2763"
    }

samc621 · 2021-06-16T01:46:31Z

@authsoftware thanks for this, I will test with this Task object.

I have seen this error from puppeteer before, particularly when using headless mode. Some websites will reject the headless user agent so we need to manually set the user agent when the page is created. I will verify that this is the issue and then put up a quick fix for it.

samc621 · 2021-06-16T15:09:04Z

@authsoftware this looks to be the case, can you confirm that it is working for you in headful mode?

authsoftware · 2021-06-16T17:58:45Z

@samc621 I did enable headful mode by making headless false in cluster.js but i have an issue with almost any site whenever you go to the product page it keeps going back to the main page for example on champs it goes to the product page then redirects to the home page then this issue comes up

TimeoutError: waiting for selector `div#bluecoreActionScreen` failed: timeout 30000ms exceeded
    at new WaitTask (C:\Users\dev\Desktop\SneakerBot-master\node_modules\puppeteer\lib\cjs\puppeteer\common\DOMWorld.js:509:34)
    at DOMWorld.waitForSelectorInPage (C:\Users\dev\Desktop\SneakerBot-master\node_modules\puppeteer\lib\cjs\puppeteer\common\DOMWorld.js:420:26)
    at Object.internalHandler.waitFor (C:\Users\dev\Desktop\SneakerBot-master\node_modules\puppeteer\lib\cjs\puppeteer\common\QueryHandler.js:31:77)
    at DOMWorld.waitForSelector (C:\Users\dev\Desktop\SneakerBot-master\node_modules\puppeteer\lib\cjs\puppeteer\common\DOMWorld.js:313:29)
    at Frame.waitForSelector (C:\Users\dev\Desktop\SneakerBot-master\node_modules\puppeteer\lib\cjs\puppeteer\common\FrameManager.js:841:51)
    at Page.waitForSelector (C:\Users\dev\Desktop\SneakerBot-master\node_modules\puppeteer\lib\cjs\puppeteer\common\Page.js:1335:33)
    at closeModal (C:\Users\dev\Desktop\SneakerBot-master\sites\footsites.js:256:16)
    at Object.exports.guestCheckout (C:\Users\dev\Desktop\SneakerBot-master\sites\footsites.js:320:5)
    at C:\Users\dev\Desktop\SneakerBot-master\helpers\cluster.js:95:57

samc621 · 2021-06-17T13:01:18Z

@authsoftware this is happening for you with the sample Champs task above? It is working fine for me, at least in headful mode.

authsoftware · 2021-06-17T13:36:19Z

@samc621 Yes it is happening to me in all sites

samc621 · 2021-06-17T14:00:11Z

Based on the selector in that stack trace, it sounds like the modal which sometimes appears on the footsites is not appearing for you. Using timeout: 0 on that selector might help. But this promise is not awaited so its weird that it wouldn't at least attempt to ATC before this error occurred. I might need to see a video of what you are seeing.

I understand that you're seeing this behaviour across sites, but I will need stack traces from the others, then. This specific error is with a selector on the footsites.

labboy0276 · 2021-07-05T18:04:41Z

I am seeing this error as well (on headless) @samc621:

TimeoutError: waiting for selector `div#bluecoreActionScreen` failed: timeout 30000ms exceeded

Here is the task JSON I used:

{
        "site_id": 2,
        "url": "https://www.champssports.com/product/converse-run-star-hike-platform-high-top-womens/170913C.html",
        "style_index": 0,
        "size": "10.0",
        "shipping_speed_index": 0,
        "billing_address_id": 1,
        "shipping_address_id": 1,
        "notification_email_address": "[email protected]"
}

edit: I am seeing this as a the body response via:

      let bodyHTML = await page.evaluate(() =>  document.documentElement.outerHTML);
      console.log(bodyHTML);

<html><head>
    <title>429 Too Many Requests</title>
  </head>
  <body>
    <h1>Error 429 Too Many Requests</h1>
    <p>Too Many Requests</p>
    <h3>Guru Mediation:</h3>
    <p>Details: cache-ewr18145-EWR 1625508467 2084869219</p>
    <hr>
    <p>Varnish cache server</p>

Is there a way to set the user agent dynamically with somehow? Maybe that is still the issue?

labboy0276 · 2021-07-05T20:46:06Z

After playing around this a little more, I was able to get past the error with the following:

Made sure my proxies were being used. (I had burned through the list)
Added the following package to my setup: user-agents && puppeteer-extra-plugin-anonymize-ua
I then add this to the helpers/cluster.js file:

const Ua = require('puppeteer-extra-plugin-anonymize-ua');
const UserAgent = require("user-agents");

puppeteer.use(Ua());
const userAgent = new UserAgent();

    const puppeteerOptions = {
      headless: true,
      defaultViewport: null,
      args: [
        '--no-sandbox',
        '--disable-setuid-sandbox',
        '--disable-web-security',
        '--disable-features=IsolateOrigins,site-per-process',
        '--user-agent=' + userAgent
      ]
    };

This got the timeout errors to go away. (note I edited this a couple times)

samc621 · 2021-07-06T01:42:52Z

@labboy0276 thanks for the updates here! There's a lot we can do around randomizing browser fingerprints, and starting with the UA is a great idea. If this has been tested and confirmed to help address a problem you are facing, please feel free to open a Pull Request. I am happy to try it on my end and then merge it as long as it works.

samc621 · 2021-07-06T01:47:46Z

@labboy0276 as for this error:

TimeoutError: waiting for selector `div#bluecoreActionScreen` failed: timeout 30000ms exceeded

Can you try the suggestion I mentioned above, which is to add timeout: 0 to the waitFor args?

So line 256 would look like:

await page.waitForSelector(modalSelector, { visible: true, timeout: 0 });

What I think is going on here is that this modal only shows on some of the footsites, but not Champs. Using the default timeout from Puppeteer, it will timeout after 30s, which will break your task midway. But with the timeout of 0, it will basically never time out, but listen asynchronously (non-blocking) and close the modal if it does ever appear.

I suggest trying this and see if it works. If it does, seems like a quick fix we can get in there (feel free to open a PR).

labboy0276 · 2021-07-06T13:21:31Z

@samc621 I put in that PR. I didn't have issues with that timeout after I fixed everything I mentioned. If it does happen again, I can try your suggestion.

samc621 · 2021-07-06T13:28:33Z

@labboy0276 thanks for that! So just to confirm, this fix is just for the footsites issue, right? Or does this address the nike issue as well?

labboy0276 · 2021-07-06T13:36:59Z

@samc621 the user agent issue fixes all sites for peoples who use headless.

cimthog · 2021-07-08T10:21:17Z

I keep getting Error 429, too many requests, whenever there's a drop.. for footsites.

Anyone knows a way round this?

samc621 · 2021-07-09T14:59:21Z

@labboy0276 thank you! I am testing that PR now.

13ROY · 2021-09-17T21:09:25Z

@samc621 and @labboy0276 , based on the discussions and the closed PR #39, I am guessing we can close this issue?

samc621 · 2021-09-18T01:37:56Z

@13ROY yes, I am closing this issue now.

samc621 closed this as completed Sep 18, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

No sites work and proxy doesnt work #33

No sites work and proxy doesnt work #33

authsoftware commented Jun 9, 2021 •

edited

Loading

samc621 commented Jun 12, 2021 •

edited

Loading

samc621 commented Jun 13, 2021

samc621 commented Jun 13, 2021

authsoftware commented Jun 13, 2021

authsoftware commented Jun 13, 2021

samc621 commented Jun 15, 2021

authsoftware commented Jun 15, 2021

authsoftware commented Jun 15, 2021

samc621 commented Jun 16, 2021

samc621 commented Jun 16, 2021

authsoftware commented Jun 16, 2021 •

edited

Loading

samc621 commented Jun 17, 2021

authsoftware commented Jun 17, 2021

samc621 commented Jun 17, 2021

labboy0276 commented Jul 5, 2021 •

edited

Loading

labboy0276 commented Jul 5, 2021 •

edited

Loading

samc621 commented Jul 6, 2021

samc621 commented Jul 6, 2021

labboy0276 commented Jul 6, 2021

samc621 commented Jul 6, 2021

labboy0276 commented Jul 6, 2021

cimthog commented Jul 8, 2021

samc621 commented Jul 9, 2021

13ROY commented Sep 17, 2021

samc621 commented Sep 18, 2021

No sites work and proxy doesnt work #33

No sites work and proxy doesnt work #33

Comments

authsoftware commented Jun 9, 2021 • edited Loading

samc621 commented Jun 12, 2021 • edited Loading

samc621 commented Jun 13, 2021

samc621 commented Jun 13, 2021

authsoftware commented Jun 13, 2021

authsoftware commented Jun 13, 2021

samc621 commented Jun 15, 2021

authsoftware commented Jun 15, 2021

authsoftware commented Jun 15, 2021

samc621 commented Jun 16, 2021

samc621 commented Jun 16, 2021

authsoftware commented Jun 16, 2021 • edited Loading

samc621 commented Jun 17, 2021

authsoftware commented Jun 17, 2021

samc621 commented Jun 17, 2021

labboy0276 commented Jul 5, 2021 • edited Loading

labboy0276 commented Jul 5, 2021 • edited Loading

samc621 commented Jul 6, 2021

samc621 commented Jul 6, 2021

labboy0276 commented Jul 6, 2021

samc621 commented Jul 6, 2021

labboy0276 commented Jul 6, 2021

cimthog commented Jul 8, 2021

samc621 commented Jul 9, 2021

13ROY commented Sep 17, 2021

samc621 commented Sep 18, 2021

authsoftware commented Jun 9, 2021 •

edited

Loading

samc621 commented Jun 12, 2021 •

edited

Loading

authsoftware commented Jun 16, 2021 •

edited

Loading

labboy0276 commented Jul 5, 2021 •

edited

Loading

labboy0276 commented Jul 5, 2021 •

edited

Loading