You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
#### The super parameter enables the use of a residential proxy for the request. When this parameter is set to true, the request will be routed through a residential IP address. This means that the IP address will typically appear as if it belongs to a mobile network provider, adding an additional layer of anonymity and making the request look more like regular web traffic.
28
+
The super parameter enables the use of a residential proxy for the request. When this parameter is set to true, the request will be routed through a residential IP address. This means that the IP address will typically appear as if it belongs to a mobile network provider, adding an additional layer of anonymity and making the request look more like regular web traffic.
#### The customHeaders option gives you full control over all headers sent to the target website. When you use customHeaders, the headers you provide will completely replace the default ones. This feature is useful when you need to define specific headers like User-Agent, Accept, Cookies, and more, ensuring that only your specified headers are sent with the request.
42
+
The geoCode parameter allows you to specify the geographic location from which the request should appear to originate. By setting a specific country code, such as "us" for the United States, the request will be routed through an IP address from that region. This is especially useful for scraping websites that serve region-specific content or pricing, allowing you to access data as if you were browsing from that location.
The regionalGeoCode parameter allows you to target requests from a broader geographic region, rather than a specific country. By specifying a regional code such as "europe" or "asia", your request will be routed through an IP address from that particular region. This is useful for scraping content that may be region-restricted, or for accessing region-specific data without the need to specify individual country codes.
The sessionId parameter enables you to use the same proxy address for multiple requests over a certain period. By passing a unique integer value (e.g., sessionId=1234), you can maintain a persistent session with a single proxy. This is useful when you need consistent IP continuity for scraping operations or interacting with websites that track user sessions.
71
+
72
+
Key points to note:
73
+
74
+
- Session ID Range: The sessionId must be an integer between 0 and 1,000,000.
75
+
- Session Timeout: If no request is made using the sessionId for 5 minutes, the session will automatically expire.
76
+
- Session Failure: If a request made with a session ID fails, a new proxy will be assigned, and the session will reset.
77
+
- Geo Targeting Compatibility: When used with Geo Targeting or Regional Geo Targeting, the session will be locked to the specified country or region.
78
+
- No session required for new proxies: If you want to use a different proxy for each request, you don’t need to set a sessionId.
79
+
- Sessions only for successful requests: A session will only be created if the initial request is successful.
The customHeaders option gives you full control over all headers sent to the target website. When you use customHeaders, the headers you provide will completely replace the default ones. This feature is useful when you need to define specific headers like User-Agent, Accept, Cookies, and more, ensuring that only your specified headers are sent with the request.
#### extraHeaders is used when you want to add one or more headers specifically required by the target website, without altering the core headers automatically generated by the service. This is useful for passing additional information while maintaining the integrity of the existing request headers.
109
+
extraHeaders is used when you want to add one or more headers specifically required by the target website, without altering the core headers automatically generated by the service. This is useful for passing additional information while maintaining the integrity of the existing request headers.
59
110
60
-
#### The following example returns the response of how you requested from httpbin.co. You should see the ‘Key’ header in the header section of the response.
111
+
The following example returns the response of how you requested from httpbin.co. You should see the ‘Key’ header in the header section of the response.
#### The forwardHeaders option is ideal when you want to forward your custom headers directly to the target website without any additional headers being generated or modified by the service. This approach makes the request appear as if it is being made directly from your end, preserving the original header structure.
127
+
The forwardHeaders option is ideal when you want to forward your custom headers directly to the target website without any additional headers being generated or modified by the service. This approach makes the request appear as if it is being made directly from your end, preserving the original header structure.
#### The render parameter allows for the execution of JavaScript during the request, enabling full browser-like rendering. When this parameter is set to true, the service will render the target webpage as if it were being loaded in a real browser, executing all JavaScript, loading dynamic content, and handling client-side interactions. This approach is particularly useful for scraping websites that rely heavily on JavaScript to display their content, providing a more accurate and “humanized” view of the page.
143
+
The render parameter allows for the execution of JavaScript during the request, enabling full browser-like rendering. When this parameter is set to true, the service will render the target webpage as if it were being loaded in a real browser, executing all JavaScript, loading dynamic content, and handling client-side interactions. This approach is particularly useful for scraping websites that rely heavily on JavaScript to display their content, providing a more accurate and “humanized” view of the page.
The statistics() method allows you to retrieve real-time usage statistics for your subscription. This API call returns details such as your current subscription status, the number of concurrent requests allowed, the total and remaining requests per month, and how many concurrent requests are still available.
158
+
159
+
Key information retrieved:
160
+
161
+
- IsActive: Indicates whether your subscription is active.
162
+
- ConcurrentRequest: The total number of concurrent requests your subscription supports.
163
+
- MaxMonthlyRequest: The maximum number of requests allowed per month.
164
+
- RemainingConcurrentRequest: The number of concurrent requests you have left at the current time.
165
+
- RemainingMonthlyRequest: The remaining number of requests you can send this month.
166
+
167
+
> [!WARNING]
168
+
> For security reasons, you can send up to 10 requests per minute to this endpoint. If you exceed this rate, you will receive a 429 Too Many Requests error.
169
+
170
+
```typescript
171
+
const client =newScrapeDo("example_token");
172
+
const stats =awaitclient.statistics();
173
+
174
+
console.log(stats);
102
175
```
103
176
104
-
### final bonus example (render, super, geoCode, playWithBrowser)
177
+
### Final bonus example (render, super, geoCode, playWithBrowser)
105
178
106
-
#### In this example, multiple parameters are combined to showcase advanced scraping capabilities. By using a combination of render, super, geoCode, and playWithBrowser, you can perform complex scraping tasks that require JavaScript execution, residential proxies, geographical targeting, and interactive browser actions:
179
+
In this example, multiple parameters are combined to showcase advanced scraping capabilities. By using a combination of render, super, geoCode, and playWithBrowser, you can perform complex scraping tasks that require JavaScript execution, residential proxies, geographical targeting, and interactive browser actions:
107
180
108
181
-[render: true](https://scrape.do/documentation/#js-render?utm_source=github&utm_medium=node-client): Enables JavaScript execution to fully render the webpage, allowing for the scraping of dynamic content that relies on client-side scripting.
109
182
-[super: true](https://scrape.do/documentation/#super-residential--mobile?utm_source=github&utm_medium=node-client): Utilizes a residential proxy, which makes the request appear as if it is coming from a typical user on a mobile network, providing enhanced anonymity and avoiding blocks from anti-scraping measures.
### This project is licensed under the MIT License. See the [LICENSE](./LICENSE) file for more details.
214
+
####This project is licensed under the MIT License. See the [LICENSE](./LICENSE) file for more details.
142
215
143
216
## Disclaimer
144
217
145
-
### Any damages arising from the use of the library or service or any other legal situation cannot be associated with the scrape.do legal entity and team. The responsibility lies entirely with the user.
218
+
####Any damages arising from the use of the library or service or any other legal situation cannot be associated with the scrape.do legal entity and team. The responsibility lies entirely with the user.
0 commit comments