You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardExpand all lines: README.md
+28-26Lines changed: 28 additions & 26 deletions
Original file line number
Diff line number
Diff line change
@@ -23,71 +23,71 @@ npm build
23
23
24
24
## Example Usages
25
25
26
-
> ### extraHeaders
26
+
### super (residental proxy)
27
27
28
-
#### extraHeaders is used when you want to add one or more headers specifically required by the target website, without altering the core headers automatically generated by the service. This is useful for passing additional information while maintaining the integrity of the existing request headers.
29
-
30
-
#### The following example returns the response of how you requested from httpbin.co. You should see the ‘Key’ header in the header section of the response.
28
+
#### The super parameter enables the use of a residential proxy for the request. When this parameter is set to true, the request will be routed through a residential IP address. This means that the IP address will typically appear as if it belongs to a mobile network provider, adding an additional layer of anonymity and making the request look more like regular web traffic.
31
29
32
30
```typescript
33
31
const client =newScrapeDo("example_token");
34
32
const response =awaitclient.sendRequest("GET", {
35
33
url: "https://httpbin.co/anything",
36
-
extraHeaders: {
37
-
Key: "Value",
38
-
},
34
+
super: true,
39
35
});
40
36
41
37
console.log(response.data);
42
38
```
43
39
44
-
> ### forwardHeaders
40
+
### customHeaders
45
41
46
-
#### The forwardHeaders option is ideal when you want to forward your custom headers directly to the target website without any additional headers being generated or modified by the service. This approach makes the request appear as if it is being made directly from your end, preserving the original header structure.
42
+
#### The customHeaders option gives you full control over all headers sent to the target website. When you use customHeaders, the headers you provide will completely replace the default ones. This feature is useful when you need to define specific headers like User-Agent, Accept, Cookies, and more, ensuring that only your specified headers are sent with the request.
47
43
48
44
```typescript
49
45
const client =newScrapeDo("example_token");
50
46
const response =awaitclient.sendRequest("GET", {
51
47
url: "https://httpbin.co/anything",
52
-
forwardHeaders: {
48
+
customHeaders: {
53
49
Key: "Value",
54
50
},
55
51
});
56
52
57
53
console.log(response.data);
58
54
```
59
55
60
-
> ### customHeaders
56
+
### extraHeaders
61
57
62
-
#### The customHeaders option gives you full control over all headers sent to the target website. When you use customHeaders, the headers you provide will completely replace the default ones. This feature is useful when you need to define specific headers like User-Agent, Accept, Cookies, and more, ensuring that only your specified headers are sent with the request.
58
+
#### extraHeaders is used when you want to add one or more headers specifically required by the target website, without altering the core headers automatically generated by the service. This is useful for passing additional information while maintaining the integrity of the existing request headers.
59
+
60
+
#### The following example returns the response of how you requested from httpbin.co. You should see the ‘Key’ header in the header section of the response.
63
61
64
62
```typescript
65
63
const client =newScrapeDo("example_token");
66
64
const response =awaitclient.sendRequest("GET", {
67
65
url: "https://httpbin.co/anything",
68
-
customHeaders: {
66
+
extraHeaders: {
69
67
Key: "Value",
70
68
},
71
69
});
72
70
73
71
console.log(response.data);
74
72
```
75
73
76
-
> ### super (residental proxy)
74
+
### forwardHeaders
77
75
78
-
#### The super parameter enables the use of a residential proxy for the request. When this parameter is set to true, the request will be routed through a residential IP address. This means that the IP address will typically appear as if it belongs to a mobile network provider, adding an additional layer of anonymity and making the request look more like regular web traffic.
76
+
#### The forwardHeaders option is ideal when you want to forward your custom headers directly to the target website without any additional headers being generated or modified by the service. This approach makes the request appear as if it is being made directly from your end, preserving the original header structure.
#### The render parameter allows for the execution of JavaScript during the request, enabling full browser-like rendering. When this parameter is set to true, the service will render the target webpage as if it were being loaded in a real browser, executing all JavaScript, loading dynamic content, and handling client-side interactions. This approach is particularly useful for scraping websites that rely heavily on JavaScript to display their content, providing a more accurate and “humanized” view of the page.
> ### final bonus example (render, super, geoCode, playWithBrowser)
104
+
### final bonus example (render, super, geoCode, playWithBrowser)
105
105
106
106
#### In this example, multiple parameters are combined to showcase advanced scraping capabilities. By using a combination of render, super, geoCode, and playWithBrowser, you can perform complex scraping tasks that require JavaScript execution, residential proxies, geographical targeting, and interactive browser actions:
107
107
108
-
- render: true: Enables JavaScript execution to fully render the webpage, allowing for the scraping of dynamic content that relies on client-side scripting.
109
-
- super: true: Utilizes a residential proxy, which makes the request appear as if it is coming from a typical user on a mobile network, providing enhanced anonymity and avoiding blocks from anti-scraping measures.
110
-
- geoCode: "us": Targets a specific geographic location for the request, in this case, the United States. This is useful for scraping content that varies by region, such as localized prices or region-specific data.
111
-
- playWithBrowser: Provides the ability to interact with the browser while rendering the page. For example, you can wait for specific elements to load or perform actions like clicking buttons. In this case, it waits for the <body> element to ensure the page is fully loaded before proceeding.
108
+
-[render: true](https://scrape.do/documentation/#js-render?utm_source=github&utm_medium=node-client): Enables JavaScript execution to fully render the webpage, allowing for the scraping of dynamic content that relies on client-side scripting.
109
+
-[super: true](https://scrape.do/documentation/#super-residential--mobile?utm_source=github&utm_medium=node-client): Utilizes a residential proxy, which makes the request appear as if it is coming from a typical user on a mobile network, providing enhanced anonymity and avoiding blocks from anti-scraping measures.
110
+
-[geoCode](https://scrape.do/documentation/#geo-targeting?utm_source=github&utm_medium=node-client): "us": Targets a specific geographic location for the request, in this case, the United States. This is useful for scraping content that varies by region, such as localized prices or region-specific data.
111
+
-[playWithBrowser](https://scrape.do/documentation/#play-with-browser?utm_source=github&utm_medium=node-client): Provides the ability to interact with the browser while rendering the page. For example, you can wait for specific elements to load or perform actions like clicking buttons. In this case, it waits for the <body> element to ensure the page is fully loaded before proceeding.
0 commit comments