Skip to content

Commit d76900c

Browse files
authored
Merge pull request #2 from scrape-do/change-request-library
Change request library
2 parents aae7ed7 + 7e39f38 commit d76900c

File tree

7 files changed

+430
-178
lines changed

7 files changed

+430
-178
lines changed

.env

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,2 @@
1+
# Only the TOKEN element is required for the test.
2+
TOKEN=<your-api-token>

README.md

Lines changed: 73 additions & 26 deletions
Original file line numberDiff line numberDiff line change
@@ -1,24 +1,37 @@
1-
# @scrape-do/client
2-
3-
#### Scrape.do's official http client for node.js
1+
<p align="center">
2+
<img width="100" height="100" src="https://avatars.githubusercontent.com/u/67231321?s=200&v=4">
3+
<h3 align="center">Scrape Do Node Client</h3>
4+
<p align="center">Get unblocked while scraping the web - we bypass anti-bots and rotate proxies while you only pay for successful requests.</p>
5+
6+
<p align="center">
7+
<img src="https://img.shields.io/npm/v/@scrape-do/client/" />
8+
<img src="https://github.com/scrape-do/node-client/actions/workflows/build-test.yml/badge.svg?branch=main" />
9+
<img src="https://img.shields.io/github/issues/scrape-do/node-client" alt="Issues" />
10+
<img src="https://img.shields.io/github/license/scrape-do/node-client" alt="License" />
11+
</p>
12+
</p>
413

514
## How to install?
615

716
```bash
8-
> npm install @scrape-do/client
9-
# or get it from github
10-
> npm install git://git@github.com/scrape-do/node-client
17+
npm i @scrape-do/client
1118
```
19+
or install with github
1220

13-
## How to build from scratch
21+
```bash
22+
npm install git://git@github.com/scrape-do/node-client
23+
```
1424

15-
#### If you want to contribute to the library or include your own customisations, you can recompile the library in this way.
25+
## How Do I Import the Library?
1626

17-
```bash
18-
> git clone https://github.com/scrape-do/node-client
19-
> npm i
20-
# build with
21-
> npm build
27+
```js
28+
// CommonJS
29+
const { ScrapeDo } = require("@scrape-do/client");
30+
```
31+
32+
```typescript
33+
// Module - TypeScript
34+
import { ScrapeDo } from '@scrape-do/client'
2235
```
2336

2437
## Example Usages
@@ -28,7 +41,9 @@
2841
The super parameter enables the use of a residential proxy for the request. When this parameter is set to true, the request will be routed through a residential IP address. This means that the IP address will typically appear as if it belongs to a mobile network provider, adding an additional layer of anonymity and making the request look more like regular web traffic.
2942

3043
```typescript
31-
const client = new ScrapeDo("example_token");
44+
const { ScrapeDo } = require("@scrape-do/client");
45+
46+
const client = new ScrapeDo("your_api_token");
3247
const response = await client.sendRequest("GET", {
3348
url: "https://httpbin.co/anything",
3449
super: true,
@@ -42,7 +57,9 @@ console.log(response);
4257
The geoCode parameter allows you to specify the geographic location from which the request should appear to originate. By setting a specific country code, such as "us" for the United States, the request will be routed through an IP address from that region. This is especially useful for scraping websites that serve region-specific content or pricing, allowing you to access data as if you were browsing from that location.
4358

4459
```typescript
45-
const client = new ScrapeDo("example_token");
60+
const { ScrapeDo } = require("@scrape-do/client");
61+
62+
const client = new ScrapeDo("your_api_token");
4663
const response = await client.sendRequest("GET", {
4764
url: "https://httpbin.co/anything",
4865
geoCode: "us",
@@ -56,7 +73,9 @@ console.log(response);
5673
The regionalGeoCode parameter allows you to target requests from a broader geographic region, rather than a specific country. By specifying a regional code such as "europe" or "asia", your request will be routed through an IP address from that particular region. This is useful for scraping content that may be region-restricted, or for accessing region-specific data without the need to specify individual country codes.
5774

5875
```typescript
59-
const client = new ScrapeDo("example_token");
76+
const { ScrapeDo } = require("@scrape-do/client");
77+
78+
const client = new ScrapeDo("your_api_token");
6079
const response = await client.sendRequest("GET", {
6180
url: "https://httpbin.co/anything",
6281
regionalGeoCode: "europe",
@@ -79,7 +98,9 @@ Key points to note:
7998
- Sessions only for successful requests: A session will only be created if the initial request is successful.
8099

81100
```typescript
82-
const client = new ScrapeDo("example_token");
101+
const { ScrapeDo } = require("@scrape-do/client");
102+
103+
const client = new ScrapeDo("your_api_token");
83104
const response = await client.sendRequest("GET", {
84105
url: "https://httpbin.co/anything",
85106
sessionId: "1234",
@@ -93,7 +114,9 @@ console.log(response);
93114
The customHeaders option gives you full control over all headers sent to the target website. When you use customHeaders, the headers you provide will completely replace the default ones. This feature is useful when you need to define specific headers like User-Agent, Accept, Cookies, and more, ensuring that only your specified headers are sent with the request.
94115

95116
```typescript
96-
const client = new ScrapeDo("example_token");
117+
const { ScrapeDo } = require("@scrape-do/client");
118+
119+
const client = new ScrapeDo("your_api_token");
97120
const response = await client.sendRequest("GET", {
98121
url: "https://httpbin.co/anything",
99122
customHeaders: {
@@ -111,7 +134,9 @@ extraHeaders is used when you want to add one or more headers specifically requi
111134
The following example returns the response of how you requested from httpbin.co. You should see the ‘Key’ header in the header section of the response.
112135

113136
```typescript
114-
const client = new ScrapeDo("example_token");
137+
const { ScrapeDo } = require("@scrape-do/client");
138+
139+
const client = new ScrapeDo("your_api_token");
115140
const response = await client.sendRequest("GET", {
116141
url: "https://httpbin.co/anything",
117142
extraHeaders: {
@@ -127,7 +152,9 @@ console.log(response);
127152
The forwardHeaders option is ideal when you want to forward your custom headers directly to the target website without any additional headers being generated or modified by the service. This approach makes the request appear as if it is being made directly from your end, preserving the original header structure.
128153

129154
```typescript
130-
const client = new ScrapeDo("example_token");
155+
const { ScrapeDo } = require("@scrape-do/client");
156+
157+
const client = new ScrapeDo("your_api_token");
131158
const response = await client.sendRequest("GET", {
132159
url: "https://httpbin.co/anything",
133160
forwardHeaders: {
@@ -143,7 +170,9 @@ console.log(response);
143170
The render parameter allows for the execution of JavaScript during the request, enabling full browser-like rendering. When this parameter is set to true, the service will render the target webpage as if it were being loaded in a real browser, executing all JavaScript, loading dynamic content, and handling client-side interactions. This approach is particularly useful for scraping websites that rely heavily on JavaScript to display their content, providing a more accurate and “humanized” view of the page.
144171

145172
```typescript
146-
const client = new ScrapeDo("example_token");
173+
const { ScrapeDo } = require("@scrape-do/client");
174+
175+
const client = new ScrapeDo("your_api_token");
147176
const response = await client.sendRequest("GET", {
148177
url: "https://httpbin.co/anything",
149178
render: true,
@@ -168,7 +197,9 @@ Key information retrieved:
168197
> For security reasons, you can send up to 10 requests per minute to this endpoint. If you exceed this rate, you will receive a 429 Too Many Requests error.
169198
170199
```typescript
171-
const client = new ScrapeDo("example_token");
200+
const { ScrapeDo } = require("@scrape-do/client");
201+
202+
const client = new ScrapeDo("your_api_token");
172203
const stats = await client.statistics();
173204

174205
console.log(stats);
@@ -178,13 +209,18 @@ console.log(stats);
178209

179210
In this example, multiple parameters are combined to showcase advanced scraping capabilities. By using a combination of render, super, geoCode, and playWithBrowser, you can perform complex scraping tasks that require JavaScript execution, residential proxies, geographical targeting, and interactive browser actions:
180211

181-
- [render: true](https://scrape.do/documentation/#js-render?utm_source=github&utm_medium=node-client): Enables JavaScript execution to fully render the webpage, allowing for the scraping of dynamic content that relies on client-side scripting.
182-
- [super: true](https://scrape.do/documentation/#super-residential--mobile?utm_source=github&utm_medium=node-client): Utilizes a residential proxy, which makes the request appear as if it is coming from a typical user on a mobile network, providing enhanced anonymity and avoiding blocks from anti-scraping measures.
212+
> [!WARNING]
213+
> The browser created with this endpoint can be detected. It can be used for simple tasks such as waiting for the page to load, interacting with the page in your scraping tasks.
214+
215+
- [render](https://scrape.do/documentation/#js-render?utm_source=github&utm_medium=node-client): Enables JavaScript execution to fully render the webpage, allowing for the scraping of dynamic content that relies on client-side scripting.
216+
- [super](https://scrape.do/documentation/#super-residential--mobile?utm_source=github&utm_medium=node-client): Utilizes a residential proxy, which makes the request appear as if it is coming from a typical user on a mobile network, providing enhanced anonymity and avoiding blocks from anti-scraping measures.
183217
- [geoCode](https://scrape.do/documentation/#geo-targeting?utm_source=github&utm_medium=node-client): "us": Targets a specific geographic location for the request, in this case, the United States. This is useful for scraping content that varies by region, such as localized prices or region-specific data.
184218
- [playWithBrowser](https://scrape.do/documentation/#play-with-browser?utm_source=github&utm_medium=node-client): Provides the ability to interact with the browser while rendering the page. For example, you can wait for specific elements to load or perform actions like clicking buttons. In this case, it waits for the <body> element to ensure the page is fully loaded before proceeding.
185219

186220
```typescript
187-
const client = new ScrapeDo("example_token");
221+
const { ScrapeDo } = require("@scrape-do/client");
222+
223+
const client = new ScrapeDo("your_api_token");
188224
const response = await client.sendRequest("GET", {
189225
url: "https://example.com",
190226
render: true,
@@ -201,6 +237,17 @@ const response = await client.sendRequest("GET", {
201237
console.log(response);
202238
```
203239

240+
## How to build from scratch
241+
If you want to contribute to the library or include your own customisations, you can recompile the library in this way.
242+
243+
```bash
244+
git clone https://github.com/scrape-do/node-client
245+
npm i
246+
# build with
247+
npm build
248+
```
249+
250+
204251
## Official links
205252

206253
- [Scrape.do](https://scrape.do?utm_source=github&utm_medium=node-client)
@@ -215,4 +262,4 @@ console.log(response);
215262

216263
## Disclaimer
217264

218-
#### Any damages arising from the use of the library or service or any other legal situation cannot be associated with the scrape.do legal entity and team. The responsibility lies entirely with the user.
265+
#### Any damages arising from the use of the library or service or any other legal situation cannot be associated with the scrape.do legal entity and team. The responsibility lies entirely with the user.

0 commit comments

Comments
 (0)