You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
The assembly jar file includes also the [WebGraph](https://webgraph.di.unimi.it/) and [LAW](https://law.di.unimi.it/software.php) packages required to process the webgraphs and compute [PageRank](https://en.wikipedia.org/wiki/PageRank) or [Harmonic Centrality](https://en.wikipedia.org/wiki/Centrality#Harmonic_centrality).
14
+
The assembly jar file includes also the [WebGraph](https://webgraph.di.unimi.it/) and [LAW](https://law.di.unimi.it/software.php) packages required to process the web graphs and compute [PageRank](https://en.wikipedia.org/wiki/PageRank) or [Harmonic Centrality](https://en.wikipedia.org/wiki/Centrality#Harmonic_centrality).
15
15
16
16
17
17
### Javadocs
@@ -26,7 +26,7 @@ Run `mvn spotless:check` and `mvn spotless:apply`, see the [Spotless Maven guide
26
26
27
27
## Memory and Disk Requirements
28
28
29
-
Note that the webgraphs are usually multiple Gigabytes in size and require for processing
29
+
Note that the web graphs are usually multiple Gigabytes in size and require for processing
30
30
- a sufficient Java heap size ([Java option](https://docs.oracle.com/en/java/javase/21/docs/specs/man/java.html#extra-options-for-java)`-Xmx`)
31
31
- enough disk space to store the graphs and temporary data.
32
32
@@ -49,7 +49,7 @@ To analyze the graph structure and calculate rankings you may further process th
49
49
50
50
A couple of scripts may help you to run the WebGraph tools to build and process the graphs are provided in [src/script/webgraph_ranking/](src/script/webgraph_ranking/). They're also used to prepare the Common Crawl web graph releases.
51
51
52
-
To process a webgraph and rank the nodes, you should first adapt the configuration to your graph and hardware setup:
52
+
To process a web graph and rank the nodes, you should first adapt the configuration to your graph and hardware setup:
53
53
```
54
54
vi ./src/script/webgraph_ranking/webgraph_config.sh
55
55
```
@@ -59,12 +59,12 @@ After running
59
59
```
60
60
the `output_dir/` should contain all generated files, eg. `graph_name.graph` and `graph_name-ranks.txt.gz`.
61
61
62
-
The shell script is easily adapted to your needs. Please refer to the [LAW dataset tutorial](https://law.di.unimi.it/tutorial.php), the [API docs of LAW](https://law.di.unimi.it/software/law-docs/index.html) and [webgraph](https://webgraph.di.unimi.it/docs/) for further information.
62
+
The shell script is easily adapted to your needs. Please refer to the [LAW dataset tutorial](https://law.di.unimi.it/tutorial.php), the [API docs of LAW](https://law.di.unimi.it/software/law-docs/index.html) and [WebGraph](https://webgraph.di.unimi.it/docs/) for further information.
63
63
64
64
65
-
## Exploring Webgraph Data Sets
65
+
## Exploring Web Graph Data Sets
66
66
67
-
The Common Crawl webgraph data sets are announced on the [Common Crawl web site](https://commoncrawl.org/tag/webgraph/).
67
+
The Common Crawl web graph data sets are announced on the [Common Crawl web site](https://commoncrawl.org/tag/webgraph/).
68
68
69
69
For instructions how to explore the web graphs using the JShell please see the tutorial [Interactive Graph Exploration](./graph-exploration-README.md).
Copy file name to clipboardExpand all lines: graph-exploration-README.md
+7-7Lines changed: 7 additions & 7 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -1,6 +1,6 @@
1
1
# Interactive Graph Exploration
2
2
3
-
A tutorial how to interactively explore the Common Crawl webgraphs – or other graphs using the webgraph format – using the [JShell](https://docs.oracle.com/en/java/javase/21/jshell/index.html) and the [GraphExplorer](src/main/java/org/commoncrawl/webgraph/explore/GraphExplorer.java) class.
3
+
A tutorial how to interactively explore the Common Crawl web graphs – or other graphs using the [WebGraph](https://webgraph.di.unimi.it/) format – using the [JShell](https://docs.oracle.com/en/java/javase/21/jshell/index.html) and the [GraphExplorer](src/main/java/org/commoncrawl/webgraph/explore/GraphExplorer.java) class.
4
4
5
5
6
6
## Quick Start
@@ -117,13 +117,13 @@ A tutorial how to interactively explore the Common Crawl webgraphs – or other
117
117
## Using the Java Classes
118
118
119
119
The Java classes "GraphExplorer" and "Graph" bundle a set of methods which help exploring the graphs:
120
-
- load the webgraph, its transpose and the vertex map
120
+
- load the web graph, its transpose and the vertex map
121
121
- access the vertices and their successors or predecessors
122
122
- utilities to import or export a list of vertices or counts from or into a file
123
123
124
124
The methods are bundled in the classes of the Java package `org.commoncrawl.webgraph.explore`. To get an overview over all provided methods, inspect the source code or see the section [Javadocs](README.md#javadocs) in the main README for how to read the Javadocs. Here only few examples are presented.
125
125
126
-
We start again with launching the JShell and loading a webgraph:
126
+
We start again with launching the JShell and loading a web graph:
One important note: Common Crawl's webgraphs list the host or domain names in [reverse domain name notation](https://en.wikipedia.org/wiki/Reverse_domain_name_notation). The vertex lists are sorted by the reversed names in lexicographic order and then numbered continuously. This gives a close-to-perfect compression of the webgraphs itself. Most of the arcs are close in terms of locality because subdomains or sites of the same region (by country-code top-level domain) are listed in one continous block. Cf. the paper [The WebGraph Framework I: Compression Techniques](https://vigna.di.unimi.it/ftp/papers/WebGraphI.pdf) by Paolo Boldi and Sebastiano Vigna.
157
+
One important note: Common Crawl's web graphs list the host or domain names in [reverse domain name notation](https://en.wikipedia.org/wiki/Reverse_domain_name_notation). The vertex lists are sorted by the reversed names in lexicographic order and then numbered continuously. This gives a close-to-perfect compression of the web graphs itself. Most of the arcs are close in terms of locality because subdomains or sites of the same region (by country-code top-level domain) are listed in one continous block. Cf. the paper [The WebGraph Framework I: Compression Techniques](https://vigna.di.unimi.it/ftp/papers/WebGraphI.pdf) by Paolo Boldi and Sebastiano Vigna.
158
158
159
159
Now, let's look how many other domains are linked from Wikipedia?
Another note: Common Crawl's webgraphs are based on sample crawls of the web. Same as the crawls, also the webgraphs are not complete and the Wikipedia may in reality link to far more domains. But 2 million linked domains is already not a small sample.
166
+
Another note: Common Crawl's web graphs are based on sample crawls of the web. Same as the crawls, also the web graphs are not complete and the Wikipedia may in reality link to far more domains. But 2 million linked domains is already not a small sample.
167
167
168
168
The Graph class also gives you access to the successors of a vertex, as array or stream of integers, but also as stream of strings (vertex labels):
169
169
@@ -215,7 +215,7 @@ abogado.super
215
215
ac.789bet
216
216
```
217
217
218
-
Technically, webgraphs only store successor lists. But the Graph class holds also two graphs: the "original" one and its transpose. In the transposed graph "successors" are "predecessors", and "outdegree" means "indegree". Some methods on a deeper level take one of the two webgraphs as argument, here it makes a difference whether you pass `g.graph` or `g.graphT`, here to a method which translates vertex IDs to labels and extracts the top-level domain:
218
+
Technically, web graphs only store successor lists. But the Graph class holds also two graphs: the "original" one and its transpose. In the transposed graph "successors" are "predecessors", and "outdegree" means "indegree". Some methods on a deeper level take one of the two web graphs as argument, here it makes a difference whether you pass `g.graph` or `g.graphT`, here to a method which translates vertex IDs to labels and extracts the top-level domain:
0 commit comments