Refactor code for improved readability and maintainability.

README.md

@@ -22,25 +22,61 @@ The goal of this project can be achieved with the following milestones:
 4. **Providing an interface**: Design a UI for API consumers and producers to initiate a search looking for APIs. Initially, the search can be done using metadata—the info object of the [OpenAPI document](https://spec.openapis.org/oas/latest.html#info-object).
 5. **Updating dataset**: Regularly update the crawl results and re-index them for better search results.
 
-## Info about GSoC’23
 
-> **Note**: This project idea is shortlisted for [Google Summer Of Code 2023](https://blog.postman.com/join-postman-at-google-summer-of-code-2023/). Find the initial conversation [here](https://github.com/postman-open-technologies/gsoc-2023/issues/7).
+# Running the Server
+
+> Fork and/or clone the OpenAPI Web Search repo and change directory into it:
+
+```js
+
+git clone https://github.com/<username>/openapi-web-search.git
+cd openapi-web-search/src/server
+
+```
+
+> Install dependencies via yarn: 
+
+```js
+
+yarn install
+
+```
+
+> Start local server:
+
+```js
+
+yarn run dev
+
+```
+
+> After launching the local server, we can use Postman to begin sending http requests to the specified endpoints. I've included a postman collection in root of the project to get you started:
+
+
+> Run the following endpoints in the specified order after configuring Postman with the collection above:
+
+```js
+
+1. http://localhost:1337/api/v1/run/crawler?latest=true
+2. http://localhost:1337/api/v1/process/index-files?skip=0&limit=20&sort=aes
+3. http://localhost:1337/api/v1/indexing
+4. http://localhost:1337/api/v1/search?q=<query>
+
+```
+
+> Explanation:
+
+1. The first endpoint will crawl the common-crawl website to get some files which include the paths to index files that are converted into the appropriate endpoints. 
+2. The second endpoint initiates the background process of downloading index files, processing them, and storing the results, which are validated openapi definitions, in mongodb. 
+3. Third endpoint begins indexing the previously gathered MongoDB results into Elasticsearch..
+4. The last endpoint is utilised to create a search query for optimum retrival.
+
+
 
-If you're an aspiring GSoC candidate, here's what you should know: 
 
-- Having said that the purpose of this project is the **discovery of APIs from lesser-known sources**, crawling is where you will spend a good chunk of time.
-- The proposal should expand on each milestone mentioned in the above section. We understand that completing all the milestones within the 12 weeks of the GSoC period may not be feasible. We can figure it out based on the timeline provided.
-- There is no restriction on the choice of language, framework, or tools for building the solution for Open API Web Search.
-- We really don’t believe in reinventing the wheel. Feel free to use an existing solution like [Common Crawl](https://commoncrawl.org/).
-- For any concerns, kindly reach out to [@vinitshahdeo](https://github.com/vinitshahdeo) or [@MikeRalphson](https://github.com/MikeRalphson).
 
-#### Qualifying task
 
-As mentioned in [`CONTRIBUTOR_GUIDANCE.md`](https://github.com/postman-open-technologies/gsoc-2023/blob/main/CONTRIBUTOR_GUIDANCE.md), please refer to **[#2](https://github.com/postman-open-technologies/openapi-web-search/issues/2)** for the qualifying task.
 
 
-## Contact
 
-If you have any questions or queries, please [create an issue](openapi-web-search) on this repo (with a prefix GSoC 2023), start a topic on [our community forums in the GSoC category](https://community.postman.com/c/open-technology/gsoc/42) or send an email to us at gsoc@postman.com.
 
-[![Twitter](https://img.shields.io/badge/Twitter-%40getpostman-orange?logo=twitter&logoColor=white)](https://twitter.com/getpostman) [![YouTube](https://img.shields.io/badge/YouTube-%40postman-orange?logo=youtube)](https://www.youtube.com/c/postman)

open api web search.postman_collection.json

@@ -0,0 +1,131 @@
+{
+	"info": {
+		"_postman_id": "a19eee01-9a5f-4228-8353-c7fcc3615531",
+		"name": "open api web search",
+		"schema": "https://schema.getpostman.com/json/collection/v2.1.0/collection.json",
+		"_exporter_id": "20917353"
+	},
+	"item": [
+		{
+			"name": "start crawling",
+			"request": {
+				"method": "POST",
+				"header": [],
+				"body": {
+					"mode": "raw",
+					"raw": "{\n    \"dataSource\": \"commonCrawl\"\n}",
+					"options": {
+						"raw": {
+							"language": "json"
+						}
+					}
+				},
+				"url": {
+					"raw": "http://localhost:1337/api/v1/run/crawler?latest=true",
+					"protocol": "http",
+					"host": [
+						"localhost"
+					],
+					"port": "1337",
+					"path": [
+						"api",
+						"v1",
+						"run",
+						"crawler"
+					],
+					"query": [
+						{
+							"key": "latest",
+							"value": "true"
+						}
+					]
+				}
+			},
+			"response": []
+		},
+		{
+			"name": "download-process-index-files",
+			"request": {
+				"method": "GET",
+				"header": [],
+				"url": {
+					"raw": "http://localhost:1337/api/v1/process/index-files?skip=0&limit=20&sort=aes",
+					"protocol": "http",
+					"host": [
+						"localhost"
+					],
+					"port": "1337",
+					"path": [
+						"api",
+						"v1",
+						"process",
+						"index-files"
+					],
+					"query": [
+						{
+							"key": "skip",
+							"value": "0"
+						},
+						{
+							"key": "limit",
+							"value": "20"
+						},
+						{
+							"key": "sort",
+							"value": "aes"
+						}
+					]
+				}
+			},
+			"response": []
+		},
+		{
+			"name": "searchController",
+			"request": {
+				"method": "GET",
+				"header": [],
+				"url": {
+					"raw": "http://localhost:1337/api/v1/search?q=austria",
+					"protocol": "http",
+					"host": [
+						"localhost"
+					],
+					"port": "1337",
+					"path": [
+						"api",
+						"v1",
+						"search"
+					],
+					"query": [
+						{
+							"key": "q",
+							"value": "austria"
+						}
+					]
+				}
+			},
+			"response": []
+		},
+		{
+			"name": "indexController",
+			"request": {
+				"method": "GET",
+				"header": [],
+				"url": {
+					"raw": "http://localhost:1337/api/v1/indexing",
+					"protocol": "http",
+					"host": [
+						"localhost"
+					],
+					"port": "1337",
+					"path": [
+						"api",
+						"v1",
+						"indexing"
+					]
+				}
+			},
+			"response": []
+		}
+	]
+}

src/server/.editorconfig

@@ -0,0 +1,31 @@
+################################################
+#   ╔═╗╔╦╗╦╔╦╗╔═╗╦═╗┌─┐┌─┐┌┐┌┌─┐┬┌─┐
+#   ║╣  ║║║ ║ ║ ║╠╦╝│  │ ││││├┤ ││ ┬
+#  o╚═╝═╩╝╩ ╩ ╚═╝╩╚═└─┘└─┘┘└┘└  ┴└─┘
+#
+# > Formatting conventions for your Sails app.
+#
+# This file (`.editorconfig`) exists to help
+# maintain consistent formatting throughout the
+# files in your Sails app.
+#
+# For the sake of convention, the Sails team's
+# preferred settings are included here out of the
+# box.  You can also change this file to fit your
+# team's preferences (for example, if all of the
+# developers on your team have a strong preference
+# for tabs over spaces),
+#
+# To review what each of these options mean, see:
+# http://editorconfig.org/
+#
+################################################
+root = true
+
+[*]
+indent_style = space
+indent_size = 2
+end_of_line = lf
+charset = utf-8
+trim_trailing_whitespace = true
+insert_final_newline = true

src/server/.eslintignore

@@ -0,0 +1 @@
+

src/server/.eslintrc

@@ -0,0 +1,85 @@
+{
+  //   ╔═╗╔═╗╦  ╦╔╗╔╔╦╗┬─┐┌─┐
+  //   ║╣ ╚═╗║  ║║║║ ║ ├┬┘│
+  //  o╚═╝╚═╝╩═╝╩╝╚╝ ╩ ┴└─└─┘
+  // A set of basic code conventions designed to encourage quality and consistency
+  // across your Sails app's code base.  These rules are checked against
+  // automatically any time you run `npm test`.
+  // 
+  // > Note: If you're using mocha, you'll want to add an extra override file to your
+  // > `test/` folder so that eslint will tolerate mocha-specific globals like `before`
+  // > and `describe`.
+  // Designed for ESLint v4.
+  // - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
+  // For more information about any of the rules below, check out the relevant
+  // reference page on eslint.org.  For example, to get details on "no-sequences",
+  // you would visit `http://eslint.org/docs/rules/no-sequences`.  If you're unsure
+  // or could use some advice, come by https://sailsjs.com/support.
+  // - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
+
+  "env": {
+    "node": true
+  },
+
+  "parserOptions": {
+    "ecmaVersion": 2018
+  },
+
+  "globals": {
+    // If "no-undef" is enabled below, be sure to list all global variables that
+    // are used in this app's backend code (including the globalIds of models):
+    // - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
+    "Promise": true,
+    "sails": true,
+    "_": true
+    // …and any others (e.g. `"Organization": true`)
+    // - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
+  },
+
+  "rules": {
+    "block-scoped-var":             ["error"],
+    "callback-return":              ["error", ["done", "proceed", "next", "onwards", "callback", "cb"]],
+    "camelcase":                    ["warn", {"properties":"always"}],
+    "comma-style":                  ["warn", "last"],
+    "curly":                        ["warn"],
+    "eqeqeq":                       ["error", "always"],
+    "eol-last":                     ["warn"],
+    "handle-callback-err":          ["error"],
+    "indent":                       ["warn", 2, {
+      "SwitchCase": 1,
+      "MemberExpression": "off",
+      "FunctionDeclaration": {"body":1, "parameters":"off"},
+      "FunctionExpression": {"body":1, "parameters":"off"},
+      "CallExpression": {"arguments":"off"},
+      "ArrayExpression": 1,
+      "ObjectExpression": 1,
+      "ignoredNodes": ["ConditionalExpression"]
+    }],
+    "linebreak-style":              ["error", "unix"],
+    "no-dupe-keys":                 ["error"],
+    "no-duplicate-case":            ["error"],
+    "no-extra-semi":                ["warn"],
+    "no-labels":                    ["error"],
+    "no-mixed-spaces-and-tabs":     [2, "smart-tabs"],
+    "no-redeclare":                 ["warn"],
+    "no-return-assign":             ["error", "always"],
+    "no-sequences":                 ["error"],
+    "no-trailing-spaces":           ["warn"],
+    "no-undef":                     ["off"],
+    // - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
+    // ^^Note: If this "no-undef" rule is enabled (set to `["error"]`), then all model globals
+    // (e.g. `"Organization": true`) should be included above under "globals".
+    // - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
+    "no-unexpected-multiline":      ["warn"],
+    "no-unreachable":               ["warn"],
+    "no-unused-vars":               ["warn", {"caughtErrors":"all", "caughtErrorsIgnorePattern": "^unused($|[A-Z].*$)", "argsIgnorePattern": "^unused($|[A-Z].*$)", "varsIgnorePattern": "^unused($|[A-Z].*$)" }],
+    "no-use-before-define":         ["error", {"functions":false}],
+    "one-var":                      ["warn", "never"],
+    "prefer-arrow-callback":        ["warn", {"allowNamedFunctions":true}],
+    "quotes":                       ["warn", "single", {"avoidEscape":false, "allowTemplateLiterals":true}],
+    "semi":                         ["warn", "always"],
+    "semi-spacing":                 ["warn", {"before":false, "after":true}],
+    "semi-style":                   ["warn", "last"]
+  }
+
+}