TelMe!

The one stop web-app to tell you all about mobile subscription plans in Singapore.

You may watch our video presentation or visit the live website.

Proposed level of achievement: Artemis

Motivation

It’s that time of the year again, your Telco subscription is up for renewal and you’re faced with a tough decision, to recontract or to change Telco? A quick google search for the best mobile recontract promotions ends up being a 1 hour search for the best deal possible. You finally decide to recontract with Singtel, but to your dismay, after going through all that hassle, you find out that M1 was offering way more mobile data capacity for lower prices.

In this day and age, information is key. However, in a bid to attract more customers, companies flood and overwhelm the market with ambiguous and irrelevant advertisements. This discourages many to do more research and choose the easiest but less worth it option.

The issue of how fast information can be acquired also comes into play. Sifting through multiple websites is extremely time consuming and users cannot look up promotions they did not even know exists. Hence, a single place where users can view and sort through all kinds of Telco subscription plans and promotions is needed.

Aim

We hope to be able to provide users with a single place where one can view and compare all the different types of Telco subscription plans and promotions. Here, they will be able to find the best available Telco subscription plan, tailor made, according to what they are looking for.

Features of our web-app

No.	Feature	Function
1	Web Scraper	To populate the database with products extracted from each individual telco site.
1.1	Starhub Scraper	Scrapes data from Starhub's web page.
1.2	Singtel Scraper	Scrapes data from Singtel’s web page.
1.3	M1 Scraper	Scrapes data from M1’s web page.
1.4	My Republic Scraper	Scrapes data from My Republic’s web page.
1.5	giga! Scraper	Scrapes data from giga!’s web page.
2	Web Application	Provides an interface for users to view Telco information in a clean and concise manner.
3	Home Page	Welcomes users and shows them popular telco subscription plans.
3.1	Featured	Users will be able to see what popular promotions are available. As of the end of Orbital, the features items are hard-coded in. Given time, this feature could be done through scraping.
3.2	Trending topics	Users will be able to view articles related to telco discussion and reviews that users have posted on our website. As of the end of Orbital, the features items are hard-coded in. Given time, this feature could be done through scraping.
4	Products Page	Lists down all the plans and gives users a brief overview of all the plans in our database. Filter is found here (Refer to 1.3)
4.1	Add to favorites button	A button where customers can add a product to their favorite products.
4.2	Filter	Users have the option of choosing and filtering to zoom in on subscription plans according to their interests.
4.2.1	Price Range	Users will be able to search for plans within a price range
4.2.2	Data Range	Users will be able to search for plan within a data range
4.2.3	Contract Length Range	Users will be able to search for plans with a specific contract length (e.g 1 year, 2 years, no contract, etc.)
4.3	Submitted Filters	Users will be able to see which filters they have submitted in one place.
4.3.1	Remove	Users can remove each applied filter either individually or stop filtering altogether.
4.4	Ordering	Users can choose between the available methods ordering how the products are displayed. They can be ordered in four different ways as of now newest, cheapest, most data, shortest contract.
4.5	Search bar	Users can search for any product from any page in the website.
4.5.1	Search in results	Users can search for any product on the products page after filtering or ordering.
4.6	More details button (Product details page)	The information provided on the product page will only touch briefly on the information of the plans. To see more details, users can click on a dropdown button on the product card to see more details.
4.7	Visit Site button	This button links back to the telco website so the user can purchase it.
5	Admin Page	Admins will be able to visit the admin page to make changes to the website i.e modify and delete data, users and reviews.
5.1	Modify/Add user data	Admins will be able to remove, edit, and add users and super users.
5.2	Modify/add product data	Admins will be able to remove, edit and add new products into the existing listing.
5.3	Modify/add review and comment data	Admins will be able to monitor the reviews and comments section, remove and flag out comments that do not adhere to the site’s code of conduct.
5.4	Import/Export Data	Through a django package, admins will be able to download any data on the site into many different file types with .csv and excel being one of them.
6	Users	Users will be able to sign in on our web page. With an account users will be able to leave comments and reviews. This will be implemented in milestone 2.
6.1	Sign Up Page	Users will be able to sign up on our web page. A prompt will be given to sign up if a user tries to make a review or comment without an account. After signing up, the user is brought back to the page he came from. This will be implemented in milestone 2.
6.2	Sign In Page	Similar to sign up, users will be prompted when trying to leave a review. After signing up, users will be brought back to the page they were trying to leave a review on.
6.3	Forgot password reset	In the case of a lost password, users will be able to enter their email to get a password reset email.
6.4	User profile page	Users will be able to update their account details here. Their email, password and profile picture can be changed. A list of all the user liked plans can be found here.
6.4.1	Personal Info	Users will be able to view their personal info here. This consists of their Displayed name and their email.
6.4.2	Change Password	Users will be able to change their password should they find the need to do so.
6.4.3	Logout	Users can logout from this page.
6.5	Favorite Products page	This is where the User is able to view the products he has favourited.
6.6	Recommended Products page	Users will be recommended similar products using machine learning models based on the user's browsing behaviour and displayed on the profile page as well.
7	Track	Users can choose to be notified via email, once there is a new plan or a plan that they may be interested in.
7.1	Email	Given an email, a user can be added to our mailing list and be informed via email if there is a new plan or a price drop. This will be implemented in milestone 3

User Stories

Priorities: High (must have) - ***, Medium (nice to have) - **, Low (unlikely to have) - *

Priority	As a...	I want to...	So that I can...
***	Customer looking to recontract my Telco subscription plan	Be able to know and compare the existing promotions in the market	Make a well-informed decision before purchase.
***	User	Save my preferences and receive tailored recommendations	Make better judgement and ease the process of selecting a new plan
***	User	Be able to focus and zoom in on only products/subscription plans I am interested in	Save precious time.
***	User	Select specific deals and be notified when a promotion for that deal comes out.	Save the hassle of coming back to the website for updates on the specific deal.
***	Admin	Be able to modify product listings on my site.	Identify false/fake promotions and remove them from the website.
***	Admin	Be able to moderate user accounts on my site.	Identify and flag out users who violate code of conduct, and remove them.
***	Admin	Be able to populate my database automatically	Save time and receive information in the shortest time possible.
**	User	Be able to view new plans at a glance.	Avoid scrolling through old plans to try to find the new ones.

Recommend feature

First we get a list of plans favorited by the requester user.

1. Then we obtain the name of the cluster the user belongs to. We do this through the User.objects.get(..).cluster_set field that references the user side of the many-to-many relationship we have with clusters. We also exclude the requester user from that list. This is not strictly needed because of what we are going to do next, but might reduce query time.

2. Then we use the previous list of names to get favorites for those users in the cluster, excluding those favorites referring to the plans we got in step 1. From the result, we get a list of plan IDs.

3. Finally, we use the previous list of IDs to retrieve all the recommended plans.

   Clustering

   We used K-means clustering as a machine learning model that made use of user similarity in order to provide better plan recommendations. When a user saves a particular plan into his favorites list, the backend triggers the machine learning model and allocates him to a cluster with users of similar profiles. The backend will then suggest favorited plans of these similar users to the current user. As such, our model will perform better with more user activity.

   We chose K-means clustering because It is a fast clustering algorithm that has parallel and scalable implementations (e.g. see Spark). And overall, it is very easy to understand what K-means does and how it works. A user cluster is just a group of users close to each other based on how they favorited plans.

Web Extraction

First, the admin must run the command heroku run python scrape.py

1. Then, the scrapers are called using their respective function calls (e.g add_starhub_products(), etc)

2. Scraping is done through the use of both bs4 and Selenium packages. Upon having their function called, the scraper will crawl through the website, extract data and store it in a dictionary.

3. Upon completion of all the scrapers, all of the data scraped is pipelined into add_products.py where we can add all the products and run validators on them.

4. A filter is run on the existing models in the database. If a product with the same title already exists, the product is not added. Otherwise, Product.objects.create() is used to make the model object and product.save() is used to save the object in the database.

5. Axios is then used to fetch the products from our API and these products are mapped to our product template in React to display all the items on the webpage.

   As of milestone 3, data was scraped from the websites of Starhub, M1, giga! and My Republic. Data from Singtel was scrapped as well but due issues related to hosting and the packages we used to scrape from Singtel’s website, we are unable to run Singtel’s scraper on the server. Hence, we had to scrape Singtel’s data locally, export it, then import it into the online website’s database.

Deployment to Production
To deploy our web application, we chose to deploy with Heroku because it provides an ecosystem of cloud services, which can be used to instantly extend our application with fully-managed services.

1. Server

   Integration with Heroku requires a Procfile file which is used to explicitly declare our application’s process types and entry points. It is located in the root of our repository. This Procfile requires Gunicorn, the production web server.

web: gunicorn myproject.wsgi

2. Database

   For our local database, the web application uses SQLite, but because Heroku provides its applications with a Postgresql database, a seperate database connection unique to production was created in settings.py

   import dj_database_url
   db_from_env = dj_database_url.config(conn_max_age=600)
   DATABASES['default'].update(db_from_env)
   
   

Testing

	Problem	Solution
Frontend	Range for filter includes either “Unlimited” or “No Contract”	When scraping, upon encountering unlimited, it is registered as “999” while No Contract is registered as “0”. Conditionals are then used in the frontend code to display “Unlimited” upon encountering “999” and “No contract” on “0” This way, the filter can work without any issues.
	No recommended or favorite products	Query database for any recommended or favorited products. If none, show a string stating that there are no products.
	Search function cancels out filters	Implement search in results
Backend	Account Registration	Email: \ The application uses an email Regex Validator to check if email keyed in by the user is valid. Password: The application checks if the user has keyed in a password of at least 8 characters. New Passwords: The application returns “Wrong Password” if the user has entered an old password wrongly.
	Machine Learning & Clustering	A test was carried out in the project video.
	Web-scraper	When scraping and creating models, if a plan exists, change price of existing model if price dropped, else, do nothing. If a field is missing, a default value is used.

Some Notable Libraries Used

Language	Library	Use
Python	django	Provides a functional backend for the project
	djangorestframework	Creates the link between the django-backend and the react-frontend through the creation of a django rest API
	Beautiful Soup 4 / Selenium	Aids in the extraction of data from websites to populate the project database
	scikit-learn	Provides K-means clustering for machine learning.
	scipy	Helps us easily build sparse matrices for data handling.
Node	react	Provides a functional frontend for the project
	react-router-dom	Enables the linking of multiple pages in react
	redux	Provides the linking between django api endpoints and data queries to react frontend.
	webpack	Aids in the deployment of the website
	material-ui	Provides the resources to style the website

Timeline

Milestone I	Milestone II	Milestone III
1. Create and format website 2. Display Telco plans from different Telcos. 3. Allow users to sort/filter information based on what they want (e.g more mobile data, handset model, etc.) 4. Allow users to like deals.	1. Create Sign-up and Log-in features 2. Implement Favorite Plans list. 3. Implement a Recommendation system using machine learning algorithms. 4. Populate the database with actual promotions that have been web-scraped.	1. Implement Tracking system. Allow users to be notified about price reduction in deals that they have previously indicated “Notify Me!” 2. Host website on Heroku.