Skip to content

Data Collection & Processing

Jump to bottom
IBXCODECAT edited this page Jan 26, 2023 · 23 revisions

Data Collection

Data is collected using the Saber-Metrics application which can be deployed to android devices for scouters to use.

Pre-Loading Team Numbers

When scouting a match, our team assigns assigns 6 scouters (one scouter per each team on the field) to scout a specific team on the field. When we scouted our first regional matches in 2022, we noticed that our scouters were accidentally selecting the same teams as others resulting in conflicting data entries and data loss. To solve this issue, our backed team came up with the idea of pre-loading the team numbers into the application using the following solution.

Accessing TBA-APIv3

In order to access the team numbers that are in a specific match we need to scrape data from the TBA API. The version of the API you will want to be scraping from is TBA-APIv3 which is the most recent version at the time of writing this wiki. In order to scrape the API you will need to create a myTBA account with The Blue Alliance by connecting TBA to your Google account.

Generating an API Key

Once you have a myTBA account you will need to generate a 'read' API Key. This is how your script can authenticate and associate it'self with the API. We are generating a 'read' key because we are only scraping the API and not making any API calls. Once you have generated your API key make sure you keep it in a safe place and never upload it to version control.

Choosing a language

When fetching data from the TBA API, we are making a web request. The Python language allows us to make these web requests easily without the headaches that come with using the Node.JS framework combined with vanilla JavaScript or Typescript. Although Node.JS would be the most logical framework to choose when making web requests, we are only making a total of three simple fetch requests so the extra features included in Node.JS are not needed.

Safely storing your API key

Your API key is how the API can authenticate and associate your application with you. Treat it like a password!

  • NEVER share your API key with anyone who is not authorized to use it.
  • NEVER use your API Key where it can be seen publicly
  • NEVER upload your API key to version control
  • ALWAYS use a solution like .env or .gitignore to verify your API Key is never made publicly available.

IF YOUR API KEY HAS BEEN STOLEN OR SHARED MAKE SURE TO REGENERATE YOUR KEY ASAP | YOU (THE DEVELOPER) WILL ALWAYS BE HELD RESPONSIBLE FOR ACTIONS TAKEN USING YOUR API KEY EVEN IF IT WASN'T YOU WHO TOOK THOSE ACTIONS

In the case of Saber Robotics we store our key in a file named api-key.txt in the current working directory (CWD) which is strictly ignored by our .gitignore so it will never be committed to this repository. We read our API key from the CWD using the following code:

import os #Import the os module
api_key = '' #Create an empty variable for our api key

with open('api-key.txt') as filestream: #Load API_KEY from file
    api_key = filestream.readlines() #Set our api_key to the contents of the file

for key in api_key: #For each key in our api_key list... (there is only one)
    api_key = key #Set our api_key to the extracted key

Fetching data from the TBA API

baseURL = 'http://www.thebluealliance.com/api/v3/' #The API we will be scraping
header = {'X-TBA-Auth-Key':api_key} #Create our request header
def getTBA(url): return requests.get(baseURL + url, headers=header).json()

Data Processing

Temporary Data Storage

When you submit data in the Saber-Metrics app, it will be stored in a serializable Kotlin data object which will be converted into the JSON format and saved to the following directory on your Android device as a *.scout file:

/storage/emulated/0/Android/data/com.example.frc_scouting/files

The name of the file written will follow the pattern of teamNumber-matchNumber.scout, where teamNumber and matchNumber are the team and match number in the form submission. Using this naming schema, we can guarantee that there will be no duplicate data entries logged, and that corrected entries will automatically overwrite an existing *.scout file.

After competitions, we normally will manually transfer these files from our scouting devices to the computer with Microsoft SQL Server installed.

Building SQL Queries

The SQL Builder is responsible for parsing all *.scout files it can find within the current working directory and building a list of queries. Once the SQL Builder has finished building it's queries, it will write them to a file named sqlBuilderOutput.sql which you will manually have to run against the database. You can run the SQL Builder on Windows by running this Windows Command File.

Example Query Output:

INSERT INTO [Scouting 2022].[dbo].[MatchMaster1](Column0, Column1)VALUES(Data0, Data1) --Generated SQL Query

Note: Each *.scout file will generate it's own query.

For more information on how the SQL Builder works check out the comments in the SQL Builder python script!

Clone this wiki locally