Cogpilot Datathon Challenge – AFWERX

A: Anyone interested in the intersection of physiology, cognition, and flying! All participants are welcome, except we are unable to enroll businesses with more than 500 employees into the Datathon at this time. The dataset will be freely available to anyone who consents to the Data Sharing Agreement on our website.

Q: Can I join the Datathon after the start?

A: You can enroll any time before Tuesday, 2 Nov 21, (1 week prior to the end of the Datathon).

Q: Is the CDO Cogpilot Datathon in-person or virtual?

A: This challenge will be hosted fully virtual.

Q: Are you able to reach the registration page from a government computer?

A: Yes, you can register for the event at https://afwerxchallenge.com/datathon

Q: Am I able to complete the challenge on my government computer?

A: Yes, the challenge will be hosted at https://cdo-datathon.circle.so/home which is accessible from a government computer

Q: When will the Datathon take place and what is the time commitment of participants?

A: The Datathon will begin on 28 September at 1100 EST, the event will be self-paced with weekly SME support sessions.

Q: What level of technical ability do I have to have to participate in this challenge?

A: Novices (for example, those that have completed a coding or data science course) are able to participate in this challenge; there are several simpler tasks like identify features of the performance metrics that may be more suitable for novices. Advanced participants and adventurous novices may choose to attempt additional machine learning tasks.

Q: What will participants do during this challenge?

A: Visualize multimodal physiological data, build models which most accurately use pilot physiology to predict performance on a flight task. Please see the Datasheet for more information.

Q: How do I access the dataset?

A: Dataset will be available for download on the Datathon website.

Q: What is SuperCloud and who will have access to it?

A: MIT Supercloud is a cloud-based high-performance computing environment. More information can be found here https://supercloud.mit.edu/.

Specific to this Datathon, for those participants who express interest in obtaining Supercloud accounts, the Datathon team will send the requests to the Supercloud team, which will review the requests.

Q: For Military and Academia users, how do I access the DataSet on Supercloud?

A: Information on how to access the dataset will be posted on the Datathon/Circle to approved participants who are granted Supercloud accounts.

Q: If my company wants to participate as a team on the Challenge, does each member register individually? Can we choose our own groups?

A: Please have all members register individually to receive access to the Circle Community.

On the Scavenger Hunt portion of the Opening Day event on Tuesday, Sept 28th, we will have a team-signup forum where you all can officially sign up as a team.

Q: How do we learn more about the challenge data, including the data collection set-up, recording, modalities, and preprocessing?

A: Please check out the reference folder that comes with the challenge dataset download. It contains a wealth of information for this data challenge.

Q: Where can we find the associated values for physiological data? ie; how much is "300 somethings?"

A: The first column in every file is time_s (time in seconds), which corresponds to the computer clock time.

Q: Is this trying to predict the difficulty of a task at the end of a recorded set of inputs? Or the difficulty of a task at the moment based on current/previous observations?

A: There are two predictive modeling tasks for you to complete in this data challenge. Given the physiological data measured from a subject during a run:

Predict the difficulty level of the run, specifically to distinguish between Level 1 and Level 4
Predict the performance error of the run

Q: Are there any recommended resources for gaining a baseline familiarity with the pertinent disciplines which will be critical to this challenge? My personal fallback would be https://digitalu.udemy.com/ but there may be a lighter lift time/attention-wise to get an initial foothold.

A: Consider coming to our office hours every Tuesday, during which we will be discussing relevant knowledge and skills. (For background, the DigitalU link in the question is free Udemy access for USAF/USSF personnel. It has excellent courses in many fields, but also some great ones on coding and ML. Follow the link in the question to self-register.)

Q1: At what point in a flight is the trainee "spawned"; How does the point of spawning relate to the start of the data set to be provided? Q2: Can you provide a process flow with approximate times that covers what the trainees go through before data recording starts until after data recording stops?

A: Trainees are first outfitted with wearable sensors. They perform a few practice ILS runs on the easiest difficulty to get familiarized with the scenario. These practice runs, usually lasting 5-10 mins, are recorded, though not included in the public data package. Trainees then get ready for the 12 runs of the experiment. The experimenter first loads the scenario (i.e., one of the 4 levels of difficulty). The scenario loads in “paused” mode so there’s no aircraft movement. The experimenter then starts the data recording. Every flight begins with the aircraft in the same position and orientation. The only differences between levels of difficulty are changes in weather (visibility and wind). The start of the data recording happens slightly before the start of the flight. In practice, the simulation is paused, the data logging begins, the simulation is then unpaused, and the participant begins the run. One could plot the aircraft airspeed to see that the initial few points have zero velocity, but it jumps to ~100 knots once the simulation is unpaused. After the trainee lands the aircraft (or crashes), the experimenter asks the trainee to take their hands off the controls and then ends the recording of the data. At this point, the next trial is loaded.

Q: Is there a preference for programming language or technology stack?

A: The starter kit has been written in Python. However, you are welcome to use any language that you are comfortable with. The dataset can be pulled out of the kit and loaded into Visual Studio for your convenience. Also, we provide the data in .mat format for participants who’d like to use Matlab.

Q: You are not implementing data imputation standards?

A: You can impute and document your method, or develop algorithms to be robust to missing data as may be possible in a real environment. If you have more specific questions on imputation, please reach out to us for a focused discussion during one of the office hours.

Q: Are there requirements for computing environment? Do we have to use Supercloud?

A: You're free to perform ML modeling in any computing environment you'd like, Supercloud is just one option we wanted to make available for those who are eligible to use it.

A: ML development is not limited to Supercloud. Participants are free to use any platform of their choosing.

Q: So we can use CDO VAULT?

A: Datathon participants who have access to the CDO VAULT can use the VAULT for code development. IL-2 data on an IL-4 platform is still IL-2 data. Any AF civilian or military personnel can request an account on VAULT using the automated process (afdatalab.af.mil). Chris Gillie is working to get the starter material and data into VAULT.

Algorithms developed using open source SW or COTS BI Tools are IL-2. Participants just need to be careful not to add operational knowledge that is > IL-2 to their code. It’s also possible that insight gained from developed solutions could be above IL-2. These last two points always hold regardless of where anyone codes.

Participants just need to be careful not to add IL-4+ operational knowledge to their code. It’s also possible that insight gained from developed solutions could be above IL-2.

We can probably get a shared folder built on VAULT Databricks so users can come and run some of the pre-built notebooks. We can also load the datasets into the Data Lake somewhere. We can work that in the AFAC channel on Teams.

Q: What problem are we trying to solve? How to improve pilot training on the ground? How to predict the in-air performance of the trained pilot before they fly?

A: The Challenge is focused on developing a model to turn physiological measures into accurate assessments of cognitive state/cognitive workload (at a high level). The implications of such a model would apply to ground training, in-flight training, and really any job or task that's cognitively demanding.

Q: I'm not familiar with Anaconda.

A: Please check out the Anaconda with Jupyter Notebook Installation Guide provided in the data package. You can find instructions for downloading and installing Anaconda on Windows and other operating systems here:

Download: https://www.anaconda.com/products/individual

Install: https://docs.anaconda.com/anaconda/install/index.html

Q: Is this data labeled?

A: The Development Set data comes with a PerfMetrics.csv file that provides, for each run, both the difficulty level label (“Difficulty”) for Challenge Task #1 and the total flight performance error (“Cumulative_Error”) for Challenge Task #2. However, these truth labels are withheld for the Evaluation Set data, and they will be used to adjudicate submissions.

Q: So we are predicting the difficulty level of the task being undertaken. Our predictors are the time series performance data of the pilot as well as the background of the pilot?

A: There are two tasks:

Predict the task difficulty level (classification task)
Predict the performance score per trial (regression task)

For both, the input features should include physiological data.

Q: Is there one final "Score" per run that we are trying to predict? Or are there multiple scores per run with the given tasks such as takeoff, landing, etc?

A: We have one continuous time-series score, as well as the aggregated final score for each run. There's only one task (approach to land) performed 12 times in a row for each subject. Your final model should predict one final score per run.

A: yes, there's a single score value per trial. We have back up slides showing the details of how that's computed, but it's also detailed in the datasheet.pdf

Q: Will the end product be to create a training suite for all pilots that are optimally designed, or to monitor the pilot going through training to keep him at an optimal state?

A: the first column in every file is time_s (time in seconds), which corresponds to the computer clock time. The time is in unix epoch timestamps (in seconds)

Q: Is there a targeted deployment platform for a final solution? (compute limitations, etc)

A: Not for this Datathon. At this stage, we’d like to use any and all resources to produce the best solution. We can downselect from there based on the end platform.

Q: Is there a limit to team size?

A: No, but usually a team of 4-8 is recommended.

Q: How long is each run? 10mn? does the difficulty level equally spread out throughout the run or is there certain period where the task is more difficult than other parts?

A: Runs are ~7-10 mins, depending on the speed of flight, errors in movement, etc. But, sometimes, a novice subject may crash, so the run may be shorter. Though, we think the crashes are valuable too, to get extreme situations that may evoke large physiological responses of attempted correction.
A: While technically the difficulty is about the same for the duration of the run (for a given difficulty level), it would be interesting to see if physiology changes more as subjects get closer to the runway and subjects need to ensure a landing.

A: each run is approximately 7 minutes (19 mile flight to land flown at ~115 knots indicated airspeed plus there's variable wind depending on difficulty). Each of the 12 runs is one of four difficulty options.

Q: Can we safely assume all data have the same starting timestamp? so we don't have to align them?

A: The data and time are aligned and the linking time is in the first column. That time in the first column is universal across streams (and subjects!). However, the first timestamp between two different files (e.g., Subject001_EDAfile vs Subject001_EMGfile) may not be the same. One stream might start slightly ahead of the other. Nevertheless, the timestamps will be the truth to figure out that slight difference.

Q: For eye tracking, is there an association of x y axis to what instrument the pilot would be looking at?

A: Excellent question! NO! :) Because of head movement and rotation, the position of the instrument panel may change in the VR space relative to the person's eye tracking gaze. We hope to have that capability for future challenges.

Q: Is a "Trial" the same as a "Run" or are there multiple trials per run?

A: Trial == Run. In the data files, we use "runs". If we use the term “trial”, it’s mean to refer to a run.

Q:Error is based on Center of Gravity of the sim Aircraft, correct? [does not consider orientation of the A/C in err calc]

A: Correct, error is independent of aircraft orientation. Error is based on 1) horizontal localizer deviation, 2) vertical glide slope deviation, and 3) speed deviation from 115 knots. Details are in the datasheet.

Q: Is there any data that indicates users previous experience with using a VR headset?

A: Yes, subjects report their VR and Real flight experience. That data is in the "total flight hours" pdf. We're really trying to add more capability than just using flight data, which instructor pilots already have. We want to determine in this challenge whether there's information in physiology (cognitive state) that can be used to guide pilot training, more than just the flight data. Adding some of that, otherwise hidden physiological information, we think would add data for instructors to guide decisions about training.

Q: What is included in the “Rest” periods

A: There is a 5-min Rest period before the 12 runs and after. During the rest periods, trainees just sit quietly with all the sensors on and logging data. We think the Rest periods would enable better baselining of physiology of an individual.

Q: Has this data been baselined against AutoML? AutoGluon?

A: No AutoML, etc.. Really just as raw of data and we can get. Doing pre-processing may impede someone's AI pipeline and impose assumptions on the data, which we didn't want to do a priori.
In the data challenge package, we have provided preliminary results using SVM for the difficulty level. We'd love to see what the community comes up with and specifically, the performance against the Eval Set

Q: Is this all the data we are going to get or will we get more data throughout the project? If 21 subjects are all we have access to, is that going to be enough data to make predictive model for all students, or is our task to predict the results of these 21 subjects?

A: Great question! We're still actively collecting data. We'd love to make more data available in time. Please stay tuned in to Circle for more info on that.

Q: Should we hold out more subjects for evaluation of our code?

A: No, for now, 6 Subjects held out for participant code evaluation, so training on the development set of 15 subjects (x 12 runs per subject = 180 runs) should be just fine. Though, you may want to do some train/test splits on the 15 subjects data you have access to.

Q: How can I participate in the data collection?

A: If you'd like to be a subject, please email cogpilot@mit.edu (data collections occur at MIT Campus in Cambridge, MA)

Q: Can participants use CDO VAULT for computing?

A: Datathon participants who have access to the CDO VAULT can use the VAULT for code development. VAULT is IL-4 certified. IL-2 data on an IL-4 platform is still IL-2 data. Any AF civilian or military personnel can request an account on VAULT using the automated process (afdatalab.af.mil). Mr. Chris Gillie is working to get the starter material and data into VAULT.

Q: If we have our own Valve Index can we run this simulator ourselves? Is this simulator the same as the publicly available one on Steam?

https://store.steampowered.com/app/269950/XPlane_11/

A: We are still working on sharing the simulation files. The main limitation is that the simulation was performed using a T-6A Texan II aircraft model developed by FliteAdvantage. This aircraft model is custom built based on the Air Force T-6 and is not available freely. So use of the simulation would require the purchase of a license. We are working on recreating the scenarios using freely available aircrafts, but that would limit the parallelism of using the ILS instruments. Therefore, we are still in discussion about how to best share the simulation.

Q: Are we able to run our models against the 6 subjects in the eval data to see how we do? Or do we need to further break down the 15 in train/validate sets?

A: Good question. The Eval Set is indeed in the data/ folder. Though, you'll note that the difficulty levels are missing. So you can run your own code on the Eval set data and submit your results. The 15 in the Dev set can be split into train/test based on your algorithms. But just note that some train/test splits may help increase generalizability for the Eval set.

Q: What if I’m not an expert in Ai/Machine Learning?

A: For those of you who are worried about your level of data skills, the teams you’ve been assigned to should have a good mix of people who can get you across the finish line — and our weekly office hours will help (plus the community will be helpful for asking questions!)

Q: Are you fitting z score separately on the train and validation sets? If so this looks like data snooping.

A: In Cells 47 & 49, the sub functions compute the z score within the train/test separately. We'd like this code to be just a starting point to help people get familiar with the data and thinking about what the challenge outcomes could look like. But please run with it and adapt the code as you see fit. We'd love to see what people come up with and novel ideas for predicting performance from physiology.

Q: Is there any particular desire towards broader concerns like explainability, parsimony, speed, etc.?

A: Explainability would be a great one. E.g., Why did the model do so well? What about the physiology is indicative of "cog overload" that impacts performance? How robust is that feature across individuals? Etc. If we better understand why your model did well, we can build that knowledge into the training pipeline.

Q: Is it still possible to request a SuperCloud account? Or is it just for people who requested one during registration?

A: Supercloud registration is closed, except members of Massachusetts Green High-Performance Computing Center (MGHPCC) Consortium institutions (MIT, Harvard, University of Massachusetts, Northeastern University, Boston University) are always eligible for access by emailing supercloud@mit.edu.

Q: If we have our own Valve Index can we run this scenario in our own sim (I know we won’t have the sensory data streaming)? Is this simulator the same as the publicly available one on Steam?