At a Glance
The upper left chart is a translation of racetrack coordinates from the video game into latitude and longitude to depict the position on the track. One line (currently the blue one in the picture) is the real-time feed to depict a racer’s session as it happens. The other line on the chart shows a reference lap to illustrate historical data (up to and including the previous lap).
We also created a leaderboard. The bottom chart shows the telemetry demonstration. Similar to the track visualization, this chart shows both real-time and a reference lap.
F1 2018 by Codemasters. This game was built to export its telemetry data to increase the immersive experience of playing the game; companies in the e-sports ecosystem sell dashboard apps for phone or tablet to display the data, and some e-sports driving wheels display the information directly on the wheel.
OmniSci Open Source Edition which includes OmniSci Core. The demo includes custom visualizations, so we’re not using OmniSci Immerse.
StreamSets for visually architecting and implementing the data flows (we are using the open-source Docker container).
Apache Kafka for processing the feeds.
OmniSci F1 Demo (authored by Randy Zwitch) built with Python, pymapd, and Dash
We hosted everything on a Microsoft Azure ND24S instance (4x P40 GPUs) with 96GB of GPU RAM.
We also bought a GTultimate v2 Simulator Cockpit from Next Level Racing to… uh… enhance the immersive experience of the demo, and not because it’s super fun to have at the office. This is not required to make the demo work.
Collection and Ingestion
- Player Races a Lap
The game itself collects telemetry data from the racing simulation; we created the demo to collect it per session (lap around the track).
- Game Streams Data to StreamSets
The game streams data via UDP to an IP address and port combination set by the player.
StreamSets receives the data from the game and writes it to Apache Kafka. This process allows data buffering, so that OmniSci doesn’t have to process every message as it comes in, but instead can process (and re-process at a later date if desired) messages in larger batches.
- Read Raw UDP Packets from Kafka and Convert Them to JSON
StreamSets also reads data from the Apache Kafka topic and processes messages from their raw states (binary) to JSON. The processing is done with Groovy (a Java scripting language). We also do minor data enrichment before writing to a different Kafka topic.
- Read JSON from Kafka and Load to OmniSci
After that, data is inserted into OmniSci using the OmniSci JDBC driver. There are 8 packet types; each packet type gets split into a different processing pipeline. Each pipeline makes records from JSON, does appropriate data type conversion as necessary, then uses the OmniSci JDBC driver to load data into OmniSci Core.
Content and Size
There are 8 packet types, which get parsed into 9 tables. Cumulatively, there are over 100 distinct attributes between the tables. The telemetry and car status data is generated at 60 packets per second, or 60 hertz. Some of the data, such as engine RPM, track position, car damage, tire pressure, etc. are provided at this rate; other data like car setup at the start of the race and lap driving times are provided less frequently.
During continuous driving, the game streams 14MB of packets of data per minute in a binary (i.e. compressed) format. Once unpacked, a single packet contains an average of 20 records. Because the demo is loading in data in real-time, there’s no baseline to how large the dataset is.
Over several hundred sessions (individual races,) the largest 4 tables have 236MM records each; others have 10 million.