[Tool] TSR File Analyzer and Replayer

[Mike Nike]

Project in progress. Ideas/wishes welcome..
Tool written in MS Excel 2007. It basicly takes the "sliders.exe -d 3" output.txt (fast fwd replay of tsr ingame won't create loss of frames!) and reduces the file to a file with "important" lines only.
10min output.txt=210MB (in 0,5*length(tsrvideo), reduced=> 20MB (in <40 sec/210MB), very reduced: not done yet. During the reducing it immediately tracks stats and displays them in excel. in the end you can push a button to instantly export the stats to an html file, if you dont activate that this happens automaticly.

The idea was to analyze stats from mainly punaball games.
Probably though this work out as a possibility to create videos without frame loss. Also possible later should be: to upload a "reduced output"-file for an online replayer.

Replayer
(excel version)
http://mikenike.mi.funpic.de/ts/output_replay.html

(latest blitz basic version)
http://mikenike.mi.funpic.de/ts/TSR-Replayer%20v1.2.jpg
http://mikenike.mi.funpic.de/ts/TSR-Replayer%20v1.1.rar
http://mikenike.mi.funpic.de/ts/TSR-Replayer.rar

old:
http://mikenike.mi.funpic.de/ts/ExcelTS ... alyzer.jpg

todo:
[done] angle of cars
[todo] carcolors (probably no correct colors and colortype possible!)
[partly done] replay functions
[todo] autoexport to html replayer (javascript, css, html..)

Analyzer
http://mikenike.mi.funpic.de/ts/output_stats.html

not much to explain here, but let's see:
the 5-3 leg from the last pb cup final Fama/Palo vs olio/dede

not yet done: auto recognize teams.
let's see.. fama drove the most "distance", olio was the driver who was too lazy to run/drive for his money. dede, pay him more!
Keys Pressed: L/R/U/D: looks like dede is working a lot with the braking-key, while olio doesn't like to do that so much. for the "little" distance, dede still accelerated a lot though.
Palo probably is circling too much "right" and olio is a bit left-orientated, it seems..
The ball was in 61% of the cases in dede/olios half, so it kinda looks the goal difference relation 5-3 (5/(5+3)=62.5%) is kinda matching.
Dede seemed to be slightly more defensive than olio and Palo slightly more defensive than Fama. Interesting: 235/4=58.75% the cars were at right side, but the ball 61%.
Messages and written characters: dede writes more in his messages than olio. missing stat: Chars/Msg. and dede write also more messages.
Pro team Fama/Palo did not talk even once. GG!
300 Collision Avoidances have taken place. I'm not sure if that's only ball/car, car/car or also ball/wall and car/wall and how many happen in a press-shot situation or when being semi-stuck..

todo:
time spent in: 16m rooms, corners

Download:
i don't feel like publishing something yet.
But if you cry loud enough, wanting a tsr file to get analyzed, pm me the link to it or send to my emailaddy..

Edit:

Lightmap (where driven) Example of a player (in this case: olio):
http://mikenike.mi.funpic.de/ts/output_ ... yer_1.html

[Tijny]

Ha, nice tool.

Interesting stats: Fama being the most active player was to be expected, and dede defending more than olio can probably be explained by the fact that he doesn't trust olio's defending so much.
What might also be interesting to see, in addition to the percentage the cars spend on each half, is the average position of every player, and how those stats would differ from each other. The expected result would be for Fama and Palo to be near the middle circle, and for dede and olio to be somewhere in the middle of their own half, but I wonder if that's really the case.

[dede]

Very nice stats Nike! Lovely job

[Mike Nike]

my plan was to create a lightmap out of the x/y position array of each driver. just creating a very reduced output file (3mb instead of 20mb now)..still reduceable by a lot if i create a binary file instead of a textfile..
also, some more stats..like who "collided" more, upload in a minute..

so, Palo and dede very "collisional"^^, but Fama plays like having no car insurance.

edit:
1st lightmap, in this case of olio
http://mikenike.mi.funpic.de/ts/output_ ... yer_1.html

[Whiplash]

I'm not too big fan of of Puna, but I always like your stats!

[Mike Nike]

allrighty, 1st replayer screenshot.
rewind in progress..
already working well: the fast fwd speed (integer 1 to unlimited)..

http://mikenike.mi.funpic.de/ts/output_replay.html

reset and continue works fine, quite enjoyable already.
well, except that the speed "1" (no frame loss) takes slightly more than 10 sec for real 10 sec.
still looking good though.

[dede]

Hmm, doesn't work here. JSON problem, hmmm..

[Mike Nike]

changed stats page to outputfiletext+"_stats.html", so new addy is:

http://mikenike.mi.funpic.de/ts/output_stats.html

json problem...maybe has to do with browser not being able to show a line break. i think in this case i reformat the stats, so there is no such linebreak in it anymore..

*k, done

litte update (for tijny):
PosXAv PosYAv PosXAv% PosYAv%

[Badeend]

very nice

[Jan]

This is very fascinating. I like. ^^

[Mike Nike]

the replay in excel doesnt seem to be running that well, so i wrote a replayer in blitzbasic.

after fixing a float/integer bug for Movements/Distance driven int output_stats, i also uploaded an example of the replayer, so you see a first running version.

unfortunately you might need some sort of windows OS and probably some sort of IE version installed to run the tool.

http://mikenike.mi.funpic.de/ts/TSR-Replayer.rar

features:
+ adjust speed (also run video backwards) [no autoadjustment done yet]
+ Step +-10 sec
+ Step to Video Start; +60 sec

todo:
- easy tsr output create and reduce (user still needs to let the whole video run in fast forward, so a 3h race would take 1.5h and take..eh..10min=210mb, so 1.8+ GB i guess..i dont know if my tool can handle this, but i wrote it to handle even 24h races^^)

- videocut (multiple methods here, such as replaying backwards,but those vids might cause problems if certain stats are wished, so i maybe wont support that videocuttype)
- more support for other arenas/cars (and other tracktypes)
- show goals and other stats during the video (in races: current position, if thats possible, as well as nice stats of driven laptimes, such as last laptime or comparison to laptime of car in front -> all what shyguy might love to have and more ;D -> and note here: i can create a seperate video output, which just shows the stats, so you can record the transparent overlay of stats in my tool and record ingame tsr if you want more details of the race, such as smoke, skidmarks and maybe another zoom level, though the zoom i can maybe support)
- autonavigate to x sec before next goal or previous goal
- navigate to videoend
- autocut goals from a video
- navigate to/ autocut highlights (like goalshots, which have been saved)
- easy mouse selection in autoscene-bar at bottom of replayer for pbmatches (that hopefully will make us create more easily cool skill videos!?)

- autocut fastest lap(of race or of player) of a racingvideo
- possibility to merge videos (and easily adjust 1 to the other, so we can easily compare fastestlaps-videos i.e.)
pb:
- add "fire" effect behind ball or player if they are suddenly way faster than they should; and maybe some kinda "blizzard" ending on the position where the car or ball suddenly accelerated crazily

*meh, i had once nice feature i forgot now...
anyway, post your wishes/ideas here


@tijny:
i cant currently open the tsr format ande once described to me in a mail, i think.
do you have it? i rly forgot the format and dont even know anymore if there are just the key inputs or also/instead the positions of the cars.
in other words, would be great to just cut the tsr files instead creating a reduced output file from a huuuuge output.txt

[Whiplash]

Damn! I didn't expect it to be so smooth. Awesome work!

[Tijny]

@tijny:
i cant currently open the tsr format ande once described to me in a mail, i think.
do you have it? i rly forgot the format and dont even know anymore if there are just the key inputs or also/instead the positions of the cars.
in other words, would be great to just cut the tsr files instead creating a reduced output file from a huuuuge output.txt

The TSR format does store positions, but only at intervals specified by the recording interval setting. By default this is 3000ms for server videos and 400ms for client ones. If you need a higher frequency, I'm afraid there's no way to reliably extract the positions from a video. Anyway, I do not have any documentation on the format, but most of the overall structure is in my head somewhere and if you need to know anything specific, I could dig it up for you.

I've also been wondering how to do this without having the game generate an output.txt, but so far I haven't really come up with anything.

[power79]

@tijny:
i cant currently open the tsr format ande once described to me in a mail, i think.
do you have it? i rly forgot the format and dont even know anymore if there are just the key inputs or also/instead the positions of the cars.
in other words, would be great to just cut the tsr files instead creating a reduced output file from a huuuuge output.txt

This tsr reading would interest me too. I also could possibly create some tools for TS if I could make some sense of those files. Is there some reason to keep those file formats a secret?

[Tijny]

This tsr reading would interest me too. I also could possibly create some tools for TS if I could make some sense of those files. Is there some reason to keep those file formats a secret?

Well, here's an introduction to the TSR format:

Overall structure
TSR videos are gzip-compressed binary files. This means that in order to be read from, they must be fed
to a gzip-decompression library; the resulting binary output can be parsed relatively trivially.

The information is stored in the form of messages, most of which comply to this format:

Code: Select all

int message_length: 4 bytes; 
int message_type: 4 bytes;
message_data: (`message_length` - 4) bytes.

For messages of this type, the first byte of the length is always zero; if it's non-zero, it's a short message that signifies a control change or a time update.

Integers are stored in a little-endian fashion, meaning that the most significant byte comes first. For example, the integer 500 will be stored as 00 00 01 F4. (hexadecimal)

Message types
A video begins with a message type of 500, representing the file header. This contains whether it's a client or server video, the game version, and the date. If you're interested in how to interpret this and you can't figure it out, ask me.

Probably the most important message is the one that follows after that. It contains pretty much all the essential data that must be known before video playback can be started. The type of this message is 5.

Its format for the data is roughly as follows:

Code: Select all

int lap_number: 2 bytes
int track_data_length: 4 bytes
int track_hash: 16 bytes
int track_name_length: 4 bytes
string track_name: `track_name_length` bytes
int track_maker_length: 4 bytes
string track_maker: `track_maker_length` bytes

int player_data_length: 4 bytes
int version: 4 bytes
int player_count: 4 bytes

unknown: 12 bytes

player_data = {
    int hash: 4 bytes
    int primary_color: 3 bytes
    int secondary_color: 3 bytes
    int tertiary_color: 3 bytes
    bool is_ghost: 1 byte
    int color_style: 1 byte
    string player_name: 17 bytes
} × `player_count

int starting_grid: `player_count` bytes

int car_data_length: 4 bytes
car_data = {
    int car_hash: 8 bytes
    int car_name_length: 4 bytes
    int car_name: `car_name_length` bytes
} × `player_count`

int records_data_length: 4 bytes
re2_format records_data: `records_data_length` bytes

The other message types follow the same principles, and often the meaning of the data can be inferred from the data itself, this is what I had to do regularly when I was writing my tools.

Other types of messages include: (type ids in parentheses)

• Time (501): Sets the race state time.
• Start (502): Starts playback
• Stop (503): Stops playback
• Race state (9) Race state update: contains information about the car's positions, but is relatively infrequent.
• Race stats (10) This contains data about player' laptimes, number of completed laps, and so on.
• Chat message (11)

I hope you understand that I'm not going to provide full documentation for all of them - if you need to know something specific, just ask, but I hope to have helped you on your way at least.

[Mike Nike]

@Tijny

i gone through the huge output.txt again, finding at least a reference to the trk file.
font and engine wav, hardly...

from startpositions (and from trk size x y, which is located in output.txt) i can take the /team arrays, in races the grid. if grids are nondefault, i can still take the positions from trk file.

so...im missing:
.car references (which i could search though in cars folder, since laprecord of track is searched in output txt with string "Antislider" -> problems exist, if there are multiple car defs for the same carname

eventually i could go through logfiles, if exist, match them with the tsr file name and hope to get some more information about the race, the normal output wouldnt give me.

car colors and colortype (stripes) i cant get all (local ones partly maybe, if there are no multiple hashs for 1 nick)

ooooor i just ask ande to create a special tool to convert tsr to my favourite tsr file type format^^


edit: allright, you posted while i wrote..ty³ (seems i should refresh page next time^^)



hmm, interesting is...
if you define the videotype as based on physical functions, you can save a lot of disk space by calculating positions during replay by intFrame and boolKeyInputs[4]...
unfortunately the calculation takes a while if you want to calculate far away from current Frame

while just saving each frames takes more space, but it is so much easier for so many things^^. i was always thinking that the tsr format uses the complicated disk space friendly method - thats right, or?


@all
now with some sort of fire shots
http://mikenike.mi.funpic.de/ts/TSR-Replayer%20v1.1.rar

[Tijny]

hmm, interesting is...
if you define the videotype as based on physical functions, you can save a lot of disk space by calculating positions during replay by intFrame and boolKeyInputs[4]...
unfortunately the calculation takes a while if you want to calculate far away from current Frame

while just saving each frames takes more space, but it is so much easier for so many things^^. i was always thinking that the tsr format uses the complicated disk space friendly method - thats right, or?

Yes, it is very disk space friendly, but it's hardly complicated. The video format is very much in line with the game's own logic, it can just reuse the same functions it also uses for the game itself, with relatively little additional code. If I'm correct, most of the messages stored in a TSR video are actually an exact copy of the data that's sent across the wire in network games.
But yeah, for us it would be easier if it were implemented differently.

[Mike Nike]

for lazy testers a little yam yam:

http://mikenike.mi.funpic.de/ts/TSR-Replayer%20v1.2.jpg

[Whiplash]

It would be nice if this could count ball possession. Every ball touch should be counted in that case. That should be done for each player and for each team. I'm not sure if replay allows you to extract that info.

[Mike Nike]

the output gives me lines of "Collision Avoidance", not telling me what collides though. but i can find that out by comparing the positions.

there might be multiple collisions at once, when press shots happen or those uber explosions when RDP-Antisliders get crazy and cars are sent to the left and right side of the screen ^^
so yeah, i could count the player(ballContacts) as well as the player(ballPossessionTime), though the possession often wouldnt really match a valueable ball possession statistic, im afraid..
some players, like me, might often let the ball untouched get to a corner until an opponent comes closer.
i could certainly count the player(closesToBallTime) though and maybe take some sort of formula of those 3 values to get a somewhat matching player(ballControlTime) statistic.

*edit: trying to go through tsr right now..

[power79]

Well, here's an introduction to the TSR format:

Thanks a lot! I don't have time to look into this right now but it seems here's enough information for me to get started.

[Whiplash]

though the possession often wouldnt really match a valueable ball possession statistic, im afraid..
some players, like me, might often let the ball untouched get to a corner until an opponent comes closer.

Well, you don't have to think that the team with a bigger possession percent must be the winning one. I actually think that ball touch IS a valuable criteria. If some of the touches your tool can't notice, that not a big problem - because few touches will not make a crucial changes to stats. So, at least, give it a try.

[Mike Nike]

well, if it is somehow possible to extract the positions of the cars from the binary file, that would be great and no ingame output recording would be needed.
just the use of a gzip decompressor.
if it ends up needing to find out the game logic, how the physics are updated, this might fail though, since im afraid im not motivated to try hard enough to do so..
then i could calculate each position relativly fast from videocut-starting points.
unless ande would actually tell me more about the physics than he did when i made the carcreator tool back then..

[Tijny]

well, if it is somehow possible to extract the positions of the cars from the binary file, that would be great and no ingame output recording would be needed.
just the use of a gzip decompressor.
if it ends up needing to find out the game logic, how the physics are updated, this might fail though, since im afraid im not motivated to try hard enough to do so..
then i could calculate each position relativly fast from videocut-starting points.
unless ande would actually tell me more about the physics than he did when i made the carcreator tool back then..

As I said, positions can be extracted from a video but for server videos the interval is probably too long to be very usable. I also don't think it's a plausible project to reimplement all of the game's physics, but if you could somehow find a way to play a video without the graphical part, it should be possible to generate some output file from a video in a matter of seconds. This sounds like a great hacking project for me, haha. Not sure how difficult it would be, need to investigate!

[Mike Nike]

well, im on my way...
eh, first 6 bytes of msgtype 500: in my tsr i have 7 184 0 0 0 29
2nd byte=184 -> 1.84 version of ts? but the others?
and byte 13-16: 0 0 1 238 -> some code for clocktime? 21.45oclock it should be