Unreal Software - Thread: Save huge table in textfile fast

Save huge table in textfile fast

Offline

03.02.21 07:06:38 pm

Hey

nrealSoftware,

I was testing some Record/Replay script to create demos of your games you can watch later.

Idea:
This script creates a big table for every "tick". This table includes every information of all the relevant hooks, including almost all parameters. This table gets saved into a textfile periodicly - the demo file.

Problem:
The problem with saving this periodicly using normal io-functions into a textfile: You get a noticable freeze when it saves duo to the huge amount of lines to print, with just ~4 players.

My solutions so far:
Saving everything for a round in the table and only write on roundstart into the textfile would sort-of decreese the freeze problem (a lag on roundstart is less impactful than mid-round) BUT the save process takes over 30seconds for like 5 players (which depends on roundtime and amount of hooks fired listed in Addhooks.lua).

How did I structure my code:
In the config.lua you will only find constant variables to make the script easy modifiable.
The Assembler.lua helps me to create strings (Used in Addhooks.lua).
The GameState.lua is every data I collect for the current tick.
The GameReplay.lua is every data for every tick (It contains an "actualState"-instance of GameState which swaps out every tick).
Addhooks.lua contains every addhook command. Every addhook command saves the data about the parameters in GameReplay.actualState.

The source code can be found here:https://github.com/Bowlinghead/cs2drecorder
Dofile "dofileRecord.lua" to let the script run; then write "!record <name>" to start record/ "!stop" for stop.

Essential code parts are quoted here:

When user types e.g. "!record footype":

1
2
3
4
5
6
7
8
9
10
11
12
13
14

function GameReplay:StartRecord(savefile)
	
	--[[
		Some is-already-recording
	]]

	-- IO stuff
	self.file = io.open(self.defaultPath..savefile..self.format,"a"); 
	io.output(self.file);

	--[[
		Some saving
	]]
end

When it saves - either done all at once (:Save()) or only for some amounts of ticks (:SaveSome()).

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29

function GameReplay:Save()
	if (GameReplay.isRecording) then
		
		for timeStamp,state in pairs(self.stateList) do 	-- for each stat to save
			if (state ~= nil) then
				io.write(tostring(timeStamp).." = "..state:ToString());
			end
		end
		self.stateList = {};
	end
end

function GameReplay:SaveSome(amount)	-- amount: actually amount of ticks to save
	local counter = 0;
	
	for timeStamp, state in pairs(self.stateList) do
		if (counter >= amount) then
			return;
		end
		
		if (state ~= nil) then
			io.Write(tostring(timeStamp).." = "..self.stateList[timeStamp]:ToString());
			
			self.stateList[timeStamp] = nil;
			
			counter = counter + 1;
		end
	end
end

When the user types "!stop":

1
2
3
4
5
6
7
8
9
10
11

function GameReplay:EndRecord()
	if (GameReplay.isRecording) then

		self:Save();  -- call the Save function from above

		io.close();
		
		msg("\169255255255Stopped the recording!");
		GameReplay.isRecording = false;
	end
end

Example of triggering save on addhook-event (Addhooks.lua):

1
2
3
4
5
6
7
8
9

addhook("attack","AttackHook")
function AttackHook(id)

	-- To succesfully simulate the game, save player angle when he attacks
	assembler = Assembler:New();
	assembler:SetEntry(player(id,"angle")); -- using my "assembler" class is kinda pointless, I know
	
	gameReplay.actualState:SetEntry("attack", id, assembler:Assemble()); 
end

The :SetEntry() function you see 3 lines above (Found in GameState.lua):

1
2
3
4

function GameState:SetEntry(strAction, intPlayerId, strParam)
	local entry = {strAction, intPlayerId, strParam };
	table.insert(self.list, entry);
end

The script will :StartRecord() then saves data into the GameReplay-object, which then :Save()s or :SaveSome(tickAmount) into the textfile. Then it will :EndRecord()

Note that this script runs perfect on my side, except for the long write-time (tested with bots). I dont know the magic behind the io.- maybe I use it completly wrong. The file remains open during the whole recording - what does it mean, to (io.)open a file essentially?

TL;DR
- How do I save big table data into a text file the fastest?
- On which factors does the speed to io.write depend on mainly? (Example: Is 1 io.write call with a big string better than 100 small io.writes?)

Re: Save huge table in textfile fast

Cebra
User Off

Offline

04.02.21 12:21:53 am

Can you save incremental?
Should reduce the data, but need more calculation (maybe each tick, depending on the implementation)

I did a small test and maybe will do some more research: for a small dataset of 100 kb the single-write-op is 10-times faster then the multi-wirte-op (the raw writing-operation, not include the creation of the diffrent types of data)

Re: Save huge table in textfile fast

MikuAuahDark
User Off

Offline

04.02.21 02:50:16 pm

Bowlinghead has written

How do I save big table data into a text file the fastest?

I've left my code review to the GitHub issue https://github.com/Bowlinghead/cs2drecorder/issues/1 and you probably want to read it as I highlight important bits in your code.

Bowlinghead has written

On which factors does the speed to io.write depend on mainly?

io.write

is actually fast, assume with default file buffering. If you disable file buffering, then it's better to write large strings in one go.

Optional

Re: Save huge table in textfile fast

Gaios
Reviewer Off

Offline

04.02.21 03:47:50 pm

[SALT] Save and Load Tables (12)

And the fastest way could be binary write.

Forum

Save huge table in textfile fast