Missing support for #include and #base directives

[hkva]

Hi.

Valve's KeyValues format supports two statements for including and merging other KeyValues files ( #include and #base), but trying to parse a vdf file using either statement causes a ParseError. Merging other files seems to be beyond the scope of this parser, but would it be possible to add an option that ignores statements like these instead of throwing an error?

Here's a stripped-down example from Team Fortress 2's population files:

minimal.pop:

#base robot_standard.pop

WaveSchedule
{

	StartingCurrency		1200
	RespawnWaveTime 		2
	CanBotsAttackWhileInSpawnRoom no
}

robot_standard.pop:

WaveSchedule
{
	Templates
	{
		// Sniper
		T_TFBot_Sniper
		{
			Class Sniper
			Skill Hard
			MaxVisionRange 2500
		}
	}
}

[CosmicHorrorDev]

Thanks for opening an issue. I remember seeing mention of #include and #base, but wasn't sure how to handle it at the time so I just ignored it.

Now that everything else is pretty established it would be good for me to actually handle these statements. I'll take a look and see how languages in other libraries handle things, but just ignoring the files may be a good solution since it seems odd for the parser to try and access other files on your computer while running.

There may also be some clean way of including the tokens for both of those statements in the parsed output as well so that consumers can manually stitch them together if they desire.

I'll look into things more this weekend and get back to you with any info!

[CosmicHorrorDev]

So I've been looking into this issue and I've got a couple of questions if you don't mind

---

So I downloaded all the free to play or already owned Valve games that I have, and from some investigation it looks like the files you are talking about are packed into a ~/.local/share/Steam/steamapps/common/Team Fortress 2/tf/tf2_misc_019.vpk file. Following the information on the Valve VPK wiki page I was able to get the help output from vpk_linux32, but I'm not able to list or extract any of the VPK files I've tried, and from what I can tell it looks like this tool no longer works on Linux.

From that I would assume that you're on Windows using the vpk.exe file instead? How do you extract these files? (It would add hundreds to thousands of extra VDF files that I can test the parser with)

---

Just making sure that I get the usage of #base and #include right, it looks like base would combine the provided base onto the matching object so from your example they would end up becoming what's below?

WaveSchedule
{

	StartingCurrency		1200
	RespawnWaveTime 		2
	CanBotsAttackWhileInSpawnRoom no
        Templates
	{
		// Sniper
		T_TFBot_Sniper
		{
			Class Sniper
			Skill Hard
			MaxVisionRange 2500
		}
	}
}

and then I would assume that #include simply includes the contents of the specified file into the location it's used?

I should be able to add support for exposing both of these in the parser, although it will involve some breaking changes (totally fine by me, the parser is just v0.1.0 after all)

[hkva]

From that I would assume that you're on Windows using the vpk.exe file instead? How do you extract these files? (It would add hundreds to thousands of extra VDF files that I can test the parser with)

I've been using the ValvePython command-line vpk tool on Linux: https://github.com/ValvePython/vpk

Just making sure that I get the usage of #base and #include right, it looks like base would combine the provided base onto the matching object so from your example they would end up becoming what's below?

Yeah. #base seems to load and merge a secondary KeyValues structure.

The C++ code for this step is surprisingly well-documented: https://github.com/ValveSoftware/source-sdk-2013/blob/0d8dceea4310fde5706b3ce1c70609d72a38efdf/sp/src/tier1/KeyValues.cpp#L2014-L2049

and then I would assume that #include simply includes the contents of the specified file into the location it's used?

I haven't been able to figure out what #include does. The Wiki Page says the statement flat-out doesn't work, and I haven't seen it in use in any of Valve's games. I can experiment with it more in a few hours.

[CosmicHorrorDev]

Thanks for linking the python tool and the extra info! I think I saw some #include directives onside some packed CSGO files, and from what I could tell it looks like all it does is some text replacement with the contents of the file it mentions

I'll dig in to things more later tonight

[CosmicHorrorDev]

Thanks for all that information. Knowing how to unpack .vpk files got me over 20k more files to test out just from that tf2_misc pack alone. I've sifted through a select few of them so far and it seems that

• The #includes were for some text file collection stuff, so I still haven't seen any VDF file use it
• There can be 0 or more #base usages
• There can be just a #base usage by itself with no object (just entirely copies it I would assume)

The things that I'm not super clear on yet

• If #base can only be used on the top object (I haven't seen any usage otherwise and it would make my life easier)
• If the files indicated by #base can have #bases in them as well. Haven't seen it yet, but it wouldn't be too bad even if that was the case

I'll be hacking together a script so that I can test the parser across all these new files and dump any files that failed to parse so that I can look at them for reference, but from what I've seen so far supporting #base should be rather easy. I doubt I'll have time to get through everything and have it tidied up this weekend, but I should have it done by the end of next weekend at the latest

[CosmicHorrorDev]

Alright so with 0ea6be9 there is now very limited support for #base macros.

Currently this is restricted to just parsing and doesn't yet have any documentation or ergonomics that I'm planning on adding down the line. The good news is that it does now parse the provided file you listed

# Cargo.toml
[package]
name = "parse-vdf-with-base"
version = "0.1.0"
edition = "2021"

[dependencies.keyvalues-parser]
git = "https://github.com/LovecraftianHorror/vdf-rs"
rev = "0ea6be99eb372932344be6210ca519f143c3649b"

// main.rs
use keyvalues_parser::{PartialVdf, Vdf};

const VDF_TEXT: &str = r#"
#base robot_standard.pop

WaveSchedule
{

	StartingCurrency		1200
	RespawnWaveTime 		2
	CanBotsAttackWhileInSpawnRoom no
}
"#;

fn main() {
    println!("{:#?}", Vdf::parse(VDF_TEXT).unwrap());
    println!("{:#?}", PartialVdf::parse(VDF_TEXT).unwrap());
}

The PartialVdf has a field that stores the paths indicated by #bases while the Vdf currently just ignores any bases it sees. The output of the PartialVdf bit is as follows

PartialVdf {
    key: "WaveSchedule",
    value: Obj(
        Obj(
            {
                "CanBotsAttackWhileInSpawnRoom": [
                    Str(
                        "no",
                    ),
                ],
                "RespawnWaveTime": [
                    Str(
                        "2",
                    ),
                ],
                "StartingCurrency": [
                    Str(
                        "1200",
                    ),
                ],
            },
        ),
    ),
    bases: [
        "robot_standard.pop",
    ],
}

---

I'm still planning on adding in more functionality later on (rendering with bases, merging, etc.), but that will take some time to get through for the next release