ZX0 with prefix at arbitrary address

[jorgegv]

Hi @Einar Saukas , i'm trying to use ZX0 compression with prefix, but as far as I can see, the prefix table must be at the beginning of the compressed data (and It Will be skipped when decompressing).

Would It be posible to use a prefix table at an arbitrary address other than the begining of data? This would allow to share a given prefix table for compressing several different datasets. This is precisely my use case in RAGE1.

I presume the decompressor would need to be modified for this, and also the compresor for outputting a separate prefix table...

Shall I open a GitHub ticket for this?

Thanks
J.

[Einar Saukas]

i'm trying to use ZX0 compression with prefix, but as far as I can see, the prefix table must be at the beginning of the compressed data (and It Will be skipped when decompressing).

Correct. Here's the idea:

Suppose a certain game have player sprites stored in memory between address 40,000 and 42,999. Before starting each new level, this game will decompress boss sprites (specific to each level) to address 43,000. In this case, it may be useful to let ZX0 know that those player sprites will already be in memory immediately before the decompressing address. This way, if there's any part of the boss sprites that's similar to some player sprite, ZX0 may simply store an instruction to copy this part, instead of storing the entire boss sprite in the compressed block. This could make the compressed block smaller.

Would It be posible to use a prefix table at an arbitrary address other than the begining of data? This would allow to share a given prefix table for compressing several different datasets.

It's possible, but probably not worth it. Using a prefix won't be efficient if the prefix area isn't stored immediately before the decompressing address. Either ZX0 will need to use larger references (thus reducing compression), or we will need a much slower larger decompressor to "fill the gap" between prefix and decompressing address.

But first, let me better understand your requirements. Why do you need to decompress different datasets at different addresses?

• If each dataset is required in a different game level, why don't you decompress one dataset at a time, always at the same address? You will use a lot less memory this way.
• Of if you need 2 datasets in memory simultaneously for the same level(s), why don't you compress them together as a single block? You will get better compression this way.

What kind of data you have in your prefix table (sprites or something else)? If the prefix and compressed data contain different kinds of data, it probably won't make noticeable different in compressed size, since there will be too few similarities between them that ZX0 could use. Another requirement is that the prefix data should be immutable. If you used a certain prefix to compress data, and anything has changed in the prefix when you use it later to decompress the same data, it won't decompress correctly.

[jorgegv]

It's possible, but probably not worth it. Using a prefix won't be efficient if the prefix area isn't stored immediately before the decompressing address. Either ZX0 will need to use larger references (thus reducing compression), or we will need a much slower larger decompressor to "fill the gap" between prefix and decompressing address.

Mmm I think i understand. I believed the prefix table was some sort of "pretraining" of the decompressor before it started to output real uncompressed bytes at the proper address, but from your explanation it seems that the decompressor counts on that previous bytes already being decompressed at the final address to output the next ones, right? I think then my request does not make much sense...

But first, let me better understand your requirements. Why do you need to decompress different datasets at different addresses?

This is my use case:

• I have some sprite, tile and screen data which are stored in a "dataset". The data here is deduplicated, but not compressed (I can ellaborate on this later, but it's not relevant for this discussion).
• Since I have lots of screens, I have several datasets (10 in my current game), each of them containing data for the screens contained in it.
• The sprite and tile data is duplicated at some level in each of the datasets, because I have freely used all of the possible game tiles in any of the screens. There is not a single tile table and a single sprite table for the whole game, but instead each dataset has its own tile and sprite tables, and the screen definitions in each dataset only reference tiles and sprites stored in that dataset. E.g. sprite "dog" can be duplicated in dataset 0 and dataset 3, and tile "house" in dataset 5 and dataset 7.
• Datasets are stored compressed (separately) in the upper banks, as ZX0 streams.
• All datasets are compiled independently (they only contain data) and linked to start at address 0x5C00. The memory window at 05C00 is approx ~9kB long, and when any dataset is in use, it is temporarily decompressed to the 0x5C00 address. THis decompression normally takes place when entering a screen, and is completely unnoticeable for the user.

And now, I was thinking that since the enemy sprites are stored repeated in several datasets, I could spare 1K in non-banked memory to put some "prefix data" (be that whatever it be) so that the compressor could squeeze my datasets a bit more.

This is my use case.

P.S. Any suggestions about changing the memory layout, putting sprites in low memory, etc. are out of the question for the moment. This is the layout that we have currently and we cannot change it without imposing a big publication delay due to the redesign.