A new data compressor called ZX0

[Alone Coder]

All sizes are in bytes, a * means that the game was paritally run to decrypt itself before saving the binary file.

Can you please upload these files?

[introspec]

I thought people might like to see a table of games that I investigated making into a ZX IF/2 Interface cartridge but then gave up due to size issues. You can see just how much better ZX0 is than ZX7.

Code: Select all

		orig	zx7	zx0	shrinkler
Arkanoid 2	33024	17407	16384	15536
Bargain Basemt	32700		21105	20108
Bruce Lee	33239	18433	17369	16404
Chessmaster 2k	34814		14775	14368
Deathstar Int	41541		21405	20052
Everyones Wally*32809		20144	19044
Exploding Fist	38915		22890	21916
Finders Keepers	36860		18831	17972
Flunky		38650	25674	23706	22884
Fox fights bk	40224		20660	19688
Ghostbusters	32064	19717	18636	17976
Gilligan's Gold	39936	20719	18968	18088
Greg Loses Clk	32900	22948	21289	20588
Hampstead	33957		17787	16460
Herbert Dummy* 	32191		20327
Hobbit		40000		27347
Hydrofool	36466		22713	21404
Iball		41436		23090	21756
Iball 2		40530		23534	22000
Knightlore*	30720	19490	18228	17064
Lunar Jetman*	31745		19509	18940
Matchday 2	34565		23526	22808
Max desktop	32099	19850	18474	17680
Minder		39001		23301	22284
Mordon's Quest	41204		28565	26988
Mugsy		40960		28582	27536
Night Gunner	40960	25971	24302	23472
Pippo		34151	34880!	30839	30780
Raid over Mscw	40282	23928	21664	20764
Roland Rat Race	36890	17955	16501	15664
Sabre Wulf*	35936		23632	22168
Sigma 7		33024	24650	22977	21904
Starion		45024		21345	20848
Starstrike	41024		20931	19148
Tapper		39724		22813	22624
Tau Ceti	40520		22899	21756
The Eye of Bain	32340	23869	22285	21212
Through TrapD	32860		20824	20112
Thrust		40351	19371	17159	16252
Trapdoor	35600		22244	21412
Travel Trashman	32768		19791	18948
Turbo Esprit	38351		27680	26636
Virus		35134	20995	18734	17920
Wheelie		39037		18066	17196

Why do you use Shrinkler as a point of reference? It is slower to decompress by 2-3 orders of magnitude, to the extent that it is about the same speed as loading uncompressed data from tape.

[Bedazzle]

Why do you use Shrinkler as a point of reference?

Because of size?

[jimmy]

Why do you use Shrinkler as a point of reference?

Because we need a theoretical minimum size - it's not going to be zero or one, and not even 1K in these cases. As Shrinkler seems to offer the best compression at present I thought it was a useful reference. I could've used .zip, but by sticking to Shrinkler I know they can be decompressed on a Spectrum with a small piece of code.

It is interesting that for complex data (eg:Pippo) there isn't much difference between ZX0 and Shrinkler, but for adventures such as Eye Of Bain, Hampstead and Hobbit the Shrinkler files are at least 1K smaller. There might be room for improvement here.

I can't update the original table, so I've posted an updated table below. Some more games should've been marked as decrypted. I've included Atic Atac, added the missing shrinkler values and included more zx7 values.

Code: Select all

		orig	zx7	zx0	shrinkler
Arkanoid 2	33024	17407	16384	15536
Atic Atac*	30209	20039	18400	17240
Bargain Basemt	32700	22495	21105	20108
Bruce Lee	33239	18433	17369	16404
Chessmaster 2k	34814	16069	14775	14368
Deathstar Int	41541	24060	21405	20052
Everyones Wally*32809	21446	20144	19044
Exploding Fist	38915	26036	22890	21916
Finders Keepers	36860	21777	18831	17972
Flunky*		38650	25674	23706	22884
Fox fights bk	40224	22269	20660	19688
Ghostbusters	32064	19717	18636	17976
Gilligan's Gold	39936	20719	18968	18088
Greg Loses Clk	32900	22948	21289	20588
Hampstead	33957	19653	17787	16460
Herbert Dummy* 	32191	21763	20327	19252
Hobbit		40000	28911	27347	26076
Hydrofool	36466		22713	21404
Iball		41436	24954	23090	21756
Iball 2		40530	25304	23534	22000
Knightlore*	30720	19490	18228	17064
Lunar Jetman*	31745	21082	19509	18940
Matchday 2	34565		23526	22808
Max desktop	32099	19850	18474	17680
Minder		39001	25111	23301	22284
Mordon's Quest	41204	29799	28565	26988
Mugsy		40960	30459	28582	27536
Night Gunner	40960	25971	24302	23472
Pippo		34151	34880!	30839	30780
Raid over Mscw	40282	23928	21664	20764
Roland Rat Race	36890	17955	16501	15664
Sabre Wulf*	35936	25318	23632	22168
Sigma 7		33024	24650	22977	21904
Starion		45024		21345	20848
Starstrike	41024		20931	19148
Tapper		39724		22813	22624
Tau Ceti	40520		22899	21756
The Eye of Bain	32340	23869	22285	21212
Through TrapD	32860		20824	20112
Thrust		40351	19371	17159	16252
Trapdoor*	35600	23836	22244	21412
Travel Trashman	32768		19791	18948
Turbo Esprit	38351	29361	27680	26636
Virus		35134	20995	18734	17920
Wheelie		39037	19074	18066	17196

[jimmy]

All sizes are in bytes, a * means that the game was paritally run to decrypt itself before saving the binary file.

Can you please upload these files?

Upload all those denenienced titles? Sorry I can't do that.
However you can recreate them just by loading the game in and stopping it just before it starts (or just after it decrypts). As an example I'll show you how I obtained a decrypted Flunky binary:

• The game has a standard loader screen$ and a hyperloader for the main code.
• Place a breakpoint at $FDE8 and start loading the game. It should stop just before it starts the hyperloader. At this point IX=$6D60 and DE=$8FF2 so we know where the game loads and how big it is.
• Place another breakpoint at $FE70 - this should trigger after the game finishes loading. At this point the binary is still encoded.
• If you look around $FEA9 you can see the decryptor routine (it modifies memory starting at $6D60 and is $8FF2 bytes long)
• If you place a final breakpoint at $FEB8 this will trigger once decryption has finished.
• The decrypted bytes can now be save as "FLUNKY" CODE 28000,36850
• If you use zx0,zx7 or shrinkler 4.6 (without parity) you should see the same sizes I listed in the table.

[NEO SPECTRUMAN]

also zx0 (v1.5) is TO SLOW too on big files

[Einar Saukas]

also zx0 (v1.5) is TO SLOW too on big files

It's slower when there are lots of matches therefore more combinations to be analyzed. Fortunately it also means more opportunities for better compression. Don't you think that compressing from 65536 to 8 bytes was worth the wait?

Even so, your times are higher than expected. I repeated this test using a more-than-a-decade-old PC and it took under a minute. Are you using the "official" Windows executable file, or did you compile it yourself? If so, what's your platform, compiler and compile options? Also could you please try the most recent version?

[NEO SPECTRUMAN]

Don't you think that compressing from 65536 to 8 bytes was worth the wait?

nope
it's only just another one experiment result table

If so, what's your platform, compiler and compile options? Also could you please try the most recent version?

i will do it later
with constant CPU frequency (I just installed a new XP on new HDD, and now i'm not have all necessary drivers)

[Einar Saukas]

Can the algorithm be made to use multiple threads?

Sure. However memory management in current C implementation is based on reference counting, which would be inefficient across threads. This will require another language with GC and lightweight threads, preferably also resizeable hash tables. I will probably try Kotlin, unless anyone has a better idea?

OK, the multi-thread version of the ZX5 compressor is now available here:

https://github.com/einar-saukas/ZX5-Kotlin

You can download the compiled JAR file directly from here:

https://github.com/einar-saukas/ZX5-Kotlin/releases

You don't need Kotlin to use it, but you must have JVM installed (Java 8 or later). I'm aware that a few people here don't like Java, but as I mentioned before, it wouldn't be efficient to use a lower level language for this!

[Einar Saukas]

Since I learned Kotlin to implement the ZX5 multi-thread compressor, why not do the same for ZX0?

The multi-thread version of the ZX0 compressor is now available here:

https://github.com/einar-saukas/ZX0-Kotlin

You can download the compiled JAR file directly from here:

https://github.com/einar-saukas/ZX0-Kotlin/releases

[jimmy]

84 hours in, with the last dot (....) under the space between "Einar" and "Saukas" it stopped with "Error: Insufficient Memory".
I'm slightly confused about this because when I last looked in Task Manager it was only taking 1.6Gb of space - it had 8Gb to play with.

So I did another test using the -quick mode on the same file and the results were:
zx0 compressed from 33887 bytes to 17007 (non-quick mode is 16356)
zx5 compressed from 33887 bytes to 17223

Whilst watching this 'quick' zx5 compression I noticed that at the point where my first attempt failed, memory usage was around 40Mb. When the dot had moved across to the last letter "s", memory usage was not much higher. However the compressor spent most of it's time on this last dot of progress - nearly 2 hours! Memory usage also increased alarmingly to over 332Mb by the time it finished.

Based on this I don't think my first test would ever have worked as the 32-bit zx5.exe would unlikely be able to address more than 4Gb. The zx5 quick mode took just over 150 minutes to complete! Even if zx5 was 64-bit, it seems to imply I'd need at least 16Gb of RAM to perform this task.

Is it true to say that if a quick zx5 result is larger/greater than a quick zx0 result, then the 'full compression' versions will also come out as zx5 is larger/greater than zx0 output? (ie: is this a quick test to save wasting hours compressing a file that zx0 would do better anyway?)

I don't have Java 8 installed so I haven't been able to test the multi-threaded versions, however I will see what I can do this weekend.

[Einar Saukas]

84 hours in, with the last dot (....) under the space between "Einar" and "Saukas" it stopped with "Error: Insufficient Memory".

Ouch!

Based on this I don't think my first test would ever have worked as the 32-bit zx5.exe would unlikely be able to address more than 4Gb. The zx5 quick mode took just over 150 minutes to complete! Even if zx5 was 64-bit, it seems to imply I'd need at least 16Gb of RAM to perform this task.

AFAIK the memory limit for 32 bit applications is 2Gb only.

Is it true to say that if a quick zx5 result is larger/greater than a quick zx0 result, then the 'full compression' versions will also come out as zx5 is larger/greater than zx0 output? (ie: is this a quick test to save wasting hours compressing a file that zx0 would do better anyway?)

No, I don't think it's a reliable estimate.

I don't have Java 8 installed so I haven't been able to test the multi-threaded versions, however I will see what I can do this weekend.

It should run much faster. Don't forget to specify memory size, as described in the project documentation!

[Einar Saukas]

I just released a ZX0 multi-thread compressor in Java, if anyone's interested:

https://github.com/einar-saukas/ZX0-Java

You can download the compiled JAR file directly from here:

https://github.com/einar-saukas/ZX0-Java/releases

The execution time is exactly the same as the Kotlin version. It means I didn't need to learn Kotlin after all

[andydansby]

I've started using ZX0 in my development scheme. Loving it.

An example project that I've started playing with is at https://github.com/andydansby/Vortex2_P ... 2_compress to compress Vortex tracker songs. Here I am creating a buffer and uncompressing the song to it to play. Next to add multiple songs.

Nice compression. Takes a 11302 byte song and compresses it to 4019 bytes.

Thanks Einar.

[Einar Saukas]

I've started using ZX0 in my development scheme. Loving it.

An example project that I've started playing with is at https://github.com/andydansby/Vortex2_P ... 2_compress to compress Vortex tracker songs. Here I am creating a buffer and uncompressing the song to it to play. Next to add multiple songs.

Nice compression. Takes a 11302 byte song and compresses it to 4019 bytes.

Thanks Einar.

You are welcome!

Thanks a lot for publishing a tutorial about using zx0 with z88dk/sccz80. I highly recommend it:

https://zxspectrumcoding.wordpress.com/ ... -lesson-7/

[Bedazzle]

Hello guys, I need a help.
Read documentation on github about decompressing data that overlaps source.
Packed screen is 4253 bytes with 3 bytes delta.
When I'm trying to unpack 6912 bytes into address 16384 all works like a charm, but when I'm changing destination to 49152 something goes wrong - rubbish is copied to the screen, and if I explore memory at 49152, picture is unpacked incorrectly.

Code: Select all

	device ZXSPECTRUM48

MEM_END  EQU 65535
displace EQU MEM_END - file_end + 1

	org 50000
	display "start ", /A, $
start:
	ld sp, 24999

	; fill screen with pattern
	ld hl, 16384
	ld de, 16385
	ld bc, 6911
	ld (hl), e
	ldir

	; move packed data and unpacker to end of memory
	ld hl, file_end-1
	ld de, MEM_END
	ld bc, file_end - file_start
	lddr
	jp unpacker + displace


	; data and code from this point will be moved
file_start:
	incbin "Andy Green - Flashback (2018).scr_norm.zx0"
	include	"dzx0_standard.asm"

unpacker:
	ld hl, file_start + displace
	ld de, 16384
	;ld de, 49152	; <=== crash
	
	push de
	call dzx0_standard + displace
	pop hl

	; copy unpacked picture to screen
	ld de, 16384
	ld bc, 6912
	ldir

	; dead end to indicate all went ok
	xor a
loop:
	xor %111
	out(254),a
	jr loop

file_end:

	savesna "test.sna", start

[Einar Saukas]

It seems you are compiling the decompressor at one address but executing it at another.

[Bedazzle]

It seems you are compiling the decompressor at one address but executing it at another.

Arrgh, silly me
Thank you!

P.S.
but wait... it does correctly extract to screen with same code (besides destination) !

[Einar Saukas]

P.S.
but wait... it does correctly extract to screen with same code (besides destination) !

When you moved the compiled code, the CALL instruction was still invoking a sub-routine at the original address. I'm guessing you did not override the old address when decompressing directly to screen, so it still worked (by accident).

However you erased the old sub-routine when you decompressed over it.

[Bedazzle]

When you moved the compiled code, the CALL instruction was still invoking a sub-routine at the original address.

Yes, you are right. Thank you again. I rewrote part of code with disp/ent, and it works now.

If anybody interested:

Code: Select all

	device ZXSPECTRUM48

MEM_END  	EQU 65535
move_size	EQU file_end - file_start

	org 50000
start:
	ld sp, 24999

	; fill screen with pattern
	ld hl, 16384
	ld de, 16385
	ld bc, 6911
	ld (hl), e
	ldir

	; move packed data and unpacker to end of memory
	ld hl, move_end-1
	ld de, MEM_END
	ld bc, move_size
	lddr
	jp unpacker

	; ---------------------------
	; data and code from this point will be moved on start
move_start:
	; compile part of program to this ORG address
	DISP 65535 - (move_size) +1
file_start:
	incbin "Andy Green - Flashback (2018).scr_norm.zx0"
	include	"dzx0_standard.asm"

unpacker:
	ld hl, file_start
	;ld de, 16384
	ld de, 49152	; <=== crash
	
	push de
	call dzx0_standard
	pop hl

	; copy unpacked picture to screen
	ld de, 16384
	ld bc, 6912
	ldir

	; dead end to indicate all went ok
	xor a
loop:
	xor %111
	out(254),a
	ld b, a
pause:
	djnz pause
	jr loop

file_end:
	ENT
move_end:
	; ---------------------------

	savesna "test.sna", start

[Einar Saukas]

Awesome! You are welcome

[Alone Coder]

calgary - 98571 (lost to Exomizer3)
canterbury - 27939
graphics - 163173 (better than others)
music - 63721
misc - 254202

RAR headers included.
Decompressor size is 342 bytes (uses 1566 bytes for temporary buffer).

TR-DOS version of ZXRar optionally supports compression with 2-byte blocks (helpful for code), with 386-byte decompressor. It has to be tested too.

Temporarily added 2-byte blocks in NedoOS version:
calgary - 98525
canterbury - 27911
graphics - 163943
music - 62740
misc - 250341

ZXRar with 2-byte blocks TOTAL = 603460, including file headers (lost to exomizer and ZX0)

mRIP archiver for PC by Eugene77 (2003 format, depacker size is 218 bytes, fits in 1-sector TR-DOS BASIC loader, working area of #5C2 bytes):
calgary - 91734 (better than exomizer etc)
canterbury - 25800 (better than exomizer etc)
graphics - 155697 (better than exomizer etc)
music - 60160 (lost to exomizer and ZX0)
misc - 245702 (lost to exomizer and ZX0)
TOTAL - 579093 (better than exomizer etc)

RIP archiver for PC by Eugene77 (2000 format, depacker size is 228 bytes, working area of #5C2 bytes):
calgary - 90794 (the best)
canterbury - 24658 (the best)
graphics - 150981 (the best)
music - 57276 (the best)
misc - 240765 (the best)
TOTAL - 565474 (the best)

So, the old native formats were not bad!

[TMD2003]

I admit I have never had a reason to try ZX0 before. But now, I do: a relatively simple screen, with lots of empty space, which has absolutely no need to take up 6912 bytes. It requires compression.

I can't run it on my regular PC - it's incompatible with 64-bit Windows.
I tried it with DOSBox and it crashed - the window sits there doing nothing and can't be interrupted (and the .zx0 file never appears).
My 15-year-old laptop running Windows 7 returned the message "Program is too big to fit in memory", even though it's only 130-odd KB.
My 23-year-old laptop running Windows 2000 is flatly refusing to start up... even in Safe Mode.

Any suggestions?

I've already tried the screen compressor in BMP2SCR, and I've disassembled the initial machine code routine. For some reason that is several light years beyond my pay grade, the screen only displays if I load it from tape and RANDOMIZE USR 50000. If I copy the code into an .ASM file, along with lines and lines and lines of DEFBs to hold the screen data and assemble it with Spin, all I get is a screen full of flashing shash. The first lines are:
CALL 6034 (which is just a RET in the ROM and nothing more)
DEC SP
DEC SP
There is no reference whatsoever to the first line of the screen data - the compressor loads the code at 50000, and the screen data starts at 50114. Is it performing some mad trick with whatever the stack pointer points to at the end of a load?
I tried replacing these first lines with LD SP,50114 but it was to no available. As soon as any SP instructions are thrown my way I'm completely in the dark.

I've also taken to analysing the 6912 bytes, piece by piece, writing a load of LD HL,***** then LD (HL),*** and the odd LDIR routine to draw it effectively byte by byte, leaving out all the huge chunks of zeroes. Effectively I'm compressing the screen by hand, which is lengthy and tedious.

[Einar Saukas]

I can't run it on my regular PC - it's incompatible with 64-bit Windows.

Check if you downloaded it correctly. You are supposed to click on the "Download" button.

Direct link:

https://github.com/einar-saukas/ZX0/raw ... in/zx0.exe

[TMD2003]

...and that's working, on my main PC that outright rejected what I had yesterday. I was using the version found here:
https://github.com/einar-saukas/ZX0/tree/main/win

"Updated executables" seemed like the right place to go...

1,923 bytes! About 600 less than BMP2SCR required. I knew this would be the right option. Cheers!