A new data compressor called ZX0
Re: A new data compressor called ZX0
84 hours in, with the last dot (....) under the space between "Einar" and "Saukas" it stopped with "Error: Insufficient Memory".
I'm slightly confused about this because when I last looked in Task Manager it was only taking 1.6Gb of space - it had 8Gb to play with.
So I did another test using the -quick mode on the same file and the results were:
zx0 compressed from 33887 bytes to 17007 (non-quick mode is 16356)
zx5 compressed from 33887 bytes to 17223
Whilst watching this 'quick' zx5 compression I noticed that at the point where my first attempt failed, memory usage was around 40Mb. When the dot had moved across to the last letter "s", memory usage was not much higher. However the compressor spent most of it's time on this last dot of progress - nearly 2 hours! Memory usage also increased alarmingly to over 332Mb by the time it finished.
Based on this I don't think my first test would ever have worked as the 32-bit zx5.exe would unlikely be able to address more than 4Gb. The zx5 quick mode took just over 150 minutes to complete! Even if zx5 was 64-bit, it seems to imply I'd need at least 16Gb of RAM to perform this task.
Is it true to say that if a quick zx5 result is larger/greater than a quick zx0 result, then the 'full compression' versions will also come out as zx5 is larger/greater than zx0 output? (ie: is this a quick test to save wasting hours compressing a file that zx0 would do better anyway?)
I don't have Java 8 installed so I haven't been able to test the multi-threaded versions, however I will see what I can do this weekend.
I'm slightly confused about this because when I last looked in Task Manager it was only taking 1.6Gb of space - it had 8Gb to play with.
So I did another test using the -quick mode on the same file and the results were:
zx0 compressed from 33887 bytes to 17007 (non-quick mode is 16356)
zx5 compressed from 33887 bytes to 17223
Whilst watching this 'quick' zx5 compression I noticed that at the point where my first attempt failed, memory usage was around 40Mb. When the dot had moved across to the last letter "s", memory usage was not much higher. However the compressor spent most of it's time on this last dot of progress - nearly 2 hours! Memory usage also increased alarmingly to over 332Mb by the time it finished.
Based on this I don't think my first test would ever have worked as the 32-bit zx5.exe would unlikely be able to address more than 4Gb. The zx5 quick mode took just over 150 minutes to complete! Even if zx5 was 64-bit, it seems to imply I'd need at least 16Gb of RAM to perform this task.
Is it true to say that if a quick zx5 result is larger/greater than a quick zx0 result, then the 'full compression' versions will also come out as zx5 is larger/greater than zx0 output? (ie: is this a quick test to save wasting hours compressing a file that zx0 would do better anyway?)
I don't have Java 8 installed so I haven't been able to test the multi-threaded versions, however I will see what I can do this weekend.
- Einar Saukas
- Bugaboo
- Posts: 3141
- Joined: Wed Nov 15, 2017 2:48 pm
Re: A new data compressor called ZX0
Ouch!
AFAIK the memory limit for 32 bit applications is 2Gb only.jimmy wrote: ↑Wed Oct 20, 2021 1:47 pm Based on this I don't think my first test would ever have worked as the 32-bit zx5.exe would unlikely be able to address more than 4Gb. The zx5 quick mode took just over 150 minutes to complete! Even if zx5 was 64-bit, it seems to imply I'd need at least 16Gb of RAM to perform this task.
No, I don't think it's a reliable estimate.jimmy wrote: ↑Wed Oct 20, 2021 1:47 pm Is it true to say that if a quick zx5 result is larger/greater than a quick zx0 result, then the 'full compression' versions will also come out as zx5 is larger/greater than zx0 output? (ie: is this a quick test to save wasting hours compressing a file that zx0 would do better anyway?)
It should run much faster. Don't forget to specify memory size, as described in the project documentation!
- Einar Saukas
- Bugaboo
- Posts: 3141
- Joined: Wed Nov 15, 2017 2:48 pm
Re: A new data compressor called ZX0
I just released a ZX0 multi-thread compressor in Java, if anyone's interested:
https://github.com/einar-saukas/ZX0-Java
You can download the compiled JAR file directly from here:
https://github.com/einar-saukas/ZX0-Java/releases
The execution time is exactly the same as the Kotlin version. It means I didn't need to learn Kotlin after all
https://github.com/einar-saukas/ZX0-Java
You can download the compiled JAR file directly from here:
https://github.com/einar-saukas/ZX0-Java/releases
The execution time is exactly the same as the Kotlin version. It means I didn't need to learn Kotlin after all
-
- Microbot
- Posts: 147
- Joined: Fri Nov 24, 2017 5:09 pm
- Location: Syracuse, NY, USA
- Contact:
Re: A new data compressor called ZX0
I've started using ZX0 in my development scheme. Loving it.
An example project that I've started playing with is at https://github.com/andydansby/Vortex2_P ... 2_compress to compress Vortex tracker songs. Here I am creating a buffer and uncompressing the song to it to play. Next to add multiple songs.
Nice compression. Takes a 11302 byte song and compresses it to 4019 bytes.
Thanks Einar.
An example project that I've started playing with is at https://github.com/andydansby/Vortex2_P ... 2_compress to compress Vortex tracker songs. Here I am creating a buffer and uncompressing the song to it to play. Next to add multiple songs.
Nice compression. Takes a 11302 byte song and compresses it to 4019 bytes.
Thanks Einar.
- Einar Saukas
- Bugaboo
- Posts: 3141
- Joined: Wed Nov 15, 2017 2:48 pm
Re: A new data compressor called ZX0
You are welcome!andydansby wrote: ↑Sat Nov 06, 2021 1:35 pm I've started using ZX0 in my development scheme. Loving it.
An example project that I've started playing with is at https://github.com/andydansby/Vortex2_P ... 2_compress to compress Vortex tracker songs. Here I am creating a buffer and uncompressing the song to it to play. Next to add multiple songs.
Nice compression. Takes a 11302 byte song and compresses it to 4019 bytes.
Thanks Einar.
Thanks a lot for publishing a tutorial about using zx0 with z88dk/sccz80. I highly recommend it:
https://zxspectrumcoding.wordpress.com/ ... -lesson-7/
Re: A new data compressor called ZX0
Hello guys, I need a help.
Read documentation on github about decompressing data that overlaps source.
Packed screen is 4253 bytes with 3 bytes delta.
When I'm trying to unpack 6912 bytes into address 16384 all works like a charm, but when I'm changing destination to 49152 something goes wrong - rubbish is copied to the screen, and if I explore memory at 49152, picture is unpacked incorrectly.
Read documentation on github about decompressing data that overlaps source.
Packed screen is 4253 bytes with 3 bytes delta.
When I'm trying to unpack 6912 bytes into address 16384 all works like a charm, but when I'm changing destination to 49152 something goes wrong - rubbish is copied to the screen, and if I explore memory at 49152, picture is unpacked incorrectly.
Code: Select all
device ZXSPECTRUM48
MEM_END EQU 65535
displace EQU MEM_END - file_end + 1
org 50000
display "start ", /A, $
start:
ld sp, 24999
; fill screen with pattern
ld hl, 16384
ld de, 16385
ld bc, 6911
ld (hl), e
ldir
; move packed data and unpacker to end of memory
ld hl, file_end-1
ld de, MEM_END
ld bc, file_end - file_start
lddr
jp unpacker + displace
; data and code from this point will be moved
file_start:
incbin "Andy Green - Flashback (2018).scr_norm.zx0"
include "dzx0_standard.asm"
unpacker:
ld hl, file_start + displace
ld de, 16384
;ld de, 49152 ; <=== crash
push de
call dzx0_standard + displace
pop hl
; copy unpacked picture to screen
ld de, 16384
ld bc, 6912
ldir
; dead end to indicate all went ok
xor a
loop:
xor %111
out(254),a
jr loop
file_end:
savesna "test.sna", start
- Einar Saukas
- Bugaboo
- Posts: 3141
- Joined: Wed Nov 15, 2017 2:48 pm
Re: A new data compressor called ZX0
It seems you are compiling the decompressor at one address but executing it at another.
Re: A new data compressor called ZX0
Arrgh, silly meEinar Saukas wrote: ↑Tue Dec 21, 2021 6:56 pm It seems you are compiling the decompressor at one address but executing it at another.
Thank you!
P.S.
but wait... it does correctly extract to screen with same code (besides destination) !
- Einar Saukas
- Bugaboo
- Posts: 3141
- Joined: Wed Nov 15, 2017 2:48 pm
Re: A new data compressor called ZX0
When you moved the compiled code, the CALL instruction was still invoking a sub-routine at the original address. I'm guessing you did not override the old address when decompressing directly to screen, so it still worked (by accident).
However you erased the old sub-routine when you decompressed over it.
Re: A new data compressor called ZX0
Yes, you are right. Thank you again. I rewrote part of code with disp/ent, and it works now.Einar Saukas wrote: ↑Tue Dec 21, 2021 9:18 pm When you moved the compiled code, the CALL instruction was still invoking a sub-routine at the original address.
If anybody interested:
Code: Select all
device ZXSPECTRUM48
MEM_END EQU 65535
move_size EQU file_end - file_start
org 50000
start:
ld sp, 24999
; fill screen with pattern
ld hl, 16384
ld de, 16385
ld bc, 6911
ld (hl), e
ldir
; move packed data and unpacker to end of memory
ld hl, move_end-1
ld de, MEM_END
ld bc, move_size
lddr
jp unpacker
; ---------------------------
; data and code from this point will be moved on start
move_start:
; compile part of program to this ORG address
DISP 65535 - (move_size) +1
file_start:
incbin "Andy Green - Flashback (2018).scr_norm.zx0"
include "dzx0_standard.asm"
unpacker:
ld hl, file_start
;ld de, 16384
ld de, 49152 ; <=== crash
push de
call dzx0_standard
pop hl
; copy unpacked picture to screen
ld de, 16384
ld bc, 6912
ldir
; dead end to indicate all went ok
xor a
loop:
xor %111
out(254),a
ld b, a
pause:
djnz pause
jr loop
file_end:
ENT
move_end:
; ---------------------------
savesna "test.sna", start
Last edited by Bedazzle on Tue Dec 21, 2021 9:35 pm, edited 1 time in total.
- Einar Saukas
- Bugaboo
- Posts: 3141
- Joined: Wed Nov 15, 2017 2:48 pm
Re: A new data compressor called ZX0
Awesome! You are welcome
-
- Manic Miner
- Posts: 401
- Joined: Fri Jan 03, 2020 10:00 am
Re: A new data compressor called ZX0
ZXRar with 2-byte blocks TOTAL = 603460, including file headers (lost to exomizer and ZX0)Alone Coder wrote: ↑Wed Jan 20, 2021 5:31 pmTemporarily added 2-byte blocks in NedoOS version:Alone Coder wrote: ↑Wed Jan 20, 2021 4:02 pm calgary - 98571 (lost to Exomizer3)
canterbury - 27939
graphics - 163173 (better than others)
music - 63721
misc - 254202
RAR headers included.
Decompressor size is 342 bytes (uses 1566 bytes for temporary buffer).
TR-DOS version of ZXRar optionally supports compression with 2-byte blocks (helpful for code), with 386-byte decompressor. It has to be tested too.
calgary - 98525
canterbury - 27911
graphics - 163943
music - 62740
misc - 250341
mRIP archiver for PC by Eugene77 (2003 format, depacker size is 218 bytes, fits in 1-sector TR-DOS BASIC loader, working area of #5C2 bytes):
calgary - 91734 (better than exomizer etc)
canterbury - 25800 (better than exomizer etc)
graphics - 155697 (better than exomizer etc)
music - 60160 (lost to exomizer and ZX0)
misc - 245702 (lost to exomizer and ZX0)
TOTAL - 579093 (better than exomizer etc)
RIP archiver for PC by Eugene77 (2000 format, depacker size is 228 bytes, working area of #5C2 bytes):
calgary - 90794 (the best)
canterbury - 24658 (the best)
graphics - 150981 (the best)
music - 57276 (the best)
misc - 240765 (the best)
TOTAL - 565474 (the best)
So, the old native formats were not bad!
Re: A new data compressor called ZX0
I admit I have never had a reason to try ZX0 before. But now, I do: a relatively simple screen, with lots of empty space, which has absolutely no need to take up 6912 bytes. It requires compression.
I can't run it on my regular PC - it's incompatible with 64-bit Windows.
I tried it with DOSBox and it crashed - the window sits there doing nothing and can't be interrupted (and the .zx0 file never appears).
My 15-year-old laptop running Windows 7 returned the message "Program is too big to fit in memory", even though it's only 130-odd KB.
My 23-year-old laptop running Windows 2000 is flatly refusing to start up... even in Safe Mode.
Any suggestions?
I've already tried the screen compressor in BMP2SCR, and I've disassembled the initial machine code routine. For some reason that is several light years beyond my pay grade, the screen only displays if I load it from tape and RANDOMIZE USR 50000. If I copy the code into an .ASM file, along with lines and lines and lines of DEFBs to hold the screen data and assemble it with Spin, all I get is a screen full of flashing shash. The first lines are:
CALL 6034 (which is just a RET in the ROM and nothing more)
DEC SP
DEC SP
There is no reference whatsoever to the first line of the screen data - the compressor loads the code at 50000, and the screen data starts at 50114. Is it performing some mad trick with whatever the stack pointer points to at the end of a load?
I tried replacing these first lines with LD SP,50114 but it was to no available. As soon as any SP instructions are thrown my way I'm completely in the dark.
I've also taken to analysing the 6912 bytes, piece by piece, writing a load of LD HL,***** then LD (HL),*** and the odd LDIR routine to draw it effectively byte by byte, leaving out all the huge chunks of zeroes. Effectively I'm compressing the screen by hand, which is lengthy and tedious.
I can't run it on my regular PC - it's incompatible with 64-bit Windows.
I tried it with DOSBox and it crashed - the window sits there doing nothing and can't be interrupted (and the .zx0 file never appears).
My 15-year-old laptop running Windows 7 returned the message "Program is too big to fit in memory", even though it's only 130-odd KB.
My 23-year-old laptop running Windows 2000 is flatly refusing to start up... even in Safe Mode.
Any suggestions?
I've already tried the screen compressor in BMP2SCR, and I've disassembled the initial machine code routine. For some reason that is several light years beyond my pay grade, the screen only displays if I load it from tape and RANDOMIZE USR 50000. If I copy the code into an .ASM file, along with lines and lines and lines of DEFBs to hold the screen data and assemble it with Spin, all I get is a screen full of flashing shash. The first lines are:
CALL 6034 (which is just a RET in the ROM and nothing more)
DEC SP
DEC SP
There is no reference whatsoever to the first line of the screen data - the compressor loads the code at 50000, and the screen data starts at 50114. Is it performing some mad trick with whatever the stack pointer points to at the end of a load?
I tried replacing these first lines with LD SP,50114 but it was to no available. As soon as any SP instructions are thrown my way I'm completely in the dark.
I've also taken to analysing the 6912 bytes, piece by piece, writing a load of LD HL,***** then LD (HL),*** and the odd LDIR routine to draw it effectively byte by byte, leaving out all the huge chunks of zeroes. Effectively I'm compressing the screen by hand, which is lengthy and tedious.
Spectribution: Dr. Jim's Sinclair computing pages.
Features my own programs, modified type-ins, RZXs, character sets & UDGs, and QL type-ins... so far!
Features my own programs, modified type-ins, RZXs, character sets & UDGs, and QL type-ins... so far!
- Einar Saukas
- Bugaboo
- Posts: 3141
- Joined: Wed Nov 15, 2017 2:48 pm
Re: A new data compressor called ZX0
Check if you downloaded it correctly. You are supposed to click on the "Download" button.
Direct link:
https://github.com/einar-saukas/ZX0/raw ... in/zx0.exe
Re: A new data compressor called ZX0
...and that's working, on my main PC that outright rejected what I had yesterday. I was using the version found here:
https://github.com/einar-saukas/ZX0/tree/main/win
"Updated executables" seemed like the right place to go...
1,923 bytes! About 600 less than BMP2SCR required. I knew this would be the right option. Cheers!
https://github.com/einar-saukas/ZX0/tree/main/win
"Updated executables" seemed like the right place to go...
1,923 bytes! About 600 less than BMP2SCR required. I knew this would be the right option. Cheers!
Spectribution: Dr. Jim's Sinclair computing pages.
Features my own programs, modified type-ins, RZXs, character sets & UDGs, and QL type-ins... so far!
Features my own programs, modified type-ins, RZXs, character sets & UDGs, and QL type-ins... so far!
Re: A new data compressor called ZX0
An easily understood (though not hugely efficient) way to compress a screen that's got lots of empty character cells is to start with the attributes, and exploit the fact that no-one in their right mind would ever use the FLASH bit. So you scan your screen for empty character cells, and if you find one, set the FLASH bit of that attribute. Then you store your data as one byte of attribute, followed by eight bytes of pixel data, but you only store the pixel data if the FLASH bit is not set. Otherwise, you move straight on to the attribute for the next character cell.TMD2003 wrote: ↑Wed Sep 21, 2022 11:49 pm I've also taken to analysing the 6912 bytes, piece by piece, writing a load of LD HL,***** then LD (HL),*** and the odd LDIR routine to draw it effectively byte by byte, leaving out all the huge chunks of zeroes. Effectively I'm compressing the screen by hand, which is lengthy and tedious.
When you come to unpack that stream of data, you first take a byte, check the top bit, and if it's set, you erase that bit, write the byte to the attribute, then write 0s into the pixel bytes for that character cell. If the top bit isn't set, then copy it up as an attribute, then copy the next eight bytes into the pixel bytes of that character cell.
Of course your screen data can't get any lower than 768 bytes (768 empty characters), but it's a start.
And then there are far more complex approaches you can take once you get the hang of it! But however elaborate the mechanism, decoding is the same. Read some signal bits or bytes from the compressed data, and depending on what you just read, either copy some more raw data from the stream or do something special like pad with blanks or copy something from earlier, or from a prepared 'dictionary' of common byte sequences.
- Lethargeek
- Manic Miner
- Posts: 743
- Joined: Wed Dec 11, 2019 6:47 am
Re: A new data compressor called ZX0
...aaand again the example is that screen compressor of mine, LgK aka Lethargeek KompaktJoefish wrote: ↑Thu Sep 22, 2022 11:28 am An easily understood (though not hugely efficient) way to compress a screen that's got lots of empty character cells is to start with the attributes, and exploit the fact that no-one in their right mind would ever use the FLASH bit. So you scan your screen for empty character cells, and if you find one, set the FLASH bit of that attribute. Then you store your data as one byte of attribute, followed by eight bytes of pixel data, but you only store the pixel data if the FLASH bit is not set. Otherwise, you move straight on to the attribute for the next character cell.
When you come to unpack that stream of data, you first take a byte, check the top bit, and if it's set, you erase that bit, write the byte to the attribute, then write 0s into the pixel bytes for that character cell. If the top bit isn't set, then copy it up as an attribute, then copy the next eight bytes into the pixel bytes of that character cell.
Of course your screen data can't get any lower than 768 bytes (768 empty characters), but it's a start.
And then there are far more complex approaches you can take once you get the hang of it! But however elaborate the mechanism, decoding is the same. Read some signal bits or bytes from the compressed data, and depending on what you just read, either copy some more raw data from the stream or do something special like pad with blanks or copy something from earlier, or from a prepared 'dictionary' of common byte sequences.
first it filters the original screen to maximize the number of empty square chunks - 8x8 then 4x4 then 2x2
then the more clean filtered picture is compressed using the variable length prefix code
the idea is that squares generally have better correlation between all their pixels than byte strips
- Einar Saukas
- Bugaboo
- Posts: 3141
- Joined: Wed Nov 15, 2017 2:48 pm
Re: A new data compressor called ZX0
In this page, clicking on a filename will take you to another page, not directly to the actual file. So if you right clicked and saved it, you actually saved a webpage, not an executable file.TMD2003 wrote: ↑Thu Sep 22, 2022 9:25 amI was using the version found here:
https://github.com/einar-saukas/ZX0/tree/main/win
"Updated executables" seemed like the right place to go...
To get the actual file, first you need to left click on the file you want. Then you will see a button called "View raw", that you can use to download it.
The Github interface can be quite confusing, that's not my fault!
Re: A new data compressor called ZX0
I've always found it unnecessarily annoying, especially since a fair few projects of this type are hosted there. Sourceforge is a lot easier to navigate.Einar Saukas wrote: ↑Fri Sep 23, 2022 2:33 pm The Github interface can be quite confusing, that's not my fault!
Anyway, the actual ZX0 has done some sterling work - four blocks of code (mostly screens and attributes) compressed to about a quarter of their size, maybe less... leaving me more room for a 12K wall of text. Of course, I have to fit the actual game in there somewhere...
Also, just a thought: how much stack space does ZX0 need when decompressing a full screen? Do tell me it's not 7K, otherwise I'll be in trouble. Does it decode a few bytes at a time and write them to their destination?
Spectribution: Dr. Jim's Sinclair computing pages.
Features my own programs, modified type-ins, RZXs, character sets & UDGs, and QL type-ins... so far!
Features my own programs, modified type-ins, RZXs, character sets & UDGs, and QL type-ins... so far!
- Einar Saukas
- Bugaboo
- Posts: 3141
- Joined: Wed Nov 15, 2017 2:48 pm
Re: A new data compressor called ZX0
The standard version of the decompressor uses 4 bytes of stack.
The other 3 versions use 2 bytes of stack.
Each decompressed byte is directly written to the destination.
Re: A new data compressor called ZX0
No problems there, then. I can afford four bytes, come what may.
Spectribution: Dr. Jim's Sinclair computing pages.
Features my own programs, modified type-ins, RZXs, character sets & UDGs, and QL type-ins... so far!
Features my own programs, modified type-ins, RZXs, character sets & UDGs, and QL type-ins... so far!
Re: A new data compressor called ZX0
Hi all, I am new in this forum, not (yet) a ZX Spectrum, but at least a Z80 guy, active on some other platforms like MSX, Enterprise and Amstrad CPC (my childhood system). Some weeks ago I discovered this fantastic ZX0 compressor and was so amazed, that I decided to fully integrate it into my project called "SymbOS", an operating system for Z80 systems (unfortunately currently not for the Spectrum).
Thanks so much to Einar Saukas for this fantastic piece of software!!
The next SymbOS version supports loading of ZX0 compressed executables (EXE, COM etc.) but also provides functions for applications for decompressing data. You can load compressed data from an opened file in a transparent way like if it would be uncompressed, which makes it very easy to update even existing applications for handling compressed data.
I added a little header to all kind of ZX0 compressed data...
- 1W: length of the compressed data
- 4B: last 4 bytes of uncompressed data
- 1W: number of bytes at the beginning, which should be skipped/shouldn't be compressed
The first word is necessary for the loading routine, which only knows the uncompressed size from the application.
The next 4 bytes are the last 4 bytes of the uncompressed data, which won't be included in the compressed ones. This makes it possible to load the compressed data exactly at the end of the area, where it will be decompressed. I never saw a delta>3, so to be safe I choosed to save 4 bytes, so there will never be a problem with the delta.
The last word is necessary for files, which have some meta-data at the beginning. These shouldn't be compressed, as you may want to load it separately without the need to decompress the whole thing. Therefore the prefix feature in ZX0 is a great thing for still using this part as dictionary.
Now all EXE (GUI executables) and COM (command line executables) files can be compressed, as well as some special executables "*.WDG" (desktop widgets) and "*.SAV" (screen saver).
Picture files (SGX) can be compressed, too, including desktop background pictures, as well as help files (HLP).
Last but not least I decided to add a header for compressed music files, which are supported in the SymAmp music player. These are PT3, ST2 (Amstrad Soundtrakker 128), SKM (Amstrad Starkos Tracker) and SA2 (Adlib Tracker 2), which can all be ZX0-compressed now.
The compression ratio is just fantastic, in most cases it's better than ZIP! Now it's possible again to place the whole operating system with all system apps, background picture and additional stuff which is loaded during booting on one Amstrad standard disc side (178K).
I am using the Turbo decompressor, which is still small and has a great speed. It's directly integrated into the SymbOS kernel, which makes it possible to decompress data with the full linear size of up to 63K!
I hope to be able to release the next SymbOS version later this year. I know that currently all this is not interesting for you Spectrum guys, but I just wanted to share this info, as I am using this awesome software now. Maybe/hopefully there will be a Spectrum Next version in the future.
Again a lot of thank you and respect to Einar Saukas (for the whole thing) and Introspec (for the Turbo decompressor)!
Thanks so much to Einar Saukas for this fantastic piece of software!!
The next SymbOS version supports loading of ZX0 compressed executables (EXE, COM etc.) but also provides functions for applications for decompressing data. You can load compressed data from an opened file in a transparent way like if it would be uncompressed, which makes it very easy to update even existing applications for handling compressed data.
I added a little header to all kind of ZX0 compressed data...
- 1W: length of the compressed data
- 4B: last 4 bytes of uncompressed data
- 1W: number of bytes at the beginning, which should be skipped/shouldn't be compressed
The first word is necessary for the loading routine, which only knows the uncompressed size from the application.
The next 4 bytes are the last 4 bytes of the uncompressed data, which won't be included in the compressed ones. This makes it possible to load the compressed data exactly at the end of the area, where it will be decompressed. I never saw a delta>3, so to be safe I choosed to save 4 bytes, so there will never be a problem with the delta.
The last word is necessary for files, which have some meta-data at the beginning. These shouldn't be compressed, as you may want to load it separately without the need to decompress the whole thing. Therefore the prefix feature in ZX0 is a great thing for still using this part as dictionary.
Now all EXE (GUI executables) and COM (command line executables) files can be compressed, as well as some special executables "*.WDG" (desktop widgets) and "*.SAV" (screen saver).
Picture files (SGX) can be compressed, too, including desktop background pictures, as well as help files (HLP).
Last but not least I decided to add a header for compressed music files, which are supported in the SymAmp music player. These are PT3, ST2 (Amstrad Soundtrakker 128), SKM (Amstrad Starkos Tracker) and SA2 (Adlib Tracker 2), which can all be ZX0-compressed now.
The compression ratio is just fantastic, in most cases it's better than ZIP! Now it's possible again to place the whole operating system with all system apps, background picture and additional stuff which is loaded during booting on one Amstrad standard disc side (178K).
I am using the Turbo decompressor, which is still small and has a great speed. It's directly integrated into the SymbOS kernel, which makes it possible to decompress data with the full linear size of up to 63K!
I hope to be able to release the next SymbOS version later this year. I know that currently all this is not interesting for you Spectrum guys, but I just wanted to share this info, as I am using this awesome software now. Maybe/hopefully there will be a Spectrum Next version in the future.
Again a lot of thank you and respect to Einar Saukas (for the whole thing) and Introspec (for the Turbo decompressor)!
- Einar Saukas
- Bugaboo
- Posts: 3141
- Joined: Wed Nov 15, 2017 2:48 pm
Re: A new data compressor called ZX0
That's awesome! Thank you!!!
Now here's an idea. Your current format is as follows:
- 1W: total length of data after decompression
- last 4 bytes of uncompressed data
- 1W: number of uncompressed bytes at the beginning
- N uncompressed bytes at the beginning
- remaining compressed bytes
My suggestion is to use a simpler header like this:
- 1W: total length of data after decompression
- 1W: number of uncompressed bytes at the beginning (at least 4)
- N uncompressed bytes at the beginning (N>=4)
- remaining bytes, compressed backwards
This way, you will only need N uncompressed bytes total, instead of N+4.
To decompress, you copy into memory exactly N-4 uncompressed bytes, immediatelly followed by the compressed block. Afterwards decompress it backwards from the top of memory area. Finally copy the remaining 4 uncompressed bytes.
Compressing backwards may sound awkward, but you should get a slightly better compression in almost every file...
Now here's an idea. Your current format is as follows:
- 1W: total length of data after decompression
- last 4 bytes of uncompressed data
- 1W: number of uncompressed bytes at the beginning
- N uncompressed bytes at the beginning
- remaining compressed bytes
My suggestion is to use a simpler header like this:
- 1W: total length of data after decompression
- 1W: number of uncompressed bytes at the beginning (at least 4)
- N uncompressed bytes at the beginning (N>=4)
- remaining bytes, compressed backwards
This way, you will only need N uncompressed bytes total, instead of N+4.
To decompress, you copy into memory exactly N-4 uncompressed bytes, immediatelly followed by the compressed block. Afterwards decompress it backwards from the top of memory area. Finally copy the remaining 4 uncompressed bytes.
Compressing backwards may sound awkward, but you should get a slightly better compression in almost every file...
Re: A new data compressor called ZX0
Hi Einar, thanks a lot for your answer and your ideas!
Using backwards compression to have the "4 bytes" (for avoiding the delta problem) at the beginning in the uncompressed area is a good idea!
I made some quick and dirty tests with everal SymbOS EXE files, and the ratio with backward compression was always a little bit worse compared to normal compression. Then I remembered, that EXE files have the relocator table at the end, which may be a bad start for the dictionary. So I did the same with bitmap files, but here there is the same result, all backward compressed files are a little bit larger than normal compressed ones.
I don't really get this, as backward or forward is just swapping LDIR and LDDR, and why should the dictionary be somehow more optimal, when you start from the beginning or from the end.
Anyway I will make some more tests, especially I like to find out, if your suggestion will save some code on OS side, which is always a very good thing
Using backwards compression to have the "4 bytes" (for avoiding the delta problem) at the beginning in the uncompressed area is a good idea!
Is it really like this?Einar Saukas wrote: ↑Wed Jun 07, 2023 2:42 am Compressing backwards may sound awkward, but you should get a slightly better compression in almost every file...
I made some quick and dirty tests with everal SymbOS EXE files, and the ratio with backward compression was always a little bit worse compared to normal compression. Then I remembered, that EXE files have the relocator table at the end, which may be a bad start for the dictionary. So I did the same with bitmap files, but here there is the same result, all backward compressed files are a little bit larger than normal compressed ones.
I don't really get this, as backward or forward is just swapping LDIR and LDDR, and why should the dictionary be somehow more optimal, when you start from the beginning or from the end.
Anyway I will make some more tests, especially I like to find out, if your suggestion will save some code on OS side, which is always a very good thing
- Einar Saukas
- Bugaboo
- Posts: 3141
- Joined: Wed Nov 15, 2017 2:48 pm
Re: A new data compressor called ZX0
Ops, I should have explained what I meant!
The compression itself shouldn't make much difference. Some files work slightly better with forward compression, others with backward compression.
However the new format I suggested (using backward compression) will allow compressing 4 more bytes per file.