A new data compressor called ZX0

[Einar Saukas]

Hi Einar, thanks for you help so far.

You are welcome!

I do wonder though if you have an example of bank switching Spectrum +3
I have attempted to use it but alas it just crashes my game

I don't. The best place to get any kind of ZX Basic support is the ZX Basic forum:

https://www.boriel.com/forum/forumdisplay.php?fid=11

Failing that any tips on using the +3 Disk for loading screens or text ?

It's too hard for anyone to give you advice to improve your program, when we know nothing about it!

Considering that ZX0 is working in your test but not your main program, it means your problems are related to ZX Basic, not ZX0. Therefore the ZX Basic forum is probably the best place to start.

By the way I loved the 128K Hobbit you did and sorry that there was a lot of not so happy souls out there.

Thanks! I didn't really give up on this project, there are still more things to be done (I never released the extended location descriptions) but I also have a few other pending projects that I want to finish first... also ZXDB takes a lot of my time nowadays

[Bedazzle]

You are using a very old version. Please try again using the latest ZX0 executables for Windows:

Silly me.
Replaced with recent exe, and errors gone.

Source data: 75*704 = 52 800 bytes
Original algorithm used kind of packing, where values were presented as 3 bit, and put together into one long bytestream 19 800 bytes long.
Packed each file separately: 10 411 bytes total
Packed with first level as prefix: 9 605 bytes + 704 prefix = 10 309 total.
Nice!
Now I'll test with different sizes of prefix, and different data as prefix to get better compression.

Thank you for great tool!

[Randomiserusr]

Hi Einar, thanks for you help so far.

You are welcome!

I do wonder though if you have an example of bank switching Spectrum +3
I have attempted to use it but alas it just crashes my game

I don't. The best place to get any kind of ZX Basic support is the ZX Basic forum:

https://www.boriel.com/forum/forumdisplay.php?fid=11

Failing that any tips on using the +3 Disk for loading screens or text ?

It's too hard for anyone to give you advice to improve your program, when we know nothing about it!

Considering that ZX0 is working in your test but not your main program, it means your problems are related to ZX Basic, not ZX0. Therefore the ZX Basic forum is probably the best place to start.

By the way I loved the 128K Hobbit you did and sorry that there was a lot of not so happy souls out there.

Thanks! I didn't really give up on this project, there are still more things to be done (I never released the extended location descriptions) but I also have a few other pending projects that I want to finish first... also ZXDB takes a lot of my time nowadays

Well thank you again for replying much appreciated.
Yes I guess what I am asking is a bit tricky to provide help.
I have asked on Boriels site but nothing has come of it.

I am very interested in what else you have done on the Hobbit and any chance of a preview?
Thanks again
Ken

[Alessandro]

I do wonder though if you have an example of bank switching Spectrum +3

What exactly would you want to do, if I may ask?

If you are trying to switch between memory banks on the 128K and later models through ZX Basic, forget it; ZX Basic does not allow for that by itself. You can always divide your program into separate sub-programs, copy variables and heap in an unused region of the RAM and recall them when you switch from one bank to another (see my Ad Lunam Plus for a practical application of this method, source files are included).

[Randomiserusr]

I do wonder though if you have an example of bank switching Spectrum +3

What exactly would you want to do, if I may ask?

If you are trying to switch between memory banks on the 128K and later models through ZX Basic, forget it; ZX Basic does not allow for that by itself. You can always divide your program into separate sub-programs, copy variables and heap in an unused region of the RAM and recall them when you switch from one bank to another (see my Ad Lunam Plus for a practical application of this method, source files are included).

Hi Alessandro

In ZX Basic (Boriel) I had two methods called :-
These were then called before say loading in a new screen into the screen ram 16384
So clearly I have done something that causes the game to crash when I call setmemorypage7()

Thanks for the other tip - I will check this out too.

Code: Select all

'sub FASTCALL setmemorypage7()
'ASM
'	LD      A,(0x5b5c)      
' 	AND     0xf8
'	OR      7               ; Bank 7
'	LD      BC,0x7ffd
'	DI
'	LD      (0x5b5c),A
'	OUT     (C),A
'	EI
'END ASM
'end sub

'sub FASTCALL setmemorypage5()
'ASM
'	LD      A,(0x5b5c)      ;Previous value of port
'	AND     0xf8
'	OR      5             ;Select bank 5
'	LD      BC,0x7ffd
'	DI
'	LD      (0x5b5c),A
'	OUT     (C),A
'	EI
'END ASM
'end sub

[Einar Saukas]

So clearly I have done something that causes the game to crash when I call setmemorypage7()

Are you doing it in the same program that occupies almost the entire memory?

This is what you wrote before:

ORG starts at 25085 the code ends at F0FA (61,690)

If it's the same program, then switching banks will make almost half your code to disappear, replacing it with an empty memory area.

[Alessandro]

Like Einar said. Each time you switch between memory banks, anything in the RAM from 49152 on will be erased. A solution to this would be, again, to divide the program in several blocks, which are then stored in the various memory banks, preferably compressed in order to occupy less space and decompressed when needed.

[Rhino]

Hello,
Congratulations to the authors for the great work!
I'm new here and I just registered to suggest two possible optimizations for the standard decompression code:

1) Replacing the dzx0s_elias subroutine with this:

Code: Select all

dzx0s_elias_backtrack:
        add     a, a
        rl      c
        rl      b
dzx0s_elias:
        add     a, a
        jr      nz, dzx0s_elias_skip
        ld      a, (hl)                 ; load another group of 8 bits
        inc     hl
        rla
dzx0s_elias_skip:
        jr	nc, dzx0s_elias_backtrack
	ret

And including an "inc c" just before each call to dzx0s_elias. According to my tests this can be about 2% faster at the cost of +2 bytes of code.

2) The second would be to replace "jr dzx0s_copy" with "jp (iy)", initializing iy with dzx0s_copy at the start of the code. This can be +0.5% faster at the cost of +3 bytes.
Both changes can be +2.5% faster at the cost of +5 bytes of code but those figures are probably not very reliable as I have done very little tests, so, maybe the author or any interested would like to test these changes better.

Thanks!

[Einar Saukas]

Congratulations to the authors for the great work!

Thank you very much!

I'm new here and I just registered to suggest two possible optimizations for the standard decompression code:

Thanks for your suggestions. However the "standard" decompressor was implemented on purpose to be as small as possible. This is the reason it uses "jr dzx0s_copy" instead of "jp dzx0s_copy", for instance. It's optimized for size instead of speed.

Speed optimizations were applied to the other ZX0 decompressors, That's the reason we provide several versions, each one with a different trade-off. In particular, your optimization ideas are very good but they were already used in these other decompressors, as follows:

1) Replacing the dzx0s_elias subroutine with this:
[...]
And including an "inc c" just before each call to dzx0s_elias.

The "turbo" decompressor includes "inc c" in a few places in the main routine, to ensure you will always have BC=1 before calling sub-routine "dzx0t_elias". It's the same idea.

2) The second would be to replace "jr dzx0s_copy" with "jp (iy)", initializing iy with dzx0s_copy at the start of the code.

The "fast" decompressor initializes IX with address of routine CopyMatch1 at the start of the code, so it can avoid using "jp CopyMatch1". It's a similar idea.

It's not easy to find optimizations that we didn't use already, since we have invested a lot of time in these routines. But please don't give up trying, you have good ideas!

[Rhino]

Ok, don't worry.
In any case, I hope my previous post can be useful to anyone who wants to gain +2% speed at the cost of only 2 bytes more of code, since I think it can be a good thing for most practical cases : )

Thanks and keep up the good work!

[Rhino]

Btw, I am looking at the "turbo" version and I see that there are many jp cc,x and one jp instructions at near addresses that could be replaced by jr in order to not only be the code smaller, but also faster since jr cc,x saves a cycle for each false condition.

Regards

[introspec]

Btw, I am looking at the "turbo" version and I see that there are many jp cc,x and one jp instructions at near addresses that could be replaced by jr in order to not only be the code smaller, but also faster since jr cc,x saves a cycle for each false condition.

No, it is not quite as simple as you think. Replacing JP by JR would only save cycles if probability of not jumping is at least about the same as probability of jumping. Let us get properly mathematical about this. Suppose you have
JP NZ,..
It would always be executed in 10 t-states. Consider replacing it by
JR NZ,..
and suppose that flag Z happens with probability X, and flag NZ happens with probability 1-X. Then the average execution time for this jump would be 7*X+12*(1-X)=12-5*X
If X is 0, this executes 12 t-states, if X is 1 this executes 7 t-states. The average execution would match 10t of JP when X=0.4. If you read the logic of turbo decompressor you would see that the probability of NZ there is actually 7/8, so X = 0.125 (1/8). Hence, the average execution time of JR in there would be 12-5*0.125 ~ 11.375t, which is quite a bit slower than using JP.

[Rhino]

No, it is not quite as simple as you think. Replacing JP by JR would only save cycles if probability of not jumping is at least about the same as probability of jumping. Let us get properly mathematical about this. Suppose you have
JP NZ,..
It would always be executed in 10 t-states. Consider replacing it by
JR NZ,..
and suppose that flag Z happens with probability X, and flag NZ happens with probability 1-X. Then the average execution time for this jump would be 7*X+12*(1-X)=12-5*X
If X is 0, this executes 12 t-states, if X is 1 this executes 7 t-states. The average execution would match 10t of JP when X=0.4. If you read the logic of turbo decompressor you would see that the probability of NZ there is actually 7/8, so X = 0.125 (1/8). Hence, the average execution time of JR in there would be 12-5*0.125 ~ 11.375t, which is quite a bit slower than using JP.

Thanks for the answer.
I forgot to mention that I am an Amstrad CPC user, where the memory snowing effect makes jp not faster than jr in any case, so that "jp" turbo version is not the most optimal for CPC.
Maybe it would be worthwhile to specify the platform from a constant in the code so that it fits better in each case?
Regards

[dfzx]

No, it is not quite as simple as you think. Replacing JP by JR would only save cycles if probability of not jumping is at least about the same as probability of jumping. Let us get properly mathematical about this. Suppose you have
JP NZ,..
It would always be executed in 10 t-states. Consider replacing it by
JR NZ,..
and suppose that flag Z happens with probability X, and flag NZ happens with probability 1-X. Then the average execution time for this jump would be 7*X+12*(1-X)=12-5*X
If X is 0, this executes 12 t-states, if X is 1 this executes 7 t-states. The average execution would match 10t of JP when X=0.4. If you read the logic of turbo decompressor you would see that the probability of NZ there is actually 7/8, so X = 0.125 (1/8). Hence, the average execution time of JR in there would be 12-5*0.125 ~ 11.375t, which is quite a bit slower than using JP.

What a brilliant explanation!

[Sol_HSA]

I forgot to mention that I am an Amstrad CPC user, where the memory snowing effect makes jp not faster than jr in any case, so that "jp" turbo version is not the most optimal for CPC.

That sounds weird. Do you have a reference link about that? I'm just curious.

[Rhino]

That sounds weird. Do you have a reference link about that? I'm just curious.

Hi! You can google amstrad cpc z80 timing for more info.

[AndyC]

I forgot to mention that I am an Amstrad CPC user, where the memory snowing effect makes jp not faster than jr in any case, so that "jp" turbo version is not the most optimal for CPC.

That sounds weird. Do you have a reference link about that? I'm just curious.

Roughly speaking, the gate array in the CPC stalls the CPU periodically which has the approximate effect of rounding instructions up to the nearest 4 clock cycles (ish, it's a bit more complex for longer instructions). It's like contention on the Speccy but much easier to predict, since you mostly just use a lookup of instructions against a slightly different timing length (commonly referred to as NOP timing since everything is a multiple of 1 NOP)

[Einar Saukas]

I forgot to mention that I am an Amstrad CPC user, where the memory snowing effect makes jp not faster than jr in any case, so that "jp" turbo version is not the most optimal for CPC.
Maybe it would be worthwhile to specify the platform from a constant in the code so that it fits better in each case?

It makes sense to have Amstrad CPC specific optimizations.

However I prefer to avoid using IFDEF otherwise the code won't be fully portable anymore. Different assemblers require a different syntax for this. I think it's important to keep the code as simple as possible, so it will work out of the box for everyone. Many people give up if they need to spend time figuring out how to make something work.

Instead, why don't you create your own source code repository with the Amstrad CPC version? You can easily create an account on github, for instance. If you do it, I will add a link to your version from the ZX0 official page.

[Sol_HSA]

Roughly speaking, the gate array in the CPC stalls the CPU periodically which has the approximate effect of rounding instructions up to the nearest 4 clock cycles (ish, it's a bit more complex for longer instructions).

Ah, that makes sense. I started wondering how just the jumps would be affected.

[Rhino]

It makes sense to have Amstrad CPC specific optimizations.

However I prefer to avoid using IFDEF otherwise the code won't be fully portable anymore. Different assemblers require a different syntax for this. I think it's important to keep the code as simple as possible, so it will work out of the box for everyone. Many people give up if they need to spend time figuring out how to make something work.

Instead, why don't you create your own source code repository with the Amstrad CPC version? You can easily create an account on github, for instance. If you do it, I will add a link to your version from the ZX0 official page.

Ok, I will let you know if I do it.
Btw, I was someone who had to make changes to the code to make it work because the "$" symbol for hexadecimal numbers is not compatible with many z80 assemblers as are most of those used in CPC. You can use decimal numbers instead to reduce the chances of error.

[Bedazzle]

However I prefer to avoid using IFDEF otherwise the code won't be fully portable anymore. Different assemblers require a different syntax for this. I think it's important to keep the code as simple as possible, so it will work out of the box for everyone. Many people give up if they need to spend time figuring out how to make something work.

Why not to mention possible Amstrad optimization at the top of source file in comment?

[introspec]

I forgot to mention that I am an Amstrad CPC user, where the memory snowing effect makes jp not faster than jr in any case, so that "jp" turbo version is not the most optimal for CPC.
Maybe it would be worthwhile to specify the platform from a constant in the code so that it fits better in each case?

Ah, so you must be Rhino^BG? I now understand where you coming from (I've heard CPC people measuring execution time in NOPs, but never paid enough attention as to why). But it is an interesting question. The thing is, both turbo and fast decompressors are heavily optimized for speed under the assumption of standard Z80 timings. I run tests in a specially instrumented emulator and I can see differences in execution times even on the order of single t-states. Almost every structural decision in the decompressor was made with consideration of the resulting efficiency on several "average" files, which I selected from a corpus of mine. Thus, given how different your Z80 timings are, I think you would benefit from re-optimizing everything fully, not just patching a decompressor here and there.

I am not volunteering to do this because my work on decompressors is progressing very slowly at present, due to my other responsibilities. In addition, I do not have an emulator with CPC timings, and I do not have much immediate intuition with regard of what would and what would not well. This is probably a kind of the project that should be undertaken by someone with CPC background, like yourself.

[Einar Saukas]

Why not to mention possible Amstrad optimization at the top of source file in comment?

If you were an Amstrad CPC user, what would you prefer?

"Download this version instead"

or

"Edit the source code to make the following changes: replace every occurrence of..."

[Bedazzle]

If you were an Amstrad CPC user, what would you prefer?

I hardly imagine user that uses sources

And I'm not sure that different sources of data will be in sync when updated from time to time.

[theNestruo]

If you were an Amstrad CPC user, what would you prefer?

Hi!
Not sure if my opinion is relevant (I'm not an Amstrad CPC user but an MSX user, and I'm currently more interested in size so I'm using the "Standard" version), but I would expect it to be in a separate file (e.g.: "dzx0_mega_cpc.asm" instead of "dzx0_mega.asm").