A text compressor/decompressor for Z80

[rastersoft]

Hi all!

I'm very happy to announce that I have extracted the text compressor/decompressor used in my game Escape from M.O.N.J.A.S. It is available, with instructions, at https://gitlab.com/rastersoft/tcz0 It's called TCZ0 (Text Compressor for Z80).

[Sol_HSA]

Interesting use case. I'm pretty sure the compression could be faster, though =)

Not having looked at the source code, I'd probably try to find a good order by trying to first compress each pair, and then pair those with the rest, etc. That should(tm) lead to a decent compression, if not the absolute optimal one, pretty quickly.

[Alessandro]

This sounds very interesting, but how do you actually compress files? There is no executable, and I do not know what to do with the non-Z80 source files.

[Sol_HSA]

The description talks about python, so I guess the compressor is one or more python scripts.

[Alessandro]

The description talks about python, so I guess the compressor is one or more python scripts.

Which I do not know how to use do they have to be assembled with some external program? Sorry but to me a python is just a kind of snake...

[Sol_HSA]

The description talks about python, so I guess the compressor is one or more python scripts.

Which I do not know how to use do they have to be assembled with some external program? Sorry but to me a python is just a kind of snake...

You just install python (or if you're on Linux you may already have it) and then you just run the python scripts. No compiling needed. As for python language, I do recommend learning it; it's fun. I remind myself that it's fun every now and then, do something with it and then forget about it for some time again..

If you're a next user and want to use my nextsync, the server is written in python. If you want to compile my speccy projects, the build scripts are in python. Doesn't matter which OS you use, the python scripts run; you just need to install python.

For said build scripts, I started them as batch files calling the compiler, then changed that to python and kinda accidentally made it build concurrently after automatically finding the changed files, recursively checking any included files... like I said, fun.

[rastersoft]

Interesting use case. I'm pretty sure the compression could be faster, though =)

Not having looked at the source code, I'd probably try to find a good order by trying to first compress each pair, and then pair those with the rest, etc. That should(tm) lead to a decent compression, if not the absolute optimal one, pretty quickly.

That sounds like combinatorial explosion Escape from M.O.N.J.A.S. has 289 sentences/text blocks, which means that just doing your first part will mean 83232 combinations. And having there the best combination doesn't mean it is the global optimum, because you can be compressing a piece in the second sentence that, if left uncompressed, will lead to a more repeated block when combined with some letters before or after.

[rastersoft]

The description talks about python, so I guess the compressor is one or more python scripts.

Which I do not know how to use do they have to be assembled with some external program? Sorry but to me a python is just a kind of snake...

You only have to go to https://www.python.org and download the installer. Then, any file ended with .py will be run by the interpreter. But I recommend you to also install pypy (https://www.pypy.org/), which is a different interpreter for the same language, but it is EXTREMELY FASTER than the official interpreter (the inconvenient is that some modules, like NUMPY, aren't compatible with it, but this compressor works with pypy, so I recommend to use it).

And I also encourage you to learn python. When I learnt it, I literally fell in love (and I was 30 years old... Do you really think that is it possible to fall in love with a programming language at that age, unless it is an INSANELY GREAT language? ) It is designed to be easy to learn and use, flexible, and what is more important: "batteries are included", which means that you have modules for nearly everything: create window applications? check. Complex mathematical functions, including fast fourier transform, matrices and more? check. Neural networks? check. HTTP clients and servers? of course. Asynchronous functions? Yep. Telegram bots? You name it...

[Sol_HSA]

That sounds like combinatorial explosion Escape from M.O.N.J.A.S. has 289 sentences/text blocks, which means that just doing your first part will mean 83232 combinations.

83232 combinations means nothing on modern computers, even running python. And these are tiny iterations as they only consider pairs of text blocks. Every subsequent pass is 1/4 steps from the previous one, so there's no explosion: 83521+21025+5329...+1 = 111744 steps, which still is nothing on modern computers.

And having there the best combination doesn't mean it is the global optimum, because you can be compressing a piece in the second sentence that, if left uncompressed, will lead to a more repeated block when combined with some letters before or after.

I know it's probably not a global optimum, but if nothing else, it should run in a sane time period and probably give a relatively good result. I'm not going to force you to do anything; this just an idea.

[Alessandro]

Thanks for all your help, folks

Honestly, I don't think I have enough time and patience to learn a programming language these days. But you might never know

[rastersoft]

That sounds like combinatorial explosion Escape from M.O.N.J.A.S. has 289 sentences/text blocks, which means that just doing your first part will mean 83232 combinations.

83232 combinations means nothing on modern computers, even running python. And these are tiny iterations as they only consider pairs of text blocks. Every subsequent pass is 1/4 steps from the previous one, so there's no explosion: 83521+21025+5329...+1 = 111744 steps, which still is nothing on modern computers.

Of course 83232 combinations aren't a big number, but the point is that it's only the first step, and the number already grew by a lot. Also, I don't see why this is better, because having two sentences ordered in the best way locally doesn't guarantee that it is a big optimum. In fact, I can think of a lot of cases where this fails.

I think that the best way is to optimize the compression of each individual block by ensuring the best combination of compressed subblocks, instead of always prioritizing the biggest one, as is done currently. That, and take advantage of unused ASCII characters to compress two-byte blocks.

Anyway, in only five-ten minutes running you achieve a very good compression level. Letting go for eight hours is only when you want to squeeze up to the last byte.

And having there the best combination doesn't mean it is the global optimum, because you can be compressing a piece in the second sentence that, if left uncompressed, will lead to a more repeated block when combined with some letters before or after.

I know it's probably not a global optimum, but if nothing else, it should run in a sane time period and probably give a relatively good result. I'm not going to force you to do anything; this just an idea.

And I thanks any idea, of course. And maybe can be used to speed up the initial setup. But I still think that testing new combinations will be a must anyway.

[rastersoft]

Thanks for all your help, folks

Honestly, I don't think I have enough time and patience to learn a programming language these days. But you might never know

Don't forget that you don't need to know python to use the compressor, anyway. Just install the interpreter and run the 'compress_sentences.py' script passing as first parameter the .asm file with the sentences, and as second parameter the output file name.

[evilpaul]

You only have to go to https://www.python.org and download the installer. Then, any file ended with .py will be run by the interpreter. But I recommend you to also install pypy (https://www.pypy.org/), which is a different interpreter for the same language, but it is EXTREMELY FASTER than the official interpreter (the inconvenient is that some modules, like NUMPY, aren't compatible with it, but this compressor works with pypy, so I recommend to use it).

Huh.. interesting! I've never used PyPy before, so I gave it a try (well, PyPy3 to be specific) on my Python port of ZX0 and I'm super impressed! In a single, particularly challenging benchmark it took 5% of the time that Python3 takes. Yay! Win!

Only problem now is that it has issues with the PIL library that I use in some of my tools.. but I imagine that's easy* to sort out

* PIL was a right pain in the ass to install for Python3, so I expect it will be another pain in the ass to get it working with PyPy.. oh well

[Bedazzle]

* PIL was a right pain in the ass to install for Python3, so I expect it will be another pain in the ass to get it working with PyPy.. oh well

Are you under Win or Nix?
I had no problem installing under Win 7 and Win 10
https://pillow.readthedocs.io/en/stable/

[evilpaul]

Are you under Win or Nix?
I had no problem installing under Win 7 and Win 10
https://pillow.readthedocs.io/en/stable/

macOS

[evilpaul]

Are you under Win or Nix?
I had no problem installing under Win 7 and Win 10
https://pillow.readthedocs.io/en/stable/

macOS

Ah.. got it working now. Wasn't too hard. And.. yay.. the full "clean" build time for my project dropped form 31 seconds to 3

[dobbo2k]

Hi rastersoft, this looks great and is exactly what I'm looking for!

However, I know nothing about assembly language. I'm writing a text adventure in C (z88dk toolkit). How would I integrate this asm code so I could call the decompression function on a given sentence within C? Any help would be appreciated.

Thanks in advance.

[rastersoft]

Hi rastersoft, this looks great and is exactly what I'm looking for!

However, I know nothing about assembly language. I'm writing a text adventure in C (z88dk toolkit). How would I integrate this asm code so I could call the decompression function on a given sentence within C? Any help would be appreciated.

That's such a good point that I uploaded just now all the changes needed to also generate C code. Now you have a C function for decompression, and .C and .H files with the compressed data and some #defines to each sentence.

[dobbo2k]

Thank you!

[rastersoft]

Thank you!

(Of course, if you detect any bug, please, drop a comment here )