ZX Recoloring project

[Ralf]

I am not really familiar the details of with Spec256 approach. I mean, I know that it emulates a sort of "shadow graphics processor", but I neither know the details nor can estimate how hard is to to something similar.

Furthermore, I am not entirely sure that the sprites are stored in the memory in the same format for all games. We know exactly that when a certain picture is displayed on the screen, it has a predefined pixel structure. When it sits in memory, it can be literally anything stored in the way the authors wanted. For some games with large sprites (like Popeye) I guess there are maybe no complete sprites at all, just separate sprite pieces later combined into a character on the fly. So in order to recolor such sprites one would need to guess these pieces and process them separately.

It looks like Spec256 website still exists You can read there a little about the used approach:
http://www.emulatronia.com/emusdaqui/sp ... ex-eng.htm

It was a closed project where authors modified themselves a few games but there wasn't open code or any editor available for everybody so he could recolour any game. So we may not know the details but they describe the general idea:

The idea is: in one hand we´ve got the speccy emulator with its memory zone, and in the other we emulate a Z80 which works with 64 bit registers instead of 8 bit, and with a memory map with positions of 64 bit instead of 8.

Each time a z80 instruction is emulated, the same instruction is simulated with data always supposed to be graphics. I´ve named this parallel processor Z80_GFX. Z80_GFX modifies its memory zone accordig to the instructions and doesn´t do anything with Z80´s memory zone. What have we archieved? A faithful emulation of Z80 and a memory zone from which we can obtain 256 colour graphics.

Each time they probably also used some custom hacks tailored for each game.

So that's Spec256. And we have also EmuZWin. I believe it uses exactly the same approach, just it enables the user to recolour any game himself.
And you are right, most time you work on chunks of graphics and not complete sprites. And you have to find these chunks yourself in a mess of pixels And it requires skill+intuition as these data may be stored in different format and you must reorganize the display to get it in the friendliest way. And yes, it's a pain

I believe you cannot avoid it, but you can improve the process with a good, friendly tool

[Ralf]

And for anybody interested, there is a recently active thread on a Russian forum about 256 colour games

Here's the English automatic translation:
https://translate.google.pl/translate?s ... ec256.html

You may find there quite a lot of recently recoloured games (with EmuZWin) and among others you may find there my almost completed (but having problems of course ) recoloured version of Renegade which I didn't announce anywhere else:

[ZXDunny]

The idea is: in one hand we´ve got the speccy emulator with its memory zone, and in the other we emulate a Z80 which works with 64 bit registers instead of 8 bit, and with a memory map with positions of 64 bit instead of 8.

Each time a z80 instruction is emulated, the same instruction is simulated with data always supposed to be graphics. I´ve named this parallel processor Z80_GFX. Z80_GFX modifies its memory zone accordig to the instructions and doesn´t do anything with Z80´s memory zone. What have we archieved? A faithful emulation of Z80 and a memory zone from which we can obtain 256 colour graphics.

Each time they probably also used some custom hacks tailored for each game.

No hacks needed for this particular approach. The 64bit z80 they talk about is misleading. You don't need to do that.

You have 8 "pages" for each RAM page. Each acts as a bitplane - the Speccy RAM contains the low bit of each bit that's read, and when you read a byte from RAM you read 8 bytes, one from each page and store them in 8 registers per speccy register. When performing memory writes, you do the same thing in reverse.

When it comes time to build the display for each frame, you take one bit for each pixel from those 8 banks of RAM and build a 256 colour pixel from those.

That's it - no hacks needed. The extra colours come from a tool which identifies sprites and graphics in RAM and allows you to colour them, eventually spitting out a "snapshot" that contains all 8 pages specially constructed from those 256 colour graphics.

You're right that it's all about the tool - if it's easy for the user to find and modify the graphics then the engine will take care of the rest.

It has limitations. Can't work on tape images. Can't recolour blank backgrounds (those need to be hacked in). Can't do 3D games.

But it's really the simplest one you can do.

[hikoki]

That Renegade remake looks good!!

About these tools, I don't know if it'd be possible a sort of rzx viewer where the user can record,pause,resume a gameplay. One could pause it and start clicking over the screen to assist the algorithm engine to build its configuration parameters about collision bounds, glitches, wrong matches, etc.

[ZXDunny]

Training a DeepMind instance to recognise sprites in a Speccy snapshot would be very, very doable. In fact, it would be an excellent solution to the problem where actually using your eyes to find them is just too hard for everyone.

[rg_software]

In fact, it would be an excellent solution to the problem where actually using your eyes to find them is just too hard for everyone.

It's funny you think exact matching is not a good solution whereas an approximate ML-based algorithm with no guarantees whatsoever and a theoretical possibility that your sprite will be substituted with something completely different is "excellent" If no one is able to recognize a sprite on the screen, by definition it's not important what's going on there!

Though I shouldn't really be chatting about these things, I'd better work on my current codebase

[obo]

Training a DeepMind instance to recognise sprites in a Speccy snapshot would be very, very doable. In fact, it would be an excellent solution to the problem where actually using your eyes to find them is just too hard for everyone.

You could probably combine the two, so ML recognises the sprites in the display image, and the emulation remembers the source of the data. It could have some built-in knowledge of sprite masks too. Playing the RZX would give it a good selection of what is needed for a game. You'd probably still want it to be a starting point, with a manual review over the results to fill gaps and correct false-positives.

Hook-all-the-things has been my approach so far, but it's a very manual process. Even given some tools like Spec256, there aren't many people willing to go through it.

[ZXDunny]

In fact, it would be an excellent solution to the problem where actually using your eyes to find them is just too hard for everyone.

It's funny you think exact matching is not a good solution whereas an approximate ML-based algorithm with no guarantees whatsoever and a theoretical possibility that your sprite will be substituted with something completely different is "excellent" If no one is able to recognize a sprite on the screen, by definition it's not important what's going on there!

Though I shouldn't really be chatting about these things, I'd better work on my current codebase

It's funny that you thought that this is what it would be used for. If you think that using an AI to detect sprites at runtime and replace them is at all less computationally expensive than your current method then I fear for any progress to be made...

To put it into terms you might better understand:

1. Nobody wants to do this because the tools suck
2. Someone creates a very, very problematic solution that involves image-recognition which will create a shed-load of problems for the vast majority of the games out there, because the tool is easy to use.
3. Users are really interested (for now!) but eventually get annoyed because the engine can't replace the graphics for very many games at all without horrific flickering.

How to fix? Well, we need a better option than image-recognition (and I've outlined a few that are better) BUT... the tools get more complex for the user. To create a recolouration, the user would either have to be able to find the graphics in RAM or Disassemble the game to find suitable hook addresses. While the latter is hard for non-coders, the former is easy with a simple graphics ripper. But it's not simple enough! Who wants to muck around looking for graphics and messing with addresses and modulus? Especially when this new tool lets you do it from screen grabs.

To address this insanely difficult task of finding graphics, we could indeed feed load of snapshots to an AI, tell it where the graphics are in those snapshots so it creates a neural net of ability to recognise sprite sequences.

Ta-da! After that initial training (which gets better the more you feed it), it can recognise the sprite addresses for you! And there's nothing to stop this from being an add-on to the usual RAM-search tool as a fallback.

I do hope you understand now.

Training a DeepMind instance to recognise sprites in a Speccy snapshot would be very, very doable. In fact, it would be an excellent solution to the problem where actually using your eyes to find them is just too hard for everyone.

You could probably combine the two, so ML recognises the sprites in the display image, and the emulation remembers the source of the data. It could have some built-in knowledge of sprite masks too. Playing the RZX would give it a good selection of what is needed for a game. You'd probably still want it to be a starting point, with a manual review over the results to fill gaps and correct false-positives.

Hook-all-the-things has been my approach so far, but it's a very manual process. Even given some tools like Spec256, there aren't many people willing to go through it.

"Hook everything" is the best way but it's not easy for users. It can recolour and make higher-res anything but because it's not identifying sprites from screen grabs it's unlikely to get any attention.

Personally, I'd go with both and a Python API built into the emulator.


But then, what do I know?

[rg_software]

If you think that using an AI to detect sprites at runtime and replace them is at all less computationally expensive than your current method

Well, as I said before, I shouldn't be really theorizing here. However, what do you mean by "AI"? Let's talk numbers. If I use Aho-Corasick method, the computational complexity is basically linear. You can't make it faster even in theory.

To address this insanely difficult task of finding graphics, we could indeed feed load of snapshots to an AI, tell it where the graphics are in those snapshots so it creates a neural net of ability to recognise sprite sequences.

Yes, I understand you want a silver bullet that would do the work instead -- let's call it "AI", "neural network" or "magic". I completely understand this wish, but I don't believe its doable. Actually, it is, let's call it "graphics filter" -- that's what we got already.

I do hope you understand now.

Yes, absolutely. I wish someone really goes through this process to evaluate its real complexity. I mean, if you/anyone can make it and measure how much time end effort it takes, demonstrate how it works and estimate how fast it is -- I will be the first person to applaude.

[ZXDunny]

If you think that using an AI to detect sprites at runtime and replace them is at all less computationally expensive than your current method

Well, as I said before, I shouldn't be really theorizing here. However, what do you mean by "AI"? Let's talk numbers. If I use Aho-Corasick method, the computational complexity is basically linear. You can't make it faster even in theory.

What has that got to do with anything?

To address this insanely difficult task of finding graphics, we could indeed feed load of snapshots to an AI, tell it where the graphics are in those snapshots so it creates a neural net of ability to recognise sprite sequences.

Yes, I understand you want a silver bullet that would do the work instead -- let's call it "AI", "neural network" or "magic". I completely understand this wish, but I don't believe its doable. Actually, it is, let's call it "graphics filter" -- that's what we got already.

Again, has nothing to do with what you're trying to achieve. We've already established that your approach will not work for the vast majority of games.

I do hope you understand now.

Yes, absolutely. I wish someone really goes through this process to evaluate its real complexity. I mean, if you/anyone can make it and measure how much time end effort it takes, demonstrate how it works and estimate how fast it is -- I will be the first person to applaude.

No, you don't understand. I'M NOT TALKING ABOUT USING AI OR ML AT 50 FRAMES PER SECOND.

Sheesh.

[hikoki]

I've made a quick search and there seem to be photo editors based on AI like pixelmator for Mac.
Another interesting experiment is this Neural photo editor on github, https://github.com/ajbrock/Neural-Photo-Editor
I guess you would have to fit your custom sprites to the original ones by selecting them manually on a rzx snapshot. Supervised machine learning I think it's called the jargon term. Looks like complicated!

[rg_software]

We've already established that your approach will not work for the vast majority of games.

Well, honestly speaking I think many of these issues still can be resolved with partial matching and other tricks, so I might be able to cover many harder cases, so it isn't done yet.

No, you don't understand. I'M NOT TALKING ABOUT USING AI OR ML AT 50 FRAMES PER SECOND.

Okay, I think I got it this time. Though, I still think it sounds very nice as a theoretical chit-chat, but can be much harder to achieve in practice. Let's say, I am not arguing, but I don't feel capable of taking this amount of work, at least, in near future.

[ZXDunny]

We've already established that your approach will not work for the vast majority of games.

Well, honestly speaking I think many of these issues still can be resolved with partial matching and other tricks, so I might be able to cover many harder cases, so it isn't done yet.

Yes, but even getting the more obvious XOR problems sorted, your algorithm will end up with so many bolted-on edge cases that it might not be fun to maintain. And even then, you're still restricting yourself to a very narrow corridor of the amount of games that people will want to convert.

I do understand that you want a method that presents the user with as easy an experience as you can, for recolouring games. But when the method used restricts people to only a handful of games that can run smoothly (with all the rest exhibiting missing/flickering sprites and holes in the background - and yes that will happen) then at some point you have to sacrifice some of that ease of use for an approach that enables more games to be meddled with.

I'm going to be watching your progress, but I can't promise I won't laugh along the way

No, you don't understand. I'M NOT TALKING ABOUT USING AI OR ML AT 50 FRAMES PER SECOND.

Okay, I think I got it this time. Though, I still think it sounds very nice as a theoretical chit-chat, but can be much harder to achieve in practice. Let's say, I am not arguing, but I don't feel capable of taking this amount of work, at least, in near future.
[/quote]

It's a fair bit of work, yeah, but might be worth it to make the tools that the user engages with much easier to use. Detecting sprites in memory from a snapshot is a perfect example of what an AI/ML model can be used for. It would simplify how the user finds those sprites without them having to worry about sprite size, interleaving etc etc.

And yeah, I presented the idea initially as a joke, but it would actually be very very useful. Not for the game rendering itself, but for the search tool the user would use for making their remake.

And as I said, when I go back to writing ZXSpin again, I'll definitely be looking at Spec256 and code hooking with an API for advanced users - I'm not as restricted by ease of use as you are.

[geecab]

Here's my Xevius test recolour. For some reason it can't replace the bullet, the crosshair or the square scenery block.
I was lazy and didn't rename it, so click on iball2.exe
Feel free to play with this and eventually finish it, for me it was just a test. I included a sprite sheet to make things easier

https://we.tl/t-kZizNQltfI

Looks good, thanks for uploading it, I gave it a go and really like the colour choice

- The bullet isn't showing because you've got the black and whites the wrong way round.
- The crosshair isn't showing because the bitmap dimensions need to be multiples of 8 (The searchBitmap, and the recolouredBitmap). Currently its, 12x8, should be in 16x8. I scratched my head with this problem in the past, the developer does mention it on his website though its easy to miss.
- The scenary block was the wrong dimensions too, I made it 16x16 but still couldn't make it show. This one is a bit odd. I might look into this a bit more later.


>>Mostly works but sometimes it defaults to the the original. It doesn't affect gameplay much but it's a bit strange

I do think you are experiencing the same bug that I've seen (The "Stormbringer Knight walking under a tree" issue I mentioned several posts ago).

In Xevious, when the player's ship flies over a patch of shaded ground, the emulator should still be able to recognise the ship as I've checked your searchBitmap and it looks correct to me.

Its strange, when the ship flies into the shaded area and whilst its completely inside the shaded area, the original bitmap is shown. Then, as the first (top left) pixel of player's ship leaves the shaded area, then the colour bitmap is shown (Correctly layered over the shaded area)...

[geecab]

>>The scenary block was the wrong dimensions too, I made it 16x16 but still couldn't make it show. This one is a bit odd. I might look into this a bit more later

Found it. The scenary blocks are named differently to what is in your settings.txt.

E.g. the lines that read:
0 pixel zx_block_1.bmp pc_block_1.bmp
0 pixel zx_block_2.bmp pc_block_2.bmp
0 pixel zx_block_3.bmp pc_block_3.bmp
should be:
0 pixel zx_block1.bmp pc_block1.bmp
0 pixel zx_block2.bmp pc_block2.bmp
0 pixel zx_block3.bmp pc_block3.bmp

Once renamed (and the bitmaps resized to 16x16 pixels), then the blocks appear

[Joefish]

Why not use the pattern-matching algorithm to search for the sprite data in a memory snapshot and present it for editing, then use a Spectrum-256 like approach to present it in-game?

I am not really familiar the details of with Spec256 approach. I mean, I know that it emulates a sort of "shadow graphics processor", but I neither know the details nor can estimate how hard is to to something similar.

I think it's already been covered, but the Spec256 approach is to run an emulation where each bit of memory is handled as a byte instead. That way, a 1-bit pixel in the emulated screen memory is actually an 8-bit colour index.(REMEMBER you'd need colour #1 to be 'black', as colour #0 is 'transparent' or a zero bit).
When emulating the Z80, you use the zero/non-zero status of a bit as 0/1. But when building the screen display, for each bit of pixel data you ignore the attribute, consider the 'bit' to be its full 8-bit representation, and look up its colour in a 256-entry palette.

Now my suggestion, to improve things, is you have a palette that is partly fixed, and partly dependent on the current attribute.
The fixed part of the palette is the first 224 colours.
Then there's a palette of another 256 colours. Each holds 16 variations on each of the 16 spectrum colours (including BRIGHT / DARK BLACK).
So the palette for a character cell consists of 224 base colours + 16 variations on the INK colour + 16 variations on the PAPER colour from the current attribute.
This allows sprites to be partially recoloured depending on their current attribute, as much or as little as desired. So for Manic Miner, your penguins could still have an attribute-dependent body-colour. But for something like 3 Weeks in Paradise, with its horrendous colour clash, you'd just use the fixed colour palette.

Furthermore, I am not entirely sure that the sprites are stored in the memory in the same format for all games. We know exactly that when a certain picture is displayed on the screen, it has a predefined pixel structure. When it sits in memory, it can be literally anything stored in the way the authors wanted. For some games with large sprites (like Popeye) I guess there are maybe no complete sprites at all, just separate sprite pieces later combined into a character on the fly. So in order to recolor such sprites one would need to guess these pieces and process them separately.

Most sprites are stored in a simple linear fashion, or a slight variant of it. Variations include alternating bytes of sprite / mask, or alternating rows, or alternating frames of sprite / mask; reversal of byte order, reversal of row order, reversal of every second row; sometimes pre-shifted with animation. But this is what the smart routine is for - to recognise the repeating shapes in memory and identify the start and end of individual sprites; which ones are similar enough to be considered animations, etc. And all done from a memory snapshot, so it can take as long as it wants, and accept hints and prompts from a user to guide it.

If sprites are stored as UDGs, with a second table to organise the UDGs into a large object, then you're pretty much stuffed. But a huge number of games could be tackled with only a few simple variations from linear storage.

You wouldn't even have to identify attribute data, as with my suggested approach sprites could be partly re-coloured by their original attributes, or with no respect to their attributes at all, as the designer wishes.

[ZXDunny]

I think that most people saw the Spec256 tool as being way too complex for them - they just want to be presented with graphics for recolouring without actually having to find them first.

Which is a shame, because it's the best "automatic" way to do it we've seen so far.

[4thRock]

Once renamed (and the bitmaps resized to 16x16 pixels), then the blocks appear

Thanks!

[4thRock]

I think that most people saw the Spec256 tool as being way too complex for them - they just want to be presented with graphics for recolouring without actually having to find them first.

That's why pattern matching is a great idea. Sure you still need to create a spritesheet, but that's very intuitive.
Once I tried to find sprites directly on TAPs and memory. Confusing because of different sizes, masks and the variations mentioned by Joefish.
So an automated approach has it's merits I think.

[ZXDunny]

I think that most people saw the Spec256 tool as being way too complex for them - they just want to be presented with graphics for recolouring without actually having to find them first.

That's why pattern matching is a great idea. Sure you still need to create a spritesheet, but that's very intuitive.
Once I tried to find sprites directly on TAPs and memory. Confusing because of different sizes, masks and the variations mentioned by Joefish.
So an automated approach has it's merits I think.

It does, but it certainly limits the games you can recolour and also the circumstances in which they can be recoloured. I'm hoping that I can be proved wrong though.

[rg_software]

Didn't have time for an update yet... However, since several people mentioned that EmuZWin has a potential to do the job "the right way", I tried to investigate the state of that project. It seems that it is effectively abandoned, and the sources are given to Denis Grachev (given that he keeps them locked somewhere). So he is perhaps the only person who can modify the emulator now, and it seems it's far from trivial given its old codebase.

P.S. People suggest using EmuZWin 2.6 instead of 2.7 -- it seems that some bugs @geecab encountered appear on the 2.7 version only.

[ZXDunny]

I would start a new emulator from scratch tbh. They're not terribly hard, just laborious to get right. If all you intend to do is play recoloured games then the accuracy doesn't have to be anything like SpecEMU or its ilk.

[hikoki]

This project might be a good opportunity for the Sam Coupe

[ZXDunny]

This project might be a good opportunity for the Sam Coupe

I'm not sure the Sam Coupe, even with an accelerator, would be able to find and replace the graphics quickly enough.

[Stefan]

This project might be a good opportunity for the Sam Coupe

I'm not sure the Sam Coupe, even with an accelerator, would be able to find and replace the graphics quickly enough.

Even an optimised SAM Coupé port of any title will have trouble since it needs to move four times the amount of graphics data around with only a slightly faster z80.