ZX Recoloring project

[rg_software]

Well, I see the discussion goes into the direction of "whether it is right approach or not"

I think the problem boils down to the fact that Spectrum doesn't have any "hardware-supported sprites" or anything like that. While for each particular game you can perhaps point out a better solution (let's say that for games like Elite you don't need sprites at all -- we can scale vector graphics wth perfection), in general case I see no real choice here.

First, on performance. Given that Spectrum screen is just 6912 bytes, I think sprite search should not be an issue. Currently I use a naive approach to searching, but I can implement something like Aho-Corasick algorithm, which should make searching of multiple sprites really fast. I can again complain about lack of time to do it, but I see interest now, so I promise to reconsider my schedule

Next, on false positives, etc, etc. Yes, it's all tough. Also, think about cases when sprites overlap and you need to decide which one goes above the other (say, in games like Fairlight). I am aware that we can find a lot of potential issues. But I really see no other "magic" approach that would solve these issues. Every game is different, sprite handling is different and so on. If I could only be sure that somewhere in the memory there is a map of sprites that can be substituted... Alas, I don't think that's the case.

But, anyway, instead of expounding my theories I'd perhaps better focus on the code... Maybe within the next 2 weeks or so I'll try to show something new. Thanks for your interest, really, I am happy to be here.

[AndyC]

In terms of "what resolution should games use" I think it's important to distinguish between real remakes and just coloured in spectrum titles. A genuine remake of the game has a low of flexibility to change how things look and adjust collision detection, movement etc to compensate. Recolouring a spectrum title is a lot more limited - it may break the game mechanics if a sprite is changed such that it no longer maps directly over the original. As Dunny mentioned, even something like smoothing curved edges might start causing collision detection to feel "off" when the underlying game engine is working to a pixel perfect hit with the original sprites. Taking it even further if you changed the graphics enough it might even become necessary to change the underlying imagery so the engine has a better concept of what is going on.

[Joefish]

Why not use the pattern-matching algorithm to search for the sprite data in a memory snapshot and present it for editing, then use a Spectrum-256 like approach to present it in-game?

[hikoki]

[mention]geecab[/mention]

Yes, I definitely think all that should be possible. You have put the idea of recoloring Scuba Dive in my head now Excellent game, and there shouldn't be many sprites to recolor either...

I would enjoy playing your mod for sure. That could be another feature idea, how to give the impression there are a greater variety of sprites just by playing with the search and replace mechanism. There could be randomisation when a match is detected like showing different treasures every time on Scuba Dive.

I don't see much of a problem in getting artifacts as long as they don't happen all the time and appear in a good looking way when possible just as a reminder of the original sprites and engine.

[4thRock]

Why not use the pattern-matching algorithm to search for the sprite data in a memory snapshot and present it for editing, then use a Spectrum-256 like approach to present it in-game?

Yep, second that.

Regarding what type of enhanced graphics are more acceptable, I'd go for what the Spectrum might display if it had hardware sprites.
So two colours (*) per 8x8 pixel block, aligned with the sprite origin. Perhaps 8x1 if we consider a Timex Hicolour mode. No sprite colour clash when placed on screen.

*- Well, if we consider paper as transparent, then it's a single colour per block but I can overlook that. No need to be 100% purist.

[4thRock]

Here's my Xevius test recolour. For some reason it can't replace the bullet, the crosshair or the square scenery block.
I was lazy and didn't rename it, so click on iball2.exe
Feel free to play with this and eventually finish it, for me it was just a test. I included a sprite sheet to make things easier

https://we.tl/t-kZizNQltfI

[ZXDunny]

I actually did something similar.

A long time ago, I wrote a remake of Sinclair BASIC. I run the emulation, and then for the parts I want to replace I hook the ROM code - when the emulator encounters one of these hooks, it hands processing off to the relevant handler, and then returns. The emulated Speccy has literally no idea that anything has happened. That way I was able to remake the BASIC editor, the error handling, get statistics about memory usage, monitor variables and do other stuff too, like a full debugging suite that offered breakpoints and watches and suchlike in real time while the BASIC program was running.

It was quite successful.

This is not really much different, but you're trading ease of use in order to be able to recolour and enhance only a very, very small number of games. There are very few out there that don't use XOR to print their sprites - any game that does that will immediately fail to be recoloured and will drop back to Speccy graphics. That's got to be a jarring experience.

As an example, Manic Miner and JSW both disallow (due to their game engines) sprites from touching anything. That's great, but the player sprite can walk through certain passable walls - as soon as it does, your recolouring goes out the window.

And you'll never be able to enhance any 3D games. Not Mercenary, nor Elite, nor Starion, nor any Freescape title. You can't enhance any Ultimate games. Nor any of the Wally Weeks. None of the Odin games (Nodes springs to mind) can be enhanced by image recognition.

To get around this you absolutely must take a different route. Intercept screen memory writes and identify sprites from there - maybe keep track of them in 16x16 (or whatever size) chunks you can examine at the end of the frame and replace in-order. That would eliminate most of the overlapping issues, but only for games that don't do any double buffering. And it wouldn't solve 3D games.

Go the Spec256 route of layering 8 bytes for every byte of RAM. It's pretty simple to do, and an editor that's well written can take the heartache out of making the games for the users. That's certainly better than any image recognition, but may fail on different coloured sprites - unless you can examine the attribute bytes when it's written. An intelligent system could maintain several "pages" of graphics to choose from based on the ATTR value.

But even that can't match the king of recolouring - code hooking. There's a very good reason I used it myself. You can catch anything - sprites, 3D, you name it. Want the Speccy Outrun port to look amazing? Easy. Want it to run faster, higher frame rate? Hook the interrupt routine, or modify T-State counts for RAM writes. The world's your Oyster.

But of course, this method is really, really hard for your average user. You could churn out your own datafiles for games, or you could go for say, a Python scripting interface and open up an API for some of the more savvy users to start converting games for everyone to play.

I know if I was doing this (and I may yet - I have plans to go back to Speccy emulation hopefully soon) I'd go for the latter. Hell, I even have a BASIC interpreter core that is as fast as Python but looks surprisingly like Sinclair BASIC which could be used for the API.

So which do you want? Easy for users to create their own, or high quality recolouring?

[rg_software]

Why not use the pattern-matching algorithm to search for the sprite data in a memory snapshot and present it for editing, then use a Spectrum-256 like approach to present it in-game?

I am not really familiar the details of with Spec256 approach. I mean, I know that it emulates a sort of "shadow graphics processor", but I neither know the details nor can estimate how hard is to to something similar.

Furthermore, I am not entirely sure that the sprites are stored in the memory in the same format for all games. We know exactly that when a certain picture is displayed on the screen, it has a predefined pixel structure. When it sits in memory, it can be literally anything stored in the way the authors wanted. For some games with large sprites (like Popeye) I guess there are maybe no complete sprites at all, just separate sprite pieces later combined into a character on the fly. So in order to recolor such sprites one would need to guess these pieces and process them separately.

[rg_software]

So which do you want? Easy for users to create their own, or high quality recolouring?

Well, you know, I'd say that "easy for users". When I duscussed my approach many years ago with Remakes.org people, some responses were "I do remakes because I like coding, that's it". Most Speccy games aren't rocket science, and it is not extraordinarily difficult to do a decent remake with "additional sauce" like scrolling or music or whatever capability you miss in the original (say, free camera mode in 3D games). Even the best recoloring ever can't get over these limitations.

Yes, you are of course correct when you speak about limitations. I wish we had an option to make recoloring both easy for users and reliable. If we are dealing with really advanced users who can dig deeply into game logic, they may simply opt for their own remake.

[geecab]

[mention]rg_software[/mention]
Great to hear you are getting your interest back into the project FYI. I am working on Rebelstar2 (One of my absolute favorite games of all time) at the moment and it works perfectly with your emulator. Loads of sprite recognitions happening fast as I can use the 8x8 'block' searching option in settings.txt, I'll post screenshots later. This means that Chaos/LazerSquad/SpaceCrusade (Excellent turn-based games that have a big fan base) are all good-to-go with the sprite recognition approach!

Yes, you are of course correct when you speak about limitations. I wish we had an option to make recoloring both easy for users and reliable. If we are dealing with really advanced users who can dig deeply into game logic, they may simply opt for their own remake.

Exactly. I feel the pure sprite recognition approach would always be a good place to start for users at any level (novice/or advanced). You might get to the end of your remake that uses only sprite recognition and realize that is all you need (Rebelstar2 for example), or maybe you do see occasional artifacts but decide that you can live with them. However, if you absolutely can not deal with 'what happens when that sprite goes there and touches that' situation, your sprite recognition work is not in vain, you have all your bitmaps drawn to help with your advanced solution.

[Ralf]

I am not really familiar the details of with Spec256 approach. I mean, I know that it emulates a sort of "shadow graphics processor", but I neither know the details nor can estimate how hard is to to something similar.

Furthermore, I am not entirely sure that the sprites are stored in the memory in the same format for all games. We know exactly that when a certain picture is displayed on the screen, it has a predefined pixel structure. When it sits in memory, it can be literally anything stored in the way the authors wanted. For some games with large sprites (like Popeye) I guess there are maybe no complete sprites at all, just separate sprite pieces later combined into a character on the fly. So in order to recolor such sprites one would need to guess these pieces and process them separately.

It looks like Spec256 website still exists You can read there a little about the used approach:
http://www.emulatronia.com/emusdaqui/sp ... ex-eng.htm

It was a closed project where authors modified themselves a few games but there wasn't open code or any editor available for everybody so he could recolour any game. So we may not know the details but they describe the general idea:

The idea is: in one hand we´ve got the speccy emulator with its memory zone, and in the other we emulate a Z80 which works with 64 bit registers instead of 8 bit, and with a memory map with positions of 64 bit instead of 8.

Each time a z80 instruction is emulated, the same instruction is simulated with data always supposed to be graphics. I´ve named this parallel processor Z80_GFX. Z80_GFX modifies its memory zone accordig to the instructions and doesn´t do anything with Z80´s memory zone. What have we archieved? A faithful emulation of Z80 and a memory zone from which we can obtain 256 colour graphics.

Each time they probably also used some custom hacks tailored for each game.

So that's Spec256. And we have also EmuZWin. I believe it uses exactly the same approach, just it enables the user to recolour any game himself.
And you are right, most time you work on chunks of graphics and not complete sprites. And you have to find these chunks yourself in a mess of pixels And it requires skill+intuition as these data may be stored in different format and you must reorganize the display to get it in the friendliest way. And yes, it's a pain

I believe you cannot avoid it, but you can improve the process with a good, friendly tool

[Ralf]

And for anybody interested, there is a recently active thread on a Russian forum about 256 colour games

Here's the English automatic translation:
https://translate.google.pl/translate?s ... ec256.html

You may find there quite a lot of recently recoloured games (with EmuZWin) and among others you may find there my almost completed (but having problems of course ) recoloured version of Renegade which I didn't announce anywhere else:

[ZXDunny]

The idea is: in one hand we´ve got the speccy emulator with its memory zone, and in the other we emulate a Z80 which works with 64 bit registers instead of 8 bit, and with a memory map with positions of 64 bit instead of 8.

Each time a z80 instruction is emulated, the same instruction is simulated with data always supposed to be graphics. I´ve named this parallel processor Z80_GFX. Z80_GFX modifies its memory zone accordig to the instructions and doesn´t do anything with Z80´s memory zone. What have we archieved? A faithful emulation of Z80 and a memory zone from which we can obtain 256 colour graphics.

Each time they probably also used some custom hacks tailored for each game.

No hacks needed for this particular approach. The 64bit z80 they talk about is misleading. You don't need to do that.

You have 8 "pages" for each RAM page. Each acts as a bitplane - the Speccy RAM contains the low bit of each bit that's read, and when you read a byte from RAM you read 8 bytes, one from each page and store them in 8 registers per speccy register. When performing memory writes, you do the same thing in reverse.

When it comes time to build the display for each frame, you take one bit for each pixel from those 8 banks of RAM and build a 256 colour pixel from those.

That's it - no hacks needed. The extra colours come from a tool which identifies sprites and graphics in RAM and allows you to colour them, eventually spitting out a "snapshot" that contains all 8 pages specially constructed from those 256 colour graphics.

You're right that it's all about the tool - if it's easy for the user to find and modify the graphics then the engine will take care of the rest.

It has limitations. Can't work on tape images. Can't recolour blank backgrounds (those need to be hacked in). Can't do 3D games.

But it's really the simplest one you can do.

[hikoki]

That Renegade remake looks good!!

About these tools, I don't know if it'd be possible a sort of rzx viewer where the user can record,pause,resume a gameplay. One could pause it and start clicking over the screen to assist the algorithm engine to build its configuration parameters about collision bounds, glitches, wrong matches, etc.

[ZXDunny]

Training a DeepMind instance to recognise sprites in a Speccy snapshot would be very, very doable. In fact, it would be an excellent solution to the problem where actually using your eyes to find them is just too hard for everyone.

[rg_software]

In fact, it would be an excellent solution to the problem where actually using your eyes to find them is just too hard for everyone.

It's funny you think exact matching is not a good solution whereas an approximate ML-based algorithm with no guarantees whatsoever and a theoretical possibility that your sprite will be substituted with something completely different is "excellent" If no one is able to recognize a sprite on the screen, by definition it's not important what's going on there!

Though I shouldn't really be chatting about these things, I'd better work on my current codebase

[obo]

Training a DeepMind instance to recognise sprites in a Speccy snapshot would be very, very doable. In fact, it would be an excellent solution to the problem where actually using your eyes to find them is just too hard for everyone.

You could probably combine the two, so ML recognises the sprites in the display image, and the emulation remembers the source of the data. It could have some built-in knowledge of sprite masks too. Playing the RZX would give it a good selection of what is needed for a game. You'd probably still want it to be a starting point, with a manual review over the results to fill gaps and correct false-positives.

Hook-all-the-things has been my approach so far, but it's a very manual process. Even given some tools like Spec256, there aren't many people willing to go through it.

[ZXDunny]

In fact, it would be an excellent solution to the problem where actually using your eyes to find them is just too hard for everyone.

It's funny you think exact matching is not a good solution whereas an approximate ML-based algorithm with no guarantees whatsoever and a theoretical possibility that your sprite will be substituted with something completely different is "excellent" If no one is able to recognize a sprite on the screen, by definition it's not important what's going on there!

Though I shouldn't really be chatting about these things, I'd better work on my current codebase

It's funny that you thought that this is what it would be used for. If you think that using an AI to detect sprites at runtime and replace them is at all less computationally expensive than your current method then I fear for any progress to be made...

To put it into terms you might better understand:

1. Nobody wants to do this because the tools suck
2. Someone creates a very, very problematic solution that involves image-recognition which will create a shed-load of problems for the vast majority of the games out there, because the tool is easy to use.
3. Users are really interested (for now!) but eventually get annoyed because the engine can't replace the graphics for very many games at all without horrific flickering.

How to fix? Well, we need a better option than image-recognition (and I've outlined a few that are better) BUT... the tools get more complex for the user. To create a recolouration, the user would either have to be able to find the graphics in RAM or Disassemble the game to find suitable hook addresses. While the latter is hard for non-coders, the former is easy with a simple graphics ripper. But it's not simple enough! Who wants to muck around looking for graphics and messing with addresses and modulus? Especially when this new tool lets you do it from screen grabs.

To address this insanely difficult task of finding graphics, we could indeed feed load of snapshots to an AI, tell it where the graphics are in those snapshots so it creates a neural net of ability to recognise sprite sequences.

Ta-da! After that initial training (which gets better the more you feed it), it can recognise the sprite addresses for you! And there's nothing to stop this from being an add-on to the usual RAM-search tool as a fallback.

I do hope you understand now.

Training a DeepMind instance to recognise sprites in a Speccy snapshot would be very, very doable. In fact, it would be an excellent solution to the problem where actually using your eyes to find them is just too hard for everyone.

You could probably combine the two, so ML recognises the sprites in the display image, and the emulation remembers the source of the data. It could have some built-in knowledge of sprite masks too. Playing the RZX would give it a good selection of what is needed for a game. You'd probably still want it to be a starting point, with a manual review over the results to fill gaps and correct false-positives.

Hook-all-the-things has been my approach so far, but it's a very manual process. Even given some tools like Spec256, there aren't many people willing to go through it.

"Hook everything" is the best way but it's not easy for users. It can recolour and make higher-res anything but because it's not identifying sprites from screen grabs it's unlikely to get any attention.

Personally, I'd go with both and a Python API built into the emulator.


But then, what do I know?

[rg_software]

If you think that using an AI to detect sprites at runtime and replace them is at all less computationally expensive than your current method

Well, as I said before, I shouldn't be really theorizing here. However, what do you mean by "AI"? Let's talk numbers. If I use Aho-Corasick method, the computational complexity is basically linear. You can't make it faster even in theory.

To address this insanely difficult task of finding graphics, we could indeed feed load of snapshots to an AI, tell it where the graphics are in those snapshots so it creates a neural net of ability to recognise sprite sequences.

Yes, I understand you want a silver bullet that would do the work instead -- let's call it "AI", "neural network" or "magic". I completely understand this wish, but I don't believe its doable. Actually, it is, let's call it "graphics filter" -- that's what we got already.

I do hope you understand now.

Yes, absolutely. I wish someone really goes through this process to evaluate its real complexity. I mean, if you/anyone can make it and measure how much time end effort it takes, demonstrate how it works and estimate how fast it is -- I will be the first person to applaude.

[ZXDunny]

If you think that using an AI to detect sprites at runtime and replace them is at all less computationally expensive than your current method

Well, as I said before, I shouldn't be really theorizing here. However, what do you mean by "AI"? Let's talk numbers. If I use Aho-Corasick method, the computational complexity is basically linear. You can't make it faster even in theory.

What has that got to do with anything?

To address this insanely difficult task of finding graphics, we could indeed feed load of snapshots to an AI, tell it where the graphics are in those snapshots so it creates a neural net of ability to recognise sprite sequences.

Yes, I understand you want a silver bullet that would do the work instead -- let's call it "AI", "neural network" or "magic". I completely understand this wish, but I don't believe its doable. Actually, it is, let's call it "graphics filter" -- that's what we got already.

Again, has nothing to do with what you're trying to achieve. We've already established that your approach will not work for the vast majority of games.

I do hope you understand now.

Yes, absolutely. I wish someone really goes through this process to evaluate its real complexity. I mean, if you/anyone can make it and measure how much time end effort it takes, demonstrate how it works and estimate how fast it is -- I will be the first person to applaude.

No, you don't understand. I'M NOT TALKING ABOUT USING AI OR ML AT 50 FRAMES PER SECOND.

Sheesh.

[hikoki]

I've made a quick search and there seem to be photo editors based on AI like pixelmator for Mac.
Another interesting experiment is this Neural photo editor on github, https://github.com/ajbrock/Neural-Photo-Editor
I guess you would have to fit your custom sprites to the original ones by selecting them manually on a rzx snapshot. Supervised machine learning I think it's called the jargon term. Looks like complicated!

[rg_software]

We've already established that your approach will not work for the vast majority of games.

Well, honestly speaking I think many of these issues still can be resolved with partial matching and other tricks, so I might be able to cover many harder cases, so it isn't done yet.

No, you don't understand. I'M NOT TALKING ABOUT USING AI OR ML AT 50 FRAMES PER SECOND.

Okay, I think I got it this time. Though, I still think it sounds very nice as a theoretical chit-chat, but can be much harder to achieve in practice. Let's say, I am not arguing, but I don't feel capable of taking this amount of work, at least, in near future.

[ZXDunny]

We've already established that your approach will not work for the vast majority of games.

Well, honestly speaking I think many of these issues still can be resolved with partial matching and other tricks, so I might be able to cover many harder cases, so it isn't done yet.

Yes, but even getting the more obvious XOR problems sorted, your algorithm will end up with so many bolted-on edge cases that it might not be fun to maintain. And even then, you're still restricting yourself to a very narrow corridor of the amount of games that people will want to convert.

I do understand that you want a method that presents the user with as easy an experience as you can, for recolouring games. But when the method used restricts people to only a handful of games that can run smoothly (with all the rest exhibiting missing/flickering sprites and holes in the background - and yes that will happen) then at some point you have to sacrifice some of that ease of use for an approach that enables more games to be meddled with.

I'm going to be watching your progress, but I can't promise I won't laugh along the way

No, you don't understand. I'M NOT TALKING ABOUT USING AI OR ML AT 50 FRAMES PER SECOND.

Okay, I think I got it this time. Though, I still think it sounds very nice as a theoretical chit-chat, but can be much harder to achieve in practice. Let's say, I am not arguing, but I don't feel capable of taking this amount of work, at least, in near future.
[/quote]

It's a fair bit of work, yeah, but might be worth it to make the tools that the user engages with much easier to use. Detecting sprites in memory from a snapshot is a perfect example of what an AI/ML model can be used for. It would simplify how the user finds those sprites without them having to worry about sprite size, interleaving etc etc.

And yeah, I presented the idea initially as a joke, but it would actually be very very useful. Not for the game rendering itself, but for the search tool the user would use for making their remake.

And as I said, when I go back to writing ZXSpin again, I'll definitely be looking at Spec256 and code hooking with an API for advanced users - I'm not as restricted by ease of use as you are.

[geecab]

Here's my Xevius test recolour. For some reason it can't replace the bullet, the crosshair or the square scenery block.
I was lazy and didn't rename it, so click on iball2.exe
Feel free to play with this and eventually finish it, for me it was just a test. I included a sprite sheet to make things easier

https://we.tl/t-kZizNQltfI

Looks good, thanks for uploading it, I gave it a go and really like the colour choice

- The bullet isn't showing because you've got the black and whites the wrong way round.
- The crosshair isn't showing because the bitmap dimensions need to be multiples of 8 (The searchBitmap, and the recolouredBitmap). Currently its, 12x8, should be in 16x8. I scratched my head with this problem in the past, the developer does mention it on his website though its easy to miss.
- The scenary block was the wrong dimensions too, I made it 16x16 but still couldn't make it show. This one is a bit odd. I might look into this a bit more later.


>>Mostly works but sometimes it defaults to the the original. It doesn't affect gameplay much but it's a bit strange

I do think you are experiencing the same bug that I've seen (The "Stormbringer Knight walking under a tree" issue I mentioned several posts ago).

In Xevious, when the player's ship flies over a patch of shaded ground, the emulator should still be able to recognise the ship as I've checked your searchBitmap and it looks correct to me.

Its strange, when the ship flies into the shaded area and whilst its completely inside the shaded area, the original bitmap is shown. Then, as the first (top left) pixel of player's ship leaves the shaded area, then the colour bitmap is shown (Correctly layered over the shaded area)...

[geecab]

>>The scenary block was the wrong dimensions too, I made it 16x16 but still couldn't make it show. This one is a bit odd. I might look into this a bit more later

Found it. The scenary blocks are named differently to what is in your settings.txt.

E.g. the lines that read:
0 pixel zx_block_1.bmp pc_block_1.bmp
0 pixel zx_block_2.bmp pc_block_2.bmp
0 pixel zx_block_3.bmp pc_block_3.bmp
should be:
0 pixel zx_block1.bmp pc_block1.bmp
0 pixel zx_block2.bmp pc_block2.bmp
0 pixel zx_block3.bmp pc_block3.bmp

Once renamed (and the bitmaps resized to 16x16 pixels), then the blocks appear