XT-Engine: small, but powerful platform game engine in asm

[ketmar]

eh… spent almost a week rewriting UrForth. moved most words out of the C kernel (they're compiling from the Forth sources on startup now; this includes things like IF/THEN, and even The Colon Word). also, the system is more like a normal Forth now: no more hidden C data structures for dictionary and such.

why? because it is easier to write Forth in Forth. ;-) i can also reuse some code from native UrForth/x86, and use native system to develop and debug various things. it will be much easier to develop AGD-like compiler this way.

also, i will prolly convert UrAsm to one-pass assembler later. there is no real reason to have two passes, and with Forth code it now has to do all the work twice, basically for nothing. it's also error-prone, because Forth code need to behave exactly the same on both passes. it is way easier to use one-pass approach instead.

and maybe i will turn it inside out: let Forth system drive the assembler, not vice versa. tbh, that was the plan for a long time now, i simply haven't had enough motivation to do it. ;-) if i'll do it that way, i will prolly change dsForth to cross-compiler then. there is little reason to develop in dsForth on the real hardware (or in an emulator). then it will be possible to develop and debug your code on PC, and simply cross-compile it to Speccy, leaving only those parts of the system your code used. i may even write a native Z80 codegen which will turn most Forth words from threaded code to native code. but this is something that should go into dsForth topic, i guess… ;-)

[ketmar]

btw, i just realised that my "the code is not public" doesn't mean what i meant to say. ;-) it is not ready to be released yet, but of course, Fossil repo is public. along with UrAsm repo.

[Wall_Axe]

what platform is dsForth for?

why make it a one-pass compiler, its already very fast. Won't it introduce new bugs?

[PeterJ]

is dsforth for the nintendoDS?

No. It's a recompiled and slightly optimised version of Abersoft Forth for the Spectrum.

viewtopic.php?p=41975&hilit=Dsforth#p41975

You seem to have edited your post after I quoted.

[ketmar]

why make it a one-pass compiler, its already very fast. Won't it introduce new bugs?

basically, there is no reason to have two passes, one is enough. to be more precise, it is either 1, or way, way more than 2. all labels can be resolved at the end of the 1st pass anyway. or, if you want your generated code to depend on yet unknown labels (for data align and such), you need way more than 2 passes to stabilize everything (that's what Flat Assembler for x86 does, for example).

as for bugs… meh. ;-) any new code will have bugs, it's not the reason to stop writing new code at all. ;-)

UrAsm is a freakin' mess, it is something i would never write in my sane mind. it was quickly thrown together to re-assemble decompiled AberForth (that's what became dsForth later), and then i kept adding features on top of the one of the worst codebases i've ever created. i always wanted to rewrite it. there was UrAsm2 project once, but it died at the prototyping stage. UrAsm history goes back to 2005, and it really sux. it works, but it is impossible to maintain or extend it properly. so i'm using my chance to nuke the old code now.

[ketmar]

eh. writing new Forth code using UrAsm as a shell is annoying. so i converted UrForth/C into properly embeddable library, which is now shared between UrAsm and standalone runner. yay, Yet Another Scripting Engine! now i'm adding various things like data structures, to write AGD-like compiler. so while this post is about UrForth, it is about XTE too. (btw, "XT" means "eXTensible" now. ;-)

btw, UrForth has built-in debugger, which is completely broken now. i should rewrite it in Forth too. it was impossible to do with the previous version, but now there are no hidden C structs in there. i am mostly ok with "DEBUG:STACK-DUMP" and "DEBUG:BACKTRACE" words yet, but sometimes i want a proper debugger.

[ketmar]

(by the way, i have to admit that writing Forth from scratch for the metal is much, much easier. without idiotic C proxy it is a matter of days, not weeks. but i don't expect people to spend their time porting my native implementation to various OSes and architectures, hence C. sigh.)

[ketmar]

just4fun (and for debugging too ;-) UrForth/C now passes ANS test suite. it also got line editor, and partially working interactive debugger with breakpoints and single-stepping.

now i am very close to my goal of writing a simple compiler (and prolly some other dev tools like room/sprite designer too).

please, somebody tell me how come this project mutated to Game Development Studio? i somehow missed that moment. ;-)

p.s.: and if i port my Z80 emulator to UrForth… oh, no. i definitely WON'T GO THAT WAY! ;-)

[Wall_Axe]

p.s.: and if i port my Z80 emulator to UrForth… oh, no. i definitely WON'T GO THAT WAY!

Then you would have a game dev tool that creates a .tap file but can also create a windows executabe that runs the game within an emulator
Without the end user realising what is going on.

How does your forth for zx spectrum handle strings?

I like the assembler way , which means you can create an array of pointers to strings. So the strings can have variable lengths. And some of the pointers can just be NULL.

[ketmar]

How does your forth for zx spectrum handle strings?

as you want it to. the whole Forth compiler written in Forth itself, there are no hidden parts there. you can use any mechanics you like: inline, tables, 0-terminated, "or 0x80"-terminated, compressed, anything. the system doesn't care, because it doesn't need strings at all. so it is up to user to manage their string resources (by extending the compiler as the user needs to). that's basically how you're writing Forth code: extending the system until it does what you want it to do.

[Wall_Axe]

as you want it to. the whole Forth compiler written in Forth itself, there are no hidden parts there. you can use any mechanics you like: inline, tables, 0-terminated, "or 0x80"-terminated, compressed, anything. the system doesn't care, because it doesn't need strings at all. so it is up to user to manage their string resources (by extending the compiler as the user needs to). that's basically how you're writing Forth code: extending the system until it does what you want it to do.

That's pretty cool, do you have a cross compiler from windows to spectrum available?

[ketmar]

That's pretty cool, do you have a cross compiler from windows to spectrum available?

nope, it's not in a state of being usable yet. maybe one day it will be. ;-)

[ketmar]

it is related, so here. ;-)

new UrAsm engine is almost ready. it is one-pass, writen completely in Forth, and succesfully assembled several samples (including label resolution). so, UrForth is the host app now, and z80 asm is just a loadable module. Just As Planned! ;-)

of course, this new asm engine still missing many features of the old one, but it's something that could be easily added. code generator (including expression evaluation, and postponed label resolution) is the main part anyway.

i will also use the opportunity to get rid of several "historical" features of old UrAsm, including macros. macro system was badly designed anyway, and it is way easier to code macros directly in Forth.

[Ralf]

Any chance that we'll see this engine soon in action?

By the way I didn't understand all this Forth discussion. If this engine is
written in Forth, would it be possible to use it in a game written in assembler?

[ketmar]

Any chance that we'll see this engine soon in action? :)

it is simply new UrAsm version, which is used to compile Z80 assembler source. of course, you'll see it used with the next XTE tech demo source code. because this is what you'll need to build the code. ;-)

otherwise, it is simply Yet Another Z80 Assembler, just written in esotheric language this time.

By the way I didn't understand all this Forth discussion. If this engine is
written in Forth, would it be possible to use it in a game written in assembler?

XTE engine itself is written in assembler. it is the assembler itself, and tools used to build the game are in Forth. think of it like this: you have cross-development tools written (and scriptable) in Forth, plus assembler XTE source. XTE itself is fairly portable, but you need some way to describe rooms, sprites, and so on. of course, you can do it in asm too, with "defb" and such… or use some tool to convert your data files into something that can be used to build the final game. UrForth/C+UrAsm is The Tool.

the idea is like this: you'll have a choice of using just XTE, and write everything else in asm manually; or use dev tools bundled with XTE to code your game in AGD-like language, and the compiler will do the rest. or mix them in any way you like (like generating asm source from your AGD-like script, and then use it as a starting point for further asm development, for example).

[PeterJ]

@ketmar,

What's the new URL of the download the archive with .tap demo and full engine source code. The link on the original post does not seem to work anymore.

If you let me know when it changes I can edit the post for you.

[ketmar]

please, just remove the download link for now, and replace it with something like "COMING SOON". XTE1 is not developed anymore, and XTE2 doesn't have a real demo yet. i'll PM you when it will be ready. thank you!

ah, btw. please, put XTE source code repository link there: http://ketmar.no-ip.org/fossil/xtengine/.

[ketmar]

Any chance that we'll see this engine soon in action? :)

ah, and new XTE demo is being developed right now. it will be small, but fully featured and beatable game, not just some room with random sprites thrown in. that's why i temporarily switched the focus: i need some new dev tools to code this demo, and i decided to write those tools myself (as i usually do ;-).

just wait a little more, and you will see XTE in action. or, even IN ACTION! ;-)

after releasing the demo, i will work on releasing first XTE public beta, with documentation, compiler and such. if Real Life will not intervene, i hope to release XTE beta in January.

[PeterJ]

please, just remove the download link for now

All done!

As an aside, please can members avoid posting general questions about learning Forth in this thread. Please start a new topic in programming. Thanks

[Ralf]

Thanks for explanation Ketmar.

You are certainly taking a long path to your goal. If you just wanted to make a game, you could do it in 3 months or so,
without making any fancy tools.

But I guess, you want to make sothing more, an engine - something resuable that other people could use too.
In such case you must remember to make it friendly. If you make just a game, then your code may be ugly, it will
be enough if you are the only man in the world who can understand it

But in case of an engine, it must be friendly or nobody will use it.
I'm not so sure if using Forth as metalanguage for your engine is really friendly. Personally I can program Z80 asm but not Forth.
I would have to learn it if I wanted to use your engine.

But you have total right to do it the way you like it. So all the best to your project !

[ketmar]

But I guess, you want to make sothing more, an engine - something resuable that other people could use too.
In such case you must remember to make it friendly. If you make just a game, then your code may be ugly, it will
be enough if you are the only man in the world who can understand it ;)

But in case of an engine, it must be friendly or nobody will use it.

yes, exactly this!

I'm not so sure if using Forth as metalanguage for your engine is really friendly. Personally I can program Z80 asm but not Forth.
I would have to learn it if I wanted to use your engine.

the main idea is that you don't have to! ;-) like, you don't have to learn C++ to use sjasm, for example. i am using Forth to create tools, but you will use the tools themselves. and tools will include the compiler for simple AGD-like language. it doesn't matter in which language the compiler itself is written. of course, if you will want to modify the compiler, or to create/extend some other tools, then you will have to learn Forth, but it is not a requirement.

that is, XTE users will write code in some simple high-level language, and use already written tools to import rooms, gfx, and so on, and then build the game. it will be possible to write Z80 asm code too to customise the engine to user needs. Forth is there, but you will not see it unless you really want to.

but at the other side… the whole system is compiled from Forth sources on startup (do not worry, it takes ~10 msecs on my 10+ years old core2duo, for example). i.e. the assembler, compiler, gfx converters, room converters — everything is there in source code form in case you'll want to modify it. no need to install C compiler, linker and other tools. actually, you can use UrForth/C as a standalone Forth system if you want to. UrAsm is just a normal bundled application, there's nothing special there.

it's somewhat like sjasm+ with Lua: you don't have to learn Lua to use sjasm+. but if you want to add something the system doesn't have yet, then you can write Lua script for that. the same with UrAsm, only it's Forth, not Lua in there.

p.s.: my explanations are not really clear, i think. just wait a little until the first release, and you'll see that it's all much easier that it looks. ;-) just imagine that i wrote my own C compiler, and then wrote Z80 assembler in C, and compiled it with my C compiler. you don't have to know C, or how my compiler works to use my assembler. and now simply replace C with Forth. ah, and you will also get my Forth compiler with the system, for free. ;-)

[Ralf]

Okay, I guess it get it.

There are actually 3 languages involved - asm, Forth and some noname ADG-like game design language.
And you use tools written in Forth to compile code written in "AGD" language to code in asm
And you don't need to know Forth for using tools written in it.

Yes, this may be friendly. I suppose most people would say that AGD is friendly and it seems similar.

[ketmar]

yes, exactly this. end user can write code in Z80 asm, in AGD-like, or in any mix of both, and use my UrAsm to compile it. most users will never ever know that there is a Forth system inside.

[ketmar]

so, i'm almost there! most of the features required to build XTE are implemented (including STRUCT support, which is quite big). the only thing missing is simple textual macros now. the assembler translator code is twice as big as Forth system startup code, lol. still, system startup time is under 20 msec (it compiles almost 300KB of Forth code on startup).

the assembler itself is slower than C version, though: it compiles 99KB of CP/M sources in 140 msecs. acceptable. it is that slow because UrForth/C is quite slow. i can make it much faster (5-7 times faster at least), but at the price of portability loss. i'll prolly do it if translation speed will start taking seconds. ;-)

[ketmar]

i'm there. XTE was completely built for the first time with new Forth-based UrAsm, and it seems to work. build time is ~460 msecs (for ~300 kb of code). ok, this is the price i am ready to pay for much better dev system.

i want to stress that it's not because Forth itself if slow, it is because UrForth/C is slow. this is the price i payed for it being portable. i.e. i cannot generate proper x86 code, and each memory access is checked. i'll try native port later, but for now it's not the top priority. i started new UrAsm project for XTE, not vice versa. ;-)