Another project using ZXDB

[PeterJ]

I came across this earlier this evening:

https://mobile.twitter.com/HermitRetro/ ... 7676404744

https://www.hermitretro.com/

[kolbeck]

Yeah, I've been communicating with them for a while now. The idea, is a virtual fs layer for linux - that's using the API for creating a "ZXDB" file system. Sounds cool

[kolbeck]

Another interesting project for those of you who got a Next - http://zxdb.remysharp.com/

/T

[descarte]

Hi,

This is a project, called ZXDBFS, to provide a virtual filesystem for UNIX systems that provides a file-like interface to ZXDB. In summary, you can effectively just browse the filesystem using Finder/File Explorer/shell and read files.

This is an offshoot of the Hermit Retro ZXZero product which is a Raspberry Pi Zero W-powered board that's custom built for ZX Spectrum and Spectrum Plus cases. The project uses a modified version of Fuse and a stripped RaspiOS core. The board supports loading of games from a second SD card. See https://facebook.com/hermitretro for videos/updates etc. (main website is https://hermitretro.com). The boards are available from ZX Renew (https://www.zxrenew.co.uk).

However, as the PI Zero W has WiFi capability and ZXDB exists, it seemed like a good opportunity to put together some sort of "load games from the network" without having to search a separate website and download them (not that this isn't a good thing!). I also thought it'd be useful to have something decoupled from Fuse as its primary concern is to emulate, not to become some sort of kitchen-sink application.

So, from the point of view of any application wanting to access games via ZXDB via the filesystem, you literally just open the file. The filesystem handles figuring out where it actually is and transfers the data via HTTPS for you.

The filesystem supports browsing and I'm just adding in the search functionality. There's opportunity to add further "magic" collections (like, top 100 games or whatever) as extra layers of curation.

Hope that provides a bit of background about the project.

The code will be released to the Hermit Retro Github (https://github.com/hermitretro) when in some sort of releasable shape under the MIT License in case any other application developers want to use it. Sales of the Hermit Retro products help justify time spent on developing these other projects

There's already another filesystem I wrote for the ZXZero project to provide a UNIX-like filesystem for the second SD card. That's already on github.

If anyone's got any questions, happy to answer them here.

A.

[dfzx]

That's very clever. I've idly pondered how I'd add a ZXDB interface onto Fuse, but a decoupled custom filesystem never occurred to me.

What are you assuming for the server end? Your filesystem code needs to be pointed at a host of the database, which needs to be running an API, right? Are you expecting the user to host their own ZXDB instance, or is there some standard-API-supporting ZXDB server out there on the internet already?

[descarte]

The server end is basically the ZXDB REST API at https://api.zxinfo.dk/v3. The ZXDBFS filesystem doesn't need anything as its essentially just a glorified HTTP client. Fuse (the filesystem layer, not the emulator) is userspace, so you actually don't even need root privileges to run it. Literally, just mount the filesystem onto a directory and browse files.

I had mused about using local copies of ZXDB but I don't think it's overly useful at this stage. I'd rather leverage the excellent work already done on the REST API and help enrich that if needed.

For the Hermit Retro stuff, the plan will be to also just copy any loaded files from the filesystem onto the second SD card anyway, so a network outage won't stop the fun entirely!

A.

[dfzx]

All understood. It's a really nice idea.

Be sure to post back here when you have something to share, I'd like to try it.

[descarte]

Further ZXDBFS update with Fuse integration. By integration, I mean, I added a text widget to let you type in a search term.... everything else is handled by the filesystem...

https://twitter.com/HermitRetro/status/ ... 1999109126

[dfzx]

I got around to having a look at this last night. Apart from an undocumented(?) dependency on "fuse3" (I had "fuse" and "libfuse3"), and hence a missing fusermount3 binary, it built and worked fine. It's really very good.

I think adding an info.txt file to each directory might be useful. There are a lot of empty directories for ZXDB entries with no screen shot or downloadable. Zero files suggests it hasn't downloaded yet, or properly, or something. It was only when I started using a web browser to navigate to the entry on SC I realised what was happening.

I also think you need to solve the problem of it not being wise to run file system utilities. The urge to find what I wanted with a

find . -iname "jetpac*"

was pretty overwhelming. Is it possible to offer a binary of pre-formatted metadata to prime the caches or something?

I'd also like to be able to access my own ZXDB instance, which I have running locally, but I guess that needs a whole pile of server side stuff implemented.

There is one significant usability flaw with it though: the files you get to are ZIPped. That means they can't be drag and dropped onto FUSE to play them. I use KDE, so can navigate to the file I want using dolphin, click in the ZIP to open it, navigate inside it to the TAP, then drag that TAP file out onto FUSE, and ... thud, error message! I not quite sure which bit of this process doesn't work, and it's clearly not your virtual FS bit, but it's still something of a show stopper that it doesn't work. Dragging the TAP out to copy it to a tmp directory and from there onto FUSE is a pain.

Also, I'm not sure what you meant in your last post by "a text widget to let you type in a search term". How does a file system have a "search widget"?

The above is offered as feedback, not complaints. It's really very good, a really nice application of technology to create a solution. If it weren't for the ZIP thing I'd probably use it quite regularly.

[descarte]

Thanks for the detailed feedback. In order:

I'll update the README about needing 'fuse3' as well. Hadn't realised you didn't get 'fusermount3' with 'libfuse'....

Good idea on the info.txt. There are a lot of empty directories generated, unfortunately. I've tried restricting the search as much as possible, but it's only as good as the data it gets back....

I did muse about a cached version but decided to avoid it for the first release. I think it'd be easiest to write a script that generates all the JSON from the SQL dump and store it in the cache directory that the filesystem references. There is some caching in play which caches the "by-letter" data, but it doesn't cache the game JSON at the moment. Easy enough to change that though. The basic mechanisms are all in there already. If you want to volunteer putting a script together, let me know There'll need to be some sort of ability to regenerate caches in a sensible way, however, when there's a new release of ZXDB. I asked Thomas to add a /version REST endpoint for tracking this sort of stuff (which he has done).

To access a local instance, you'd need the code for the REST interface as well plus ElasticSearch. Again, it's possible to alter the filesystem code to use, say, SQLite instead of HTTP but that's not something I'm going to implement myself, to be honest.

ZIPs should load in Fuse directly. That's what the Hermit Retro ZXZero code is doing. Just do an "Open" in Fuse, navigate to the TAP and select it. It'll load direct. You don't need to manually decompress, as far as I know. No idea why drag and drop doesn't work. Possibly not hooked in Fuse, perhaps.

The "text widget" bit is the integration in Fuse (the emulator). It's nothing to do with the underlying filesystem. From the point of view of whatever the application is, it's just looking at files.