File sizes in ZXDB

This is the place for general discussion and updates about the ZXDB Database. This forum is not specific to Spectrum Computing.

Moderator: druellan

Post Reply
Patrik Rak
Microbot
Posts: 112
Joined: Mon Apr 13, 2020 3:07 pm

File sizes in ZXDB

Post by Patrik Rak »

Hello there, I was wondering, is there any reason why the non-WoS files listed in ZXDB do not have their file sizes filled in?

I have just recently finally converted ZXDB into ZXDS listings, so ZXDS users can conveniently access them, however the lack of file sizes makes the experience a bit lacking. Of course, I could perhaps pick the sizes by doing HEAD requests for each such URL (and risk getting blocked), but it would be much easier if they were already stored in the database.

Thanks for considering this.

Patrik
User avatar
Einar Saukas
Bugaboo
Posts: 3070
Joined: Wed Nov 15, 2017 2:48 pm

Re: File sizes in ZXDB

Post by Einar Saukas »

Patrik Rak wrote: Mon Apr 13, 2020 3:13 pm Hello there, I was wondering, is there any reason why the non-WoS files listed in ZXDB do not have their file sizes filled in?
Just for convenience. It's easier for us to add new files to ZXDB if we don't need to keep track of file sizes. Also it gives us the freedom to update games by just replacing a file with the same name, without having to wait for a new ZXDB release. For instance Pavel is constantly updating game inlays with even better scans, without depending on ZXDB updates to do so. Whenever Alessandro Grussu updates one of his games, he sends us literally dozens of files to be replaced, and it would be more work for us if we had to update their sizes in ZXDB.

Basically we never had a reason to keep track of new file sizes. Although we preserved the file size information from old WoS files that was already available.

Patrik Rak wrote: Mon Apr 13, 2020 3:13 pm I have just recently finally converted ZXDB into ZXDS listings, so ZXDS users can conveniently access them, however the lack of file sizes makes the experience a bit lacking. Of course, I could perhaps pick the sizes by doing HEAD requests for each such URL (and risk getting blocked), but it would be much easier if they were already stored in the database.
Why exactly do you need this information? If it's really important for you, we can review our process to see if we can keep track of this...
User avatar
PeterJ
Site Admin
Posts: 6854
Joined: Thu Nov 09, 2017 7:19 pm
Location: Surrey, UK

Re: File sizes in ZXDB

Post by PeterJ »

SC is on an Ubuntu Server and the linux ls (same as dir in Windows) command is very flexible so if it helps I could export a list of all files with sizes including subdirectories, then pipe it to a text file with ls -laShR
Patrik Rak
Microbot
Posts: 112
Joined: Mon Apr 13, 2020 3:07 pm

Re: File sizes in ZXDB

Post by Patrik Rak »

Einar Saukas wrote: Wed Apr 15, 2020 5:37 pm Why exactly do you need this information? If it's really important for you, we can review our process to see if we can keep track of this...
In ZXDS, you can access online files by browsing directories as if they were local directories, except that their content is in fact fetched from remote servers. Both the files themselves as well as the content of these "virtual" directories are fetched this way.

Normally, the browsing requester shows both name and file size, as you would expect. Without knowing the size, all I can show is a file size 0. It is not critical, but knowing the size gives you a hint how large the file is, before downloading it. For example. if the demo is micro demo entry or if a RZX is small or huge.

I really don't need it for anything else, it doesn't really have to match the actual size at all, it's just a convenience for the users. So I wouldn't mind if it is not completely up to date, some size would still be better than no size for me...
PeterJ wrote: Wed Apr 15, 2020 6:30 pm SC is on an Ubuntu Server and the linux ls (same as dir in Windows) command is very flexible so if it helps I could export a list of all files with sizes including subdirectories, then pipe it to a text file with ls -laShR
Well, if you could provide me with such a list, I could combine it with the database entries myself, just for the use of ZXDS. OTOH, if that was part of the database, other people might benefit from it, too, I guess. Especially if the fields for that are already all in place.

Thanks for considering this.

Patrik
User avatar
Einar Saukas
Bugaboo
Posts: 3070
Joined: Wed Nov 15, 2017 2:48 pm

Re: File sizes in ZXDB

Post by Einar Saukas »

OK, we got this sorted out (thanks [mention]PeterJ[/mention]!) and it will be in the next ZXDB release.
Patrik Rak
Microbot
Posts: 112
Joined: Mon Apr 13, 2020 3:07 pm

Re: File sizes in ZXDB

Post by Patrik Rak »

Big thanks, guys!

BTW, if you started tagging the releases on github, people could get notified whenever there is a new release...

https://help.github.com/en/github/admin ... repository

Thanks again,

Patrik
User avatar
Einar Saukas
Bugaboo
Posts: 3070
Joined: Wed Nov 15, 2017 2:48 pm

Re: File sizes in ZXDB

Post by Einar Saukas »

Patrik Rak wrote: Wed Apr 22, 2020 9:54 am BTW, if you started tagging the releases on github, people could get notified whenever there is a new release...
I always tag software releases (for instance see ZX-Ed).

However ZXDB is not a software. The concept of "release" isn't really appropriate here.

In ZXDB, every commit is a "release", and every "release" is the entire content of the ZXDB repository.

If you want to receive notifications, either subscribe to this thread at SC forum (I always notify all ZXDB updates here), or fork ZXDB at github so you can enable commit notifications there.
Patrik Rak
Microbot
Posts: 112
Joined: Mon Apr 13, 2020 3:07 pm

Re: File sizes in ZXDB

Post by Patrik Rak »

Hmm, without wanting to nitpick, checking the commit history, I see several commits between "release" commits from time to time. :)

Just using git tag with the appropriate version would do all the work (creating the zip and tarball for that version, notifying subscribers, etc.) and you could easily refer to each version retrospectively if needed. If it is software or not is irrelevant IMO, you are using versions already, so you perhaps do that for some reason...

But if you think it's too much hassle, no problem. I sure can keep track of new releases in one way or another. ;)
Patrik Rak
Microbot
Posts: 112
Joined: Mon Apr 13, 2020 3:07 pm

Re: File sizes in ZXDB

Post by Patrik Rak »

BTW, thanks for adding the file sizes. I have regenerated the ZXDS listings from the latest release and it makes quite a difference!
User avatar
Einar Saukas
Bugaboo
Posts: 3070
Joined: Wed Nov 15, 2017 2:48 pm

Re: File sizes in ZXDB

Post by Einar Saukas »

You are welcome!
Post Reply