New Database Model ZXDB

This is the place for general discussion and updates about the ZXDB Database. This forum is not specific to Spectrum Computing.

Moderator: druellan

Post Reply
User avatar
druellan
Dynamite Dan
Posts: 1466
Joined: Tue Apr 03, 2018 7:19 pm

Re: New Database Model ZXDB

Post by druellan »

R-Tape wrote: Fri May 24, 2019 9:57 am As the main person currently doing corrections at the moment, what do you prefer @druellan ? Would it be useful to have a volunteer (perhaps @StooB ?) to keep on top of a finalised corrections spreadsheet, that is basically a list of ZXDB ID and a concise explanation of the correction needed? (Or would you rather be left alone to get on with it?!)
I really don't like the idea to have the information spread on different mediums, that usually end up with discrepancies and an overhead of work. I like the flexibility of the public spreadsheet, but:

A spreadsheet is really not a good bug reporting platform at all, it makes the information difficult to post, edit and read. Not everyone has a Google account, that means to leave everything open and editable, and that's kind of concerning, not because someone is going to sabotage or post nonsense (can happen), but it is easy delete or alter the information by mistake if you are just looking.

The sheet works best as an index for all the bugs reported and solved (or not) on the forum. I'm already doing that myself, just to keep track on everything I'm looking at, perhaps is not a big deal to open a public spreadsheet and add everything there.

Could be nice to use such a sheet as a triage process for the reports, but makes no sense of doing that if I'm the only one focusing on this area.

Besides that, I think we still need to inform something to the person that posted the report, without the hassle of looking on a spreadsheet, this also helps me to quickly catch posts I missed, something that can happen when a discussion fires up. If we can't find a plugin or a hack to "mark" a post, the best way to do it is just edit the post and add some kind of tag, like [IN PROGRESS] or [DONE]. Those tags can also be color PNGs and added to the forum as Smilies.

But for my own requirements, the "like" approach works, so we can start with a public sheet of reports solved or in progress if you like, and go on from there.
User avatar
druellan
Dynamite Dan
Posts: 1466
Joined: Tue Apr 03, 2018 7:19 pm

Re: New Database Model ZXDB

Post by druellan »

druellan wrote: Fri May 24, 2019 2:16 pm The sheet works best as an index for all the bugs reported and solved (or not) on the forum. I'm already doing that myself, just to keep track on everything I'm looking at, perhaps is not a big deal to open a public spreadsheet and add everything there.
Just to be clear, I'm doing this by REPORT, and not by ENTRY, using the post ID and author as reference. By entry can be problematic, since a report can target a company or a publisher, not only a title.
User avatar
Einar Saukas
Bugaboo
Posts: 3070
Joined: Wed Nov 15, 2017 2:48 pm

Re: New Database Model ZXDB

Post by Einar Saukas »

druellan wrote: Fri May 24, 2019 2:16 pmThe sheet works best as an index for all the bugs reported and solved (or not) on the forum. I'm already doing that myself, just to keep track on everything I'm looking at, perhaps is not a big deal to open a public spreadsheet and add everything there.
Instead of opening a new public spreadsheet, you could simply share your already existing spreadsheet, as read-only. Thus you wouldn't have any extra work besides what you are already doing.

Although I'm still convinced the ideal solution would be some kind of "fixed" tag in each post.
User avatar
R-Tape
Site Admin
Posts: 6353
Joined: Thu Nov 09, 2017 11:46 am

Re: New Database Model ZXDB

Post by R-Tape »

Sorry [mention]druellan[/mention] [mention]Einar Saukas[/mention] , I actually meant someone else (not me, you, or Dru) does the work of distilling correction discussions into a concise list (spreadsheet or whatever), so that it's easier for Dru (or myself when I find time) to work through them.

My thinking was that it is already enough work for someone to create the SQL for the corrections without having to constantly keep on top of the forum discussions.

But maybe Dru is already happy with the current situation. Sorry! I don't know enough about what's happening there, corrections are not my priority, and I shouldn't have poked my nose in :D
hikoki
Manic Miner
Posts: 576
Joined: Thu Nov 16, 2017 10:54 am

Re: New Database Model ZXDB

Post by hikoki »

[mention]druellan[/mention] you may want to give bookmarks.google.com a try as it allows to add tags and comments to every post url that you bookmark. From time to time you could export to html and publish that page. An example: speccy21.updog.co/googlebookmarks.html
User avatar
PeterJ
Site Admin
Posts: 6852
Joined: Thu Nov 09, 2017 7:19 pm
Location: Surrey, UK

Re: New Database Model ZXDB

Post by PeterJ »

Lets wait for [mention]druellan[/mention] to reply until we go into detailed options. I'm a fan of keep it simple.
User avatar
druellan
Dynamite Dan
Posts: 1466
Joined: Tue Apr 03, 2018 7:19 pm

Re: New Database Model ZXDB

Post by druellan »

PeterJ wrote: Fri May 24, 2019 4:02 pm Lets wait for @druellan to reply until we go into detaied options. I'm a fan of keep it simple.
I'm OK on keeping a spreadsheet about the reports I'm working on. Just for the record, I'm currently not using a sheet but just a plain simple TXT file, but I have no problem on using a sheet instead.

About a way to proper "mark" the posts I'm working on, the "like" approach is not ideal but works for me, but for other users is kind of confusing, since all "likes" look identical, so, a better way to mark those reports is welcome.

If you don't want to mess with the forum and install a MOD, there are a couple of solutions involving Javascript that might work (just detect a post was liked by me an trigger some actions), but the simplest way is to edit the original POST (I'm going to need permissions to do that on the topic) and add a label like [DONE], [WON'T FIX], etc. We can use the smilies to add a nice graphical one.
User avatar
PeterJ
Site Admin
Posts: 6852
Joined: Thu Nov 09, 2017 7:19 pm
Location: Surrey, UK

Re: New Database Model ZXDB

Post by PeterJ »

druellan wrote: Sun May 26, 2019 12:07 pm
PeterJ wrote: Fri May 24, 2019 4:02 pm Lets wait for @druellan to reply until we go into detailed options. I'm a fan of keep it simple.
About a way to proper "mark" the posts I'm working on, the "like" approach is not ideal but works for me, but for other users is kind of confusing, since all "likes" look identical, so, a better way to mark those reports is welcome.

If you don't want to mess with the forum and install a MOD, there are a couple of solutions involving Javascript that might work (just detect a post was liked by me an trigger some actions), but the simplest way is to edit the original POST (I'm going to need permissions to do that on the topic) and add a label like [DONE], [WON'T FIX], etc. We can use the smilies to add a nice graphical one.
OK, [mention]druellan[/mention] now has appropriate rights to this sub-forum so will be able to edit posts to mark as appropriate.
User avatar
Einar Saukas
Bugaboo
Posts: 3070
Joined: Wed Nov 15, 2017 2:48 pm

Re: New Database Model ZXDB

Post by Einar Saukas »

A new ZXDB update is available!

This new version contains game review scores from all major magazines (many thanks to Chris from ZXSR for this information!), an updated list of all available RZX recordings (many thanks to Daren from RZX Archive for this information!), and additional hires inlays (many thanks to Pavel as usual!)
User avatar
PeterJ
Site Admin
Posts: 6852
Joined: Thu Nov 09, 2017 7:19 pm
Location: Surrey, UK

Re: New Database Model ZXDB

Post by PeterJ »

This latest update has been applied. Thanks again to Chris AKA [mention]Vampyre[/mention] for giving us access to the ZXSR database, [mention]Daren[/mention] for all his videos, and [mention]pavero[/mention] for the continuous flow of updated images. We will be working on adding the additional fields from ZXSR to the game pages.
User avatar
Juan F. Ramirez
Bugaboo
Posts: 5101
Joined: Tue Nov 14, 2017 6:55 am
Location: Málaga, Spain

Re: New Database Model ZXDB

Post by Juan F. Ramirez »

Einar Saukas wrote: Sat Jun 01, 2019 5:43 am A new ZXDB update is available!

This new version contains game review scores from all major magazines (many thanks to Chris from ZXSR for this information!)
Great, a cool feature quite expected (at least for me!)
User avatar
PeterJ
Site Admin
Posts: 6852
Joined: Thu Nov 09, 2017 7:19 pm
Location: Surrey, UK

Re: New Database Model ZXDB

Post by PeterJ »

We recently spoke about the 'little bug' threads and some way of knowing they have been processed.

[mention]druellan[/mention] (who kindly tracks and maintains the changes) now has rights to edit posts in the 'ZXDB Discussion' sub-forum (Note - he can edit this section of the forum only).

When a change is accepted and has been processed for entry into ZXDB he will add the following to the end of the post:
✓ Checked
User avatar
Einar Saukas
Bugaboo
Posts: 3070
Joined: Wed Nov 15, 2017 2:48 pm

Re: New Database Model ZXDB

Post by Einar Saukas »

Yet another ZXDB update is available!

This release mostly contains latest bugfixes compiled by [mention]druellan[/mention] based on people's contributions in this forum. :)
User avatar
R-Tape
Site Admin
Posts: 6353
Joined: Thu Nov 09, 2017 11:46 am

Re: New Database Model ZXDB

Post by R-Tape »

Einar Saukas wrote: Wed Jun 05, 2019 3:23 am Yet another ZXDB update is available!

This release mostly contains latest bugfixes compiled by @druellan based on people's contributions in this forum. :)
Fantastic! Bloody well done Dru! You're the Tasmanian Devil of bugfixes (erm—in a good way that is).
User avatar
druellan
Dynamite Dan
Posts: 1466
Joined: Tue Apr 03, 2018 7:19 pm

Re: New Database Model ZXDB

Post by druellan »

R-Tape wrote: Wed Jun 05, 2019 9:45 am Fantastic! Bloody well done Dru! You're the Tasmanian Devil of bugfixes (erm—in a good way that is).
And the site didn't explode! yay!
User avatar
Einar Saukas
Bugaboo
Posts: 3070
Joined: Wed Nov 15, 2017 2:48 pm

Re: New Database Model ZXDB

Post by Einar Saukas »

In the next upcoming ZXDB update, column "formattype_id" won't be used anymore. This column will still exist in all file-related tables (so it doesn't break SQL queries in sites that were not updated yet), but it will be always NULL.
User avatar
Einar Saukas
Bugaboo
Posts: 3070
Joined: Wed Nov 15, 2017 2:48 pm

Re: New Database Model ZXDB

Post by Einar Saukas »

A new ZXDB update is available!

Special thanks to [mention]R-Tape[/mention] (new titles) and [mention]pavero[/mention] (hires inlays).
User avatar
Einar Saukas
Bugaboo
Posts: 3070
Joined: Wed Nov 15, 2017 2:48 pm

Re: New Database Model ZXDB

Post by Einar Saukas »

I forgot to mention that I have fixed references to magazine "Personal Computer News" in this last update!

These references were originally imported from Martijn's WoS database. The problem is, "Personal Computer News" was a weekly magazine, but Martijn's WoS only supported monthly magazines, so it couldn't distinguish between different issues released during the same month.

For instance, if you look at references to "Personal Computer News" for "Pssst" at WoS, you will notice that these full page links are missing:

http://www.worldofspectrum.org/infoseek ... id=0009401

Now if you look at the magazine sections for the same game at SC, you will notice that all "Personal Computer News" links work properly:

https://spectrumcomputing.co.uk/index.p ... 96&id=9401

For each page, you have the option to open just the specific page image, or download the entire magazine issue PDF, or view the PDF directly inside the browser. Or you can click on the magazine issue to see a list of other game references in that specific issue. Or even click on the magazine name to get a list of all magazine issues...

If this explanation above sounds confusing, just try it yourself. It's quite intuitive in practice.

Support for magazines is certainly not a new feature at SC. It has existed for ages, but I bet lots of users never noticed it :)
User avatar
PeterJ
Site Admin
Posts: 6852
Joined: Thu Nov 09, 2017 7:19 pm
Location: Surrey, UK

Re: New Database Model ZXDB

Post by PeterJ »

Einar Saukas wrote: Tue Jun 18, 2019 10:55 pm I forgot to mention that I have fixed references to magazine "Personal Computer News" in this last update!
Fantastic [mention]Einar Saukas[/mention]

PCN was always one of my favourites. It had a very different style to the other weeklies.
User avatar
djnzx48
Manic Miner
Posts: 729
Joined: Wed Dec 06, 2017 2:13 am
Location: New Zealand

Re: New Database Model ZXDB

Post by djnzx48 »

Not sure if this is a bug or not, but in the past I've noticed the magazine references often link to one past the intended page. For example, the Pssst link takes me to page 79 when the actual reference is on page 78. It seems like the page numbers in archive.org links start at zero, with no page number specified for the first page, so subtracting one from the page number in the SC links might fix the problem.
User avatar
Einar Saukas
Bugaboo
Posts: 3070
Joined: Wed Nov 15, 2017 2:48 pm

Re: New Database Model ZXDB

Post by Einar Saukas »

djnzx48 wrote: Wed Jun 19, 2019 6:27 am Not sure if this is a bug or not, but in the past I've noticed the magazine references often link to one past the intended page. For example, the Pssst link takes me to page 79 when the actual reference is on page 78. It seems like the page numbers in archive.org links start at zero, with no page number specified for the first page, so subtracting one from the page number in the SC links might fix the problem.
Yes, that's a problem. And it's something that needs to be fixed in ZXDB, not SC.

Let me explain:

When you click on a page number, SC opens the corresponding image file. So if it needs to show page 4 of issue #52 of magazine X, it simply has to open a file like "magx05200004.jpg" or something similar. That's easy.

However when you click on the "VIEW" link next to the page number, SC will try to open the corresponding page of the PDF file inside the browser. But how can it find out what's the corresponding page?

* In certain magazines, the cover is not included in numbering. Therefore page number 4 is actually the 5th page of the PDF file (as if cover was page number zero).

* In other magazines, the cover is considered as page number 1. Therefore page number 4 coincides with the 4th page of the PDF.

* In other magazines, page number 4 could be the 8th page of the PDF, due to additional index and advertising pages without numbering at the beginning (as if cover was page number -3), for instance.

The best way to solve this problem is to store this information in ZXDB for each magazine issue. The magazines currently indexed as PDF in ZXDB are listed below:

Crash

Jogos 80

Micro Mart

Mundo Spectrum

Personal Computer News

Planeta Sinclair Almanaque

RetroMagazine

Sinclair User

Spectrum Today (EN)

The Spectrum Show

If anyone is willing to help, by checking "what would be the corresponding page number for the cover?" for even a few of these issues, it would help us a lot!
User avatar
PeterJ
Site Admin
Posts: 6852
Joined: Thu Nov 09, 2017 7:19 pm
Location: Surrey, UK

Re: New Database Model ZXDB

Post by PeterJ »

Thanks for the great explanation [mention]Einar Saukas[/mention]

When I go to our MicroMart Page (I didn't know that was included in ZXDB).

https://spectrumcomputing.co.uk/index.p ... mag_id=280

I click on any PDF and it says Page not Found on archive.org. Is that something with the path? Every PDF link seems to be trying to go to:

https://archive.org/download/Micro-Mart ... pecial.pdf
User avatar
djnzx48
Manic Miner
Posts: 729
Joined: Wed Dec 06, 2017 2:13 am
Location: New Zealand

Re: New Database Model ZXDB

Post by djnzx48 »

Einar Saukas wrote: Wed Jun 19, 2019 7:25 pm
djnzx48 wrote: Wed Jun 19, 2019 6:27 am Not sure if this is a bug or not, but in the past I've noticed the magazine references often link to one past the intended page. For example, the Pssst link takes me to page 79 when the actual reference is on page 78. It seems like the page numbers in archive.org links start at zero, with no page number specified for the first page, so subtracting one from the page number in the SC links might fix the problem.
Yes, that's a problem. And it's something that needs to be fixed in ZXDB, not SC.
OK, I didn't know whether the links were stored in the database or autogenerated on the website. But are you sure the problem isn't just a simple off-by-one error? Having a brief look, all the magazines I've seen so far have the correct page numbers on archive.org, except for:

PCN: the numbering starts at the contents page rather than the cover. So the archive.org numbers are two pages ahead.

The Spectrum Show Magazine (issues #0 and #1 only): the numbering starts on the next page after the cover. So the archive.org numbers are one page ahead.

In the archive.org URLs, #page/n1 refers to the second page of the magazine, #page/n2 refers to the third, and so on. For the title page, this parameter is simply omitted.

EDIT: Only, some magazine scans don't even have consistent numbering within themselves. For example, this Crash #40 scan has an extra page inserted after page 13 (some kind of fold-out poster?) Without verifying every scan, you can't be sure whether the page numbers are all correct.
User avatar
djnzx48
Manic Miner
Posts: 729
Joined: Wed Dec 06, 2017 2:13 am
Location: New Zealand

Re: New Database Model ZXDB

Post by djnzx48 »

PeterJ wrote: Wed Jun 19, 2019 9:27 pm Thanks for the great explanation @Einar Saukas

When I go to our MicroMart Page (I didn't know that was included in ZXDB).

https://spectrumcomputing.co.uk/index.p ... mag_id=280

I click on any PDF and it says Page not Found on archive.org. Is that something with the path? Every PDF link seems to be trying to go to:

https://archive.org/download/Micro-Mart ... pecial.pdf
It's because all the links have '-Special' appended to them, when only a few of them are actually specials. Try for instance the 2015/1/22 link and it works.
hikoki
Manic Miner
Posts: 576
Joined: Thu Nov 16, 2017 10:54 am

Re: New Database Model ZXDB

Post by hikoki »

I wonder if the archive.org api search could be used for page numbers.
https://openlibrary.org/dev/docs/bookurls
I guess the search term would have to contain footer or header words, characteristic of every magazine.

For example, a search link for page 54 on a Crash magazine containing the quoted term "54 Crash"
https://archive.org/details/crash-magazine-40/search/%2254+crash%22
Last edited by hikoki on Thu Jun 20, 2019 10:31 am, edited 5 times in total.
Post Reply