ZXInfo API v3

[kolbeck]

I'm happy to announce that ZXInfo API v3 is now open to the public. With this announcement, the old thread has been locked - as discussion will continue here.

TL;DR

A strong recommendation going forward, each client of the API should identify itself by using a unique and descriptive User-Agent string.

The ZXInfo API has been updated, you can find more information and try out yourself in the documentation found here: ZXInfo V3
Going forward the previous version changes status to deprecated, which means it will no longer be update - and you can expect it to close down in 6 months time from today. I believe this timeframe is sufficient for every client to upgrade using v3.

What is ZXInfo API?
This API makes it possible to access ZXDB information, making it possible to build other types of apps without the need to understand the SQL structure of ZXDB, installing required software such as SQL database and having to maintain your own running instance of ZXDB. With this you can easily integrate ZXDB in your own programs, from simple lookup to full featured websites such as ZXInfo or apps like Zx App.

What's new?
The main changes are done to the JSON format returned by the API
- cleanup and better structure around elements returning array/list
- added more information, such as new relations introduced in ZXDB during time
- added back magazine references

A new endpoint to get list of entries based on starting letter has been added.
- /games/byletter/{letter}

The web site ZXInfo has been switched to the use the new V3 API, and some other clients has been testing as well. (You know who you are )

As always I'm open for discussion of future requirements of ZXInfo API, as I believe that community driven effort will make this the best API for accessing ZXDB data.

[kolbeck]

It's going fast now!

Since the release of ZXInfo API v3, collaboration with developers of apps using the API has been close. Most (known) users of the API are almost done upgrading to V3, and together we have already implemented some new features:

New endpoint:
/games/byletter - lets you fetch ZXDB entry by first letter.

New output format:
The new parameter 'output' can be used if JSON is not your thing. For now it supports:
- simple - just a simple list of {id, title}
- flat - a flatten version of the JSON

New constant values:
The /search endpoint, now supports the following constans to make requests simpler:
- machinetype=ZXSPECTRUM, ZX81, PENTAGON
- genretype=GAMES

New Wiki with additional documentation.

Default values for mode, size, offset and sort - which means these are no longer mandatory.

--

[kolbeck]

New possibility with ZXInfo API.

Are you not the organized type and have files over the place? Do you try to keep up with to which software title it belongs? Fear no more - you can now lookup entries in ZXDB using md5hash values.

https://api.zxinfo.dk/v3/filecheck/{hash} - and it returns found entry, it it's known. For now it knows about every file on the WoS Mirror from 2017 as well as all files on SC, even the files part of archives (ZIP files).

On many systems you can use the command 'md5' to calculate the hash on a file.

Code: Select all

md5 randomfile.tzx
MD5 (randomfile.tzx) = 82bb33587530d337323ef3cd4456d4c4

Perform lookup: https://api.zxinfo.dk/v3/filecheck/82bb ... cd4456d4c4
And it tells you the match:

Code: Select all

{
"entry_id": "0002259",
"title": "Head over Heels",
"file": {
    "filename": "Head Over Heels.tzx",
    "archive": "HeadOverHeels.tzx.zip",
    "md5": "82bb33587530d337323ef3cd4456d4c4"
    }
}

In this case the file 'randomfile.tzx' has been identified as "Head Over Heels.tzx" inside the ZIP archive "HeadOverHeels.tzx.zip" for the title "Head over Heels"

Neat?

/Thomas

[Morkin]

I've been trying to learn how to do XMLHttpRequests in Javascript.

I figured that Speccy games are the perfect subject matter to practice with, so a big thanks for providing your ZXInfo API

I'm trying to figure out how release year works - it returns it OK for most, but titles like this one seem to return a null.

Just wondered what I'm missing here? I notice that the ZXInfo record is missing this value?

[kolbeck]

Thanks for trying out the API

This case looks like an edge case, will have a look when upgrading with next ZXDB release. Stay tuned and reach out if any help needed

/T

[arjun]

I've been working on integrating ZXDB via ZXInfo into the Zero emulator. Thanks for the very cool API!

I have some queries regarding the results the API currently returns:

1) The search results are pretty extensive but not very correlated IMO. For eg: Searching for Decathlon returns the following titles in order of appearance (rel_sort):
1) Daley Thompson's Decathlon (_score: 572.3201) - the correct result!
2) Lothlorien (_score: 140.19092) - hmm?
3) Polecat (_score: 115.15386)
4) Widecat +3 (_score: 115.15386)
etc..

Similarly searching for Jet Pac returns:
1) Jetpac (_score: 1004.8825) - the correct result!
2) Mario Islands (_score: 633.50903) - wtf?
3) Jet Pac John (_score: 411.6985) - better than Mario Islands!
4) Jet Paco (_score: 303.9324) - better than both of the above?

Looking at the above, the returned results for Decathlon make no sense after the first entry. I'm not even sure how they are being matched. For Jet Pac, it looks like it's matching on the name but the second entry baffles me. How is Mario Islands a better match than the other two?

I was hoping the _score would give me an indication on how to correlate the results but based on the above a simple "higher is better" function doesn't seem to a good one. Any help in how to interpret the results would be appreciated!

The second query I have is that when looking for suitable images to correspond to the title, it's not easy to pick figure out which is an appropriate one. To illustrate: When searching for a suitable Inlay - Front image to show with the Jet Pac result, using the first entry in the returned results for Inlay - Front would give you the Microbyte inlay, which clearly isn't optimal. You would want the second entry in reality:

Code: Select all

	    {
              "path": "/pub/sinclair/games-inlays/Rereleases/j/Jetpac(Microbyte1).jpg",
              "size": 68517,
              "format": "Picture (JPG)",
              "language": null,
              "type": "Inlay - Front"
            },
            {
              "path": "/pub/sinclair/games-inlays/j/Jetpac_2.jpg",
              "size": 82518,
              "format": "Picture (JPG)",
              "language": null,
              "type": "Inlay - Front"
            },
 

But how does one arrive at that conclusion? I suppose one could scan for "Rereleases" in the "path" and ignore it if present, but I would argue this is a bit presumptuous way to do it.

I notice that Spectrum Computing does show the *correct* front inlay for Jet Pac on its page: https://spectrumcomputing.co.uk/entry/9 ... rum/Jetpac

It shows the correct inlay as the first option and the Microbyte inlay comes further down the line somewhere. I have no idea how @PeterJ has managed it if he is relying on the same info as the one returned by ZXInfo API, unless his source of zxdb info is different?

I'm inclined to think SC does use a different query for zxdb than ZXInfo, because the returned results for "JetPac" are different. For instance, "Mario Island" isn't the second entry in the results unlike ZXInfo. OTOH, "JetPac" isn't the first returned result either in SC (unlike ZXInfo) - rather it's the fifth (which is still okay given the other results do have Jet Pac in them).

Anyway, hope all that made sense! Just trying to figure out the best way to interpret the results in a meaningful way for the end-user.

[PeterJ]

Hi @arjun,

We don't use the API, we use the original ZXDB database in MySQL format.

I've played with the API via Python, but not done anything serious with it @kolbeck wrote the API, so hopefully he can help. I've not seen him on the forums for a while though.

Regarding screens, we moved towards using .scr for as many screens as possible some time back. The old gif screens are still available in other downloads.

[arjun]

We don't use the API, we use the original ZXDB database in MySQL format.

That makes sense, although I'd hoped both SC and ZXInfo API were using similar sort of queries. Doesn't look like it, so I guess I'll just have to wait for kolbeck to reply.

Regarding screens, we moved towards using .scr for as many screens as possible some time back. The old gif screens are still available in other downloads.

Looks like morkin and me are using a similar method to get the gif screens from kolbeck's website itself, which seems to work without issue. At least for now. Cheers!

[kolbeck]

We don't use the API, we use the original ZXDB database in MySQL format.

That makes sense, although I'd hoped both SC and ZXInfo API were using similar sort of queries. Doesn't look like it, so I guess I'll just have to wait for kolbeck to reply.

Regarding screens, we moved towards using .scr for as many screens as possible some time back. The old gif screens are still available in other downloads.

Looks like morkin and me are using a similar method to get the gif screens from kolbeck's website itself, which seems to work without issue. At least for now. Cheers!

Yes, I'm still here - and the ZXInfo is very much still alive, just real life kicking in....
The short version is, that ZXInfo does not use SQL to lookup as it is based on a real search engine - so they will never get the same result when doing the search/lookup. Think about google - it always gives you something back

One thing you could try is to use the secret "titlesonly=true" which make the search only look for titles. You are more than welcome to contact me privately, in case you need more clairty

/Thomas

[kolbeck]

Just been updating and tweaking...

Without limiting the search to titles only, searching for 'Decathlon" now gives the following result:

* Daley Thompson's Decathlon (a good match)
* Hitsville (a match, because Daley Thompson's Decathlon is mentioned in the comments)
* Sports Day (a match because Decathlon is mentioned in the comments)

And for 'Jet Pac'

* Jetpac (good match, and probaly what we were looking for)
* Jet Pac John (because it stills contains what we searched for)
* Bac-Pac (a match, because Pac is part of what is searched for)

Before tweaking, Mario Islands was actually a valid hit - as one of the authors is named 'Jet', but this type of match is now lower priority. The reason for trying the "titlesonly=true" flag, as it limits the search to titles. However, this disables the possibility to find entries by a specific author - try for example to search for 'jon ritman'

As can be seen, the engine indexes the different parts of the search term and matches them against the data. For example if searching for 'beach head' on SC, you only get the two games 'beach head i + ii'. ZXInfo goes beyond that, and also returns titles that contains either beach or head in the title, giving a richer and wider output from searching.

Hope it gave some clarity on the approach for ZXInfo

/T

[kolbeck]

Hello, just a short message before going on vacation

The ZXInfo API is still very much alive and I'm happy to be able to announce two new clients

* Microdrive Emulator OqtaDrive - emulates a bank of 8 Microdrives for use with a Sinclair ZX Spectrum with Interface 1, or with a Sinclair QL.
* ZX Spectrum Gamer for Android and iOS - an app that makes it easier to access Sinclair ZX Spectrum games and magazines

/Thomas

[Einar Saukas]

Cool!!!

[Einar Saukas]

I'm trying to figure out how release year works - it returns it OK for most, but titles like this one seem to return a null.

Just wondered what I'm missing here? I notice that the ZXInfo record is missing this value?

Sorry I never noticed this question before...

This program has an indirect original release date. Since this program was originally released as part of a covertape, the original release date must be obtained from the magazine that published it.

Technically all original release dates (direct or indirect) can be easily obtained using this query:

Code: Select all

select * from entries e 
inner join search_by_origins s on e.id = s.entry_id
where e.id = ?

[Morkin]

Ah cool - that explains it -thanks..!

[kolbeck]

Sorry, forgot about this as well - ZXInfo has just been updated with newest release, and will also now consider indirect release date.

[kolbeck]

While looking at the indirect release date, I found this example

It contains different release date for "release" and for indirect release. Release year 1988 vs 1989. @Einar Saukas - what is suggested "correct" date to display in this case?

[Einar Saukas]

While looking at the indirect release date, I found this example

This particular case is tricky. It was a fake title in 1988 but a real title in 1989

However ZXDB still contains quite a few inconsistencies like this, that we inherited from Martijn's old WoS. There's a script in ZXDB called "ZXDB_health_check.sql" that lists all of them. We are slowly going through each case, checking and fixing them.

It contains different release date for "release" and for indirect release. Release year 1988 vs 1989. @Einar Saukas - what is suggested "correct" date to display in this case?

There's no general rule, thus the reason we are still checking them one-by-one.

SpectrumComputing currently displays the direct release date in these cases. I suggest you do the same for now, otherwise people may think there's something wrong with the API.

[kolbeck]

SpectrumComputing currently displays the direct release date in these cases. I suggest you do the same for now, otherwise people may think there's something wrong with the API.

Thanks @Einar Saukas - that's also how it is implemented

/Thomas

[terka]

Hello! Thanks for the beautiful api.

Can you please tell me more about file hash search? Seems like a search going across TOSEC and ZXDB entries? Their files have different md5 and filesize. So you have both ZXDB and Lady Eclipse TOSEC sqlite3 database available for search?

[kolbeck]

Hello! Thanks for the beautiful api.

Can you please tell me more about file hash search? Seems like a search going across TOSEC and ZXDB entries? Their files have different md5 and filesize. So you have both ZXDB and Lady Eclipse TOSEC sqlite3 database available for search?

The general search does not know about TOSEC, BUT in the API all TOSEC files has been indexed and added to the relevant entry. This means that the filehash search knows about them as well as files in ZXDB, and yes TOSEC files are (mostly) different from the files you will find on SC - which explains different md5 and filesize.

/Thomas

[kolbeck]

ZXInfo API v4

The next iteration on the API has started with the following main topics:
* rename a few endPoints to make it more logic (eg. /games/ becomes /entries/ - which is the correct ZXDB term)
* adding a few specific search endPoints - for example, if you want to search within titles only
* consistency in filtering and aggregations across all search functions
* splitting "category" into main category and subcategory for use in filtering
* updating the API documentation to include v4
* clean-up

One new feature will be that search expands to screens as well. Work has been started to "scan" all screenshots for text to be indexed and searchable. For now it only covers text written in the typical standard 8x8 font style and a few custom fonts as well, but over time hopefully all 8x8 fonts will be recognized for text indexing. I'm not sure this feature has any real value, but fun it is

If you want to see it in action, head over to ZXInfo and search for the famous "moonprince" - and again, this is the first iteration, lots of custom fonts in games etc. has not yet been detected (slow process).

What is ZXInfo API?
This API makes it possible to access ZXDB information online, making it possible to build other types of apps without the need to understand the SQL structure of ZXDB, installing required software such as SQL database and having to maintain your own running instance of ZXDB. With this you can easily integrate ZXDB in your own programs, from simple lookup to full featured websites such as ZXInfo or apps like Zx App.
Currently there are around 10 clients using the API in very different ways.

The current v3 API will continue to be working with updated ZXDB data, as for now no date has been set for decomission

[electricbolt]

Hi,

The v3 swagger doc does not define any json response structures. Is this something we could get added to the swagger doc? The lack of json response structures means that any generated client libraries are suboptimal. As a test I defined a response for getSuggestions:

Code: Select all

  /suggest/{term}:
    get:
      tags:
        - "zxinfo"
      summary: "Returns list of suggestions"
      description: |
        Fetches a list of suggestions for input term. Returns suggestions for titles, publishers and authors to be used in type-as-you-go search fields.

      operationId: "getSuggestions"
      produces:
        - "application/json"
      parameters:
        - name: "term"
          in: path
          description: input for suggestions
          type: string
          required: true
      responses:
        # Response code
        200:
          description: OK
          schema:
            $ref: '#/definitions/suggestions'

...

components:
  schemas:
    suggestions:
      type: array
      items:
        type: object
        properties:
          text:
            type: string
            description: Suggestion text
          labeltype:
            type: string
            description: Person,Company,<blank>,...
          type: 
            type: string
            description: AUTHOR,SOFTWARE,PUBLISHER
          entry_id:
            type: string
            description: Optional ZXDB entry id

Is there a github repo that PRs could be done against?

Thanks,
Matthew.

[kolbeck]

Hi @electricbolt

It's not something I have planned, as the output is a mixture of Elasticsearch type mappings and specific Elasticsearch output - out of my control. However if you want to give it a try - have a look at the swagger on GitHub