Wikidata/Development/JS Library

This page represents the current status of discussions within the Wikidata project team. It is used as a base for development. To avoid confusion please only update it to reflect updates of this status. If you want to discuss it, please use the Wikidata mailing list or the talk page. Thank you!

During Phase One of Wikidata, one of the main goals was to get a sufficiently functional and pretty first version of the user interface for editing basic aspects of data items working. Due to tight timelines and early conceptional uncertainties, this has evolved into a rather clustered and frontend-centric JavaScript codebase.

This page represents a proposal to separate the Wikidata UI implementation (widgets) from what could be described as JavaScript backend. Both parts combined would make the Wikibase JS Library. The main goals of this proposal are to ease further development of the Wikibase frontend as well as offering support to the gadget development community sooner rather than later.

Note:

This proposal does not yet fully embrace all Phase Two aspects but it should be flexible enough to allow building on top of it without requiring huge conventional changes as long as the overall data model doesn't change in any drastic ways. It aims to finish Phase One gracefully, ready for Phase Two to build on it as specs are getting more solid, allowing concentrating on the essentials rather than continuing struggling with current inflexibilities and restrictions.

Basic Principles

• The Wikibase JS Library will follow the Wikidata data model.
• Depends on MediaWikis existing built-in JavaScript modules.
• Attempt to stay close to the PHP backend implementation without abandoning proven JavaScript patterns and without the goal or guaranty to offer a full coverage of the PHP backends diversity of the data model.
• Makes use of the Wikidata web-API for communication with the wiki.
  • It is desirable to keep the required API queries as well as the size of the actual traffic as low as possible. This is achieved by not necessarily fully loading entities but instead fetching parts of their contents of one revision in chosen languages only.
  • Uses the API to validate and normalize data. The basic principle is that data values cannot be manipulated directly. They can be handed around, removed from and assigned to entities. Their values can usually not be changed directly. Instead, new data values can be created through API validation and normalization of provided raw values. This assures that entity objects are always valid since they can only consist of valid data values.

Restrictions

• This is not a framework providing automated real time synchronization between items on the server and on the client side. This is simply not desired since the Wikibase edit interface does not work anything like that.
• Lots of the validation and normalization is happening server-sided and therefore web-API calls are required. It would be possible to do all of this on the client side as well, but it would probably cause significantly more effort in development and could be more fault-prone. However, mechanisms for doing at least some of this client-sided are part of these specs.

Client-API Modules

( new mw.Api ).wikibase or ( new mw.Api ).wb offers an abstract way of client-sided communication with the server sided web-API in JavaScript. It has to be loaded through the resource loader module wikibase.api and will then extend the mediaWiki.Api prototype as mediaWiki.Api.prototype.wikibase which just serves as a collection for all Wikibase client-API functions offered by the Wikibase JS library. This collection of functions can be split into further sub-modules of wikibase.api similar as it is done for the different MW core API modules.

Like MW core JS-API functions, the functions introduced here might be far more abstract than the actual Web-API equivalents. It is possible to have just one function using several Web-API modules depending on the functions arguments. This makes it easier for developers who do not have to grasp the full range of the Web-API when dealing with Wikibase in JavaScript. Of course it is still possible to query the Web-API directly, simply by using mw.Api's low-level functions.

Wikibase.api.getitems

This module introduces the function mw.Api.prototype.wikibase.getItems. It offers an equivalent to the server-sided wbgetitems web-API module which allows getting a set of existing items.

getItems()

Used to fetch items fully or partially.

Parameters:

• items: array or single one of mw.Title, wb.Site, wb.SiteLink or Number (item-id)
• languages: (optional) array of language codes implying in which languages the item should be fetched.
• properties (optional) array to specify which properties should be fetched.
• ... possibly more, allowing for fallback chain handling and fetching old versions of items

Promise: This will return a $.Promise with additional parameters in case of success:

• items: Array with objects of type wb.FetchedItem representing the fetched items with information about source and fetched language-coverage.

wikibase.api.storeitem

This module serves functions for storing changes on items or creating new ones on the server which were previously constructed as wb.Item in JavaScript. It is an equivalent to the server-sided wbsetitem web-API module. At the same time it allows to handle most of the use-cases of the wbsetlanguageattribute, wbsetsitelink and wbsetaliases web-API modules.

storeItem()

Can be used to store an entirely new item.

Parameters:

• item: wb.Item which was created in JavaScript and should be stored in the database.

Promise: This will return a $.Promise with additional parameters in case of success:

• item: wb.FetchedItem the verified item, stored in the database

updateItem()

Can be used to update an existing item which was previously fetched via the API or which is known otherwise. The latter option will either overwrite values of an item or extend it.

Parameters:

• item: wb.Item|wb.FetchedItem to be stored.
• targetKey: (optional) mixed, information to select the item, e.g. id of the item or a sitelink. Only necessary if not fetched item is given.
• ... possibly more, allowing for some conflict handling behavior

Promise:

This will return a $.Promise with additional parameters in case of success:

• item: wb.FetchedItem the verified item

In case of a failure, depending on the failure, the given arguments can imply further details about conflicts, depending on the initial options for conflict treatment.

wikibase.api.buildValue

This module allows building wb.dataValues.Value instances which can then be used on wb.Item. It is likely that there will be additional factories for creating data values or even unverified draft data values based on user input. The exact implementation of this system will become clearer as Phase Two discussion on the implementation of data types progresses.

This will also require additional modules in the Web-API.

Further API Modules

The above API modules who should have a high priority and are required in the libraries higher layers. More modules for developer convenience are thinkable and could easily be added later, also by the Gadget development community or other parties.

For fetching other kinds of entities than items, we could either rename respective to use the entity term instead or simply implementing similar ones for all kinds of entities. This is just a detail implementation decision which should be made with extensibility in mind (e.g. how would extensions register other kinds of entities).

JS Backend Constructors and Prototypes

This includes the description for the most important constructors and prototypes/objects required to sufficiently present the Wikibase data model in JavaScript and for implementing the mechanism for fetching entities.

Notes:

• Definitions marked with a leading * are part of the system since Phase One around Wikimania 2012 already.
• Events are implemented by using jQuery's trigger() and can be registered with jQuery's on().

wb (wikibase)

The wikibase (or the shorthand wb) singleton (similar to a static class in other languages) is representing the Wikibase extension and contains most of the extensions related constructors and objects in a tree-like fashion. It might also offer some basic events widget developers can listen to.

Prototype Members:

* getSites()    : wb.Site[] : returns all sites known to the setup
* getSite( id ) : wb.Site|null
* hasSite( id ) : Boolean
getLanguages : String[] : all languages supported for multilingual text values
getItems: wb.Item[] : returns all items (usually just one) displayed on the current page
...

Events:

* newItemCreated( event, item ) : Triggered after a new wb.Item has been stored to the database successfully.
...

wb.Site

Represent a site known to the Wikibase Repo extension.

Note: The wb.Site constructor was implemented during early Phase One already. It might need some adjustments though. All the known sites are sent to the client JS using mw.config as wbSiteDetails through the site's html. This could be done more performant by adding this as a custom resource loader module so it will be heavily cached once received by the client.

wb.Entity

wb.Entity.prototype is the base for all kinds of Wikibase entities. It can be used to produce instances representing new entities which have no information about where they come from or where they go to, it is always considered as a “new” entity. It does never directly represent an entity stored within the system. Since Entities are not technically required to hold any data at all, it is possible to have empty instances of wb.Entity or having instances representing the changes between two fetched entities.

Members: wb.Entity.SPECIAL_ENTITIES holds an array of wb.Property with respective system properties usable with this type of entity.

Prototype Members:

isEmpty() : Boolean
setContent() : wb.Entity
getType() : string
getIdPrefix() : string : returns the prefix of this kind of entity (e.g. <code>q</code> for items).
addContent( wb.Entity : content, Boolean : overwrite := false ) : wb.Entity : returns the changed content
diffTo( wb.Entity : entity ) : wb.Entity : returns the difference to another entity
...

# The following functions will be overwritten by wb.FetchedEntity in case of a fetched entity:
getId() := null
isNew() := true
sameAs( wb.Entity : entity ) := false

wb.FetchedEntity

• Inherits from wb.Entity
• implements wb.FetchableContent

wb.FetchedEntity.prototype inherits from wbEntity and implements the wb.FetchableContent object to keep track over which parts of the entity are fetched from the server and which parts are added on top of it. This allows reading the entity's data in chosen languages only but offers to load further languages on demand. wb.FetchedEntity goes even one step further and allows fetching of statements of certain types only.

wb.FetchedEntity may only be created from an entities DOM or by the Wikibase JS-API module which fetches the entity and is able to add the necessary information.

Prototype Members:

getId() : String
isNew() : Boolean : true if the content was constructed in JavaScript and is not yet stored
getFetchSource() : mw.Title (? Note: could be an Object with the title + repository info considering items of different repos could be loaded in one application )
getFetchedRevision() : Number : The revision of the contents page which holds the source of the content
getRespectiveTitle() : mw.Title : If the related content is fetched in an old version, the content could have been merged with another one in a newer revision.
getFetchedProperties() : wb.Property[] : returns a list of properties for which statements were fetched previously
fetchProperties( properties ): $.Promise : fetches additional statements using certain properties not requested previously
getFetchedBase(): wb.FetchedEntity : see wb.FetchableContent

wb.FetchableContent

One problem with entities representations in JavaScript is that it is not necessarily desired to load the item's full content and in all languages (e.g. when an item was extracted from the current data page's DOM which only holds the full data in one language usually or to save bandwidth). It also has to be tracked which version of an entity has been fetched from the web-API. This task is handled here and can be re-used for all kinds of content.

When fetching content in one language, the fallback chain can optionally be taken into account. This means some parts of other languages might be fetched as well but these languages won't be marked as fully fetched except if they really are.

Note:

It might be worth implementing this in MW core once it has proven useful since the same mechanism could be used in similar scenarios for other kinds of content.

Prototype Members:

sameAs( wb.FetchableContent : content ) : Boolean : true if same source and equal revision, independent from the amount of fetched information
getFetchSource() : mixed : returns an object somehow representing where the content was fetched from. This could be a mw.Title.
getFetchedLanguages() : String[]
fetchLanguages( languages ) : $.Promise : fetches the entities data in additional languages
update( rebase : bool ) : $.Promise : if isHead() is true, this will update the currently fetched content by using the API. rebase: allows to apply changes already made to the content on top.
hasChanges() : Boolean : true if the content has changed since last stored
isHead() : Boolean : true if the newest state of the content was requested explicitly. Of course this could have changed in the meanwhile, but since we do not sync in real-time, this is simply not considered here.
wasFullyFetched() : true if all information was requested (not only specific languages).
getFetchedBase(): mixed : returns the originally fetched content as received from the server. If no changes have been applied, the returned value is just a copy of the content.

wb.Property

Inherits from wb.Entity Instances of the wb.Property constructor represent a property in Wikibase. This can be an existent property as well as a new or modified one constructed by JavaScript. This can also represent a special (system) property such as label or description.

For system properties some additional information

Prototype Members:

isSpecial() : Boolean : Whether the object describes a property used for system-statements such as ''label'' or ''description''
getCompatibleEntityTypes() : Array : returns a list of entities which can store statements using this property. For non-special properties, this is just the ''item'' entity type.
...

wb.Item

Inherits from wb.Entity

Instances of the wb.Item constructor represent an item in Wikibase. This can be an existent item as well as a new or modified one constructed by JavaScript.

getCustomStatements() : wb.Statement[]
...

wb.Datatype

Inherits from wb.Entity Another kind of entity. See Wikidata/Data model#Datatypes for now

wb.Statement

See Wikidata/Data model#Statements for now

wb.Statement.Claim

An instance of wb.Statement.Claim simply holds a collection of snaks where one represents the main snak which can be accompanied by several optional auxiliary snaks.

wb.Snak

See Wikidata/Data model#Snaks and Wikidata/Notes/Entities_and_Snaks for now

wb.dataValues

Collection of constructors for different types of data values for different data types. One thing to point out is that most data values will be language independent. As of now, there is only the MultilingualText type which will represent text in different languages.

For values which can have different values in different languages, the same system as used for handling wb.Entity and wb.FetchedEntity applies. If an entity was fetched from the web-API, it will be available as wb.FetchedEntity to keep track of which of the entities languages were fetched as well as other information about its affiliation (e.g. source wiki). This allows fetching a few needed languages and statements only. Respectively, statements with properties having language-specific values have to keep track and offer information about which languages were fetched already. This information is only related to the fetched entity, therefore when getting it from one entity and applying it to another one, all information about wb.FetchableContent will not be transferred since they lose their meaning in the targets context.

Note:

Even though this is system is only required for one type of data value, it might be wise to implement this system for all kinds of data values. This will allow to track more information about fetched values more directly, e.g. whether a value is new. It also allows for more generic handling of data values. In the following sections, this concept is only shown on the basis of the MultilingualText type.

wb.dataValues.MultilingualText

• Inherits from probably some common abstract base value, e.g. wb.dataValues.Value

Constructor for data values representing a text in multiple languages.

Prototype Members:

isNew() := true
...

wb.dataValues.FetchedMultilingualText

• Inherits from wb.dataValues.MultilingualText
• implements wb.FetchableContent

Prototype Members:

isNew() : Boolean : true if the value was constructed in JavaScript and is not yet stored
getFetchSource() : wb.FetchableEntity

JS Backend and the User Interface

The following describes how the existing JavaScript user interface is supposed to interact with the new JavaScript backend evolving from this proposal. Basically, a lot of what this proposal covers is done by the current frontend already. Therefore, the existing code has to be adjusted to use the new library objects instead. The task of the UI widgets will be to communicate with the user, taking the user's input and turning it into valid object structures representing the data.

Things within the current UI implementation which will experience some changes:

• The whole internal web-API handling necessary for sending changes to the server can be removed from the JS frontend.
• The current validation and normalization system implemented in wb.ui.PropertyEditTool.EditableValue which is quite ugly (especially when it comes to empty values) since it is working with strings and arrays as values.
• The setValue and getValue functions which currently use the mentioned validation system. These functions will be re-designed to take and return wb.DataValue objects, also for representing empty values.
• EditableValue and EditableValue.Interface's valueCompare functions will be obsolete.
• SiteLink handling will be less complex and moves closer to other EditableValue implementations.

See also

• bug 38174[1] (links from mw m w)