Grants:IEG/StrepHit: Wikidata Statements Validation via References/Renewal


This project is requesting a 12-month renewal of the grant, to continue work in the areas described below.

The Whys and Wherefores

edit

StrepHit[1] was born to improve the quality of Wikidata.
Serving as an artificial intelligence, its mission is to read authoritative Web sources, understand them, and ultimately guarantee reliable references that corroborate Wikidata content.
So far, StrepHit has read 1.8 million pages across 53 sources, and has produced more than 2.6 million Wikidata facts.[2]

Of course, StrepHit is not as intelligent as a human, and its understanding should be validated by the community.
In principle, such operation may be achieved via the primary sources tool:[3] not only it offers a data curation facility to Wikidata editors, but it also plays a crucial role in the inclusion of external data releases.

However, the tool is definitely lacking adoption.[4]
Feedback collected up to now motivates the need for a substantial amount of work.[5]

Therefore, StrepHit and its sister datasets are simply unprofitable if the tool is unusable!

Rationale

edit

The primary sources tool was first built by Google to facilitate the eventual migration of Freebase to Wikidata.[6]

The tool is actually conceived to host more datasets and ultimately aims at standardizing the data release flow from third-party providers.[7] StrepHit has pioneered this direction via its Web Sources Knowledge Base[8] and by engaging more data donors, as part of its measures of success.[9]

Critical Issues

edit

We believe the tool has a huge potential, which is however flawed by the following critical issues.

  1. It is frustrating and almost unusable from an end-user perspective;[10][11]
  2. its maintenance and improvement processes are fairly sub-optimal, as:
    1. the core team at Google has explicitly stated[12][13][14] not to have enough time to tackle bugs and feature requests;
    2. the pull request/merge flow is insufficient alone, since both the back-end and the front-end must also be deployed in production, eventually by someone else.

This is supported by several discussions in the Wikidata mailing list. In chronological order from the most recent: [15],[16], [17], [18], [19], and [20].

Hence, the following requirements are essential for the tool sustainability:

  1. to understand the back-end code,[21] written in C++;
  2. to understand the front-end code,[22] written in JavaScript;
  3. access to the WMF Labs machine to deploy the back-end;[23]
  4. a Wikidata administrator to deploy the front-end;[24]
  5. centralized and exhaustive documentation.

As part of StrepHit's initial goals,[25] the grantees have already worked on:

  • requirement #1, as demonstrated by the merged pull requests;[26][27][28][29][30]
  • requirement #3, since the project leader has been granted access to the machine.[31]

The remaining ones still need to be taken in charge.

Collaboration with ContentMine

edit

It is worth to mention that ContentMine, an Open Data organization, has explicitly stated[32] its interest in a data release on Wikidata via the primary sources tool. Hence, we foresee to make the most of this opportunity under the umbrella of an IEG. Moreover, ContentMine is clearly a strategical partner, since it is currently proposing a project for Wikidata,[33] where the collaboration with StrepHit is specifically mentioned.[34][35]

Scope

edit

We first focus on a major restructuring of the primary sources tool. Then, we reserve a set of activities for the quality of StrepHit datasets, which are closely intertwined with the testing phase of the tool, as they represent its current data backbone (besides Freebase).

  • Primary sources tool:
    • take control over the codebase;
    • tackle usability issues;[36][37]
    • implement known feature requests;[38]
    • fix open issues.[39]
  • StrepHit:
    • interact with the community to improve the mapping between extracted facts and Wikidata properties;
    • improve the performance of the supervised extraction;
    • fix open technical issues.[40]

Work Package

edit

The activities stated above can be grouped into 4 high-level categories:

  1. end-user experience;
  2. technical requirements;
  3. data provider experience;
  4. StrepHit datasets.

They are broken down into the following critical tasks, prefixed with U, T, D, and S respectively. Those related to the primary sources tool stem from known user requests, and are referenced accordingly.

Primary Sources Tool

edit
ID Title Goal Month Effort
U1 Tool workflow Sketch the complete flow by integrating all the already collected requirements,[41] step back to the community with mock-ups, redesign if needed M1-M2 10%
U2 Filtering facility Allow users to curate items based on their interests[42][43] M3-M4 2.5%
U3 Primary Sources list Make this sub-tool usable[44][45] M3-M5 5%
U4 Effective curation flow All the components needed for curation should be shown in one single step[46] M3-M6 7.5%
U5 Data de-duplication Avoid showing duplicate statements[47][48][49][50] M3-M4 2.5%
U6 Community dissemination Promote the tool and engage its key stakeholders M1-M12 7.5%
T1 Codebase review Understand both the back-end (C++) and the front-end (JavaScript) code architectures M1-M2 2.5%
T2 Back-end redesign Decide with the community whether the C++ implementation is sustainable or it should be replaced with a more adopted language M2 2.5%
T3 Front-end redesign Work with strategic partners to create a suitable front-end, based on U1 outcomes M3-M8 10%
T4 Sources whitelist Build a list of reliable sources[51] M6 2.5%
T5 Back-end documentation Improve technical documentation of the back-end module[52][53] M6-M12 2.5%
T6 Front-end documentation Improve technical documentation of the front-end module[54][55] M6-M12 2.5%
T7 Developer community Engage volunteer contributors M3-M12 10%
D1 Data provider workflow Work with ContentMine to consolidate the flow for third-party data providers M1-M6 5%
D2 Data release tutorial Write an exhaustive walkthrough for third-party data releases M7 2.5%

StrepHit

edit
ID Title Goal Month Effort
S1 Lexical database Coordinate with the community to improve the mapping of extracted facts into Wikidata properties M7-M12 10%
S2 Training set Keep adding training data via crowdsourced annotation jobs M7-M12 2.5%
S3 Classification performance Train and test the supervised classifiers to improve their precision M7-M12 2.5%
S4 Datasets for direct inclusion Isolate with the community the best StrepHit datasets to be uploaded through a bot M10-M12 5%
S5 Unresolved entities dataset Decide with the community whether to mint new Wikidata Items based on the unresolved entities[56] M10-M12 5%

N.B.: overlap between tasks (in terms of timing) are needed for iterative planning.

Budget

edit

The total amount requested is 50,630 USD.

Budget Breakdown

edit
Item Description Commitment Cost
Project Leader Responsible for the whole work package Full time (40 hrs/week), 12 PM(1) 41,379 €
Front-end Developer Assistant for the user interface implementation Part time (20 hrs/week), 6 PM(2) Provided by the hosting research center
Wikimedia Developer Summit 2017 Travel, board & lodging Una tantum 1,196 €
Wikimania 2017 Registration, travel, board & lodging Una tantum 2,079 €
Training Set Crowdsourced job payment for the annotation of training sentences Una tantum 550 €
Total 45,204

(1) Person Months
(2) The need for an assistant was also suggested by a community member[57]

The item costs are computed as follows.

  • The project leader's gross salary is estimated upon the hosting research center (i.e., Fondazione Bruno Kessler) standard salaries,[58] namely "Quadro direttivo" (project manager). The salary complies both with (a) the provincial collective agreement as per the provincial law n. 14,[59] and with (b) the national collective agreement as per the national law n. 240.[60] These laws respectively regulate research and innovation activities in the area where the research center is located (i.e., Trentino, Italy), and at a national level. More specifically, the position is set to a gross labor rate of 21.55 € per hour. The rate is in line with other national research institutions, such as the universities of Trieste,[61] Firenze,[62] and Roma;[63]
  • the front-end developer will be either chosen from available human resources or directly hired by the hosting research center, thus minimizing the impact on the requested budget;
  • Wikimedia Developer Summit 2017 has been suggested by LZia (WMF),[64] and sums the sub-items below:
    • 871 €, i.e., the average cost of 1 round-trip flight from Milan to San Francisco, retrieved at Skyscanner;[65]
    • 325 €, i.e., the average cost of 1 single room for 5 nights in a 3-star hotel, retrieved at Trivago;[66]
    • registration is free.[67]
  • Wikimania 2017 sums the sub-items below:
    • 1,002 €, i.e., the average cost of 1 round-trip flight from Milan to Montréal, retrieved at Skyscanner;[68]
    • 877 €, i.e., the average cost of 1 single room for 5 nights in a 3-star hotel, retrieved at Trivago;[69]
    • 200 €, i.e., the Wikimania registration cost for 3 conference days, based on Wikimania 2016;[70]
  • the training set construction job has an average cost of 4.35 ¢ per annotated sentence: the item is an estimate to reach a threshold of 500 sentences for each of the current 49 frames.

The total budget expressed in Euros is approximately equivalent to the requested amount in U.S. Dollars, given an exchange rate of 1.12 USD = 1 €.

N.B.: Fondazione Bruno Kessler will be physically hosting the human resources, but it will not be directly involved into this proposal with respect to the requested budget: the project leader will serve as the main grantee and will appropriately allocate the funding.

Strategic Partners

edit

We target the following list of partners to maximize favorable outcomes of the work package.

Partner Tasks
ContentMine U1, U6, T2, T3(3)
Wikidata Developers team U4, T2, T3
WMF Design Research team U1, T3
WMF Discovery (User Experience) team U1, T3

(3) These tasks respectively match C2, T6, T7, C5 in Grants:Project/WikiFactMine#Activities

Measures of Success

edit

Local Metrics

edit

The self-sustainability of the primary sources tool codebase should have the highest priority. Hence, the most important measure of success is the number of technical contributors, which should reach 4 people.

All the numerical success metrics apply to the primary sources tool and can be assessed via the status page.

  1. 25,000 new curated statements, computed as the sum of approvals and rejects. Currently, they are 231,707 over a total of 16,547,445;(4)
  2. full engagement of 2 Open Data organizations, namely ContentMine and Openpolis, in the form of dataset releases via the primary sources tool;
  3. 30 new primary sources tool users may represent a stretch goal, as suggested by LZia (WMF);[71]

From a qualitative perspective, we will keep collecting feedback through the currently open request for comment.

(4) the displayed numbers were looked up on July 19th 2016.

Global Metrics

edit

The following global metrics are inherently connected to the local ones:

  1. the number of articles created or improved in Wikimedia projects measure will be based on Wikidata statements;
  2. the number of active editors involved can be determined from the primary sources tool user log database;
  3. the number of individuals involved will include an heterogeneous set of people, ranging from dissemination activities attendees, to hackathon participants, all the way to software contributors.

References

edit
  1. Grants:IEG/StrepHit:_Wikidata_Statements_Validation_via_References
  2. Grants:IEG/StrepHit:_Wikidata_Statements_Validation_via_References/Final#Summary
  3. wikidata:Wikidata:Primary_sources_tool
  4. https://tools.wmflabs.org/wikidata-primary-sources/status claims only a few active users (cf. the topusers field), out of 427 total users (cf. the total_users field)
  5. wikidata:Wikidata:Requests_for_comment/Semi-automatic_Addition_of_References_to_Wikidata_Statements
  6. Pellissier Tanon, T., Vrandečić, D., Schaffert, S., Steiner, T., and Pintscher, L. (2016, April). From Freebase to Wikidata: The Great Migration. In Proceedings of the 25th International Conference on World Wide Web (pp. 1419-1428). ACM (2016)
  7. wikidata:Wikidata:Data_donation#3._Work_with_the_Wikidata_community_to_import_the_data
  8. Grants:IEG/StrepHit:_Wikidata_Statements_Validation_via_References#Project_Goals
  9. Grants:IEG/StrepHit:_Wikidata_Statements_Validation_via_References#Measures_of_Success
  10. wikidata:Wikidata:Requests_for_comment/Semi-automatic_Addition_of_References_to_Wikidata_Statements#UI
  11. wikidata:Wikidata_talk:Primary_sources_tool#Too_slow
  12. https://lists.wikimedia.org/pipermail/wikidata/2016-February/008316.html
  13. https://lists.wikimedia.org/pipermail/wikidata/2016-May/008822.html
  14. https://github.com/Wikidata/primarysources/issues/106#issuecomment-222612530
  15. https://lists.wikimedia.org/pipermail/wikidata/2016-July/thread.html#9084
  16. https://lists.wikimedia.org/pipermail/wikidata/2016-June/thread.html#8956
  17. https://lists.wikimedia.org/pipermail/wikidata/2016-May/008825.html
  18. https://lists.wikimedia.org/pipermail/wikidata/2016-May/thread.html#8814
  19. https://lists.wikimedia.org/pipermail/wikidata/2016-February/thread.html#8316
  20. https://lists.wikimedia.org/pipermail/wikidata/2016-February/008316.html
  21. https://github.com/google/primarysources/tree/master/backend
  22. https://github.com/google/primarysources/tree/master/frontend
  23. https://tools.wmflabs.org/wikidata-primary-sources
  24. https://github.com/google/primarysources/tree/master/frontend#deployment-on-wikidata
  25. Grants:IEG/StrepHit:_Wikidata_Statements_Validation_via_References#Project_Goals
  26. https://github.com/google/primarysources/pull/86
  27. https://github.com/google/primarysources/pull/87
  28. https://github.com/google/primarysources/pull/97
  29. https://github.com/google/primarysources/pull/100
  30. https://github.com/google/primarysources/pull/102
  31. cf. the list of maintainers for wikidata-primary-sources at https://tools.wmflabs.org/?list
  32. Grants_talk:IEG/StrepHit:_Wikidata_Statements_Validation_via_References#Support_from_ContentMine
  33. Grants:Project/WikiFactMine
  34. Grants:Project/WikiFactMine#Technical_Activities
  35. Grants:Project/WikiFactMine#Community_Engagement_Activities
  36. wikidata:Wikidata:Requests_for_comment/Semi-automatic_Addition_of_References_to_Wikidata_Statements#User_interface_feature_requests
  37. wikidata:Wikidata_talk:Primary_sources_tool
  38. https://github.com/Wikidata/primarysources/issues?q=is%3Aopen+is%3Aissue+label%3A%22open+task%22
  39. https://github.com/Wikidata/primarysources/issues
  40. https://github.com/Wikidata/StrepHit/issues
  41. wikidata:Wikidata:Requests_for_comment/Semi-automatic_Addition_of_References_to_Wikidata_Statements
  42. wikidata:Wikidata_talk:Primary_sources_tool/Archive/2015#give_us_filtering_capabilities
  43. wikidata:Wikidata:Requests_for_comment/Semi-automatic_Addition_of_References_to_Wikidata_Statements#Filter_for_Items
  44. wikidata:Wikidata_talk:Primary_sources_tool#Primary_Sources_list
  45. phabricator:T148165
  46. phabricator:T148163
  47. wikidata:Wikidata:Requests_for_comment/Semi-automatic_Addition_of_References_to_Wikidata_Statements#Dataset
  48. https://lists.wikimedia.org/pipermail/wikidata/2016-May/008814.html
  49. phabricator:T148175
  50. phabricator:T148168
  51. wikidata:Wikidata:Requests_for_comment/Semi-automatic_Addition_of_References_to_Wikidata_Statements#A_whitelist_for_sources
  52. https://github.com/Wikidata/primarysources/tree/master/backend
  53. phabricator:T148174
  54. https://github.com/Wikidata/primarysources/tree/master/frontend
  55. phabricator:T148174
  56. Grants:IEG/StrepHit:_Wikidata_Statements_Validation_via_References/Final#Bonus_Outcomes
  57. Grants_talk:IEG/StrepHit:_Wikidata_Statements_Validation_via_References/Renewal#Hiring_an_additional_person
  58. http://hr.fbk.eu/sites/hr.fbk.eu/files/ccpl_28set07_aggiornato_2009.pdf - page 82, Tabella B
  59. http://www.consiglio.provincia.tn.it/doc/clex_26185.pdf
  60. http://www.camera.it/parlam/leggi/10240l.htm
  61. https://www.units.it/intra/personale/tabelle_stipendiali/?file=tab.php&ruolo=RD - ninth item of the central dropdown menu
  62. http://www.unifi.it/upload/sub/money/2016/td_tabelle_costi_tipologia_B.xlsx
  63. https://web.uniroma2.it/modules.php?name=Content&action=showattach&attach_id=15798
  64. Grants_talk:IEG/StrepHit:_Wikidata_Statements_Validation_via_References/Renewal#Feedback_.233
  65. https://www.skyscanner.com/transport/flights/lin/sfo/170107/170112/airfares-from-milan-linate-to-san-francisco-international-in-january-2017.html?adults=1&children=0&adultsv2=1&childrenv2=&infants=0&cabinclass=economy&rtn=1&preferdirects=false&outboundaltsenabled=false&inboundaltsenabled=false&ref=home#results
  66. http://www.trivago.com/go/c5jchsk
  67. mw:Wikimedia_Developer_Summit#Join_us
  68. https://www.skyscanner.com/transport/flights/lin/yul/170810/170815/airfares-from-milan-linate-to-montreal-pierre-elliott-trudeau-in-august-2017.html?adults=1&children=0&adultsv2=1&childrenv2=&infants=0&cabinclass=economy&rtn=1&preferdirects=false&outboundaltsenabled=false&inboundaltsenabled=false&ref=home#results
  69. http://www.trivago.com/go/c5gmvgc
  70. https://www.eventbrite.com/e/wikimania-2016-esino-lario-conference-only-tickets-22711535815#tickets
  71. Grants_talk:IEG/StrepHit:_Wikidata_Statements_Validation_via_References/Renewal#Feedback_.233

Community Discussion

edit

Notification

edit

The following list of links includes both notifications and discussions, and is sorted in descending order of relevance.

Statements of Support

edit

Endorsements

edit

Do you think this project should be continued for another 6 months? Please add your name and comments here. Other feedback, questions or concerns from community members are also highly valued, but please post them on the talk page of this proposal.

See also Grants:Project/StrepHit_IEG_renewal#Endorsements and Grants:IEG/StrepHit:_Wikidata_Statements_Validation_via_References#Endorsements.

  • The work done as part of this IEG has been great for us and I'd like to recommend it for an extension to further boost this important area of Wikidata. --Lydia Pintscher (WMDE) (talk) 15:36, 25 July 2016 (UTC)
  • The StrepHit project should be continued in order to improve the Primary Sources Tool. --tomayac (talk)
  • There is a strong need for enhancements to usability of the current primary sources tool. As part of the prior Freebase Experts community, I want to give my recommendation for further IEG extension. Thadguidry (talk) 13:14, 26 July 2016 (UTC)
  • Magnus Manske (talk) 13:35, 26 July 2016 (UTC)
  • When considering the long (or short!) term, adding sources manually is untenable, and unscalable - as evidenced by the current status. I urge wikimedia/data to compromise on its sources policy, or invest in an automated solution like this. The 'google-the-fact' plan is of questionable value, and is a considerable drain on the long-term viability of this project. Spencerk (talk) 13:53, 26 July 2016 (UTC)
  • It would be fantastic if the Primary Sources Tool could be updated. The tool is vastly underutilized, and I think it is a great way to increase the quality and quantity of references in Wikidata, and thus improve the reliability of Wikidata. --denny (talk) 16:59, 26 July 2016 (UTC)
  • This work is not just critical for Wikidata but also more broadly for the various initiatives to improve the quality, volume and machine-readability of sources we started under the WikiCite umbrella. --Dario (WMF) (talk) 17:36, 26 July 2016 (UTC)
  • I too would like to see a renewed commitment and funding of this work. In my opinion we not only gain useful tools but also more experience in this most interesting area of computing. Especially usability and accessibility seem important,so more people can participate in these new developments. --Tobias1984 (talk) 21:17, 26 July 2016 (UTC)
  • A very useful and helpful tool and an updated version would be great! --Todrobbins (talk) 22:13, 26 July 2016 (UTC)
  • I would like to see more tools being developed to help Wikidata end-users make the best possible use of their time (for those who have little time to spare) and user-friendly (for those who have plenty of time to spare, like my Wikitherapy participants). As my new plans include "breaking into" Wikidata especially in the medical field, I endorse this renewal.--Saintfevrier (talk) 04:37, 27 July 2016 (UTC)
  • This tool is very important for the future of Wikidata and is not currently used as much as it should because of usability issues. Tpt (talk) 07:40, 27 July 2016 (UTC)
  • I wasn't able to use this tool in a successful way, but it's very interesting. I support the endorsement so it could be improved and implemented like a normal beta feature.Coentor (talk) 18:13, 27 July 2016 (UTC)
  • I strongly endorse this project. I think that improvements to the Primary Sources Tool will facilitate the addition of new data into Wikidata - including the data I am currently working with. Epantaleo (talk) 13:39, 28 July 2016 (UTC)
  • Yiyi (talk) 14:52, 28 July 2016 (UTC)
  • Mark Hurd (talk) 15:06, 28 July 2016 (UTC)
  • Scott_WorldUnivAndSch (talk) 21:26, 28 July 2016 (UTC)
  • I have tried the tool and I have found it useful! --Jaqen (talk) 11:03, 29 July 2016 (UTC)
  • I'm glad to endorse this project and I hope more tools like this are developed and supported --Steko (talk) 13:50, 29 July 2016 (UTC)
  • As I can see, this tool improves the abilities of wikidata significantly. Please go on! Lantus (talk) 11:44, 31 July 2016 (UTC)
  • Epìdosis 18:05, 31 July 2016 (UTC)
  • Very useful --Davidpar (talk) 23:36, 31 July 2016 (UTC)
  • Pretty much what everyone else said; improving the primary sources tool's usability is likely the most effective way to increase the user base of the tool itself and, ultimately, of Wikidata --Edorigatti (talk) 10.14, 1 August 2016 (UTC)
  • The tool is a big help. It's not perfect and some of the usability issues mentioned by others above need to be addressed but I think it's a great start and does a lot of good. Reguyla 138.162.8.58 17:02, 2 August 2016 (UTC)
  • Love to see the result. I hope that'll improve Wikidata a lot.-Biyanto Rebin (WMID) (talk) 05:05, 3 August 2016 (UTC)
  • I find the project very interesting. It would be even more useful if it helped the insertion of new data in an automated fashion. Nvitucci (talk) 09:06, 3 August 2016 (UTC)
  • I strongly endorse the continuation of this project and its new focus on improving the primary sources tool. This could be one of the most important gateways for gathering a validating new data and references for wikidata --i9606 (talk)
  • I'm particularly keen to endorse this project and its work on improving primary sources tool because it's the best way to curate the addition of machine generated data, both from StrepHit and other textmining pipelines Bomarrow1 (talk) 10:53, 4 August 2016 (UTC)
  • Needed work. TomT0m (talk) 13:02, 4 August 2016 (UTC)
  • I strongly support this effort. - PKM (talk) 17:13, 4 August 2016 (UTC)
  • --Ermanon (talk) 23:50, 4 August 2016 (UTC)
  • --Melderick (talk) 10:50, 5 August 2016 (UTC)
  • Lymantria (talk) 12:24, 5 August 2016 (UTC)
  • Primary Sources Tool is important and need to be improved.
    Example of bugs :
    The link to geohack of geographical coord. is often disabled when I use Primary Sources Tool ;
    The button "Reject" or "Approve" often interchange their places (The button Reject can be at the place of button Approve...)) ;
    The Primary Sources statement list need a automatically refresh when I use it ;
    Lot of result are exactly redundant with wikidata's data and need to be automatically reject/treat ;
    When I add a reference with PST it doesn't add it to the statement but create a new statement ;
    (New Bug) The statements is sometime display empty, and force to refresh the page if I want to use PST ;
    etc... Nouill (talk) 16:55, 6 August 2016 (UTC)
  • The Primary Sources Tool has a lot of bugs at the moment, this will help a lot! Sjoerd de Bruin (talk) 09:28, 7 August 2016 (UTC)
  • Very valuable tool to improve the reliability of Wikidata; such improvements definitely deserve the effort. -- Laddo (talk) 21:05, 7 August 2016 (UTC)
  • Happy to endorse the continued improvement to this tool: definitely going to be an important part of developing Wikidata, Sadads (talk · contribs)
  • H4stings (talk) 06:11, 9 August 2016 (UTC)
  • Been a pleasure to use so far. Keep going! Telaneo (User talk page) 22:10, 9 August 2016 (UTC)
  • Of course! -- Bodhisattwa (talk) 18:34, 10 August 2016 (UTC)
  • I strongly endorse the continued use and development of StrepHit for the improvement of Wikidata and other Wikimedia Projects --- Nimbosa (talk) 13:42, 14 August 2016 (UTC)
  • Just started playing with the tool; I think there is a lot of opportunity here and I would like to see a good thing get better! harej (talk) 00:24, 18 August 2016 (UTC)
  • Very useful Afernand74 (talk) 20:39, 18 August 2016 (UTC)
  • The StrepHit project has very worthy goals and a good track record but still needs a lot of work in terms of usability and integration with existing and emerging workflows, at least for the areas that I am most active in, i.e. sciences and medicine. This renewal has the potential to help significantly to move things forward in this regard. -- Daniel Mietchen (talk) 01:00, 19 August 2016 (UTC)
  • The project has lot of potential but needs some more work -- usability to be improved, integration with other pipelines and wrokflows, as many others pointed out. -- Giulio Petrucci 22 August 2016
  • --Crazy1880 (talk) 17:46, 22 August 2016 (UTC)
  • YES, this project should be continued for another 6 months. Good work! Projekt ANA (talk) 26 August 2016
  • +1 --Kippelboy (talk) 10:18, 29 August 2016 (UTC)
  • --Uomovariabile (talk) 17:40, 31 August 2016 (UTC)
  • Also yes ·addshore· talk to me! 12:12, 2 September 2016 (UTC)
  • It's a helpful tool but it still needs a lot of work to reach it's promise. Founding it seems vital. ChristianKl (talk) 09:43, 12 September 2016 (UTC)
  • This tool is valuable, but it needs improvement. --Molarus (talk) 19:53, 15 September 2016 (UTC)
  • Yes, this tool is very useful, even if it needs some work. Mateon1 (talk) 17:57, 17 September 2016 (UTC)
  • Conny (talk) 19:27, 17 September 2016 (UTC).
  • Yes, the tool is useful and its improvement should definitely be funded. Mushroom (talk) 09:42, 19 September 2016 (UTC)
  • --CristianNX (talk) 10:01, 14 October 2016 (UTC)
  • Pajn (talk) 19:39, 21 October 2016 (UTC)
  • Sadads (talk) 15:02, 20 October 2017 (UTC)